niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #511
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestScriptTokenEmptyCommand119--- PASS: TestScriptTokenEmptyCommand (0.00s)120=== CONT TestStreamPushReportsEveryPath121=== CONT TestScriptTokenScriptFails122=== CONT TestStreamPushAnswersEveryLineWhenCanceled123=== CONT TestScriptTokenBadJSON124=== CONT TestScriptTokenEmptyToken125=== CONT TestScriptTokenCachesUntilRefresh126--- PASS: TestStreamPushReportsEveryPath (0.00s)127=== CONT TestShellSplitErrors128=== CONT TestDoWithRetry_BodyReplayedViaGetBody1292000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2130=== CONT TestScriptTokenNoExpiryRerunsEveryCall131=== CONT TestClientSignaturesByStorePath132=== CONT TestFileTokenEmpty133=== CONT TestStreamPushReportsSkippedPaths134=== CONT TestFileTokenMissing135=== CONT TestFileTokenReadsAndCaches136=== CONT TestStreamPushReportsSignatures137=== CONT TestStaticToken138=== CONT TestStreamPushRequestLine139=== CONT TestSetClientTLSErrors140=== CONT TestStreamPushGivesUpOnDeadServer141=== CONT TestSetClientTLSDoesNotMutateDefaultTransport1422026/11/12 17:02:16 ERROR Upload failed error=boom count=11432026/11/12 17:02:16 ERROR Upload failed error=boom count=11442026/11/12 17:02:16 ERROR Upload failed error="connection refused" count=201452026/11/12 17:02:16 ERROR Server seems unavailable, giving up on batch untried=171462026/11/12 17:02:16 WARN Rate limiter enabled after throttle name=server-test rate=51472026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:37809148=== CONT TestStreamPushIsolatesFailures149=== CONT TestSetClientTLS150=== CONT TestRegisterUploadedObject_OutlivesPush1512026/11/12 17:02:16 ERROR Upload failed error="bad path" count=3152=== CONT TestStreamPushBatchesUnderLoad1532026/11/12 17:02:16 WARN Rate limiter backed off name=server-test rate=5154=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false1552026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:37809156=== CONT TestDumpPathMatchesNix157--- PASS: TestShellSplitErrors (0.00s)158=== CONT TestShellSplit159=== CONT TestRetryTimeoutOutlastsMaxRetries160=== CONT TestDoWithRetry_FinalResponseBodyReadable161=== CONT TestUploadMultipart_FailedProducerIsNotCompleted162=== CONT TestPathInfoCACompatibility163=== CONT TestUploadMultipart_PartsInParallel164=== CONT TestResolveStorePath165=== CONT TestDumpPathWriterError166=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer167=== CONT TestDumpPathSingleFile168=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part169=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess1702026/11/12 17:02:16 WARN Rate limiter enabled after throttle name=server-test rate=5171=== RUN TestPathInfoCACompatibility/null_ca_field172=== CONT TestUploadMultipart_ReturnsAfterDroppedBody1732026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:336771742026/11/12 17:02:16 WARN Rate limiter enabled after throttle name=server-test rate=5175=== CONT TestStreamPushStopsWhenCanceled176=== PAUSE TestPathInfoCACompatibility/null_ca_field177=== RUN TestPathInfoCACompatibility/old_string_format_-_text178=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false179--- PASS: TestFileTokenMissing (0.00s)180=== CONT TestPartSizeForNAR1812026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:33611182=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true183=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true184=== RUN TestPartSizeForNAR/zero_stays_at_minimum185=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text186=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum187=== CONT TestPathInfoHashCompatibility1882026/11/12 17:02:16 WARN Rate limiter backed off name=server-test rate=5189=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part190=== CONT TestParsePathInfoJSONMultiplePaths191=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths192=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths193=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary194=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary195=== CONT TestParsePathInfoJSON196=== RUN TestParsePathInfoJSON/Nix_format197=== CONT TestRateLimiterFeedback198=== RUN TestRateLimiterFeedback/429_enables_limiter199=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun200=== RUN TestSetClientTLSErrors/missing_cert_file201=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive202=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive203=== RUN TestPartSizeForNAR/small_stays_at_minimum204=== PAUSE TestPartSizeForNAR/small_stays_at_minimum205=== RUN TestPathInfoCACompatibility/new_structured_format_-_text206=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text207=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method208=== CONT TestFilterOversizedClosures209=== RUN TestFilterOversizedClosures/no_limit_keeps_everything210=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything211=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)212--- PASS: TestScriptTokenScriptFails (0.00s)213--- PASS: TestStaticToken (0.00s)214=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths215=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)216=== CONT TestGetStorePathHash217=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon218=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon219=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI220=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI221=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512222=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512223=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths224=== CONT TestRegisterUploadedObjectReusesConnections2252026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:33611226=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry2272026/11/12 17:02:16 INFO Garbage collection started228=== CONT TestEncodeNixBase32WithRealHash229=== CONT TestEncodeNixBase32230=== RUN TestGetStorePathHash/valid_store_path231=== PAUSE TestGetStorePathHash/valid_store_path232=== RUN TestGetStorePathHash/basename_without_hyphen_should_error233=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error234=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error235=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error236=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error2372026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:33611238=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error239=== PAUSE TestParsePathInfoJSON/Nix_format240=== RUN TestParsePathInfoJSON/Lix_format241=== PAUSE TestParsePathInfoJSON/Lix_format242=== RUN TestParsePathInfoJSON/empty_input243=== PAUSE TestParsePathInfoJSON/empty_input244=== RUN TestParsePathInfoJSON/whitespace_only245=== PAUSE TestParsePathInfoJSON/whitespace_only246=== RUN TestParsePathInfoJSON/invalid_JSON247=== PAUSE TestParsePathInfoJSON/invalid_JSON248=== PAUSE TestRateLimiterFeedback/429_enables_limiter249=== RUN TestRateLimiterFeedback/503_enables_limiter250=== PAUSE TestRateLimiterFeedback/503_enables_limiter251=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter2522026/11/12 17:02:16 WARN Rate limiter enabled after throttle name=s3-test rate=5253=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter2542026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36077255=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter256=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter257=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false258=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true259=== CONT TestCaseHackSuffix260=== CONT TestConvertHashToNix32261=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum262=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method263=== RUN TestSetClientTLS/rejects_connection_without_client_cert264=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped265--- PASS: TestClientSignaturesByStorePath (0.00s)266=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica267=== RUN TestEncodeNixBase32/test_string_hash268=== PAUSE TestEncodeNixBase32/test_string_hash269=== RUN TestEncodeNixBase32/empty_input270=== PAUSE TestEncodeNixBase32/empty_input271=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum272=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts273=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts274=== RUN TestPartSizeForNAR/1_TiB275=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert276=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)277=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths278--- PASS: TestStreamPushReportsSkippedPaths (0.00s)279--- PASS: TestStreamPushReportsSignatures (0.00s)280--- PASS: TestScriptTokenBadJSON (0.00s)281--- PASS: TestScriptTokenEmptyToken (0.00s)282--- PASS: TestFileTokenReadsAndCaches (0.00s)283--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)284--- PASS: TestDoServerRequestAttachesToken (0.01s)285--- PASS: TestStreamPushIsolatesFailures (0.00s)286--- PASS: TestShellSplit (0.00s)287--- PASS: TestFileTokenEmpty (0.00s)288--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.00s)289--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)290--- PASS: TestStreamPushStopsWhenCanceled (0.00s)291--- PASS: TestResolveStorePath (0.00s)292--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.03s)293--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.03s)294--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.04s)295--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.03s)296--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)297--- PASS: TestEncodeNixBase32WithRealHash (0.00s)298--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)299=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths3002026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:33611301--- PASS: TestScriptTokenCachesUntilRefresh (0.04s)302=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary303=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon3042026/11/12 17:02:16 INFO Garbage collection started305=== CONT TestParsePathInfoJSON/Nix_format306=== CONT TestGetStorePathHash/basename_without_hyphen_should_error307=== CONT TestRateLimiterFeedback/429_enables_limiter308=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512309=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped310=== RUN TestFilterOversizedClosures/all_closures_skipped311=== PAUSE TestFilterOversizedClosures/all_closures_skipped312=== CONT TestParsePathInfoJSON/Lix_format313=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter3142026/11/12 17:02:16 WARN Rate limiter enabled after throttle name=server-test rate=53152026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:45607316=== CONT TestGetStorePathHash/valid_store_path317=== CONT TestParsePathInfoJSON/invalid_JSON3182026/11/12 17:02:16 WARN Rate limiter backed off name=server-test rate=5319=== CONT TestParsePathInfoJSON/whitespace_only320=== CONT TestParsePathInfoJSON/empty_input321=== CONT TestEncodeNixBase32/test_string_hash322=== CONT TestPathInfoCACompatibility/null_ca_field323=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method324=== CONT TestPathInfoCACompatibility/new_structured_format_-_text325=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive326=== CONT TestPathInfoCACompatibility/old_string_format_-_text327=== CONT TestFilterOversizedClosures/no_limit_keeps_everything328=== CONT TestFilterOversizedClosures/all_closures_skipped3292026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:336113302026/11/12 17:02:16 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50331=== CONT TestRateLimiterFeedback/503_enables_limiter332=== CONT TestEncodeNixBase32/empty_input333=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error334=== PAUSE TestPartSizeForNAR/1_TiB335=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3362026/11/12 17:02:16 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=03372026/11/12 17:02:16 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000338=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter3392026/11/12 17:02:16 WARN Rate limiter enabled after throttle name=server-test rate=53402026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:38473341=== RUN TestConvertHashToNix32/SRI_format_to_Nix32342=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32343=== RUN TestConvertHashToNix32/already_Nix32_format344=== PAUSE TestConvertHashToNix32/already_Nix32_format345=== RUN TestConvertHashToNix32/invalid_format346=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part347=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA348=== PAUSE TestConvertHashToNix32/invalid_format349=== CONT TestConvertHashToNix32/SRI_format_to_Nix32350=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI351--- PASS: TestStreamPushBatchesUnderLoad (0.03s)352 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)353 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)354--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.04s)3552026/11/12 17:02:16 WARN Rate limiter backed off name=server-test rate=5356=== PAUSE TestSetClientTLSErrors/missing_cert_file357--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)358 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)359 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)360=== RUN TestSetClientTLSErrors/missing_key_file361=== PAUSE TestSetClientTLSErrors/missing_key_file362=== RUN TestSetClientTLSErrors/missing_ca_file363=== PAUSE TestSetClientTLSErrors/missing_ca_file364=== RUN TestSetClientTLSErrors/invalid_ca_file365=== PAUSE TestSetClientTLSErrors/invalid_ca_file366=== CONT TestSetClientTLSErrors/missing_cert_file367=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA368=== RUN TestSetClientTLS/preserves_debug_logging_transport369=== CONT TestSetClientTLSErrors/missing_key_file370--- PASS: TestParsePathInfoJSON (0.00s)371 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)372 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)373 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)374 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)375 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)376--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.03s)377 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)378 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)379=== CONT TestSetClientTLSErrors/invalid_ca_file380=== CONT TestSetClientTLSErrors/missing_ca_file3812026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:336113822026/11/12 17:02:16 WARN Garbage collection finished on another replica. Its statistics are not available here383=== RUN TestPartSizeForNAR/5_TiB_S3_max_object384=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object385=== RUN TestPartSizeForNAR/capped_at_5_GiB386=== PAUSE TestPartSizeForNAR/capped_at_5_GiB387=== CONT TestPartSizeForNAR/zero_stays_at_minimum388--- PASS: TestPathInfoCACompatibility (0.03s)389 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)390 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)391 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)392 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)393 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)394=== CONT TestPartSizeForNAR/capped_at_5_GiB395=== CONT TestPartSizeForNAR/5_TiB_S3_max_object396=== CONT TestPartSizeForNAR/1_TiB397=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts398=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum399=== CONT TestPartSizeForNAR/small_stays_at_minimum400=== PAUSE TestSetClientTLS/preserves_debug_logging_transport401=== CONT TestSetClientTLS/rejects_connection_without_client_cert402=== CONT TestConvertHashToNix32/invalid_format403=== CONT TestSetClientTLS/preserves_debug_logging_transport404=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error405=== CONT TestConvertHashToNix32/already_Nix32_format406--- PASS: TestFilterOversizedClosures (0.00s)407 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)408 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)409 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)410=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA411--- PASS: TestConvertHashToNix32 (0.00s)412 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)413 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)414 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)415--- PASS: TestPathInfoHashCompatibility (0.00s)416 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)417 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)418 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)419 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)420--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.00s)421--- PASS: TestPartSizeForNAR (0.01s)422 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)423 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)424 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)425 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)426 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)427 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)428 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)429--- PASS: TestGetStorePathHash (0.00s)430 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)431 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)432 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)433 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)434--- PASS: TestEncodeNixBase32 (0.00s)435 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)436 --- PASS: TestEncodeNixBase32/empty_input (0.00s)437--- PASS: TestRateLimiterFeedback (0.00s)438 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)439 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)440 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)441 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)4422026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:33611443--- PASS: TestSetClientTLSErrors (0.04s)444 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)445 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)446 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)447 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)4482026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:336114492026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:336114502026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:336114512026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:336114522026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:336114532026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:336114542026/11/12 17:02:16 http: TLS handshake error from 127.0.0.1:51908: remote error: tls: bad certificate4552026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:33611456--- PASS: TestSetClientTLS (0.04s)457 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)458 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)459 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4602026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:33611461--- PASS: TestRegisterUploadedObjectReusesConnections (0.03s)4622026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:33611463--- PASS: TestStreamPushRequestLine (0.06s)4642026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:33611465--- PASS: TestDumpPathSingleFile (0.07s)466--- PASS: TestCaseHackSuffix (0.04s)4672026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:336114682026/11/12 17:02:16 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:33611469--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.07s)470--- PASS: TestDumpPathWriterError (0.08s)4712026/11/12 17:02:16 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:37453/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:37453/api/uploads/complete4722026/11/12 17:02:16 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4732026/11/12 17:02:16 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:37453/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:37453/api/uploads/complete4742026/11/12 17:02:16 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4752026/11/12 17:02:16 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:37453/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:37453/api/uploads/complete4762026/11/12 17:02:16 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"477--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)478--- PASS: TestDumpPathMatchesNix (0.12s)479--- PASS: TestUploadMultipart_PartsInParallel (0.66s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres860297736/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres860297736/data -l logfile start509510/build/postgres860297736:5432 - no response5112026-11-12 17:02:18.024 UTC [129] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5122026-11-12 17:02:18.025 UTC [129] LOG: listening on Unix socket "/build/postgres860297736/.s.PGSQL.5432"5132026-11-12 17:02:18.030 UTC [136] LOG: database system was shut down at 2026-11-12 17:02:17 UTC5142026-11-12 17:02:18.034 UTC [129] LOG: database system is ready to accept connections515/build/postgres860297736:5432 - accepting connections516{"timestamp":"2026-11-12T17:02:18.223876538Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"14d7c574-5aec-4eb9-9402-1aa399f86012","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(203)"}517=== RUN TestAbortSucceeded518=== PAUSE TestAbortSucceeded519=== RUN TestService_AuthMiddleware520=== PAUSE TestService_AuthMiddleware521=== RUN TestService_AuthMiddleware_MTLSProxyHeader522=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader523=== RUN TestService_AuthMiddleware_MTLSBoundSubjects524=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects525=== RUN TestService_ReadAuthMiddleware526=== PAUSE TestService_ReadAuthMiddleware527=== RUN TestService_AuthMiddleware_OIDC528=== PAUSE TestService_AuthMiddleware_OIDC529=== RUN TestService_RequireScope_OIDC530=== PAUSE TestService_RequireScope_OIDC531=== RUN TestService_ReadScope_PublicByDefault532=== PAUSE TestService_ReadScope_PublicByDefault533=== RUN TestPinScopedPrincipal534=== PAUSE TestPinScopedPrincipal535=== RUN TestCacheConfigHandler536=== PAUSE TestCacheConfigHandler537=== RUN TestCacheStatsHandler538=== PAUSE TestCacheStatsHandler539=== RUN TestClientCADerivations540=== PAUSE TestClientCADerivations541=== RUN TestClientErrorHandling542=== PAUSE TestClientErrorHandling543=== RUN TestClientIntegration544=== PAUSE TestClientIntegration545=== RUN TestClientMultipleUploads546=== PAUSE TestClientMultipleUploads547=== RUN TestClientWithDependencies548=== PAUSE TestClientWithDependencies549=== RUN TestClientSharedPathCommittedMidPush550=== PAUSE TestClientSharedPathCommittedMidPush551=== RUN TestPinProtectsFromGC552=== PAUSE TestPinProtectsFromGC553=== RUN TestClientPushesUseOnePush554=== PAUSE TestClientPushesUseOnePush555=== RUN TestClientFallsBackToClosures556=== PAUSE TestClientFallsBackToClosures557=== RUN TestDeleteClosuresRacingPins558=== PAUSE TestDeleteClosuresRacingPins559=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock560=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock561=== RUN TestResolveDBConnectionString562=== PAUSE TestResolveDBConnectionString563=== RUN TestConnectWaitsForAPeerMigration564=== PAUSE TestConnectWaitsForAPeerMigration565=== RUN TestLeadElectsOneAndHandsOver566=== PAUSE TestLeadElectsOneAndHandsOver567=== RUN TestLeadIncumbentWinsAfterRestart5682026/11/12 17:02:18 INFO lead: acquired remote=192.0.2.1:12345692026/11/12 17:02:19 INFO lead: released remote=192.0.2.1:12345702026/11/12 17:02:19 INFO lead: acquired remote=192.0.2.1:12345712026/11/12 17:02:20 INFO lead: released remote=192.0.2.1:1234572--- PASS: TestLeadIncumbentWinsAfterRestart (1.90s)573=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent574=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent575=== RUN TestLeadIncumbentAnnouncesAtOnce576=== PAUSE TestLeadIncumbentAnnouncesAtOnce577=== RUN TestLeadEndsOnShutdown578=== PAUSE TestLeadEndsOnShutdown579=== RUN TestLeadEndsWhenItsConnectionHangs5802026/11/12 17:02:20 INFO lead: acquired remote=192.0.2.1:12345812026-11-12 17:02:21.741 UTC [383] FATAL: terminating connection due to administrator command5822026/11/12 17:02:21 INFO lead: acquired remote=192.0.2.1:12345832026/11/12 17:02:22 WARN lead: lock connection lost error="timeout: context deadline exceeded"5842026/11/12 17:02:22 INFO lead: released remote=192.0.2.1:12345852026/11/12 17:02:23 INFO lead: released remote=192.0.2.1:1234586--- PASS: TestLeadEndsWhenItsConnectionHangs (3.11s)587=== RUN TestGCAdvisoryLockBlocksConcurrentRun588=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun589=== RUN TestGCBugBareHashReferences590=== PAUSE TestGCBugBareHashReferences591=== RUN TestGCMetrics592=== PAUSE TestGCMetrics593=== RUN TestPushDedupSurvivesConcurrentGC594=== PAUSE TestPushDedupSurvivesConcurrentGC595=== RUN TestDeduplicatedObjectsRecordedAsPending596=== PAUSE TestDeduplicatedObjectsRecordedAsPending597=== RUN TestGCSweepSkipsPendingObjects598=== PAUSE TestGCSweepSkipsPendingObjects599=== RUN TestTombstonedObjectOfferedWithoutWaiting600=== PAUSE TestTombstonedObjectOfferedWithoutWaiting601=== RUN TestGCSweepDeliversEachKeyOnce602=== PAUSE TestGCSweepDeliversEachKeyOnce603=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection604=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection605=== RUN TestForceGCDuringPushOffersSweptObject606=== PAUSE TestForceGCDuringPushOffersSweptObject607=== RUN TestFailedS3DeleteKeepsTombstone608=== PAUSE TestFailedS3DeleteKeepsTombstone609=== RUN TestSweepSkipsRowsLockedByPush610=== PAUSE TestSweepSkipsRowsLockedByPush611=== RUN TestRegisterCompletedObjectNeedsPendingRow612=== PAUSE TestRegisterCompletedObjectNeedsPendingRow613=== RUN TestGetExistingObjectsLocksRows614=== PAUSE TestGetExistingObjectsLocksRows615=== RUN TestSweepRechecksPendingAfterLocking616=== PAUSE TestSweepRechecksPendingAfterLocking617=== RUN TestPushWaitsForSweepHoldingRow618=== PAUSE TestPushWaitsForSweepHoldingRow619=== RUN TestCommitRacingPendingCleanupKeepsObjects620=== PAUSE TestCommitRacingPendingCleanupKeepsObjects621=== RUN TestGCEndsOnShutdown622=== PAUSE TestGCEndsOnShutdown623=== RUN TestGCTaskStore_StartNew624=== PAUSE TestGCTaskStore_StartNew625=== RUN TestGCTaskStore_DeduplicateSameParams626=== PAUSE TestGCTaskStore_DeduplicateSameParams627=== RUN TestGCTaskStore_ConflictDifferentParams628=== PAUSE TestGCTaskStore_ConflictDifferentParams629=== RUN TestGCTaskStore_GetEmpty630=== PAUSE TestGCTaskStore_GetEmpty631=== RUN TestGCTaskStore_GetReturnsLatest632=== PAUSE TestGCTaskStore_GetReturnsLatest633=== RUN TestGCTaskStore_CompletedAllowsNewTask634=== PAUSE TestGCTaskStore_CompletedAllowsNewTask635=== RUN TestGCTaskStore_PhaseUpdates636=== PAUSE TestGCTaskStore_PhaseUpdates637=== RUN TestGCTaskStore_Fail638=== PAUSE TestGCTaskStore_Fail639=== RUN TestGracefulShutdownDrainsInflight640=== PAUSE TestGracefulShutdownDrainsInflight641=== RUN TestService_healthCheckHandler642=== PAUSE TestService_healthCheckHandler643=== RUN TestService_readinessHandler644=== PAUSE TestService_readinessHandler645=== RUN TestGenerateLandingPage646=== PAUSE TestGenerateLandingPage647=== RUN TestEnsureAbortUploadsRule648=== PAUSE TestEnsureAbortUploadsRule649=== RUN TestValidatePinReadNeedsWriteSubjects650=== PAUSE TestValidatePinReadNeedsWriteSubjects651=== RUN TestParseArgsReadPinFlag652--- PASS: TestParseArgsReadPinFlag (0.00s)653=== RUN TestCacheConfigHandlerMaxNarSize654=== PAUSE TestCacheConfigHandlerMaxNarSize655=== RUN TestCreatePendingClosureRejectsOversizedNAR656=== PAUSE TestCreatePendingClosureRejectsOversizedNAR657=== RUN TestNARDeduplicationMetadataUploadBug658=== PAUSE TestNARDeduplicationMetadataUploadBug659=== RUN TestMetricsInventory660=== PAUSE TestMetricsInventory661=== RUN TestMetricsCommit662=== PAUSE TestMetricsCommit663=== RUN TestService_NativeMTLS664=== PAUSE TestService_NativeMTLS665=== RUN TestServerTLSConfig666=== PAUSE TestServerTLSConfig667=== RUN TestMultipartCleanup668=== PAUSE TestMultipartCleanup669=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded670=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded671=== RUN TestPendingCleanupUsesOneCutoff672=== PAUSE TestPendingCleanupUsesOneCutoff673=== RUN TestObjectStatsTrigger674=== PAUSE TestObjectStatsTrigger675=== RUN TestObjectStatsUpdatedOncePerCommit676=== PAUSE TestObjectStatsUpdatedOncePerCommit677=== RUN TestOrphanedObjectsGC678=== PAUSE TestOrphanedObjectsGC679=== RUN TestOrphanedObjectsGCStressTest680=== PAUSE TestOrphanedObjectsGCStressTest681=== RUN TestPinObjectsReachable682=== PAUSE TestPinObjectsReachable683=== RUN TestPinObjectsGrace684=== PAUSE TestPinObjectsGrace685=== RUN TestPinObjectsTwoMoves686=== PAUSE TestPinObjectsTwoMoves687=== RUN TestPinObjectsSameTarget688=== PAUSE TestPinObjectsSameTarget689=== RUN TestPinObjectsSelfReference690=== PAUSE TestPinObjectsSelfReference691=== RUN TestPinObjectsOtherPin692=== PAUSE TestPinObjectsOtherPin693=== RUN TestPinObjectsDelete694=== PAUSE TestPinObjectsDelete695=== RUN TestPinObjectsBackfill696=== PAUSE TestPinObjectsBackfill697=== RUN TestPinObjectsSharedTarget698=== PAUSE TestPinObjectsSharedTarget699=== RUN TestGCDeletesExpiredPinObjects700=== PAUSE TestGCDeletesExpiredPinObjects701=== RUN TestGlobToLike702=== PAUSE TestGlobToLike703=== RUN TestPinScopedReads704=== PAUSE TestPinScopedReads705=== RUN TestPinScopedReadRedirect706=== PAUSE TestPinScopedReadRedirect707=== RUN TestFullReaderUnaffected708=== PAUSE TestFullReaderUnaffected709=== RUN TestPinScopedSubstringAttack710=== PAUSE TestPinScopedSubstringAttack711=== RUN TestParsePinReadRule712=== PAUSE TestParsePinReadRule713=== RUN TestCommonName714=== PAUSE TestCommonName715=== RUN TestCreatePin_ReservedPins716=== PAUSE TestCreatePin_ReservedPins717=== RUN TestPresentReportsOnlyClosureRoots718=== PAUSE TestPresentReportsOnlyClosureRoots719=== RUN TestPresentRefreshesOnlyStaleClosures720=== PAUSE TestPresentRefreshesOnlyStaleClosures721=== RUN TestPresentNotReportedWhileGCDeletesClosure722=== PAUSE TestPresentNotReportedWhileGCDeletesClosure723=== RUN TestParseSingleRange724=== PAUSE TestParseSingleRange725=== RUN TestProxyHeadersOnlyTrustedOnSocket7262026/11/12 17:02:23 INFO Starting HTTP server address=127.0.0.1:425717272026/11/12 17:02:23 INFO Starting HTTP server address=proxy.sock7282026/11/12 17:02:23 WARN mTLS auth: subject not in bound subjects subject="CN=someone"7292026/11/12 17:02:23 INFO Shutdown signal received, draining in-flight requests timeout=10s730--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)731=== RUN TestIsValidCachePath732=== PAUSE TestIsValidCachePath733=== RUN TestReadProxyNarinfo734=== PAUSE TestReadProxyNarinfo735=== RUN TestReadProxyNarinfoAlreadyDecompressed736=== PAUSE TestReadProxyNarinfoAlreadyDecompressed737=== RUN TestReadProxyContentNegotiation738=== PAUSE TestReadProxyContentNegotiation739=== RUN TestReadProxyNarStreaming740=== PAUSE TestReadProxyNarStreaming741=== RUN TestReadProxy404742=== PAUSE TestReadProxy404743=== RUN TestReadProxyInvalidPath744=== PAUSE TestReadProxyInvalidPath745=== RUN TestReadProxyHead746=== PAUSE TestReadProxyHead747=== RUN TestReadProxyOutlastsServerWriteTimeout748=== PAUSE TestReadProxyOutlastsServerWriteTimeout749=== RUN TestReadProxyConditionalGet750=== PAUSE TestReadProxyConditionalGet751=== RUN TestReadProxyRootRedirectsToIndexHTML752=== PAUSE TestReadProxyRootRedirectsToIndexHTML753=== RUN TestReadProxyDisabled754=== PAUSE TestReadProxyDisabled755=== RUN TestReadRedirectNar756=== PAUSE TestReadRedirectNar757=== RUN TestReadRedirectKeepsNarinfoProxied758=== PAUSE TestReadRedirectKeepsNarinfoProxied759=== RUN TestReadProxyRangeRequest760=== PAUSE TestReadProxyRangeRequest761=== RUN TestReadRedirectUsesPublicS3URL762=== PAUSE TestReadRedirectUsesPublicS3URL763=== RUN TestPush_OverlappingRootsStoreOneRowPerKey764=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey765=== RUN TestPush_CompleteCommitsEveryRoot766=== PAUSE TestPush_CompleteCommitsEveryRoot767=== RUN TestPush_RepeatedRootCommitsOnce768=== PAUSE TestPush_RepeatedRootCommitsOnce769=== RUN TestPush_CompleteWaitsForCleanupOfThePush770=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush771=== RUN TestPush_RootsAreLockedInByteOrder772=== PAUSE TestPush_RootsAreLockedInByteOrder773=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit774=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit775=== RUN TestPush_RejectsBadRequests776=== PAUSE TestPush_RejectsBadRequests777=== RUN TestPush_SignsNarinfosOfItsPendingObjects778=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects779=== RUN TestCommitCanBeRepeatedUntilCleanup780=== PAUSE TestCommitCanBeRepeatedUntilCleanup781=== RUN TestSharedNarBothUploadsComplete782=== PAUSE TestSharedNarBothUploadsComplete783=== RUN TestCompleteMultipartUpload_ErrorButObjectExists784=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists785=== RUN TestCompletedNarNotReofferedAcrossClosures786=== PAUSE TestCompletedNarNotReofferedAcrossClosures787=== RUN TestPresignedUploadRegisteredBeforeCommit788=== PAUSE TestPresignedUploadRegisteredBeforeCommit789=== RUN TestService_Rustfstest790=== PAUSE TestService_Rustfstest791=== RUN TestParseSize792=== PAUSE TestParseSize793=== RUN TestSkippedUploadsHandler794=== PAUSE TestSkippedUploadsHandler795=== RUN TestSystemdListenerNotActivated796--- PASS: TestSystemdListenerNotActivated (0.00s)797=== RUN TestWatchdogBeatsWhenHealthy798--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)799=== RUN TestWatchdogSkipsWhenUnhealthy8002026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8012026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8022026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8032026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8042026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8052026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8062026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8072026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8082026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"8092026/11/12 17:02:23 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"810--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)811=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle812=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle813=== RUN TestProxyWriteTimeout814=== PAUSE TestProxyWriteTimeout815=== RUN TestPendingClosureWriteTimeout816=== PAUSE TestPendingClosureWriteTimeout817=== RUN TestIsValidUploadKey818=== PAUSE TestIsValidUploadKey819=== RUN TestUploadHandlersRejectInvalidKeys820=== PAUSE TestUploadHandlersRejectInvalidKeys821=== RUN TestUploadHandlersRejectOversizedBody822=== PAUSE TestUploadHandlersRejectOversizedBody823=== RUN TestService_cleanupPendingClosuresHandler824=== PAUSE TestService_cleanupPendingClosuresHandler825=== RUN TestService_createPendingClosureHandler826=== PAUSE TestService_createPendingClosureHandler827=== RUN TestService_verifyS3Integrity828=== PAUSE TestService_verifyS3Integrity829=== RUN TestService_verifyS3ReuploadsStaleNarinfo830=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo831=== RUN TestCompleteMultipartUnregistered832=== PAUSE TestCompleteMultipartUnregistered833=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT834=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT835=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle836=== CONT TestService_Rustfstest837=== CONT TestService_cleanupPendingClosuresHandler838=== CONT TestCompletedNarNotReofferedAcrossClosures839=== CONT TestIsValidUploadKey840=== RUN TestIsValidUploadKey/narinfo841=== CONT TestSkippedUploadsHandler842=== PAUSE TestIsValidUploadKey/narinfo843=== RUN TestIsValidUploadKey/nar_zst844=== PAUSE TestIsValidUploadKey/nar_zst845=== RUN TestIsValidUploadKey/nar_xz846=== PAUSE TestIsValidUploadKey/nar_xz847=== CONT TestParseSize848--- PASS: TestParseSize (0.00s)849=== CONT TestEnsureAbortUploadsRule850=== CONT TestUploadHandlersRejectOversizedBody851=== CONT TestUploadHandlersRejectInvalidKeys852=== CONT TestService_verifyS3ReuploadsStaleNarinfo8532026/11/12 17:02:23 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000854=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed855=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT856=== CONT TestCompleteMultipartUnregistered857=== CONT TestPendingClosureWriteTimeout858=== CONT TestProxyWriteTimeout859=== CONT TestService_verifyS3Integrity860=== CONT TestService_createPendingClosureHandler861=== CONT TestDeduplicatedObjectsRecordedAsPending862=== CONT TestNARDeduplicationMetadataUploadBug863=== CONT TestCreatePendingClosureRejectsOversizedNAR864=== CONT TestMetricsInventory865=== CONT TestCacheConfigHandlerMaxNarSize866=== CONT TestPresignedUploadRegisteredBeforeCommit867=== CONT TestValidatePinReadNeedsWriteSubjects868=== CONT TestAbortSucceeded869=== RUN TestIsValidUploadKey/nar_plain870=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info871=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed872=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed873=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed874=== RUN TestProxyWriteTimeout/narinfo875--- PASS: TestAbortSucceeded (0.00s)876=== CONT TestGenerateLandingPage877=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost878=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost879=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed880=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed881=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed882=== PAUSE TestIsValidUploadKey/nar_plain883=== RUN TestIsValidUploadKey/listing884--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)885--- PASS: TestSkippedUploadsHandler (0.01s)886=== CONT TestService_healthCheckHandler887--- PASS: TestGenerateLandingPage (0.00s)888=== CONT TestGracefulShutdownDrainsInflight889=== CONT TestService_readinessHandler8902026/11/12 17:02:23 INFO Received uploads request method=POST path=/api/pending_closures891--- PASS: TestValidatePinReadNeedsWriteSubjects (0.00s)892=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed8932026/11/12 17:02:23 INFO Starting HTTP server address=127.0.0.1:44183894=== CONT TestGCTaskStore_Fail895--- PASS: TestGCTaskStore_Fail (0.00s)896=== CONT TestGCTaskStore_CompletedAllowsNewTask897--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)898=== CONT TestGCTaskStore_GetReturnsLatest899--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)900--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)901=== CONT TestGCTaskStore_ConflictDifferentParams902--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)9032026/11/12 17:02:23 INFO Shutdown signal received, draining in-flight requests timeout=10s904=== CONT TestGCTaskStore_DeduplicateSameParams905--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)906=== CONT TestGCTaskStore_StartNew907--- PASS: TestGCTaskStore_StartNew (0.00s)908=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info909=== CONT TestGCTaskStore_GetEmpty910--- PASS: TestGCTaskStore_GetEmpty (0.00s)911=== CONT TestCommitRacingPendingCleanupKeepsObjects912=== RUN TestPendingClosureWriteTimeout/empty913=== PAUSE TestPendingClosureWriteTimeout/empty914=== RUN TestPendingClosureWriteTimeout/400_objects915=== PAUSE TestProxyWriteTimeout/narinfo916=== PAUSE TestIsValidUploadKey/listing917=== CONT TestGCTaskStore_PhaseUpdates918=== RUN TestIsValidUploadKey/build_log919=== PAUSE TestIsValidUploadKey/build_log920=== RUN TestIsValidUploadKey/build_log_home-manager_file921=== PAUSE TestIsValidUploadKey/build_log_home-manager_file922=== RUN TestIsValidUploadKey/build_log_plus_in_name923--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)924=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal925=== PAUSE TestPendingClosureWriteTimeout/400_objects926=== RUN TestPendingClosureWriteTimeout/670k_objects927=== RUN TestProxyWriteTimeout/1_GiB_nar928=== CONT TestPushWaitsForSweepHoldingRow929=== CONT TestGCEndsOnShutdown930=== RUN TestGCEndsOnShutdown/before_the_run931=== PAUSE TestGCEndsOnShutdown/before_the_run932=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete933=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete934=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark935=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark936=== CONT TestSweepRechecksPendingAfterLocking937=== PAUSE TestIsValidUploadKey/build_log_plus_in_name938=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal939=== PAUSE TestPendingClosureWriteTimeout/670k_objects940=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline941=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline942=== PAUSE TestProxyWriteTimeout/1_GiB_nar943=== RUN TestProxyWriteTimeout/10_GiB_nar944=== PAUSE TestProxyWriteTimeout/10_GiB_nar945=== RUN TestProxyWriteTimeout/unknown_size946=== PAUSE TestProxyWriteTimeout/unknown_size947=== CONT TestGetExistingObjectsLocksRows948=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key949=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key950=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key951=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key952=== RUN TestIsValidUploadKey/build_log_question_mark953=== PAUSE TestIsValidUploadKey/build_log_question_mark954=== RUN TestIsValidUploadKey/build_log_equals955=== PAUSE TestIsValidUploadKey/build_log_equals956=== RUN TestIsValidUploadKey/realisation957=== PAUSE TestIsValidUploadKey/realisation958=== RUN TestIsValidUploadKey/realisation_plus_in_output959=== CONT TestRegisterCompletedObjectNeedsPendingRow960=== PAUSE TestIsValidUploadKey/realisation_plus_in_output961=== RUN TestIsValidUploadKey/nix-cache-info962=== PAUSE TestIsValidUploadKey/nix-cache-info963=== RUN TestIsValidUploadKey/index.html964=== PAUSE TestIsValidUploadKey/index.html965=== RUN TestIsValidUploadKey/narinfo_key,_nar_type966=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type967=== RUN TestIsValidUploadKey/nar_key,_narinfo_type968=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type969=== RUN TestIsValidUploadKey/listing_key,_narinfo_type970=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type971=== RUN TestIsValidUploadKey/traversal972=== PAUSE TestIsValidUploadKey/traversal973=== RUN TestIsValidUploadKey/traversal_nar974=== PAUSE TestIsValidUploadKey/traversal_nar975=== CONT TestSweepSkipsRowsLockedByPush976=== RUN TestIsValidUploadKey/absolute977=== PAUSE TestIsValidUploadKey/absolute978=== RUN TestIsValidUploadKey/empty_key979=== PAUSE TestIsValidUploadKey/empty_key980=== RUN TestIsValidUploadKey/unknown_type981=== PAUSE TestIsValidUploadKey/unknown_type982=== CONT TestFailedS3DeleteKeepsTombstone983--- PASS: TestGracefulShutdownDrainsInflight (0.14s)984=== CONT TestForceGCDuringPushOffersSweptObject985=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows986=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows987=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check988=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check989=== CONT TestClientSharedPathCommittedMidPush990=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure991=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure992=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart993=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart994=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts995=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts996=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection9972026/11/12 17:02:23 INFO Received cleanup request method=DELETE path=/api/pending_closures9982026/11/12 17:02:23 INFO Aborted multipart uploads count=0 kept=09992026/11/12 17:02:23 INFO Received uploads request method=POST path=/api/pending_closures10002026/11/12 17:02:23 INFO Received cleanup request method=DELETE path=/api/pending_closures10012026/11/12 17:02:23 INFO Aborted multipart uploads count=1 kept=010022026/11/12 17:02:23 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10032026-11-12 17:02:23.965 UTC [449] ERROR: Closure does not exist: id=110042026-11-12 17:02:23.965 UTC [449] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 17 at RAISE10052026-11-12 17:02:23.965 UTC [449] STATEMENT: -- name: CommitPendingClosure :one1006 SELECT commit_pending_closure($1::bigint)::bigint1007 1008--- PASS: TestService_cleanupPendingClosuresHandler (0.29s)1009=== CONT TestPushDedupSurvivesConcurrentGC1010--- PASS: TestService_Rustfstest (0.29s)1011=== CONT TestGCSweepDeliversEachKeyOnce10122026/11/12 17:02:23 INFO Received uploads request method=POST path=/api/pending_closures10132026/11/12 17:02:24 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket7 days=21014--- PASS: TestEnsureAbortUploadsRule (0.34s)1015=== CONT TestGCMetrics10162026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures1017=== NAME TestNARDeduplicationMetadataUploadBug1018 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug299529866/001/store/fgaajgq76ky68nbqf5myravvrp3q3ss3-file1.txt10192026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures10202026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures10212026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures10222026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures10232026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures10242026/11/12 17:02:24 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10252026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures1026--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.45s)1027=== CONT TestTombstonedObjectOfferedWithoutWaiting1028--- PASS: TestService_healthCheckHandler (0.45s)1029=== CONT TestGCBugBareHashReferences10302026/11/12 17:02:24 INFO Received push request method=POST path=/api/pushes10312026/11/12 17:02:24 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)10322026/11/12 17:02:24 INFO Uploading fgaajgq76ky68nbqf5myravvrp3q3ss3-file1.txt (160B)10332026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"10342026/11/12 17:02:24 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10352026/11/12 17:02:24 WARN Failed to register uploaded object key=fgaajgq76ky68nbqf5myravvrp3q3ss3.ls error="server returned 404: 404 page not found\n"10362026/11/12 17:02:24 INFO Signed narinfos id=1 count=110372026/11/12 17:02:24 INFO Uploading 1 narinfos10382026/11/12 17:02:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10392026/11/12 17:02:24 INFO Received complete push request method=POST path=/api/pushes/1/complete10402026/11/12 17:02:24 WARN Failed to register uploaded object key=fgaajgq76ky68nbqf5myravvrp3q3ss3.narinfo error="server returned 404: 404 page not found\n"10412026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures10422026/11/12 17:02:24 INFO Completed push id=1 objects=3 duration=7.286848ms10432026/11/12 17:02:24 INFO Upload complete. (64ms)1044=== NAME TestNARDeduplicationMetadataUploadBug1045 metadata_upload_test.go:54: Retrieved narinfo from S3:1046 StorePath: /build/TestNARDeduplicationMetadataUploadBug299529866/001/store/fgaajgq76ky68nbqf5myravvrp3q3ss3-file1.txt1047 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1048 Compression: zstd1049 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1050 NarSize: 1601051 References: 1052 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1053--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.49s)1054=== CONT TestGCSweepSkipsPendingObjects1055=== NAME TestNARDeduplicationMetadataUploadBug1056 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1057 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1058 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1059--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.42s)1060=== CONT TestGCAdvisoryLockBlocksConcurrentRun1061=== NAME TestNARDeduplicationMetadataUploadBug1062 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug299529866/001/store/82n51ghvdr4kxxmsz0lx49dv4g4nls1n-file2.txt1063--- PASS: TestMetricsInventory (0.46s)1064=== CONT TestLeadEndsOnShutdown10652026/11/12 17:02:24 WARN readiness check failed error="closed pool"1066--- PASS: TestService_readinessHandler (0.47s)1067=== CONT TestPinScopedPrincipal10682026/11/12 17:02:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10692026/11/12 17:02:24 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1070--- PASS: TestCompleteMultipartUnregistered (0.49s)1071=== CONT TestLeadIncumbentAnnouncesAtOnce10722026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures10732026/11/12 17:02:24 INFO Received push request method=POST path=/api/pushes10742026/11/12 17:02:24 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)10752026/11/12 17:02:24 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign10762026/11/12 17:02:24 WARN Failed to register uploaded object key=82n51ghvdr4kxxmsz0lx49dv4g4nls1n.ls error="server returned 404: 404 page not found\n"10772026/11/12 17:02:24 INFO Signed narinfos id=2 count=110782026/11/12 17:02:24 INFO Uploading 1 narinfos10792026/11/12 17:02:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10802026/11/12 17:02:24 INFO Received complete push request method=POST path=/api/pushes/2/complete10812026/11/12 17:02:24 WARN Failed to register uploaded object key=82n51ghvdr4kxxmsz0lx49dv4g4nls1n.narinfo error="server returned 404: 404 page not found\n"10822026/11/12 17:02:24 INFO Completed push id=2 objects=3 duration=2.521144ms10832026/11/12 17:02:24 INFO Upload complete. (54ms)1084=== NAME TestNARDeduplicationMetadataUploadBug1085 metadata_upload_test.go:76: Retrieved narinfo from S3:1086 StorePath: /build/TestNARDeduplicationMetadataUploadBug299529866/001/store/82n51ghvdr4kxxmsz0lx49dv4g4nls1n-file2.txt1087 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1088 Compression: zstd1089 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1090 NarSize: 1601091 References: 1092 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1093 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1094 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1095 {"version":1,"root":{"type":"regular","size":44}}1096--- PASS: TestNARDeduplicationMetadataUploadBug (0.62s)1097=== CONT TestClientWithDependencies10982026/11/12 17:02:24 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NDg1Y2M3ZmItZDY0Mi00NjlkLWJhZDQtODc1Y2YwYWViMDRlLjQ3YTRmNzk3LWI1OWEtNDFiYy05Y2VkLTdkNDYwOWI1ZGVlNXgxNzk0NTAyOTQzOTkxMzA5MDcx parts=1010992026/11/12 17:02:24 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11002026/11/12 17:02:24 INFO Completed upload id=1 objects=2 duration=4.056858ms11012026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures11022026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures11032026/11/12 17:02:24 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo11042026/11/12 17:02:24 WARN Found objects in DB but missing from S3, will re-upload count=11105--- PASS: TestService_verifyS3Integrity (0.64s)1106=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent11072026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=011082026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period11092026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011102026/11/12 17:02:24 INFO Vacuumed table table=pending_closures11112026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts11122026/11/12 17:02:24 INFO Vacuumed table table=pending_objects11132026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads11142026/11/12 17:02:24 INFO Vacuumed table table=closures11152026/11/12 17:02:24 INFO Vacuumed table table=objects11162026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=011172026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period11182026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=01119--- PASS: TestGetExistingObjectsLocksRows (0.60s)1120=== CONT TestClientMultipleUploads11212026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period11222026/11/12 17:02:24 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."11232026/11/12 17:02:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."11242026/11/12 17:02:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1125--- PASS: TestFailedS3DeleteKeepsTombstone (0.61s)1126=== CONT TestLeadElectsOneAndHandsOver11272026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=011282026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=011292026/11/12 17:02:24 INFO Vacuumed table table=pending_closures11302026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts11312026/11/12 17:02:24 INFO Vacuumed table table=pending_objects11322026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads11332026/11/12 17:02:24 INFO Vacuumed table table=closures11342026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period11352026/11/12 17:02:24 INFO Vacuumed table table=objects1136--- PASS: TestSweepSkipsRowsLockedByPush (0.62s)1137=== CONT TestClientIntegration11382026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011392026/11/12 17:02:24 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NDg1Y2M3ZmItZDY0Mi00NjlkLWJhZDQtODc1Y2YwYWViMDRlLjdmNzRiYjcyLTk1ZmYtNGVjNC05OTgxLTU2N2JjOGIzNjYyMngxNzk0NTAyOTQ0MDY3OTE1ODYy parts=1011402026/11/12 17:02:24 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11412026/11/12 17:02:24 INFO Vacuumed table table=pending_closures11422026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts11432026/11/12 17:02:24 INFO Vacuumed table table=pending_objects11442026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads11452026/11/12 17:02:24 INFO Completed upload id=1 objects=2 duration=3.648794ms11462026/11/12 17:02:24 INFO Vacuumed table table=closures11472026/11/12 17:02:24 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000011482026/11/12 17:02:24 INFO Vacuumed table table=objects11492026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=011502026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures11512026/11/12 17:02:24 INFO Starting cleanup of old closures method=DELETE path=/api/closures11522026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period11532026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures11542026/11/12 17:02:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1155--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.65s)1156=== CONT TestConnectWaitsForAPeerMigration11572026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=011582026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011592026/11/12 17:02:24 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NDg1Y2M3ZmItZDY0Mi00NjlkLWJhZDQtODc1Y2YwYWViMDRlLjU2YjQwODY2LTRmOTUtNGZmMC05MWE2LWRjM2VjZWUyMDRmNXgxNzk0NTAyOTQ0MDQ2MjY2NjYx parts=1211602026/11/12 17:02:24 INFO Vacuumed table table=pending_closures11612026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures11622026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures11632026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts11642026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011652026/11/12 17:02:24 INFO Vacuumed table table=pending_objects1166--- PASS: TestCompletedNarNotReofferedAcrossClosures (0.80s)1167=== CONT TestClientErrorHandling1168=== RUN TestClientErrorHandling/InvalidStorePath1169=== PAUSE TestClientErrorHandling/InvalidStorePath1170=== RUN TestClientErrorHandling/InvalidAuthToken1171=== PAUSE TestClientErrorHandling/InvalidAuthToken1172=== RUN TestClientErrorHandling/ServerNotAvailable1173=== PAUSE TestClientErrorHandling/ServerNotAvailable1174=== CONT TestResolveDBConnectionString1175=== RUN TestResolveDBConnectionString/flag_wins1176=== PAUSE TestResolveDBConnectionString/flag_wins1177=== RUN TestResolveDBConnectionString/file_when_flag_empty1178=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1179=== RUN TestResolveDBConnectionString/missing_file_is_an_error1180=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1181=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1182=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1183=== RUN TestResolveDBConnectionString/nothing_configured1184=== PAUSE TestResolveDBConnectionString/nothing_configured1185=== CONT TestClientCADerivations11862026/11/12 17:02:24 INFO Vacuumed table table=pending_closures11872026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts11882026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads1189--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.73s)1190=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock11912026/11/12 17:02:24 INFO Vacuumed table table=pending_objects11922026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads11932026/11/12 17:02:24 INFO Vacuumed table table=closures11942026/11/12 17:02:24 INFO Vacuumed table table=closures11952026/11/12 17:02:24 INFO Vacuumed table table=objects11962026/11/12 17:02:24 INFO Vacuumed table table=objects1197--- PASS: TestSweepRechecksPendingAfterLocking (0.74s)1198=== CONT TestCacheStatsHandler11992026/11/12 17:02:24 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001200--- PASS: TestService_createPendingClosureHandler (0.82s)1201=== CONT TestDeleteClosuresRacingPins12022026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures12032026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures12042026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=012052026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=012062026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period12072026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period12082026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012092026/11/12 17:02:24 INFO Vacuumed table table=pending_closures12102026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts12112026/11/12 17:02:24 INFO Vacuumed table table=pending_objects12122026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures12132026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads12142026/11/12 17:02:24 INFO Vacuumed table table=closures12152026/11/12 17:02:24 INFO Vacuumed table table=objects1216--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.49s)1217=== CONT TestClientFallsBackToClosures1218--- PASS: TestGCMetrics (0.60s)1219=== CONT TestClientPushesUseOnePush12202026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=012212026/11/12 17:02:24 INFO Vacuumed table table=pending_closures12222026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts12232026/11/12 17:02:24 INFO Vacuumed table table=pending_objects12242026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads12252026/11/12 17:02:24 INFO Vacuumed table table=closures12262026/11/12 17:02:24 INFO Vacuumed table table=objects1227--- PASS: TestGCSweepDeliversEachKeyOnce (0.67s)1228=== CONT TestCacheConfigHandler1229=== RUN TestCacheConfigHandler/full_config,_no_issuer1230=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1231=== RUN TestCacheConfigHandler/no_cache_url_configured1232=== PAUSE TestCacheConfigHandler/no_cache_url_configured1233=== RUN TestCacheConfigHandler/no_signing_keys1234=== PAUSE TestCacheConfigHandler/no_signing_keys1235=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1236=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1237=== CONT TestPresentRefreshesOnlyStaleClosures12382026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=012392026/11/12 17:02:24 INFO Received push request method=POST path=/api/pushes12402026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012412026/11/12 17:02:24 INFO Vacuumed table table=pending_closures12422026/11/12 17:02:24 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)12432026/11/12 17:02:24 INFO Uploading xk3gpxnrg3cnl58ycmv60515g2ik9aw0-shared-dep (136B)12442026/11/12 17:02:24 INFO Uploading hmvzfha44dgm0157lh8brlzdvn1hliw2-top (224B)12452026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts12462026/11/12 17:02:24 INFO Received uploads request method=POST path=/api/pending_closures12472026/11/12 17:02:24 INFO Vacuumed table table=pending_objects12482026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads12492026/11/12 17:02:24 WARN Failed to register uploaded object key=xk3gpxnrg3cnl58ycmv60515g2ik9aw0.ls error="server returned 404: 404 page not found\n"12502026/11/12 17:02:24 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12512026/11/12 17:02:24 WARN Failed to register uploaded object key=hmvzfha44dgm0157lh8brlzdvn1hliw2.ls error="server returned 404: 404 page not found\n"12522026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/171z18jrgy41ny9lnz8i0j35pclfmsq9yf75cycjps67sslq7xbx.nar.zst error="server returned 404: 404 page not found\n"12532026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12542026/11/12 17:02:24 INFO Signed narinfos id=1 count=212552026/11/12 17:02:24 INFO Uploading 2 narinfos12562026/11/12 17:02:24 WARN Failed to register uploaded object key=xk3gpxnrg3cnl58ycmv60515g2ik9aw0.narinfo error="server returned 404: 404 page not found\n"12572026/11/12 17:02:24 INFO Received complete push request method=POST path=/api/pushes/1/complete12582026/11/12 17:02:24 WARN Failed to register uploaded object key=hmvzfha44dgm0157lh8brlzdvn1hliw2.narinfo error="server returned 404: 404 page not found\n"12592026/11/12 17:02:24 INFO Vacuumed table table=closures12602026/11/12 17:02:24 INFO Vacuumed table table=objects12612026/11/12 17:02:24 INFO Completed push id=1 objects=6 duration=8.015655ms12622026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=012632026/11/12 17:02:24 INFO Upload complete. (63ms)1264=== NAME TestClientSharedPathCommittedMidPush1265 client_integration_test.go:740: Retrieved narinfo from S3:1266 StorePath: /build/TestClientSharedPathCommittedMidPush2476687929/001/store/xk3gpxnrg3cnl58ycmv60515g2ik9aw0-shared-dep1267 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1268 Compression: zstd1269 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821270 NarSize: 1361271 References: 1272 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n12732026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period12742026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01275 client_integration_test.go:740: Retrieved narinfo from S3:1276 StorePath: /build/TestClientSharedPathCommittedMidPush2476687929/001/store/hmvzfha44dgm0157lh8brlzdvn1hliw2-top1277 URL: nar/171z18jrgy41ny9lnz8i0j35pclfmsq9yf75cycjps67sslq7xbx.nar.zst1278 Compression: zstd1279 NarHash: sha256:171z18jrgy41ny9lnz8i0j35pclfmsq9yf75cycjps67sslq7xbx1280 NarSize: 2241281 References: /build/TestClientSharedPathCommittedMidPush2476687929/001/store/xk3gpxnrg3cnl58ycmv60515g2ik9aw0-shared-dep1282 CA: text:sha256:1yydybzlmf795a0sxqwaqh905k9yl2g8afl33iflv81ap9n99kxf12832026/11/12 17:02:24 INFO Vacuumed table table=pending_closures12842026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts12852026/11/12 17:02:24 INFO Vacuumed table table=pending_objects1286--- PASS: TestClientSharedPathCommittedMidPush (0.86s)1287=== CONT TestReadRedirectNar12882026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads12892026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=012902026/11/12 17:02:24 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1291--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.51s)1292=== CONT TestPinProtectsFromGC12932026/11/12 17:02:24 INFO Vacuumed table table=closures12942026/11/12 17:02:24 INFO Vacuumed table table=objects1295--- PASS: TestPushDedupSurvivesConcurrentGC (0.72s)1296=== CONT TestCompleteMultipartUpload_ErrorButObjectExists12972026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period12982026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012992026/11/12 17:02:24 INFO lead: acquired remote=192.0.2.1:123413002026/11/12 17:02:24 INFO Vacuumed table table=pending_closures13012026/11/12 17:02:24 INFO Vacuumed table table=commit_receipts13022026/11/12 17:02:24 INFO Vacuumed table table=pending_objects13032026/11/12 17:02:24 INFO Vacuumed table table=multipart_uploads13042026/11/12 17:02:24 INFO Vacuumed table table=closures13052026/11/12 17:02:24 INFO Vacuumed table table=objects1306--- PASS: TestGCSweepSkipsPendingObjects (0.54s)1307=== CONT TestPush_CompleteWaitsForCleanupOfThePush13082026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"13092026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"13102026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"13112026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"13122026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"13132026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"13142026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=bad/name"13152026/11/12 17:02:24 WARN mTLS auth: subject not in bound subjects subject="CN=bad/name"1316--- PASS: TestPinScopedPrincipal (0.49s)1317=== CONT TestReadProxyDisabled1318--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.91s)1319=== CONT TestSharedNarBothUploadsComplete13202026/11/12 17:02:24 INFO lead: acquired remote=192.0.2.1:123413212026/11/12 17:02:24 INFO lead: released remote=192.0.2.1:12341322--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.50s)1323=== CONT TestReadProxyRootRedirectsToIndexHTML13242026/11/12 17:02:24 INFO lead: acquired remote=192.0.2.1:12341325=== NAME TestClientMultipleUploads1326 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1645259108/001/store/c4md618idbfg7yk8bhwj6laz1plad7vm-test-file-0.txt13272026/11/12 17:02:24 INFO lead: acquired remote=192.0.2.1:12341328=== NAME TestClientWithDependencies1329 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies1295745038/001/store/dqv9xg51yyfr8zg1x9czz3wjlk9lr5a3-test-script1330--- PASS: TestGCBugBareHashReferences (0.72s)1331=== CONT TestCommitCanBeRepeatedUntilCleanup1332=== RUN TestCommitCanBeRepeatedUntilCleanup/push1333=== PAUSE TestCommitCanBeRepeatedUntilCleanup/push1334=== RUN TestCommitCanBeRepeatedUntilCleanup/pending_closure1335=== PAUSE TestCommitCanBeRepeatedUntilCleanup/pending_closure1336=== CONT TestReadProxyConditionalGet1337=== NAME TestClientIntegration1338 client_integration_test.go:292: Created store path: /build/TestClientIntegration4268264202/002/store/jrbmj706df11hm57x13ijqfpan6alanx-test-file.txt1339=== NAME TestClientWithDependencies1340 client_integration_test.go:663: Found 1 dependencies (including self)1341=== NAME TestClientMultipleUploads1342 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1645259108/001/store/ap1is5vbhp41qjidrdyzm18id7qcg41p-test-file-1.txt1343--- PASS: TestPushWaitsForSweepHoldingRow (1.14s)1344=== CONT TestPush_RepeatedRootCommitsOnce1345=== NAME TestClientMultipleUploads1346 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1645259108/001/store/ll34sy8lq1l3briknkji1fiyrhw5qdac-test-file-2.txt1347--- PASS: TestCacheStatsHandler (0.43s)1348=== CONT TestPush_SignsNarinfosOfItsPendingObjects13492026/11/12 17:02:24 INFO Received push request method=POST path=/api/pushes13502026/11/12 17:02:24 INFO Received push request method=POST path=/api/pushes13512026/11/12 17:02:24 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13522026/11/12 17:02:24 INFO Uploading jrbmj706df11hm57x13ijqfpan6alanx-test-file.txt (152B)1353=== NAME TestClientCADerivations1354 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations932063081/001/store/hxs64mbap41ynb86rl4z1srvkazncsa6-ca-test13552026/11/12 17:02:24 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13562026/11/12 17:02:24 INFO Uploading dqv9xg51yyfr8zg1x9czz3wjlk9lr5a3-test-script (136B)13572026/11/12 17:02:24 WARN Failed to register uploaded object key=jrbmj706df11hm57x13ijqfpan6alanx.ls error="server returned 404: 404 page not found\n"13582026/11/12 17:02:24 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13592026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"13602026/11/12 17:02:24 INFO Signed narinfos id=1 count=113612026/11/12 17:02:24 INFO Uploading 1 narinfos13622026/11/12 17:02:24 WARN Failed to register uploaded object key=dqv9xg51yyfr8zg1x9czz3wjlk9lr5a3.ls error="server returned 404: 404 page not found\n"13632026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13642026/11/12 17:02:24 WARN Failed to register uploaded object key=log/lspa8rvpmv26jnkr78hdcnah8lwx9imx-test-script.drv error="server returned 404: 404 page not found\n"13652026/11/12 17:02:24 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13662026/11/12 17:02:24 INFO Signed narinfos id=1 count=113672026/11/12 17:02:24 INFO Uploading 1 narinfos13682026/11/12 17:02:24 INFO Received complete push request method=POST path=/api/pushes/1/complete13692026/11/12 17:02:24 WARN Failed to register uploaded object key=jrbmj706df11hm57x13ijqfpan6alanx.narinfo error="server returned 404: 404 page not found\n"13702026/11/12 17:02:24 INFO Received complete push request method=POST path=/api/pushes/1/complete13712026/11/12 17:02:24 WARN Failed to register uploaded object key=dqv9xg51yyfr8zg1x9czz3wjlk9lr5a3.narinfo error="server returned 404: 404 page not found\n"13722026/11/12 17:02:24 INFO Completed push id=1 objects=3 duration=8.754781ms13732026/11/12 17:02:24 INFO Upload complete. (61ms)13742026/11/12 17:02:24 INFO Completed push id=1 objects=4 duration=8.58068ms13752026/11/12 17:02:24 INFO Upload complete. (61ms)1376 client_ca_test.go:139: Found 1 dependencies (including self)13772026/11/12 17:02:24 INFO Aborted multipart uploads count=0 kept=013782026/11/12 17:02:24 INFO Received push request method=POST path=/api/pushes13792026/11/12 17:02:24 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)13802026/11/12 17:02:24 INFO Uploading ll34sy8lq1l3briknkji1fiyrhw5qdac-test-file-2.txt (160B)13812026/11/12 17:02:24 INFO Uploading c4md618idbfg7yk8bhwj6laz1plad7vm-test-file-0.txt (160B)13822026/11/12 17:02:24 INFO Uploading ap1is5vbhp41qjidrdyzm18id7qcg41p-test-file-1.txt (160B)13832026/11/12 17:02:24 WARN Force mode enabled - objects will be deleted immediately without grace period13842026/11/12 17:02:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013852026/11/12 17:02:24 WARN Failed to register uploaded object key=ll34sy8lq1l3briknkji1fiyrhw5qdac.ls error="server returned 404: 404 page not found\n"13862026/11/12 17:02:24 WARN Failed to register uploaded object key=c4md618idbfg7yk8bhwj6laz1plad7vm.ls error="server returned 404: 404 page not found\n"13872026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"13882026/11/12 17:02:24 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13892026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"13902026/11/12 17:02:24 WARN Failed to register uploaded object key=ap1is5vbhp41qjidrdyzm18id7qcg41p.ls error="server returned 404: 404 page not found\n"13912026/11/12 17:02:24 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"13922026/11/12 17:02:24 INFO Signed narinfos id=1 count=313932026/11/12 17:02:24 INFO Uploading 3 narinfos13942026/11/12 17:02:25 INFO Vacuumed table table=pending_closures1395--- PASS: TestPresentRefreshesOnlyStaleClosures (0.36s)1396=== CONT TestReadProxyOutlastsServerWriteTimeout13972026/11/12 17:02:25 INFO All 1 paths already cached1398=== NAME TestClientIntegration1399 client_integration_test.go:318: Retrieved narinfo from S3:1400 StorePath: /build/TestClientIntegration4268264202/002/store/jrbmj706df11hm57x13ijqfpan6alanx-test-file.txt1401 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1402 Compression: zstd1403 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11404 NarSize: 1521405 References: 1406 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk114072026/11/12 17:02:25 INFO Vacuumed table table=commit_receipts14082026/11/12 17:02:25 WARN Failed to register uploaded object key=ap1is5vbhp41qjidrdyzm18id7qcg41p.narinfo error="server returned 404: 404 page not found\n"14092026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/1/complete14102026/11/12 17:02:25 WARN Failed to register uploaded object key=c4md618idbfg7yk8bhwj6laz1plad7vm.narinfo error="server returned 404: 404 page not found\n"14112026/11/12 17:02:25 WARN Failed to register uploaded object key=ll34sy8lq1l3briknkji1fiyrhw5qdac.narinfo error="server returned 404: 404 page not found\n"1412 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1413 client_integration_test.go:319: Decompressed .ls content (64 bytes):1414 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}14152026/11/12 17:02:25 INFO Vacuumed table table=pending_objects14162026/11/12 17:02:25 INFO Vacuumed table table=multipart_uploads14172026/11/12 17:02:25 INFO Vacuumed table table=closures14182026/11/12 17:02:25 INFO Completed push id=1 objects=9 duration=7.768392ms14192026/11/12 17:02:25 INFO Vacuumed table table=objects14202026/11/12 17:02:25 INFO Upload complete. (68ms)1421=== NAME TestClientMultipleUploads1422 client_integration_test.go:417: Uploaded 3 paths in 99.852227ms1423=== NAME TestClientWithDependencies1424 client_integration_test.go:677: Skipping nix copy test - isolated store (/build/TestClientWithDependencies1295745038/001/store) requires matching store prefix1425--- PASS: TestClientWithDependencies (0.72s)1426=== CONT TestPush_CompleteCommitsEveryRoot1427--- PASS: TestClientMultipleUploads (0.67s)1428=== CONT TestPush_RejectsBadRequests1429--- PASS: TestReadRedirectNar (0.34s)1430=== CONT TestReadProxyHead14312026/11/12 17:02:25 INFO Received uploads request method=POST path=/api/pending_closures14322026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes14332026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes1434--- PASS: TestReadProxyDisabled (0.37s)1435=== CONT TestPush_OverlappingRootsStoreOneRowPerKey14362026/11/12 17:02:25 INFO Object in database but missing from S3 key=jrbmj706df11hm57x13ijqfpan6alanx.ls14372026/11/12 17:02:25 WARN Found objects in DB but missing from S3, will re-upload count=114382026/11/12 17:02:25 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14392026/11/12 17:02:25 INFO Uploading hxs64mbap41ynb86rl4z1srvkazncsa6-ca-test (144B)14402026/11/12 17:02:25 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)14412026/11/12 17:02:25 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14422026/11/12 17:02:25 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"14432026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/2/complete14442026/11/12 17:02:25 WARN Failed to register uploaded object key=hxs64mbap41ynb86rl4z1srvkazncsa6.ls error="server returned 404: 404 page not found\n"14452026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14462026/11/12 17:02:25 WARN Failed to register uploaded object key=log/ssb23jpiqdm7ccrj5l372j9qi8y22jh5-ca-test.drv error="server returned 404: 404 page not found\n"14472026/11/12 17:02:25 WARN Failed to register uploaded object key=jrbmj706df11hm57x13ijqfpan6alanx.ls error="server returned 404: 404 page not found\n"14482026/11/12 17:02:25 INFO Signed narinfos id=1 count=114492026/11/12 17:02:25 INFO Uploading 1 narinfos14502026/11/12 17:02:25 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NDg1Y2M3ZmItZDY0Mi00NjlkLWJhZDQtODc1Y2YwYWViMDRlLjU2MzkxZDYzLTdlNzctNGQwOS05MDQ3LTNkNDE0ZDdjNTYyNHgxNzk0NTAyOTQ1MDY3MjM0NzIx14512026/11/12 17:02:25 INFO Completed push id=2 objects=3 duration=5.570552ms14522026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/1/complete14532026/11/12 17:02:25 INFO Upload complete. (50ms)14542026/11/12 17:02:25 WARN Failed to register uploaded object key=hxs64mbap41ynb86rl4z1srvkazncsa6.narinfo error="server returned 404: 404 page not found\n"1455=== NAME TestClientIntegration1456 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1457 client_integration_test.go:336: Decompressed .ls content (49 bytes):1458 {"version":1,"root":{"type":"regular","size":39}}14592026/11/12 17:02:25 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NDg1Y2M3ZmItZDY0Mi00NjlkLWJhZDQtODc1Y2YwYWViMDRlLjU2MzkxZDYzLTdlNzctNGQwOS05MDQ3LTNkNDE0ZDdjNTYyNHgxNzk0NTAyOTQ1MDY3MjM0NzIx parts=114602026/11/12 17:02:25 INFO Completed push id=1 objects=4 duration=8.369638ms14612026/11/12 17:02:25 INFO Upload complete. (99ms)1462=== NAME TestPinProtectsFromGC1463 client_integration_test.go:791: Pinned store path: /build/TestPinProtectsFromGC2821246088/001/store/24nslyqgv86zc29a0zah09sz4d2a7lsf-pinned-file.txt1464 client_integration_test.go:792: Unpinned store path: /build/TestPinProtectsFromGC2821246088/001/store/hjxr1f55ylmlfr8zhli8j6pwyv7ikm4h-unpinned-file.txt1465--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.42s)1466=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit14672026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes1468=== NAME TestClientCADerivations1469 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations932063081/001/store/hxs64mbap41ynb86rl4z1srvkazncsa6-ca-test1470 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1471 Compression: zstd1472 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1473 NarSize: 1441474 References: 1475 Deriver: /build/TestClientCADerivations932063081/001/store/ssb23jpiqdm7ccrj5l372j9qi8y22jh5-ca-test.drv1476 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1477 client_ca_test.go:185: Checking for realisation files in S3...1478 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1479 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache14802026/11/12 17:02:25 INFO Received uploads request method=POST path=/api/pending_closures14812026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes14822026/11/12 17:02:25 INFO Received uploads request method=POST path=/api/pending_closures14832026/11/12 17:02:25 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)14842026/11/12 17:02:25 INFO Uploading w3zn0ph4gi9r6m43l5n0327izmp1mkv7-a (216B)14852026/11/12 17:02:25 INFO Uploading 7b1d9y4fzyxlgacqgw291r9k2dch7q7w-shared-dep (136B)14862026/11/12 17:02:25 WARN Failed to register uploaded object key=g5vwjigr3swirdfrw72k7rmshcbqvbxx.ls error="server returned 404: 404 page not found\n"14872026/11/12 17:02:25 WARN Failed to register uploaded object key=nar/022482shv47lv48dyyyz3ajvi6b52i7vqryqwpil1q2kkz97m4md.nar.zst error="server returned 404: 404 page not found\n"14882026/11/12 17:02:25 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14892026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14902026/11/12 17:02:25 WARN Failed to register uploaded object key=w3zn0ph4gi9r6m43l5n0327izmp1mkv7.ls error="server returned 404: 404 page not found\n"14912026/11/12 17:02:25 WARN Failed to register uploaded object key=7b1d9y4fzyxlgacqgw291r9k2dch7q7w.ls error="server returned 404: 404 page not found\n"14922026/11/12 17:02:25 INFO Signed narinfos id=1 count=314932026/11/12 17:02:25 INFO Uploading 3 narinfos14942026/11/12 17:02:25 INFO Received uploads request method=POST path=/api/pending_closures14952026/11/12 17:02:25 WARN Failed to register uploaded object key=g5vwjigr3swirdfrw72k7rmshcbqvbxx.narinfo error="server returned 404: 404 page not found\n"14962026/11/12 17:02:25 WARN Failed to register uploaded object key=7b1d9y4fzyxlgacqgw291r9k2dch7q7w.narinfo error="server returned 404: 404 page not found\n"14972026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/1/complete14982026/11/12 17:02:25 WARN Failed to register uploaded object key=w3zn0ph4gi9r6m43l5n0327izmp1mkv7.narinfo error="server returned 404: 404 page not found\n"14992026/11/12 17:02:25 INFO Received uploads request method=POST path=/api/pending_closures15002026/11/12 17:02:25 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)15012026/11/12 17:02:25 INFO Uploading gkvygm2rps58zpjyzbpcd28kcd09nayz-shared-dep (136B)15022026/11/12 17:02:25 INFO Uploading p6gry6gfwsyh7mnz2d600cadgks4v6b7-b (216B)1503--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.42s)1504=== CONT TestReadProxyInvalidPath15052026/11/12 17:02:25 INFO Completed push id=1 objects=8 duration=7.953194ms15062026/11/12 17:02:25 INFO Upload complete. (67ms)15072026/11/12 17:02:25 WARN Failed to register uploaded object key=nar/1w1whvfzw7dl5dm5smmwc63l1v3lpr0xvn27ci9bzwry8wkd2183.nar.zst error="server returned 404: 404 page not found\n"1508=== NAME TestClientPushesUseOnePush1509 client_pushes_test.go:97: Retrieved narinfo from S3:1510 StorePath: /build/TestClientPushesUseOnePush3644460904/001/store/7b1d9y4fzyxlgacqgw291r9k2dch7q7w-shared-dep1511 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1512 Compression: zstd1513 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821514 NarSize: 1361515 References: 1516 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n15172026/11/12 17:02:25 WARN Failed to register uploaded object key=gkvygm2rps58zpjyzbpcd28kcd09nayz.ls error="server returned 404: 404 page not found\n"15182026/11/12 17:02:25 WARN Failed to register uploaded object key=p691hzfr26wgc45xwc2c0i8sf3ph9cln.ls error="server returned 404: 404 page not found\n"15192026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign15202026/11/12 17:02:25 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15212026/11/12 17:02:25 WARN Failed to register uploaded object key=p6gry6gfwsyh7mnz2d600cadgks4v6b7.ls error="server returned 404: 404 page not found\n"15222026/11/12 17:02:25 INFO Signed narinfos id=2 count=215232026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign15242026/11/12 17:02:25 INFO Signed narinfos id=1 count=215252026/11/12 17:02:25 INFO Uploading 4 narinfos1526 client_pushes_test.go:97: Retrieved narinfo from S3:1527 StorePath: /build/TestClientPushesUseOnePush3644460904/001/store/w3zn0ph4gi9r6m43l5n0327izmp1mkv7-a1528 URL: nar/022482shv47lv48dyyyz3ajvi6b52i7vqryqwpil1q2kkz97m4md.nar.zst1529 Compression: zstd1530 NarHash: sha256:022482shv47lv48dyyyz3ajvi6b52i7vqryqwpil1q2kkz97m4md1531 NarSize: 2161532 References: /build/TestClientPushesUseOnePush3644460904/001/store/7b1d9y4fzyxlgacqgw291r9k2dch7q7w-shared-dep1533 CA: text:sha256:19nmdi3cfzhhc7r1jarg03nip741wwg36f43c7rrwjp9kxvzz4391534 client_pushes_test.go:97: Retrieved narinfo from S3:15352026/11/12 17:02:25 WARN Failed to register uploaded object key=p691hzfr26wgc45xwc2c0i8sf3ph9cln.narinfo error="server returned 404: 404 page not found\n"1536 StorePath: /build/TestClientPushesUseOnePush3644460904/001/store/g5vwjigr3swirdfrw72k7rmshcbqvbxx-b1537 URL: nar/022482shv47lv48dyyyz3ajvi6b52i7vqryqwpil1q2kkz97m4md.nar.zst1538 Compression: zstd15392026/11/12 17:02:25 WARN Failed to register uploaded object key=p6gry6gfwsyh7mnz2d600cadgks4v6b7.narinfo error="server returned 404: 404 page not found\n"1540 NarHash: sha256:022482shv47lv48dyyyz3ajvi6b52i7vqryqwpil1q2kkz97m4md1541 NarSize: 21615422026/11/12 17:02:25 WARN Failed to register uploaded object key=gkvygm2rps58zpjyzbpcd28kcd09nayz.narinfo error="server returned 404: 404 page not found\n"1543 References: /build/TestClientPushesUseOnePush3644460904/001/store/7b1d9y4fzyxlgacqgw291r9k2dch7q7w-shared-dep1544 CA: text:sha256:19nmdi3cfzhhc7r1jarg03nip741wwg36f43c7rrwjp9kxvzz4391545--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.70s)1546=== CONT TestReadRedirectUsesPublicS3URL15472026/11/12 17:02:25 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15482026/11/12 17:02:25 WARN Failed to register uploaded object key=gkvygm2rps58zpjyzbpcd28kcd09nayz.narinfo error="server returned 404: 404 page not found\n"1549--- PASS: TestClientPushesUseOnePush (0.57s)1550=== CONT TestPush_RootsAreLockedInByteOrder15512026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes15522026/11/12 17:02:25 INFO Completed upload id=1 objects=6 duration=5.667693ms15532026/11/12 17:02:25 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete15542026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes15552026/11/12 17:02:25 INFO Completed upload id=2 objects=6 duration=865.048µs15562026/11/12 17:02:25 INFO Upload complete. (69ms)15572026/11/12 17:02:25 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1558=== NAME TestClientFallsBackToClosures1559 client_pushes_test.go:112: Retrieved narinfo from S3:1560 StorePath: /build/TestClientFallsBackToClosures942959866/001/store/gkvygm2rps58zpjyzbpcd28kcd09nayz-shared-dep1561 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1562 Compression: zstd1563 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821564 NarSize: 1361565 References: 1566 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1567--- PASS: TestReadProxyConditionalGet (0.34s)1568=== CONT TestReadProxy4041569=== NAME TestClientFallsBackToClosures1570 client_pushes_test.go:112: Retrieved narinfo from S3:1571 StorePath: /build/TestClientFallsBackToClosures942959866/001/store/p691hzfr26wgc45xwc2c0i8sf3ph9cln-a1572 URL: nar/1w1whvfzw7dl5dm5smmwc63l1v3lpr0xvn27ci9bzwry8wkd2183.nar.zst1573 Compression: zstd1574 NarHash: sha256:1w1whvfzw7dl5dm5smmwc63l1v3lpr0xvn27ci9bzwry8wkd21831575 NarSize: 2161576 References: /build/TestClientFallsBackToClosures942959866/001/store/gkvygm2rps58zpjyzbpcd28kcd09nayz-shared-dep1577 CA: text:sha256:06ma1lrqlqxd0fj1r150ynvg4akn8wr3dqhd10im4a5m8sha2ak215782026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/3/complete15792026/11/12 17:02:25 WARN Failed to register uploaded object key=jrbmj706df11hm57x13ijqfpan6alanx.ls error="server returned 404: 404 page not found\n"1580 client_pushes_test.go:112: Retrieved narinfo from S3:1581 StorePath: /build/TestClientFallsBackToClosures942959866/001/store/p6gry6gfwsyh7mnz2d600cadgks4v6b7-b1582 URL: nar/1w1whvfzw7dl5dm5smmwc63l1v3lpr0xvn27ci9bzwry8wkd2183.nar.zst1583 Compression: zstd1584 NarHash: sha256:1w1whvfzw7dl5dm5smmwc63l1v3lpr0xvn27ci9bzwry8wkd21831585 NarSize: 2161586 References: /build/TestClientFallsBackToClosures942959866/001/store/gkvygm2rps58zpjyzbpcd28kcd09nayz-shared-dep1587 CA: text:sha256:06ma1lrqlqxd0fj1r150ynvg4akn8wr3dqhd10im4a5m8sha2ak215882026/11/12 17:02:25 INFO Completed push id=3 objects=3 duration=5.095088ms15892026/11/12 17:02:25 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15902026/11/12 17:02:25 INFO Upload complete. (43ms)15912026/11/12 17:02:25 INFO Uploading 24nslyqgv86zc29a0zah09sz4d2a7lsf-pinned-file.txt (128B)1592=== NAME TestClientIntegration1593 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1594--- PASS: TestClientFallsBackToClosures (0.59s)1595=== NAME TestClientIntegration1596 client_integration_test.go:353: Decompressed .ls content (49 bytes):1597 {"version":1,"root":{"type":"regular","size":39}}1598=== CONT TestReadProxyRangeRequest1599=== NAME TestClientIntegration1600 client_integration_test.go:364: Testing garbage collection...16012026/11/12 17:02:25 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"16022026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16032026/11/12 17:02:25 WARN Failed to register uploaded object key=24nslyqgv86zc29a0zah09sz4d2a7lsf.ls error="server returned 404: 404 page not found\n"16042026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes16052026/11/12 17:02:25 INFO Signed narinfos id=1 count=116062026/11/12 17:02:25 INFO Uploading 1 narinfos16072026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/1/complete16082026/11/12 17:02:25 WARN Failed to register uploaded object key=24nslyqgv86zc29a0zah09sz4d2a7lsf.narinfo error="server returned 404: 404 page not found\n"1609=== NAME TestClientCADerivations1610 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1611 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1612 error: binary cache 's3://bucket43?endpoint=http://localhost:45279®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations932063081/001/store'1613 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 116142026/11/12 17:02:25 INFO Completed push id=1 objects=3 duration=7.586791ms16152026/11/12 17:02:25 INFO Upload complete. (81ms)1616--- PASS: TestClientCADerivations (0.75s)1617=== CONT TestService_ReadAuthMiddleware16182026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/1/complete16192026/11/12 17:02:25 INFO Starting cleanup of old closures method=DELETE path=/api/closures16202026/11/12 17:02:25 INFO Garbage collection started16212026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes16222026/11/12 17:02:25 INFO Completed push id=1 objects=4 duration=11.161304ms16232026/11/12 17:02:25 INFO Aborted multipart uploads count=0 kept=01624--- PASS: TestPush_RepeatedRootCommitsOnce (0.35s)1625=== CONT TestReadRedirectKeepsNarinfoProxied16262026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16272026/11/12 17:02:25 INFO Signed narinfos id=1 count=116282026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16292026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16302026/11/12 17:02:25 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"16312026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16322026/11/12 17:02:25 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"16332026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16342026/11/12 17:02:25 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=116352026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16362026/11/12 17:02:25 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1637--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.34s)1638=== CONT TestReadProxyNarStreaming16392026/11/12 17:02:25 WARN Force mode enabled - objects will be deleted immediately without grace period16402026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes1641--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.20s)1642=== CONT TestService_ReadScope_PublicByDefault16432026/11/12 17:02:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016442026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes16452026/11/12 17:02:25 INFO Vacuumed table table=pending_closures16462026/11/12 17:02:25 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16472026/11/12 17:02:25 INFO Uploading hjxr1f55ylmlfr8zhli8j6pwyv7ikm4h-unpinned-file.txt (128B)16482026/11/12 17:02:25 WARN Failed to register uploaded object key=hjxr1f55ylmlfr8zhli8j6pwyv7ikm4h.ls error="server returned 404: 404 page not found\n"16492026/11/12 17:02:25 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign16502026/11/12 17:02:25 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"16512026/11/12 17:02:25 INFO Signed narinfos id=2 count=116522026/11/12 17:02:25 INFO Uploading 1 narinfos16532026/11/12 17:02:25 INFO Vacuumed table table=commit_receipts16542026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/2/complete16552026/11/12 17:02:25 WARN Failed to register uploaded object key=hjxr1f55ylmlfr8zhli8j6pwyv7ikm4h.narinfo error="server returned 404: 404 page not found\n"16562026/11/12 17:02:25 INFO Vacuumed table table=pending_objects16572026/11/12 17:02:25 INFO Vacuumed table table=multipart_uploads16582026/11/12 17:02:25 INFO Completed push id=2 objects=3 duration=2.603064ms16592026/11/12 17:02:25 INFO Upload complete. (55ms)16602026/11/12 17:02:25 INFO Vacuumed table table=closures16612026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes16622026/11/12 17:02:25 INFO Vacuumed table table=objects16632026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/1/complete1664=== RUN TestPush_RejectsBadRequests/no_roots1665=== PAUSE TestPush_RejectsBadRequests/no_roots1666=== RUN TestPush_RejectsBadRequests/no_objects1667=== PAUSE TestPush_RejectsBadRequests/no_objects1668=== RUN TestPush_RejectsBadRequests/bad_root1669=== PAUSE TestPush_RejectsBadRequests/bad_root1670=== RUN TestPush_RejectsBadRequests/root_not_in_objects1671=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1672=== CONT TestReadProxyContentNegotiation16732026/11/12 17:02:25 INFO Completed push id=1 objects=6 duration=10.954602ms1674--- PASS: TestPush_CompleteCommitsEveryRoot (0.34s)1675=== CONT TestService_RequireScope_OIDC16762026/11/12 17:02:25 INFO Starting cleanup of old closures method=DELETE path=/api/closures16772026/11/12 17:02:25 INFO Garbage collection started16782026/11/12 17:02:25 INFO Aborted multipart uploads count=0 kept=01679--- PASS: TestReadProxyHead (0.36s)1680=== CONT TestIsValidCachePath1681=== RUN TestIsValidCachePath/narinfo1682=== PAUSE TestIsValidCachePath/narinfo1683=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1684=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1685=== RUN TestIsValidCachePath/nar_zst1686=== PAUSE TestIsValidCachePath/nar_zst1687=== RUN TestIsValidCachePath/nar_xz1688=== PAUSE TestIsValidCachePath/nar_xz1689=== RUN TestIsValidCachePath/nar_bz21690=== PAUSE TestIsValidCachePath/nar_bz21691=== RUN TestIsValidCachePath/nar_uncompressed1692=== PAUSE TestIsValidCachePath/nar_uncompressed1693=== RUN TestIsValidCachePath/ls1694=== PAUSE TestIsValidCachePath/ls1695=== RUN TestIsValidCachePath/log1696=== PAUSE TestIsValidCachePath/log1697=== RUN TestIsValidCachePath/realisation1698=== PAUSE TestIsValidCachePath/realisation1699=== RUN TestIsValidCachePath/nix-cache-info1700=== PAUSE TestIsValidCachePath/nix-cache-info1701=== RUN TestIsValidCachePath/index.html1702=== PAUSE TestIsValidCachePath/index.html1703=== RUN TestIsValidCachePath/traversal_parent1704=== PAUSE TestIsValidCachePath/traversal_parent1705=== RUN TestIsValidCachePath/traversal_in_middle1706=== PAUSE TestIsValidCachePath/traversal_in_middle1707=== RUN TestIsValidCachePath/invalid_char_e1708=== PAUSE TestIsValidCachePath/invalid_char_e1709=== RUN TestIsValidCachePath/invalid_char_u1710=== PAUSE TestIsValidCachePath/invalid_char_u1711=== RUN TestIsValidCachePath/random_path1712=== PAUSE TestIsValidCachePath/random_path1713=== RUN TestIsValidCachePath/empty1714=== PAUSE TestIsValidCachePath/empty1715=== RUN TestIsValidCachePath/leading_slash1716=== PAUSE TestIsValidCachePath/leading_slash1717=== RUN TestIsValidCachePath/wrong_extension1718=== PAUSE TestIsValidCachePath/wrong_extension1719=== RUN TestIsValidCachePath/short_hash1720=== PAUSE TestIsValidCachePath/short_hash1721=== RUN TestIsValidCachePath/pins_substring_attack1722=== PAUSE TestIsValidCachePath/pins_substring_attack1723=== RUN TestIsValidCachePath/xpins_substring_attack1724=== PAUSE TestIsValidCachePath/xpins_substring_attack1725=== RUN TestIsValidCachePath/log_pins_substring_attack1726=== PAUSE TestIsValidCachePath/log_pins_substring_attack1727=== CONT TestParseSingleRange1728=== RUN TestParseSingleRange/none1729=== PAUSE TestParseSingleRange/none1730=== RUN TestParseSingleRange/unknown_unit1731=== PAUSE TestParseSingleRange/unknown_unit1732=== RUN TestParseSingleRange/multi-range_ignored1733=== PAUSE TestParseSingleRange/multi-range_ignored1734=== RUN TestParseSingleRange/malformed_no_dash1735=== PAUSE TestParseSingleRange/malformed_no_dash1736=== RUN TestParseSingleRange/malformed_both_empty1737=== PAUSE TestParseSingleRange/malformed_both_empty1738=== RUN TestParseSingleRange/malformed_end_before_start1739=== PAUSE TestParseSingleRange/malformed_end_before_start1740=== RUN TestParseSingleRange/closed1741=== PAUSE TestParseSingleRange/closed1742=== RUN TestParseSingleRange/open-ended17432026/11/12 17:02:25 WARN Force mode enabled - objects will be deleted immediately without grace period1744=== PAUSE TestParseSingleRange/open-ended1745=== RUN TestParseSingleRange/end_clamped_to_size1746=== PAUSE TestParseSingleRange/end_clamped_to_size1747=== RUN TestParseSingleRange/suffix1748=== PAUSE TestParseSingleRange/suffix1749=== RUN TestParseSingleRange/suffix_exceeds_size1750=== PAUSE TestParseSingleRange/suffix_exceeds_size1751=== RUN TestParseSingleRange/single_byte1752=== PAUSE TestParseSingleRange/single_byte1753=== RUN TestParseSingleRange/start_past_EOF1754=== PAUSE TestParseSingleRange/start_past_EOF1755=== RUN TestParseSingleRange/start_far_past_EOF1756=== PAUSE TestParseSingleRange/start_far_past_EOF1757=== CONT TestService_AuthMiddleware_OIDC17582026/11/12 17:02:25 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36679/oidc17592026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes17602026/11/12 17:02:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=017612026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/1/complete17622026/11/12 17:02:25 INFO Vacuumed table table=pending_closures17632026/11/12 17:02:25 INFO Vacuumed table table=commit_receipts1764--- PASS: TestReadProxyInvalidPath (0.26s)1765=== CONT TestReadProxyNarinfoAlreadyDecompressed17662026/11/12 17:02:25 INFO Vacuumed table table=pending_objects17672026/11/12 17:02:25 INFO Vacuumed table table=multipart_uploads17682026/11/12 17:02:25 INFO Completed push id=1 objects=4 duration=15.533924ms17692026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes17702026-11-12 17:02:25.436 UTC [1393] ERROR: Push does not exist: id=117712026-11-12 17:02:25.436 UTC [1393] CONTEXT: PL/pgSQL function commit_push(bigint) line 15 at RAISE17722026-11-12 17:02:25.436 UTC [1393] STATEMENT: SELECT commit_push($1)1773--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.73s)1774=== CONT TestPresentNotReportedWhileGCDeletesClosure1775=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1776=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1777=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1778=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1779=== CONT TestService_AuthMiddleware_MTLSProxyHeader17802026/11/12 17:02:25 INFO Vacuumed table table=closures17812026/11/12 17:02:25 INFO Vacuumed table table=objects17822026/11/12 17:02:25 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37417/oidc17832026/11/12 17:02:25 INFO Aborted multipart uploads count=0 kept=017842026/11/12 17:02:25 INFO Received push request method=POST path=/api/pushes17852026/11/12 17:02:25 WARN Force mode enabled - objects will be deleted immediately without grace period17862026/11/12 17:02:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=017872026/11/12 17:02:25 INFO Vacuumed table table=pending_closures17882026/11/12 17:02:25 INFO Vacuumed table table=commit_receipts17892026/11/12 17:02:25 INFO Vacuumed table table=pending_objects17902026/11/12 17:02:25 INFO Vacuumed table table=multipart_uploads1791--- PASS: TestReadProxy404 (0.36s)1792=== CONT TestReadProxyNarinfo17932026/11/12 17:02:25 INFO Vacuumed table table=closures17942026/11/12 17:02:25 INFO Vacuumed table table=objects17952026/11/12 17:02:25 INFO Received complete push request method=POST path=/api/pushes/2/complete17962026/11/12 17:02:25 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17972026/11/12 17:02:25 INFO Completed push id=2 objects=4 duration=3.319111ms1798--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.45s)1799=== CONT TestService_AuthMiddleware18002026-11-12 17:02:25.565 UTC [1744] ERROR: could not obtain lock on row in relation "closures"18012026-11-12 17:02:25.565 UTC [1744] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18022026-11-12 17:02:25.566 UTC [1744] ERROR: could not obtain lock on row in relation "closures"18032026-11-12 17:02:25.566 UTC [1744] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18042026-11-12 17:02:25.566 UTC [1744] ERROR: could not obtain lock on row in relation "closures"18052026-11-12 17:02:25.566 UTC [1744] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18062026-11-12 17:02:25.566 UTC [1744] ERROR: could not obtain lock on row in relation "closures"18072026-11-12 17:02:25.566 UTC [1744] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18082026-11-12 17:02:25.567 UTC [1744] ERROR: could not obtain lock on row in relation "closures"18092026-11-12 17:02:25.567 UTC [1744] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1810--- PASS: TestReadRedirectUsesPublicS3URL (0.39s)1811=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1812--- PASS: TestPush_RootsAreLockedInByteOrder (0.39s)1813=== CONT TestPinScopedReadRedirect18142026/11/12 17:02:25 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NDg1Y2M3ZmItZDY0Mi00NjlkLWJhZDQtODc1Y2YwYWViMDRlLjZjYjljYmQwLTQxMmMtNDFlYy05NzgxLTg5NDExZmQzOTZlM3gxNzk0NTAyOTQ1MTQ0OTY4ODQy parts=121815--- PASS: TestReadProxyRangeRequest (0.38s)1816=== CONT TestPinObjectsSharedTarget1817--- PASS: TestService_ReadAuthMiddleware (0.37s)1818=== CONT TestPinObjectsSelfReference1819--- PASS: TestReadProxyNarStreaming (0.38s)1820=== CONT TestPresentReportsOnlyClosureRoots1821--- PASS: TestReadRedirectKeepsNarinfoProxied (0.41s)1822=== CONT TestPinScopedReads1823--- PASS: TestService_ReadScope_PublicByDefault (0.39s)1824=== CONT TestPinObjectsBackfill1825=== RUN TestService_RequireScope_OIDC/builder_may_write1826=== PAUSE TestService_RequireScope_OIDC/builder_may_write1827=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1828=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1829=== RUN TestService_RequireScope_OIDC/ops_may_admin1830=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1831=== RUN TestService_RequireScope_OIDC/ops_may_not_write1832=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1833=== RUN TestService_RequireScope_OIDC/reader_may_not_write1834=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1835=== RUN TestService_RequireScope_OIDC/static_token_may_admin1836=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1837=== RUN TestService_RequireScope_OIDC/static_token_may_write1838=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1839=== RUN TestService_RequireScope_OIDC/reader_may_read1840=== PAUSE TestService_RequireScope_OIDC/reader_may_read1841=== RUN TestService_RequireScope_OIDC/writer_implies_read1842=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1843=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1844=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1845=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1846=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1847=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1848=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1849=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1850=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1851=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1852=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1853=== CONT TestCreatePin_ReservedPins1854--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.33s)1855=== CONT TestGlobToLike1856--- PASS: TestGlobToLike (0.00s)1857=== CONT TestPinObjectsDelete1858--- PASS: TestReadProxyContentNegotiation (0.41s)1859=== CONT TestCommonName1860=== RUN TestCommonName/CN=web-01,O=acme1861=== PAUSE TestCommonName/CN=web-01,O=acme1862=== RUN TestCommonName/O=acme1863=== PAUSE TestCommonName/O=acme1864=== RUN TestCommonName/O=x\,CN=evil,CN=real1865=== PAUSE TestCommonName/O=x\,CN=evil,CN=real1866=== RUN TestCommonName/CN=a\,b,O=x1867=== PAUSE TestCommonName/CN=a\,b,O=x1868=== RUN TestCommonName/CN=a\\,O=b1869=== PAUSE TestCommonName/CN=a\\,O=b1870=== RUN TestCommonName/CN=a+O=b1871=== PAUSE TestCommonName/CN=a+O=b1872=== CONT TestGCDeletesExpiredPinObjects1873--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.33s)1874=== CONT TestPinObjectsOtherPin18752026/11/12 17:02:25 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33911/oidc1876=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1877=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1878=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1879=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1880=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1881=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1882=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1883=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1884=== CONT TestParsePinReadRule1885=== RUN TestParsePinReadRule/CN=*=host-{cn}1886=== PAUSE TestParsePinReadRule/CN=*=host-{cn}1887=== RUN TestParsePinReadRule/CN=web-*,O=x=web-*1888=== PAUSE TestParsePinReadRule/CN=web-*,O=x=web-*1889=== RUN TestParsePinReadRule/CN=*=1890=== PAUSE TestParsePinReadRule/CN=*=1891=== RUN TestParsePinReadRule/=host1892=== PAUSE TestParsePinReadRule/=host1893=== RUN TestParsePinReadRule/CN=*=host/{cn}1894=== PAUSE TestParsePinReadRule/CN=*=host/{cn}1895=== CONT TestPinScopedSubstringAttack18962026/11/12 17:02:25 INFO lead: released remote=192.0.2.1:12341897--- PASS: TestLeadEndsOnShutdown (1.58s)1898=== CONT TestObjectStatsUpdatedOncePerCommit18992026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-web-0119002026/11/12 17:02:25 INFO Created/updated pin name=host-web-01 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo19012026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-db-0119022026/11/12 17:02:25 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"19032026/11/12 17:02:25 WARN mTLS auth: bound subjects configured but subject DN unavailable19042026/11/12 17:02:25 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1905--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.28s)1906=== CONT TestFullReaderUnaffected19072026/11/12 17:02:25 INFO Created/updated pin name=host-db-01 store_path=/nix/store/cccccccccccccccccccccccccccccccc-test narinfo_key=cccccccccccccccccccccccccccccccc.narinfo19082026/11/12 17:02:25 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19092026/11/12 17:02:25 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1910--- PASS: TestService_AuthMiddleware (0.31s)1911=== CONT TestMultipartCleanup19122026/11/12 17:02:25 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"1913--- PASS: TestPinScopedReadRedirect (0.31s)1914=== CONT TestPinObjectsSameTarget19152026/11/12 17:02:25 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1916--- PASS: TestReadProxyNarinfo (0.33s)1917=== CONT TestPinObjectsTwoMoves19182026/11/12 17:02:25 INFO lead: released remote=192.0.2.1:123419192026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-119202026/11/12 17:02:25 INFO Created/updated pin name=host-1 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo19212026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-219222026/11/12 17:02:25 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19232026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-119242026/11/12 17:02:25 INFO Created/updated pin name=host-2 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo19252026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-119262026/11/12 17:02:25 INFO lead: acquired remote=192.0.2.1:123419272026/11/12 17:02:25 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NDg1Y2M3ZmItZDY0Mi00NjlkLWJhZDQtODc1Y2YwYWViMDRlLjdkODBmYzk4LTUwNzItNGE0NS1hZjRhLTI0ZDQ5NzM1YmY2OXgxNzk0NTAyOTQ1MTUzMjEyNTU5 parts=121928--- PASS: TestSharedNarBothUploadsComplete (1.20s)1929=== CONT TestPendingCleanupUsesOneCutoff19302026/11/12 17:02:25 INFO Created/updated pin name=host-1 store_path=/nix/store/cccccccccccccccccccccccccccccccc-test narinfo_key=cccccccccccccccccccccccccccccccc.narinfo19312026/11/12 17:02:25 INFO Created/updated pin name=host-1 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo1932--- PASS: TestPinObjectsSelfReference (0.35s)1933=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1934--- PASS: TestPinObjectsSharedTarget (0.37s)1935=== CONT TestPinObjectsGrace1936--- PASS: TestPresentReportsOnlyClosureRoots (0.32s)1937=== CONT TestObjectStatsTrigger19382026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-web-0119392026/11/12 17:02:25 INFO lead: given back to the incumbent remote=192.0.2.1:123419402026/11/12 17:02:25 INFO lead: acquired remote=192.0.2.1:123419412026/11/12 17:02:25 INFO Created/updated pin name=host-web-01 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo19422026/11/12 17:02:25 INFO Received create pin request method=POST path=/api/pins/host-db-0119432026/11/12 17:02:25 INFO Created/updated pin name=host-db-01 store_path=/nix/store/cccccccccccccccccccccccccccccccc-test narinfo_key=cccccccccccccccccccccccccccccccc.narinfo1944--- PASS: TestPinObjectsBackfill (0.34s)1945=== CONT TestPinObjectsReachable19462026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/host-119472026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19482026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19492026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19502026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19512026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19522026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19532026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19542026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"19552026/11/12 17:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=web-01"1956--- PASS: TestPinScopedReads (0.39s)1957=== CONT TestOrphanedObjectsGCStressTest19582026/11/12 17:02:26 INFO Created/updated pin name=host-1 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo19592026/11/12 17:02:26 INFO Received delete pin request method=DELETE path=/api/pins/host-119602026/11/12 17:02:26 INFO lead: released remote=192.0.2.1:123419612026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/host-119622026/11/12 17:02:26 INFO Deleted pin name=host-11963--- PASS: TestPinObjectsDelete (0.36s)1964=== CONT TestService_NativeMTLS19652026/11/12 17:02:26 INFO Created/updated pin name=host-1 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo19662026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/host-119672026/11/12 17:02:26 INFO Created/updated pin name=host-1 store_path=/nix/store/cccccccccccccccccccccccccccccccc-test narinfo_key=cccccccccccccccccccccccccccccccc.narinfo19682026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/host-119692026/11/12 17:02:26 INFO deleted expired pin objects count=119702026/11/12 17:02:26 INFO Aborted multipart uploads count=0 kept=019712026/11/12 17:02:26 INFO Created/updated pin name=host-1 store_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-test narinfo_key=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo19722026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/host-219732026/11/12 17:02:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019742026/11/12 17:02:26 INFO Vacuumed table table=pending_closures19752026/11/12 17:02:26 INFO Vacuumed table table=commit_receipts19762026/11/12 17:02:26 INFO lead: acquired remote=192.0.2.1:123419772026/11/12 17:02:26 INFO Vacuumed table table=pending_objects19782026/11/12 17:02:26 INFO Vacuumed table table=multipart_uploads19792026/11/12 17:02:26 INFO Created/updated pin name=host-2 store_path=/nix/store/cccccccccccccccccccccccccccccccc-test narinfo_key=cccccccccccccccccccccccccccccccc.narinfo19802026/11/12 17:02:26 INFO Vacuumed table table=closures1981--- PASS: TestPinObjectsOtherPin (0.39s)1982=== CONT TestOrphanedObjectsGC19832026/11/12 17:02:26 INFO Vacuumed table table=objects1984--- PASS: TestGCDeletesExpiredPinObjects (0.39s)1985=== CONT TestServerTLSConfig1986=== RUN TestServerTLSConfig/no_client_CA1987=== PAUSE TestServerTLSConfig/no_client_CA1988=== RUN TestServerTLSConfig/missing_CA_file1989=== PAUSE TestServerTLSConfig/missing_CA_file1990=== RUN TestServerTLSConfig/not_a_PEM_file1991=== PAUSE TestServerTLSConfig/not_a_PEM_file1992=== CONT TestMetricsCommit19932026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux19942026/11/12 17:02:26 WARN Refused reserved pin name=worker-x86_64-linux19952026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux19962026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/my-app19972026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1998--- PASS: TestCreatePin_ReservedPins (0.42s)1999=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed20002026/11/12 17:02:26 INFO Received create pin request method=POST path=/api/pins/host-db-01
loading 1,073 hidden lines…
30742026/11/12 17:02:40 INFO Received uploads request method=POST path=/30752026/11/12 17:02:40 INFO Received push request method=POST path=/30762026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=030772026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=030782026/11/12 17:02:40 INFO Vacuumed table table=pending_closures30792026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts30802026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period30812026/11/12 17:02:40 INFO Received push request method=POST path=/30822026/11/12 17:02:40 INFO Vacuumed table table=pending_objects30832026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads30842026/11/12 17:02:40 INFO Vacuumed table table=closures30852026/11/12 17:02:40 INFO Vacuumed table table=objects30862026/11/12 17:02:40 INFO Received push request method=POST path=/30872026/11/12 17:02:40 INFO Received uploads request method=POST path=/30882026/11/12 17:02:40 INFO Received complete upload request method=POST path=/30892026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=030902026/11/12 17:02:40 INFO Received push request method=POST path=/30912026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=030922026/11/12 17:02:40 INFO Completed upload id=4 objects=14 duration=2.675025ms30932026/11/12 17:02:40 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30942026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30952026/11/12 17:02:40 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30962026/11/12 17:02:40 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30972026/11/12 17:02:40 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30982026/11/12 17:02:40 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30992026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31002026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31012026/11/12 17:02:40 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31022026/11/12 17:02:40 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31032026/11/12 17:02:40 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31042026/11/12 17:02:40 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31052026/11/12 17:02:40 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31062026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."31072026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31082026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31092026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31102026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31112026/11/12 17:02:40 INFO Received complete upload request method=POST path=/31122026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."31132026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31142026/11/12 17:02:40 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."31152026/11/12 17:02:40 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31162026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31172026/11/12 17:02:40 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."31182026/11/12 17:02:40 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31192026/11/12 17:02:40 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31202026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31212026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31222026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31232026/11/12 17:02:40 INFO Received push request method=POST path=/31242026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period31252026/11/12 17:02:40 INFO Completed upload id=10 objects=6 duration=2.374722ms31262026/11/12 17:02:40 INFO Received push request method=POST path=/31272026/11/12 17:02:40 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31282026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31292026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031302026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=031312026/11/12 17:02:40 INFO Received complete upload request method=POST path=/31322026/11/12 17:02:40 INFO Vacuumed table table=pending_closures31332026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts31342026/11/12 17:02:40 INFO Completed upload id=9 objects=7 duration=1.495774ms31352026/11/12 17:02:40 INFO Vacuumed table table=pending_objects31362026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=031372026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads31382026/11/12 17:02:40 INFO Received complete upload request method=POST path=/31392026/11/12 17:02:40 INFO Received push request method=POST path=/31402026/11/12 17:02:40 INFO Vacuumed table table=closures31412026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period31422026/11/12 17:02:40 INFO Completed upload id=2 objects=2 duration=1.05321ms31432026/11/12 17:02:40 INFO Received push request method=POST path=/31442026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=031452026/11/12 17:02:40 INFO Vacuumed table table=objects31462026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period31472026/11/12 17:02:40 INFO Received uploads request method=POST path=/31482026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031492026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031502026/11/12 17:02:40 INFO Vacuumed table table=pending_closures31512026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts31522026/11/12 17:02:40 INFO Received complete upload request method=POST path=/31532026/11/12 17:02:40 INFO Vacuumed table table=pending_closures31542026/11/12 17:02:40 INFO Vacuumed table table=pending_objects31552026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads31562026/11/12 17:02:40 INFO Completed upload id=2 objects=10 duration=1.369812ms31572026/11/12 17:02:40 INFO Received uploads request method=POST path=/31582026/11/12 17:02:40 INFO Vacuumed table table=closures31592026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts31602026/11/12 17:02:40 INFO Vacuumed table table=objects31612026/11/12 17:02:40 INFO Received uploads request method=POST path=/31622026/11/12 17:02:40 INFO Vacuumed table table=pending_objects31632026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads31642026/11/12 17:02:40 INFO Vacuumed table table=closures31652026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=031662026/11/12 17:02:40 INFO Vacuumed table table=objects31672026/11/12 17:02:40 INFO Vacuumed table table=pending_closures31682026/11/12 17:02:40 INFO Received uploads request method=POST path=/31692026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts31702026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=031712026/11/12 17:02:40 INFO Received complete push request method=POST path=/31722026/11/12 17:02:40 INFO Vacuumed table table=pending_objects31732026/11/12 17:02:40 INFO Received uploads request method=POST path=/31742026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period31752026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads31762026/11/12 17:02:40 INFO Vacuumed table table=closures31772026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031782026/11/12 17:02:40 INFO Vacuumed table table=objects31792026/11/12 17:02:40 INFO Received complete push request method=POST path=/31802026/11/12 17:02:40 INFO Received complete push request method=POST path=/31812026/11/12 17:02:40 INFO Completed push id=6 objects=5 duration=2.648264ms31822026/11/12 17:02:40 INFO Received push request method=POST path=/31832026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=031842026/11/12 17:02:40 INFO Vacuumed table table=pending_closures31852026/11/12 17:02:40 INFO Completed push id=6 objects=5 duration=1.609515ms31862026/11/12 17:02:40 INFO Completed push id=7 objects=27 duration=1.696796ms31872026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts31882026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=031892026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031902026/11/12 17:02:40 INFO Vacuumed table table=pending_objects31912026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period31922026/11/12 17:02:40 INFO Received uploads request method=POST path=/31932026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads31942026/11/12 17:02:40 INFO Vacuumed table table=pending_closures31952026/11/12 17:02:40 INFO Vacuumed table table=closures31962026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts31972026/11/12 17:02:40 INFO Vacuumed table table=objects31982026/11/12 17:02:40 INFO Vacuumed table table=pending_objects31992026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads32002026/11/12 17:02:40 INFO Vacuumed table table=closures32012026/11/12 17:02:40 INFO Vacuumed table table=objects32022026/11/12 17:02:40 INFO Received push request method=POST path=/32032026/11/12 17:02:40 INFO Received push request method=POST path=/32042026/11/12 17:02:40 INFO Received uploads request method=POST path=/32052026/11/12 17:02:40 INFO Received push request method=POST path=/32062026/11/12 17:02:40 INFO Received uploads request method=POST path=/32072026/11/12 17:02:40 INFO Received uploads request method=POST path=/32082026/11/12 17:02:40 INFO Received complete push request method=POST path=/32092026/11/12 17:02:40 INFO Received complete upload request method=POST path=/32102026/11/12 17:02:40 INFO Received complete push request method=POST path=/32112026/11/12 17:02:40 INFO Completed push id=12 objects=3 duration=1.573435ms32122026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=032132026/11/12 17:02:40 INFO Completed push id=8 objects=13 duration=1.462354ms32142026/11/12 17:02:40 INFO Vacuumed table table=pending_closures32152026/11/12 17:02:40 INFO Completed upload id=1 objects=15 duration=3.797215ms32162026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts32172026/11/12 17:02:40 INFO Received push request method=POST path=/32182026/11/12 17:02:40 INFO Received complete upload request method=POST path=/32192026/11/12 17:02:40 INFO Vacuumed table table=pending_objects32202026/11/12 17:02:40 INFO Completed upload id=3 objects=9 duration=1.310992ms32212026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads32222026/11/12 17:02:40 INFO Received push request method=POST path=/32232026/11/12 17:02:40 INFO Vacuumed table table=closures32242026/11/12 17:02:40 INFO Vacuumed table table=objects32252026/11/12 17:02:40 INFO Received complete push request method=POST path=/32262026/11/12 17:02:40 INFO Received push request method=POST path=/32272026/11/12 17:02:40 INFO Received push request method=POST path=/32282026/11/12 17:02:40 INFO Completed push id=3 objects=5 duration=1.426313ms32292026/11/12 17:02:40 INFO Received uploads request method=POST path=/32302026/11/12 17:02:40 INFO Received uploads request method=POST path=/32312026/11/12 17:02:40 INFO Received push request method=POST path=/32322026/11/12 17:02:40 INFO Received push request method=POST path=/32332026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=032342026/11/12 17:02:40 INFO Received push request method=POST path=/32352026/11/12 17:02:40 INFO Received push request method=POST path=/32362026/11/12 17:02:40 INFO Received complete upload request method=POST path=/32372026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=032382026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=032392026/11/12 17:02:40 INFO Completed upload id=3 objects=24 duration=1.320832ms32402026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032412026/11/12 17:02:40 INFO Received uploads request method=POST path=/32422026/11/12 17:02:40 INFO Received push request method=POST path=/32432026/11/12 17:02:40 INFO Vacuumed table table=pending_closures32442026/11/12 17:02:40 INFO Received uploads request method=POST path=/32452026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts32462026/11/12 17:02:40 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32472026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32482026/11/12 17:02:40 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32492026/11/12 17:02:40 INFO Vacuumed table table=pending_objects32502026/11/12 17:02:40 INFO Received uploads request method=POST path=/32512026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads32522026/11/12 17:02:40 INFO Vacuumed table table=closures32532026/11/12 17:02:40 INFO Vacuumed table table=objects32542026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032552026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=032562026/11/12 17:02:40 INFO Vacuumed table table=pending_closures32572026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period32582026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts32592026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=032602026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032612026/11/12 17:02:40 INFO Vacuumed table table=pending_objects32622026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads32632026/11/12 17:02:40 INFO Received complete push request method=POST path=/32642026/11/12 17:02:40 INFO Vacuumed table table=pending_closures32652026/11/12 17:02:40 INFO Received push request method=POST path=/32662026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts32672026/11/12 17:02:40 INFO Vacuumed table table=closures32682026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period32692026/11/12 17:02:40 INFO Completed push id=5 objects=6 duration=1.490073ms32702026/11/12 17:02:40 INFO Vacuumed table table=pending_objects32712026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads32722026/11/12 17:02:40 INFO Vacuumed table table=objects32732026/11/12 17:02:40 INFO Vacuumed table table=closures32742026/11/12 17:02:40 INFO Received uploads request method=POST path=/32752026/11/12 17:02:40 INFO Vacuumed table table=objects32762026/11/12 17:02:40 INFO Received complete upload request method=POST path=/32772026/11/12 17:02:40 INFO Received push request method=POST path=/32782026/11/12 17:02:40 INFO Completed upload id=10 objects=23 duration=1.645075ms32792026/11/12 17:02:40 INFO Received complete push request method=POST path=/32802026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=032812026/11/12 17:02:40 INFO Received push request method=POST path=/32822026/11/12 17:02:40 INFO Completed push id=5 objects=8 duration=2.799426ms32832026/11/12 17:02:40 INFO Received uploads request method=POST path=/32842026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=032852026/11/12 17:02:40 INFO Received complete upload request method=POST path=/32862026/11/12 17:02:40 INFO Received uploads request method=POST path=/32872026/11/12 17:02:40 INFO Completed upload id=11 objects=6 duration=1.768676ms32882026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period32892026/11/12 17:02:40 INFO Received push request method=POST path=/32902026/11/12 17:02:40 INFO Received uploads request method=POST path=/32912026/11/12 17:02:40 INFO Received push request method=POST path=/32922026/11/12 17:02:40 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32932026/11/12 17:02:40 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."32942026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32952026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."32962026/11/12 17:02:40 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."32972026/11/12 17:02:40 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."32982026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."32992026/11/12 17:02:40 INFO Received uploads request method=POST path=/33002026/11/12 17:02:40 INFO Received complete push request method=POST path=/33012026/11/12 17:02:40 INFO Completed push id=4 objects=4 duration=1.490674ms33022026/11/12 17:02:40 INFO Received uploads request method=POST path=/33032026/11/12 17:02:40 INFO Received push request method=POST path=/33042026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=033052026/11/12 17:02:40 INFO Received complete push request method=POST path=/33062026/11/12 17:02:40 INFO Received uploads request method=POST path=/33072026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period33082026/11/12 17:02:40 INFO Completed push id=10 objects=2 duration=1.832197ms33092026/11/12 17:02:40 INFO Received complete upload request method=POST path=/33102026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033112026/11/12 17:02:40 INFO Vacuumed table table=pending_closures33122026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts33132026/11/12 17:02:40 INFO Received complete upload request method=POST path=/33142026/11/12 17:02:40 INFO Completed upload id=10 objects=4 duration=2.323262ms33152026/11/12 17:02:40 INFO Vacuumed table table=pending_objects33162026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=033172026/11/12 17:02:40 INFO Received uploads request method=POST path=/33182026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads33192026/11/12 17:02:40 INFO Vacuumed table table=closures33202026/11/12 17:02:40 INFO Completed upload id=16 objects=23 duration=1.199331ms33212026/11/12 17:02:40 INFO Received uploads request method=POST path=/33222026/11/12 17:02:40 INFO Vacuumed table table=objects33232026/11/12 17:02:40 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33242026/11/12 17:02:40 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33252026/11/12 17:02:40 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33262026/11/12 17:02:40 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33272026/11/12 17:02:40 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."33282026/11/12 17:02:40 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33292026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."33302026/11/12 17:02:40 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33312026/11/12 17:02:40 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33322026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33332026/11/12 17:02:40 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33342026/11/12 17:02:40 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33352026/11/12 17:02:40 INFO Received uploads request method=POST path=/33362026/11/12 17:02:40 INFO Received complete push request method=POST path=/33372026/11/12 17:02:40 INFO Completed push id=7 objects=20 duration=2.12768ms33382026/11/12 17:02:40 INFO Received push request method=POST path=/33392026/11/12 17:02:40 INFO Received complete upload request method=POST path=/33402026/11/12 17:02:40 INFO Received push request method=POST path=/33412026/11/12 17:02:40 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."33422026/11/12 17:02:40 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."33432026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33442026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."33452026/11/12 17:02:40 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."33462026/11/12 17:02:40 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."33472026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."33482026/11/12 17:02:40 INFO Completed upload id=11 objects=6 duration=1.375313ms33492026/11/12 17:02:40 INFO Received complete upload request method=POST path=/33502026/11/12 17:02:40 INFO Received complete upload request method=POST path=/33512026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=033522026/11/12 17:02:40 INFO Received push request method=POST path=/33532026/11/12 17:02:40 INFO Received uploads request method=POST path=/33542026/11/12 17:02:40 INFO Completed upload id=15 objects=6 duration=1.01015ms33552026/11/12 17:02:40 INFO Completed upload id=12 objects=21 duration=1.287872ms33562026/11/12 17:02:40 INFO Received push request method=POST path=/33572026/11/12 17:02:40 INFO Received complete push request method=POST path=/33582026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033592026/11/12 17:02:40 INFO Vacuumed table table=pending_closures33602026/11/12 17:02:40 INFO Completed push id=13 objects=12 duration=3.008828ms33612026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts33622026/11/12 17:02:40 INFO Vacuumed table table=pending_objects33632026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads33642026/11/12 17:02:40 INFO Vacuumed table table=closures33652026/11/12 17:02:40 INFO Vacuumed table table=objects33662026/11/12 17:02:40 INFO Received uploads request method=POST path=/33672026/11/12 17:02:40 INFO Received push request method=POST path=/33682026/11/12 17:02:40 INFO Received push request method=POST path=/33692026/11/12 17:02:40 INFO Received push request method=POST path=/33702026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=033712026/11/12 17:02:40 INFO Received push request method=POST path=/33722026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=033732026/11/12 17:02:40 INFO Received push request method=POST path=/33742026/11/12 17:02:40 INFO Received uploads request method=POST path=/33752026/11/12 17:02:40 INFO Received push request method=POST path=/33762026/11/12 17:02:40 INFO Vacuumed table table=pending_closures33772026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts33782026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033792026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=033802026/11/12 17:02:40 INFO Vacuumed table table=pending_objects33812026/11/12 17:02:40 INFO Vacuumed table table=pending_closures33822026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads33832026/11/12 17:02:40 INFO Vacuumed table table=closures33842026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts33852026/11/12 17:02:40 INFO Vacuumed table table=objects33862026/11/12 17:02:40 INFO Received push request method=POST path=/33872026/11/12 17:02:40 INFO Vacuumed table table=pending_objects33882026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads33892026/11/12 17:02:40 INFO Received complete push request method=POST path=/33902026/11/12 17:02:40 INFO Vacuumed table table=closures33912026/11/12 17:02:40 INFO Received push request method=POST path=/33922026/11/12 17:02:40 INFO Completed push id=9 objects=5 duration=1.563394ms33932026/11/12 17:02:40 INFO Received uploads request method=POST path=/33942026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=033952026/11/12 17:02:40 INFO Vacuumed table table=objects33962026/11/12 17:02:40 INFO Received push request method=POST path=/33972026/11/12 17:02:40 INFO Received uploads request method=POST path=/33982026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033992026/11/12 17:02:40 INFO Vacuumed table table=pending_closures34002026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts34012026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=034022026/11/12 17:02:40 INFO Vacuumed table table=pending_closures34032026/11/12 17:02:40 INFO Vacuumed table table=pending_objects34042026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads34052026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts34062026/11/12 17:02:40 INFO Vacuumed table table=closures34072026/11/12 17:02:40 INFO Vacuumed table table=pending_objects34082026/11/12 17:02:40 INFO Vacuumed table table=objects34092026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads34102026/11/12 17:02:40 INFO Vacuumed table table=closures34112026/11/12 17:02:40 INFO Received complete upload request method=POST path=/34122026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034132026/11/12 17:02:40 INFO Vacuumed table table=objects34142026/11/12 17:02:40 INFO Received complete upload request method=POST path=/34152026/11/12 17:02:40 INFO Received push request method=POST path=/34162026/11/12 17:02:40 INFO Received uploads request method=POST path=/34172026/11/12 17:02:40 INFO Received complete push request method=POST path=/34182026/11/12 17:02:40 INFO Received complete push request method=POST path=/34192026/11/12 17:02:40 INFO Completed upload id=17 objects=7 duration=989.709µs34202026/11/12 17:02:40 INFO Completed upload id=5 objects=7 duration=2.08038ms34212026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period34222026/11/12 17:02:40 INFO Completed push id=13 objects=23 duration=1.468334ms34232026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034242026/11/12 17:02:40 INFO Completed push id=12 objects=15 duration=1.620735ms34252026/11/12 17:02:40 INFO Vacuumed table table=pending_closures34262026/11/12 17:02:40 INFO Received uploads request method=POST path=/34272026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts34282026/11/12 17:02:40 INFO Vacuumed table table=pending_objects34292026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads34302026/11/12 17:02:40 INFO Vacuumed table table=closures34312026/11/12 17:02:40 INFO Vacuumed table table=objects34322026/11/12 17:02:40 INFO Received push request method=POST path=/34332026/11/12 17:02:40 INFO Received push request method=POST path=/34342026/11/12 17:02:40 INFO Received complete push request method=POST path=/34352026/11/12 17:02:40 INFO Received push request method=POST path=/34362026/11/12 17:02:40 INFO Received uploads request method=POST path=/34372026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034382026/11/12 17:02:40 INFO Received push request method=POST path=/34392026/11/12 17:02:40 INFO Completed push id=5 objects=3 duration=1.584554ms34402026/11/12 17:02:40 INFO Received complete push request method=POST path=/34412026/11/12 17:02:40 INFO Received uploads request method=POST path=/34422026/11/12 17:02:40 INFO Received complete upload request method=POST path=/34432026/11/12 17:02:40 INFO Completed push id=5 objects=3 duration=2.634524ms34442026/11/12 17:02:40 INFO Completed upload id=6 objects=9 duration=1.104431ms34452026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034462026/11/12 17:02:40 INFO Received complete push request method=POST path=/34472026/11/12 17:02:40 INFO Received push request method=POST path=/34482026/11/12 17:02:40 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34492026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34502026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034512026/11/12 17:02:40 INFO Received complete upload request method=POST path=/34522026/11/12 17:02:40 INFO Completed push id=3 objects=21 duration=3.24279ms34532026/11/12 17:02:40 INFO Completed upload id=11 objects=14 duration=1.235432ms34542026/11/12 17:02:40 INFO Received push request method=POST path=/34552026/11/12 17:02:40 INFO Received push request method=POST path=/34562026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."34572026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=034582026/11/12 17:02:40 INFO Vacuumed table table=pending_closures34592026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts34602026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034612026/11/12 17:02:40 INFO Vacuumed table table=pending_objects34622026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads34632026/11/12 17:02:40 INFO Vacuumed table table=closures34642026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period34652026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034662026/11/12 17:02:40 INFO Vacuumed table table=objects34672026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034682026/11/12 17:02:40 INFO Vacuumed table table=pending_closures34692026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts34702026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034712026/11/12 17:02:40 INFO Vacuumed table table=pending_objects34722026/11/12 17:02:40 INFO Received push request method=POST path=/34732026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads34742026/11/12 17:02:40 INFO Vacuumed table table=closures34752026/11/12 17:02:40 INFO Received uploads request method=POST path=/34762026/11/12 17:02:40 INFO Received uploads request method=POST path=/34772026/11/12 17:02:40 INFO Vacuumed table table=objects34782026/11/12 17:02:40 INFO Received complete upload request method=POST path=/34792026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034802026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034812026/11/12 17:02:40 INFO Vacuumed table table=pending_closures34822026/11/12 17:02:40 INFO Vacuumed table table=pending_closures34832026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts34842026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034852026/11/12 17:02:40 INFO Completed upload id=8 objects=3 duration=2.25824ms34862026/11/12 17:02:40 INFO Vacuumed table table=pending_objects34872026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts34882026/11/12 17:02:40 INFO Received push request method=POST path=/34892026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads34902026/11/12 17:02:40 INFO Vacuumed table table=closures34912026/11/12 17:02:40 INFO Received uploads request method=POST path=/34922026/11/12 17:02:40 INFO Vacuumed table table=objects34932026/11/12 17:02:40 INFO Vacuumed table table=pending_objects34942026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads34952026/11/12 17:02:40 INFO Vacuumed table table=closures34962026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=034972026/11/12 17:02:40 INFO Vacuumed table table=objects34982026/11/12 17:02:40 INFO Received push request method=POST path=/34992026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=035002026/11/12 17:02:40 INFO Received push request method=POST path=/35012026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=035022026/11/12 17:02:40 INFO Received complete upload request method=POST path=/35032026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35042026/11/12 17:02:40 INFO Received push request method=POST path=/35052026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35062026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=035072026/11/12 17:02:40 INFO Completed upload id=14 objects=6 duration=1.236812ms35082026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35092026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35102026/11/12 17:02:40 INFO Received uploads request method=POST path=/35112026/11/12 17:02:40 INFO Received push request method=POST path=/35122026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35132026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35142026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35152026/11/12 17:02:40 INFO Received push request method=POST path=/35162026/11/12 17:02:40 INFO Received complete push request method=POST path=/35172026/11/12 17:02:40 INFO Received push request method=POST path=/35182026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35192026/11/12 17:02:40 INFO Vacuumed table table=closures35202026/11/12 17:02:40 INFO Received complete upload request method=POST path=/35212026/11/12 17:02:40 INFO Vacuumed table table=closures35222026/11/12 17:02:40 INFO Vacuumed table table=objects35232026/11/12 17:02:40 INFO Completed upload id=3 objects=5 duration=1.272192ms35242026/11/12 17:02:40 INFO Vacuumed table table=objects35252026/11/12 17:02:40 INFO Completed push id=19 objects=6 duration=2.845527ms35262026/11/12 17:02:40 INFO Received complete upload request method=POST path=/35272026/11/12 17:02:40 INFO Received push request method=POST path=/35282026/11/12 17:02:40 INFO Completed upload id=10 objects=2 duration=2.652485ms35292026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=035302026/11/12 17:02:40 INFO Received push request method=POST path=/35312026/11/12 17:02:40 INFO Received uploads request method=POST path=/35322026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035332026/11/12 17:02:40 INFO Received push request method=POST path=/35342026/11/12 17:02:40 INFO Received complete push request method=POST path=/35352026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35362026/11/12 17:02:40 INFO Completed push id=8 objects=6 duration=1.569974ms35372026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35382026/11/12 17:02:40 INFO Received push request method=POST path=/35392026/11/12 17:02:40 INFO Received complete push request method=POST path=/35402026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35412026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35422026/11/12 17:02:40 INFO Vacuumed table table=closures35432026/11/12 17:02:40 INFO Received push request method=POST path=/35442026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=035452026/11/12 17:02:40 INFO Vacuumed table table=objects35462026/11/12 17:02:40 INFO Completed push id=8 objects=6 duration=2.661725ms35472026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=035482026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35492026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=035502026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35512026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period35522026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35532026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35542026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035552026/11/12 17:02:40 INFO Vacuumed table table=closures35562026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=035572026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35582026/11/12 17:02:40 INFO Received uploads request method=POST path=/35592026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35602026/11/12 17:02:40 INFO Vacuumed table table=objects35612026/11/12 17:02:40 INFO Received push request method=POST path=/35622026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035632026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35642026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35652026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35662026/11/12 17:02:40 INFO Vacuumed table table=closures35672026/11/12 17:02:40 INFO Received uploads request method=POST path=/35682026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35692026/11/12 17:02:40 INFO Vacuumed table table=objects35702026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35712026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35722026/11/12 17:02:40 INFO Received push request method=POST path=/35732026/11/12 17:02:40 INFO Vacuumed table table=closures35742026/11/12 17:02:40 INFO Vacuumed table table=objects35752026/11/12 17:02:40 INFO Received complete upload request method=POST path=/35762026/11/12 17:02:40 INFO Completed upload id=7 objects=3 duration=777.247µs35772026/11/12 17:02:40 INFO Received complete push request method=POST path=/35782026/11/12 17:02:40 INFO Received uploads request method=POST path=/35792026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=035802026/11/12 17:02:40 INFO Received uploads request method=POST path=/35812026/11/12 17:02:40 INFO Completed push id=6 objects=6 duration=1.634275ms35822026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35832026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35842026/11/12 17:02:40 INFO Received push request method=POST path=/35852026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35862026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35872026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=035882026/11/12 17:02:40 INFO Vacuumed table table=closures35892026/11/12 17:02:40 INFO Received push request method=POST path=/35902026/11/12 17:02:40 INFO Vacuumed table table=objects35912026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period35922026/11/12 17:02:40 INFO Received uploads request method=POST path=/35932026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=035942026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035952026/11/12 17:02:40 INFO Vacuumed table table=pending_closures35962026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts35972026/11/12 17:02:40 INFO Vacuumed table table=pending_objects35982026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads35992026/11/12 17:02:40 INFO Received push request method=POST path=/36002026/11/12 17:02:40 INFO Vacuumed table table=closures36012026/11/12 17:02:40 INFO Received uploads request method=POST path=/36022026/11/12 17:02:40 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36032026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36042026/11/12 17:02:40 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."36052026/11/12 17:02:40 INFO Vacuumed table table=objects36062026/11/12 17:02:40 INFO Received uploads request method=POST path=/36072026/11/12 17:02:40 INFO Received uploads request method=POST path=/36082026/11/12 17:02:40 INFO Received complete push request method=POST path=/36092026/11/12 17:02:40 INFO Completed push id=18 objects=23 duration=1.693056ms36102026/11/12 17:02:40 INFO Received complete push request method=POST path=/36112026/11/12 17:02:40 INFO Received push request method=POST path=/36122026/11/12 17:02:40 INFO Completed push id=15 objects=5 duration=1.880357ms36132026/11/12 17:02:40 INFO Received uploads request method=POST path=/36142026/11/12 17:02:40 INFO Received complete push request method=POST path=/36152026/11/12 17:02:40 INFO Received push request method=POST path=/36162026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=036172026/11/12 17:02:40 INFO Received uploads request method=POST path=/36182026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period36192026/11/12 17:02:40 INFO Completed push id=2 objects=10 duration=3.050148ms36202026/11/12 17:02:40 INFO Received push request method=POST path=/36212026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=036222026/11/12 17:02:40 INFO Received complete push request method=POST path=/36232026/11/12 17:02:40 INFO Received uploads request method=POST path=/36242026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period36252026/11/12 17:02:40 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36262026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36272026/11/12 17:02:40 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."36282026/11/12 17:02:40 INFO Received uploads request method=POST path=/36292026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036302026/11/12 17:02:40 INFO Completed push id=14 objects=3 duration=3.006328ms36312026/11/12 17:02:40 INFO Received complete upload request method=POST path=/36322026/11/12 17:02:40 INFO Vacuumed table table=pending_closures36332026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts36342026/11/12 17:02:40 INFO Vacuumed table table=pending_objects36352026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads36362026/11/12 17:02:40 INFO Completed upload id=15 objects=4 duration=1.946398ms36372026/11/12 17:02:40 INFO Received complete push request method=POST path=/36382026/11/12 17:02:40 INFO Vacuumed table table=closures36392026/11/12 17:02:40 INFO Vacuumed table table=objects36402026/11/12 17:02:40 INFO Received uploads request method=POST path=/36412026/11/12 17:02:40 INFO Received complete push request method=POST path=/36422026/11/12 17:02:40 INFO Completed push id=5 objects=9 duration=3.885676ms36432026/11/12 17:02:40 INFO Received push request method=POST path=/36442026/11/12 17:02:40 INFO Completed push id=15 objects=24 duration=3.145429ms36452026/11/12 17:02:40 INFO Received complete push request method=POST path=/36462026/11/12 17:02:40 INFO Completed push id=10 objects=15 duration=1.469653ms36472026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=036482026/11/12 17:02:40 INFO Received push request method=POST path=/36492026/11/12 17:02:40 INFO Received complete push request method=POST path=/36502026/11/12 17:02:40 INFO Received complete push request method=POST path=/36512026/11/12 17:02:40 INFO Received complete push request method=POST path=/36522026/11/12 17:02:40 INFO Completed push id=11 objects=9 duration=1.737076ms36532026/11/12 17:02:40 INFO Completed push id=9 objects=24 duration=1.697196ms36542026/11/12 17:02:40 INFO Received push request method=POST path=/36552026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036562026/11/12 17:02:40 INFO Received push request method=POST path=/36572026/11/12 17:02:40 INFO Completed push id=11 objects=8 duration=3.176529ms36582026/11/12 17:02:40 INFO Vacuumed table table=pending_closures36592026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts36602026/11/12 17:02:40 INFO Received uploads request method=POST path=/36612026/11/12 17:02:40 INFO Vacuumed table table=pending_objects36622026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads36632026/11/12 17:02:40 INFO Vacuumed table table=closures36642026/11/12 17:02:40 INFO Vacuumed table table=objects36652026/11/12 17:02:40 INFO Received complete push request method=POST path=/36662026/11/12 17:02:40 INFO Completed push id=5 objects=6 duration=1.332592ms36672026/11/12 17:02:40 INFO Received push request method=POST path=/36682026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=036692026/11/12 17:02:40 INFO Received complete upload request method=POST path=/36702026/11/12 17:02:40 INFO Received uploads request method=POST path=/36712026/11/12 17:02:40 INFO Received push request method=POST path=/36722026/11/12 17:02:40 INFO Completed upload id=10 objects=17 duration=1.124511ms36732026/11/12 17:02:40 INFO Received push request method=POST path=/36742026/11/12 17:02:40 INFO Received complete upload request method=POST path=/36752026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036762026/11/12 17:02:40 INFO Received uploads request method=POST path=/36772026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=036782026/11/12 17:02:40 INFO Completed upload id=15 objects=6 duration=751.347µs36792026/11/12 17:02:40 INFO Received push request method=POST path=/36802026/11/12 17:02:40 INFO Vacuumed table table=pending_closures36812026/11/12 17:02:40 INFO Received complete push request method=POST path=/36822026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts36832026/11/12 17:02:40 INFO Vacuumed table table=pending_objects36842026/11/12 17:02:40 INFO Completed push id=8 objects=7 duration=1.804417ms36852026/11/12 17:02:40 INFO Received uploads request method=POST path=/36862026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads36872026/11/12 17:02:40 INFO Vacuumed table table=closures36882026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036892026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=036902026/11/12 17:02:40 INFO Vacuumed table table=objects36912026/11/12 17:02:40 INFO Vacuumed table table=pending_closures36922026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts36932026/11/12 17:02:40 INFO Vacuumed table table=pending_objects36942026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period36952026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads36962026/11/12 17:02:40 INFO Received uploads request method=POST path=/36972026/11/12 17:02:40 INFO Vacuumed table table=closures36982026/11/12 17:02:40 INFO Received complete push request method=POST path=/36992026/11/12 17:02:40 INFO Vacuumed table table=objects37002026/11/12 17:02:40 INFO Received complete upload request method=POST path=/37012026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037022026/11/12 17:02:40 INFO Completed upload id=16 objects=3 duration=953.329µs37032026/11/12 17:02:40 INFO Completed push id=15 objects=7 duration=1.855318ms37042026/11/12 17:02:40 INFO Received push request method=POST path=/37052026/11/12 17:02:40 INFO Vacuumed table table=pending_closures37062026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts37072026/11/12 17:02:40 INFO Vacuumed table table=pending_objects37082026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads37092026/11/12 17:02:40 INFO Vacuumed table table=closures37102026/11/12 17:02:40 INFO Received push request method=POST path=/37112026/11/12 17:02:40 INFO Vacuumed table table=objects37122026/11/12 17:02:40 INFO Received uploads request method=POST path=/37132026/11/12 17:02:40 INFO Received uploads request method=POST path=/37142026/11/12 17:02:40 INFO Received uploads request method=POST path=/37152026/11/12 17:02:40 INFO Received push request method=POST path=/37162026/11/12 17:02:40 INFO Received push request method=POST path=/37172026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=037182026/11/12 17:02:40 INFO Received uploads request method=POST path=/37192026/11/12 17:02:40 INFO Received complete push request method=POST path=/37202026/11/12 17:02:40 INFO Received uploads request method=POST path=/37212026/11/12 17:02:40 INFO Received complete push request method=POST path=/37222026/11/12 17:02:40 INFO Received uploads request method=POST path=/37232026/11/12 17:02:40 INFO Received complete push request method=POST path=/37242026/11/12 17:02:40 INFO Completed push id=13 objects=6 duration=2.667505ms37252026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037262026/11/12 17:02:40 INFO Received complete upload request method=POST path=/37272026/11/12 17:02:40 INFO Received uploads request method=POST path=/37282026/11/12 17:02:40 INFO Vacuumed table table=pending_closures37292026/11/12 17:02:40 INFO Completed push id=9 objects=4 duration=2.871027ms37302026/11/12 17:02:40 INFO Completed upload id=10 objects=10 duration=1.155611ms37312026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts37322026/11/12 17:02:40 INFO Received uploads request method=POST path=/37332026/11/12 17:02:40 INFO Completed push id=9 objects=7 duration=3.28065ms37342026/11/12 17:02:40 INFO Vacuumed table table=pending_objects37352026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads37362026/11/12 17:02:40 INFO Received uploads request method=POST path=/37372026/11/12 17:02:40 INFO Vacuumed table table=closures37382026/11/12 17:02:40 INFO Vacuumed table table=objects37392026/11/12 17:02:40 INFO Received push request method=POST path=/37402026/11/12 17:02:40 INFO Received push request method=POST path=/37412026/11/12 17:02:40 INFO Received complete upload request method=POST path=/37422026/11/12 17:02:40 INFO Completed upload id=9 objects=16 duration=1.325432ms37432026/11/12 17:02:40 INFO Received uploads request method=POST path=/37442026/11/12 17:02:40 INFO Received push request method=POST path=/37452026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=037462026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=037472026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period37482026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period37492026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037502026/11/12 17:02:40 INFO Vacuumed table table=pending_closures37512026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts37522026/11/12 17:02:40 INFO Vacuumed table table=pending_objects37532026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads37542026/11/12 17:02:40 INFO Received push request method=POST path=/37552026/11/12 17:02:40 INFO Vacuumed table table=closures37562026/11/12 17:02:40 INFO Vacuumed table table=objects37572026/11/12 17:02:40 INFO Received uploads request method=POST path=/37582026/11/12 17:02:40 INFO Received complete upload request method=POST path=/37592026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037602026/11/12 17:02:40 INFO Completed upload id=6 objects=6 duration=964.409µs37612026/11/12 17:02:40 INFO Received uploads request method=POST path=/37622026/11/12 17:02:40 INFO Received complete push request method=POST path=/37632026/11/12 17:02:40 INFO Vacuumed table table=pending_closures37642026/11/12 17:02:40 INFO Received push request method=POST path=/37652026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts37662026/11/12 17:02:40 INFO Received complete upload request method=POST path=/37672026/11/12 17:02:40 INFO Completed push id=14 objects=6 duration=1.258691ms37682026/11/12 17:02:40 INFO Vacuumed table table=pending_objects37692026/11/12 17:02:40 INFO Completed upload id=10 objects=7 duration=1.02355ms37702026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads37712026/11/12 17:02:40 INFO Vacuumed table table=closures37722026/11/12 17:02:40 INFO Vacuumed table table=objects37732026/11/12 17:02:40 INFO Received complete push request method=POST path=/37742026/11/12 17:02:40 INFO Completed push id=16 objects=27 duration=1.406733ms37752026/11/12 17:02:40 INFO Received uploads request method=POST path=/37762026/11/12 17:02:40 INFO Received uploads request method=POST path=/37772026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=037782026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=037792026/11/12 17:02:40 INFO Received push request method=POST path=/37802026/11/12 17:02:40 INFO Received push request method=POST path=/37812026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period37822026/11/12 17:02:40 INFO Received uploads request method=POST path=/37832026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period37842026/11/12 17:02:40 INFO Received push request method=POST path=/37852026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037862026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037872026/11/12 17:02:40 INFO Vacuumed table table=pending_closures37882026/11/12 17:02:40 INFO Received push request method=POST path=/37892026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts37902026/11/12 17:02:40 INFO Vacuumed table table=pending_closures37912026/11/12 17:02:40 INFO Received uploads request method=POST path=/37922026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts37932026/11/12 17:02:40 INFO Vacuumed table table=pending_objects37942026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads37952026/11/12 17:02:40 INFO Vacuumed table table=closures37962026/11/12 17:02:40 INFO Vacuumed table table=pending_objects37972026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads37982026/11/12 17:02:40 INFO Received push request method=POST path=/37992026/11/12 17:02:40 INFO Vacuumed table table=objects38002026/11/12 17:02:40 INFO Vacuumed table table=closures38012026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038022026/11/12 17:02:40 INFO Vacuumed table table=objects38032026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period38042026/11/12 17:02:40 INFO Received complete upload request method=POST path=/38052026/11/12 17:02:40 INFO Received complete push request method=POST path=/38062026/11/12 17:02:40 INFO Completed upload id=18 objects=4 duration=1.482574ms38072026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038082026/11/12 17:02:40 INFO Vacuumed table table=pending_closures38092026/11/12 17:02:40 INFO Completed push id=17 objects=6 duration=1.581095ms38102026/11/12 17:02:40 INFO Received push request method=POST path=/38112026/11/12 17:02:40 INFO Received complete push request method=POST path=/38122026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts38132026/11/12 17:02:40 INFO Completed push id=10 objects=3 duration=1.177291ms38142026/11/12 17:02:40 INFO Vacuumed table table=pending_objects38152026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads38162026/11/12 17:02:40 INFO Received complete upload request method=POST path=/38172026/11/12 17:02:40 INFO Vacuumed table table=closures38182026/11/12 17:02:40 INFO Vacuumed table table=objects38192026/11/12 17:02:40 INFO Completed upload id=22 objects=2 duration=928.168µs38202026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038212026/11/12 17:02:40 INFO Received push request method=POST path=/38222026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period38232026/11/12 17:02:40 INFO Received push request method=POST path=/38242026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038252026/11/12 17:02:40 INFO Received complete upload request method=POST path=/38262026/11/12 17:02:40 INFO Received push request method=POST path=/38272026/11/12 17:02:40 INFO Vacuumed table table=pending_closures38282026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts38292026/11/12 17:02:40 INFO Vacuumed table table=pending_objects38302026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads38312026/11/12 17:02:40 INFO Received push request method=POST path=/38322026/11/12 17:02:40 INFO Completed upload id=10 objects=6 duration=2.004018ms38332026/11/12 17:02:40 INFO Vacuumed table table=closures38342026/11/12 17:02:40 INFO Vacuumed table table=objects38352026/11/12 17:02:40 INFO Received uploads request method=POST path=/38362026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038372026/11/12 17:02:40 INFO Received uploads request method=POST path=/38382026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038392026/11/12 17:02:40 INFO Received uploads request method=POST path=/38402026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=038412026/11/12 17:02:40 INFO Vacuumed table table=pending_closures38422026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts38432026/11/12 17:02:40 INFO Vacuumed table table=pending_objects38442026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads38452026/11/12 17:02:40 INFO Received complete push request method=POST path=/38462026/11/12 17:02:40 INFO Vacuumed table table=closures38472026/11/12 17:02:40 INFO Received uploads request method=POST path=/38482026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038492026/11/12 17:02:40 INFO Received complete push request method=POST path=/38502026/11/12 17:02:40 INFO Vacuumed table table=objects38512026/11/12 17:02:40 INFO Received complete push request method=POST path=/38522026/11/12 17:02:40 INFO Vacuumed table table=pending_closures38532026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts38542026/11/12 17:02:40 INFO Received complete upload request method=POST path=/38552026/11/12 17:02:40 INFO Completed push id=4 objects=7 duration=1.450733ms38562026/11/12 17:02:40 INFO Completed push id=6 objects=26 duration=3.114129ms38572026/11/12 17:02:40 INFO Received complete push request method=POST path=/38582026/11/12 17:02:40 INFO Completed push id=17 objects=6 duration=1.440413ms38592026/11/12 17:02:40 INFO Vacuumed table table=pending_objects38602026/11/12 17:02:40 INFO Received complete upload request method=POST path=/38612026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads38622026/11/12 17:02:40 INFO Completed push id=18 objects=10 duration=1.228631ms38632026/11/12 17:02:40 INFO Completed upload id=18 objects=4 duration=950.349µs38642026/11/12 17:02:40 INFO Vacuumed table table=closures38652026/11/12 17:02:40 INFO Completed upload id=23 objects=15 duration=1.910438ms38662026/11/12 17:02:40 INFO Vacuumed table table=objects38672026/11/12 17:02:40 INFO Received push request method=POST path=/38682026/11/12 17:02:40 INFO Received uploads request method=POST path=/38692026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038702026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period38712026/11/12 17:02:40 INFO Received uploads request method=POST path=/38722026/11/12 17:02:40 INFO Received uploads request method=POST path=/38732026/11/12 17:02:40 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."38742026/11/12 17:02:40 INFO Received uploads request method=POST path=/38752026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038762026/11/12 17:02:40 INFO Received push request method=POST path=/38772026/11/12 17:02:40 INFO Received complete push request method=POST path=/38782026/11/12 17:02:40 INFO Received uploads request method=POST path=/38792026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period38802026/11/12 17:02:40 INFO Completed push id=17 objects=22 duration=1.917418ms38812026/11/12 17:02:40 INFO Received complete push request method=POST path=/38822026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038832026/11/12 17:02:40 INFO Received complete push request method=POST path=/38842026/11/12 17:02:40 INFO Received complete push request method=POST path=/38852026/11/12 17:02:40 INFO Vacuumed table table=pending_closures38862026/11/12 17:02:40 INFO Completed push id=14 objects=3 duration=929.569µs38872026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts38882026/11/12 17:02:40 INFO Received push request method=POST path=/38892026/11/12 17:02:40 INFO Completed push id=14 objects=3 duration=2.420323ms38902026/11/12 17:02:40 INFO Vacuumed table table=pending_objects38912026/11/12 17:02:40 INFO Received push request method=POST path=/38922026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads38932026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038942026/11/12 17:02:40 INFO Vacuumed table table=closures38952026/11/12 17:02:40 INFO Completed push id=11 objects=11 duration=3.080668ms38962026/11/12 17:02:40 INFO Vacuumed table table=objects38972026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=038982026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period38992026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period39002026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039012026/11/12 17:02:40 INFO Received complete push request method=POST path=/39022026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39032026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039042026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39052026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39062026/11/12 17:02:40 INFO Completed push id=8 objects=6 duration=1.365732ms39072026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39082026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39092026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads39102026/11/12 17:02:40 INFO Received complete upload request method=POST path=/39112026/11/12 17:02:40 INFO Vacuumed table table=closures39122026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39132026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads39142026/11/12 17:02:40 INFO Vacuumed table table=objects39152026/11/12 17:02:40 INFO Completed upload id=19 objects=6 duration=1.09439ms39162026/11/12 17:02:40 INFO Vacuumed table table=closures39172026/11/12 17:02:40 INFO Vacuumed table table=objects39182026/11/12 17:02:40 INFO Received push request method=POST path=/39192026/11/12 17:02:40 INFO Received push request method=POST path=/39202026/11/12 17:02:40 INFO Received uploads request method=POST path=/39212026/11/12 17:02:40 INFO Received complete push request method=POST path=/39222026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=039232026/11/12 17:02:40 INFO Received uploads request method=POST path=/39242026/11/12 17:02:40 INFO Received complete upload request method=POST path=/39252026/11/12 17:02:40 INFO Completed push id=21 objects=9 duration=1.916858ms39262026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=039272026/11/12 17:02:40 INFO Completed upload id=14 objects=8 duration=1.903337ms39282026/11/12 17:02:40 INFO Received push request method=POST path=/39292026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039302026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period39312026/11/12 17:02:40 INFO Received complete push request method=POST path=/39322026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39332026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39342026/11/12 17:02:40 INFO Completed push id=12 objects=7 duration=1.153271ms39352026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039362026/11/12 17:02:40 INFO Received uploads request method=POST path=/39372026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39382026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads39392026/11/12 17:02:40 INFO Vacuumed table table=closures39402026/11/12 17:02:40 INFO Vacuumed table table=objects39412026/11/12 17:02:40 INFO Received complete upload request method=POST path=/39422026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39432026/11/12 17:02:40 INFO Received push request method=POST path=/39442026/11/12 17:02:40 INFO Completed upload id=22 objects=7 duration=676.686µs39452026/11/12 17:02:40 INFO Received uploads request method=POST path=/39462026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39472026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39482026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads39492026/11/12 17:02:40 INFO Vacuumed table table=closures39502026/11/12 17:02:40 INFO Received complete push request method=POST path=/39512026/11/12 17:02:40 INFO Vacuumed table table=objects39522026/11/12 17:02:40 INFO Received uploads request method=POST path=/39532026/11/12 17:02:40 INFO Completed push id=4 objects=6 duration=1.441854ms39542026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=039552026/11/12 17:02:40 INFO Received complete push request method=POST path=/39562026/11/12 17:02:40 INFO Received push request method=POST path=/39572026/11/12 17:02:40 INFO Completed push id=7 objects=7 duration=1.191712ms39582026/11/12 17:02:40 INFO Received push request method=POST path=/39592026/11/12 17:02:40 INFO Received complete push request method=POST path=/39602026/11/12 17:02:40 INFO Received uploads request method=POST path=/39612026/11/12 17:02:40 INFO Completed push id=12 objects=2 duration=2.562683ms39622026/11/12 17:02:40 INFO Received uploads request method=POST path=/39632026/11/12 17:02:40 INFO Received uploads request method=POST path=/39642026/11/12 17:02:40 INFO Received complete upload request method=POST path=/39652026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=039662026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=039672026/11/12 17:02:40 INFO Completed upload id=7 objects=10 duration=1.229671ms39682026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period39692026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39702026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39712026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039722026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=039732026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39742026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39752026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=039762026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads39772026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39782026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=039792026/11/12 17:02:40 INFO Vacuumed table table=closures39802026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39812026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period39822026/11/12 17:02:40 INFO Vacuumed table table=objects39832026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads39842026/11/12 17:02:40 INFO Received uploads request method=POST path=/39852026/11/12 17:02:40 INFO Received uploads request method=POST path=/39862026/11/12 17:02:40 INFO Vacuumed table table=closures39872026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period39882026/11/12 17:02:40 INFO Vacuumed table table=objects39892026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039902026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039912026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39922026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39932026/11/12 17:02:40 INFO Vacuumed table table=pending_closures39942026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts39952026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39962026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads39972026/11/12 17:02:40 INFO Vacuumed table table=pending_objects39982026/11/12 17:02:40 INFO Vacuumed table table=closures39992026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads40002026/11/12 17:02:40 INFO Vacuumed table table=closures40012026/11/12 17:02:40 INFO Received complete push request method=POST path=/40022026/11/12 17:02:40 INFO Vacuumed table table=objects40032026/11/12 17:02:40 INFO Vacuumed table table=objects40042026/11/12 17:02:40 INFO Completed push id=1 objects=11 duration=1.513574ms40052026/11/12 17:02:40 INFO Received complete upload request method=POST path=/40062026/11/12 17:02:40 INFO Received complete push request method=POST path=/40072026/11/12 17:02:40 INFO Completed upload id=9 objects=2 duration=1.991438ms40082026/11/12 17:02:40 INFO Completed push id=24 objects=9 duration=1.584735ms40092026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=040102026/11/12 17:02:40 INFO Received push request method=POST path=/40112026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40122026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period40132026/11/12 17:02:40 INFO Received push request method=POST path=/40142026/11/12 17:02:40 INFO Received push request method=POST path=/40152026/11/12 17:02:40 INFO Received push request method=POST path=/40162026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040172026/11/12 17:02:40 INFO Received complete upload request method=POST path=/40182026/11/12 17:02:40 INFO Vacuumed table table=pending_closures40192026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts40202026/11/12 17:02:40 INFO Completed upload id=18 objects=11 duration=1.05761ms40212026/11/12 17:02:40 INFO Vacuumed table table=pending_objects40222026/11/12 17:02:40 INFO Received complete push request method=POST path=/40232026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads40242026/11/12 17:02:40 INFO Vacuumed table table=closures40252026/11/12 17:02:40 INFO Vacuumed table table=objects40262026/11/12 17:02:40 INFO Completed push id=8 objects=3 duration=1.455114ms40272026/11/12 17:02:40 INFO Received complete upload request method=POST path=/40282026/11/12 17:02:40 INFO Completed upload id=16 objects=3 duration=846.668µs40292026/11/12 17:02:40 INFO Received push request method=POST path=/40302026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=040312026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period40322026/11/12 17:02:40 INFO Received uploads request method=POST path=/40332026/11/12 17:02:40 INFO Received push request method=POST path=/40342026/11/12 17:02:40 INFO Received push request method=POST path=/40352026/11/12 17:02:40 INFO Received push request method=POST path=/40362026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040372026/11/12 17:02:40 INFO Vacuumed table table=pending_closures40382026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts40392026/11/12 17:02:40 INFO Received uploads request method=POST path=/40402026/11/12 17:02:40 INFO Received uploads request method=POST path=/40412026/11/12 17:02:40 INFO Vacuumed table table=pending_objects40422026/11/12 17:02:40 INFO Received uploads request method=POST path=/40432026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads40442026/11/12 17:02:40 INFO Received push request method=POST path=/40452026/11/12 17:02:40 INFO Received push request method=POST path=/40462026/11/12 17:02:40 INFO Vacuumed table table=closures40472026/11/12 17:02:40 INFO Received push request method=POST path=/40482026/11/12 17:02:40 INFO Vacuumed table table=objects40492026/11/12 17:02:40 INFO Received push request method=POST path=/40502026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=040512026/11/12 17:02:40 INFO Received push request method=POST path=/40522026/11/12 17:02:40 INFO Received uploads request method=POST path=/40532026/11/12 17:02:40 INFO Received push request method=POST path=/40542026/11/12 17:02:40 INFO Received push request method=POST path=/40552026/11/12 17:02:40 INFO Received push request method=POST path=/40562026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040572026/11/12 17:02:40 INFO Received uploads request method=POST path=/40582026/11/12 17:02:40 INFO Received uploads request method=POST path=/40592026/11/12 17:02:40 INFO Vacuumed table table=pending_closures40602026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts40612026/11/12 17:02:40 INFO Vacuumed table table=pending_objects40622026/11/12 17:02:40 INFO Received uploads request method=POST path=/40632026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads40642026/11/12 17:02:40 INFO Vacuumed table table=closures40652026/11/12 17:02:40 INFO Received push request method=POST path=/40662026/11/12 17:02:40 INFO Vacuumed table table=objects40672026/11/12 17:02:40 INFO Received uploads request method=POST path=/40682026/11/12 17:02:40 INFO Received complete upload request method=POST path=/40692026/11/12 17:02:40 INFO Received complete upload request method=POST path=/40702026/11/12 17:02:40 INFO Completed upload id=11 objects=7 duration=1.09715ms40712026/11/12 17:02:40 INFO Completed upload id=19 objects=25 duration=2.22474ms40722026/11/12 17:02:40 INFO Received complete upload request method=POST path=/40732026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=040742026/11/12 17:02:40 INFO Completed upload id=13 objects=10 duration=759.007µs40752026/11/12 17:02:40 INFO Received uploads request method=POST path=/40762026/11/12 17:02:40 INFO Received uploads request method=POST path=/40772026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040782026/11/12 17:02:40 INFO Received push request method=POST path=/40792026/11/12 17:02:40 INFO Vacuumed table table=pending_closures40802026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=040812026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts40822026/11/12 17:02:40 INFO Received uploads request method=POST path=/40832026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period40842026/11/12 17:02:40 INFO Vacuumed table table=pending_objects40852026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads40862026/11/12 17:02:40 INFO Received uploads request method=POST path=/40872026/11/12 17:02:40 INFO Vacuumed table table=closures40882026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040892026/11/12 17:02:40 INFO Vacuumed table table=objects40902026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=040912026/11/12 17:02:40 INFO Vacuumed table table=pending_closures40922026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts40932026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=040942026/11/12 17:02:40 INFO Vacuumed table table=pending_objects40952026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period40962026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads40972026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period40982026/11/12 17:02:40 INFO Vacuumed table table=closures40992026/11/12 17:02:40 INFO Vacuumed table table=objects41002026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041012026/11/12 17:02:40 INFO Received uploads request method=POST path=/41022026/11/12 17:02:40 INFO Received push request method=POST path=/41032026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period41042026/11/12 17:02:40 INFO Received complete upload request method=POST path=/41052026/11/12 17:02:40 INFO Received complete upload request method=POST path=/41062026/11/12 17:02:40 INFO Received uploads request method=POST path=/41072026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041082026/11/12 17:02:40 INFO Completed upload id=17 objects=6 duration=1.777737ms41092026/11/12 17:02:40 INFO Completed upload id=23 objects=16 duration=2.05994ms41102026/11/12 17:02:40 INFO Vacuumed table table=pending_closures41112026/11/12 17:02:40 INFO Received uploads request method=POST path=/41122026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts41132026/11/12 17:02:40 INFO Vacuumed table table=pending_objects41142026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads41152026/11/12 17:02:40 INFO Vacuumed table table=closures41162026/11/12 17:02:40 INFO Received uploads request method=POST path=/41172026/11/12 17:02:40 INFO Vacuumed table table=objects41182026/11/12 17:02:40 INFO Received uploads request method=POST path=/41192026/11/12 17:02:40 INFO Received complete upload request method=POST path=/41202026/11/12 17:02:40 INFO Completed upload id=16 objects=17 duration=2.16166ms41212026/11/12 17:02:40 INFO Received push request method=POST path=/41222026/11/12 17:02:40 INFO Received push request method=POST path=/41232026/11/12 17:02:40 INFO Received push request method=POST path=/41242026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."41252026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041262026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period41272026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041282026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041292026/11/12 17:02:40 INFO Vacuumed table table=pending_closures41302026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts41312026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041322026/11/12 17:02:40 INFO Vacuumed table table=pending_objects41332026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads41342026/11/12 17:02:40 INFO Vacuumed table table=pending_closures41352026/11/12 17:02:40 INFO Vacuumed table table=closures41362026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts41372026/11/12 17:02:40 INFO Vacuumed table table=objects41382026/11/12 17:02:40 INFO Vacuumed table table=pending_objects41392026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads41402026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041412026/11/12 17:02:40 INFO Vacuumed table table=closures41422026/11/12 17:02:40 INFO Received uploads request method=POST path=/41432026/11/12 17:02:40 INFO Vacuumed table table=objects41442026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period41452026/11/12 17:02:40 INFO Received uploads request method=POST path=/41462026/11/12 17:02:40 INFO Received uploads request method=POST path=/41472026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041482026/11/12 17:02:40 INFO Received push request method=POST path=/41492026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041502026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period41512026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041522026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=041532026/11/12 17:02:40 INFO Vacuumed table table=pending_closures41542026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts41552026/11/12 17:02:40 INFO Vacuumed table table=pending_closures41562026/11/12 17:02:40 INFO Received push request method=POST path=/41572026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts41582026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041592026/11/12 17:02:40 INFO Received uploads request method=POST path=/41602026/11/12 17:02:40 INFO Vacuumed table table=pending_objects41612026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads41622026/11/12 17:02:40 INFO Vacuumed table table=pending_closures41632026/11/12 17:02:40 INFO Vacuumed table table=pending_objects41642026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads41652026/11/12 17:02:40 INFO Received uploads request method=POST path=/41662026/11/12 17:02:40 INFO Vacuumed table table=closures41672026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts41682026/11/12 17:02:40 INFO Vacuumed table table=closures41692026/11/12 17:02:40 INFO Vacuumed table table=objects41702026/11/12 17:02:40 INFO Vacuumed table table=pending_objects41712026/11/12 17:02:40 INFO Vacuumed table table=objects41722026/11/12 17:02:40 INFO Received complete push request method=POST path=/41732026/11/12 17:02:40 INFO Received push request method=POST path=/41742026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads41752026/11/12 17:02:40 INFO Received uploads request method=POST path=/41762026/11/12 17:02:40 INFO Vacuumed table table=closures41772026/11/12 17:02:40 INFO Vacuumed table table=objects41782026/11/12 17:02:40 INFO Completed push id=22 objects=3 duration=2.472263ms41792026/11/12 17:02:40 INFO Received complete push request method=POST path=/41802026/11/12 17:02:40 INFO Received complete push request method=POST path=/41812026/11/12 17:02:40 INFO Completed push id=4 objects=6 duration=1.334713ms41822026/11/12 17:02:40 INFO Completed push id=15 objects=4 duration=880.768µs41832026/11/12 17:02:40 INFO Received complete push request method=POST path=/41842026/11/12 17:02:40 INFO Completed push id=21 objects=3 duration=1.173951ms41852026/11/12 17:02:40 INFO Received complete upload request method=POST path=/41862026/11/12 17:02:40 INFO Completed upload id=13 objects=8 duration=1.04325ms41872026/11/12 17:02:40 INFO Received push request method=POST path=/41882026/11/12 17:02:40 INFO Received uploads request method=POST path=/41892026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041902026/11/12 17:02:40 INFO Received uploads request method=POST path=/41912026/11/12 17:02:40 INFO Received push request method=POST path=/41922026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period41932026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=041942026/11/12 17:02:40 INFO Received push request method=POST path=/41952026/11/12 17:02:40 INFO Received uploads request method=POST path=/41962026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period41972026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041982026/11/12 17:02:40 INFO Vacuumed table table=pending_closures41992026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts42002026/11/12 17:02:40 INFO Received uploads request method=POST path=/42012026/11/12 17:02:40 INFO Vacuumed table table=pending_objects42022026/11/12 17:02:40 INFO Received uploads request method=POST path=/42032026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads42042026/11/12 17:02:40 INFO Vacuumed table table=closures42052026/11/12 17:02:40 INFO Received push request method=POST path=/42062026/11/12 17:02:40 INFO Vacuumed table table=objects42072026/11/12 17:02:40 INFO Received uploads request method=POST path=/42082026/11/12 17:02:40 INFO Received uploads request method=POST path=/42092026/11/12 17:02:40 INFO Received uploads request method=POST path=/42102026/11/12 17:02:40 INFO Received push request method=POST path=/42112026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=042122026/11/12 17:02:40 INFO Vacuumed table table=pending_closures42132026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts42142026/11/12 17:02:40 INFO Received uploads request method=POST path=/42152026/11/12 17:02:40 INFO Received push request method=POST path=/42162026/11/12 17:02:40 INFO Vacuumed table table=pending_objects42172026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads42182026/11/12 17:02:40 INFO Vacuumed table table=closures42192026/11/12 17:02:40 INFO Vacuumed table table=objects42202026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42212026/11/12 17:02:40 INFO Received uploads request method=POST path=/42222026/11/12 17:02:40 INFO Received complete push request method=POST path=/42232026/11/12 17:02:40 INFO Received uploads request method=POST path=/42242026/11/12 17:02:40 INFO Received complete upload request method=POST path=/42252026/11/12 17:02:40 INFO Received complete push request method=POST path=/42262026/11/12 17:02:40 INFO Received uploads request method=POST path=/42272026/11/12 17:02:40 INFO Completed push id=14 objects=4 duration=2.751285ms42282026/11/12 17:02:40 INFO Received uploads request method=POST path=/42292026/11/12 17:02:40 INFO Completed upload id=23 objects=18 duration=2.13024ms42302026/11/12 17:02:40 INFO Received complete push request method=POST path=/42312026/11/12 17:02:40 INFO Completed push id=19 objects=8 duration=3.124929ms42322026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=042332026/11/12 17:02:40 INFO Completed push id=13 objects=15 duration=1.637715ms42342026/11/12 17:02:40 INFO Received push request method=POST path=/42352026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=042362026/11/12 17:02:40 INFO Received uploads request method=POST path=/42372026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042382026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=042392026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period42402026/11/12 17:02:40 INFO Vacuumed table table=pending_closures42412026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts42422026/11/12 17:02:40 INFO Received push request method=POST path=/42432026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042442026/11/12 17:02:40 INFO Vacuumed table table=pending_objects42452026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads42462026/11/12 17:02:40 INFO Vacuumed table table=pending_closures42472026/11/12 17:02:40 INFO Received push request method=POST path=/42482026/11/12 17:02:40 INFO Vacuumed table table=closures42492026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts42502026/11/12 17:02:40 INFO Received complete push request method=POST path=/42512026/11/12 17:02:40 INFO Vacuumed table table=objects42522026/11/12 17:02:40 INFO Vacuumed table table=pending_objects42532026/11/12 17:02:40 INFO Received push request method=POST path=/42542026/11/12 17:02:40 INFO Completed push id=14 objects=3 duration=1.001469ms42552026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads42562026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042572026/11/12 17:02:40 INFO Received uploads request method=POST path=/42582026/11/12 17:02:40 INFO Received uploads request method=POST path=/42592026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=042602026/11/12 17:02:40 INFO Vacuumed table table=closures42612026/11/12 17:02:40 INFO Vacuumed table table=pending_closures42622026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts42632026/11/12 17:02:40 INFO Vacuumed table table=objects42642026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period42652026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=042662026/11/12 17:02:40 INFO Received complete push request method=POST path=/42672026/11/12 17:02:40 INFO Vacuumed table table=pending_objects42682026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads42692026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042702026/11/12 17:02:40 INFO Vacuumed table table=closures42712026/11/12 17:02:40 INFO Vacuumed table table=pending_closures42722026/11/12 17:02:40 INFO Received push request method=POST path=/42732026/11/12 17:02:40 INFO Completed push id=26 objects=14 duration=2.011679ms42742026/11/12 17:02:40 INFO Vacuumed table table=objects42752026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts42762026/11/12 17:02:40 INFO Vacuumed table table=pending_objects42772026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads42782026/11/12 17:02:40 INFO Vacuumed table table=closures42792026/11/12 17:02:40 INFO Vacuumed table table=objects42802026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42812026/11/12 17:02:40 INFO Received push request method=POST path=/42822026/11/12 17:02:40 INFO Received uploads request method=POST path=/42832026/11/12 17:02:40 INFO Received complete push request method=POST path=/42842026/11/12 17:02:40 INFO Received push request method=POST path=/42852026/11/12 17:02:40 INFO Received complete push request method=POST path=/42862026/11/12 17:02:40 INFO Received uploads request method=POST path=/42872026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=042882026/11/12 17:02:40 INFO Completed push id=18 objects=6 duration=2.760626ms42892026/11/12 17:02:40 INFO Completed push id=7 objects=15 duration=2.747325ms42902026/11/12 17:02:40 INFO Received uploads request method=POST path=/42912026/11/12 17:02:40 INFO Received push request method=POST path=/42922026/11/12 17:02:40 INFO Received complete upload request method=POST path=/42932026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042942026/11/12 17:02:40 INFO Received uploads request method=POST path=/42952026/11/12 17:02:40 INFO Vacuumed table table=pending_closures42962026/11/12 17:02:40 INFO Completed upload id=15 objects=3 duration=1.729536ms42972026/11/12 17:02:40 INFO Received push request method=POST path=/42982026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts42992026/11/12 17:02:40 INFO Vacuumed table table=pending_objects43002026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads43012026/11/12 17:02:40 INFO Vacuumed table table=closures43022026/11/12 17:02:40 INFO Vacuumed table table=objects43032026/11/12 17:02:40 INFO Received complete upload request method=POST path=/43042026/11/12 17:02:40 INFO Received complete upload request method=POST path=/43052026/11/12 17:02:40 INFO Completed upload id=15 objects=9 duration=815.488µs43062026/11/12 17:02:40 INFO Completed upload id=33 objects=5 duration=1.944298ms43072026/11/12 17:02:40 INFO Received complete push request method=POST path=/43082026/11/12 17:02:40 INFO Received complete upload request method=POST path=/43092026/11/12 17:02:40 INFO Completed push id=5 objects=28 duration=1.648275ms43102026/11/12 17:02:40 INFO Completed upload id=22 objects=22 duration=1.337132ms43112026/11/12 17:02:40 INFO Received complete upload request method=POST path=/43122026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=043132026/11/12 17:02:40 INFO Completed upload id=23 objects=9 duration=742.487µs43142026/11/12 17:02:40 INFO Received uploads request method=POST path=/43152026/11/12 17:02:40 INFO Received complete upload request method=POST path=/43162026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period43172026/11/12 17:02:40 INFO Completed upload id=20 objects=4 duration=785.128µs43182026/11/12 17:02:40 INFO Received complete push request method=POST path=/43192026/11/12 17:02:40 INFO Received push request method=POST path=/43202026/11/12 17:02:40 INFO Completed push id=20 objects=3 duration=1.361372ms43212026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043222026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=043232026/11/12 17:02:40 INFO Vacuumed table table=pending_closures43242026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts43252026/11/12 17:02:40 INFO Received uploads request method=POST path=/43262026/11/12 17:02:40 INFO Vacuumed table table=pending_objects43272026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period43282026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads43292026/11/12 17:02:40 INFO Vacuumed table table=closures43302026/11/12 17:02:40 INFO Vacuumed table table=objects43312026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043322026/11/12 17:02:40 INFO Vacuumed table table=pending_closures43332026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts43342026/11/12 17:02:40 INFO Received complete push request method=POST path=/43352026/11/12 17:02:40 INFO Vacuumed table table=pending_objects43362026/11/12 17:02:40 INFO Received push request method=POST path=/43372026/11/12 17:02:40 INFO Received push request method=POST path=/43382026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads43392026/11/12 17:02:40 INFO Received push request method=POST path=/43402026/11/12 17:02:40 INFO Completed push id=10 objects=2 duration=1.417413ms43412026/11/12 17:02:40 INFO Vacuumed table table=closures43422026/11/12 17:02:40 INFO Vacuumed table table=objects43432026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=043442026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=043452026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=043462026/11/12 17:02:40 INFO Received complete push request method=POST path=/43472026/11/12 17:02:40 INFO Received complete push request method=POST path=/43482026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043492026/11/12 17:02:40 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."43502026/11/12 17:02:40 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."43512026/11/12 17:02:40 INFO Vacuumed table table=pending_closures43522026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period43532026/11/12 17:02:40 INFO Completed push id=30 objects=4 duration=2.924407ms43542026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts43552026/11/12 17:02:40 INFO Aborted multipart uploads count=0 kept=043562026/11/12 17:02:40 INFO Completed push id=16 objects=4 duration=2.701705ms43572026/11/12 17:02:40 INFO Vacuumed table table=pending_objects43582026/11/12 17:02:40 WARN Force mode enabled - objects will be deleted immediately without grace period43592026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads43602026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043612026/11/12 17:02:40 INFO Received push request method=POST path=/43622026/11/12 17:02:40 INFO Vacuumed table table=closures43632026/11/12 17:02:40 INFO Received complete push request method=POST path=/43642026/11/12 17:02:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043652026/11/12 17:02:40 INFO Vacuumed table table=pending_closures43662026/11/12 17:02:40 INFO Vacuumed table table=objects43672026/11/12 17:02:40 INFO Received push request method=POST path=/43682026/11/12 17:02:40 INFO Received complete push request method=POST path=/43692026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts43702026/11/12 17:02:40 INFO Vacuumed table table=pending_closures43712026/11/12 17:02:40 INFO Received complete push request method=POST path=/43722026/11/12 17:02:40 INFO Vacuumed table table=commit_receipts43732026/11/12 17:02:40 INFO Completed push id=21 objects=17 duration=1.849138ms43742026/11/12 17:02:40 INFO Received push request method=POST path=/43752026/11/12 17:02:40 INFO Completed push id=26 objects=5 duration=1.752077ms43762026/11/12 17:02:40 INFO Vacuumed table table=pending_objects43772026/11/12 17:02:40 INFO Completed push id=16 objects=5 duration=1.10957ms43782026/11/12 17:02:40 INFO Vacuumed table table=pending_objects43792026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads43802026/11/12 17:02:40 INFO Received complete upload request method=POST path=/43812026/11/12 17:02:40 INFO Vacuumed table table=multipart_uploads43822026/11/12 17:02:40 INFO Vacuumed table table=closures43832026/11/12 17:02:40 INFO Vacuumed table table=closures43842026/11/12 17:02:40 INFO Completed upload id=17 objects=5 duration=1.138951ms43852026/11/12 17:02:40 INFO Vacuumed table table=objects43862026/11/12 17:02:40 INFO Vacuumed table table=objects43872026/11/12 17:02:41 INFO Received uploads request method=POST path=/43882026/11/12 17:02:41 INFO Received complete upload request method=POST path=/43892026/11/12 17:02:41 INFO Received uploads request method=POST path=/43902026/11/12 17:02:41 INFO Completed upload id=21 objects=25 duration=1.759937ms43912026/11/12 17:02:41 INFO Received uploads request method=POST path=/43922026/11/12 17:02:41 INFO Received uploads request method=POST path=/43932026/11/12 17:02:41 INFO Received complete push request method=POST path=/43942026/11/12 17:02:41 INFO Received complete upload request method=POST path=/43952026/11/12 17:02:41 INFO Completed push id=19 objects=7 duration=1.372513ms43962026/11/12 17:02:41 INFO Completed upload id=24 objects=14 duration=890.228µs43972026/11/12 17:02:41 INFO Received uploads request method=POST path=/43982026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=043992026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period44002026/11/12 17:02:41 INFO Received complete push request method=POST path=/44012026/11/12 17:02:41 INFO Completed push id=19 objects=3 duration=1.06805ms44022026/11/12 17:02:41 INFO Received complete push request method=POST path=/44032026/11/12 17:02:41 INFO Received uploads request method=POST path=/44042026/11/12 17:02:41 INFO Received uploads request method=POST path=/44052026/11/12 17:02:41 INFO Received complete upload request method=POST path=/44062026/11/12 17:02:41 INFO Received push request method=POST path=/44072026/11/12 17:02:41 INFO Received push request method=POST path=/44082026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=044092026/11/12 17:02:41 INFO Received complete upload request method=POST path=/44102026/11/12 17:02:41 INFO Completed push id=17 objects=22 duration=2.222821ms44112026/11/12 17:02:41 INFO Received push request method=POST path=/44122026/11/12 17:02:41 INFO Received push request method=POST path=/44132026/11/12 17:02:41 INFO Completed upload id=20 objects=5 duration=950.568µs44142026/11/12 17:02:41 INFO Completed upload id=17 objects=25 duration=2.670945ms44152026/11/12 17:02:41 INFO Received complete push request method=POST path=/44162026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=044172026/11/12 17:02:41 INFO Completed push id=19 objects=3 duration=878.848µs44182026/11/12 17:02:41 INFO Vacuumed table table=pending_closures44192026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts44202026/11/12 17:02:41 INFO Vacuumed table table=pending_objects44212026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads44222026/11/12 17:02:41 INFO Vacuumed table table=closures44232026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044242026/11/12 17:02:41 INFO Vacuumed table table=objects44252026/11/12 17:02:41 INFO Vacuumed table table=pending_closures44262026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts44272026/11/12 17:02:41 INFO Vacuumed table table=pending_objects44282026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads44292026/11/12 17:02:41 INFO Received uploads request method=POST path=/44302026/11/12 17:02:41 INFO Vacuumed table table=closures44312026/11/12 17:02:41 INFO Vacuumed table table=objects44322026/11/12 17:02:41 INFO Received uploads request method=POST path=/44332026/11/12 17:02:41 INFO Received push request method=POST path=/44342026/11/12 17:02:41 INFO Received complete push request method=POST path=/44352026/11/12 17:02:41 INFO Received uploads request method=POST path=/44362026/11/12 17:02:41 INFO Received push request method=POST path=/44372026/11/12 17:02:41 INFO Received push request method=POST path=/44382026/11/12 17:02:41 INFO Received push request method=POST path=/44392026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=044402026/11/12 17:02:41 INFO Completed push id=16 objects=16 duration=3.454833ms44412026/11/12 17:02:41 INFO Received push request method=POST path=/44422026/11/12 17:02:41 INFO Received complete push request method=POST path=/44432026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period44442026/11/12 17:02:41 INFO Completed push id=20 objects=4 duration=969.989µs44452026/11/12 17:02:41 INFO Received complete upload request method=POST path=/44462026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044472026/11/12 17:02:41 INFO Vacuumed table table=pending_closures44482026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts44492026/11/12 17:02:41 INFO Completed upload id=21 objects=7 duration=2.211561ms44502026/11/12 17:02:41 INFO Vacuumed table table=pending_objects44512026/11/12 17:02:41 INFO Received push request method=POST path=/44522026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=044532026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads44542026/11/12 17:02:41 INFO Vacuumed table table=closures44552026/11/12 17:02:41 INFO Vacuumed table table=objects44562026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044572026/11/12 17:02:41 INFO Received push request method=POST path=/44582026/11/12 17:02:41 INFO Vacuumed table table=pending_closures44592026/11/12 17:02:41 INFO Received push request method=POST path=/44602026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts44612026/11/12 17:02:41 INFO Received complete push request method=POST path=/44622026/11/12 17:02:41 INFO Vacuumed table table=pending_objects44632026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads44642026/11/12 17:02:41 INFO Completed push id=32 objects=7 duration=1.06581ms44652026/11/12 17:02:41 INFO Vacuumed table table=closures44662026/11/12 17:02:41 INFO Received push request method=POST path=/44672026/11/12 17:02:41 INFO Received uploads request method=POST path=/44682026/11/12 17:02:41 INFO Received complete upload request method=POST path=/44692026/11/12 17:02:41 INFO Received complete upload request method=POST path=/44702026/11/12 17:02:41 INFO Vacuumed table table=objects44712026/11/12 17:02:41 INFO Completed upload id=21 objects=2 duration=884.049µs44722026/11/12 17:02:41 INFO Completed upload id=12 objects=6 duration=1.06279ms44732026/11/12 17:02:41 INFO Received uploads request method=POST path=/44742026/11/12 17:02:41 INFO Received uploads request method=POST path=/44752026/11/12 17:02:41 INFO Received complete upload request method=POST path=/44762026/11/12 17:02:41 INFO Completed upload id=22 objects=8 duration=1.358473ms44772026/11/12 17:02:41 INFO Received uploads request method=POST path=/44782026/11/12 17:02:41 INFO Received push request method=POST path=/44792026/11/12 17:02:41 INFO Received complete push request method=POST path=/44802026/11/12 17:02:41 INFO Received complete push request method=POST path=/44812026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=044822026/11/12 17:02:41 INFO Completed push id=26 objects=4 duration=1.11255ms44832026/11/12 17:02:41 INFO Received complete upload request method=POST path=/44842026/11/12 17:02:41 INFO Completed push id=19 objects=14 duration=1.854738ms44852026/11/12 17:02:41 INFO Completed upload id=22 objects=6 duration=741.687µs44862026/11/12 17:02:41 INFO Received complete push request method=POST path=/44872026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=044882026/11/12 17:02:41 INFO Completed push id=17 objects=3 duration=1.296972ms44892026/11/12 17:02:41 INFO Received complete push request method=POST path=/44902026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period44912026/11/12 17:02:41 INFO Received push request method=POST path=/44922026/11/12 17:02:41 INFO Completed push id=16 objects=26 duration=1.03877ms44932026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044942026/11/12 17:02:41 INFO Vacuumed table table=pending_closures44952026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts44962026/11/12 17:02:41 INFO Vacuumed table table=pending_objects44972026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads44982026/11/12 17:02:41 INFO Vacuumed table table=closures44992026/11/12 17:02:41 INFO Vacuumed table table=objects45002026/11/12 17:02:41 INFO Received complete push request method=POST path=/45012026/11/12 17:02:41 INFO Completed push id=23 objects=5 duration=859.188µs45022026/11/12 17:02:41 INFO Received uploads request method=POST path=/45032026/11/12 17:02:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."45042026/11/12 17:02:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."45052026/11/12 17:02:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."45062026/11/12 17:02:41 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."45072026/11/12 17:02:41 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45082026/11/12 17:02:41 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."45092026/11/12 17:02:41 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."45102026/11/12 17:02:41 INFO Received uploads request method=POST path=/45112026/11/12 17:02:41 INFO Received push request method=POST path=/45122026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45132026/11/12 17:02:41 INFO Completed upload id=28 objects=13 duration=1.03423ms45142026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=045152026/11/12 17:02:41 INFO Received uploads request method=POST path=/45162026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45172026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period45182026/11/12 17:02:41 INFO Received push request method=POST path=/45192026/11/12 17:02:41 INFO Received complete push request method=POST path=/45202026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=045212026/11/12 17:02:41 INFO Completed upload id=12 objects=26 duration=2.366322ms45222026/11/12 17:02:41 INFO Completed push id=23 objects=5 duration=1.249892ms45232026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period45242026/11/12 17:02:41 INFO Received push request method=POST path=/45252026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045262026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45272026/11/12 17:02:41 INFO Vacuumed table table=pending_closures45282026/11/12 17:02:41 INFO Received uploads request method=POST path=/45292026/11/12 17:02:41 INFO Completed upload id=24 objects=8 duration=837.767µs45302026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts45312026/11/12 17:02:41 INFO Vacuumed table table=pending_objects45322026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads45332026/11/12 17:02:41 INFO Received complete push request method=POST path=/45342026/11/12 17:02:41 INFO Vacuumed table table=closures45352026/11/12 17:02:41 INFO Completed push id=29 objects=3 duration=1.07913ms45362026/11/12 17:02:41 INFO Vacuumed table table=objects45372026/11/12 17:02:41 INFO Received uploads request method=POST path=/45382026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45392026/11/12 17:02:41 INFO Completed upload id=25 objects=4 duration=1.00509ms45402026/11/12 17:02:41 INFO Received complete push request method=POST path=/45412026/11/12 17:02:41 INFO Completed push id=19 objects=14 duration=1.408213ms45422026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45432026/11/12 17:02:41 INFO Completed upload id=24 objects=3 duration=833.368µs45442026/11/12 17:02:41 INFO Received push request method=POST path=/45452026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=045462026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45472026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045482026/11/12 17:02:41 INFO Completed upload id=28 objects=5 duration=874.108µs45492026/11/12 17:02:41 INFO Received uploads request method=POST path=/45502026/11/12 17:02:41 INFO Received uploads request method=POST path=/45512026/11/12 17:02:41 INFO Vacuumed table table=pending_closures45522026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts45532026/11/12 17:02:41 INFO Vacuumed table table=pending_objects45542026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads45552026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45562026/11/12 17:02:41 INFO Vacuumed table table=closures45572026/11/12 17:02:41 INFO Received complete push request method=POST path=/45582026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=045592026/11/12 17:02:41 INFO Vacuumed table table=objects45602026/11/12 17:02:41 INFO Completed upload id=31 objects=2 duration=825.768µs45612026/11/12 17:02:41 INFO Received uploads request method=POST path=/45622026/11/12 17:02:41 INFO Vacuumed table table=pending_closures45632026/11/12 17:02:41 INFO Received push request method=POST path=/45642026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts45652026/11/12 17:02:41 INFO Completed push id=27 objects=6 duration=2.810626ms45662026/11/12 17:02:41 INFO Received uploads request method=POST path=/45672026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45682026/11/12 17:02:41 INFO Received complete push request method=POST path=/45692026/11/12 17:02:41 INFO Vacuumed table table=pending_objects45702026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads45712026/11/12 17:02:41 INFO Completed push id=25 objects=4 duration=867.068µs45722026/11/12 17:02:41 INFO Received uploads request method=POST path=/45732026/11/12 17:02:41 INFO Completed upload id=14 objects=16 duration=1.751276ms45742026/11/12 17:02:41 INFO Vacuumed table table=closures45752026/11/12 17:02:41 INFO Vacuumed table table=objects45762026/11/12 17:02:41 INFO Received complete push request method=POST path=/45772026/11/12 17:02:41 INFO Received complete push request method=POST path=/45782026/11/12 17:02:41 INFO Received push request method=POST path=/45792026/11/12 17:02:41 INFO Received push request method=POST path=/45802026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=045812026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45822026/11/12 17:02:41 INFO Received complete push request method=POST path=/45832026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=045842026/11/12 17:02:41 INFO Completed push id=27 objects=25 duration=2.826447ms45852026/11/12 17:02:41 INFO Completed push id=7 objects=24 duration=3.22349ms45862026/11/12 17:02:41 INFO Completed upload id=30 objects=5 duration=831.187µs45872026/11/12 17:02:41 INFO Received complete upload request method=POST path=/45882026/11/12 17:02:41 INFO Completed push id=33 objects=6 duration=952.529µs45892026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period45902026/11/12 17:02:41 INFO Completed upload id=27 objects=4 duration=1.268212ms45912026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045922026/11/12 17:02:41 INFO Vacuumed table table=pending_closures45932026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts45942026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045952026/11/12 17:02:41 INFO Vacuumed table table=pending_objects45962026/11/12 17:02:41 INFO Vacuumed table table=pending_closures45972026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads45982026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts45992026/11/12 17:02:41 INFO Vacuumed table table=closures46002026/11/12 17:02:41 INFO Received push request method=POST path=/46012026/11/12 17:02:41 INFO Vacuumed table table=objects46022026/11/12 17:02:41 INFO Received complete push request method=POST path=/46032026/11/12 17:02:41 INFO Vacuumed table table=pending_objects46042026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads46052026/11/12 17:02:41 INFO Vacuumed table table=closures46062026/11/12 17:02:41 INFO Completed push id=16 objects=8 duration=1.565134ms46072026/11/12 17:02:41 INFO Vacuumed table table=objects46082026/11/12 17:02:41 INFO Received push request method=POST path=/46092026/11/12 17:02:41 INFO Received complete upload request method=POST path=/46102026/11/12 17:02:41 INFO Received uploads request method=POST path=/46112026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=046122026/11/12 17:02:41 INFO Completed upload id=20 objects=6 duration=925.048µs46132026/11/12 17:02:41 INFO Received complete push request method=POST path=/46142026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=046152026/11/12 17:02:41 INFO Received complete upload request method=POST path=/46162026/11/12 17:02:41 INFO Completed push id=13 objects=11 duration=1.423693ms46172026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period46182026/11/12 17:02:41 INFO Completed upload id=15 objects=2 duration=1.05171ms46192026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046202026/11/12 17:02:41 INFO Received complete push request method=POST path=/46212026/11/12 17:02:41 INFO Vacuumed table table=pending_closures46222026/11/12 17:02:41 INFO Completed push id=20 objects=8 duration=1.008569ms46232026/11/12 17:02:41 INFO Received complete push request method=POST path=/46242026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts46252026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046262026/11/12 17:02:41 INFO Completed push id=31 objects=3 duration=815.268µs46272026/11/12 17:02:41 INFO Vacuumed table table=pending_objects46282026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads46292026/11/12 17:02:41 INFO Vacuumed table table=pending_closures46302026/11/12 17:02:41 INFO Received uploads request method=POST path=/46312026/11/12 17:02:41 INFO Vacuumed table table=closures46322026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts46332026/11/12 17:02:41 INFO Received uploads request method=POST path=/46342026/11/12 17:02:41 INFO Vacuumed table table=objects46352026/11/12 17:02:41 INFO Vacuumed table table=pending_objects46362026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads46372026/11/12 17:02:41 INFO Vacuumed table table=closures46382026/11/12 17:02:41 INFO Vacuumed table table=objects46392026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=046402026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=046412026/11/12 17:02:41 INFO Received uploads request method=POST path=/46422026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period46432026/11/12 17:02:41 INFO Received uploads request method=POST path=/46442026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046452026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046462026/11/12 17:02:41 INFO Received uploads request method=POST path=/46472026/11/12 17:02:41 INFO Vacuumed table table=pending_closures46482026/11/12 17:02:41 INFO Vacuumed table table=pending_closures46492026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts46502026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts46512026/11/12 17:02:41 INFO Vacuumed table table=pending_objects46522026/11/12 17:02:41 INFO Vacuumed table table=pending_objects46532026/11/12 17:02:41 INFO Received push request method=POST path=/46542026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads46552026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads46562026/11/12 17:02:41 INFO Vacuumed table table=closures46572026/11/12 17:02:41 INFO Received complete push request method=POST path=/46582026/11/12 17:02:41 INFO Received uploads request method=POST path=/46592026/11/12 17:02:41 INFO Vacuumed table table=closures46602026/11/12 17:02:41 INFO Vacuumed table table=objects46612026/11/12 17:02:41 INFO Received complete push request method=POST path=/46622026/11/12 17:02:41 INFO Vacuumed table table=objects46632026/11/12 17:02:41 INFO Completed push id=17 objects=11 duration=1.627615ms46642026/11/12 17:02:41 INFO Completed push id=26 objects=23 duration=1.08467ms46652026/11/12 17:02:41 INFO Received complete push request method=POST path=/46662026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=046672026/11/12 17:02:41 INFO Completed push id=24 objects=5 duration=2.326081ms46682026/11/12 17:02:41 INFO Received uploads request method=POST path=/46692026/11/12 17:02:41 INFO Received uploads request method=POST path=/46702026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046712026/11/12 17:02:41 INFO Vacuumed table table=pending_closures46722026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts46732026/11/12 17:02:41 INFO Received uploads request method=POST path=/46742026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=046752026/11/12 17:02:41 INFO Vacuumed table table=pending_objects46762026/11/12 17:02:41 INFO Received complete push request method=POST path=/46772026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads46782026/11/12 17:02:41 INFO Received push request method=POST path=/46792026/11/12 17:02:41 INFO Vacuumed table table=closures46802026/11/12 17:02:41 INFO Completed push id=32 objects=6 duration=1.07105ms46812026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period46822026/11/12 17:02:41 INFO Vacuumed table table=objects46832026/11/12 17:02:41 INFO Received push request method=POST path=/46842026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046852026/11/12 17:02:41 INFO Received push request method=POST path=/46862026/11/12 17:02:41 INFO Vacuumed table table=pending_closures46872026/11/12 17:02:41 INFO Received push request method=POST path=/46882026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts46892026/11/12 17:02:41 INFO Vacuumed table table=pending_objects46902026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads46912026/11/12 17:02:41 INFO Vacuumed table table=closures46922026/11/12 17:02:41 INFO Vacuumed table table=objects46932026/11/12 17:02:41 INFO Received uploads request method=POST path=/46942026/11/12 17:02:41 INFO Received complete push request method=POST path=/46952026/11/12 17:02:41 INFO Received uploads request method=POST path=/46962026/11/12 17:02:41 INFO Completed push id=11 objects=6 duration=2.673205ms46972026/11/12 17:02:41 INFO Received push request method=POST path=/46982026/11/12 17:02:41 INFO Received complete upload request method=POST path=/46992026/11/12 17:02:41 INFO Completed upload id=33 objects=7 duration=1.02881ms47002026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=047012026/11/12 17:02:41 INFO Received push request method=POST path=/47022026/11/12 17:02:41 INFO Received complete push request method=POST path=/47032026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period47042026/11/12 17:02:41 INFO Completed push id=27 objects=15 duration=1.172871ms47052026/11/12 17:02:41 INFO Received uploads request method=POST path=/47062026/11/12 17:02:41 INFO Received uploads request method=POST path=/47072026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47082026/11/12 17:02:41 INFO Completed upload id=21 objects=2 duration=782.547µs47092026/11/12 17:02:41 INFO Received push request method=POST path=/47102026/11/12 17:02:41 INFO Received uploads request method=POST path=/47112026/11/12 17:02:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."47122026/11/12 17:02:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."47132026/11/12 17:02:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."47142026/11/12 17:02:41 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."47152026/11/12 17:02:41 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."47162026/11/12 17:02:41 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."47172026/11/12 17:02:41 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."47182026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47192026/11/12 17:02:41 INFO Received push request method=POST path=/47202026/11/12 17:02:41 INFO Completed upload id=34 objects=3 duration=825.027µs47212026/11/12 17:02:41 INFO Received complete push request method=POST path=/47222026/11/12 17:02:41 INFO Received push request method=POST path=/47232026/11/12 17:02:41 INFO Completed push id=18 objects=6 duration=957.689µs47242026/11/12 17:02:41 INFO Received complete push request method=POST path=/47252026/11/12 17:02:41 INFO Completed push id=27 objects=11 duration=958.009µs47262026/11/12 17:02:41 INFO Received uploads request method=POST path=/47272026/11/12 17:02:41 INFO Received uploads request method=POST path=/47282026/11/12 17:02:41 INFO Received uploads request method=POST path=/47292026/11/12 17:02:41 INFO Received complete push request method=POST path=/47302026/11/12 17:02:41 INFO Completed push id=13 objects=2 duration=1.05769ms47312026/11/12 17:02:41 INFO Received complete push request method=POST path=/47322026/11/12 17:02:41 INFO Completed push id=35 objects=11 duration=931.809µs47332026/11/12 17:02:41 INFO Received uploads request method=POST path=/47342026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47352026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=047362026/11/12 17:02:41 INFO Received complete push request method=POST path=/47372026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47382026/11/12 17:02:41 INFO Completed upload id=26 objects=6 duration=1.10919ms47392026/11/12 17:02:41 INFO Completed push id=21 objects=26 duration=1.761477ms47402026/11/12 17:02:41 INFO Completed upload id=29 objects=26 duration=1.09115ms47412026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=047422026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047432026/11/12 17:02:41 INFO Received push request method=POST path=/47442026/11/12 17:02:41 INFO Vacuumed table table=pending_closures47452026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047462026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts47472026/11/12 17:02:41 INFO Received complete push request method=POST path=/47482026/11/12 17:02:41 INFO Vacuumed table table=pending_closures47492026/11/12 17:02:41 INFO Vacuumed table table=pending_objects47502026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47512026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts47522026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads47532026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=047542026/11/12 17:02:41 INFO Vacuumed table table=closures47552026/11/12 17:02:41 INFO Completed upload id=21 objects=3 duration=866.768µs47562026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47572026/11/12 17:02:41 INFO Vacuumed table table=pending_objects47582026/11/12 17:02:41 INFO Vacuumed table table=objects47592026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads47602026/11/12 17:02:41 INFO Completed push id=16 objects=14 duration=2.617325ms47612026/11/12 17:02:41 INFO Vacuumed table table=closures47622026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period47632026/11/12 17:02:41 INFO Completed upload id=27 objects=13 duration=1.247852ms47642026/11/12 17:02:41 INFO Vacuumed table table=objects47652026/11/12 17:02:41 INFO Received complete push request method=POST path=/47662026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47672026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047682026/11/12 17:02:41 INFO Completed push id=28 objects=19 duration=1.10889ms47692026/11/12 17:02:41 INFO Vacuumed table table=pending_closures47702026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=047712026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts47722026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=047732026/11/12 17:02:41 INFO Completed upload id=13 objects=14 duration=2.319402ms47742026/11/12 17:02:41 INFO Received uploads request method=POST path=/47752026/11/12 17:02:41 INFO Vacuumed table table=pending_objects47762026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads47772026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period47782026/11/12 17:02:41 INFO Vacuumed table table=closures47792026/11/12 17:02:41 INFO Received uploads request method=POST path=/47802026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period47812026/11/12 17:02:41 INFO Received complete push request method=POST path=/47822026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047832026/11/12 17:02:41 INFO Vacuumed table table=objects47842026/11/12 17:02:41 INFO Completed push id=30 objects=8 duration=954.749µs47852026/11/12 17:02:41 INFO Vacuumed table table=pending_closures47862026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047872026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts47882026/11/12 17:02:41 INFO Vacuumed table table=pending_objects47892026/11/12 17:02:41 INFO Vacuumed table table=pending_closures47902026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads47912026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=047922026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts47932026/11/12 17:02:41 INFO Vacuumed table table=closures47942026/11/12 17:02:41 INFO Received complete upload request method=POST path=/47952026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period47962026/11/12 17:02:41 INFO Vacuumed table table=objects47972026/11/12 17:02:41 INFO Completed upload id=28 objects=17 duration=1.06729ms47982026/11/12 17:02:41 INFO Vacuumed table table=pending_objects47992026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48002026/11/12 17:02:41 INFO Vacuumed table table=closures48012026/11/12 17:02:41 INFO Received complete upload request method=POST path=/48022026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048032026/11/12 17:02:41 INFO Vacuumed table table=objects48042026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048052026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048062026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period48072026/11/12 17:02:41 INFO Completed upload id=20 objects=3 duration=2.035959ms48082026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period48092026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048102026/11/12 17:02:41 INFO Received complete upload request method=POST path=/48112026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048122026/11/12 17:02:41 INFO Completed upload id=29 objects=5 duration=805.487µs48132026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048142026/11/12 17:02:41 INFO Vacuumed table table=pending_closures48152026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048162026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts48172026/11/12 17:02:41 INFO Vacuumed table table=pending_closures48182026/11/12 17:02:41 INFO Received complete upload request method=POST path=/48192026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048202026/11/12 17:02:41 INFO Vacuumed table table=pending_objects48212026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts48222026/11/12 17:02:41 INFO Vacuumed table table=pending_closures48232026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48242026/11/12 17:02:41 INFO Received complete upload request method=POST path=/48252026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts48262026/11/12 17:02:41 INFO Completed upload id=26 objects=8 duration=1.222931ms48272026/11/12 17:02:41 INFO Vacuumed table table=pending_closures48282026/11/12 17:02:41 INFO Completed upload id=31 objects=5 duration=679.666µs48292026/11/12 17:02:41 INFO Vacuumed table table=pending_objects48302026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=048312026/11/12 17:02:41 INFO Vacuumed table table=closures48322026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48332026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts48342026/11/12 17:02:41 INFO Vacuumed table table=pending_objects48352026/11/12 17:02:41 INFO Vacuumed table table=objects48362026/11/12 17:02:41 INFO Vacuumed table table=closures48372026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48382026/11/12 17:02:41 INFO Vacuumed table table=pending_objects48392026/11/12 17:02:41 INFO Vacuumed table table=pending_closures48402026/11/12 17:02:41 INFO Vacuumed table table=closures48412026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48422026/11/12 17:02:41 INFO Vacuumed table table=objects48432026/11/12 17:02:41 INFO Vacuumed table table=closures48442026/11/12 17:02:41 INFO Vacuumed table table=objects48452026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts48462026/11/12 17:02:41 INFO Received complete push request method=POST path=/48472026/11/12 17:02:41 INFO Received complete push request method=POST path=/48482026/11/12 17:02:41 INFO Vacuumed table table=objects48492026/11/12 17:02:41 INFO Completed push id=29 objects=2 duration=964.429µs48502026/11/12 17:02:41 INFO Completed push id=29 objects=10 duration=1.212891ms48512026/11/12 17:02:41 INFO Vacuumed table table=pending_objects48522026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48532026/11/12 17:02:41 INFO Vacuumed table table=closures48542026/11/12 17:02:41 INFO Vacuumed table table=objects48552026/11/12 17:02:41 INFO Received push request method=POST path=/48562026/11/12 17:02:41 INFO Received uploads request method=POST path=/48572026/11/12 17:02:41 INFO Received complete upload request method=POST path=/48582026/11/12 17:02:41 INFO Received complete push request method=POST path=/48592026/11/12 17:02:41 INFO Received complete push request method=POST path=/48602026/11/12 17:02:41 INFO Completed upload id=19 objects=2 duration=2.17042ms48612026/11/12 17:02:41 INFO Completed push id=28 objects=14 duration=1.406953ms48622026/11/12 17:02:41 INFO Completed push id=32 objects=3 duration=890.449µs48632026/11/12 17:02:41 INFO Received push request method=POST path=/48642026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048652026/11/12 17:02:41 INFO Received uploads request method=POST path=/48662026/11/12 17:02:41 INFO Received complete push request method=POST path=/48672026/11/12 17:02:41 INFO Received complete push request method=POST path=/48682026/11/12 17:02:41 INFO Completed push id=16 objects=7 duration=1.460554ms48692026/11/12 17:02:41 INFO Completed push id=31 objects=6 duration=935.909µs48702026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048712026/11/12 17:02:41 INFO Vacuumed table table=pending_closures48722026/11/12 17:02:41 INFO Received complete push request method=POST path=/48732026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts48742026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048752026/11/12 17:02:41 INFO Vacuumed table table=pending_objects48762026/11/12 17:02:41 INFO Received complete upload request method=POST path=/48772026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48782026/11/12 17:02:41 INFO Vacuumed table table=closures48792026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period48802026/11/12 17:02:41 INFO Completed upload id=30 objects=6 duration=874.188µs48812026/11/12 17:02:41 INFO Completed push id=31 objects=6 duration=2.545844ms48822026/11/12 17:02:41 INFO Vacuumed table table=objects48832026/11/12 17:02:41 INFO Received uploads request method=POST path=/48842026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048852026/11/12 17:02:41 INFO Vacuumed table table=pending_closures48862026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts48872026/11/12 17:02:41 INFO Received push request method=POST path=/48882026/11/12 17:02:41 INFO Vacuumed table table=pending_objects48892026/11/12 17:02:41 INFO Received push request method=POST path=/48902026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048912026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads48922026/11/12 17:02:41 INFO Vacuumed table table=closures48932026/11/12 17:02:41 INFO Received push request method=POST path=/48942026/11/12 17:02:41 INFO Vacuumed table table=objects48952026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period48962026/11/12 17:02:41 INFO Received complete push request method=POST path=/48972026/11/12 17:02:41 INFO Received push request method=POST path=/48982026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=048992026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049002026/11/12 17:02:41 INFO Completed push id=32 objects=9 duration=1.348933ms49012026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period49022026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49032026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49042026/11/12 17:02:41 INFO Received uploads request method=POST path=/49052026/11/12 17:02:41 INFO Received complete upload request method=POST path=/49062026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049072026/11/12 17:02:41 INFO Received push request method=POST path=/49082026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49092026/11/12 17:02:41 INFO Completed upload id=31 objects=5 duration=920.528µs49102026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49112026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49122026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49132026/11/12 17:02:41 INFO Received complete upload request method=POST path=/49142026/11/12 17:02:41 INFO Vacuumed table table=closures49152026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49162026/11/12 17:02:41 INFO Received complete push request method=POST path=/49172026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49182026/11/12 17:02:41 INFO Completed upload id=24 objects=7 duration=886.848µs49192026/11/12 17:02:41 INFO Vacuumed table table=objects49202026/11/12 17:02:41 INFO Vacuumed table table=closures49212026/11/12 17:02:41 INFO Vacuumed table table=objects49222026/11/12 17:02:41 INFO Completed push id=33 objects=7 duration=1.460094ms49232026/11/12 17:02:41 INFO Received complete push request method=POST path=/49242026/11/12 17:02:41 INFO Received uploads request method=POST path=/49252026/11/12 17:02:41 INFO Received complete push request method=POST path=/49262026/11/12 17:02:41 INFO Completed push id=23 objects=6 duration=1.609135ms49272026/11/12 17:02:41 INFO Received push request method=POST path=/49282026/11/12 17:02:41 INFO Received complete push request method=POST path=/49292026/11/12 17:02:41 INFO Completed push id=24 objects=24 duration=2.563644ms49302026/11/12 17:02:41 INFO Completed push id=31 objects=14 duration=2.698125ms49312026/11/12 17:02:41 INFO Received push request method=POST path=/49322026/11/12 17:02:41 INFO Received complete upload request method=POST path=/49332026/11/12 17:02:41 INFO Received complete upload request method=POST path=/49342026/11/12 17:02:41 INFO Completed upload id=33 objects=9 duration=834.008µs49352026/11/12 17:02:41 INFO Completed upload id=34 objects=3 duration=719.446µs49362026/11/12 17:02:41 INFO Received complete upload request method=POST path=/49372026/11/12 17:02:41 INFO Received uploads request method=POST path=/49382026/11/12 17:02:41 INFO Received push request method=POST path=/49392026/11/12 17:02:41 INFO Completed upload id=26 objects=3 duration=822.328µs49402026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=049412026/11/12 17:02:41 INFO Received complete push request method=POST path=/49422026/11/12 17:02:41 INFO Completed push id=24 objects=6 duration=834.048µs49432026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=049442026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049452026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=049462026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period49472026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49482026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49492026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049502026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49512026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period49522026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49532026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49542026/11/12 17:02:41 INFO Vacuumed table table=closures49552026/11/12 17:02:41 INFO Received uploads request method=POST path=/49562026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49572026/11/12 17:02:41 INFO Vacuumed table table=objects49582026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49592026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049602026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49612026/11/12 17:02:41 INFO Vacuumed table table=closures49622026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49632026/11/12 17:02:41 INFO Vacuumed table table=objects49642026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=049652026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49662026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period49672026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=049682026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=049692026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49702026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049712026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49722026/11/12 17:02:41 INFO Received complete upload request method=POST path=/49732026/11/12 17:02:41 INFO Received complete push request method=POST path=/49742026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period49752026/11/12 17:02:41 INFO Vacuumed table table=closures49762026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49772026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period49782026/11/12 17:02:41 INFO Vacuumed table table=objects49792026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49802026/11/12 17:02:41 INFO Completed upload id=25 objects=6 duration=1.533214ms49812026/11/12 17:02:41 INFO Received uploads request method=POST path=/49822026/11/12 17:02:41 INFO Completed push id=22 objects=3 duration=1.848057ms49832026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049842026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49852026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049862026/11/12 17:02:41 INFO Received uploads request method=POST path=/49872026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49882026/11/12 17:02:41 INFO Vacuumed table table=closures49892026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49902026/11/12 17:02:41 INFO Vacuumed table table=pending_closures49912026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49922026/11/12 17:02:41 INFO Vacuumed table table=objects49932026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts49942026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49952026/11/12 17:02:41 INFO Received uploads request method=POST path=/49962026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49972026/11/12 17:02:41 INFO Vacuumed table table=pending_objects49982026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads49992026/11/12 17:02:41 INFO Received uploads request method=POST path=/50002026/11/12 17:02:41 INFO Vacuumed table table=closures50012026/11/12 17:02:41 INFO Vacuumed table table=closures50022026/11/12 17:02:41 INFO Received uploads request method=POST path=/50032026/11/12 17:02:41 INFO Vacuumed table table=objects50042026/11/12 17:02:41 INFO Vacuumed table table=objects50052026/11/12 17:02:41 INFO Received complete push request method=POST path=/50062026/11/12 17:02:41 INFO Completed push id=25 objects=7 duration=1.223791ms50072026/11/12 17:02:41 INFO Received uploads request method=POST path=/50082026/11/12 17:02:41 INFO Received uploads request method=POST path=/50092026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=050102026/11/12 17:02:41 INFO Received complete push request method=POST path=/50112026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50122026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period50132026/11/12 17:02:41 INFO Completed push id=1 objects=9 duration=1.287532ms50142026/11/12 17:02:41 INFO Completed upload id=29 objects=4 duration=1.746856ms50152026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050162026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50172026/11/12 17:02:41 INFO Received push request method=POST path=/50182026/11/12 17:02:41 INFO Vacuumed table table=pending_closures50192026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts50202026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50212026/11/12 17:02:41 INFO Received uploads request method=POST path=/50222026/11/12 17:02:41 INFO Completed upload id=21 objects=9 duration=2.16638ms50232026/11/12 17:02:41 INFO Vacuumed table table=pending_objects50242026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads50252026/11/12 17:02:41 INFO Vacuumed table table=closures50262026/11/12 17:02:41 INFO Completed upload id=31 objects=3 duration=2.09158ms50272026/11/12 17:02:41 INFO Vacuumed table table=objects50282026/11/12 17:02:41 INFO Received push request method=POST path=/50292026/11/12 17:02:41 INFO Received uploads request method=POST path=/50302026/11/12 17:02:41 INFO Received complete push request method=POST path=/50312026/11/12 17:02:41 INFO Received uploads request method=POST path=/50322026/11/12 17:02:41 INFO Received complete push request method=POST path=/50332026/11/12 17:02:41 INFO Completed push id=7 objects=7 duration=1.494634ms50342026/11/12 17:02:41 INFO Completed push id=25 objects=10 duration=2.006059ms50352026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50362026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50372026/11/12 17:02:41 INFO Completed upload id=22 objects=2 duration=930.069µs50382026/11/12 17:02:41 INFO Completed upload id=26 objects=24 duration=1.140231ms50392026/11/12 17:02:41 INFO Received complete push request method=POST path=/50402026/11/12 17:02:41 INFO Received complete push request method=POST path=/50412026/11/12 17:02:41 INFO Received complete push request method=POST path=/50422026/11/12 17:02:41 INFO Completed push id=9 objects=9 duration=2.998688ms50432026/11/12 17:02:41 INFO Completed push id=24 objects=3 duration=1.210812ms50442026/11/12 17:02:41 INFO Received uploads request method=POST path=/50452026/11/12 17:02:41 INFO Completed push id=23 objects=28 duration=3.406651ms50462026/11/12 17:02:41 INFO Received complete push request method=POST path=/50472026/11/12 17:02:41 INFO Received complete push request method=POST path=/50482026/11/12 17:02:41 INFO Completed push id=11 objects=9 duration=1.360992ms50492026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50502026/11/12 17:02:41 INFO Completed push id=12 objects=6 duration=1.137211ms50512026/11/12 17:02:41 INFO Received push request method=POST path=/50522026/11/12 17:02:41 INFO Completed upload id=29 objects=5 duration=938.788µs50532026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=050542026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period50552026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050562026/11/12 17:02:41 INFO Received push request method=POST path=/50572026/11/12 17:02:41 INFO Vacuumed table table=pending_closures50582026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts50592026/11/12 17:02:41 INFO Vacuumed table table=pending_objects50602026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads50612026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50622026/11/12 17:02:41 INFO Vacuumed table table=closures50632026/11/12 17:02:41 INFO Completed upload id=21 objects=8 duration=1.10761ms50642026/11/12 17:02:41 INFO Vacuumed table table=objects50652026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50662026/11/12 17:02:41 INFO Received uploads request method=POST path=/50672026/11/12 17:02:41 INFO Completed upload id=30 objects=2 duration=987.949µs50682026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50692026/11/12 17:02:41 INFO Received complete push request method=POST path=/50702026/11/12 17:02:41 INFO Received complete push request method=POST path=/50712026/11/12 17:02:41 INFO Completed upload id=2 objects=7 duration=773.507µs50722026/11/12 17:02:41 INFO Completed push id=15 objects=24 duration=1.404933ms50732026/11/12 17:02:41 INFO Completed push id=11 objects=6 duration=1.711296ms50742026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50752026/11/12 17:02:41 INFO Completed upload id=27 objects=26 duration=1.00551ms50762026/11/12 17:02:41 INFO Received complete push request method=POST path=/50772026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50782026/11/12 17:02:41 INFO Completed push id=22 objects=6 duration=1.061669ms50792026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50802026/11/12 17:02:41 INFO Completed upload id=27 objects=8 duration=878.688µs50812026/11/12 17:02:41 INFO Completed upload id=18 objects=2 duration=2.15206ms50822026/11/12 17:02:41 INFO Received complete push request method=POST path=/50832026/11/12 17:02:41 INFO Completed push id=31 objects=20 duration=974.949µs50842026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=050852026/11/12 17:02:41 INFO Received complete upload request method=POST path=/50862026/11/12 17:02:41 INFO Completed upload id=19 objects=2 duration=798.407µs50872026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period50882026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=050892026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050902026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period50912026/11/12 17:02:41 INFO Vacuumed table table=pending_closures50922026/11/12 17:02:41 INFO Received complete push request method=POST path=/50932026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts50942026/11/12 17:02:41 INFO Received complete push request method=POST path=/50952026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050962026/11/12 17:02:41 INFO Received complete push request method=POST path=/50972026/11/12 17:02:41 INFO Received complete push request method=POST path=/50982026/11/12 17:02:41 INFO Vacuumed table table=pending_objects50992026/11/12 17:02:41 INFO Completed push id=28 objects=6 duration=1.192531ms51002026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads51012026/11/12 17:02:41 INFO Vacuumed table table=pending_closures51022026/11/12 17:02:41 INFO Completed push id=24 objects=8 duration=1.492494ms51032026/11/12 17:02:41 INFO Vacuumed table table=closures51042026/11/12 17:02:41 INFO Completed push id=20 objects=13 duration=1.452853ms51052026/11/12 17:02:41 INFO Completed push id=13 objects=22 duration=1.638495ms51062026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts51072026/11/12 17:02:41 INFO Vacuumed table table=objects51082026/11/12 17:02:41 INFO Received push request method=POST path=/51092026/11/12 17:02:41 INFO Received push request method=POST path=/51102026/11/12 17:02:41 INFO Vacuumed table table=pending_objects51112026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads51122026/11/12 17:02:41 INFO Vacuumed table table=closures51132026/11/12 17:02:41 INFO Vacuumed table table=objects51142026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51152026/11/12 17:02:41 INFO Completed upload id=32 objects=5 duration=924.169µs51162026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51172026/11/12 17:02:41 INFO Completed upload id=24 objects=7 duration=781.447µs51182026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51192026/11/12 17:02:41 INFO Completed upload id=8 objects=6 duration=1.012329ms51202026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51212026/11/12 17:02:41 INFO Completed upload id=25 objects=9 duration=938.029µs51222026/11/12 17:02:41 INFO Received complete push request method=POST path=/51232026/11/12 17:02:41 INFO Completed push id=29 objects=6 duration=902.129µs51242026/11/12 17:02:41 INFO Received complete push request method=POST path=/51252026/11/12 17:02:41 INFO Received complete push request method=POST path=/51262026/11/12 17:02:41 INFO Completed push id=22 objects=4 duration=985.669µs51272026/11/12 17:02:41 INFO Completed push id=21 objects=6 duration=939.648µs51282026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=051292026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51302026/11/12 17:02:41 INFO Received complete push request method=POST path=/51312026/11/12 17:02:41 INFO Completed upload id=33 objects=3 duration=917.689µs51322026/11/12 17:02:41 INFO Completed push id=26 objects=3 duration=1.06393ms51332026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051342026/11/12 17:02:41 INFO Vacuumed table table=pending_closures51352026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts51362026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51372026/11/12 17:02:41 INFO Vacuumed table table=pending_objects51382026/11/12 17:02:41 INFO Completed upload id=27 objects=3 duration=902.529µs51392026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads51402026/11/12 17:02:41 INFO Vacuumed table table=closures51412026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51422026/11/12 17:02:41 INFO Vacuumed table table=objects51432026/11/12 17:02:41 INFO Completed upload id=13 objects=7 duration=730.607µs51442026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51452026/11/12 17:02:41 INFO Received push request method=POST path=/51462026/11/12 17:02:41 INFO Completed upload id=22 objects=7 duration=1.04885ms51472026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51482026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51492026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=051502026/11/12 17:02:41 INFO Completed upload id=28 objects=8 duration=843.307µs51512026/11/12 17:02:41 INFO Completed upload id=23 objects=7 duration=896.948µs51522026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period51532026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51542026/11/12 17:02:41 INFO Completed upload id=28 objects=6 duration=794.427µs51552026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051562026/11/12 17:02:41 INFO Vacuumed table table=pending_closures51572026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts51582026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51592026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51602026/11/12 17:02:41 INFO Completed upload id=20 objects=4 duration=825.708µs51612026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=051622026/11/12 17:02:41 INFO Vacuumed table table=pending_objects51632026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads51642026/11/12 17:02:41 INFO Completed upload id=30 objects=8 duration=900.688µs51652026/11/12 17:02:41 INFO Vacuumed table table=closures51662026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=051672026/11/12 17:02:41 INFO Received complete push request method=POST path=/51682026/11/12 17:02:41 INFO Vacuumed table table=objects51692026/11/12 17:02:41 INFO Completed push id=24 objects=4 duration=865.948µs51702026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=051712026/11/12 17:02:41 INFO Vacuumed table table=pending_closures51722026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51732026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts51742026/11/12 17:02:41 INFO Vacuumed table table=pending_objects51752026/11/12 17:02:41 INFO Completed upload id=29 objects=10 duration=764.007µs51762026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads51772026/11/12 17:02:41 INFO Received complete push request method=POST path=/51782026/11/12 17:02:41 INFO Vacuumed table table=closures51792026/11/12 17:02:41 INFO Received complete push request method=POST path=/51802026/11/12 17:02:41 INFO Completed push id=24 objects=5 duration=992.65µs51812026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51822026/11/12 17:02:41 INFO Vacuumed table table=objects51832026/11/12 17:02:41 INFO Completed push id=30 objects=4 duration=926.009µs51842026/11/12 17:02:41 INFO Completed upload id=23 objects=7 duration=2.1414ms51852026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51862026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=051872026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51882026/11/12 17:02:41 INFO Completed upload id=12 objects=9 duration=922.908µs51892026/11/12 17:02:41 INFO Vacuumed table table=pending_closures51902026/11/12 17:02:41 INFO Completed upload id=23 objects=7 duration=1.354413ms51912026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts51922026/11/12 17:02:41 INFO Vacuumed table table=pending_objects51932026/11/12 17:02:41 INFO Received complete push request method=POST path=/51942026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads51952026/11/12 17:02:41 INFO Vacuumed table table=closures51962026/11/12 17:02:41 INFO Completed push id=25 objects=9 duration=973.689µs51972026/11/12 17:02:41 INFO Vacuumed table table=objects51982026/11/12 17:02:41 INFO Received complete upload request method=POST path=/51992026/11/12 17:02:41 INFO Completed upload id=31 objects=6 duration=873.668µs52002026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52012026/11/12 17:02:41 INFO Received complete push request method=POST path=/52022026/11/12 17:02:41 INFO Completed upload id=30 objects=20 duration=799.347µs52032026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52042026/11/12 17:02:41 INFO Completed push id=31 objects=7 duration=862.848µs52052026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52062026/11/12 17:02:41 INFO Completed upload id=25 objects=2 duration=706.686µs52072026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52082026/11/12 17:02:41 INFO Completed upload id=26 objects=5 duration=812.767µs52092026/11/12 17:02:41 INFO Completed upload id=25 objects=16 duration=2.0768ms52102026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52112026/11/12 17:02:41 INFO Completed upload id=26 objects=3 duration=588.426µs52122026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52132026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52142026/11/12 17:02:41 INFO Received complete push request method=POST path=/52152026/11/12 17:02:41 INFO Completed upload id=32 objects=4 duration=780.847µs52162026/11/12 17:02:41 INFO Completed upload id=32 objects=5 duration=658.866µs52172026/11/12 17:02:41 INFO Completed push id=31 objects=11 duration=932.468µs52182026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52192026/11/12 17:02:41 INFO Completed upload id=26 objects=20 duration=721.546µs52202026/11/12 17:02:41 INFO Received complete push request method=POST path=/52212026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=052222026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period52232026/11/12 17:02:41 INFO Completed push id=27 objects=3 duration=1.451234ms52242026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052252026/11/12 17:02:41 INFO Received uploads request method=POST path=/52262026/11/12 17:02:41 INFO Vacuumed table table=pending_closures52272026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts52282026/11/12 17:02:41 INFO Vacuumed table table=pending_objects52292026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads52302026/11/12 17:02:41 INFO Vacuumed table table=closures52312026/11/12 17:02:41 INFO Vacuumed table table=objects52322026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52332026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52342026/11/12 17:02:41 INFO Completed upload id=32 objects=2 duration=646.786µs52352026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52362026/11/12 17:02:41 INFO Completed upload id=24 objects=18 duration=965.929µs52372026/11/12 17:02:41 INFO Completed upload id=27 objects=7 duration=712.786µs52382026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52392026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52402026/11/12 17:02:41 INFO Completed upload id=33 objects=10 duration=867.348µs52412026/11/12 17:02:41 INFO Completed upload id=28 objects=5 duration=695.086µs52422026/11/12 17:02:41 INFO Received complete push request method=POST path=/52432026/11/12 17:02:41 INFO Completed push id=28 objects=7 duration=873.148µs52442026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52452026/11/12 17:02:41 INFO Completed upload id=26 objects=2 duration=1.734916ms52462026/11/12 17:02:41 INFO Received push request method=POST path=/52472026/11/12 17:02:41 INFO Received complete push request method=POST path=/52482026/11/12 17:02:41 INFO Received complete push request method=POST path=/52492026/11/12 17:02:41 INFO Completed push id=33 objects=3 duration=860.368µs52502026/11/12 17:02:41 INFO Completed push id=25 objects=2 duration=1.584435ms52512026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52522026/11/12 17:02:41 INFO Completed upload id=28 objects=3 duration=758.707µs52532026/11/12 17:02:41 INFO Received complete push request method=POST path=/52542026/11/12 17:02:41 INFO Completed push id=29 objects=17 duration=993.909µs52552026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=052562026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52572026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052582026/11/12 17:02:41 INFO Completed upload id=30 objects=6 duration=891.309µs52592026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52602026/11/12 17:02:41 INFO Vacuumed table table=pending_closures52612026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52622026/11/12 17:02:41 INFO Completed upload id=34 objects=24 duration=958.389µs52632026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52642026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts52652026/11/12 17:02:41 INFO Completed upload id=34 objects=3 duration=630.226µs52662026/11/12 17:02:41 INFO Received complete push request method=POST path=/52672026/11/12 17:02:41 INFO Completed upload id=34 objects=23 duration=1.039729ms52682026/11/12 17:02:41 INFO Received push request method=POST path=/52692026/11/12 17:02:41 INFO Completed push id=27 objects=4 duration=795.587µs52702026/11/12 17:02:41 INFO Vacuumed table table=pending_objects52712026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads52722026/11/12 17:02:41 INFO Vacuumed table table=closures52732026/11/12 17:02:41 INFO Vacuumed table table=objects52742026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=052752026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period52762026/11/12 17:02:41 INFO Received complete push request method=POST path=/52772026/11/12 17:02:41 INFO Received complete upload request method=POST path=/52782026/11/12 17:02:41 INFO Completed push id=29 objects=23 duration=1.156811ms52792026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052802026/11/12 17:02:41 INFO Completed upload id=18 objects=22 duration=1.07541ms52812026/11/12 17:02:41 INFO Received complete push request method=POST path=/52822026/11/12 17:02:41 INFO Vacuumed table table=pending_closures52832026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts52842026/11/12 17:02:41 INFO Completed push id=31 objects=5 duration=861.648µs52852026/11/12 17:02:41 INFO Vacuumed table table=pending_objects52862026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads52872026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=052882026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=052892026/11/12 17:02:41 INFO Vacuumed table table=closures52902026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period52912026/11/12 17:02:41 INFO Vacuumed table table=objects52922026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period52932026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052942026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=052952026/11/12 17:02:41 INFO Vacuumed table table=pending_closures52962026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts52972026/11/12 17:02:41 INFO Vacuumed table table=pending_closures52982026/11/12 17:02:41 INFO Vacuumed table table=pending_objects52992026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads53002026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts53012026/11/12 17:02:41 INFO Received complete push request method=POST path=/53022026/11/12 17:02:41 INFO Vacuumed table table=closures53032026/11/12 17:02:41 INFO Completed push id=35 objects=6 duration=1.03499ms53042026/11/12 17:02:41 INFO Vacuumed table table=pending_objects53052026/11/12 17:02:41 INFO Vacuumed table table=objects53062026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads53072026/11/12 17:02:41 INFO Vacuumed table table=closures53082026/11/12 17:02:41 INFO Vacuumed table table=objects53092026/11/12 17:02:41 INFO Received complete push request method=POST path=/53102026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53112026/11/12 17:02:41 INFO Completed upload id=21 objects=8 duration=818.787µs53122026/11/12 17:02:41 INFO Completed push id=30 objects=6 duration=1.09681ms53132026/11/12 17:02:41 INFO Received push request method=POST path=/53142026/11/12 17:02:41 INFO Received complete push request method=POST path=/53152026/11/12 17:02:41 INFO Completed push id=28 objects=26 duration=1.412613ms53162026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=053172026/11/12 17:02:41 INFO Received complete push request method=POST path=/53182026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period53192026/11/12 17:02:41 INFO Completed push id=32 objects=26 duration=1.292972ms53202026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53212026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=053222026/11/12 17:02:41 INFO Completed upload id=22 objects=7 duration=900.768µs53232026/11/12 17:02:41 INFO Received complete push request method=POST path=/53242026/11/12 17:02:41 INFO Received complete push request method=POST path=/53252026/11/12 17:02:41 INFO Vacuumed table table=pending_closures53262026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts53272026/11/12 17:02:41 INFO Completed push id=33 objects=3 duration=1.246411ms53282026/11/12 17:02:41 INFO Received uploads request method=POST path=/53292026/11/12 17:02:41 INFO Completed push id=30 objects=3 duration=1.141671ms53302026/11/12 17:02:41 INFO Vacuumed table table=pending_objects53312026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads53322026/11/12 17:02:41 INFO Vacuumed table table=closures53332026/11/12 17:02:41 INFO Vacuumed table table=objects53342026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53352026/11/12 17:02:41 INFO Completed upload id=33 objects=2 duration=763.907µs53362026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=053372026/11/12 17:02:41 INFO Received complete push request method=POST path=/53382026/11/12 17:02:41 INFO Completed push id=24 objects=5 duration=908.388µs53392026/11/12 17:02:41 INFO Received complete push request method=POST path=/53402026/11/12 17:02:41 INFO Received complete push request method=POST path=/53412026/11/12 17:02:41 INFO Completed push id=34 objects=6 duration=923.389µs53422026/11/12 17:02:41 INFO Completed push id=32 objects=19 duration=1.228432ms53432026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53442026/11/12 17:02:41 INFO Completed upload id=22 objects=5 duration=1.745716ms53452026/11/12 17:02:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."53462026/11/12 17:02:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."53472026/11/12 17:02:41 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."53482026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53492026/11/12 17:02:41 INFO Completed upload id=34 objects=5 duration=674.366µs53502026/11/12 17:02:41 INFO Received complete push request method=POST path=/53512026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=053522026/11/12 17:02:41 INFO Completed push id=35 objects=8 duration=966.049µs53532026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53542026/11/12 17:02:41 INFO Completed upload id=25 objects=8 duration=987.469µs53552026/11/12 17:02:41 INFO Received complete push request method=POST path=/53562026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period53572026/11/12 17:02:41 INFO Received push request method=POST path=/53582026/11/12 17:02:41 INFO Completed push id=28 objects=3 duration=1.599595ms53592026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=053602026/11/12 17:02:41 INFO Vacuumed table table=pending_closures53612026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53622026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts53632026/11/12 17:02:41 INFO Completed upload id=35 objects=11 duration=845.348µs53642026/11/12 17:02:41 INFO Vacuumed table table=pending_objects53652026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads53662026/11/12 17:02:41 INFO Vacuumed table table=closures53672026/11/12 17:02:41 INFO Vacuumed table table=objects53682026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53692026/11/12 17:02:41 INFO Completed upload id=36 objects=9 duration=706.846µs53702026/11/12 17:02:41 INFO Received complete push request method=POST path=/53712026/11/12 17:02:41 INFO Completed push id=26 objects=4 duration=684.606µs53722026/11/12 17:02:41 INFO Received complete push request method=POST path=/53732026/11/12 17:02:41 INFO Completed push id=29 objects=15 duration=974.249µs53742026/11/12 17:02:41 INFO Received complete upload request method=POST path=/53752026/11/12 17:02:41 INFO Completed upload id=36 objects=3 duration=565.385µs53762026/11/12 17:02:41 INFO Received push request method=POST path=/53772026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=053782026/11/12 17:02:41 INFO Received complete push request method=POST path=/53792026/11/12 17:02:41 INFO Completed push id=30 objects=2 duration=933.049µs53802026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period53812026/11/12 17:02:41 INFO Received complete push request method=POST path=/53822026/11/12 17:02:41 INFO Completed push id=30 objects=6 duration=983.449µs53832026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=053842026/11/12 17:02:41 INFO Vacuumed table table=pending_closures53852026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts53862026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=053872026/11/12 17:02:41 INFO Vacuumed table table=pending_objects53882026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period53892026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads53902026/11/12 17:02:41 INFO Vacuumed table table=closures53912026/11/12 17:02:41 INFO Vacuumed table table=objects53922026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=053932026/11/12 17:02:41 INFO Received complete push request method=POST path=/53942026/11/12 17:02:41 INFO Vacuumed table table=pending_closures53952026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts53962026/11/12 17:02:41 INFO Completed push id=28 objects=11 duration=2.10494ms53972026/11/12 17:02:41 INFO Vacuumed table table=pending_objects53982026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads53992026/11/12 17:02:41 INFO Vacuumed table table=closures54002026/11/12 17:02:41 INFO Received complete upload request method=POST path=/54012026/11/12 17:02:41 INFO Completed upload id=31 objects=3 duration=805.127µs54022026/11/12 17:02:41 INFO Vacuumed table table=objects54032026/11/12 17:02:41 INFO Received complete push request method=POST path=/54042026/11/12 17:02:41 INFO Completed push id=32 objects=7 duration=1.05569ms54052026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=054062026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period54072026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=054082026/11/12 17:02:41 INFO Vacuumed table table=pending_closures54092026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts54102026/11/12 17:02:41 INFO Vacuumed table table=pending_objects54112026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads54122026/11/12 17:02:41 INFO Received complete upload request method=POST path=/54132026/11/12 17:02:41 INFO Vacuumed table table=closures54142026/11/12 17:02:41 INFO Completed upload id=33 objects=6 duration=736.667µs54152026/11/12 17:02:41 INFO Vacuumed table table=objects54162026/11/12 17:02:41 INFO Received complete push request method=POST path=/54172026/11/12 17:02:41 INFO Completed push id=29 objects=24 duration=1.284592ms54182026/11/12 17:02:41 INFO Received complete upload request method=POST path=/54192026/11/12 17:02:41 INFO Completed upload id=34 objects=3 duration=533.245µs54202026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=054212026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period54222026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=054232026/11/12 17:02:41 INFO Aborted multipart uploads count=0 kept=054242026/11/12 17:02:41 INFO Vacuumed table table=pending_closures54252026/11/12 17:02:41 WARN Force mode enabled - objects will be deleted immediately without grace period54262026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts54272026/11/12 17:02:41 INFO Vacuumed table table=pending_objects54282026/11/12 17:02:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=054292026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads54302026/11/12 17:02:41 INFO Vacuumed table table=closures54312026/11/12 17:02:41 INFO Vacuumed table table=pending_closures54322026/11/12 17:02:41 INFO Vacuumed table table=objects54332026/11/12 17:02:41 INFO Vacuumed table table=commit_receipts54342026/11/12 17:02:41 INFO Vacuumed table table=pending_objects54352026/11/12 17:02:41 INFO Vacuumed table table=multipart_uploads54362026/11/12 17:02:41 INFO Vacuumed table table=closures54372026/11/12 17:02:41 INFO Vacuumed table table=objects5438--- PASS: FuzzPushGC (0.00s)5439 --- PASS: FuzzPushGC/seed#14 (1.66s)5440 --- PASS: FuzzPushGC/seed#8 (1.74s)5441 --- PASS: FuzzPushGC/seed#3 (1.78s)5442 --- PASS: FuzzPushGC/seed#0 (1.82s)5443 --- PASS: FuzzPushGC/seed#13 (1.82s)5444 --- PASS: FuzzPushGC/seed#6 (1.94s)5445 --- PASS: FuzzPushGC/seed#1 (2.02s)5446 --- PASS: FuzzPushGC/seed#5 (2.02s)5447 --- PASS: FuzzPushGC/seed#9 (2.03s)5448 --- PASS: FuzzPushGC/seed#12 (2.05s)5449 --- PASS: FuzzPushGC/seed#7 (2.08s)5450 --- PASS: FuzzPushGC/seed#4 (2.10s)5451 --- PASS: FuzzPushGC/seed#11 (2.11s)5452 --- PASS: FuzzPushGC/seed#15 (2.13s)5453 --- PASS: FuzzPushGC/seed#2 (2.16s)5454 --- PASS: FuzzPushGC/seed#10 (2.16s)5455PASS54562026-11-12 17:02:42.002 UTC [129] LOG: received smart shutdown request54572026-11-12 17:02:42.007 UTC [129] LOG: background worker "logical replication launcher" (PID 139) exited with exit code 154582026-11-12 17:02:43.127 UTC [134] LOG: checkpoint complete: wrote 1768 buffers (10.8%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=15.800 s, sync=1.070 s, total=16.914 s; sync files=48810, longest=0.026 s, average=0.001 s; distance=534185 kB, estimate=534185 kB; lsn=0/2BCACE88, redo lsn=0/2211028854592026-11-12 17:02:43.131 UTC [134] LOG: shutting down54602026-11-12 17:02:43.131 UTC [134] LOG: checkpoint starting: shutdown immediate54612026-11-12 17:02:43.886 UTC [134] LOG: checkpoint complete: wrote 12950 buffers (79.0%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 9 recycled; write=0.196 s, sync=0.544 s, total=0.756 s; sync files=4268, longest=0.002 s, average=0.001 s; distance=159347 kB, estimate=496701 kB; lsn=0/2BCACF00, redo lsn=0/2BCACF0054622026-11-12 17:02:43.972 UTC [129] LOG: database system is shut down5463Running OIDC tests...5464=== RUN TestAudienceForIssuer5465=== PAUSE TestAudienceForIssuer5466=== RUN TestProviderClientGivesUpOnSilentIssuer5467=== PAUSE TestProviderClientGivesUpOnSilentIssuer5468=== RUN TestGlobMatch5469=== PAUSE TestGlobMatch5470=== RUN TestValidateToken_ValidToken5471=== PAUSE TestValidateToken_ValidToken5472=== RUN TestValidateToken_WrongAudience5473=== PAUSE TestValidateToken_WrongAudience5474=== RUN TestValidateToken_Expired5475=== PAUSE TestValidateToken_Expired5476=== RUN TestValidateToken_BoundClaimsMismatch5477=== PAUSE TestValidateToken_BoundClaimsMismatch5478=== RUN TestValidateToken_BoundSubjectMismatch5479=== PAUSE TestValidateToken_BoundSubjectMismatch5480=== RUN TestValidateToken_MultipleProviders5481=== PAUSE TestValidateToken_MultipleProviders5482=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5483=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5484=== RUN TestValidateToken_GoogleSchemelessIssuer5485=== PAUSE TestValidateToken_GoogleSchemelessIssuer5486=== RUN TestValidateToken_KubernetesServiceAccount5487=== PAUSE TestValidateToken_KubernetesServiceAccount5488=== RUN TestNewValidator_KubernetesRequiresCA5489=== PAUSE TestNewValidator_KubernetesRequiresCA5490=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5491=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5492=== RUN TestPins_ReservedForMatchingRule5493=== PAUSE TestPins_ReservedForMatchingRule5494=== RUN TestPins_TopLevelShorthand5495=== PAUSE TestPins_TopLevelShorthand5496=== RUN TestPins_ConfigValidation5497=== PAUSE TestPins_ConfigValidation5498=== RUN TestScopes_LegacyProviderDefaultsToWrite5499=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5500=== RUN TestScopes_Rules5501=== PAUSE TestScopes_Rules5502=== RUN TestScopes_ConfigValidation5503=== PAUSE TestScopes_ConfigValidation5504=== CONT TestAudienceForIssuer5505=== CONT TestValidateToken_GoogleSchemelessIssuer5506=== CONT TestScopes_ConfigValidation5507=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5508=== CONT TestScopes_Rules5509=== CONT TestNewValidator_KubernetesRequiresCA5510=== CONT TestPins_ReservedForMatchingRule5511=== CONT TestValidateToken_KubernetesServiceAccount5512--- PASS: TestAudienceForIssuer (0.00s)5513--- PASS: TestScopes_ConfigValidation (0.00s)5514=== CONT TestPins_TopLevelShorthand5515=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5516=== CONT TestValidateToken_MultipleProviders5517=== CONT TestValidateToken_BoundSubjectMismatch5518=== CONT TestValidateToken_BoundClaimsMismatch5519=== CONT TestValidateToken_Expired5520=== CONT TestValidateToken_WrongAudience5521=== CONT TestValidateToken_ValidToken5522=== CONT TestGlobMatch5523=== CONT TestProviderClientGivesUpOnSilentIssuer5524=== CONT TestScopes_LegacyProviderDefaultsToWrite5525=== CONT TestPins_ConfigValidation5526=== RUN TestGlobMatch/foo_foo5527=== PAUSE TestGlobMatch/foo_foo5528=== RUN TestGlobMatch/foo_bar5529=== PAUSE TestGlobMatch/foo_bar5530=== RUN TestGlobMatch/*_5531=== PAUSE TestGlobMatch/*_5532=== RUN TestGlobMatch/*_anything5533=== PAUSE TestGlobMatch/*_anything5534=== RUN TestGlobMatch/foo*_foo5535=== PAUSE TestGlobMatch/foo*_foo5536=== RUN TestGlobMatch/foo*_foobar5537=== PAUSE TestGlobMatch/foo*_foobar5538=== RUN TestGlobMatch/foo*_bar5539=== PAUSE TestGlobMatch/foo*_bar5540=== RUN TestGlobMatch/*bar_bar5541=== PAUSE TestGlobMatch/*bar_bar5542=== RUN TestGlobMatch/*bar_foobar5543=== PAUSE TestGlobMatch/*bar_foobar5544=== RUN TestGlobMatch/*bar_foo5545=== PAUSE TestGlobMatch/*bar_foo5546=== RUN TestGlobMatch/foo*bar_foobar5547=== PAUSE TestGlobMatch/foo*bar_foobar5548=== RUN TestGlobMatch/foo*bar_foo123bar5549=== PAUSE TestGlobMatch/foo*bar_foo123bar5550=== RUN TestGlobMatch/foo*bar_foobarbaz5551=== PAUSE TestGlobMatch/foo*bar_foobarbaz5552=== RUN TestGlobMatch/*/*_foo/bar5553=== PAUSE TestGlobMatch/*/*_foo/bar5554=== RUN TestGlobMatch/*/*_foo5555=== PAUSE TestGlobMatch/*/*_foo5556=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5557--- PASS: TestPins_ConfigValidation (0.00s)5558--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)5559=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5560=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05561=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05562=== RUN TestGlobMatch/refs/*/main_refs/heads/main5563=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5564=== RUN TestGlobMatch/fo?_foo5565=== PAUSE TestGlobMatch/fo?_foo5566=== RUN TestGlobMatch/fo?_fo5567=== PAUSE TestGlobMatch/fo?_fo5568=== RUN TestGlobMatch/fo?_fooo5569=== PAUSE TestGlobMatch/fo?_fooo5570=== RUN TestGlobMatch/?oo_foo5571=== PAUSE TestGlobMatch/?oo_foo5572=== RUN TestGlobMatch/?oo_boo5573=== PAUSE TestGlobMatch/?oo_boo5574=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5575=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5576=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5577=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5578=== CONT TestGlobMatch/foo_foo5579=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5580=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5581=== CONT TestGlobMatch/?oo_boo5582=== CONT TestGlobMatch/foo*bar_foobar5583=== CONT TestGlobMatch/*bar_foo5584=== CONT TestGlobMatch/*bar_foobar5585=== CONT TestGlobMatch/*bar_bar5586=== CONT TestGlobMatch/foo*_bar5587=== CONT TestGlobMatch/foo*_foobar5588=== CONT TestGlobMatch/foo*_foo5589=== CONT TestGlobMatch/*_anything5590=== CONT TestGlobMatch/*_5591=== CONT TestGlobMatch/foo_bar55922026/11/12 17:02:46 INFO OIDC provider initialized name=known issuer=https://known.example5593=== CONT TestGlobMatch/refs/*/main_refs/heads/main5594=== CONT TestGlobMatch/foo*bar_foo123bar5595=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05596=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5597=== CONT TestGlobMatch/*/*_foo5598=== CONT TestGlobMatch/*/*_foo/bar5599=== CONT TestGlobMatch/foo*bar_foobarbaz5600=== CONT TestGlobMatch/?oo_foo5601=== CONT TestGlobMatch/fo?_fooo5602=== CONT TestGlobMatch/fo?_fo5603=== CONT TestGlobMatch/fo?_foo5604--- PASS: TestGlobMatch (0.00s)5605 --- PASS: TestGlobMatch/foo_foo (0.00s)5606 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5607 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5608 --- PASS: TestGlobMatch/?oo_boo (0.00s)5609 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5610 --- PASS: TestGlobMatch/*bar_foo (0.00s)5611 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5612 --- PASS: TestGlobMatch/*bar_bar (0.00s)5613 --- PASS: TestGlobMatch/foo*_bar (0.00s)5614 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5615 --- PASS: TestGlobMatch/foo*_foo (0.00s)5616 --- PASS: TestGlobMatch/*_anything (0.00s)5617 --- PASS: TestGlobMatch/*_ (0.00s)5618 --- PASS: TestGlobMatch/foo_bar (0.00s)5619 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5620 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5621 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5622 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5623 --- PASS: TestGlobMatch/*/*_foo (0.00s)5624 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5625 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5626 --- PASS: TestGlobMatch/?oo_foo (0.00s)5627 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5628 --- PASS: TestGlobMatch/fo?_fo (0.00s)5629 --- PASS: TestGlobMatch/fo?_foo (0.00s)56302026/11/12 17:02:46 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5631--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.03s)5632--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.07s)56332026/11/12 17:02:46 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:447715634--- PASS: TestValidateToken_KubernetesServiceAccount (0.08s)56352026/11/12 17:02:46 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235636--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.10s)56372026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42479/oidc56382026/11/12 17:02:46 http: TLS handshake error from 127.0.0.1:44934: remote error: tls: bad certificate5639--- PASS: TestNewValidator_KubernetesRequiresCA (0.10s)5640--- PASS: TestValidateToken_WrongAudience (0.10s)56412026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35039/oidc5642--- PASS: TestValidateToken_BoundSubjectMismatch (0.10s)56432026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46391/oidc5644--- PASS: TestValidateToken_ValidToken (0.11s)56452026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34715/oidc5646--- PASS: TestPins_TopLevelShorthand (0.13s)56472026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38461/oidc56482026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42645/oidc5649--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.19s)5650--- PASS: TestPins_ReservedForMatchingRule (0.20s)56512026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38975/oidc5652--- PASS: TestScopes_Rules (0.23s)56532026/11/12 17:02:46 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:45183/oidc56542026/11/12 17:02:46 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:38399/oidc5655--- PASS: TestValidateToken_MultipleProviders (0.24s)56562026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35099/oidc5657--- PASS: TestValidateToken_Expired (0.28s)56582026/11/12 17:02:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35739/oidc5659--- PASS: TestValidateToken_BoundClaimsMismatch (0.30s)5660PASS5661Running hook tests...5662=== RUN TestSendPathsEmpty5663=== PAUSE TestSendPathsEmpty5664=== RUN TestQueueEnqueueAndFetch5665=== PAUSE TestQueueEnqueueAndFetch5666=== RUN TestQueueDeduplication5667=== PAUSE TestQueueDeduplication5668=== RUN TestQueueRemove5669=== PAUSE TestQueueRemove5670=== RUN TestQueueFetchBatchLimit5671=== PAUSE TestQueueFetchBatchLimit5672=== RUN TestQueueRetryMovesToBack5673=== PAUSE TestQueueRetryMovesToBack5674=== RUN TestQueueFetchRemoveLifecycle5675=== PAUSE TestQueueFetchRemoveLifecycle5676=== RUN TestQueueConcurrentWriters5677=== PAUSE TestQueueConcurrentWriters5678=== RUN TestQueueRemoveLargeClosure5679=== PAUSE TestQueueRemoveLargeClosure5680=== RUN TestServerClientIntegration5681--- PASS: TestServerClientIntegration (0.00s)5682=== RUN TestServerQueueError56832026/11/12 17:02:46 ERROR Failed to queue paths error="permission denied" count=15684--- PASS: TestServerQueueError (0.00s)5685=== RUN TestGetListenerSocketActivation5686 server_test.go:220: === RUN TestGetListenerSocketActivation5687 --- PASS: TestGetListenerSocketActivation (0.00s)5688 PASS5689 5690--- PASS: TestGetListenerSocketActivation (0.01s)5691=== RUN TestServerStalledClientDoesNotBlockShutdown5692=== PAUSE TestServerStalledClientDoesNotBlockShutdown5693=== RUN TestServerBacksOffOnAcceptErrors5694=== PAUSE TestServerBacksOffOnAcceptErrors5695=== RUN TestDrainIsolatesPoisonPath5696=== PAUSE TestDrainIsolatesPoisonPath5697=== RUN TestRunNotBlockedByPoisonHead5698=== PAUSE TestRunNotBlockedByPoisonHead5699=== RUN TestDrainGivesUpWhenServerDown5700=== PAUSE TestDrainGivesUpWhenServerDown5701=== RUN TestFailedPathPrunedByLaterClosure5702=== PAUSE TestFailedPathPrunedByLaterClosure5703=== RUN TestWorkerUploadsAndRemoves5704=== PAUSE TestWorkerUploadsAndRemoves5705=== RUN TestWorkerSkipsGCdPaths5706=== PAUSE TestWorkerSkipsGCdPaths5707=== RUN TestWorkerPrunesClosureDeps5708=== PAUSE TestWorkerPrunesClosureDeps5709=== RUN TestDrainTimeout5710=== PAUSE TestDrainTimeout5711=== RUN TestWorkerRemoveFailureIsNotProgress5712=== PAUSE TestWorkerRemoveFailureIsNotProgress5713=== RUN TestWorkerKeepsPathItCannotStat5714=== PAUSE TestWorkerKeepsPathItCannotStat5715=== CONT TestSendPathsEmpty5716=== CONT TestDrainTimeout5717--- PASS: TestSendPathsEmpty (0.00s)5718=== CONT TestWorkerKeepsPathItCannotStat5719=== CONT TestWorkerRemoveFailureIsNotProgress5720=== CONT TestWorkerPrunesClosureDeps5721=== CONT TestQueueRemoveLargeClosure5722=== CONT TestQueueConcurrentWriters5723=== CONT TestQueueFetchRemoveLifecycle5724=== CONT TestQueueDeduplication5725=== CONT TestQueueEnqueueAndFetch5726=== CONT TestQueueRemove5727=== CONT TestQueueRetryMovesToBack5728=== CONT TestQueueFetchBatchLimit5729=== CONT TestDrainGivesUpWhenServerDown5730=== CONT TestWorkerUploadsAndRemoves5731=== CONT TestFailedPathPrunedByLaterClosure5732=== CONT TestWorkerSkipsGCdPaths5733=== CONT TestDrainIsolatesPoisonPath5734=== CONT TestServerBacksOffOnAcceptErrors5735=== CONT TestServerStalledClientDoesNotBlockShutdown57362000/01/01 00:00:00 ERROR Accept failed error="too many open files"57372000/01/01 00:00:00 ERROR Accept failed error="too many open files"57382000/01/01 00:00:00 ERROR Accept failed error="too many open files"57392000/01/01 00:00:00 ERROR Accept failed error="too many open files"57402000/01/01 00:00:00 ERROR Accept failed error="too many open files"57412000/01/01 00:00:00 ERROR Accept failed error="too many open files"57422000/01/01 00:00:00 ERROR Accept failed error="too many open files"57432000/01/01 00:00:00 ERROR Accept failed error="too many open files"5744=== CONT TestRunNotBlockedByPoisonHead5745--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)57462026/11/12 17:02:46 INFO Upload queue status pending=257472026/11/12 17:02:46 INFO Uploading batch count=257482026/11/12 17:02:46 INFO Uploading batch count=457492026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=457502026/11/12 17:02:46 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3307252423/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3307252423/002/locked/aaa: permission denied"57512026/11/12 17:02:46 INFO Upload queue status pending=257522026/11/12 17:02:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths3685600273/002/nonexistent57532026/11/12 17:02:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath309724691/002/bbb57542026/11/12 17:02:46 INFO Uploading batch count=157552026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=157562026/11/12 17:02:46 INFO Upload queue status pending=257572026/11/12 17:02:46 INFO Uploading batch count=157582026/11/12 17:02:46 INFO Upload queue status pending=357592026/11/12 17:02:46 INFO Uploading batch count=157602026/11/12 17:02:46 INFO Uploading batch count=157612026/11/12 17:02:46 INFO Uploading batch count=257622026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=257632026/11/12 17:02:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3397749188/002/a57642026/11/12 17:02:46 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3307252423/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3307252423/002/locked/aaa: permission denied"5765--- PASS: TestQueueEnqueueAndFetch (0.01s)57662026/11/12 17:02:46 INFO Uploading batch count=157672026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=157682000/01/01 00:00:00 INFO Uploading batch count=257692000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=257702000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45771--- PASS: TestQueueFetchRemoveLifecycle (0.01s)57722026/11/12 17:02:46 INFO Uploading batch count=15773--- PASS: TestQueueDeduplication (0.02s)5774--- PASS: TestQueueRemove (0.02s)57752026/11/12 17:02:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3397749188/002/b57762026/11/12 17:02:46 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3307252423/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3307252423/002/locked/aaa: permission denied"5777--- PASS: TestDrainTimeout (0.02s)5778--- PASS: TestQueueRetryMovesToBack (0.02s)57792026/11/12 17:02:46 ERROR Drain finished with paths left in queue remaining=15780--- PASS: TestQueueFetchBatchLimit (0.02s)57812026/11/12 17:02:46 INFO Uploading batch count=157822026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=157832026/11/12 17:02:46 INFO Uploading batch count=257842026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=257852026/11/12 17:02:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3397749188/002/c57862026/11/12 17:02:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1295397072/002/nonexistent57872026/11/12 17:02:46 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=157882026/11/12 17:02:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1295397072/002/nonexistent57892026/11/12 17:02:46 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=157902026/11/12 17:02:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1295397072/002/nonexistent57912026/11/12 17:02:46 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=157922026/11/12 17:02:46 INFO Uploading batch count=157932026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=157942026/11/12 17:02:46 ERROR Drain finished with paths left in queue remaining=157952026/11/12 17:02:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3397749188/002/d57962026/11/12 17:02:46 INFO Uploading batch count=157972026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=157982026/11/12 17:02:46 INFO Uploading batch count=257992026/11/12 17:02:46 ERROR Upload failed error="upload failed" count=258002026/11/12 17:02:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3397749188/002/e5801--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5802--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)58032026/11/12 17:02:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3397749188/002/f58042026/11/12 17:02:46 ERROR Drain finished with paths left in queue remaining=15805--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)58062026/11/12 17:02:46 ERROR Drain finished with paths left in queue remaining=105807--- PASS: TestDrainIsolatesPoisonPath (0.02s)5808--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5809--- PASS: TestWorkerUploadsAndRemoves (0.03s)5810--- PASS: TestWorkerSkipsGCdPaths (0.03s)5811--- PASS: TestWorkerPrunesClosureDeps (0.04s)5812--- PASS: TestQueueRemoveLargeClosure (0.10s)5813--- PASS: TestQueueConcurrentWriters (0.10s)58142026/11/12 17:02:46 ERROR Failed to decode request error="read unix /build/hook1978314320/test.sock->@: i/o timeout"58152026/11/12 17:02:46 ERROR Failed to write response error="write unix /build/hook1978314320/test.sock->@: i/o timeout"5816--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)58172026/11/12 17:02:47 INFO Uploading batch count=158182026/11/12 17:02:47 INFO Uploading batch count=158192026/11/12 17:02:47 INFO Uploading batch count=158202026/11/12 17:02:47 ERROR Upload failed error="upload failed" count=158212026/11/12 17:02:47 INFO Uploading batch count=158222026/11/12 17:02:47 ERROR Upload failed error="upload failed" count=158232026/11/12 17:02:47 INFO Uploading batch count=158242026/11/12 17:02:47 ERROR Upload failed error="upload failed" count=158252026/11/12 17:02:47 INFO Uploading batch count=158262026/11/12 17:02:47 ERROR Upload failed error="upload failed" count=158272026/11/12 17:02:47 ERROR Drain finished with paths left in queue remaining=15828--- PASS: TestRunNotBlockedByPoisonHead (1.02s)5829PASS5830Running signing tests...5831=== RUN TestExternalSigner5832=== PAUSE TestExternalSigner5833=== RUN TestExternalSignerWrongPublicKey5834=== PAUSE TestExternalSignerWrongPublicKey5835=== RUN TestExternalSignerErrors5836=== PAUSE TestExternalSignerErrors5837=== RUN TestExternalSignerRestartsAfterCrash5838=== PAUSE TestExternalSignerRestartsAfterCrash5839=== RUN TestExternalSignerProgramErrorKeepsProcess5840=== PAUSE TestExternalSignerProgramErrorKeepsProcess5841=== RUN TestExternalSignerClose5842=== PAUSE TestExternalSignerClose5843=== RUN TestExternalSignerRejectsInvalidPublicKey5844=== PAUSE TestExternalSignerRejectsInvalidPublicKey5845=== RUN TestExternalSignerCancellation5846=== PAUSE TestExternalSignerCancellation5847=== RUN TestGenerateFingerprint5848=== PAUSE TestGenerateFingerprint5849=== RUN TestParseSigningKey5850=== PAUSE TestParseSigningKey5851=== RUN TestSign5852=== PAUSE TestSign5853=== RUN TestSignBatch5854=== PAUSE TestSignBatch5855=== RUN TestNilKeyPublicKey5856=== PAUSE TestNilKeyPublicKey5857=== RUN TestSignErrors5858=== PAUSE TestSignErrors5859=== CONT TestExternalSigner5860=== CONT TestSignBatch5861=== CONT TestExternalSignerProgramErrorKeepsProcess5862=== CONT TestExternalSignerCancellation5863=== CONT TestSign5864=== CONT TestParseSigningKey5865=== RUN TestParseSigningKey/valid_32-byte_key5866=== PAUSE TestParseSigningKey/valid_32-byte_key5867=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5868=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5869=== RUN TestParseSigningKey/no_colon5870=== CONT TestGenerateFingerprint5871=== RUN TestGenerateFingerprint/basic_with_references5872=== CONT TestSignErrors5873=== CONT TestExternalSignerErrors5874=== CONT TestExternalSignerRestartsAfterCrash5875=== RUN TestExternalSignerErrors/exit_on_request5876=== PAUSE TestExternalSignerErrors/exit_on_request5877=== RUN TestExternalSignerErrors/null5878=== PAUSE TestExternalSignerErrors/null5879=== CONT TestExternalSignerRejectsInvalidPublicKey5880=== CONT TestNilKeyPublicKey5881=== CONT TestExternalSignerWrongPublicKey5882=== CONT TestExternalSignerClose5883=== PAUSE TestParseSigningKey/no_colon5884=== RUN TestParseSigningKey/empty_name5885=== PAUSE TestParseSigningKey/empty_name5886=== RUN TestParseSigningKey/invalid_base645887=== PAUSE TestParseSigningKey/invalid_base645888=== RUN TestParseSigningKey/wrong_length5889=== PAUSE TestParseSigningKey/wrong_length5890=== CONT TestParseSigningKey/valid_32-byte_key5891=== PAUSE TestGenerateFingerprint/basic_with_references5892--- PASS: TestNilKeyPublicKey (0.00s)5893=== RUN TestGenerateFingerprint/no_references5894=== PAUSE TestGenerateFingerprint/no_references5895=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5896=== CONT TestParseSigningKey/wrong_length5897=== CONT TestParseSigningKey/invalid_base645898=== CONT TestParseSigningKey/empty_name5899=== CONT TestParseSigningKey/no_colon5900=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5901=== RUN TestExternalSignerErrors/program_error5902=== RUN TestExternalSignerRejectsInvalidPublicKey/#005903=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005904=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5905=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5906=== PAUSE TestExternalSignerErrors/program_error5907=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5908=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5909=== RUN TestExternalSignerErrors/invalid_JSON5910=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5911=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5912=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5913=== PAUSE TestExternalSignerErrors/invalid_JSON5914=== RUN TestExternalSignerErrors/wrong_shape5915=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5916=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5917=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5918=== RUN TestGenerateFingerprint/invalid_nar_hash_length5919=== PAUSE TestExternalSignerErrors/wrong_shape5920=== RUN TestExternalSignerErrors/missing_signatures5921=== PAUSE TestExternalSignerErrors/missing_signatures5922=== RUN TestExternalSignerErrors/null_signatures5923=== PAUSE TestExternalSignerErrors/null_signatures5924=== RUN TestExternalSignerErrors/invalid_signature_base645925=== PAUSE TestExternalSignerErrors/invalid_signature_base645926=== RUN TestExternalSignerErrors/wrong_signatures_shape5927=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5928=== CONT TestExternalSignerRejectsInvalidPublicKey/#005929=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5930=== RUN TestGenerateFingerprint/invalid_store_path_prefix5931=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5932=== RUN TestGenerateFingerprint/invalid_reference_prefix5933=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5934=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5935=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5936=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5937=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5938--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5939 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5940 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5941 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5942 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5943 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5944=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5945=== CONT TestGenerateFingerprint/basic_with_references5946=== RUN TestExternalSignerErrors/too_few_signatures5947=== CONT TestGenerateFingerprint/invalid_nar_hash_length5948=== CONT TestGenerateFingerprint/invalid_reference_prefix5949=== PAUSE TestExternalSignerErrors/too_few_signatures5950=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5951--- PASS: TestParseSigningKey (0.00s)5952 --- PASS: TestParseSigningKey/wrong_length (0.00s)5953 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5954 --- PASS: TestParseSigningKey/empty_name (0.00s)5955 --- PASS: TestParseSigningKey/no_colon (0.00s)5956 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5957 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5958=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5959=== CONT TestGenerateFingerprint/no_references5960=== RUN TestSignErrors/nil_key5961=== PAUSE TestSignErrors/nil_key5962=== CONT TestGenerateFingerprint/invalid_store_path_prefix5963=== RUN TestExternalSignerErrors/unrequested_key5964=== PAUSE TestExternalSignerErrors/unrequested_key5965=== RUN TestExternalSignerErrors/missing_signature_name5966--- PASS: TestSign (0.00s)5967=== RUN TestSignErrors/nil_narinfo5968=== PAUSE TestExternalSignerErrors/missing_signature_name5969=== RUN TestExternalSignerErrors/wrong_signature_name5970--- PASS: TestGenerateFingerprint (0.00s)5971 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5972 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5973 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5974 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5975 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5976 --- PASS: TestGenerateFingerprint/no_references (0.00s)5977 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5978=== PAUSE TestSignErrors/nil_narinfo5979=== RUN TestSignErrors/invalid_narinfo5980=== PAUSE TestExternalSignerErrors/wrong_signature_name5981=== RUN TestExternalSignerErrors/exit_at_startup5982=== PAUSE TestExternalSignerErrors/exit_at_startup5983=== RUN TestExternalSignerErrors/wrong_id5984=== PAUSE TestExternalSignerErrors/wrong_id5985=== PAUSE TestSignErrors/invalid_narinfo5986--- PASS: TestSignBatch (0.00s)5987=== RUN TestExternalSignerErrors/wrong_signature_length5988--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5989=== PAUSE TestExternalSignerErrors/wrong_signature_length5990=== CONT TestExternalSignerErrors/exit_on_request5991=== CONT TestExternalSignerErrors/too_few_signatures5992=== CONT TestExternalSignerErrors/exit_at_startup5993=== CONT TestExternalSignerErrors/null5994=== CONT TestExternalSignerErrors/unrequested_key5995=== CONT TestExternalSignerErrors/missing_signatures5996=== CONT TestExternalSignerErrors/wrong_signatures_shape5997=== CONT TestExternalSignerErrors/invalid_signature_base645998=== CONT TestExternalSignerErrors/null_signatures5999=== CONT TestExternalSignerErrors/missing_signature_name6000=== CONT TestExternalSignerErrors/wrong_signature_name6001=== CONT TestExternalSignerErrors/wrong_signature_length6002=== RUN TestSignErrors/canceled6003=== CONT TestExternalSignerErrors/invalid_JSON6004=== CONT TestExternalSignerErrors/wrong_shape6005=== CONT TestExternalSignerErrors/wrong_id6006=== CONT TestExternalSignerErrors/program_error6007=== PAUSE TestSignErrors/canceled6008=== CONT TestSignErrors/nil_key6009=== CONT TestSignErrors/canceled6010--- PASS: TestExternalSignerClose (0.00s)6011=== CONT TestSignErrors/nil_narinfo6012=== CONT TestSignErrors/invalid_narinfo6013--- PASS: TestSignErrors (0.00s)6014 --- PASS: TestSignErrors/nil_key (0.00s)6015 --- PASS: TestSignErrors/canceled (0.00s)6016 --- PASS: TestSignErrors/nil_narinfo (0.00s)6017 --- PASS: TestSignErrors/invalid_narinfo (0.00s)6018--- PASS: TestExternalSignerWrongPublicKey (0.00s)6019--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)6020--- PASS: TestExternalSigner (0.01s)6021--- PASS: TestExternalSignerErrors (0.00s)6022 --- PASS: TestExternalSignerErrors/exit_on_request (0.00s)6023 --- PASS: TestExternalSignerErrors/too_few_signatures (0.00s)6024 --- PASS: TestExternalSignerErrors/exit_at_startup (0.00s)6025 --- PASS: TestExternalSignerErrors/unrequested_key (0.00s)6026 --- PASS: TestExternalSignerErrors/missing_signatures (0.00s)6027 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)6028 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)6029 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)6030 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)6031 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)6032 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)6033 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)6034 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)6035 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)6036 --- PASS: TestExternalSignerErrors/program_error (0.01s)6037 --- PASS: TestExternalSignerErrors/null (0.01s)6038--- PASS: TestExternalSignerCancellation (0.02s)6039PASS6040Running OpenBao signer tests...6041=== RUN TestSign6042=== PAUSE TestSign6043=== RUN TestSignErrors6044=== PAUSE TestSignErrors6045=== RUN TestServe6046=== PAUSE TestServe6047=== RUN TestServeRejectsGarbage6048=== PAUSE TestServeRejectsGarbage6049=== CONT TestSign6050=== CONT TestSignErrors6051=== CONT TestServe6052=== RUN TestSignErrors/invalid_base646053=== CONT TestServeRejectsGarbage6054=== PAUSE TestSignErrors/invalid_base646055=== RUN TestSignErrors/http_error6056=== PAUSE TestSignErrors/http_error6057=== RUN TestSignErrors/invalid_json6058=== PAUSE TestSignErrors/invalid_json6059=== RUN TestSignErrors/no_prefix6060=== PAUSE TestSignErrors/no_prefix6061=== CONT TestSignErrors/invalid_base646062=== CONT TestSignErrors/no_prefix6063--- PASS: TestServeRejectsGarbage (0.00s)6064=== CONT TestSignErrors/invalid_json6065=== CONT TestSignErrors/http_error6066--- PASS: TestServe (0.00s)6067--- PASS: TestSign (0.00s)6068--- PASS: TestSignErrors (0.00s)6069 --- PASS: TestSignErrors/invalid_json (0.00s)6070 --- PASS: TestSignErrors/http_error (0.00s)6071 --- PASS: TestSignErrors/no_prefix (0.00s)6072 --- PASS: TestSignErrors/invalid_base64 (0.00s)6073PASS