niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #502
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (2.37s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestScriptTokenEmptyCommand118=== CONT TestDoServerRequestAttachesToken119--- PASS: TestScriptTokenEmptyCommand (0.00s)120=== CONT TestSetClientTLSDoesNotMutateDefaultTransport121=== CONT TestResolveStorePath122--- PASS: TestResolveStorePath (0.00s)123=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun124=== CONT TestRegisterUploadedObject_OutlivesPush125=== CONT TestDoWithRetry_BodyReplayedViaGetBody126=== CONT TestScriptTokenScriptFails127--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)128=== CONT TestSetClientTLS129=== CONT TestScriptTokenBadJSON130=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer131=== CONT TestScriptTokenEmptyToken132=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess133--- PASS: TestScriptTokenScriptFails (4.46s)134=== CONT TestScriptTokenCachesUntilRefresh135=== CONT TestDumpPathMatchesNix136=== CONT TestScriptTokenNoExpiryRerunsEveryCall137=== CONT TestPartSizeForNAR138=== CONT TestFileTokenEmpty139=== CONT TestUploadMultipart_FailedProducerIsNotCompleted140=== CONT TestFileTokenMissing141=== CONT TestUploadMultipart_ReturnsAfterDroppedBody142=== CONT TestFileTokenReadsAndCaches143=== CONT TestUploadMultipart_PartsInParallel144=== CONT TestStaticToken145=== CONT TestFilterOversizedClosures146=== CONT TestSetClientTLSErrors147=== CONT TestCaseHackSuffix148=== RUN TestPartSizeForNAR/zero_stays_at_minimum149=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum150=== RUN TestPartSizeForNAR/small_stays_at_minimum151=== PAUSE TestPartSizeForNAR/small_stays_at_minimum152=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum153=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum154=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts155=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts156=== RUN TestPartSizeForNAR/1_TiB157=== PAUSE TestPartSizeForNAR/1_TiB158=== RUN TestPartSizeForNAR/5_TiB_S3_max_object159=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object160=== RUN TestPartSizeForNAR/capped_at_5_GiB161=== PAUSE TestPartSizeForNAR/capped_at_5_GiB162=== CONT TestStreamPushStopsWhenCanceled163=== RUN TestFilterOversizedClosures/no_limit_keeps_everything164=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything165=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped166=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped167=== RUN TestFilterOversizedClosures/all_closures_skipped168=== PAUSE TestFilterOversizedClosures/all_closures_skipped169=== CONT TestRegisterUploadedObjectReusesConnections1702026/11/12 00:34:58 WARN Rate limiter enabled after throttle name=server-test rate=5171=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part172=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry1732026/11/12 00:35:02 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:42457/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:42457/api/uploads/complete174=== CONT TestClientSignaturesByStorePath175=== CONT TestStreamPushReportsSkippedPaths1762026/11/12 00:35:02 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"1772026/11/12 00:35:02 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:42457/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:42457/api/uploads/complete1782026/11/12 00:35:02 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:42457/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:42457/api/uploads/complete1792026/11/12 00:35:02 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"1802026/11/12 00:35:02 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"181=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part182=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary183=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary184--- PASS: TestStaticToken (0.00s)185=== CONT TestStreamPushReportsSignatures186=== CONT TestStreamPushAnswersEveryLineWhenCanceled1872026/11/12 00:35:06 WARN Rate limiter enabled after throttle name=server-test rate=51882026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:45299189=== RUN TestSetClientTLSErrors/missing_cert_file190=== PAUSE TestSetClientTLSErrors/missing_cert_file191=== RUN TestSetClientTLSErrors/missing_key_file192=== PAUSE TestSetClientTLSErrors/missing_key_file193=== RUN TestSetClientTLSErrors/missing_ca_file194=== PAUSE TestSetClientTLSErrors/missing_ca_file195=== RUN TestSetClientTLSErrors/invalid_ca_file196=== PAUSE TestSetClientTLSErrors/invalid_ca_file197--- PASS: TestStreamPushStopsWhenCanceled (7.32s)198=== CONT TestStreamPushReportsEveryPath199--- PASS: TestFileTokenReadsAndCaches (5.22s)200--- PASS: TestFileTokenEmpty (0.00s)201--- PASS: TestClientSignaturesByStorePath (0.00s)202=== RUN TestSetClientTLS/rejects_connection_without_client_cert203=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert204=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA205=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA206=== RUN TestSetClientTLS/preserves_debug_logging_transport207=== PAUSE TestSetClientTLS/preserves_debug_logging_transport208=== CONT TestShellSplitErrors209--- PASS: TestShellSplitErrors (0.00s)210=== CONT TestStreamPushGivesUpOnDeadServer211=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica2122026/11/12 00:35:06 WARN Rate limiter backed off name=server-test rate=52132026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:45299214--- PASS: TestFileTokenMissing (0.00s)215=== CONT TestShellSplit216--- PASS: TestShellSplit (0.00s)217=== CONT TestStreamPushIsolatesFailures218--- PASS: TestScriptTokenBadJSON (13.08s)219=== CONT TestDoWithRetry_FinalResponseBodyReadable2202026/11/12 00:35:06 WARN Rate limiter enabled after throttle name=server-test rate=5221--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (13.20s)2222026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:42439223=== CONT TestStreamPushBatchesUnderLoad224=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false225=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false226=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true227=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true228=== CONT TestRetryTimeoutOutlastsMaxRetries229--- PASS: TestRegisterUploadedObject_OutlivesPush (13.20s)230=== CONT TestPathInfoHashCompatibility231=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)232=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)233=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon234=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon235=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI236=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI237=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512238=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512239=== CONT TestRateLimiterFeedback240=== RUN TestRateLimiterFeedback/429_enables_limiter241=== PAUSE TestRateLimiterFeedback/429_enables_limiter242=== RUN TestRateLimiterFeedback/503_enables_limiter243=== PAUSE TestRateLimiterFeedback/503_enables_limiter244=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter245=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter246=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter247=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter248=== CONT TestEncodeNixBase32WithRealHash249--- PASS: TestEncodeNixBase32WithRealHash (0.00s)250=== CONT TestPathInfoCACompatibility251=== RUN TestPathInfoCACompatibility/null_ca_field252=== PAUSE TestPathInfoCACompatibility/null_ca_field253=== RUN TestPathInfoCACompatibility/old_string_format_-_text254=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text255=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive2562026/11/12 00:35:06 ERROR Upload failed error="connection refused" count=202572026/11/12 00:35:06 ERROR Server seems unavailable, giving up on batch untried=17258--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (13.09s)259=== CONT TestParsePathInfoJSON260=== RUN TestParsePathInfoJSON/Nix_format261=== PAUSE TestParsePathInfoJSON/Nix_format262=== RUN TestParsePathInfoJSON/Lix_format263=== PAUSE TestParsePathInfoJSON/Lix_format264=== RUN TestParsePathInfoJSON/empty_input265=== PAUSE TestParsePathInfoJSON/empty_input266=== RUN TestParsePathInfoJSON/whitespace_only267=== PAUSE TestParsePathInfoJSON/whitespace_only268=== RUN TestParsePathInfoJSON/invalid_JSON269=== PAUSE TestParsePathInfoJSON/invalid_JSON270=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive271--- PASS: TestStreamPushReportsSkippedPaths (3.52s)272--- PASS: TestStreamPushReportsEveryPath (0.01s)273=== RUN TestPathInfoCACompatibility/new_structured_format_-_text274--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)275=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text2762026/11/12 00:35:06 INFO Garbage collection started277=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method278=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method279=== CONT TestEncodeNixBase32280=== RUN TestEncodeNixBase32/test_string_hash281=== CONT TestStreamPushRequestLine282=== CONT TestDumpPathWriterError2832026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:410512842000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=22852026/11/12 00:35:06 ERROR Upload failed error=boom count=1286--- PASS: TestStreamPushReportsSignatures (3.52s)287=== CONT TestPartSizeForNAR/zero_stays_at_minimum288=== CONT TestPartSizeForNAR/1_TiB289=== CONT TestPartSizeForNAR/capped_at_5_GiB290=== CONT TestPartSizeForNAR/5_TiB_S3_max_object291=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum292=== CONT TestParsePathInfoJSONMultiplePaths293--- PASS: TestDoServerRequestAttachesToken (13.20s)294--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (3.52s)295=== CONT TestPartSizeForNAR/small_stays_at_minimum296=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths297=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths298=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths299=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths300=== CONT TestFilterOversizedClosures/no_limit_keeps_everything301=== CONT TestConvertHashToNix32302=== RUN TestConvertHashToNix32/SRI_format_to_Nix32303=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32304=== RUN TestConvertHashToNix32/already_Nix32_format305=== PAUSE TestConvertHashToNix32/already_Nix32_format306=== RUN TestConvertHashToNix32/invalid_format307=== PAUSE TestConvertHashToNix32/invalid_format308=== CONT TestFilterOversizedClosures/all_closures_skipped3092026/11/12 00:35:06 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50310=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary311--- PASS: TestScriptTokenEmptyToken (13.09s)312=== CONT TestSetClientTLSErrors/missing_cert_file313=== CONT TestSetClientTLS/rejects_connection_without_client_cert314=== CONT TestSetClientTLS/preserves_debug_logging_transport315=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts316=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA3172026/11/12 00:35:06 INFO Garbage collection started3182026/11/12 00:35:06 ERROR Upload failed error=boom count=1319=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3202026/11/12 00:35:06 ERROR Upload failed error="bad path" count=3321=== CONT TestDumpPathSingleFile322=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false3232026/11/12 00:35:06 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000324=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true3252026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:410513262026/11/12 00:35:06 WARN Rate limiter backed off name=server-test rate=5327--- PASS: TestCaseHackSuffix (8.75s)328=== CONT TestGetStorePathHash329=== RUN TestGetStorePathHash/valid_store_path330=== PAUSE TestGetStorePathHash/valid_store_path331=== RUN TestGetStorePathHash/basename_without_hyphen_should_error332=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error333=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error334=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error335=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error336=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error337=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part338--- PASS: TestPartSizeForNAR (0.00s)339 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)340 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)341 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)342 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)343 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)344 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)345 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)346=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)347=== CONT TestRateLimiterFeedback/429_enables_limiter348=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512349--- PASS: TestStreamPushIsolatesFailures (0.01s)350=== PAUSE TestEncodeNixBase32/test_string_hash351=== RUN TestEncodeNixBase32/empty_input352=== PAUSE TestEncodeNixBase32/empty_input353=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon354=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI355--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.02s)356--- PASS: TestScriptTokenCachesUntilRefresh (0.02s)357--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (3.54s)358=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter359=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter360--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.04s)361=== CONT TestParsePathInfoJSON/whitespace_only362=== CONT TestParsePathInfoJSON/empty_input363=== CONT TestPathInfoCACompatibility/null_ca_field364=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method365=== CONT TestPathInfoCACompatibility/new_structured_format_-_text366--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (13.24s)367=== CONT TestParsePathInfoJSON/Lix_format368=== CONT TestPathInfoCACompatibility/old_string_format_-_text369=== CONT TestSetClientTLSErrors/invalid_ca_file370=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths371=== CONT TestConvertHashToNix32/SRI_format_to_Nix32372=== CONT TestSetClientTLSErrors/missing_ca_file373=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths374--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)375 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)376 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)377=== CONT TestConvertHashToNix32/invalid_format378=== CONT TestConvertHashToNix32/already_Nix32_format379--- PASS: TestConvertHashToNix32 (0.00s)380 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)381 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)382 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)3832026/11/12 00:35:06 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0384--- PASS: TestFilterOversizedClosures (0.02s)385 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)386 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)387 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)3882026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:41051389=== CONT TestGetStorePathHash/valid_store_path390--- PASS: TestPathInfoHashCompatibility (0.00s)391 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)392 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)393 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)394 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.02s)395=== CONT TestSetClientTLSErrors/missing_key_file396=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error397=== CONT TestEncodeNixBase32/test_string_hash398=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive399=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error400=== CONT TestGetStorePathHash/basename_without_hyphen_should_error401--- PASS: TestGetStorePathHash (0.00s)402 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)403 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)404 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)405 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)406--- PASS: TestStreamPushBatchesUnderLoad (0.00s)407 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.01s)408 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.01s)409--- PASS: TestPathInfoCACompatibility (0.00s)410 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)411 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)412 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)413 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)414 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)415=== CONT TestEncodeNixBase32/empty_input416--- PASS: TestEncodeNixBase32 (0.02s)417 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)418 --- PASS: TestEncodeNixBase32/empty_input (0.00s)419--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.00s)420 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.01s)421 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.02s)422=== CONT TestRateLimiterFeedback/503_enables_limiter423--- PASS: TestSetClientTLSErrors (3.50s)424 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)425 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)426 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)427 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)4282026/11/12 00:35:06 WARN Rate limiter enabled after throttle name=server-test rate=54292026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:410514302026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:338554312026/11/12 00:35:06 WARN Garbage collection finished on another replica. Its statistics are not available here432--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.04s)4332026/11/12 00:35:06 WARN Rate limiter backed off name=server-test rate=54342026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:41051435=== CONT TestParsePathInfoJSON/Nix_format436=== CONT TestParsePathInfoJSON/invalid_JSON437--- PASS: TestParsePathInfoJSON (0.00s)438 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)439 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)440 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)441 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)442 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)4432026/11/12 00:35:06 WARN Rate limiter enabled after throttle name=server-test rate=54442026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:465534452026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:410514462026/11/12 00:35:06 WARN Rate limiter backed off name=server-test rate=5447--- PASS: TestRateLimiterFeedback (0.00s)448 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)449 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)450 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)451 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.02s)4522026/11/12 00:35:06 WARN Rate limiter enabled after throttle name=s3-test rate=54532026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:409314542026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:41051455--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (3.56s)4562026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:410514572026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:410514582026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:410514592026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:410514602026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:410514612026/11/12 00:35:06 http: TLS handshake error from 127.0.0.1:54312: remote error: tls: bad certificate462--- PASS: TestSetClientTLS (13.20s)463 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.03s)464 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.04s)465 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.06s)4662026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:41051467--- PASS: TestStreamPushRequestLine (0.07s)468--- PASS: TestRegisterUploadedObjectReusesConnections (3.59s)4692026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:410514702026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:410514712026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:410514722026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:410514732026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:410514742026/11/12 00:35:06 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:41051475--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.08s)476--- PASS: TestDumpPathWriterError (0.08s)477--- PASS: TestDumpPathSingleFile (0.24s)478--- PASS: TestUploadMultipart_PartsInParallel (9.39s)479--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (13.89s)480--- PASS: TestDumpPathMatchesNix (4.57s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres4249212890/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres4249212890/data -l logfile start509510/build/postgres4249212890:5432 - no response511/build/postgres4249212890:5432 - no response5122026-11-12 00:35:17.915 UTC [132] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5132026-11-12 00:35:17.916 UTC [132] LOG: listening on Unix socket "/build/postgres4249212890/.s.PGSQL.5432"5142026-11-12 00:35:17.921 UTC [140] LOG: database system was shut down at 2026-11-12 00:35:17 UTC5152026-11-12 00:35:17.925 UTC [132] LOG: database system is ready to accept connections516/build/postgres4249212890:5432 - accepting connections517{"timestamp":"2026-11-12T00:35:55.695636701Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"9e8413d6-ac0e-4f14-ad2f-2f243f16766a","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":9,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(208)"}518{"timestamp":"2026-11-12T00:35:55.804190661Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3000d89c-a10a-48bc-a0e4-3e51c7059ed7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(206)"}519=== RUN TestAbortSucceeded520=== PAUSE TestAbortSucceeded521=== RUN TestService_AuthMiddleware522=== PAUSE TestService_AuthMiddleware523=== RUN TestService_AuthMiddleware_MTLSProxyHeader524=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader525=== RUN TestService_AuthMiddleware_MTLSBoundSubjects526=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects527=== RUN TestService_ReadAuthMiddleware528=== PAUSE TestService_ReadAuthMiddleware529=== RUN TestService_AuthMiddleware_OIDC530=== PAUSE TestService_AuthMiddleware_OIDC531=== RUN TestService_RequireScope_OIDC532=== PAUSE TestService_RequireScope_OIDC533=== RUN TestService_ReadScope_PublicByDefault534=== PAUSE TestService_ReadScope_PublicByDefault535=== RUN TestCacheConfigHandler536=== PAUSE TestCacheConfigHandler537=== RUN TestCacheStatsHandler538=== PAUSE TestCacheStatsHandler539=== RUN TestClientCADerivations540=== PAUSE TestClientCADerivations541=== RUN TestClientErrorHandling542=== PAUSE TestClientErrorHandling543=== RUN TestClientIntegration544=== PAUSE TestClientIntegration545=== RUN TestClientMultipleUploads546=== PAUSE TestClientMultipleUploads547=== RUN TestClientWithDependencies548=== PAUSE TestClientWithDependencies549=== RUN TestClientSharedPathCommittedMidPush550=== PAUSE TestClientSharedPathCommittedMidPush551=== RUN TestPinProtectsFromGC552=== PAUSE TestPinProtectsFromGC553=== RUN TestClientPushesUseOnePush554=== PAUSE TestClientPushesUseOnePush555=== RUN TestClientFallsBackToClosures556=== PAUSE TestClientFallsBackToClosures557=== RUN TestDeleteClosuresRacingPins558=== PAUSE TestDeleteClosuresRacingPins559=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock560=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock561=== RUN TestResolveDBConnectionString562=== PAUSE TestResolveDBConnectionString563=== RUN TestConnectWaitsForAPeerMigration564=== PAUSE TestConnectWaitsForAPeerMigration565=== RUN TestLeadElectsOneAndHandsOver566=== PAUSE TestLeadElectsOneAndHandsOver567=== RUN TestLeadIncumbentWinsAfterRestart5682026/11/12 00:35:58 INFO lead: acquired remote=192.0.2.1:1234 generation=15692026/11/12 00:35:59 INFO lead: released remote=192.0.2.1:1234 generation=15702026/11/12 00:35:59 INFO lead: acquired remote=192.0.2.1:1234 generation=25712026/11/12 00:36:00 INFO lead: released remote=192.0.2.1:1234 generation=2572--- PASS: TestLeadIncumbentWinsAfterRestart (4.40s)573=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent574=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent575=== RUN TestLeadIncumbentAnnouncesAtOnce576=== PAUSE TestLeadIncumbentAnnouncesAtOnce577=== RUN TestLeadGenerationGrowsWithEveryGrant578=== PAUSE TestLeadGenerationGrowsWithEveryGrant579=== RUN TestLeadEndsOnShutdown580=== PAUSE TestLeadEndsOnShutdown581=== RUN TestLeadEndsWhenItsConnectionHangs5822026/11/12 00:36:01 INFO lead: acquired remote=192.0.2.1:1234 generation=15832026-11-12 00:36:02.977 UTC [391] FATAL: terminating connection due to administrator command5842026/11/12 00:36:03 INFO lead: acquired remote=192.0.2.1:1234 generation=25852026/11/12 00:36:04 WARN lead: lock connection lost error="timeout: context deadline exceeded"5862026/11/12 00:36:04 INFO lead: released remote=192.0.2.1:1234 generation=15872026/11/12 00:36:04 INFO lead: released remote=192.0.2.1:1234 generation=2588--- PASS: TestLeadEndsWhenItsConnectionHangs (4.24s)589=== RUN TestGCAdvisoryLockBlocksConcurrentRun590=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun591=== RUN TestGCBugBareHashReferences592=== PAUSE TestGCBugBareHashReferences593=== RUN TestGCMetrics594=== PAUSE TestGCMetrics595=== RUN TestPushDedupSurvivesConcurrentGC596=== PAUSE TestPushDedupSurvivesConcurrentGC597=== RUN TestDeduplicatedObjectsRecordedAsPending598=== PAUSE TestDeduplicatedObjectsRecordedAsPending599=== RUN TestGCSweepSkipsPendingObjects600=== PAUSE TestGCSweepSkipsPendingObjects601=== RUN TestTombstonedObjectOfferedWithoutWaiting602=== PAUSE TestTombstonedObjectOfferedWithoutWaiting603=== RUN TestGCSweepDeliversEachKeyOnce604=== PAUSE TestGCSweepDeliversEachKeyOnce605=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection606=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection607=== RUN TestForceGCDuringPushOffersSweptObject608=== PAUSE TestForceGCDuringPushOffersSweptObject609=== RUN TestFailedS3DeleteKeepsTombstone610=== PAUSE TestFailedS3DeleteKeepsTombstone611=== RUN TestSweepSkipsRowsLockedByPush612=== PAUSE TestSweepSkipsRowsLockedByPush613=== RUN TestRegisterCompletedObjectNeedsPendingRow614=== PAUSE TestRegisterCompletedObjectNeedsPendingRow615=== RUN TestGetExistingObjectsLocksRows616=== PAUSE TestGetExistingObjectsLocksRows617=== RUN TestSweepRechecksPendingAfterLocking618=== PAUSE TestSweepRechecksPendingAfterLocking619=== RUN TestPushWaitsForSweepHoldingRow620=== PAUSE TestPushWaitsForSweepHoldingRow621=== RUN TestCommitRacingPendingCleanupKeepsObjects622=== PAUSE TestCommitRacingPendingCleanupKeepsObjects623=== RUN TestGCEndsOnShutdown624=== PAUSE TestGCEndsOnShutdown625=== RUN TestGCTaskStore_StartNew626=== PAUSE TestGCTaskStore_StartNew627=== RUN TestGCTaskStore_DeduplicateSameParams628=== PAUSE TestGCTaskStore_DeduplicateSameParams629=== RUN TestGCTaskStore_ConflictDifferentParams630=== PAUSE TestGCTaskStore_ConflictDifferentParams631=== RUN TestGCTaskStore_GetEmpty632=== PAUSE TestGCTaskStore_GetEmpty633=== RUN TestGCTaskStore_GetReturnsLatest634=== PAUSE TestGCTaskStore_GetReturnsLatest635=== RUN TestGCTaskStore_CompletedAllowsNewTask636=== PAUSE TestGCTaskStore_CompletedAllowsNewTask637=== RUN TestGCTaskStore_PhaseUpdates638=== PAUSE TestGCTaskStore_PhaseUpdates639=== RUN TestGCTaskStore_Fail640=== PAUSE TestGCTaskStore_Fail641=== RUN TestGracefulShutdownDrainsInflight642=== PAUSE TestGracefulShutdownDrainsInflight643=== RUN TestService_healthCheckHandler644=== PAUSE TestService_healthCheckHandler645=== RUN TestService_readinessHandler646=== PAUSE TestService_readinessHandler647=== RUN TestGenerateLandingPage648=== PAUSE TestGenerateLandingPage649=== RUN TestEnsureAbortUploadsRule650=== PAUSE TestEnsureAbortUploadsRule651=== RUN TestCacheConfigHandlerMaxNarSize652=== PAUSE TestCacheConfigHandlerMaxNarSize653=== RUN TestCreatePendingClosureRejectsOversizedNAR654=== PAUSE TestCreatePendingClosureRejectsOversizedNAR655=== RUN TestNARDeduplicationMetadataUploadBug656=== PAUSE TestNARDeduplicationMetadataUploadBug657=== RUN TestMetricsInventory658=== PAUSE TestMetricsInventory659=== RUN TestMetricsCommit660=== PAUSE TestMetricsCommit661=== RUN TestService_NativeMTLS662=== PAUSE TestService_NativeMTLS663=== RUN TestServerTLSConfig664=== PAUSE TestServerTLSConfig665=== RUN TestMultipartCleanup666=== PAUSE TestMultipartCleanup667=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded668=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded669=== RUN TestPendingCleanupUsesOneCutoff670=== PAUSE TestPendingCleanupUsesOneCutoff671=== RUN TestObjectStatsTrigger672=== PAUSE TestObjectStatsTrigger673=== RUN TestObjectStatsUpdatedOncePerCommit674=== PAUSE TestObjectStatsUpdatedOncePerCommit675=== RUN TestOrphanedObjectsGC676=== PAUSE TestOrphanedObjectsGC677=== RUN TestOrphanedObjectsGCStressTest678=== PAUSE TestOrphanedObjectsGCStressTest679=== RUN TestCreatePin_ReservedPins680=== PAUSE TestCreatePin_ReservedPins681=== RUN TestPresentReportsOnlyClosureRoots682=== PAUSE TestPresentReportsOnlyClosureRoots683=== RUN TestPresentRefreshesOnlyStaleClosures684=== PAUSE TestPresentRefreshesOnlyStaleClosures685=== RUN TestPresentNotReportedWhileGCDeletesClosure686=== PAUSE TestPresentNotReportedWhileGCDeletesClosure687=== RUN TestParseSingleRange688=== PAUSE TestParseSingleRange689=== RUN TestProxyHeadersOnlyTrustedOnSocket6902026/11/12 00:36:06 INFO Starting HTTP server address=127.0.0.1:454296912026/11/12 00:36:06 INFO Starting HTTP server address=proxy.sock6922026/11/12 00:36:06 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6932026/11/12 00:36:06 INFO Shutdown signal received, draining in-flight requests timeout=10s694--- PASS: TestProxyHeadersOnlyTrustedOnSocket (1.67s)695=== RUN TestIsValidCachePath696=== PAUSE TestIsValidCachePath697=== RUN TestReadProxyNarinfo698=== PAUSE TestReadProxyNarinfo699=== RUN TestReadProxyNarinfoAlreadyDecompressed700=== PAUSE TestReadProxyNarinfoAlreadyDecompressed701=== RUN TestReadProxyContentNegotiation702=== PAUSE TestReadProxyContentNegotiation703=== RUN TestReadProxyNarStreaming704=== PAUSE TestReadProxyNarStreaming705=== RUN TestReadProxy404706=== PAUSE TestReadProxy404707=== RUN TestReadProxyInvalidPath708=== PAUSE TestReadProxyInvalidPath709=== RUN TestReadProxyHead710=== PAUSE TestReadProxyHead711=== RUN TestReadProxyOutlastsServerWriteTimeout712=== PAUSE TestReadProxyOutlastsServerWriteTimeout713=== RUN TestReadProxyConditionalGet714=== PAUSE TestReadProxyConditionalGet715=== RUN TestReadProxyRootRedirectsToIndexHTML716=== PAUSE TestReadProxyRootRedirectsToIndexHTML717=== RUN TestReadProxyDisabled718=== PAUSE TestReadProxyDisabled719=== RUN TestReadRedirectNar720=== PAUSE TestReadRedirectNar721=== RUN TestReadRedirectKeepsNarinfoProxied722=== PAUSE TestReadRedirectKeepsNarinfoProxied723=== RUN TestReadProxyRangeRequest724=== PAUSE TestReadProxyRangeRequest725=== RUN TestReadRedirectUsesPublicS3URL726=== PAUSE TestReadRedirectUsesPublicS3URL727=== RUN TestPush_OverlappingRootsStoreOneRowPerKey728=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey729=== RUN TestPush_CompleteCommitsEveryRoot730=== PAUSE TestPush_CompleteCommitsEveryRoot731=== RUN TestPush_RepeatedRootCommitsOnce732=== PAUSE TestPush_RepeatedRootCommitsOnce733=== RUN TestPush_CompleteWaitsForCleanupOfThePush734=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush735=== RUN TestPush_RootsAreLockedInByteOrder736=== PAUSE TestPush_RootsAreLockedInByteOrder737=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit738=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit739=== RUN TestPush_RejectsBadRequests740=== PAUSE TestPush_RejectsBadRequests741=== RUN TestPush_SignsNarinfosOfItsPendingObjects742=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects743=== RUN TestSharedNarBothUploadsComplete744=== PAUSE TestSharedNarBothUploadsComplete745=== RUN TestCompleteMultipartUpload_ErrorButObjectExists746=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists747=== RUN TestCompletedNarNotReofferedAcrossClosures748=== PAUSE TestCompletedNarNotReofferedAcrossClosures749=== RUN TestPresignedUploadRegisteredBeforeCommit750=== PAUSE TestPresignedUploadRegisteredBeforeCommit751=== RUN TestService_Rustfstest752=== PAUSE TestService_Rustfstest753=== RUN TestParseSize754=== PAUSE TestParseSize755=== RUN TestSkippedUploadsHandler756=== PAUSE TestSkippedUploadsHandler757=== RUN TestSystemdListenerNotActivated758--- PASS: TestSystemdListenerNotActivated (0.00s)759=== RUN TestWatchdogBeatsWhenHealthy760--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)761=== RUN TestWatchdogSkipsWhenUnhealthy7622026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7702026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7712026/11/12 00:36:06 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"772--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)773=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle774=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle775=== RUN TestProxyWriteTimeout776=== PAUSE TestProxyWriteTimeout777=== RUN TestPendingClosureWriteTimeout778=== PAUSE TestPendingClosureWriteTimeout779=== RUN TestIsValidUploadKey780=== PAUSE TestIsValidUploadKey781=== RUN TestUploadHandlersRejectInvalidKeys782=== PAUSE TestUploadHandlersRejectInvalidKeys783=== RUN TestUploadHandlersRejectOversizedBody784=== PAUSE TestUploadHandlersRejectOversizedBody785=== RUN TestService_cleanupPendingClosuresHandler786=== PAUSE TestService_cleanupPendingClosuresHandler787=== RUN TestService_createPendingClosureHandler788=== PAUSE TestService_createPendingClosureHandler789=== RUN TestService_verifyS3Integrity790=== PAUSE TestService_verifyS3Integrity791=== RUN TestService_verifyS3ReuploadsStaleNarinfo792=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo793=== RUN TestCompleteMultipartUnregistered794=== PAUSE TestCompleteMultipartUnregistered795=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT796=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT797=== CONT TestPendingClosureWriteTimeout798=== CONT TestService_verifyS3Integrity799=== CONT TestAbortSucceeded800=== CONT TestProxyWriteTimeout801=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle802=== RUN TestProxyWriteTimeout/narinfo803=== PAUSE TestProxyWriteTimeout/narinfo804=== CONT TestSkippedUploadsHandler805=== CONT TestParseSize806--- PASS: TestAbortSucceeded (0.00s)807=== CONT TestService_Rustfstest808=== CONT TestPresignedUploadRegisteredBeforeCommit809=== CONT TestCompletedNarNotReofferedAcrossClosures810=== CONT TestService_createPendingClosureHandler811=== CONT TestCompleteMultipartUpload_ErrorButObjectExists812=== CONT TestSharedNarBothUploadsComplete813=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT814=== CONT TestPush_SignsNarinfosOfItsPendingObjects815=== CONT TestCompleteMultipartUnregistered816=== CONT TestPush_RejectsBadRequests817=== CONT TestService_verifyS3ReuploadsStaleNarinfo818=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit819=== CONT TestPush_RootsAreLockedInByteOrder820=== CONT TestPush_CompleteWaitsForCleanupOfThePush821=== CONT TestPush_RepeatedRootCommitsOnce822=== CONT TestPush_CompleteCommitsEveryRoot823=== CONT TestPush_OverlappingRootsStoreOneRowPerKey824=== RUN TestPendingClosureWriteTimeout/empty825=== RUN TestProxyWriteTimeout/1_GiB_nar826--- PASS: TestParseSize (0.00s)827=== CONT TestReadProxyRangeRequest828=== CONT TestReadRedirectUsesPublicS3URL8292026/11/12 00:36:06 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000830=== CONT TestGCEndsOnShutdown831=== RUN TestGCEndsOnShutdown/before_the_run832=== PAUSE TestGCEndsOnShutdown/before_the_run833=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete834=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete835=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark836=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark837=== CONT TestCommitRacingPendingCleanupKeepsObjects838=== PAUSE TestPendingClosureWriteTimeout/empty839=== RUN TestPendingClosureWriteTimeout/400_objects840=== PAUSE TestPendingClosureWriteTimeout/400_objects841=== RUN TestPendingClosureWriteTimeout/670k_objects842=== PAUSE TestPendingClosureWriteTimeout/670k_objects843=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline844=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline845=== CONT TestPushWaitsForSweepHoldingRow846--- PASS: TestSkippedUploadsHandler (0.01s)847=== PAUSE TestProxyWriteTimeout/1_GiB_nar848=== RUN TestProxyWriteTimeout/10_GiB_nar849=== PAUSE TestProxyWriteTimeout/10_GiB_nar850=== RUN TestProxyWriteTimeout/unknown_size851=== PAUSE TestProxyWriteTimeout/unknown_size852=== CONT TestSweepRechecksPendingAfterLocking853=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed854=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed855=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed856=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed857=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost858=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost859=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed860=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed861=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed862=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed863=== CONT TestGetExistingObjectsLocksRows8642026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes8652026/11/12 00:36:09 INFO Received uploads request method=POST path=/api/pending_closures866=== RUN TestPush_RejectsBadRequests/no_objects867=== PAUSE TestPush_RejectsBadRequests/no_objects868=== RUN TestPush_RejectsBadRequests/bad_root869=== PAUSE TestPush_RejectsBadRequests/bad_root870=== RUN TestPush_RejectsBadRequests/root_not_in_objects871=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects872=== RUN TestPush_RejectsBadRequests/no_roots873=== PAUSE TestPush_RejectsBadRequests/no_roots874=== CONT TestRegisterCompletedObjectNeedsPendingRow8752026/11/12 00:36:09 INFO Received uploads request method=POST path=/api/pending_closures876--- PASS: TestService_Rustfstest (3.04s)877=== CONT TestSweepSkipsRowsLockedByPush8782026/11/12 00:36:09 INFO Received uploads request method=POST path=/api/pending_closures8792026/11/12 00:36:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete8802026/11/12 00:36:09 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst881--- PASS: TestCompleteMultipartUnregistered (2.90s)882=== CONT TestFailedS3DeleteKeepsTombstone8832026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes8842026/11/12 00:36:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8852026/11/12 00:36:09 INFO Signed narinfos id=1 count=18862026/11/12 00:36:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8872026/11/12 00:36:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8882026/11/12 00:36:09 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"8892026/11/12 00:36:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8902026/11/12 00:36:09 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"8912026/11/12 00:36:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8922026/11/12 00:36:09 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=18932026/11/12 00:36:09 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8942026/11/12 00:36:09 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo895--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (2.99s)896=== CONT TestForceGCDuringPushOffersSweptObject897=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows898=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows899=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check900=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check901=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection9022026/11/12 00:36:09 INFO Received uploads request method=POST path=/api/pending_closures903--- PASS: TestGetExistingObjectsLocksRows (2.96s)904=== CONT TestGCTaskStore_StartNew905--- PASS: TestGCTaskStore_StartNew (0.00s)906=== CONT TestReadRedirectKeepsNarinfoProxied9072026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes908--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (2.98s)909=== CONT TestReadRedirectNar9102026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes9112026/11/12 00:36:09 INFO Received complete push request method=POST path=/api/pushes/1/complete9122026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes9132026/11/12 00:36:09 INFO Completed push id=1 objects=4 duration=18.862458ms9142026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes9152026/11/12 00:36:09 INFO Received uploads request method=POST path=/api/pending_closures9162026/11/12 00:36:09 INFO Received uploads request method=POST path=/api/pending_closures9172026/11/12 00:36:09 INFO Received uploads request method=POST path=/api/pending_closures9182026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes9192026-11-12 00:36:09.803 UTC [496] ERROR: Push does not exist: id=19202026-11-12 00:36:09.803 UTC [496] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE9212026-11-12 00:36:09.803 UTC [496] STATEMENT: SELECT commit_push($1)922--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (3.32s)923=== CONT TestGCSweepDeliversEachKeyOnce9242026/11/12 00:36:09 INFO Aborted multipart uploads count=0 kept=09252026/11/12 00:36:09 INFO Received complete push request method=POST path=/api/pushes/1/complete9262026/11/12 00:36:09 INFO Completed push id=1 objects=4 duration=12.122873ms927--- PASS: TestPush_RepeatedRootCommitsOnce (3.20s)928=== CONT TestReadProxyDisabled9292026/11/12 00:36:09 INFO Received push request method=POST path=/api/pushes9302026/11/12 00:36:09 INFO Received complete push request method=POST path=/api/pushes/1/complete9312026/11/12 00:36:09 INFO Aborted multipart uploads count=0 kept=09322026/11/12 00:36:09 INFO Received complete multipart upload request method=POST path=/api/multipart/complete933--- PASS: TestReadRedirectUsesPublicS3URL (3.52s)934=== CONT TestTombstonedObjectOfferedWithoutWaiting9352026/11/12 00:36:10 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLjhjZTY0MDFjLTJjMDktNDhmOS1hZjA2LTRkMTU2NjU5ZTg3OHgxNzk0NDQzNzY5NTQ3MTc5NjYz parts=109362026/11/12 00:36:10 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9372026/11/12 00:36:10 INFO Completed upload id=1 objects=2 duration=5.102047ms9382026/11/12 00:36:10 INFO Aborted multipart uploads count=0 kept=09392026/11/12 00:36:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9402026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures9412026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures9422026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures9432026/11/12 00:36:10 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo9442026/11/12 00:36:10 WARN Found objects in DB but missing from S3, will re-upload count=1945--- PASS: TestService_verifyS3Integrity (3.59s)946=== CONT TestReadProxyRootRedirectsToIndexHTML9472026/11/12 00:36:10 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLjUzNTE5NTdlLTBhYzYtNDFiOS1hYzNmLWQ1YjU3Nzk5MTA0MngxNzk0NDQzNzY5NDg0NjcwMDc2 parts=129482026/11/12 00:36:10 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst9492026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures950--- PASS: TestPresignedUploadRegisteredBeforeCommit (3.62s)951=== CONT TestGCSweepSkipsPendingObjects9522026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures953--- PASS: TestPushWaitsForSweepHoldingRow (3.47s)954=== CONT TestReadProxyConditionalGet9552026/11/12 00:36:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9562026/11/12 00:36:10 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLmQzNzU5NDcwLTM3MDktNDY4ZC05ZWUwLTBlYjE1YjBlMDIxY3gxNzk0NDQzNzcwMTE1MDI5ODE29572026/11/12 00:36:10 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLmQzNzU5NDcwLTM3MDktNDY4ZC05ZWUwLTBlYjE1YjBlMDIxY3gxNzk0NDQzNzcwMTE1MDI5ODE2 parts=1958--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (3.71s)959=== CONT TestReadProxyOutlastsServerWriteTimeout9602026/11/12 00:36:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9612026/11/12 00:36:10 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLjhlZmIzMzYwLWE5MTMtNDg1Yy1hYWEyLTc0NGQ0YzkxOTFjM3gxNzk0NDQzNzY5NzgzNjI3OTI0 parts=109622026/11/12 00:36:10 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9632026/11/12 00:36:10 INFO Completed upload id=1 objects=2 duration=6.533402ms9642026/11/12 00:36:10 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000009652026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures9662026/11/12 00:36:10 INFO Starting cleanup of old closures method=DELETE path=/api/closures9672026/11/12 00:36:10 INFO Aborted multipart uploads count=0 kept=09682026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures969--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (3.78s)970=== CONT TestDeduplicatedObjectsRecordedAsPending9712026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures9722026/11/12 00:36:10 INFO Received uploads request method=POST path=/api/pending_closures973--- PASS: TestReadProxyRangeRequest (3.90s)974=== CONT TestReadProxyHead9752026/11/12 00:36:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9762026/11/12 00:36:10 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9772026/11/12 00:36:10 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLmYwMzhjN2RiLWRlMzQtNDVlYy1iMTg5LTdmYzdiNzZmMDJmNXgxNzk0NDQzNzY5NTAxOTk1MDc5 parts=12978--- PASS: TestSharedNarBothUploadsComplete (4.15s)979=== CONT TestPushDedupSurvivesConcurrentGC9802026/11/12 00:36:11 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9812026/11/12 00:36:11 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLmU3NGI5ZTg5LTZjZDktNDI4Mi1hZGRhLTUzYWE1Zjk1ZGQzMXgxNzk0NDQzNzcwNTQ5NjIwNTc4 parts=129822026/11/12 00:36:11 INFO Received uploads request method=POST path=/api/pending_closures983--- PASS: TestCompletedNarNotReofferedAcrossClosures (4.74s)984=== CONT TestReadProxyInvalidPath9852026/11/12 00:36:12 WARN Force mode enabled - objects will be deleted immediately without grace period9862026-11-12 00:36:12.197 UTC [519] ERROR: could not obtain lock on row in relation "closures"9872026-11-12 00:36:12.197 UTC [519] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9882026/11/12 00:36:12 WARN Force mode enabled - objects will be deleted immediately without grace period9892026-11-12 00:36:12.199 UTC [519] ERROR: could not obtain lock on row in relation "closures"9902026-11-12 00:36:12.199 UTC [519] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9912026-11-12 00:36:12.200 UTC [519] ERROR: could not obtain lock on row in relation "closures"9922026-11-12 00:36:12.200 UTC [519] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9932026-11-12 00:36:12.200 UTC [519] ERROR: could not obtain lock on row in relation "closures"9942026-11-12 00:36:12.200 UTC [519] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9952026-11-12 00:36:12.200 UTC [519] ERROR: could not obtain lock on row in relation "closures"9962026-11-12 00:36:12.200 UTC [519] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9972026/11/12 00:36:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=0998--- PASS: TestPush_RootsAreLockedInByteOrder (5.53s)999=== CONT TestGCMetrics10002026/11/12 00:36:12 INFO Vacuumed table table=pending_closures10012026/11/12 00:36:12 INFO Vacuumed table table=pending_objects10022026/11/12 00:36:12 INFO Vacuumed table table=multipart_uploads10032026/11/12 00:36:12 INFO Vacuumed table table=closures10042026/11/12 00:36:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010052026/11/12 00:36:12 INFO Vacuumed table table=objects10062026/11/12 00:36:12 INFO Vacuumed table table=pending_closures10072026/11/12 00:36:12 INFO Vacuumed table table=pending_objects10082026/11/12 00:36:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=010092026/11/12 00:36:12 INFO Vacuumed table table=multipart_uploads10102026/11/12 00:36:12 INFO Vacuumed table table=closures10112026/11/12 00:36:12 INFO Vacuumed table table=objects10122026/11/12 00:36:12 INFO Vacuumed table table=pending_closures1013--- PASS: TestSweepRechecksPendingAfterLocking (5.58s)1014=== CONT TestService_NativeMTLS10152026/11/12 00:36:12 INFO Vacuumed table table=pending_objects10162026/11/12 00:36:12 INFO Vacuumed table table=multipart_uploads10172026/11/12 00:36:12 INFO Vacuumed table table=closures10182026/11/12 00:36:12 INFO Vacuumed table table=objects10192026/11/12 00:36:12 INFO Received complete push request method=POST path=/api/pushes/2/complete10202026/11/12 00:36:12 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001021--- PASS: TestService_createPendingClosureHandler (5.80s)1022=== CONT TestGCBugBareHashReferences10232026/11/12 00:36:12 INFO Completed push id=2 objects=4 duration=5.451292ms1024--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (5.60s)1025=== CONT TestMetricsCommit1026--- PASS: TestReadProxyInvalidPath (1.07s)1027=== CONT TestGCAdvisoryLockBlocksConcurrentRun10282026/11/12 00:36:12 INFO Received uploads request method=POST path=/api/pending_closures10292026/11/12 00:36:12 INFO Received uploads request method=POST path=/api/pending_closures1030--- PASS: TestDeduplicatedObjectsRecordedAsPending (1.92s)1031=== CONT TestMetricsInventory1032--- PASS: TestReadProxyRootRedirectsToIndexHTML (2.33s)1033=== CONT TestLeadEndsOnShutdown10342026/11/12 00:36:12 INFO Aborted multipart uploads count=0 kept=01035--- PASS: TestReadProxyHead (1.86s)1036=== CONT TestNARDeduplicationMetadataUploadBug10372026/11/12 00:36:12 INFO Received uploads request method=POST path=/api/pending_closures1038--- PASS: TestTombstonedObjectOfferedWithoutWaiting (2.48s)1039=== CONT TestLeadGenerationGrowsWithEveryGrant1040--- PASS: TestReadRedirectKeepsNarinfoProxied (2.84s)1041=== CONT TestCreatePendingClosureRejectsOversizedNAR10422026/11/12 00:36:12 INFO Received uploads request method=POST path=/api/pending_closures1043--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1044=== CONT TestLeadIncumbentAnnouncesAtOnce10452026/11/12 00:36:12 INFO Received uploads request method=POST path=/api/pending_closures10462026/11/12 00:36:12 INFO Aborted multipart uploads count=0 kept=01047--- PASS: TestReadRedirectNar (2.89s)1048=== CONT TestCacheConfigHandlerMaxNarSize1049--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1050=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent1051--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (3.13s)1052=== CONT TestLeadElectsOneAndHandsOver10532026/11/12 00:36:14 WARN Rate limiter enabled after throttle name=s3-test rate=510542026/11/12 00:36:14 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1055=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1056 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101057 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001058--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (8.36s)1059=== CONT TestServerTLSConfig1060=== RUN TestServerTLSConfig/no_client_CA1061=== PAUSE TestServerTLSConfig/no_client_CA1062=== RUN TestServerTLSConfig/missing_CA_file1063=== PAUSE TestServerTLSConfig/missing_CA_file1064=== RUN TestServerTLSConfig/not_a_PEM_file1065=== PAUSE TestServerTLSConfig/not_a_PEM_file1066=== CONT TestConnectWaitsForAPeerMigration10672026/11/12 00:36:15 WARN Force mode enabled - objects will be deleted immediately without grace period10682026/11/12 00:36:15 WARN Force mode enabled - objects will be deleted immediately without grace period10692026/11/12 00:36:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010702026/11/12 00:36:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010712026/11/12 00:36:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010722026/11/12 00:36:15 INFO Vacuumed table table=pending_closures10732026/11/12 00:36:15 INFO Vacuumed table table=pending_objects10742026/11/12 00:36:15 INFO Vacuumed table table=multipart_uploads10752026/11/12 00:36:15 INFO Vacuumed table table=closures10762026/11/12 00:36:15 INFO Completed push id=1 objects=6 duration=5.684187985s10772026/11/12 00:36:15 INFO Vacuumed table table=pending_closures10782026/11/12 00:36:15 INFO Vacuumed table table=pending_closures1079--- PASS: TestPush_CompleteCommitsEveryRoot (8.96s)1080=== CONT TestResolveDBConnectionString1081=== RUN TestResolveDBConnectionString/flag_wins1082=== PAUSE TestResolveDBConnectionString/flag_wins1083=== RUN TestResolveDBConnectionString/file_when_flag_empty1084=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1085=== RUN TestResolveDBConnectionString/missing_file_is_an_error1086=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1087=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1088=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1089=== RUN TestResolveDBConnectionString/nothing_configured1090=== PAUSE TestResolveDBConnectionString/nothing_configured1091=== CONT TestReadProxy40410922026/11/12 00:36:15 INFO Vacuumed table table=objects10932026/11/12 00:36:15 INFO Vacuumed table table=pending_objects10942026/11/12 00:36:15 INFO Vacuumed table table=pending_objects10952026/11/12 00:36:15 INFO Vacuumed table table=multipart_uploads10962026/11/12 00:36:15 INFO Vacuumed table table=multipart_uploads10972026/11/12 00:36:15 INFO Vacuumed table table=closures10982026/11/12 00:36:15 INFO Vacuumed table table=objects10992026/11/12 00:36:15 INFO Vacuumed table table=closures11002026/11/12 00:36:15 INFO Aborted multipart uploads count=0 kept=011012026/11/12 00:36:15 INFO Vacuumed table table=objects11022026/11/12 00:36:15 INFO Aborted multipart uploads count=0 kept=011032026/11/12 00:36:15 WARN Force mode enabled - objects will be deleted immediately without grace period11042026/11/12 00:36:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011052026/11/12 00:36:15 INFO Vacuumed table table=pending_closures11062026/11/12 00:36:15 INFO Vacuumed table table=pending_objects11072026/11/12 00:36:15 INFO Vacuumed table table=multipart_uploads11082026/11/12 00:36:15 INFO Vacuumed table table=closures11092026/11/12 00:36:15 INFO Vacuumed table table=objects1110--- PASS: TestPushDedupSurvivesConcurrentGC (5.07s)1111=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock11122026/11/12 00:36:15 INFO lead: acquired remote=192.0.2.1:1234 generation=111132026/11/12 00:36:15 INFO lead: acquired remote=192.0.2.1:1234 generation=111142026/11/12 00:36:15 INFO lead: acquired remote=192.0.2.1:1234 generation=111152026/11/12 00:36:15 INFO lead: released remote=192.0.2.1:1234 generation=11116--- PASS: TestLeadIncumbentAnnouncesAtOnce (3.29s)1117=== CONT TestReadProxyNarStreaming11182026/11/12 00:36:15 INFO lead: acquired remote=192.0.2.1:1234 generation=111192026/11/12 00:36:15 INFO Received push request method=POST path=/api/pushes1120--- PASS: TestMetricsInventory (3.49s)1121=== CONT TestDeleteClosuresRacingPins11222026/11/12 00:36:15 INFO Received complete push request method=POST path=/api/pushes/1/complete11232026/11/12 00:36:15 INFO Aborted multipart uploads count=0 kept=011242026/11/12 00:36:15 INFO lead: acquired remote=192.0.2.1:1234 generation=111252026/11/12 00:36:15 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1126--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (3.63s)1127=== CONT TestReadProxyContentNegotiation11282026/11/12 00:36:15 INFO Received uploads request method=POST path=/api/pending_closures11292026/11/12 00:36:15 INFO Received uploads request method=POST path=/api/pending_closures1130--- PASS: TestRegisterCompletedObjectNeedsPendingRow (6.48s)1131=== CONT TestClientFallsBackToClosures11322026/11/12 00:36:15 INFO Aborted multipart uploads count=0 kept=01133--- PASS: TestReadProxyConditionalGet (5.85s)1134=== CONT TestReadProxyNarinfoAlreadyDecompressed11352026/11/12 00:36:16 INFO Aborted multipart uploads count=0 kept=011362026/11/12 00:36:16 WARN mTLS auth: subject not in bound subjects subject="CN=reader"11372026/11/12 00:36:16 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1138--- PASS: TestService_NativeMTLS (3.78s)1139=== CONT TestReadProxyNarinfo1140--- PASS: TestReadProxyDisabled (6.24s)1141=== CONT TestEnsureAbortUploadsRule11422026/11/12 00:36:16 INFO Aborted multipart uploads count=0 kept=01143--- PASS: TestReadProxyOutlastsServerWriteTimeout (6.44s)1144=== CONT TestClientPushesUseOnePush11452026/11/12 00:36:16 INFO lead: released remote=192.0.2.1:1234 generation=11146--- PASS: TestLeadEndsOnShutdown (4.43s)1147=== CONT TestGenerateLandingPage1148--- PASS: TestGenerateLandingPage (0.01s)1149=== CONT TestPinProtectsFromGC11502026/11/12 00:36:16 INFO lead: released remote=192.0.2.1:1234 generation=111512026/11/12 00:36:16 INFO lead: acquired remote=192.0.2.1:1234 generation=211522026/11/12 00:36:16 INFO lead: released remote=192.0.2.1:1234 generation=111532026/11/12 00:36:17 INFO lead: acquired remote=192.0.2.1:1234 generation=211542026/11/12 00:36:17 INFO lead: released remote=192.0.2.1:1234 generation=111552026/11/12 00:36:17 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=211562026/11/12 00:36:17 INFO lead: acquired remote=192.0.2.1:1234 generation=311572026/11/12 00:36:17 INFO lead: acquired remote=192.0.2.1:1234 generation=21158--- PASS: TestGCBugBareHashReferences (5.14s)1159=== CONT TestService_readinessHandler11602026/11/12 00:36:17 WARN Force mode enabled - objects will be deleted immediately without grace period11612026/11/12 00:36:17 WARN Force mode enabled - objects will be deleted immediately without grace period11622026/11/12 00:36:17 WARN Force mode enabled - objects will be deleted immediately without grace period11632026/11/12 00:36:17 WARN Force mode enabled - objects will be deleted immediately without grace period11642026/11/12 00:36:17 WARN Force mode enabled - objects will be deleted immediately without grace period11652026/11/12 00:36:17 INFO Completed push id=1 objects=4 duration=1.545613417s11662026/11/12 00:36:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011672026/11/12 00:36:17 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete11682026/11/12 00:36:17 INFO Vacuumed table table=pending_closures11692026/11/12 00:36:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011702026/11/12 00:36:17 INFO Vacuumed table table=pending_objects11712026/11/12 00:36:17 INFO Vacuumed table table=multipart_uploads11722026/11/12 00:36:17 INFO Vacuumed table table=closures11732026/11/12 00:36:17 INFO Vacuumed table table=objects11742026/11/12 00:36:17 INFO Completed upload id=2 objects=2 duration=4.122559ms11752026/11/12 00:36:17 INFO Vacuumed table table=pending_closures1176--- PASS: TestMetricsCommit (5.15s)1177=== CONT TestClientSharedPathCommittedMidPush1178--- PASS: TestGCMetrics (5.22s)1179=== CONT TestService_healthCheckHandler11802026/11/12 00:36:17 INFO Vacuumed table table=pending_objects1181--- PASS: TestCommitRacingPendingCleanupKeepsObjects (10.95s)1182=== CONT TestClientWithDependencies11832026/11/12 00:36:17 INFO Vacuumed table table=multipart_uploads11842026/11/12 00:36:17 INFO Vacuumed table table=closures11852026/11/12 00:36:17 INFO Vacuumed table table=objects11862026/11/12 00:36:17 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."11872026/11/12 00:36:17 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1188--- PASS: TestFailedS3DeleteKeepsTombstone (7.88s)1189=== CONT TestGracefulShutdownDrainsInflight1190--- PASS: TestGCSweepSkipsPendingObjects (7.35s)1191=== CONT TestIsValidCachePath1192=== RUN TestIsValidCachePath/narinfo1193=== PAUSE TestIsValidCachePath/narinfo1194=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1195=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1196=== RUN TestIsValidCachePath/nar_zst1197=== PAUSE TestIsValidCachePath/nar_zst1198=== RUN TestIsValidCachePath/nar_xz1199=== PAUSE TestIsValidCachePath/nar_xz1200=== RUN TestIsValidCachePath/nar_bz21201=== PAUSE TestIsValidCachePath/nar_bz21202=== RUN TestIsValidCachePath/nar_uncompressed1203=== PAUSE TestIsValidCachePath/nar_uncompressed1204=== RUN TestIsValidCachePath/ls1205=== PAUSE TestIsValidCachePath/ls1206=== RUN TestIsValidCachePath/log1207=== PAUSE TestIsValidCachePath/log1208=== RUN TestIsValidCachePath/realisation1209=== PAUSE TestIsValidCachePath/realisation1210=== RUN TestIsValidCachePath/nix-cache-info1211=== PAUSE TestIsValidCachePath/nix-cache-info1212=== RUN TestIsValidCachePath/index.html1213=== PAUSE TestIsValidCachePath/index.html1214=== RUN TestIsValidCachePath/traversal_parent1215=== PAUSE TestIsValidCachePath/traversal_parent12162026/11/12 00:36:17 INFO Starting HTTP server address=127.0.0.1:377631217=== RUN TestIsValidCachePath/traversal_in_middle1218=== PAUSE TestIsValidCachePath/traversal_in_middle1219=== RUN TestIsValidCachePath/invalid_char_e1220=== PAUSE TestIsValidCachePath/invalid_char_e1221=== RUN TestIsValidCachePath/invalid_char_u1222=== PAUSE TestIsValidCachePath/invalid_char_u1223=== RUN TestIsValidCachePath/random_path1224=== PAUSE TestIsValidCachePath/random_path1225=== RUN TestIsValidCachePath/empty1226=== PAUSE TestIsValidCachePath/empty1227=== RUN TestIsValidCachePath/leading_slash1228=== PAUSE TestIsValidCachePath/leading_slash1229=== RUN TestIsValidCachePath/wrong_extension1230=== PAUSE TestIsValidCachePath/wrong_extension1231=== RUN TestIsValidCachePath/short_hash1232=== PAUSE TestIsValidCachePath/short_hash1233=== CONT TestGCTaskStore_Fail1234--- PASS: TestGCTaskStore_Fail (0.00s)1235=== CONT TestParseSingleRange1236=== RUN TestParseSingleRange/none1237=== PAUSE TestParseSingleRange/none1238=== RUN TestParseSingleRange/unknown_unit1239=== PAUSE TestParseSingleRange/unknown_unit1240=== RUN TestParseSingleRange/multi-range_ignored1241=== PAUSE TestParseSingleRange/multi-range_ignored1242=== RUN TestParseSingleRange/malformed_no_dash1243=== PAUSE TestParseSingleRange/malformed_no_dash1244=== RUN TestParseSingleRange/malformed_both_empty1245=== PAUSE TestParseSingleRange/malformed_both_empty1246=== RUN TestParseSingleRange/malformed_end_before_start1247=== PAUSE TestParseSingleRange/malformed_end_before_start1248=== RUN TestParseSingleRange/closed1249=== PAUSE TestParseSingleRange/closed1250=== RUN TestParseSingleRange/open-ended1251=== PAUSE TestParseSingleRange/open-ended1252=== RUN TestParseSingleRange/end_clamped_to_size1253=== PAUSE TestParseSingleRange/end_clamped_to_size12542026/11/12 00:36:17 INFO Shutdown signal received, draining in-flight requests timeout=10s1255=== RUN TestParseSingleRange/suffix1256=== PAUSE TestParseSingleRange/suffix1257=== RUN TestParseSingleRange/suffix_exceeds_size1258=== PAUSE TestParseSingleRange/suffix_exceeds_size1259=== RUN TestParseSingleRange/single_byte1260=== PAUSE TestParseSingleRange/single_byte1261=== RUN TestParseSingleRange/start_past_EOF1262=== PAUSE TestParseSingleRange/start_past_EOF1263=== RUN TestParseSingleRange/start_far_past_EOF1264=== PAUSE TestParseSingleRange/start_far_past_EOF1265=== CONT TestClientMultipleUploads1266=== NAME TestNARDeduplicationMetadataUploadBug1267 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3044770412/001/store/kwilxw0v53ardsm8k2ikzvjsxyw7nbzr-file1.txt12682026/11/12 00:36:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=012692026/11/12 00:36:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=012702026/11/12 00:36:17 INFO Vacuumed table table=pending_closures12712026/11/12 00:36:17 INFO Vacuumed table table=pending_objects1272--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1273=== CONT TestObjectStatsUpdatedOncePerCommit12742026/11/12 00:36:17 INFO Vacuumed table table=multipart_uploads12752026/11/12 00:36:17 INFO Vacuumed table table=closures12762026/11/12 00:36:17 INFO Vacuumed table table=pending_closures12772026/11/12 00:36:17 INFO Vacuumed table table=pending_objects12782026/11/12 00:36:17 INFO Vacuumed table table=multipart_uploads12792026/11/12 00:36:17 INFO Vacuumed table table=closures12802026/11/12 00:36:17 INFO Vacuumed table table=objects1281--- PASS: TestGCSweepDeliversEachKeyOnce (7.72s)1282=== CONT TestClientIntegration12832026/11/12 00:36:17 INFO Vacuumed table table=objects1284--- PASS: TestSweepSkipsRowsLockedByPush (8.01s)1285=== CONT TestObjectStatsTrigger1286--- PASS: TestReadProxyNarinfoAlreadyDecompressed (1.69s)1287=== CONT TestClientErrorHandling1288=== RUN TestClientErrorHandling/InvalidStorePath1289=== PAUSE TestClientErrorHandling/InvalidStorePath1290=== RUN TestClientErrorHandling/InvalidAuthToken1291=== PAUSE TestClientErrorHandling/InvalidAuthToken1292=== RUN TestClientErrorHandling/ServerNotAvailable1293=== PAUSE TestClientErrorHandling/ServerNotAvailable1294=== CONT TestGCTaskStore_PhaseUpdates1295--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1296=== CONT TestClientCADerivations12972026/11/12 00:36:17 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1298--- PASS: TestReadProxyNarinfo (1.66s)1299=== CONT TestCacheStatsHandler1300--- PASS: TestReadProxyContentNegotiation (1.82s)1301=== CONT TestCacheConfigHandler1302=== RUN TestCacheConfigHandler/full_config,_no_issuer1303=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1304=== RUN TestCacheConfigHandler/no_cache_url_configured1305=== PAUSE TestCacheConfigHandler/no_cache_url_configured1306=== RUN TestCacheConfigHandler/no_signing_keys1307=== PAUSE TestCacheConfigHandler/no_signing_keys1308=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1309=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1310=== CONT TestGCTaskStore_CompletedAllowsNewTask1311--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1312=== CONT TestGCTaskStore_GetReturnsLatest1313--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1314=== CONT TestService_ReadScope_PublicByDefault13152026/11/12 00:36:17 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket66 days=21316--- PASS: TestEnsureAbortUploadsRule (1.76s)1317=== CONT TestGCTaskStore_GetEmpty1318--- PASS: TestGCTaskStore_GetEmpty (0.00s)1319=== CONT TestService_RequireScope_OIDC1320--- PASS: TestReadProxy404 (2.19s)1321=== CONT TestGCTaskStore_ConflictDifferentParams1322--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1323=== CONT TestPendingCleanupUsesOneCutoff1324--- PASS: TestReadProxyNarStreaming (2.06s)1325=== CONT TestGCTaskStore_DeduplicateSameParams1326--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1327=== CONT TestOrphanedObjectsGC13282026/11/12 00:36:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41897/oidc13292026/11/12 00:36:18 INFO lead: released remote=192.0.2.1:1234 generation=21330--- PASS: TestService_healthCheckHandler (0.58s)1331=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded13322026/11/12 00:36:18 INFO lead: acquired remote=192.0.2.1:1234 generation=313332026/11/12 00:36:18 INFO lead: released remote=192.0.2.1:1234 generation=31334--- PASS: TestLeadGenerationGrowsWithEveryGrant (5.56s)1335=== CONT TestUploadHandlersRejectOversizedBody1336--- PASS: TestObjectStatsTrigger (0.54s)1337=== CONT TestPresentNotReportedWhileGCDeletesClosure1338=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1339=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1340=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1341=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1342=== CONT TestService_cleanupPendingClosuresHandler1343--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (2.46s)1344=== CONT TestPresentRefreshesOnlyStaleClosures1345=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1346=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1347=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1348=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1349=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1350=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1351=== CONT TestUploadHandlersRejectInvalidKeys1352=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1353=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1354=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1355=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1356=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1357=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1358=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1359=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1360=== CONT TestPresentReportsOnlyClosureRoots13612026/11/12 00:36:18 INFO lead: released remote=192.0.2.1:1234 generation=21362--- PASS: TestLeadElectsOneAndHandsOver (5.48s)1363=== CONT TestIsValidUploadKey1364=== RUN TestIsValidUploadKey/narinfo1365=== PAUSE TestIsValidUploadKey/narinfo1366=== RUN TestIsValidUploadKey/nar_zst1367=== PAUSE TestIsValidUploadKey/nar_zst1368=== RUN TestIsValidUploadKey/nar_xz1369=== PAUSE TestIsValidUploadKey/nar_xz1370=== RUN TestIsValidUploadKey/nar_plain1371=== PAUSE TestIsValidUploadKey/nar_plain1372=== RUN TestIsValidUploadKey/listing1373=== PAUSE TestIsValidUploadKey/listing1374=== RUN TestIsValidUploadKey/build_log1375=== PAUSE TestIsValidUploadKey/build_log1376=== RUN TestIsValidUploadKey/build_log_home-manager_file1377=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1378=== RUN TestIsValidUploadKey/build_log_plus_in_name1379=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1380=== RUN TestIsValidUploadKey/build_log_question_mark1381=== PAUSE TestIsValidUploadKey/build_log_question_mark1382=== RUN TestIsValidUploadKey/build_log_equals1383=== PAUSE TestIsValidUploadKey/build_log_equals1384=== RUN TestIsValidUploadKey/realisation1385=== PAUSE TestIsValidUploadKey/realisation1386=== RUN TestIsValidUploadKey/realisation_plus_in_output1387=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1388=== RUN TestIsValidUploadKey/nix-cache-info1389=== PAUSE TestIsValidUploadKey/nix-cache-info1390=== RUN TestIsValidUploadKey/index.html1391=== PAUSE TestIsValidUploadKey/index.html1392=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1393=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1394=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1395=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1396=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1397=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1398=== RUN TestIsValidUploadKey/traversal1399=== PAUSE TestIsValidUploadKey/traversal1400=== RUN TestIsValidUploadKey/traversal_nar1401=== PAUSE TestIsValidUploadKey/traversal_nar1402=== RUN TestIsValidUploadKey/absolute1403=== PAUSE TestIsValidUploadKey/absolute1404=== RUN TestIsValidUploadKey/empty_key1405=== PAUSE TestIsValidUploadKey/empty_key1406=== RUN TestIsValidUploadKey/unknown_type1407=== PAUSE TestIsValidUploadKey/unknown_type1408=== CONT TestCreatePin_ReservedPins1409=== NAME TestPinProtectsFromGC1410 client_integration_test.go:791: Pinned store path: /build/TestPinProtectsFromGC1714890210/001/store/p49zvszyvgyr4960d3d5cf2pvch70bqz-pinned-file.txt1411 client_integration_test.go:792: Unpinned store path: /build/TestPinProtectsFromGC1714890210/001/store/6aqzz7b71nsk648ii2111af8cdc2xpls-unpinned-file.txt1412=== NAME TestClientCADerivations1413 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations2741273989/001/store/a90wj283c3vbb217w0zq068z5ax077wm-ca-test1414--- PASS: TestService_ReadScope_PublicByDefault (0.55s)1415=== CONT TestMultipartCleanup14162026/11/12 00:36:18 INFO Received uploads request method=POST path=/api/pending_closures14172026/11/12 00:36:18 INFO Received uploads request method=POST path=/api/pending_closures14182026/11/12 00:36:18 INFO Received cleanup request method=DELETE path=/api/pending_closures14192026/11/12 00:36:18 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34195/oidc1420--- PASS: TestCacheStatsHandler (0.77s)1421=== CONT TestOrphanedObjectsGCStressTest1422--- PASS: TestObjectStatsUpdatedOncePerCommit (0.96s)1423=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1424=== RUN TestService_RequireScope_OIDC/builder_may_write1425=== PAUSE TestService_RequireScope_OIDC/builder_may_write1426=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1427=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1428=== RUN TestService_RequireScope_OIDC/ops_may_admin1429=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1430=== RUN TestService_RequireScope_OIDC/ops_may_not_write1431=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1432=== RUN TestService_RequireScope_OIDC/reader_may_not_write1433=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1434=== RUN TestService_RequireScope_OIDC/static_token_may_admin1435=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1436=== RUN TestService_RequireScope_OIDC/static_token_may_write1437=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1438=== RUN TestService_RequireScope_OIDC/reader_may_read1439=== PAUSE TestService_RequireScope_OIDC/reader_may_read1440=== RUN TestService_RequireScope_OIDC/writer_implies_read1441=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1442=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1443=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1444=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1445=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1446=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1447=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1448=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1449=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1450=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1451=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1452=== CONT TestService_ReadAuthMiddleware14532026/11/12 00:36:18 WARN readiness check failed error="closed pool"1454--- PASS: TestService_readinessHandler (1.12s)1455=== CONT TestService_AuthMiddleware_MTLSProxyHeader1456=== NAME TestClientCADerivations1457 client_ca_test.go:139: Found 1 dependencies (including self)1458=== NAME TestClientIntegration1459 client_integration_test.go:292: Created store path: /build/TestClientIntegration1183620928/002/store/mfsm8kjkphxkcx2xfw3ggs2s16z845w7-test-file.txt14602026/11/12 00:36:18 INFO Received cleanup request method=DELETE path=/api/pending_closures14612026/11/12 00:36:18 INFO Aborted multipart uploads count=0 kept=014622026/11/12 00:36:18 INFO Received uploads request method=POST path=/api/pending_closures14632026/11/12 00:36:18 INFO Received cleanup request method=DELETE path=/api/pending_closures14642026/11/12 00:36:18 INFO Aborted multipart uploads count=1 kept=014652026/11/12 00:36:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete14662026-11-12 00:36:18.602 UTC [878] ERROR: Closure does not exist: id=114672026-11-12 00:36:18.602 UTC [878] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE14682026-11-12 00:36:18.602 UTC [878] STATEMENT: -- name: CommitPendingClosure :one1469 SELECT commit_pending_closure($1::bigint)::bigint1470 1471--- PASS: TestService_cleanupPendingClosuresHandler (0.53s)1472=== CONT TestService_AuthMiddleware14732026/11/12 00:36:18 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux14742026/11/12 00:36:18 WARN Refused reserved pin name=worker-x86_64-linux14752026/11/12 00:36:18 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux14762026/11/12 00:36:18 INFO Received create pin request method=POST path=/api/pins/my-app14772026/11/12 00:36:18 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1478--- PASS: TestCreatePin_ReservedPins (0.40s)1479=== CONT TestService_AuthMiddleware_OIDC14802026/11/12 00:36:18 INFO Received uploads request method=POST path=/api/pending_closures14812026/11/12 00:36:18 INFO Received cleanup request method=DELETE path=/api/pending_closures14822026/11/12 00:36:18 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket84/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""14832026/11/12 00:36:18 INFO Aborted multipart uploads count=0 kept=114842026/11/12 00:36:18 INFO Received cleanup request method=DELETE path=/api/pending_closures14852026/11/12 00:36:18 INFO Received push request method=POST path=/api/pushes14862026/11/12 00:36:18 INFO Aborted multipart uploads count=1 kept=014872026/11/12 00:36:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1488--- PASS: TestMultipartCleanup (0.49s)1489=== CONT TestGCEndsOnShutdown/before_the_run14902026/11/12 00:36:18 INFO Uploading kwilxw0v53ardsm8k2ikzvjsxyw7nbzr-file1.txt (160B)1491=== NAME TestClientWithDependencies1492 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies2245205416/001/store/grwg9bigall563qi2vjz699j0g0xjk5q-test-script14932026/11/12 00:36:18 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38493/oidc14942026/11/12 00:36:18 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14952026/11/12 00:36:18 WARN Failed to register uploaded object key=kwilxw0v53ardsm8k2ikzvjsxyw7nbzr.ls error="server returned 404: 404 page not found\n"14962026/11/12 00:36:18 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"14972026/11/12 00:36:18 INFO Signed narinfos id=1 count=114982026/11/12 00:36:18 INFO Uploading 1 narinfos14992026/11/12 00:36:18 INFO Received complete push request method=POST path=/api/pushes/1/complete15002026/11/12 00:36:18 WARN Failed to register uploaded object key=kwilxw0v53ardsm8k2ikzvjsxyw7nbzr.narinfo error="server returned 404: 404 page not found\n"15012026/11/12 00:36:18 INFO Completed push id=1 objects=3 duration=10.818161ms15022026/11/12 00:36:18 INFO Upload complete. (566ms)1503=== NAME TestNARDeduplicationMetadataUploadBug1504 metadata_upload_test.go:54: Retrieved narinfo from S3:1505 StorePath: /build/TestNARDeduplicationMetadataUploadBug3044770412/001/store/kwilxw0v53ardsm8k2ikzvjsxyw7nbzr-file1.txt1506 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1507 Compression: zstd1508 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1509 NarSize: 1601510 References: 1511 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1512 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1513 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1514 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1515--- PASS: TestService_ReadAuthMiddleware (0.35s)1516=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1517--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.34s)1518=== CONT TestPendingClosureWriteTimeout/empty1519=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1520=== NAME TestNARDeduplicationMetadataUploadBug1521 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug3044770412/001/store/wdk3yd81pyij1mp6rq5y4zp69lnmjlqy-file2.txt15222026/11/12 00:36:18 INFO Received uploads request method=POST path=/api/pending_closures1523=== NAME TestClientMultipleUploads1524 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads4111505433/001/store/i9izanly525ywm30pgp6n7qrxvaq7l1n-test-file-0.txt1525--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.93s)1526=== CONT TestPendingClosureWriteTimeout/670k_objects1527=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline15282026/11/12 00:36:18 INFO lead: released remote=192.0.2.1:1234 generation=31529--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (6.39s)1530=== CONT TestProxyWriteTimeout/narinfo1531=== CONT TestPendingClosureWriteTimeout/400_objects1532=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed15332026/11/12 00:36:19 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1534--- PASS: TestService_AuthMiddleware (0.45s)1535=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1536--- PASS: TestPresentReportsOnlyClosureRoots (0.90s)1537=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1538=== NAME TestClientWithDependencies1539 client_integration_test.go:663: Found 1 dependencies (including self)15402026/11/12 00:36:19 INFO Received uploads request method=POST path=/api/pending_closures15412026/11/12 00:36:19 INFO Received uploads request method=POST path=/api/pending_closures1542=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1543=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1544=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1545=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1546=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1547=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1548=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1549=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1550=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost15512026/11/12 00:36:19 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"15522026/11/12 00:36:19 WARN mTLS auth: bound subjects configured but subject DN unavailable15532026/11/12 00:36:19 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1554--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.94s)1555=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1556--- PASS: TestPresentRefreshesOnlyStaleClosures (1.31s)1557=== CONT TestProxyWriteTimeout/10_GiB_nar1558=== CONT TestProxyWriteTimeout/unknown_size1559=== CONT TestProxyWriteTimeout/1_GiB_nar1560--- PASS: TestProxyWriteTimeout (0.21s)1561 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1562 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1563 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1564 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1565=== CONT TestPush_RejectsBadRequests/no_objects15662026/11/12 00:36:19 INFO Received push request method=POST path=/api/pushes1567=== CONT TestPush_RejectsBadRequests/root_not_in_objects15682026/11/12 00:36:19 INFO Received push request method=POST path=/api/pushes1569=== CONT TestPush_RejectsBadRequests/bad_root15702026/11/12 00:36:19 INFO Received push request method=POST path=/api/pushes1571=== CONT TestPush_RejectsBadRequests/no_roots15722026/11/12 00:36:19 INFO Received push request method=POST path=/api/pushes1573=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1574--- PASS: TestPush_RejectsBadRequests (2.89s)1575 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1576 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1577 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1578 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)15792026/11/12 00:36:19 INFO Received uploads request method=POST path=/api/pending_closures15802026/11/12 00:36:19 INFO Received uploads request method=POST path=/api/pending_closures15812026/11/12 00:36:19 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15822026/11/12 00:36:19 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLmUzNTNhZjIzLTg2NDctNDM0Ny1hMTIyLTQ4NDU0YmNmNjU3Y3gxNzk0NDQzNzc5MzAzMjQ1MzEy parts=1015832026/11/12 00:36:19 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15842026/11/12 00:36:19 INFO Completed upload id=1 objects=3 duration=3.039988ms15852026/11/12 00:36:19 INFO Received uploads request method=POST path=/api/pending_closures15862026/11/12 00:36:19 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst15872026/11/12 00:36:19 WARN Found objects in DB but missing from S3, will re-upload count=11588=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check15892026/11/12 00:36:19 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15902026/11/12 00:36:19 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLmFhZGI5MjRjLTcxODktNDcxZC1hMjkzLTU2ZmVlZDQwMGU0ZXgxNzk0NDQzNzc5NDkxNjM5Mjgy parts=1015912026/11/12 00:36:19 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15922026/11/12 00:36:19 INFO Completed upload id=1 objects=3 duration=15.591166ms15932026/11/12 00:36:19 INFO Received uploads request method=POST path=/api/pending_closures15942026/11/12 00:36:19 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15952026/11/12 00:36:20 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLjY0ZDQyZjYzLWM3NjctNDZlZS05NzBjLTUxZmM1YTVmMmVmY3gxNzk0NDQzNzc5NTI4MTY1ODA1 parts=1015962026/11/12 00:36:20 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15972026/11/12 00:36:20 INFO Completed upload id=1 objects=3 duration=6.083618ms15982026/11/12 00:36:20 INFO Received uploads request method=POST path=/api/pending_closures15992026/11/12 00:36:20 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst16002026/11/12 00:36:20 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0001.narinfo1601=== CONT TestServerTLSConfig/no_client_CA1602=== CONT TestServerTLSConfig/missing_CA_file1603=== CONT TestServerTLSConfig/not_a_PEM_file1604--- PASS: TestServerTLSConfig (0.00s)1605 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1606 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1607 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.01s)1608=== CONT TestResolveDBConnectionString/flag_wins1609=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1610=== CONT TestResolveDBConnectionString/nothing_configured1611=== CONT TestResolveDBConnectionString/missing_file_is_an_error1612=== CONT TestResolveDBConnectionString/file_when_flag_empty1613=== CONT TestIsValidCachePath/narinfo1614=== CONT TestIsValidCachePath/index.html1615=== CONT TestIsValidCachePath/short_hash1616=== CONT TestIsValidCachePath/random_path1617--- PASS: TestResolveDBConnectionString (0.00s)1618 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1619 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1620 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1621 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1622 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1623=== CONT TestIsValidCachePath/invalid_char_u1624=== CONT TestIsValidCachePath/wrong_extension1625=== CONT TestIsValidCachePath/invalid_char_e1626=== CONT TestIsValidCachePath/leading_slash1627=== CONT TestIsValidCachePath/traversal_parent1628=== CONT TestIsValidCachePath/empty1629=== CONT TestIsValidCachePath/nar_uncompressed1630=== CONT TestIsValidCachePath/nix-cache-info1631=== CONT TestIsValidCachePath/nar_xz1632=== CONT TestIsValidCachePath/realisation1633=== CONT TestIsValidCachePath/nar_bz21634=== CONT TestIsValidCachePath/log1635=== CONT TestIsValidCachePath/traversal_in_middle1636=== CONT TestIsValidCachePath/ls1637=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1638=== CONT TestIsValidCachePath/nar_zst1639--- PASS: TestIsValidCachePath (0.00s)1640 --- PASS: TestIsValidCachePath/narinfo (0.00s)1641 --- PASS: TestIsValidCachePath/index.html (0.00s)1642 --- PASS: TestIsValidCachePath/short_hash (0.00s)1643 --- PASS: TestIsValidCachePath/random_path (0.00s)1644 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1645 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1646 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1647 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1648 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1649 --- PASS: TestIsValidCachePath/empty (0.00s)1650 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1651 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1652 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1653 --- PASS: TestIsValidCachePath/realisation (0.00s)1654 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1655 --- PASS: TestIsValidCachePath/log (0.00s)1656 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1657 --- PASS: TestIsValidCachePath/ls (0.00s)1658 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1659 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1660=== CONT TestParseSingleRange/none1661=== CONT TestParseSingleRange/start_far_past_EOF1662=== CONT TestParseSingleRange/start_past_EOF1663=== CONT TestParseSingleRange/single_byte1664=== CONT TestParseSingleRange/suffix_exceeds_size1665=== CONT TestParseSingleRange/suffix1666=== CONT TestParseSingleRange/end_clamped_to_size1667=== CONT TestParseSingleRange/open-ended1668=== CONT TestParseSingleRange/closed1669=== CONT TestParseSingleRange/malformed_end_before_start1670=== CONT TestParseSingleRange/malformed_both_empty1671=== CONT TestParseSingleRange/malformed_no_dash1672=== CONT TestParseSingleRange/multi-range_ignored1673=== CONT TestParseSingleRange/unknown_unit1674--- PASS: TestParseSingleRange (0.00s)1675 --- PASS: TestParseSingleRange/none (0.00s)1676 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1677 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1678 --- PASS: TestParseSingleRange/single_byte (0.00s)1679 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1680 --- PASS: TestParseSingleRange/suffix (0.00s)1681 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1682 --- PASS: TestParseSingleRange/open-ended (0.00s)1683 --- PASS: TestParseSingleRange/closed (0.00s)1684 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1685 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1686 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1687 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1688 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1689=== CONT TestClientErrorHandling/InvalidStorePath1690--- PASS: TestPendingClosureWriteTimeout (0.16s)1691 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)1692 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)1693 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)1694 --- PASS: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (1.23s)1695=== CONT TestClientErrorHandling/ServerNotAvailable1696--- PASS: TestConnectWaitsForAPeerMigration (5.53s)1697=== CONT TestClientErrorHandling/InvalidAuthToken16982026/11/12 00:36:20 INFO Received push request method=POST path=/api/pushes16992026/11/12 00:36:20 INFO Received push request method=POST path=/api/pushes17002026/11/12 00:36:20 INFO Received uploads request method=POST path=/api/pending_closures17012026/11/12 00:36:20 INFO Received push request method=POST path=/api/pushes1702=== NAME TestClientMultipleUploads1703 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads4111505433/001/store/wxvggljj2jr3za93lrjpwp1hf1kqg8pj-test-file-1.txt17042026/11/12 00:36:20 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17052026/11/12 00:36:20 INFO Uploading mfsm8kjkphxkcx2xfw3ggs2s16z845w7-test-file.txt (152B)17062026/11/12 00:36:20 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)17072026/11/12 00:36:20 INFO Uploading rwhq24xkbq2kbzghwq1m0bg62jw4x93r-b (216B)17082026/11/12 00:36:20 INFO Received uploads request method=POST path=/api/pending_closures17092026/11/12 00:36:20 INFO Uploading z9ym6yyiiyhi3zkpw3qpscqc8b4r3q4q-shared-dep (136B)17102026/11/12 00:36:20 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17112026/11/12 00:36:20 INFO Uploading p49zvszyvgyr4960d3d5cf2pvch70bqz-pinned-file.txt (128B)17122026/11/12 00:36:20 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)17132026/11/12 00:36:20 INFO Uploading 9plmnm9d8f3lr4xdsx4kzkayvqhhln2a-shared-dep (136B)17142026/11/12 00:36:20 INFO Uploading hgxpmjsvivksgwpbnxsfy76wvzmrhksr-b (216B)17152026/11/12 00:36:20 WARN Failed to register uploaded object key=mfsm8kjkphxkcx2xfw3ggs2s16z845w7.ls error="server returned 404: 404 page not found\n"17162026/11/12 00:36:20 WARN Failed to register uploaded object key=rwhq24xkbq2kbzghwq1m0bg62jw4x93r.ls error="server returned 404: 404 page not found\n"17172026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"17182026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17192026/11/12 00:36:20 WARN Failed to register uploaded object key=k6mls4f1saca1c3n6sv57060migp56v1.ls error="server returned 404: 404 page not found\n"17202026/11/12 00:36:20 INFO Signed narinfos id=1 count=117212026/11/12 00:36:20 INFO Uploading 1 narinfos17222026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/1cr1ka2b7z3b9dwvkrbgc7lbs9vrh7mrdmjk45y7zwjyrspdvgvj.nar.zst error="server returned 404: 404 page not found\n"17232026/11/12 00:36:20 WARN Failed to register uploaded object key=p49zvszyvgyr4960d3d5cf2pvch70bqz.ls error="server returned 404: 404 page not found\n"17242026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17252026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"17262026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17272026/11/12 00:36:20 INFO Signed narinfos id=1 count=317282026/11/12 00:36:20 INFO Uploading 3 narinfos17292026/11/12 00:36:20 INFO Signed narinfos id=1 count=117302026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"17312026/11/12 00:36:20 WARN Failed to register uploaded object key=z9ym6yyiiyhi3zkpw3qpscqc8b4r3q4q.ls error="server returned 404: 404 page not found\n"17322026/11/12 00:36:20 INFO Uploading 1 narinfos17332026/11/12 00:36:20 WARN Failed to register uploaded object key=hgxpmjsvivksgwpbnxsfy76wvzmrhksr.ls error="server returned 404: 404 page not found\n"17342026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/16h8z3ryhqi2cqrsnikl4pszwbs8257bn4ngjq24s4c4grqs2zhi.nar.zst error="server returned 404: 404 page not found\n"17352026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"17362026/11/12 00:36:20 WARN Failed to register uploaded object key=gy3n9684sw2n841fdnykwfbniacp79qg.ls error="server returned 404: 404 page not found\n"17372026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign17382026/11/12 00:36:20 WARN Failed to register uploaded object key=9plmnm9d8f3lr4xdsx4kzkayvqhhln2a.ls error="server returned 404: 404 page not found\n"17392026/11/12 00:36:20 INFO Received complete push request method=POST path=/api/pushes/1/complete17402026/11/12 00:36:20 WARN Failed to register uploaded object key=k6mls4f1saca1c3n6sv57060migp56v1.narinfo error="server returned 404: 404 page not found\n"17412026/11/12 00:36:20 WARN Failed to register uploaded object key=z9ym6yyiiyhi3zkpw3qpscqc8b4r3q4q.narinfo error="server returned 404: 404 page not found\n"17422026/11/12 00:36:20 INFO Received complete push request method=POST path=/api/pushes/1/complete17432026/11/12 00:36:20 INFO Received complete push request method=POST path=/api/pushes/1/complete17442026/11/12 00:36:20 WARN Failed to register uploaded object key=rwhq24xkbq2kbzghwq1m0bg62jw4x93r.narinfo error="server returned 404: 404 page not found\n"17452026/11/12 00:36:20 WARN Failed to register uploaded object key=p49zvszyvgyr4960d3d5cf2pvch70bqz.narinfo error="server returned 404: 404 page not found\n"17462026/11/12 00:36:20 INFO Signed narinfos id=1 count=217472026/11/12 00:36:20 WARN Failed to register uploaded object key=mfsm8kjkphxkcx2xfw3ggs2s16z845w7.narinfo error="server returned 404: 404 page not found\n"17482026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign17492026/11/12 00:36:20 INFO Signed narinfos id=2 count=217502026/11/12 00:36:20 INFO Uploading 4 narinfos17512026/11/12 00:36:20 WARN Failed to register uploaded object key=9plmnm9d8f3lr4xdsx4kzkayvqhhln2a.narinfo error="server returned 404: 404 page not found\n"17522026/11/12 00:36:20 WARN Failed to register uploaded object key=gy3n9684sw2n841fdnykwfbniacp79qg.narinfo error="server returned 404: 404 page not found\n"17532026/11/12 00:36:20 WARN Failed to register uploaded object key=hgxpmjsvivksgwpbnxsfy76wvzmrhksr.narinfo error="server returned 404: 404 page not found\n"17542026/11/12 00:36:20 INFO Completed push id=1 objects=3 duration=8.991944ms17552026/11/12 00:36:20 INFO Upload complete. (1.221s)17562026/11/12 00:36:20 INFO Completed push id=1 objects=3 duration=10.694441ms17572026/11/12 00:36:20 INFO Upload complete. (1.537s)17582026/11/12 00:36:20 INFO Completed push id=1 objects=8 duration=12.964082ms17592026/11/12 00:36:20 INFO Upload complete. (1.54s)17602026/11/12 00:36:20 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17612026/11/12 00:36:20 WARN Failed to register uploaded object key=9plmnm9d8f3lr4xdsx4kzkayvqhhln2a.narinfo error="server returned 404: 404 page not found\n"1762=== NAME TestClientPushesUseOnePush1763 client_pushes_test.go:97: Retrieved narinfo from S3:1764 StorePath: /build/TestClientPushesUseOnePush1158050628/001/store/z9ym6yyiiyhi3zkpw3qpscqc8b4r3q4q-shared-dep1765 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1766 Compression: zstd1767 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821768 NarSize: 1361769 References: 1770 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1771 client_pushes_test.go:97: Retrieved narinfo from S3:1772 StorePath: /build/TestClientPushesUseOnePush1158050628/001/store/k6mls4f1saca1c3n6sv57060migp56v1-a1773 URL: nar/1cr1ka2b7z3b9dwvkrbgc7lbs9vrh7mrdmjk45y7zwjyrspdvgvj.nar.zst1774 Compression: zstd1775 NarHash: sha256:1cr1ka2b7z3b9dwvkrbgc7lbs9vrh7mrdmjk45y7zwjyrspdvgvj1776 NarSize: 2161777 References: /build/TestClientPushesUseOnePush1158050628/001/store/z9ym6yyiiyhi3zkpw3qpscqc8b4r3q4q-shared-dep1778 CA: text:sha256:1i18pbyalnmri43dla4ns5sbdn01ym4kfz464ak2m5yzc51izz0l17792026/11/12 00:36:20 INFO Completed upload id=1 objects=6 duration=8.934423ms17802026/11/12 00:36:20 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete1781 client_pushes_test.go:97: Retrieved narinfo from S3:1782 StorePath: /build/TestClientPushesUseOnePush1158050628/001/store/rwhq24xkbq2kbzghwq1m0bg62jw4x93r-b1783 URL: nar/1cr1ka2b7z3b9dwvkrbgc7lbs9vrh7mrdmjk45y7zwjyrspdvgvj.nar.zst1784 Compression: zstd1785 NarHash: sha256:1cr1ka2b7z3b9dwvkrbgc7lbs9vrh7mrdmjk45y7zwjyrspdvgvj1786 NarSize: 2161787 References: /build/TestClientPushesUseOnePush1158050628/001/store/z9ym6yyiiyhi3zkpw3qpscqc8b4r3q4q-shared-dep1788 CA: text:sha256:1i18pbyalnmri43dla4ns5sbdn01ym4kfz464ak2m5yzc51izz0l17892026/11/12 00:36:20 INFO Completed upload id=2 objects=6 duration=1.946718ms17902026/11/12 00:36:20 INFO Upload complete. (1.237s)17912026/11/12 00:36:20 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1792=== NAME TestClientFallsBackToClosures1793 client_pushes_test.go:112: Retrieved narinfo from S3:1794 StorePath: /build/TestClientFallsBackToClosures2636553488/001/store/9plmnm9d8f3lr4xdsx4kzkayvqhhln2a-shared-dep1795 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1796 Compression: zstd1797 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821798 NarSize: 1361799 References: 1800 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1801--- PASS: TestClientPushesUseOnePush (3.83s)1802=== CONT TestCacheConfigHandler/full_config,_no_issuer1803=== NAME TestClientFallsBackToClosures1804 client_pushes_test.go:112: Retrieved narinfo from S3:1805 StorePath: /build/TestClientFallsBackToClosures2636553488/001/store/gy3n9684sw2n841fdnykwfbniacp79qg-a1806 URL: nar/16h8z3ryhqi2cqrsnikl4pszwbs8257bn4ngjq24s4c4grqs2zhi.nar.zst1807 Compression: zstd1808 NarHash: sha256:16h8z3ryhqi2cqrsnikl4pszwbs8257bn4ngjq24s4c4grqs2zhi1809 NarSize: 2161810 References: /build/TestClientFallsBackToClosures2636553488/001/store/9plmnm9d8f3lr4xdsx4kzkayvqhhln2a-shared-dep1811 CA: text:sha256:1b0gzhbkpy2912rxhqx37vqy77dlfvxgmsbb80mhyf47qmanpz1m1812=== CONT TestCacheConfigHandler/no_signing_keys1813=== CONT TestCacheConfigHandler/no_cache_url_configured1814=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1815--- PASS: TestCacheConfigHandler (0.00s)1816 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1817 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1818 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1819 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1820=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1821=== NAME TestClientFallsBackToClosures1822 client_pushes_test.go:112: Retrieved narinfo from S3:1823 StorePath: /build/TestClientFallsBackToClosures2636553488/001/store/hgxpmjsvivksgwpbnxsfy76wvzmrhksr-b1824 URL: nar/16h8z3ryhqi2cqrsnikl4pszwbs8257bn4ngjq24s4c4grqs2zhi.nar.zst1825 Compression: zstd1826 NarHash: sha256:16h8z3ryhqi2cqrsnikl4pszwbs8257bn4ngjq24s4c4grqs2zhi1827 NarSize: 2161828 References: /build/TestClientFallsBackToClosures2636553488/001/store/9plmnm9d8f3lr4xdsx4kzkayvqhhln2a-shared-dep1829 CA: text:sha256:1b0gzhbkpy2912rxhqx37vqy77dlfvxgmsbb80mhyf47qmanpz1m18302026/11/12 00:36:20 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100400000000000000000000.nar.zst upload_id=NjBjYjFkMTAtYjE0Zi00NWRmLWJiNjYtNDZkMGJlZTc1ZjIxLjJmNGNkMTlmLTA4OGUtNGU4ZS1hNDBhLTc3YzE3Nzk5OTFhN3gxNzk0NDQzNzc5OTYxMzM5OTE0 parts=1018312026/11/12 00:36:20 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18322026/11/12 00:36:20 INFO Completed upload id=2 objects=3 duration=1.631216ms1833--- PASS: TestClientFallsBackToClosures (4.49s)1834=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh18352026/11/12 00:36:20 INFO Received push request method=POST path=/api/pushes18362026/11/12 00:36:20 INFO Received push request method=POST path=/api/pushes1837=== NAME TestClientMultipleUploads1838 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads4111505433/001/store/giiqrgnn9c17p4nzqz2bzpyd433qaxdc-test-file-2.txt18392026/11/12 00:36:20 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present18402026/11/12 00:36:20 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18412026/11/12 00:36:20 INFO Received uploads request method=POST path=/api/pending_closures18422026/11/12 00:36:20 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18432026/11/12 00:36:20 INFO Uploading a90wj283c3vbb217w0zq068z5ax077wm-ca-test (144B)18442026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign18452026/11/12 00:36:20 WARN Failed to register uploaded object key=wdk3yd81pyij1mp6rq5y4zp69lnmjlqy.ls error="server returned 404: 404 page not found\n"18462026/11/12 00:36:20 INFO Signed narinfos id=2 count=118472026/11/12 00:36:20 INFO Uploading 1 narinfos1848=== NAME TestOrphanedObjectsGC1849 orphaned_objects_gc_test.go:287: GC Test Summary:1850 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1851 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1852 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1853 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1854 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1855--- PASS: TestOrphanedObjectsGC (2.65s)1856=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure18572026/11/12 00:36:20 INFO Received uploads request method=POST path=/18582026/11/12 00:36:20 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst18592026/11/12 00:36:20 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0004.narinfo1860=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts18612026/11/12 00:36:20 INFO Received request for more parts method=POST path=/18622026/11/12 00:36:20 INFO Received push request method=POST path=/api/pushes18632026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"18642026/11/12 00:36:20 INFO Received push request method=POST path=/api/pushes18652026/11/12 00:36:20 WARN Failed to register uploaded object key=a90wj283c3vbb217w0zq068z5ax077wm.ls error="server returned 404: 404 page not found\n"18662026/11/12 00:36:20 INFO Received complete push request method=POST path=/api/pushes/2/complete18672026/11/12 00:36:20 WARN Failed to register uploaded object key=log/fic8aciz5w5fy4sn2ny72y2i3bpmykka-ca-test.drv error="server returned 404: 404 page not found\n"18682026/11/12 00:36:20 WARN Failed to register uploaded object key=wdk3yd81pyij1mp6rq5y4zp69lnmjlqy.narinfo error="server returned 404: 404 page not found\n"18692026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign18702026/11/12 00:36:20 INFO Signed narinfos id=1 count=118712026/11/12 00:36:20 INFO Uploading 1 narinfos18722026/11/12 00:36:20 INFO Received uploads request method=POST path=/api/pending_closures18732026/11/12 00:36:20 INFO Received complete push request method=POST path=/api/pushes/1/complete18742026/11/12 00:36:20 WARN Failed to register uploaded object key=a90wj283c3vbb217w0zq068z5ax077wm.narinfo error="server returned 404: 404 page not found\n"18752026/11/12 00:36:20 INFO Completed push id=2 objects=3 duration=12.142095ms18762026/11/12 00:36:20 INFO Upload complete. (129ms)18772026/11/12 00:36:20 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18782026/11/12 00:36:20 INFO Uploading grwg9bigall563qi2vjz699j0g0xjk5q-test-script (136B)18792026/11/12 00:36:20 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)18802026/11/12 00:36:20 INFO Uploading nnnjf30a8zyvilcb24a9g74bk1sdai2l-top (224B)18812026/11/12 00:36:20 INFO Uploading nzsgpv3cn40168g032x0xzsw7wj5a0z6-shared-dep (136B)1882=== NAME TestNARDeduplicationMetadataUploadBug1883 metadata_upload_test.go:76: Retrieved narinfo from S3:1884 StorePath: /build/TestNARDeduplicationMetadataUploadBug3044770412/001/store/wdk3yd81pyij1mp6rq5y4zp69lnmjlqy-file2.txt1885 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1886 Compression: zstd1887 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1888 NarSize: 1601889 References: 1890 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1891 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1892 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1893 {"version":1,"root":{"type":"regular","size":44}}18942026/11/12 00:36:20 INFO Completed push id=1 objects=4 duration=14.301334ms18952026/11/12 00:36:20 INFO Upload complete. (1.313s)1896--- PASS: TestNARDeduplicationMetadataUploadBug (8.08s)1897=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart18982026/11/12 00:36:20 INFO Received complete multipart upload request method=POST path=/18992026/11/12 00:36:20 WARN Failed to register uploaded object key=grwg9bigall563qi2vjz699j0g0xjk5q.ls error="server returned 404: 404 page not found\n"19002026/11/12 00:36:20 WARN Failed to register uploaded object key=nnnjf30a8zyvilcb24a9g74bk1sdai2l.ls error="server returned 404: 404 page not found\n"19012026/11/12 00:36:20 WARN Failed to register uploaded object key=nzsgpv3cn40168g032x0xzsw7wj5a0z6.ls error="server returned 404: 404 page not found\n"19022026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"19032026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19042026/11/12 00:36:20 WARN Failed to register uploaded object key=nar/0890d2qbmzj7ss0pgcbz22hnl661dv35s04g8j4if956s1470lsj.nar.zst error="server returned 404: 404 page not found\n"19052026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19062026/11/12 00:36:20 INFO Signed narinfos id=1 count=219072026/11/12 00:36:20 INFO Uploading 2 narinfos1908=== NAME TestClientCADerivations1909 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations2741273989/001/store/a90wj283c3vbb217w0zq068z5ax077wm-ca-test1910 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1911 Compression: zstd1912 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1913 NarSize: 1441914 References: 1915 Deriver: /build/TestClientCADerivations2741273989/001/store/fic8aciz5w5fy4sn2ny72y2i3bpmykka-ca-test.drv1916 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1917 client_ca_test.go:185: Checking for realisation files in S3...19182026/11/12 00:36:20 WARN Failed to register uploaded object key=log/710pzq66zy9p6p0gm69wdsfq948pb7qc-test-script.drv error="server returned 404: 404 page not found\n"19192026/11/12 00:36:20 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19202026/11/12 00:36:20 INFO Received uploads request method=POST path=/api/pending_closures19212026/11/12 00:36:20 INFO Signed narinfos id=1 count=119222026/11/12 00:36:20 INFO Uploading 1 narinfos19232026/11/12 00:36:20 WARN Failed to register uploaded object key=nnnjf30a8zyvilcb24a9g74bk1sdai2l.narinfo error="server returned 404: 404 page not found\n"19242026/11/12 00:36:20 INFO Received complete push request method=POST path=/api/pushes/1/complete19252026/11/12 00:36:20 WARN Failed to register uploaded object key=nzsgpv3cn40168g032x0xzsw7wj5a0z6.narinfo error="server returned 404: 404 page not found\n"19262026/11/12 00:36:20 INFO Received complete push request method=POST path=/api/pushes/1/complete19272026/11/12 00:36:20 WARN Failed to register uploaded object key=grwg9bigall563qi2vjz699j0g0xjk5q.narinfo error="server returned 404: 404 page not found\n"19282026/11/12 00:36:20 INFO Aborted multipart uploads count=0 kept=019292026/11/12 00:36:20 INFO Completed push id=1 objects=6 duration=10.507419ms19302026/11/12 00:36:20 INFO Completed push id=1 objects=4 duration=9.109125ms19312026/11/12 00:36:20 INFO Upload complete. (159ms)19322026/11/12 00:36:20 INFO Upload complete. (159ms)1933=== NAME TestClientSharedPathCommittedMidPush1934 client_integration_test.go:740: Retrieved narinfo from S3:1935 StorePath: /build/TestClientSharedPathCommittedMidPush282116200/001/store/nzsgpv3cn40168g032x0xzsw7wj5a0z6-shared-dep1936 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1937 Compression: zstd1938 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821939 NarSize: 1361940 References: 1941 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1942=== NAME TestClientCADerivations1943 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1944 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1945=== NAME TestClientSharedPathCommittedMidPush1946 client_integration_test.go:740: Retrieved narinfo from S3:1947 StorePath: /build/TestClientSharedPathCommittedMidPush282116200/001/store/nnnjf30a8zyvilcb24a9g74bk1sdai2l-top1948 URL: nar/0890d2qbmzj7ss0pgcbz22hnl661dv35s04g8j4if956s1470lsj.nar.zst1949 Compression: zstd1950 NarHash: sha256:0890d2qbmzj7ss0pgcbz22hnl661dv35s04g8j4if956s1470lsj1951 NarSize: 2241952 References: /build/TestClientSharedPathCommittedMidPush282116200/001/store/nzsgpv3cn40168g032x0xzsw7wj5a0z6-shared-dep1953 CA: text:sha256:19jskjdcys0a0sd4pabnfhlr0dkp627xxz4xd45scpfrfwkf9c4p19542026/11/12 00:36:20 INFO Received uploads request method=POST path=/api/pending_closures1955--- PASS: TestClientSharedPathCommittedMidPush (3.13s)1956=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info19572026/11/12 00:36:20 INFO Received uploads request method=POST path=/1958=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key19592026/11/12 00:36:20 INFO Aborted multipart uploads count=0 kept=019602026/11/12 00:36:20 INFO Received complete multipart upload request method=POST path=/1961=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal19622026/11/12 00:36:20 INFO Received uploads request method=POST path=/1963=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key19642026/11/12 00:36:20 INFO Received request for more parts method=POST path=/1965--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1966 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1967 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1968 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1969 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1970=== CONT TestIsValidUploadKey/narinfo1971=== CONT TestIsValidUploadKey/unknown_type1972=== CONT TestIsValidUploadKey/empty_key1973=== CONT TestIsValidUploadKey/absolute1974=== CONT TestIsValidUploadKey/traversal_nar1975=== CONT TestIsValidUploadKey/traversal1976=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1977=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1978=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1979=== CONT TestIsValidUploadKey/index.html1980=== CONT TestIsValidUploadKey/nix-cache-info1981=== CONT TestIsValidUploadKey/realisation_plus_in_output1982=== CONT TestIsValidUploadKey/realisation1983=== CONT TestIsValidUploadKey/build_log_equals1984=== CONT TestIsValidUploadKey/build_log_question_mark1985=== CONT TestIsValidUploadKey/build_log_plus_in_name1986=== CONT TestIsValidUploadKey/build_log_home-manager_file1987=== CONT TestIsValidUploadKey/build_log1988=== CONT TestIsValidUploadKey/listing1989=== CONT TestIsValidUploadKey/nar_plain1990=== CONT TestIsValidUploadKey/nar_xz1991=== CONT TestIsValidUploadKey/nar_zst1992=== CONT TestService_RequireScope_OIDC/builder_may_write1993--- PASS: TestIsValidUploadKey (0.00s)1994 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1995 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1996 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1997 --- PASS: TestIsValidUploadKey/absolute (0.00s)1998 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1999 --- PASS: TestIsValidUploadKey/traversal (0.00s)2000 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)
loading 660 hidden lines…
26612026/11/12 00:36:35 INFO Vacuumed table table=pending_objects26622026/11/12 00:36:35 INFO Vacuumed table table=closures26632026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads26642026/11/12 00:36:35 INFO Vacuumed table table=pending_objects26652026/11/12 00:36:35 INFO Vacuumed table table=closures26662026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads26672026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads26682026/11/12 00:36:35 INFO Vacuumed table table=closures26692026/11/12 00:36:35 INFO Vacuumed table table=closures26702026/11/12 00:36:35 INFO Vacuumed table table=objects26712026/11/12 00:36:35 INFO Vacuumed table table=objects26722026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=026732026/11/12 00:36:35 INFO Vacuumed table table=objects26742026/11/12 00:36:35 INFO Vacuumed table table=objects26752026/11/12 00:36:35 INFO Vacuumed table table=pending_closures26762026/11/12 00:36:35 INFO Vacuumed table table=pending_objects26772026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads26782026/11/12 00:36:35 INFO Received uploads request method=POST path=/26792026/11/12 00:36:35 INFO Vacuumed table table=closures26802026/11/12 00:36:35 INFO Vacuumed table table=objects26812026/11/12 00:36:35 INFO Received complete push request method=POST path=/26822026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=026832026/11/12 00:36:35 INFO Received push request method=POST path=/26842026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period26852026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026862026/11/12 00:36:35 INFO Completed push id=6 objects=5 duration=4.097238ms26872026/11/12 00:36:35 INFO Vacuumed table table=pending_closures26882026/11/12 00:36:35 INFO Received uploads request method=POST path=/26892026/11/12 00:36:35 INFO Vacuumed table table=pending_objects26902026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads26912026/11/12 00:36:35 INFO Vacuumed table table=closures26922026/11/12 00:36:35 INFO Vacuumed table table=objects26932026/11/12 00:36:35 INFO Received push request method=POST path=/26942026/11/12 00:36:35 INFO Received push request method=POST path=/26952026/11/12 00:36:35 INFO Received push request method=POST path=/26962026/11/12 00:36:35 INFO Received uploads request method=POST path=/26972026/11/12 00:36:35 INFO Received uploads request method=POST path=/26982026/11/12 00:36:35 INFO Received complete upload request method=POST path=/26992026/11/12 00:36:35 INFO Completed upload id=4 objects=14 duration=3.149369ms27002026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=027012026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period27022026/11/12 00:36:35 INFO Received complete upload request method=POST path=/27032026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027042026/11/12 00:36:35 INFO Vacuumed table table=pending_closures27052026/11/12 00:36:35 INFO Vacuumed table table=pending_objects27062026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads27072026/11/12 00:36:35 INFO Vacuumed table table=closures27082026/11/12 00:36:35 INFO Vacuumed table table=objects27092026/11/12 00:36:35 INFO Completed upload id=2 objects=26 duration=5.632393ms27102026/11/12 00:36:35 INFO Received push request method=POST path=/27112026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=027122026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period27132026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=027142026/11/12 00:36:35 INFO Received complete push request method=POST path=/27152026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period27162026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027172026/11/12 00:36:35 INFO Vacuumed table table=pending_closures27182026/11/12 00:36:35 INFO Completed push id=6 objects=8 duration=3.657214ms27192026/11/12 00:36:35 INFO Vacuumed table table=pending_objects27202026/11/12 00:36:35 INFO Received push request method=POST path=/27212026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads27222026/11/12 00:36:35 INFO Received uploads request method=POST path=/27232026/11/12 00:36:35 INFO Vacuumed table table=closures27242026/11/12 00:36:35 INFO Vacuumed table table=objects27252026/11/12 00:36:35 INFO Received uploads request method=POST path=/27262026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=027272026/11/12 00:36:35 INFO Received uploads request method=POST path=/27282026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period27292026/11/12 00:36:35 INFO Received push request method=POST path=/27302026/11/12 00:36:35 INFO Received uploads request method=POST path=/27312026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027322026/11/12 00:36:35 INFO Vacuumed table table=pending_closures27332026/11/12 00:36:35 INFO Vacuumed table table=pending_objects27342026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads27352026/11/12 00:36:35 INFO Vacuumed table table=closures27362026/11/12 00:36:35 INFO Vacuumed table table=objects27372026/11/12 00:36:35 INFO Received complete upload request method=POST path=/27382026/11/12 00:36:35 INFO Received push request method=POST path=/27392026/11/12 00:36:35 INFO Received push request method=POST path=/27402026/11/12 00:36:35 INFO Received uploads request method=POST path=/27412026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=027422026/11/12 00:36:35 INFO Completed upload id=10 objects=4 duration=3.711495ms27432026/11/12 00:36:35 INFO Received push request method=POST path=/27442026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027452026/11/12 00:36:35 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27462026/11/12 00:36:35 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27472026/11/12 00:36:35 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27482026/11/12 00:36:35 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27492026/11/12 00:36:35 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27502026/11/12 00:36:35 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27512026/11/12 00:36:35 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27522026/11/12 00:36:35 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27532026/11/12 00:36:35 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27542026/11/12 00:36:35 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27552026/11/12 00:36:35 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27562026/11/12 00:36:35 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27572026/11/12 00:36:35 INFO Vacuumed table table=pending_closures27582026/11/12 00:36:35 INFO Vacuumed table table=pending_objects27592026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads27602026/11/12 00:36:35 INFO Received uploads request method=POST path=/27612026/11/12 00:36:35 INFO Vacuumed table table=closures27622026/11/12 00:36:35 INFO Vacuumed table table=objects27632026/11/12 00:36:35 INFO Received uploads request method=POST path=/27642026/11/12 00:36:35 INFO Received push request method=POST path=/27652026/11/12 00:36:35 INFO Received push request method=POST path=/27662026/11/12 00:36:35 INFO Received uploads request method=POST path=/27672026/11/12 00:36:35 INFO Received uploads request method=POST path=/27682026/11/12 00:36:35 INFO Received uploads request method=POST path=/27692026/11/12 00:36:35 INFO Received push request method=POST path=/27702026/11/12 00:36:35 INFO Received push request method=POST path=/27712026/11/12 00:36:35 INFO Received complete push request method=POST path=/27722026/11/12 00:36:35 INFO Received uploads request method=POST path=/27732026/11/12 00:36:35 INFO Received push request method=POST path=/27742026/11/12 00:36:35 INFO Received complete upload request method=POST path=/27752026/11/12 00:36:35 INFO Completed push id=5 objects=8 duration=3.658015ms27762026/11/12 00:36:35 INFO Received complete upload request method=POST path=/27772026/11/12 00:36:35 INFO Completed upload id=7 objects=4 duration=1.756137ms27782026/11/12 00:36:35 INFO Received complete upload request method=POST path=/27792026/11/12 00:36:35 INFO Completed upload id=2 objects=7 duration=5.563913ms27802026/11/12 00:36:35 INFO Completed upload id=5 objects=7 duration=2.662526ms27812026/11/12 00:36:35 INFO Received push request method=POST path=/27822026/11/12 00:36:35 INFO Received push request method=POST path=/27832026/11/12 00:36:35 INFO Received complete push request method=POST path=/27842026/11/12 00:36:35 INFO Received uploads request method=POST path=/27852026/11/12 00:36:35 INFO Received uploads request method=POST path=/27862026/11/12 00:36:35 INFO Received push request method=POST path=/27872026/11/12 00:36:35 INFO Received push request method=POST path=/27882026/11/12 00:36:35 INFO Completed push id=3 objects=5 duration=5.393951ms27892026/11/12 00:36:35 INFO Received push request method=POST path=/27902026/11/12 00:36:35 INFO Received push request method=POST path=/27912026/11/12 00:36:35 INFO Received uploads request method=POST path=/27922026/11/12 00:36:35 INFO Received complete push request method=POST path=/27932026/11/12 00:36:35 INFO Received complete upload request method=POST path=/27942026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=027952026/11/12 00:36:35 INFO Completed upload id=6 objects=9 duration=1.10231ms27962026/11/12 00:36:35 INFO Received uploads request method=POST path=/27972026/11/12 00:36:35 INFO Completed push id=1 objects=2 duration=3.16091ms27982026/11/12 00:36:35 INFO Received push request method=POST path=/27992026/11/12 00:36:35 INFO Received push request method=POST path=/28002026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028012026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028022026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period28032026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28042026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28052026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28062026/11/12 00:36:35 INFO Vacuumed table table=closures28072026/11/12 00:36:35 INFO Vacuumed table table=objects28082026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028092026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028102026/11/12 00:36:35 INFO Received complete upload request method=POST path=/28112026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028122026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028132026/11/12 00:36:35 INFO Completed upload id=11 objects=14 duration=1.864357ms28142026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28152026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28162026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028172026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28182026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28192026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28202026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028212026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28222026/11/12 00:36:35 INFO Vacuumed table table=closures28232026/11/12 00:36:35 INFO Vacuumed table table=objects28242026/11/12 00:36:35 INFO Vacuumed table table=closures28252026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028262026/11/12 00:36:35 INFO Vacuumed table table=objects28272026/11/12 00:36:35 INFO Received complete upload request method=POST path=/28282026/11/12 00:36:35 INFO Received uploads request method=POST path=/28292026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=028302026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28312026/11/12 00:36:35 INFO Received uploads request method=POST path=/28322026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28332026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28342026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28352026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28362026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28372026/11/12 00:36:35 INFO Completed upload id=10 objects=6 duration=2.744725ms28382026/11/12 00:36:35 INFO Received complete upload request method=POST path=/28392026/11/12 00:36:35 INFO Vacuumed table table=closures28402026/11/12 00:36:35 INFO Vacuumed table table=closures28412026/11/12 00:36:35 INFO Vacuumed table table=objects28422026/11/12 00:36:35 INFO Vacuumed table table=objects28432026/11/12 00:36:35 INFO Completed upload id=4 objects=7 duration=2.805527ms28442026/11/12 00:36:35 INFO Received push request method=POST path=/28452026/11/12 00:36:35 INFO Received push request method=POST path=/28462026/11/12 00:36:35 INFO Received push request method=POST path=/28472026/11/12 00:36:35 INFO Received complete upload request method=POST path=/28482026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028492026/11/12 00:36:35 INFO Completed upload id=3 objects=5 duration=1.211771ms28502026/11/12 00:36:35 INFO Received push request method=POST path=/28512026/11/12 00:36:35 INFO Received push request method=POST path=/28522026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028532026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28542026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28552026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28562026/11/12 00:36:35 INFO Vacuumed table table=closures28572026/11/12 00:36:35 INFO Vacuumed table table=objects28582026/11/12 00:36:35 INFO Received uploads request method=POST path=/28592026/11/12 00:36:35 INFO Received complete upload request method=POST path=/28602026/11/12 00:36:35 INFO Received uploads request method=POST path=/28612026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028622026/11/12 00:36:35 INFO Completed upload id=10 objects=2 duration=2.718606ms28632026/11/12 00:36:35 INFO Received uploads request method=POST path=/28642026/11/12 00:36:35 INFO Received push request method=POST path=/28652026/11/12 00:36:35 INFO Received uploads request method=POST path=/28662026/11/12 00:36:35 INFO Received push request method=POST path=/28672026/11/12 00:36:35 INFO Received push request method=POST path=/28682026/11/12 00:36:35 INFO Received push request method=POST path=/28692026/11/12 00:36:35 INFO Received push request method=POST path=/28702026/11/12 00:36:35 INFO Received uploads request method=POST path=/28712026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028722026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28732026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28742026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028752026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28762026/11/12 00:36:35 INFO Received push request method=POST path=/28772026/11/12 00:36:35 INFO Vacuumed table table=closures28782026/11/12 00:36:35 INFO Received complete push request method=POST path=/28792026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period28802026/11/12 00:36:35 INFO Vacuumed table table=objects28812026/11/12 00:36:35 INFO Completed push id=8 objects=6 duration=1.991879ms28822026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028832026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028842026/11/12 00:36:35 INFO Vacuumed table table=pending_closures28852026/11/12 00:36:35 INFO Received uploads request method=POST path=/28862026/11/12 00:36:35 INFO Vacuumed table table=pending_objects28872026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads28882026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=028892026/11/12 00:36:35 INFO Vacuumed table table=closures28902026/11/12 00:36:35 INFO Received push request method=POST path=/28912026/11/12 00:36:35 INFO Vacuumed table table=objects28922026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period28932026/11/12 00:36:35 INFO Received uploads request method=POST path=/28942026/11/12 00:36:35 INFO Received push request method=POST path=/28952026/11/12 00:36:35 INFO Received complete push request method=POST path=/28962026/11/12 00:36:35 INFO Completed push id=12 objects=7 duration=2.130061ms28972026/11/12 00:36:35 INFO Received complete upload request method=POST path=/28982026/11/12 00:36:35 INFO Completed upload id=8 objects=3 duration=2.859387ms28992026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=029002026/11/12 00:36:35 INFO Received complete upload request method=POST path=/29012026/11/12 00:36:35 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29022026/11/12 00:36:35 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29032026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period29042026/11/12 00:36:35 INFO Received complete push request method=POST path=/29052026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=029062026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029072026/11/12 00:36:35 INFO Received uploads request method=POST path=/29082026/11/12 00:36:35 INFO Completed upload id=23 objects=18 duration=2.658185ms29092026/11/12 00:36:35 INFO Vacuumed table table=pending_closures29102026/11/12 00:36:35 INFO Completed push id=2 objects=10 duration=3.489112ms29112026/11/12 00:36:35 INFO Received uploads request method=POST path=/29122026/11/12 00:36:35 INFO Vacuumed table table=pending_objects29132026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads29142026/11/12 00:36:35 INFO Vacuumed table table=closures29152026/11/12 00:36:35 INFO Vacuumed table table=objects29162026/11/12 00:36:35 INFO Received uploads request method=POST path=/29172026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=029182026/11/12 00:36:35 INFO Received complete push request method=POST path=/29192026/11/12 00:36:35 INFO Vacuumed table table=pending_closures29202026/11/12 00:36:35 INFO Received uploads request method=POST path=/29212026/11/12 00:36:35 INFO Vacuumed table table=pending_objects29222026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads29232026/11/12 00:36:35 INFO Completed push id=14 objects=3 duration=2.509504ms29242026/11/12 00:36:35 INFO Vacuumed table table=closures29252026/11/12 00:36:35 INFO Vacuumed table table=objects29262026/11/12 00:36:35 INFO Received complete upload request method=POST path=/29272026/11/12 00:36:35 INFO Received push request method=POST path=/29282026/11/12 00:36:35 INFO Completed upload id=3 objects=8 duration=2.390283ms29292026/11/12 00:36:35 INFO Received push request method=POST path=/29302026/11/12 00:36:35 INFO Received push request method=POST path=/29312026/11/12 00:36:35 INFO Received push request method=POST path=/29322026/11/12 00:36:35 INFO Received uploads request method=POST path=/29332026/11/12 00:36:35 INFO Received push request method=POST path=/29342026/11/12 00:36:35 INFO Received complete push request method=POST path=/29352026/11/12 00:36:35 INFO Received complete push request method=POST path=/29362026/11/12 00:36:35 INFO Completed push id=8 objects=7 duration=1.742796ms29372026/11/12 00:36:35 INFO Received complete push request method=POST path=/29382026/11/12 00:36:35 INFO Completed push id=6 objects=6 duration=1.879438ms29392026/11/12 00:36:35 INFO Received complete upload request method=POST path=/29402026/11/12 00:36:35 INFO Completed push id=5 objects=6 duration=2.856227ms29412026/11/12 00:36:35 INFO Received uploads request method=POST path=/29422026/11/12 00:36:35 INFO Completed upload id=22 objects=22 duration=2.293982ms29432026/11/12 00:36:35 INFO Received uploads request method=POST path=/29442026/11/12 00:36:35 INFO Received push request method=POST path=/29452026/11/12 00:36:35 INFO Received push request method=POST path=/29462026/11/12 00:36:35 INFO Received uploads request method=POST path=/29472026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=029482026/11/12 00:36:35 INFO Received uploads request method=POST path=/29492026/11/12 00:36:35 INFO Received uploads request method=POST path=/29502026/11/12 00:36:35 INFO Received push request method=POST path=/29512026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period29522026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029532026/11/12 00:36:35 INFO Vacuumed table table=pending_closures29542026/11/12 00:36:35 INFO Received uploads request method=POST path=/29552026/11/12 00:36:35 INFO Vacuumed table table=pending_objects29562026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads29572026/11/12 00:36:35 INFO Vacuumed table table=closures29582026/11/12 00:36:35 INFO Vacuumed table table=objects29592026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=029602026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period29612026/11/12 00:36:35 INFO Received complete push request method=POST path=/29622026/11/12 00:36:35 INFO Completed push id=26 objects=5 duration=1.865498ms29632026/11/12 00:36:35 INFO Received complete upload request method=POST path=/29642026/11/12 00:36:35 INFO Completed upload id=9 objects=16 duration=1.423893ms29652026/11/12 00:36:35 INFO Received uploads request method=POST path=/29662026/11/12 00:36:35 INFO Received uploads request method=POST path=/29672026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=029682026/11/12 00:36:35 INFO Vacuumed table table=pending_closures29692026/11/12 00:36:35 INFO Vacuumed table table=pending_objects29702026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads29712026/11/12 00:36:35 INFO Vacuumed table table=closures29722026/11/12 00:36:35 INFO Vacuumed table table=objects29732026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=029742026/11/12 00:36:35 INFO Received complete upload request method=POST path=/29752026/11/12 00:36:35 INFO Received complete push request method=POST path=/29762026/11/12 00:36:35 INFO Completed upload id=24 objects=14 duration=957.709µs29772026/11/12 00:36:35 INFO Completed push id=10 objects=15 duration=1.852117ms29782026/11/12 00:36:35 INFO Received push request method=POST path=/29792026/11/12 00:36:35 INFO Received complete upload request method=POST path=/29802026/11/12 00:36:35 INFO Received uploads request method=POST path=/29812026/11/12 00:36:35 INFO Received push request method=POST path=/29822026/11/12 00:36:35 INFO Completed upload id=10 objects=6 duration=2.780466ms29832026/11/12 00:36:35 INFO Received complete push request method=POST path=/29842026/11/12 00:36:35 INFO Completed push id=19 objects=3 duration=1.06499ms29852026/11/12 00:36:35 INFO Received push request method=POST path=/29862026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=029872026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period29882026/11/12 00:36:35 INFO Received uploads request method=POST path=/29892026/11/12 00:36:35 INFO Received push request method=POST path=/29902026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029912026/11/12 00:36:35 INFO Vacuumed table table=pending_closures29922026/11/12 00:36:35 INFO Vacuumed table table=pending_objects29932026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads29942026/11/12 00:36:35 INFO Vacuumed table table=closures29952026/11/12 00:36:35 INFO Vacuumed table table=objects29962026/11/12 00:36:35 INFO Received complete push request method=POST path=/29972026/11/12 00:36:35 INFO Completed push id=10 objects=3 duration=1.758076ms29982026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=029992026/11/12 00:36:35 INFO Received push request method=POST path=/30002026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030012026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30022026/11/12 00:36:35 INFO Vacuumed table table=pending_objects30032026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads30042026/11/12 00:36:35 INFO Vacuumed table table=closures30052026/11/12 00:36:35 INFO Vacuumed table table=objects30062026/11/12 00:36:35 INFO Received uploads request method=POST path=/30072026/11/12 00:36:35 INFO Received push request method=POST path=/30082026/11/12 00:36:35 INFO Received uploads request method=POST path=/30092026/11/12 00:36:35 INFO Received uploads request method=POST path=/30102026/11/12 00:36:35 INFO Received uploads request method=POST path=/30112026/11/12 00:36:35 INFO Received uploads request method=POST path=/30122026/11/12 00:36:35 INFO Received push request method=POST path=/30132026/11/12 00:36:35 INFO Received push request method=POST path=/30142026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030152026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30162026/11/12 00:36:35 INFO Received uploads request method=POST path=/30172026/11/12 00:36:35 INFO Received push request method=POST path=/30182026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030192026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30202026/11/12 00:36:35 INFO Received uploads request method=POST path=/30212026/11/12 00:36:35 INFO Vacuumed table table=pending_objects30222026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads30232026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030242026/11/12 00:36:35 INFO Vacuumed table table=closures30252026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030262026/11/12 00:36:35 INFO Vacuumed table table=objects30272026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30282026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30292026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030302026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30312026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030322026/11/12 00:36:35 INFO Vacuumed table table=pending_objects30332026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads30342026/11/12 00:36:35 INFO Received uploads request method=POST path=/30352026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30362026/11/12 00:36:35 INFO Vacuumed table table=closures30372026/11/12 00:36:35 INFO Received complete push request method=POST path=/30382026/11/12 00:36:35 INFO Vacuumed table table=objects30392026/11/12 00:36:35 INFO Vacuumed table table=pending_objects30402026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads30412026/11/12 00:36:35 INFO Received complete upload request method=POST path=/30422026/11/12 00:36:35 INFO Vacuumed table table=closures30432026/11/12 00:36:35 INFO Completed upload id=25 objects=4 duration=1.05969ms30442026/11/12 00:36:35 INFO Vacuumed table table=objects30452026/11/12 00:36:35 INFO Completed push id=6 objects=26 duration=3.682874ms30462026/11/12 00:36:35 INFO Received push request method=POST path=/30472026/11/12 00:36:35 INFO Received push request method=POST path=/30482026/11/12 00:36:35 INFO Received uploads request method=POST path=/30492026/11/12 00:36:35 INFO Received push request method=POST path=/30502026/11/12 00:36:35 INFO Received push request method=POST path=/30512026/11/12 00:36:35 INFO Received complete push request method=POST path=/30522026/11/12 00:36:35 INFO Received complete upload request method=POST path=/30532026/11/12 00:36:35 INFO Completed push id=19 objects=8 duration=3.998118ms30542026/11/12 00:36:35 INFO Completed upload id=11 objects=6 duration=2.015319ms30552026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030562026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30572026/11/12 00:36:35 INFO Received uploads request method=POST path=/30582026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030592026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030602026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30612026/11/12 00:36:35 INFO Received uploads request method=POST path=/30622026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30632026/11/12 00:36:35 INFO Vacuumed table table=pending_objects30642026/11/12 00:36:35 INFO Received uploads request method=POST path=/30652026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads30662026/11/12 00:36:35 INFO Vacuumed table table=closures30672026/11/12 00:36:35 INFO Vacuumed table table=objects30682026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030692026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30702026/11/12 00:36:35 INFO Vacuumed table table=pending_objects30712026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads30722026/11/12 00:36:35 INFO Vacuumed table table=closures30732026/11/12 00:36:35 INFO Vacuumed table table=objects30742026/11/12 00:36:35 INFO Received complete push request method=POST path=/30752026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030762026/11/12 00:36:35 INFO Completed push id=20 objects=8 duration=1.12571ms30772026/11/12 00:36:35 INFO Received uploads request method=POST path=/30782026/11/12 00:36:35 INFO Received uploads request method=POST path=/30792026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30802026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030812026/11/12 00:36:35 INFO Received uploads request method=POST path=/30822026/11/12 00:36:35 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30832026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030842026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30852026/11/12 00:36:35 INFO Vacuumed table table=pending_objects30862026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads30872026/11/12 00:36:35 INFO Vacuumed table table=closures30882026/11/12 00:36:35 INFO Vacuumed table table=objects30892026/11/12 00:36:35 INFO Received push request method=POST path=/30902026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030912026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=030922026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30932026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period30942026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030952026/11/12 00:36:35 INFO Received uploads request method=POST path=/30962026/11/12 00:36:35 INFO Vacuumed table table=pending_closures30972026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030982026/11/12 00:36:35 INFO Received push request method=POST path=/30992026/11/12 00:36:35 INFO Vacuumed table table=pending_objects31002026/11/12 00:36:35 INFO Vacuumed table table=pending_closures31012026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads31022026/11/12 00:36:35 INFO Vacuumed table table=closures31032026/11/12 00:36:35 INFO Vacuumed table table=pending_objects31042026/11/12 00:36:35 INFO Vacuumed table table=objects31052026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads31062026/11/12 00:36:35 INFO Vacuumed table table=closures31072026/11/12 00:36:35 INFO Vacuumed table table=objects31082026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=031092026/11/12 00:36:35 INFO Received complete push request method=POST path=/31102026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period31112026/11/12 00:36:35 INFO Completed push id=9 objects=5 duration=2.04848ms31122026/11/12 00:36:35 INFO Received complete upload request method=POST path=/31132026/11/12 00:36:35 INFO Completed upload id=14 objects=8 duration=1.949478ms31142026/11/12 00:36:35 INFO Received push request method=POST path=/31152026/11/12 00:36:35 INFO Received push request method=POST path=/31162026/11/12 00:36:35 INFO Received push request method=POST path=/31172026/11/12 00:36:35 INFO Aborted multipart uploads count=0 kept=031182026/11/12 00:36:35 WARN Force mode enabled - objects will be deleted immediately without grace period31192026/11/12 00:36:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031202026/11/12 00:36:35 INFO Vacuumed table table=pending_closures31212026/11/12 00:36:35 INFO Vacuumed table table=pending_objects31222026/11/12 00:36:35 INFO Vacuumed table table=multipart_uploads31232026/11/12 00:36:35 INFO Vacuumed table table=closures31242026/11/12 00:36:35 INFO Vacuumed table table=objects31252026/11/12 00:36:35 INFO Received uploads request method=POST path=/31262026/11/12 00:36:35 INFO Received uploads request method=POST path=/31272026/11/12 00:36:35 INFO Received uploads request method=POST path=/31282026/11/12 00:36:35 INFO Received uploads request method=POST path=/31292026/11/12 00:36:35 INFO Received uploads request method=POST path=/31302026/11/12 00:36:35 INFO Received complete push request method=POST path=/31312026/11/12 00:36:35 INFO Completed push id=27 objects=11 duration=1.178652ms31322026/11/12 00:36:36 INFO Received complete upload request method=POST path=/31332026/11/12 00:36:36 INFO Received push request method=POST path=/31342026/11/12 00:36:36 INFO Completed upload id=17 objects=6 duration=3.329251ms31352026/11/12 00:36:36 INFO Received push request method=POST path=/31362026/11/12 00:36:36 INFO Received complete push request method=POST path=/31372026/11/12 00:36:36 INFO Received complete upload request method=POST path=/31382026/11/12 00:36:36 INFO Completed upload id=29 objects=26 duration=1.03909ms31392026/11/12 00:36:36 INFO Completed push id=3 objects=21 duration=5.26969ms31402026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=031412026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period31422026/11/12 00:36:36 INFO Received uploads request method=POST path=/31432026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031442026/11/12 00:36:36 INFO Vacuumed table table=pending_closures31452026/11/12 00:36:36 INFO Vacuumed table table=pending_objects31462026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads31472026/11/12 00:36:36 INFO Vacuumed table table=closures31482026/11/12 00:36:36 INFO Vacuumed table table=objects31492026/11/12 00:36:36 INFO Received complete push request method=POST path=/31502026/11/12 00:36:36 INFO Received uploads request method=POST path=/31512026/11/12 00:36:36 INFO Completed push id=30 objects=8 duration=1.261512ms31522026/11/12 00:36:36 INFO Received push request method=POST path=/31532026/11/12 00:36:36 INFO Received complete upload request method=POST path=/31542026/11/12 00:36:36 INFO Completed upload id=31 objects=5 duration=918.109µs31552026/11/12 00:36:36 INFO Received push request method=POST path=/31562026/11/12 00:36:36 INFO Received complete push request method=POST path=/31572026/11/12 00:36:36 INFO Received uploads request method=POST path=/31582026/11/12 00:36:36 INFO Completed push id=32 objects=3 duration=1.078371ms31592026/11/12 00:36:36 INFO Received uploads request method=POST path=/31602026/11/12 00:36:36 INFO Received push request method=POST path=/31612026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=031622026/11/12 00:36:36 INFO Received uploads request method=POST path=/31632026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period31642026/11/12 00:36:36 INFO Received complete upload request method=POST path=/31652026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031662026/11/12 00:36:36 INFO Received complete upload request method=POST path=/31672026/11/12 00:36:36 INFO Vacuumed table table=pending_closures31682026/11/12 00:36:36 INFO Received uploads request method=POST path=/31692026/11/12 00:36:36 INFO Vacuumed table table=pending_objects31702026/11/12 00:36:36 INFO Completed upload id=16 objects=17 duration=6.255359ms31712026/11/12 00:36:36 INFO Completed upload id=14 objects=16 duration=4.1908ms31722026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads31732026/11/12 00:36:36 INFO Vacuumed table table=closures31742026/11/12 00:36:36 INFO Vacuumed table table=objects31752026/11/12 00:36:36 INFO Received push request method=POST path=/31762026/11/12 00:36:36 INFO Received complete push request method=POST path=/31772026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=031782026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period31792026/11/12 00:36:36 INFO Completed push id=15 objects=5 duration=17.02476ms31802026/11/12 00:36:36 INFO Received uploads request method=POST path=/31812026/11/12 00:36:36 INFO Received uploads request method=POST path=/31822026/11/12 00:36:36 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31832026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31842026/11/12 00:36:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31852026/11/12 00:36:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31862026/11/12 00:36:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31872026/11/12 00:36:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31882026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31892026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31902026/11/12 00:36:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31912026/11/12 00:36:36 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31922026/11/12 00:36:36 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31932026/11/12 00:36:36 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31942026/11/12 00:36:36 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31952026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."31962026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31972026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31982026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31992026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."32002026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."32012026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32022026/11/12 00:36:36 INFO Received complete upload request method=POST path=/32032026/11/12 00:36:36 INFO Completed upload id=15 objects=9 duration=1.929598ms32042026/11/12 00:36:36 INFO Received push request method=POST path=/32052026/11/12 00:36:36 INFO Received push request method=POST path=/32062026/11/12 00:36:36 INFO Received complete push request method=POST path=/32072026/11/12 00:36:36 INFO Completed push id=17 objects=11 duration=7.710373ms32082026/11/12 00:36:36 INFO Received uploads request method=POST path=/32092026/11/12 00:36:36 INFO Received complete push request method=POST path=/32102026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=032112026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period32122026/11/12 00:36:36 INFO Completed push id=21 objects=17 duration=2.1297ms32132026/11/12 00:36:36 INFO Received push request method=POST path=/32142026/11/12 00:36:36 INFO Received complete push request method=POST path=/32152026/11/12 00:36:36 INFO Completed push id=15 objects=24 duration=3.458792ms32162026/11/12 00:36:36 INFO Received complete push request method=POST path=/32172026/11/12 00:36:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."32182026/11/12 00:36:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."32192026/11/12 00:36:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32202026/11/12 00:36:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."32212026/11/12 00:36:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."32222026/11/12 00:36:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."32232026/11/12 00:36:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."32242026/11/12 00:36:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."32252026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32262026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."32272026/11/12 00:36:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."32282026/11/12 00:36:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."32292026/11/12 00:36:36 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."32302026/11/12 00:36:36 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."32312026/11/12 00:36:36 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."32322026/11/12 00:36:36 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."32332026/11/12 00:36:36 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."32342026/11/12 00:36:36 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."32352026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."32362026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."32372026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."32382026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32392026/11/12 00:36:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."32402026/11/12 00:36:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."32412026/11/12 00:36:36 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."32422026/11/12 00:36:36 INFO Received push request method=POST path=/32432026/11/12 00:36:36 INFO Completed push id=19 objects=7 duration=1.944118ms32442026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=032452026/11/12 00:36:36 INFO Received uploads request method=POST path=/32462026/11/12 00:36:36 INFO Vacuumed table table=pending_closures32472026/11/12 00:36:36 INFO Received uploads request method=POST path=/32482026/11/12 00:36:36 INFO Vacuumed table table=pending_objects32492026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads32502026/11/12 00:36:36 INFO Vacuumed table table=closures32512026/11/12 00:36:36 INFO Vacuumed table table=objects32522026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=032532026/11/12 00:36:36 INFO Vacuumed table table=pending_closures32542026/11/12 00:36:36 INFO Vacuumed table table=pending_objects32552026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads32562026/11/12 00:36:36 INFO Vacuumed table table=closures32572026/11/12 00:36:36 INFO Received push request method=POST path=/32582026/11/12 00:36:36 INFO Vacuumed table table=objects32592026/11/12 00:36:36 INFO Received uploads request method=POST path=/32602026/11/12 00:36:36 INFO Received push request method=POST path=/32612026/11/12 00:36:36 INFO Received uploads request method=POST path=/32622026/11/12 00:36:36 INFO Received push request method=POST path=/32632026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=032642026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=032652026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period32662026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032672026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period32682026/11/12 00:36:36 INFO Vacuumed table table=pending_closures32692026/11/12 00:36:36 INFO Vacuumed table table=pending_objects32702026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032712026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads32722026/11/12 00:36:36 INFO Received uploads request method=POST path=/32732026/11/12 00:36:36 INFO Vacuumed table table=closures32742026/11/12 00:36:36 INFO Vacuumed table table=pending_closures32752026/11/12 00:36:36 INFO Vacuumed table table=objects32762026/11/12 00:36:36 INFO Received complete upload request method=POST path=/32772026/11/12 00:36:36 INFO Vacuumed table table=pending_objects32782026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads32792026/11/12 00:36:36 INFO Vacuumed table table=closures32802026/11/12 00:36:36 INFO Completed upload id=26 objects=6 duration=1.203731ms32812026/11/12 00:36:36 INFO Vacuumed table table=objects32822026/11/12 00:36:36 INFO Received push request method=POST path=/32832026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=032842026/11/12 00:36:36 INFO Received push request method=POST path=/32852026/11/12 00:36:36 INFO Received complete upload request method=POST path=/32862026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period32872026/11/12 00:36:36 INFO Completed upload id=15 objects=3 duration=1.632895ms32882026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032892026/11/12 00:36:36 INFO Received uploads request method=POST path=/32902026/11/12 00:36:36 INFO Vacuumed table table=pending_closures32912026/11/12 00:36:36 INFO Received uploads request method=POST path=/32922026/11/12 00:36:36 INFO Vacuumed table table=pending_objects32932026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads32942026/11/12 00:36:36 INFO Vacuumed table table=closures32952026/11/12 00:36:36 INFO Received uploads request method=POST path=/32962026/11/12 00:36:36 INFO Received push request method=POST path=/32972026/11/12 00:36:36 INFO Vacuumed table table=objects32982026/11/12 00:36:36 INFO Received push request method=POST path=/32992026/11/12 00:36:36 INFO Received complete upload request method=POST path=/33002026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=033012026/11/12 00:36:36 INFO Completed upload id=29 objects=5 duration=1.002329ms33022026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period33032026/11/12 00:36:36 INFO Received complete upload request method=POST path=/33042026/11/12 00:36:36 INFO Received push request method=POST path=/33052026/11/12 00:36:36 INFO Completed upload id=6 objects=6 duration=1.917698ms33062026/11/12 00:36:36 INFO Received uploads request method=POST path=/33072026/11/12 00:36:36 INFO Received uploads request method=POST path=/33082026/11/12 00:36:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33092026/11/12 00:36:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33102026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33112026/11/12 00:36:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."33122026/11/12 00:36:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33132026/11/12 00:36:36 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33142026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33152026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33162026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33172026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=033182026/11/12 00:36:36 INFO Received push request method=POST path=/33192026/11/12 00:36:36 INFO Received complete upload request method=POST path=/33202026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period33212026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=033222026/11/12 00:36:36 INFO Completed upload id=12 objects=26 duration=5.021667ms33232026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period33242026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033252026/11/12 00:36:36 INFO Vacuumed table table=pending_closures33262026/11/12 00:36:36 INFO Vacuumed table table=pending_objects33272026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads33282026/11/12 00:36:36 INFO Vacuumed table table=closures33292026/11/12 00:36:36 INFO Vacuumed table table=objects33302026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=033312026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period33322026/11/12 00:36:36 INFO Received push request method=POST path=/33332026/11/12 00:36:36 INFO Received complete push request method=POST path=/33342026/11/12 00:36:36 INFO Completed push id=14 objects=3 duration=3.25421ms33352026/11/12 00:36:36 INFO Received push request method=POST path=/33362026/11/12 00:36:36 INFO Received uploads request method=POST path=/33372026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=033382026/11/12 00:36:36 INFO Vacuumed table table=pending_closures33392026/11/12 00:36:36 INFO Vacuumed table table=pending_objects33402026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads33412026/11/12 00:36:36 INFO Vacuumed table table=closures33422026/11/12 00:36:36 INFO Vacuumed table table=objects33432026/11/12 00:36:36 INFO Received uploads request method=POST path=/33442026/11/12 00:36:36 INFO Received complete push request method=POST path=/33452026/11/12 00:36:36 INFO Received complete push request method=POST path=/33462026/11/12 00:36:36 INFO Received push request method=POST path=/33472026/11/12 00:36:36 INFO Completed push id=21 objects=9 duration=8.389499ms33482026/11/12 00:36:36 INFO Received push request method=POST path=/33492026/11/12 00:36:36 INFO Completed push id=17 objects=22 duration=9.919573ms33502026/11/12 00:36:36 INFO Received uploads request method=POST path=/33512026/11/12 00:36:36 INFO Received uploads request method=POST path=/33522026/11/12 00:36:36 INFO Received complete push request method=POST path=/33532026/11/12 00:36:36 INFO Completed push id=16 objects=8 duration=2.646065ms33542026/11/12 00:36:36 INFO Received uploads request method=POST path=/33552026/11/12 00:36:36 INFO Received push request method=POST path=/33562026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=033572026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=033582026/11/12 00:36:36 INFO Received uploads request method=POST path=/33592026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period33602026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033612026/11/12 00:36:36 INFO Received uploads request method=POST path=/33622026/11/12 00:36:36 INFO Vacuumed table table=pending_closures33632026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033642026/11/12 00:36:36 INFO Vacuumed table table=pending_objects33652026/11/12 00:36:36 INFO Vacuumed table table=pending_closures33662026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads33672026/11/12 00:36:36 INFO Vacuumed table table=closures33682026/11/12 00:36:36 INFO Vacuumed table table=pending_objects33692026/11/12 00:36:36 INFO Vacuumed table table=objects33702026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads33712026/11/12 00:36:36 INFO Vacuumed table table=closures33722026/11/12 00:36:36 INFO Vacuumed table table=objects33732026/11/12 00:36:36 INFO Received complete upload request method=POST path=/33742026/11/12 00:36:36 INFO Completed upload id=31 objects=3 duration=2.0639ms33752026/11/12 00:36:36 INFO Received push request method=POST path=/33762026/11/12 00:36:36 INFO Received push request method=POST path=/33772026/11/12 00:36:36 INFO Received complete upload request method=POST path=/33782026/11/12 00:36:36 INFO Completed upload id=22 objects=2 duration=1.02559ms33792026/11/12 00:36:36 INFO Received complete push request method=POST path=/33802026/11/12 00:36:36 INFO Completed push id=11 objects=6 duration=3.895056ms33812026/11/12 00:36:36 INFO Received push request method=POST path=/33822026/11/12 00:36:36 INFO Received push request method=POST path=/33832026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=033842026/11/12 00:36:36 INFO Received push request method=POST path=/33852026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period33862026/11/12 00:36:36 INFO Received complete push request method=POST path=/33872026/11/12 00:36:36 INFO Received uploads request method=POST path=/33882026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033892026/11/12 00:36:36 INFO Completed push id=19 objects=14 duration=1.602415ms33902026/11/12 00:36:36 INFO Vacuumed table table=pending_closures33912026/11/12 00:36:36 INFO Vacuumed table table=pending_objects33922026/11/12 00:36:36 INFO Received complete push request method=POST path=/33932026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads33942026/11/12 00:36:36 INFO Vacuumed table table=closures33952026/11/12 00:36:36 INFO Completed push id=18 objects=6 duration=1.316773ms33962026/11/12 00:36:36 INFO Vacuumed table table=objects33972026/11/12 00:36:36 INFO Received uploads request method=POST path=/33982026/11/12 00:36:36 INFO Received push request method=POST path=/33992026/11/12 00:36:36 INFO Received uploads request method=POST path=/34002026/11/12 00:36:36 INFO Received complete push request method=POST path=/34012026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34022026/11/12 00:36:36 INFO Completed upload id=18 objects=2 duration=1.777337ms34032026/11/12 00:36:36 INFO Completed push id=13 objects=12 duration=3.354471ms34042026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=034052026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=034062026/11/12 00:36:36 INFO Received push request method=POST path=/34072026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34082026/11/12 00:36:36 INFO Completed upload id=20 objects=6 duration=1.138811ms34092026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=034102026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=034112026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34122026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period34132026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34142026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads34152026/11/12 00:36:36 INFO Vacuumed table table=closures34162026/11/12 00:36:36 INFO Vacuumed table table=objects34172026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034182026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34192026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34202026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads34212026/11/12 00:36:36 INFO Vacuumed table table=closures34222026/11/12 00:36:36 INFO Vacuumed table table=objects34232026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34242026/11/12 00:36:36 INFO Completed upload id=13 objects=7 duration=1.09493ms34252026/11/12 00:36:36 INFO Received uploads request method=POST path=/34262026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34272026/11/12 00:36:36 INFO Received push request method=POST path=/34282026/11/12 00:36:36 INFO Completed upload id=20 objects=4 duration=9.159626ms34292026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=034302026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034312026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34322026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34332026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads34342026/11/12 00:36:36 INFO Vacuumed table table=closures34352026/11/12 00:36:36 INFO Vacuumed table table=objects34362026/11/12 00:36:36 INFO Received push request method=POST path=/34372026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34382026/11/12 00:36:36 INFO Completed upload id=23 objects=7 duration=1.00111ms34392026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34402026/11/12 00:36:36 INFO Completed upload id=13 objects=8 duration=1.296952ms34412026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34422026/11/12 00:36:36 INFO Completed upload id=21 objects=2 duration=969.969µs34432026/11/12 00:36:36 INFO Received complete push request method=POST path=/34442026/11/12 00:36:36 INFO Completed push id=19 objects=6 duration=3.973718ms34452026/11/12 00:36:36 INFO Received uploads request method=POST path=/34462026/11/12 00:36:36 INFO Received uploads request method=POST path=/34472026/11/12 00:36:36 INFO Received complete push request method=POST path=/34482026/11/12 00:36:36 INFO Completed push id=9 objects=4 duration=4.347641ms34492026/11/12 00:36:36 INFO Received complete upload request method=POST path=/34502026/11/12 00:36:36 INFO Completed upload id=24 objects=18 duration=1.656996ms34512026/11/12 00:36:36 INFO Received uploads request method=POST path=/34522026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=034532026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period34542026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034552026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34562026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34572026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads34582026/11/12 00:36:36 INFO Vacuumed table table=closures34592026/11/12 00:36:36 INFO Vacuumed table table=objects34602026/11/12 00:36:36 INFO Received push request method=POST path=/34612026/11/12 00:36:36 INFO Received complete push request method=POST path=/34622026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=034632026/11/12 00:36:36 INFO Completed push id=25 objects=2 duration=1.742317ms34642026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period34652026/11/12 00:36:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34662026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34672026/11/12 00:36:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34682026/11/12 00:36:36 INFO Received complete push request method=POST path=/34692026/11/12 00:36:36 INFO Completed push id=27 objects=4 duration=1.008969ms34702026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=034712026/11/12 00:36:36 INFO Received uploads request method=POST path=/34722026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period34732026/11/12 00:36:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34742026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34752026/11/12 00:36:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34762026/11/12 00:36:36 INFO Received complete push request method=POST path=/34772026/11/12 00:36:36 INFO Received complete push request method=POST path=/34782026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034792026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034802026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034812026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34822026/11/12 00:36:36 INFO Completed push id=5 objects=28 duration=3.536613ms34832026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34842026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34852026/11/12 00:36:36 INFO Completed push id=28 objects=26 duration=1.375273ms34862026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34872026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34882026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads34892026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34902026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034912026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads34922026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads34932026/11/12 00:36:36 INFO Vacuumed table table=closures34942026/11/12 00:36:36 INFO Vacuumed table table=closures34952026/11/12 00:36:36 INFO Vacuumed table table=pending_closures34962026/11/12 00:36:36 INFO Received push request method=POST path=/34972026/11/12 00:36:36 INFO Vacuumed table table=pending_objects34982026/11/12 00:36:36 INFO Vacuumed table table=closures34992026/11/12 00:36:36 INFO Vacuumed table table=objects35002026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35012026/11/12 00:36:36 INFO Vacuumed table table=objects35022026/11/12 00:36:36 INFO Vacuumed table table=closures35032026/11/12 00:36:36 INFO Vacuumed table table=objects35042026/11/12 00:36:36 INFO Vacuumed table table=objects35052026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period35062026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period35072026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035082026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=035092026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35102026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35112026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35122026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35132026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35142026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35152026/11/12 00:36:36 INFO Vacuumed table table=closures35162026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035172026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035182026/11/12 00:36:36 INFO Vacuumed table table=closures35192026/11/12 00:36:36 INFO Received complete push request method=POST path=/35202026/11/12 00:36:36 INFO Vacuumed table table=objects35212026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035222026/11/12 00:36:36 INFO Vacuumed table table=objects35232026/11/12 00:36:36 INFO Completed push id=30 objects=3 duration=1.159951ms35242026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35252026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35262026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35272026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35282026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35292026/11/12 00:36:36 INFO Vacuumed table table=closures35302026/11/12 00:36:36 INFO Received push request method=POST path=/35312026/11/12 00:36:36 INFO Vacuumed table table=objects35322026/11/12 00:36:36 INFO Received uploads request method=POST path=/35332026/11/12 00:36:36 INFO Received push request method=POST path=/35342026/11/12 00:36:36 INFO Received complete push request method=POST path=/35352026/11/12 00:36:36 INFO Received complete upload request method=POST path=/35362026/11/12 00:36:36 INFO Received complete push request method=POST path=/35372026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35382026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35392026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35402026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35412026/11/12 00:36:36 INFO Vacuumed table table=closures35422026/11/12 00:36:36 INFO Completed push id=10 objects=2 duration=1.607395ms35432026/11/12 00:36:36 INFO Vacuumed table table=objects35442026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35452026/11/12 00:36:36 INFO Completed upload id=20 objects=3 duration=4.770125ms35462026/11/12 00:36:36 INFO Completed push id=24 objects=24 duration=5.143948ms35472026/11/12 00:36:36 INFO Vacuumed table table=closures35482026/11/12 00:36:36 INFO Vacuumed table table=objects35492026/11/12 00:36:36 INFO Received push request method=POST path=/35502026/11/12 00:36:36 INFO Received complete push request method=POST path=/35512026/11/12 00:36:36 INFO Received uploads request method=POST path=/35522026/11/12 00:36:36 INFO Received push request method=POST path=/35532026/11/12 00:36:36 INFO Completed push id=32 objects=19 duration=1.212292ms35542026/11/12 00:36:36 INFO Received complete push request method=POST path=/35552026/11/12 00:36:36 INFO Received complete upload request method=POST path=/35562026/11/12 00:36:36 INFO Completed push id=8 objects=6 duration=2.894327ms35572026/11/12 00:36:36 INFO Received uploads request method=POST path=/35582026/11/12 00:36:36 INFO Received push request method=POST path=/35592026/11/12 00:36:36 INFO Completed upload id=9 objects=7 duration=2.252261ms35602026/11/12 00:36:36 INFO Received push request method=POST path=/35612026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=035622026/11/12 00:36:36 INFO Received push request method=POST path=/35632026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period35642026/11/12 00:36:36 INFO Received complete push request method=POST path=/35652026/11/12 00:36:36 INFO Received push request method=POST path=/35662026/11/12 00:36:36 INFO Received complete upload request method=POST path=/35672026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035682026/11/12 00:36:36 INFO Completed upload id=17 objects=5 duration=995.129µs35692026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35702026/11/12 00:36:36 INFO Completed push id=15 objects=7 duration=2.060999ms35712026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35722026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35732026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=035742026/11/12 00:36:36 INFO Vacuumed table table=closures35752026/11/12 00:36:36 INFO Received uploads request method=POST path=/35762026/11/12 00:36:36 INFO Vacuumed table table=objects35772026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period35782026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035792026/11/12 00:36:36 INFO Received complete push request method=POST path=/35802026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=035812026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35822026/11/12 00:36:36 INFO Completed push id=17 objects=22 duration=2.085319ms35832026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35842026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35852026/11/12 00:36:36 INFO Vacuumed table table=closures35862026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035872026/11/12 00:36:36 INFO Received uploads request method=POST path=/35882026/11/12 00:36:36 INFO Vacuumed table table=objects35892026/11/12 00:36:36 INFO Vacuumed table table=pending_closures35902026/11/12 00:36:36 INFO Received push request method=POST path=/35912026/11/12 00:36:36 INFO Vacuumed table table=pending_objects35922026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads35932026/11/12 00:36:36 INFO Vacuumed table table=closures35942026/11/12 00:36:36 INFO Vacuumed table table=objects35952026/11/12 00:36:36 INFO Received complete push request method=POST path=/35962026/11/12 00:36:36 INFO Received uploads request method=POST path=/35972026/11/12 00:36:36 INFO Received complete push request method=POST path=/35982026/11/12 00:36:36 INFO Completed push id=11 objects=8 duration=2.915548ms35992026/11/12 00:36:36 INFO Completed push id=19 objects=3 duration=1.15273ms36002026/11/12 00:36:36 INFO Received uploads request method=POST path=/36012026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036022026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36032026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period36042026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36052026/11/12 00:36:36 INFO Completed upload id=19 objects=25 duration=2.803126ms36062026/11/12 00:36:36 INFO Completed upload id=24 objects=7 duration=1.06737ms36072026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036082026/11/12 00:36:36 INFO Vacuumed table table=pending_closures36092026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036102026/11/12 00:36:36 INFO Vacuumed table table=pending_objects36112026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads36122026/11/12 00:36:36 INFO Vacuumed table table=closures36132026/11/12 00:36:36 INFO Vacuumed table table=objects36142026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036152026/11/12 00:36:36 INFO Received uploads request method=POST path=/36162026/11/12 00:36:36 INFO Vacuumed table table=pending_closures36172026/11/12 00:36:36 INFO Vacuumed table table=pending_objects36182026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads36192026/11/12 00:36:36 INFO Vacuumed table table=closures36202026/11/12 00:36:36 INFO Vacuumed table table=objects36212026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36222026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036232026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period36242026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036252026/11/12 00:36:36 INFO Completed upload id=6 objects=7 duration=4.592283ms36262026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036272026/11/12 00:36:36 INFO Received complete push request method=POST path=/36282026/11/12 00:36:36 INFO Completed push id=20 objects=4 duration=1.11675ms36292026/11/12 00:36:36 INFO Received push request method=POST path=/36302026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036312026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036322026/11/12 00:36:36 INFO Vacuumed table table=pending_closures36332026/11/12 00:36:36 INFO Vacuumed table table=pending_closures36342026/11/12 00:36:36 INFO Vacuumed table table=pending_objects36352026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads36362026/11/12 00:36:36 INFO Vacuumed table table=closures36372026/11/12 00:36:36 INFO Vacuumed table table=pending_objects36382026/11/12 00:36:36 INFO Vacuumed table table=objects36392026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads36402026/11/12 00:36:36 INFO Vacuumed table table=closures36412026/11/12 00:36:36 INFO Vacuumed table table=objects36422026/11/12 00:36:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."36432026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36442026/11/12 00:36:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36452026/11/12 00:36:36 INFO Received complete push request method=POST path=/36462026/11/12 00:36:36 INFO Received uploads request method=POST path=/36472026/11/12 00:36:36 INFO Received push request method=POST path=/36482026/11/12 00:36:36 INFO Received complete push request method=POST path=/36492026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36502026/11/12 00:36:36 INFO Received push request method=POST path=/36512026/11/12 00:36:36 INFO Completed push id=13 objects=6 duration=3.725094ms36522026/11/12 00:36:36 INFO Received push request method=POST path=/36532026/11/12 00:36:36 INFO Completed push id=9 objects=24 duration=2.091799ms36542026/11/12 00:36:36 INFO Received uploads request method=POST path=/36552026/11/12 00:36:36 INFO Completed upload id=6 objects=24 duration=3.028029ms36562026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36572026/11/12 00:36:36 INFO Completed upload id=7 objects=10 duration=1.421074ms36582026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36592026/11/12 00:36:36 INFO Completed upload id=22 objects=8 duration=935.629µs36602026/11/12 00:36:36 INFO Received push request method=POST path=/36612026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36622026/11/12 00:36:36 INFO Completed upload id=22 objects=2 duration=1.126671ms36632026/11/12 00:36:36 INFO Received push request method=POST path=/36642026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036652026/11/12 00:36:36 INFO Received uploads request method=POST path=/36662026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period36672026/11/12 00:36:36 INFO Received uploads request method=POST path=/36682026/11/12 00:36:36 INFO Received push request method=POST path=/36692026/11/12 00:36:36 INFO Received push request method=POST path=/36702026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036712026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036722026/11/12 00:36:36 INFO Received uploads request method=POST path=/36732026/11/12 00:36:36 INFO Vacuumed table table=pending_closures36742026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period36752026/11/12 00:36:36 INFO Received complete push request method=POST path=/36762026/11/12 00:36:36 INFO Vacuumed table table=pending_objects36772026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads36782026/11/12 00:36:36 INFO Vacuumed table table=closures36792026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036802026/11/12 00:36:36 INFO Completed push id=23 objects=5 duration=1.449793ms36812026/11/12 00:36:36 INFO Vacuumed table table=objects36822026/11/12 00:36:36 INFO Vacuumed table table=pending_closures36832026/11/12 00:36:36 INFO Vacuumed table table=pending_objects36842026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads36852026/11/12 00:36:36 INFO Vacuumed table table=closures36862026/11/12 00:36:36 INFO Vacuumed table table=objects36872026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036882026/11/12 00:36:36 INFO Received complete upload request method=POST path=/36892026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=036902026/11/12 00:36:36 INFO Received complete push request method=POST path=/36912026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period36922026/11/12 00:36:36 INFO Received complete push request method=POST path=/36932026/11/12 00:36:36 INFO Received complete push request method=POST path=/36942026/11/12 00:36:36 INFO Completed upload id=1 objects=15 duration=5.655813ms36952026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036962026/11/12 00:36:36 INFO Completed push id=10 objects=2 duration=2.024679ms36972026/11/12 00:36:36 INFO Completed push id=8 objects=6 duration=2.07644ms36982026/11/12 00:36:36 INFO Received uploads request method=POST path=/36992026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37002026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037012026/11/12 00:36:36 INFO Vacuumed table table=pending_closures37022026/11/12 00:36:36 INFO Received uploads request method=POST path=/37032026/11/12 00:36:36 INFO Completed upload id=24 objects=8 duration=1.006169ms37042026/11/12 00:36:36 INFO Vacuumed table table=pending_objects37052026/11/12 00:36:36 INFO Vacuumed table table=pending_closures37062026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads37072026/11/12 00:36:36 INFO Vacuumed table table=closures37082026/11/12 00:36:36 INFO Vacuumed table table=pending_objects37092026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads37102026/11/12 00:36:36 INFO Vacuumed table table=objects37112026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=037122026/11/12 00:36:36 INFO Vacuumed table table=closures37132026/11/12 00:36:36 WARN Force mode enabled - objects will be deleted immediately without grace period37142026/11/12 00:36:36 INFO Vacuumed table table=objects37152026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037162026/11/12 00:36:36 INFO Completed push id=3 objects=22 duration=8.698501ms37172026/11/12 00:36:36 INFO Vacuumed table table=pending_closures37182026/11/12 00:36:36 INFO Received uploads request method=POST path=/37192026/11/12 00:36:36 INFO Vacuumed table table=pending_objects37202026/11/12 00:36:36 INFO Vacuumed table table=multipart_uploads37212026/11/12 00:36:36 INFO Received complete push request method=POST path=/37222026/11/12 00:36:36 INFO Vacuumed table table=closures37232026/11/12 00:36:36 INFO Vacuumed table table=objects37242026/11/12 00:36:36 INFO Completed push id=23 objects=28 duration=3.260851ms37252026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37262026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37272026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37282026/11/12 00:36:36 INFO Completed upload id=12 objects=21 duration=1.05239ms37292026/11/12 00:36:36 INFO Completed upload id=28 objects=5 duration=983.389µs37302026/11/12 00:36:36 INFO Completed upload id=9 objects=3 duration=1.263672ms37312026/11/12 00:36:36 INFO Received push request method=POST path=/37322026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37332026/11/12 00:36:36 INFO Completed upload id=10 objects=10 duration=789.488µs37342026/11/12 00:36:36 INFO Received uploads request method=POST path=/37352026/11/12 00:36:36 INFO Received complete push request method=POST path=/37362026/11/12 00:36:36 INFO Completed push id=17 objects=6 duration=1.728456ms37372026/11/12 00:36:36 INFO Received push request method=POST path=/37382026/11/12 00:36:36 INFO Received uploads request method=POST path=/37392026/11/12 00:36:36 INFO Received push request method=POST path=/37402026/11/12 00:36:36 INFO Received push request method=POST path=/37412026/11/12 00:36:36 INFO Received push request method=POST path=/37422026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37432026/11/12 00:36:36 INFO Completed upload id=30 objects=5 duration=902.768µs37442026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37452026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=037462026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37472026/11/12 00:36:36 INFO Completed upload id=3 objects=24 duration=1.400573ms37482026/11/12 00:36:36 INFO Received push request method=POST path=/37492026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37502026/11/12 00:36:36 INFO Completed upload id=11 objects=7 duration=1.277532ms37512026/11/12 00:36:36 INFO Completed upload id=27 objects=8 duration=1.087471ms37522026/11/12 00:36:36 INFO Received push request method=POST path=/37532026/11/12 00:36:36 INFO Received complete push request method=POST path=/37542026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=037552026/11/12 00:36:36 INFO Received uploads request method=POST path=/37562026/11/12 00:36:36 INFO Completed push id=4 objects=7 duration=1.526735ms37572026/11/12 00:36:36 INFO Received complete push request method=POST path=/37582026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37592026/11/12 00:36:36 INFO Completed push id=13 objects=23 duration=1.655636ms37602026/11/12 00:36:36 INFO Completed upload id=23 objects=7 duration=2.512144ms37612026/11/12 00:36:36 INFO Received push request method=POST path=/37622026/11/12 00:36:36 INFO Received uploads request method=POST path=/37632026/11/12 00:36:36 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."37642026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37652026/11/12 00:36:36 INFO Received complete push request method=POST path=/37662026/11/12 00:36:36 INFO Received push request method=POST path=/37672026/11/12 00:36:36 INFO Received push request method=POST path=/37682026/11/12 00:36:36 INFO Completed push id=31 objects=3 duration=1.025009ms37692026/11/12 00:36:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."37702026/11/12 00:36:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."37712026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37722026/11/12 00:36:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."37732026/11/12 00:36:36 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."37742026/11/12 00:36:36 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."37752026/11/12 00:36:36 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."37762026/11/12 00:36:36 INFO Received push request method=POST path=/37772026/11/12 00:36:36 INFO Received uploads request method=POST path=/37782026/11/12 00:36:36 INFO Received push request method=POST path=/37792026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37802026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37812026/11/12 00:36:36 INFO Completed upload id=26 objects=5 duration=806.728µs37822026/11/12 00:36:36 INFO Completed upload id=23 objects=16 duration=2.21178ms37832026/11/12 00:36:36 INFO Received uploads request method=POST path=/37842026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=037852026/11/12 00:36:36 INFO Received uploads request method=POST path=/37862026/11/12 00:36:36 INFO Received push request method=POST path=/37872026/11/12 00:36:36 INFO Received uploads request method=POST path=/37882026/11/12 00:36:36 INFO Aborted multipart uploads count=0 kept=037892026/11/12 00:36:36 INFO Received complete upload request method=POST path=/37902026/11/12 00:36:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037912026/11/12 00:36:36 INFO Completed upload id=2 objects=2 duration=1.762177ms37922026/11/12 00:36:36 INFO Vacuumed table table=pending_closures37932026/11/12 00:36:36 INFO Received push request method=POST path=/37942026/11/12 00:36:37 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."37952026/11/12 00:36:37 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."37962026/11/12 00:36:37 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37972026/11/12 00:36:37 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."37982026/11/12 00:36:37 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."37992026/11/12 00:36:37 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."38002026/11/12 00:36:37 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."38012026/11/12 00:36:37 INFO Vacuumed table table=pending_objects38022026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads38032026/11/12 00:36:37 INFO Received complete push request method=POST path=/38042026/11/12 00:36:37 INFO Vacuumed table table=closures38052026/11/12 00:36:37 INFO Received uploads request method=POST path=/38062026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038072026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period38082026/11/12 00:36:37 INFO Vacuumed table table=objects38092026/11/12 00:36:37 INFO Received complete push request method=POST path=/38102026/11/12 00:36:37 INFO Completed push id=27 objects=3 duration=1.712056ms38112026/11/12 00:36:37 INFO Completed push id=32 objects=6 duration=928.149µs38122026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038132026/11/12 00:36:37 INFO Vacuumed table table=pending_closures38142026/11/12 00:36:37 INFO Vacuumed table table=pending_objects38152026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads38162026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038172026/11/12 00:36:37 INFO Vacuumed table table=closures38182026/11/12 00:36:37 INFO Vacuumed table table=objects38192026/11/12 00:36:37 INFO Received uploads request method=POST path=/38202026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038212026/11/12 00:36:37 INFO Vacuumed table table=pending_closures38222026/11/12 00:36:37 INFO Vacuumed table table=pending_objects38232026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads38242026/11/12 00:36:37 INFO Received complete push request method=POST path=/38252026/11/12 00:36:37 INFO Vacuumed table table=closures38262026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038272026/11/12 00:36:37 INFO Vacuumed table table=objects38282026/11/12 00:36:37 INFO Completed push id=6 objects=5 duration=2.12892ms38292026/11/12 00:36:37 INFO Received complete push request method=POST path=/38302026/11/12 00:36:37 INFO Received complete upload request method=POST path=/38312026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038322026/11/12 00:36:37 INFO Completed upload id=28 objects=5 duration=911.648µs38332026/11/12 00:36:37 INFO Received complete upload request method=POST path=/38342026/11/12 00:36:37 INFO Completed push id=12 objects=3 duration=2.11774ms38352026/11/12 00:36:37 INFO Completed upload id=33 objects=7 duration=797.008µs38362026/11/12 00:36:37 INFO Received push request method=POST path=/38372026/11/12 00:36:37 INFO Received complete push request method=POST path=/38382026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038392026/11/12 00:36:37 INFO Vacuumed table table=pending_closures38402026/11/12 00:36:37 INFO Received uploads request method=POST path=/38412026/11/12 00:36:37 INFO Received complete push request method=POST path=/38422026/11/12 00:36:37 INFO Completed push id=7 objects=20 duration=2.657985ms38432026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=038442026/11/12 00:36:37 INFO Vacuumed table table=pending_objects38452026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads38462026/11/12 00:36:37 INFO Vacuumed table table=closures38472026/11/12 00:36:37 INFO Vacuumed table table=objects38482026/11/12 00:36:37 INFO Vacuumed table table=pending_closures38492026/11/12 00:36:37 INFO Received push request method=POST path=/38502026/11/12 00:36:37 INFO Completed push id=16 objects=4 duration=2.799906ms38512026/11/12 00:36:37 INFO Vacuumed table table=pending_objects38522026/11/12 00:36:37 INFO Received push request method=POST path=/38532026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads38542026/11/12 00:36:37 INFO Vacuumed table table=closures38552026/11/12 00:36:37 INFO Vacuumed table table=objects38562026/11/12 00:36:37 INFO Received uploads request method=POST path=/38572026/11/12 00:36:37 INFO Received uploads request method=POST path=/38582026/11/12 00:36:37 INFO Received complete push request method=POST path=/38592026/11/12 00:36:37 INFO Completed push id=29 objects=17 duration=1.11539ms38602026/11/12 00:36:37 INFO Received complete upload request method=POST path=/38612026/11/12 00:36:37 INFO Completed upload id=34 objects=3 duration=860.508µs38622026/11/12 00:36:37 INFO Received complete push request method=POST path=/38632026/11/12 00:36:37 INFO Received uploads request method=POST path=/38642026/11/12 00:36:37 INFO Received push request method=POST path=/38652026/11/12 00:36:37 INFO Completed push id=7 objects=7 duration=1.379493ms38662026/11/12 00:36:37 INFO Received complete upload request method=POST path=/38672026/11/12 00:36:37 INFO Completed upload id=25 objects=16 duration=2.389482ms38682026/11/12 00:36:37 INFO Received push request method=POST path=/38692026/11/12 00:36:37 INFO Received push request method=POST path=/38702026/11/12 00:36:37 INFO Received push request method=POST path=/38712026/11/12 00:36:37 INFO Received uploads request method=POST path=/38722026/11/12 00:36:37 INFO Received complete push request method=POST path=/38732026/11/12 00:36:37 INFO Completed push id=14 objects=3 duration=1.457734ms38742026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038752026/11/12 00:36:37 INFO Received uploads request method=POST path=/38762026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period38772026/11/12 00:36:37 INFO Received complete push request method=POST path=/38782026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038792026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038802026/11/12 00:36:37 INFO Completed push id=35 objects=11 duration=1.072091ms38812026/11/12 00:36:37 INFO Received push request method=POST path=/38822026/11/12 00:36:37 INFO Vacuumed table table=pending_closures38832026/11/12 00:36:37 INFO Received complete upload request method=POST path=/38842026/11/12 00:36:37 INFO Vacuumed table table=pending_objects38852026/11/12 00:36:37 INFO Completed upload id=20 objects=5 duration=1.168451ms38862026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038872026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads38882026/11/12 00:36:37 INFO Vacuumed table table=closures38892026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038902026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=038912026/11/12 00:36:37 INFO Vacuumed table table=pending_closures38922026/11/12 00:36:37 INFO Vacuumed table table=objects38932026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period38942026/11/12 00:36:37 INFO Vacuumed table table=pending_objects38952026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads38962026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038972026/11/12 00:36:37 INFO Vacuumed table table=closures38982026/11/12 00:36:37 INFO Vacuumed table table=objects38992026/11/12 00:36:37 INFO Received push request method=POST path=/39002026/11/12 00:36:37 INFO Vacuumed table table=pending_closures39012026/11/12 00:36:37 INFO Vacuumed table table=pending_objects39022026/11/12 00:36:37 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."39032026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads39042026/11/12 00:36:37 INFO Vacuumed table table=closures39052026/11/12 00:36:37 INFO Vacuumed table table=objects39062026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039072026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period39082026/11/12 00:36:37 INFO Received uploads request method=POST path=/39092026/11/12 00:36:37 INFO Received uploads request method=POST path=/39102026/11/12 00:36:37 INFO Received uploads request method=POST path=/39112026/11/12 00:36:37 INFO Received uploads request method=POST path=/39122026/11/12 00:36:37 INFO Received complete push request method=POST path=/39132026/11/12 00:36:37 INFO Completed push id=13 objects=15 duration=1.434493ms39142026/11/12 00:36:37 INFO Received complete push request method=POST path=/39152026/11/12 00:36:37 INFO Received complete upload request method=POST path=/39162026/11/12 00:36:37 INFO Received push request method=POST path=/39172026/11/12 00:36:37 INFO Completed upload id=18 objects=11 duration=1.192771ms39182026/11/12 00:36:37 INFO Completed push id=5 objects=3 duration=2.529844ms39192026/11/12 00:36:37 INFO Received push request method=POST path=/39202026/11/12 00:36:37 INFO Received uploads request method=POST path=/39212026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039222026/11/12 00:36:37 INFO Received uploads request method=POST path=/39232026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039242026/11/12 00:36:37 INFO Received complete upload request method=POST path=/39252026/11/12 00:36:37 INFO Received uploads request method=POST path=/39262026/11/12 00:36:37 INFO Received push request method=POST path=/39272026/11/12 00:36:37 INFO Completed upload id=16 objects=23 duration=1.302613ms39282026/11/12 00:36:37 INFO Received uploads request method=POST path=/39292026/11/12 00:36:37 INFO Received push request method=POST path=/39302026/11/12 00:36:37 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."39312026/11/12 00:36:37 INFO Received push request method=POST path=/39322026/11/12 00:36:37 INFO Received complete upload request method=POST path=/39332026/11/12 00:36:37 INFO Completed upload id=15 objects=4 duration=1.949579ms39342026/11/12 00:36:37 INFO Received push request method=POST path=/39352026/11/12 00:36:37 INFO Received complete push request method=POST path=/39362026/11/12 00:36:37 INFO Received complete upload request method=POST path=/39372026/11/12 00:36:37 INFO Received push request method=POST path=/39382026/11/12 00:36:37 INFO Completed upload id=15 objects=6 duration=963.969µs39392026/11/12 00:36:37 INFO Completed push id=7 objects=15 duration=2.455623ms39402026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039412026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=039422026/11/12 00:36:37 INFO Vacuumed table table=pending_closures39432026/11/12 00:36:37 INFO Vacuumed table table=pending_objects39442026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period39452026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads39462026/11/12 00:36:37 INFO Received push request method=POST path=/39472026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039482026/11/12 00:36:37 INFO Vacuumed table table=closures39492026/11/12 00:36:37 INFO Received push request method=POST path=/39502026/11/12 00:36:37 INFO Vacuumed table table=pending_closures39512026/11/12 00:36:37 INFO Received push request method=POST path=/39522026/11/12 00:36:37 INFO Received push request method=POST path=/39532026/11/12 00:36:37 INFO Vacuumed table table=pending_objects39542026/11/12 00:36:37 INFO Vacuumed table table=objects39552026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads39562026/11/12 00:36:37 INFO Received uploads request method=POST path=/39572026/11/12 00:36:37 INFO Vacuumed table table=closures39582026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039592026/11/12 00:36:37 INFO Vacuumed table table=objects39602026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period39612026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039622026/11/12 00:36:37 INFO Vacuumed table table=pending_closures39632026/11/12 00:36:37 INFO Received complete push request method=POST path=/39642026/11/12 00:36:37 INFO Vacuumed table table=pending_objects39652026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads39662026/11/12 00:36:37 INFO Completed push id=20 objects=3 duration=1.346133ms39672026/11/12 00:36:37 INFO Vacuumed table table=closures39682026/11/12 00:36:37 INFO Vacuumed table table=objects39692026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039702026/11/12 00:36:37 INFO Received complete push request method=POST path=/39712026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period39722026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039732026/11/12 00:36:37 INFO Received uploads request method=POST path=/39742026/11/12 00:36:37 INFO Completed push id=8 objects=6 duration=1.406953ms39752026/11/12 00:36:37 INFO Received push request method=POST path=/39762026/11/12 00:36:37 INFO Received push request method=POST path=/39772026/11/12 00:36:37 INFO Received complete upload request method=POST path=/39782026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039792026/11/12 00:36:37 INFO Received uploads request method=POST path=/39802026/11/12 00:36:37 INFO Vacuumed table table=pending_closures39812026/11/12 00:36:37 INFO Completed upload id=17 objects=7 duration=1.952219ms39822026/11/12 00:36:37 INFO Vacuumed table table=pending_objects39832026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads39842026/11/12 00:36:37 INFO Vacuumed table table=closures39852026/11/12 00:36:37 INFO Vacuumed table table=objects39862026/11/12 00:36:37 INFO Received complete upload request method=POST path=/39872026/11/12 00:36:37 INFO Received push request method=POST path=/39882026/11/12 00:36:37 INFO Completed upload id=16 objects=3 duration=766.567µs39892026/11/12 00:36:37 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."39902026/11/12 00:36:37 INFO Received uploads request method=POST path=/39912026/11/12 00:36:37 INFO Received uploads request method=POST path=/39922026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039932026/11/12 00:36:37 INFO Received complete upload request method=POST path=/39942026/11/12 00:36:37 INFO Received push request method=POST path=/39952026/11/12 00:36:37 INFO Completed upload id=22 objects=5 duration=1.868437ms39962026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039972026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=039982026/11/12 00:36:37 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."39992026/11/12 00:36:37 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40002026/11/12 00:36:37 INFO Received uploads request method=POST path=/40012026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period40022026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040032026/11/12 00:36:37 INFO Received uploads request method=POST path=/40042026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040052026/11/12 00:36:37 INFO Received uploads request method=POST path=/40062026/11/12 00:36:37 INFO Vacuumed table table=pending_closures40072026/11/12 00:36:37 INFO Vacuumed table table=pending_closures40082026/11/12 00:36:37 INFO Received uploads request method=POST path=/40092026/11/12 00:36:37 INFO Received complete push request method=POST path=/40102026/11/12 00:36:37 INFO Vacuumed table table=pending_objects40112026/11/12 00:36:37 INFO Vacuumed table table=pending_objects40122026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads40132026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads40142026/11/12 00:36:37 INFO Received complete push request method=POST path=/40152026/11/12 00:36:37 INFO Vacuumed table table=closures40162026/11/12 00:36:37 INFO Vacuumed table table=closures40172026/11/12 00:36:37 INFO Received push request method=POST path=/40182026/11/12 00:36:37 INFO Vacuumed table table=objects40192026/11/12 00:36:37 INFO Vacuumed table table=objects40202026/11/12 00:36:37 INFO Completed push id=28 objects=3 duration=1.712976ms40212026/11/12 00:36:37 INFO Completed push id=5 objects=9 duration=4.395101ms40222026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=040232026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040242026/11/12 00:36:37 INFO Received complete push request method=POST path=/40252026/11/12 00:36:37 INFO Vacuumed table table=pending_closures40262026/11/12 00:36:37 INFO Completed push id=15 objects=4 duration=1.603875ms40272026/11/12 00:36:37 INFO Vacuumed table table=pending_objects40282026/11/12 00:36:37 INFO Received push request method=POST path=/40292026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=040302026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads40312026/11/12 00:36:37 INFO Received push request method=POST path=/40322026/11/12 00:36:37 INFO Vacuumed table table=closures40332026/11/12 00:36:37 INFO Vacuumed table table=objects40342026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period40352026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=040362026/11/12 00:36:37 INFO Received push request method=POST path=/40372026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=040382026/11/12 00:36:37 INFO Received uploads request method=POST path=/40392026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period40402026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040412026/11/12 00:36:37 INFO Received complete push request method=POST path=/40422026/11/12 00:36:37 INFO Vacuumed table table=pending_closures40432026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040442026/11/12 00:36:37 INFO Completed push id=29 objects=15 duration=1.146131ms40452026/11/12 00:36:37 INFO Vacuumed table table=pending_objects40462026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads40472026/11/12 00:36:37 INFO Vacuumed table table=pending_closures40482026/11/12 00:36:37 INFO Vacuumed table table=closures40492026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040502026/11/12 00:36:37 INFO Vacuumed table table=pending_objects40512026/11/12 00:36:37 INFO Vacuumed table table=objects40522026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads40532026/11/12 00:36:37 INFO Vacuumed table table=pending_closures40542026/11/12 00:36:37 INFO Received complete upload request method=POST path=/40552026/11/12 00:36:37 INFO Vacuumed table table=closures40562026/11/12 00:36:37 INFO Received push request method=POST path=/40572026/11/12 00:36:37 INFO Vacuumed table table=objects40582026/11/12 00:36:37 INFO Completed upload id=14 objects=6 duration=998.389µs40592026/11/12 00:36:37 INFO Vacuumed table table=pending_objects40602026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads40612026/11/12 00:36:37 INFO Vacuumed table table=closures40622026/11/12 00:36:37 INFO Vacuumed table table=objects40632026/11/12 00:36:37 INFO Received push request method=POST path=/40642026/11/12 00:36:37 INFO Received push request method=POST path=/40652026/11/12 00:36:37 INFO Received push request method=POST path=/40662026/11/12 00:36:37 INFO Received complete push request method=POST path=/40672026/11/12 00:36:37 INFO Completed push id=30 objects=6 duration=930.589µs40682026/11/12 00:36:37 INFO Received push request method=POST path=/40692026/11/12 00:36:37 INFO Received uploads request method=POST path=/40702026/11/12 00:36:37 INFO Received uploads request method=POST path=/40712026/11/12 00:36:37 INFO Received push request method=POST path=/40722026/11/12 00:36:37 INFO Received push request method=POST path=/40732026/11/12 00:36:37 INFO Received complete upload request method=POST path=/40742026/11/12 00:36:37 INFO Received complete upload request method=POST path=/40752026/11/12 00:36:37 INFO Completed upload id=22 objects=6 duration=946.468µs40762026/11/12 00:36:37 INFO Completed upload id=33 objects=5 duration=2.395962ms40772026/11/12 00:36:37 INFO Received complete upload request method=POST path=/40782026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=040792026/11/12 00:36:37 INFO Received complete upload request method=POST path=/40802026/11/12 00:36:37 INFO Completed upload id=31 objects=3 duration=983.949µs40812026/11/12 00:36:37 INFO Received complete upload request method=POST path=/40822026/11/12 00:36:37 INFO Completed upload id=18 objects=4 duration=1.630876ms40832026/11/12 00:36:37 INFO Completed upload id=10 objects=17 duration=1.07759ms40842026/11/12 00:36:37 INFO Received uploads request method=POST path=/40852026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040862026/11/12 00:36:37 INFO Vacuumed table table=pending_closures40872026/11/12 00:36:37 INFO Received complete push request method=POST path=/40882026/11/12 00:36:37 INFO Vacuumed table table=pending_objects40892026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads40902026/11/12 00:36:37 INFO Completed push id=19 objects=14 duration=1.830597ms40912026/11/12 00:36:37 INFO Vacuumed table table=closures40922026/11/12 00:36:37 INFO Vacuumed table table=objects40932026/11/12 00:36:37 INFO Received uploads request method=POST path=/40942026/11/12 00:36:37 INFO Received complete upload request method=POST path=/40952026/11/12 00:36:37 INFO Received uploads request method=POST path=/40962026/11/12 00:36:37 INFO Completed upload id=7 objects=3 duration=847.588µs40972026/11/12 00:36:37 INFO Received complete push request method=POST path=/40982026/11/12 00:36:37 INFO Completed push id=14 objects=3 duration=1.05459ms40992026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041002026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period41012026/11/12 00:36:37 INFO Received push request method=POST path=/41022026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041032026/11/12 00:36:37 INFO Received uploads request method=POST path=/41042026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041052026/11/12 00:36:37 INFO Received push request method=POST path=/41062026/11/12 00:36:37 INFO Vacuumed table table=pending_closures41072026/11/12 00:36:37 INFO Received push request method=POST path=/41082026/11/12 00:36:37 INFO Vacuumed table table=pending_objects41092026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads41102026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period41112026/11/12 00:36:37 INFO Vacuumed table table=closures41122026/11/12 00:36:37 INFO Vacuumed table table=objects41132026/11/12 00:36:37 INFO Received push request method=POST path=/41142026/11/12 00:36:37 INFO Received complete push request method=POST path=/41152026/11/12 00:36:37 INFO Received complete push request method=POST path=/41162026/11/12 00:36:37 INFO Completed push id=17 objects=6 duration=1.735376ms41172026/11/12 00:36:37 INFO Completed push id=18 objects=23 duration=2.043379ms41182026/11/12 00:36:37 INFO Received uploads request method=POST path=/41192026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041202026/11/12 00:36:37 INFO Received complete upload request method=POST path=/41212026/11/12 00:36:37 INFO Received uploads request method=POST path=/41222026/11/12 00:36:37 INFO Completed upload id=21 objects=25 duration=1.581175ms41232026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041242026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041252026/11/12 00:36:37 INFO Vacuumed table table=pending_closures41262026/11/12 00:36:37 INFO Vacuumed table table=pending_objects41272026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads41282026/11/12 00:36:37 INFO Received push request method=POST path=/41292026/11/12 00:36:37 INFO Vacuumed table table=closures41302026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period41312026/11/12 00:36:37 INFO Vacuumed table table=objects41322026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041332026/11/12 00:36:37 INFO Received uploads request method=POST path=/41342026/11/12 00:36:37 INFO Vacuumed table table=pending_closures41352026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=041362026/11/12 00:36:37 INFO Vacuumed table table=pending_objects41372026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads41382026/11/12 00:36:37 INFO Vacuumed table table=pending_closures41392026/11/12 00:36:37 INFO Received complete upload request method=POST path=/41402026/11/12 00:36:37 INFO Vacuumed table table=closures41412026/11/12 00:36:37 INFO Completed upload id=20 objects=4 duration=793.147µs41422026/11/12 00:36:37 INFO Vacuumed table table=objects41432026/11/12 00:36:37 INFO Vacuumed table table=pending_objects41442026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads41452026/11/12 00:36:37 INFO Vacuumed table table=closures41462026/11/12 00:36:37 INFO Received uploads request method=POST path=/41472026/11/12 00:36:37 INFO Vacuumed table table=objects41482026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041492026/11/12 00:36:37 INFO Received push request method=POST path=/41502026/11/12 00:36:37 INFO Received push request method=POST path=/41512026/11/12 00:36:37 INFO Received complete push request method=POST path=/41522026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period41532026/11/12 00:36:37 INFO Received push request method=POST path=/41542026/11/12 00:36:37 INFO Completed push id=14 objects=6 duration=1.557075ms41552026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041562026/11/12 00:36:37 INFO Vacuumed table table=pending_closures41572026/11/12 00:36:37 INFO Vacuumed table table=pending_objects41582026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads41592026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041602026/11/12 00:36:37 INFO Vacuumed table table=closures41612026/11/12 00:36:37 INFO Vacuumed table table=objects41622026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period41632026/11/12 00:36:37 INFO Received uploads request method=POST path=/41642026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041652026/11/12 00:36:37 INFO Received complete push request method=POST path=/41662026/11/12 00:36:37 INFO Vacuumed table table=pending_closures41672026/11/12 00:36:37 INFO Received push request method=POST path=/41682026/11/12 00:36:37 INFO Vacuumed table table=pending_objects41692026/11/12 00:36:37 INFO Completed push id=11 objects=9 duration=1.506175ms41702026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads41712026/11/12 00:36:37 INFO Vacuumed table table=closures41722026/11/12 00:36:37 INFO Received push request method=POST path=/41732026/11/12 00:36:37 INFO Vacuumed table table=objects41742026/11/12 00:36:37 INFO Received uploads request method=POST path=/41752026/11/12 00:36:37 INFO Received push request method=POST path=/41762026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041772026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041782026/11/12 00:36:37 INFO Received complete push request method=POST path=/41792026/11/12 00:36:37 INFO Vacuumed table table=pending_closures41802026/11/12 00:36:37 INFO Completed push id=16 objects=5 duration=1.05855ms41812026/11/12 00:36:37 INFO Vacuumed table table=pending_objects41822026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads41832026/11/12 00:36:37 INFO Vacuumed table table=closures41842026/11/12 00:36:37 INFO Vacuumed table table=objects41852026/11/12 00:36:37 INFO Received push request method=POST path=/41862026/11/12 00:36:37 INFO Received uploads request method=POST path=/41872026/11/12 00:36:37 INFO Received uploads request method=POST path=/41882026/11/12 00:36:37 INFO Received push request method=POST path=/41892026/11/12 00:36:37 INFO Received complete upload request method=POST path=/41902026/11/12 00:36:37 INFO Completed upload id=12 objects=6 duration=1.132651ms41912026/11/12 00:36:37 INFO Received uploads request method=POST path=/41922026/11/12 00:36:37 INFO Received push request method=POST path=/41932026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=041942026/11/12 00:36:37 INFO Received complete push request method=POST path=/41952026/11/12 00:36:37 INFO Received push request method=POST path=/41962026/11/12 00:36:37 INFO Completed push id=18 objects=6 duration=2.880007ms41972026/11/12 00:36:37 INFO Received uploads request method=POST path=/41982026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041992026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42002026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=042012026/11/12 00:36:37 INFO Vacuumed table table=pending_objects42022026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads42032026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=042042026/11/12 00:36:37 INFO Vacuumed table table=closures42052026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=042062026/11/12 00:36:37 INFO Received complete push request method=POST path=/42072026/11/12 00:36:37 INFO Vacuumed table table=objects42082026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042092026/11/12 00:36:37 INFO Completed push id=17 objects=3 duration=1.609335ms42102026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42112026/11/12 00:36:37 INFO Vacuumed table table=pending_objects42122026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads42132026/11/12 00:36:37 INFO Vacuumed table table=closures42142026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=042152026/11/12 00:36:37 INFO Vacuumed table table=objects42162026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42172026/11/12 00:36:37 INFO Received push request method=POST path=/42182026/11/12 00:36:37 INFO Vacuumed table table=pending_objects42192026/11/12 00:36:37 INFO Received complete push request method=POST path=/42202026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads42212026/11/12 00:36:37 INFO Vacuumed table table=closures42222026/11/12 00:36:37 INFO Vacuumed table table=objects42232026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042242026/11/12 00:36:37 INFO Received uploads request method=POST path=/42252026/11/12 00:36:37 INFO Received uploads request method=POST path=/42262026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42272026/11/12 00:36:37 INFO Completed push id=11 objects=11 duration=4.082138ms42282026/11/12 00:36:37 INFO Received complete upload request method=POST path=/42292026/11/12 00:36:37 INFO Vacuumed table table=pending_objects42302026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads42312026/11/12 00:36:37 INFO Vacuumed table table=closures42322026/11/12 00:36:37 INFO Completed upload id=23 objects=9 duration=1.279353ms42332026/11/12 00:36:37 INFO Vacuumed table table=objects42342026/11/12 00:36:37 INFO Received complete push request method=POST path=/42352026/11/12 00:36:37 INFO Received complete push request method=POST path=/42362026/11/12 00:36:37 INFO Completed push id=23 objects=5 duration=1.190971ms42372026/11/12 00:36:37 INFO Received complete push request method=POST path=/42382026/11/12 00:36:37 INFO Completed push id=31 objects=14 duration=2.309901ms42392026/11/12 00:36:37 INFO Completed push id=9 objects=7 duration=4.25066ms42402026/11/12 00:36:37 INFO Received uploads request method=POST path=/42412026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=042422026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=042432026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period42442026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042452026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42462026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042472026/11/12 00:36:37 INFO Vacuumed table table=pending_objects42482026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads42492026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42502026/11/12 00:36:37 INFO Vacuumed table table=closures42512026/11/12 00:36:37 INFO Vacuumed table table=pending_objects42522026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads42532026/11/12 00:36:37 INFO Vacuumed table table=objects42542026/11/12 00:36:37 INFO Received complete upload request method=POST path=/42552026/11/12 00:36:37 INFO Received push request method=POST path=/42562026/11/12 00:36:37 INFO Vacuumed table table=closures42572026/11/12 00:36:37 INFO Completed upload id=24 objects=3 duration=813.327µs42582026/11/12 00:36:37 INFO Vacuumed table table=objects42592026/11/12 00:36:37 INFO Received uploads request method=POST path=/42602026/11/12 00:36:37 INFO Received push request method=POST path=/42612026/11/12 00:36:37 INFO Received complete upload request method=POST path=/42622026/11/12 00:36:37 INFO Received complete push request method=POST path=/42632026/11/12 00:36:37 INFO Completed upload id=10 objects=7 duration=1.400813ms42642026/11/12 00:36:37 INFO Received uploads request method=POST path=/42652026/11/12 00:36:37 INFO Completed push id=4 objects=6 duration=1.867998ms42662026/11/12 00:36:37 INFO Received complete push request method=POST path=/42672026/11/12 00:36:37 INFO Received uploads request method=POST path=/42682026/11/12 00:36:37 INFO Completed push id=25 objects=4 duration=1.213411ms42692026/11/12 00:36:37 INFO Received uploads request method=POST path=/42702026/11/12 00:36:37 INFO Received uploads request method=POST path=/42712026/11/12 00:36:37 INFO Received push request method=POST path=/42722026/11/12 00:36:37 INFO Received push request method=POST path=/42732026/11/12 00:36:37 INFO Received uploads request method=POST path=/42742026/11/12 00:36:37 INFO Received uploads request method=POST path=/42752026/11/12 00:36:37 INFO Received complete upload request method=POST path=/42762026/11/12 00:36:37 INFO Received uploads request method=POST path=/42772026/11/12 00:36:37 INFO Completed upload id=13 objects=14 duration=2.925228ms42782026/11/12 00:36:37 INFO Received uploads request method=POST path=/42792026/11/12 00:36:37 INFO Received push request method=POST path=/42802026/11/12 00:36:37 INFO Received push request method=POST path=/42812026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=042822026/11/12 00:36:37 INFO Received complete push request method=POST path=/42832026/11/12 00:36:37 INFO Completed push id=12 objects=2 duration=2.895506ms42842026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042852026/11/12 00:36:37 INFO Received complete upload request method=POST path=/42862026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42872026/11/12 00:36:37 INFO Vacuumed table table=pending_objects42882026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads42892026/11/12 00:36:37 INFO Completed upload id=9 objects=2 duration=1.228992ms42902026/11/12 00:36:37 INFO Vacuumed table table=closures42912026/11/12 00:36:37 INFO Vacuumed table table=objects42922026/11/12 00:36:37 INFO Received push request method=POST path=/42932026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=042942026/11/12 00:36:37 INFO Received uploads request method=POST path=/42952026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042962026/11/12 00:36:37 INFO Received push request method=POST path=/42972026/11/12 00:36:37 INFO Vacuumed table table=pending_closures42982026/11/12 00:36:37 INFO Received complete push request method=POST path=/42992026/11/12 00:36:37 INFO Received complete push request method=POST path=/43002026/11/12 00:36:37 INFO Vacuumed table table=pending_objects43012026/11/12 00:36:37 INFO Received complete upload request method=POST path=/43022026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads43032026/11/12 00:36:37 INFO Completed push id=26 objects=23 duration=1.319452ms43042026/11/12 00:36:37 INFO Completed push id=7 objects=7 duration=1.557535ms43052026/11/12 00:36:37 INFO Vacuumed table table=closures43062026/11/12 00:36:37 INFO Vacuumed table table=objects43072026/11/12 00:36:37 INFO Received uploads request method=POST path=/43082026/11/12 00:36:37 INFO Completed upload id=23 objects=15 duration=1.761017ms43092026/11/12 00:36:37 INFO Received complete push request method=POST path=/43102026/11/12 00:36:37 INFO Completed push id=8 objects=3 duration=1.671696ms43112026/11/12 00:36:37 INFO Received complete push request method=POST path=/43122026/11/12 00:36:37 INFO Completed push id=16 objects=7 duration=1.679015ms43132026/11/12 00:36:37 INFO Received complete push request method=POST path=/43142026/11/12 00:36:37 INFO Received complete upload request method=POST path=/43152026/11/12 00:36:37 INFO Received push request method=POST path=/43162026/11/12 00:36:37 INFO Completed upload id=18 objects=4 duration=1.08631ms43172026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=043182026/11/12 00:36:37 INFO Completed push id=31 objects=6 duration=3.18693ms43192026/11/12 00:36:37 INFO Received push request method=POST path=/43202026/11/12 00:36:37 INFO Received complete push request method=POST path=/43212026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043222026/11/12 00:36:37 INFO Completed push id=12 objects=6 duration=887.148µs43232026/11/12 00:36:37 INFO Vacuumed table table=pending_closures43242026/11/12 00:36:37 INFO Vacuumed table table=pending_objects43252026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads43262026/11/12 00:36:37 INFO Vacuumed table table=closures43272026/11/12 00:36:37 INFO Vacuumed table table=objects43282026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=043292026/11/12 00:36:37 INFO Received complete push request method=POST path=/43302026/11/12 00:36:37 INFO Received uploads request method=POST path=/43312026/11/12 00:36:37 INFO Received uploads request method=POST path=/43322026/11/12 00:36:37 INFO Completed push id=16 objects=26 duration=1.343133ms43332026/11/12 00:36:37 INFO Received complete push request method=POST path=/43342026/11/12 00:36:37 INFO Received uploads request method=POST path=/43352026/11/12 00:36:37 INFO Completed push id=27 objects=15 duration=1.171671ms43362026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043372026/11/12 00:36:37 INFO Vacuumed table table=pending_closures43382026/11/12 00:36:37 INFO Received complete push request method=POST path=/43392026/11/12 00:36:37 INFO Vacuumed table table=pending_objects43402026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads43412026/11/12 00:36:37 INFO Completed push id=23 objects=6 duration=1.12207ms43422026/11/12 00:36:37 INFO Vacuumed table table=closures43432026/11/12 00:36:37 INFO Vacuumed table table=objects43442026/11/12 00:36:37 INFO Received complete upload request method=POST path=/43452026/11/12 00:36:37 INFO Completed upload id=19 objects=6 duration=1.11429ms43462026/11/12 00:36:37 INFO Received push request method=POST path=/43472026/11/12 00:36:37 INFO Received complete upload request method=POST path=/43482026/11/12 00:36:37 INFO Completed upload id=13 objects=10 duration=962.629µs43492026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=043502026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=043512026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period43522026/11/12 00:36:37 INFO Received complete push request method=POST path=/43532026/11/12 00:36:37 INFO Completed push id=24 objects=6 duration=1.00323ms43542026/11/12 00:36:37 INFO Received push request method=POST path=/43552026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043562026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043572026/11/12 00:36:37 INFO Vacuumed table table=pending_closures43582026/11/12 00:36:37 INFO Vacuumed table table=pending_closures43592026/11/12 00:36:37 INFO Vacuumed table table=pending_objects43602026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads43612026/11/12 00:36:37 INFO Vacuumed table table=pending_objects43622026/11/12 00:36:37 INFO Vacuumed table table=closures43632026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads43642026/11/12 00:36:37 INFO Vacuumed table table=closures43652026/11/12 00:36:37 INFO Vacuumed table table=objects43662026/11/12 00:36:37 INFO Received push request method=POST path=/43672026/11/12 00:36:37 INFO Received complete push request method=POST path=/43682026/11/12 00:36:37 INFO Vacuumed table table=objects43692026/11/12 00:36:37 INFO Completed push id=13 objects=22 duration=1.377373ms43702026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=043712026/11/12 00:36:37 INFO Received complete upload request method=POST path=/43722026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period43732026/11/12 00:36:37 INFO Completed upload id=22 objects=7 duration=933.649µs43742026/11/12 00:36:37 INFO Received uploads request method=POST path=/43752026/11/12 00:36:37 INFO Received push request method=POST path=/43762026/11/12 00:36:37 INFO Received complete push request method=POST path=/43772026/11/12 00:36:37 INFO Received uploads request method=POST path=/43782026/11/12 00:36:37 INFO Received complete push request method=POST path=/43792026/11/12 00:36:37 INFO Completed push id=28 objects=19 duration=1.151051ms43802026/11/12 00:36:37 INFO Received complete push request method=POST path=/43812026/11/12 00:36:37 INFO Received complete push request method=POST path=/43822026/11/12 00:36:37 INFO Completed push id=21 objects=6 duration=1.07873ms43832026/11/12 00:36:37 INFO Completed push id=27 objects=6 duration=2.499503ms43842026/11/12 00:36:37 INFO Completed push id=25 objects=7 duration=1.236712ms43852026/11/12 00:36:37 INFO Received push request method=POST path=/43862026/11/12 00:36:37 INFO Received push request method=POST path=/43872026/11/12 00:36:37 INFO Received uploads request method=POST path=/43882026/11/12 00:36:37 INFO Received complete upload request method=POST path=/43892026/11/12 00:36:37 INFO Received complete push request method=POST path=/43902026/11/12 00:36:37 INFO Completed upload id=22 objects=7 duration=872.148µs43912026/11/12 00:36:37 INFO Completed push id=1 objects=9 duration=1.14653ms43922026/11/12 00:36:37 INFO Received complete push request method=POST path=/43932026/11/12 00:36:37 INFO Completed push id=29 objects=10 duration=1.998039ms43942026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=043952026/11/12 00:36:37 INFO Vacuumed table table=pending_closures43962026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=043972026/11/12 00:36:37 INFO Received complete push request method=POST path=/43982026/11/12 00:36:37 INFO Vacuumed table table=pending_objects43992026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads44002026/11/12 00:36:37 INFO Vacuumed table table=closures44012026/11/12 00:36:37 INFO Vacuumed table table=objects44022026/11/12 00:36:37 INFO Completed push id=26 objects=14 duration=2.552084ms44032026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44042026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044052026/11/12 00:36:37 INFO Received complete push request method=POST path=/44062026/11/12 00:36:37 INFO Completed upload id=15 objects=2 duration=953.529µs44072026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44082026/11/12 00:36:37 INFO Received complete push request method=POST path=/44092026/11/12 00:36:37 INFO Vacuumed table table=pending_closures44102026/11/12 00:36:37 INFO Completed push id=24 objects=4 duration=1.079651ms44112026/11/12 00:36:37 INFO Completed upload id=26 objects=24 duration=1.275872ms44122026/11/12 00:36:37 INFO Vacuumed table table=pending_objects44132026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads44142026/11/12 00:36:37 INFO Vacuumed table table=closures44152026/11/12 00:36:37 INFO Completed push id=1 objects=11 duration=1.940678ms44162026/11/12 00:36:37 INFO Vacuumed table table=objects44172026/11/12 00:36:37 INFO Received complete push request method=POST path=/44182026/11/12 00:36:37 INFO Completed push id=24 objects=9 duration=1.920078ms44192026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=044202026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044212026/11/12 00:36:37 INFO Received push request method=POST path=/44222026/11/12 00:36:37 INFO Vacuumed table table=pending_closures44232026/11/12 00:36:37 INFO Vacuumed table table=pending_objects44242026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44252026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads44262026/11/12 00:36:37 INFO Vacuumed table table=closures44272026/11/12 00:36:37 INFO Completed upload id=30 objects=6 duration=1.191951ms44282026/11/12 00:36:37 INFO Vacuumed table table=objects44292026/11/12 00:36:37 INFO Received push request method=POST path=/44302026/11/12 00:36:37 INFO Received complete push request method=POST path=/44312026/11/12 00:36:37 INFO Received complete push request method=POST path=/44322026/11/12 00:36:37 INFO Completed push id=25 objects=9 duration=736.187µs44332026/11/12 00:36:37 INFO Received push request method=POST path=/44342026/11/12 00:36:37 INFO Completed push id=22 objects=3 duration=2.753206ms44352026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44362026/11/12 00:36:37 INFO Completed upload id=16 objects=3 duration=968.389µs44372026/11/12 00:36:37 INFO Received push request method=POST path=/44382026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=044392026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44402026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period44412026/11/12 00:36:37 INFO Completed upload id=31 objects=5 duration=810.328µs44422026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044432026/11/12 00:36:37 INFO Vacuumed table table=pending_closures44442026/11/12 00:36:37 INFO Received uploads request method=POST path=/44452026/11/12 00:36:37 INFO Vacuumed table table=pending_objects44462026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44472026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads44482026/11/12 00:36:37 INFO Completed upload id=26 objects=3 duration=820.288µs44492026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=044502026/11/12 00:36:37 INFO Vacuumed table table=closures44512026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44522026/11/12 00:36:37 INFO Received push request method=POST path=/44532026/11/12 00:36:37 INFO Vacuumed table table=objects44542026/11/12 00:36:37 INFO Received push request method=POST path=/44552026/11/12 00:36:37 INFO Completed upload id=2 objects=7 duration=1.09219ms44562026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period44572026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44582026/11/12 00:36:37 INFO Completed upload id=27 objects=26 duration=966.869µs44592026/11/12 00:36:37 INFO Received complete push request method=POST path=/44602026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44612026/11/12 00:36:37 INFO Completed upload id=34 objects=3 duration=887.628µs44622026/11/12 00:36:37 INFO Completed push id=14 objects=4 duration=3.697834ms44632026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44642026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=044652026/11/12 00:36:37 INFO Completed upload id=19 objects=2 duration=2.040699ms44662026/11/12 00:36:37 INFO Received push request method=POST path=/44672026/11/12 00:36:37 INFO Received complete push request method=POST path=/44682026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044692026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44702026/11/12 00:36:37 INFO Completed push id=28 objects=6 duration=911.048µs44712026/11/12 00:36:37 INFO Completed upload id=27 objects=7 duration=1.08977ms44722026/11/12 00:36:37 INFO Vacuumed table table=pending_closures44732026/11/12 00:36:37 INFO Vacuumed table table=pending_objects44742026/11/12 00:36:37 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."44752026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads44762026/11/12 00:36:37 INFO Vacuumed table table=closures44772026/11/12 00:36:37 INFO Vacuumed table table=objects44782026/11/12 00:36:37 INFO Received uploads request method=POST path=/44792026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=044802026/11/12 00:36:37 INFO Received push request method=POST path=/44812026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period44822026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044832026/11/12 00:36:37 INFO Received uploads request method=POST path=/44842026/11/12 00:36:37 INFO Vacuumed table table=pending_closures44852026/11/12 00:36:37 INFO Received complete push request method=POST path=/44862026/11/12 00:36:37 INFO Completed push id=29 objects=6 duration=1.203131ms44872026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44882026/11/12 00:36:37 INFO Vacuumed table table=pending_objects44892026/11/12 00:36:37 INFO Received complete upload request method=POST path=/44902026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads44912026/11/12 00:36:37 INFO Completed upload id=8 objects=6 duration=1.242772ms44922026/11/12 00:36:37 INFO Received uploads request method=POST path=/44932026/11/12 00:36:37 INFO Completed upload id=28 objects=3 duration=759.927µs44942026/11/12 00:36:37 INFO Vacuumed table table=closures44952026/11/12 00:36:37 INFO Received push request method=POST path=/44962026/11/12 00:36:37 INFO Vacuumed table table=objects44972026/11/12 00:36:37 INFO Received complete push request method=POST path=/44982026/11/12 00:36:37 INFO Completed push id=33 objects=7 duration=1.743616ms44992026/11/12 00:36:37 INFO Received uploads request method=POST path=/45002026/11/12 00:36:37 INFO Received uploads request method=POST path=/45012026/11/12 00:36:37 INFO Received push request method=POST path=/45022026/11/12 00:36:37 INFO Received complete push request method=POST path=/45032026/11/12 00:36:37 INFO Completed push id=29 objects=23 duration=1.177551ms45042026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45052026/11/12 00:36:37 INFO Completed upload id=26 objects=3 duration=1.643836ms45062026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45072026/11/12 00:36:37 INFO Completed upload id=30 objects=8 duration=1.03285ms45082026/11/12 00:36:37 INFO Received uploads request method=POST path=/45092026/11/12 00:36:37 INFO Received uploads request method=POST path=/45102026/11/12 00:36:37 INFO Received push request method=POST path=/45112026/11/12 00:36:37 INFO Received complete push request method=POST path=/45122026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045132026/11/12 00:36:37 INFO Completed push id=30 objects=6 duration=960.629µs45142026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period45152026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045162026/11/12 00:36:37 INFO Vacuumed table table=pending_closures45172026/11/12 00:36:37 INFO Vacuumed table table=pending_objects45182026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads45192026/11/12 00:36:37 INFO Vacuumed table table=closures45202026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45212026/11/12 00:36:37 INFO Vacuumed table table=objects45222026/11/12 00:36:37 INFO Completed upload id=12 objects=9 duration=1.168971ms45232026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45242026/11/12 00:36:37 INFO Completed upload id=26 objects=8 duration=985.29µs45252026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45262026/11/12 00:36:37 INFO Completed upload id=31 objects=6 duration=904.989µs45272026/11/12 00:36:37 INFO Received push request method=POST path=/45282026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045292026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period45302026/11/12 00:36:37 INFO Received complete push request method=POST path=/45312026/11/12 00:36:37 INFO Completed push id=4 objects=6 duration=1.508094ms45322026/11/12 00:36:37 INFO Received complete push request method=POST path=/45332026/11/12 00:36:37 INFO Received complete push request method=POST path=/45342026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045352026/11/12 00:36:37 INFO Completed push id=21 objects=3 duration=1.225171ms45362026/11/12 00:36:37 INFO Completed push id=33 objects=3 duration=1.323132ms45372026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45382026/11/12 00:36:37 INFO Received complete push request method=POST path=/45392026/11/12 00:36:37 INFO Completed upload id=32 objects=4 duration=1.200731ms45402026/11/12 00:36:37 INFO Completed push id=28 objects=14 duration=1.889757ms45412026/11/12 00:36:37 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."45422026/11/12 00:36:37 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."45432026/11/12 00:36:37 INFO Received uploads request method=POST path=/45442026/11/12 00:36:37 INFO Received complete push request method=POST path=/45452026/11/12 00:36:37 INFO Received uploads request method=POST path=/45462026/11/12 00:36:37 INFO Completed push id=34 objects=6 duration=1.447454ms45472026/11/12 00:36:37 INFO Received push request method=POST path=/45482026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45492026/11/12 00:36:37 INFO Completed upload id=33 objects=10 duration=964.329µs45502026/11/12 00:36:37 INFO Received push request method=POST path=/45512026/11/12 00:36:37 INFO Received complete push request method=POST path=/45522026/11/12 00:36:37 INFO Received complete push request method=POST path=/45532026/11/12 00:36:37 INFO Completed push id=35 objects=8 duration=1.11677ms45542026/11/12 00:36:37 INFO Completed push id=9 objects=9 duration=2.981008ms45552026/11/12 00:36:37 INFO Received uploads request method=POST path=/45562026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045572026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period45582026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045592026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period45602026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045612026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45622026/11/12 00:36:37 INFO Vacuumed table table=pending_closures45632026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=045642026/11/12 00:36:37 INFO Completed upload id=36 objects=9 duration=908.868µs45652026/11/12 00:36:37 INFO Vacuumed table table=pending_objects45662026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45672026/11/12 00:36:37 INFO Vacuumed table table=pending_closures45682026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads45692026/11/12 00:36:37 INFO Received push request method=POST path=/45702026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45712026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045722026/11/12 00:36:37 INFO Vacuumed table table=closures45732026/11/12 00:36:37 INFO Completed upload id=34 objects=23 duration=1.11859ms45742026/11/12 00:36:37 INFO Vacuumed table table=pending_objects45752026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads45762026/11/12 00:36:37 INFO Completed upload id=21 objects=8 duration=1.199091ms45772026/11/12 00:36:37 INFO Vacuumed table table=closures45782026/11/12 00:36:37 INFO Vacuumed table table=objects45792026/11/12 00:36:37 INFO Vacuumed table table=objects45802026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045812026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045822026/11/12 00:36:37 INFO Vacuumed table table=pending_closures45832026/11/12 00:36:37 INFO Vacuumed table table=pending_objects45842026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period45852026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads45862026/11/12 00:36:37 INFO Vacuumed table table=closures45872026/11/12 00:36:37 INFO Vacuumed table table=objects45882026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045892026/11/12 00:36:37 INFO Vacuumed table table=pending_closures45902026/11/12 00:36:37 INFO Received complete push request method=POST path=/45912026/11/12 00:36:37 INFO Completed push id=22 objects=6 duration=1.11199ms45922026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=045932026/11/12 00:36:37 INFO Vacuumed table table=pending_objects45942026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads45952026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period45962026/11/12 00:36:37 INFO Vacuumed table table=closures45972026/11/12 00:36:37 INFO Received complete upload request method=POST path=/45982026/11/12 00:36:37 INFO Vacuumed table table=objects45992026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46002026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046012026/11/12 00:36:37 INFO Completed upload id=18 objects=22 duration=1.10487ms46022026/11/12 00:36:37 INFO Received uploads request method=POST path=/46032026/11/12 00:36:37 INFO Vacuumed table table=pending_closures46042026/11/12 00:36:37 INFO Completed upload id=21 objects=7 duration=2.702505ms46052026/11/12 00:36:37 INFO Received complete push request method=POST path=/46062026/11/12 00:36:37 INFO Received complete push request method=POST path=/46072026/11/12 00:36:37 INFO Completed push id=35 objects=6 duration=1.114471ms46082026/11/12 00:36:37 INFO Vacuumed table table=pending_objects46092026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads46102026/11/12 00:36:37 INFO Vacuumed table table=closures46112026/11/12 00:36:37 INFO Completed push id=27 objects=25 duration=2.404763ms46122026/11/12 00:36:37 INFO Vacuumed table table=objects46132026/11/12 00:36:37 INFO Received complete push request method=POST path=/46142026/11/12 00:36:37 INFO Completed push id=24 objects=8 duration=1.007029ms46152026/11/12 00:36:37 INFO Received push request method=POST path=/46162026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46172026/11/12 00:36:37 INFO Completed upload id=21 objects=8 duration=609.486µs46182026/11/12 00:36:37 INFO Received uploads request method=POST path=/46192026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=046202026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=046212026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period46222026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period46232026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046242026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46252026/11/12 00:36:37 INFO Vacuumed table table=pending_closures46262026/11/12 00:36:37 INFO Completed upload id=25 objects=9 duration=979.209µs46272026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=046282026/11/12 00:36:37 INFO Vacuumed table table=pending_objects46292026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads46302026/11/12 00:36:37 INFO Vacuumed table table=closures46312026/11/12 00:36:37 INFO Vacuumed table table=objects46322026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=046332026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046342026/11/12 00:36:37 INFO Vacuumed table table=pending_closures46352026/11/12 00:36:37 INFO Received push request method=POST path=/46362026/11/12 00:36:37 INFO Received push request method=POST path=/46372026/11/12 00:36:37 INFO Vacuumed table table=pending_closures46382026/11/12 00:36:37 INFO Vacuumed table table=pending_objects46392026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads46402026/11/12 00:36:37 INFO Vacuumed table table=pending_objects46412026/11/12 00:36:37 INFO Vacuumed table table=closures46422026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads46432026/11/12 00:36:37 INFO Vacuumed table table=closures46442026/11/12 00:36:37 INFO Vacuumed table table=objects46452026/11/12 00:36:37 INFO Vacuumed table table=objects46462026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46472026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46482026/11/12 00:36:37 INFO Completed upload id=22 objects=7 duration=870.768µs46492026/11/12 00:36:37 INFO Completed upload id=21 objects=9 duration=2.157301ms46502026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46512026/11/12 00:36:37 INFO Completed upload id=27 objects=3 duration=954.789µs46522026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=046532026/11/12 00:36:37 INFO Received complete push request method=POST path=/46542026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period46552026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046562026/11/12 00:36:37 INFO Received complete push request method=POST path=/46572026/11/12 00:36:37 INFO Completed push id=30 objects=4 duration=2.869907ms46582026/11/12 00:36:37 INFO Received complete push request method=POST path=/46592026/11/12 00:36:37 INFO Vacuumed table table=pending_closures46602026/11/12 00:36:37 INFO Received complete push request method=POST path=/46612026/11/12 00:36:37 INFO Vacuumed table table=pending_objects46622026/11/12 00:36:37 INFO Completed push id=24 objects=5 duration=950.629µs46632026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads46642026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46652026/11/12 00:36:37 INFO Vacuumed table table=closures46662026/11/12 00:36:37 INFO Completed push id=24 objects=5 duration=2.408503ms46672026/11/12 00:36:37 INFO Completed upload id=28 objects=6 duration=787.767µs46682026/11/12 00:36:37 INFO Vacuumed table table=objects46692026/11/12 00:36:37 INFO Completed push id=25 objects=10 duration=1.725436ms46702026/11/12 00:36:37 INFO Received uploads request method=POST path=/46712026/11/12 00:36:37 INFO Received uploads request method=POST path=/46722026/11/12 00:36:37 INFO Received complete push request method=POST path=/46732026/11/12 00:36:37 INFO Completed push id=30 objects=4 duration=911.168µs46742026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46752026/11/12 00:36:37 INFO Completed upload id=29 objects=5 duration=905.008µs46762026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46772026/11/12 00:36:37 INFO Completed upload id=25 objects=8 duration=753.547µs46782026/11/12 00:36:37 INFO Received uploads request method=POST path=/46792026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46802026/11/12 00:36:37 INFO Received complete push request method=POST path=/46812026/11/12 00:36:37 INFO Completed push id=31 objects=7 duration=972.389µs46822026/11/12 00:36:37 INFO Received push request method=POST path=/46832026/11/12 00:36:37 INFO Completed upload id=17 objects=25 duration=2.375822ms46842026/11/12 00:36:37 INFO Received uploads request method=POST path=/46852026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46862026/11/12 00:36:37 INFO Completed upload id=30 objects=2 duration=1.09191ms46872026/11/12 00:36:37 INFO Received complete push request method=POST path=/46882026/11/12 00:36:37 INFO Completed push id=26 objects=4 duration=958.529µs46892026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46902026/11/12 00:36:37 INFO Completed upload id=27 objects=4 duration=862.309µs46912026/11/12 00:36:37 INFO Received complete upload request method=POST path=/46922026/11/12 00:36:37 INFO Completed upload id=32 objects=5 duration=858.608µs46932026/11/12 00:36:37 INFO Received push request method=POST path=/46942026/11/12 00:36:37 INFO Received push request method=POST path=/46952026/11/12 00:36:37 INFO Received complete push request method=POST path=/46962026/11/12 00:36:37 INFO Received complete push request method=POST path=/46972026/11/12 00:36:37 INFO Received complete push request method=POST path=/46982026/11/12 00:36:37 INFO Completed push id=31 objects=20 duration=1.390073ms46992026/11/12 00:36:37 INFO Completed push id=30 objects=2 duration=984.969µs47002026/11/12 00:36:37 INFO Completed push id=13 objects=11 duration=1.664455ms47012026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47022026/11/12 00:36:37 INFO Received complete push request method=POST path=/47032026/11/12 00:36:37 INFO Completed upload id=21 objects=2 duration=1.009449ms47042026/11/12 00:36:37 INFO Completed push id=32 objects=7 duration=1.567735ms47052026/11/12 00:36:37 INFO Received complete push request method=POST path=/47062026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47072026/11/12 00:36:37 INFO Received uploads request method=POST path=/47082026/11/12 00:36:37 INFO Completed upload id=32 objects=5 duration=995.77µs47092026/11/12 00:36:37 INFO Completed push id=21 objects=26 duration=1.413193ms47102026/11/12 00:36:37 INFO Received complete push request method=POST path=/47112026/11/12 00:36:37 INFO Completed push id=32 objects=7 duration=1.207152ms47122026/11/12 00:36:37 INFO Received complete push request method=POST path=/47132026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=047142026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period47152026/11/12 00:36:37 INFO Completed push id=26 objects=4 duration=943.829µs47162026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047172026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47182026/11/12 00:36:37 INFO Vacuumed table table=pending_closures47192026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=047202026/11/12 00:36:37 INFO Completed upload id=33 objects=3 duration=935.569µs47212026/11/12 00:36:37 INFO Vacuumed table table=pending_objects47222026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads47232026/11/12 00:36:37 INFO Vacuumed table table=closures47242026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period47252026/11/12 00:36:37 INFO Vacuumed table table=objects47262026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47272026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47282026/11/12 00:36:37 INFO Completed upload id=34 objects=24 duration=1.08073ms47292026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047302026/11/12 00:36:37 INFO Completed upload id=28 objects=17 duration=1.12543ms47312026/11/12 00:36:37 INFO Vacuumed table table=pending_closures47322026/11/12 00:36:37 INFO Vacuumed table table=pending_objects47332026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads47342026/11/12 00:36:37 INFO Vacuumed table table=closures47352026/11/12 00:36:37 INFO Vacuumed table table=objects47362026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47372026/11/12 00:36:37 INFO Completed upload id=28 objects=13 duration=1.015309ms47382026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=047392026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47402026/11/12 00:36:37 INFO Completed upload id=33 objects=6 duration=918.709µs47412026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period47422026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=047432026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period47442026/11/12 00:36:37 INFO Received complete push request method=POST path=/47452026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047462026/11/12 00:36:37 INFO Completed push id=29 objects=2 duration=1.09137ms47472026/11/12 00:36:37 INFO Vacuumed table table=pending_closures47482026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047492026/11/12 00:36:37 INFO Vacuumed table table=pending_objects47502026/11/12 00:36:37 INFO Vacuumed table table=pending_closures47512026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads47522026/11/12 00:36:37 INFO Vacuumed table table=closures47532026/11/12 00:36:37 INFO Vacuumed table table=pending_objects47542026/11/12 00:36:37 INFO Vacuumed table table=objects47552026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads47562026/11/12 00:36:37 INFO Vacuumed table table=closures47572026/11/12 00:36:37 INFO Vacuumed table table=objects47582026/11/12 00:36:37 INFO Received complete push request method=POST path=/47592026/11/12 00:36:37 INFO Completed push id=29 objects=3 duration=869.648µs47602026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47612026/11/12 00:36:37 INFO Completed upload id=34 objects=3 duration=781.967µs47622026/11/12 00:36:37 INFO Received complete push request method=POST path=/47632026/11/12 00:36:37 INFO Received uploads request method=POST path=/47642026/11/12 00:36:37 INFO Completed push id=31 objects=6 duration=1.00201ms47652026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47662026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=047672026/11/12 00:36:37 INFO Completed upload id=31 objects=2 duration=1.031549ms47682026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period47692026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047702026/11/12 00:36:37 INFO Received complete push request method=POST path=/47712026/11/12 00:36:37 INFO Vacuumed table table=pending_closures47722026/11/12 00:36:37 INFO Completed push id=32 objects=9 duration=1.212152ms47732026/11/12 00:36:37 INFO Vacuumed table table=pending_objects47742026/11/12 00:36:37 INFO Received complete push request method=POST path=/47752026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads47762026/11/12 00:36:37 INFO Vacuumed table table=closures47772026/11/12 00:36:37 INFO Vacuumed table table=objects47782026/11/12 00:36:37 INFO Completed push id=7 objects=24 duration=3.17181ms47792026/11/12 00:36:37 INFO Received complete push request method=POST path=/47802026/11/12 00:36:37 INFO Received complete push request method=POST path=/47812026/11/12 00:36:37 INFO Completed push id=16 objects=14 duration=3.090569ms47822026/11/12 00:36:37 INFO Completed push id=33 objects=6 duration=943.928µs47832026/11/12 00:36:37 INFO Received uploads request method=POST path=/47842026/11/12 00:36:37 INFO Received complete upload request method=POST path=/47852026/11/12 00:36:37 INFO Completed upload id=33 objects=9 duration=933.268µs47862026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=047872026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period47882026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=047892026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047902026/11/12 00:36:37 INFO Vacuumed table table=pending_closures47912026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period47922026/11/12 00:36:37 INFO Vacuumed table table=pending_objects47932026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads47942026/11/12 00:36:37 INFO Vacuumed table table=closures47952026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047962026/11/12 00:36:37 INFO Vacuumed table table=objects47972026/11/12 00:36:37 INFO Vacuumed table table=pending_closures47982026/11/12 00:36:37 INFO Vacuumed table table=pending_objects47992026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads48002026/11/12 00:36:37 INFO Received uploads request method=POST path=/48012026/11/12 00:36:37 INFO Vacuumed table table=closures48022026/11/12 00:36:37 INFO Vacuumed table table=objects48032026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=048042026/11/12 00:36:37 WARN Force mode enabled - objects will be deleted immediately without grace period48052026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048062026/11/12 00:36:37 INFO Aborted multipart uploads count=0 kept=048072026/11/12 00:36:37 INFO Vacuumed table table=pending_closures48082026/11/12 00:36:37 INFO Vacuumed table table=pending_objects48092026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads48102026/11/12 00:36:37 INFO Vacuumed table table=closures48112026/11/12 00:36:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048122026/11/12 00:36:37 INFO Vacuumed table table=objects48132026/11/12 00:36:37 INFO Vacuumed table table=pending_closures48142026/11/12 00:36:37 INFO Vacuumed table table=pending_objects48152026/11/12 00:36:37 INFO Vacuumed table table=multipart_uploads48162026/11/12 00:36:37 INFO Vacuumed table table=closures48172026/11/12 00:36:37 INFO Received uploads request method=POST path=/48182026/11/12 00:36:37 INFO Vacuumed table table=objects48192026/11/12 00:36:37 INFO Received push request method=POST path=/48202026/11/12 00:36:37 INFO Received push request method=POST path=/48212026/11/12 00:36:37 INFO Received uploads request method=POST path=/48222026/11/12 00:36:37 INFO Received uploads request method=POST path=/48232026/11/12 00:36:37 INFO Received complete upload request method=POST path=/48242026/11/12 00:36:37 INFO Completed upload id=25 objects=6 duration=1.331433ms48252026/11/12 00:36:37 INFO Received complete push request method=POST path=/48262026/11/12 00:36:37 INFO Completed push id=13 objects=2 duration=1.268292ms48272026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48282026/11/12 00:36:38 INFO Completed upload id=27 objects=13 duration=863.408µs48292026/11/12 00:36:38 INFO Received push request method=POST path=/48302026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=048312026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048322026/11/12 00:36:38 INFO Vacuumed table table=pending_closures48332026/11/12 00:36:38 INFO Vacuumed table table=pending_objects48342026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads48352026/11/12 00:36:38 INFO Vacuumed table table=closures48362026/11/12 00:36:38 INFO Vacuumed table table=objects48372026/11/12 00:36:38 INFO Received uploads request method=POST path=/48382026/11/12 00:36:38 INFO Received uploads request method=POST path=/48392026/11/12 00:36:38 INFO Received complete push request method=POST path=/48402026/11/12 00:36:38 INFO Completed push id=11 objects=9 duration=1.366073ms48412026/11/12 00:36:38 INFO Received push request method=POST path=/48422026/11/12 00:36:38 INFO Received complete push request method=POST path=/48432026/11/12 00:36:38 INFO Completed push id=15 objects=24 duration=3.308331ms48442026/11/12 00:36:38 INFO Received push request method=POST path=/48452026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48462026/11/12 00:36:38 INFO Completed upload id=19 objects=2 duration=5.662574ms48472026/11/12 00:36:38 INFO Received uploads request method=POST path=/48482026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48492026/11/12 00:36:38 INFO Completed upload id=24 objects=7 duration=8.452239ms48502026/11/12 00:36:38 INFO Received complete push request method=POST path=/48512026/11/12 00:36:38 INFO Completed push id=26 objects=3 duration=6.37876ms48522026/11/12 00:36:38 INFO Received uploads request method=POST path=/48532026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48542026/11/12 00:36:38 INFO Completed upload id=28 objects=8 duration=1.283792ms48552026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48562026/11/12 00:36:38 INFO Completed upload id=29 objects=4 duration=1.731716ms48572026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48582026/11/12 00:36:38 INFO Completed upload id=29 objects=10 duration=991.19µs48592026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48602026/11/12 00:36:38 INFO Received uploads request method=POST path=/48612026/11/12 00:36:38 INFO Completed upload id=30 objects=20 duration=1.000709ms48622026/11/12 00:36:38 INFO Received complete push request method=POST path=/48632026/11/12 00:36:38 INFO Received uploads request method=POST path=/48642026/11/12 00:36:38 INFO Completed push id=31 objects=11 duration=906.348µs48652026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48662026/11/12 00:36:38 INFO Completed upload id=32 objects=2 duration=904.928µs48672026/11/12 00:36:38 INFO Received complete push request method=POST path=/48682026/11/12 00:36:38 INFO Completed push id=11 objects=6 duration=1.570554ms48692026/11/12 00:36:38 INFO Received complete push request method=POST path=/48702026/11/12 00:36:38 INFO Completed push id=33 objects=3 duration=854.628µs48712026/11/12 00:36:38 INFO Received complete push request method=POST path=/48722026/11/12 00:36:38 INFO Completed push id=20 objects=13 duration=1.461274ms48732026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48742026/11/12 00:36:38 INFO Completed upload id=34 objects=3 duration=1.201092ms48752026/11/12 00:36:38 INFO Received complete push request method=POST path=/48762026/11/12 00:36:38 INFO Completed push id=22 objects=4 duration=1.152991ms48772026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=048782026/11/12 00:36:38 WARN Force mode enabled - objects will be deleted immediately without grace period48792026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048802026/11/12 00:36:38 INFO Vacuumed table table=pending_closures48812026/11/12 00:36:38 INFO Vacuumed table table=pending_objects48822026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads48832026/11/12 00:36:38 INFO Vacuumed table table=closures48842026/11/12 00:36:38 INFO Vacuumed table table=objects48852026/11/12 00:36:38 INFO Received complete upload request method=POST path=/48862026/11/12 00:36:38 INFO Completed upload id=23 objects=7 duration=1.223332ms48872026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048882026/11/12 00:36:38 INFO Vacuumed table table=pending_closures48892026/11/12 00:36:38 INFO Vacuumed table table=pending_objects48902026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads48912026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048922026/11/12 00:36:38 INFO Vacuumed table table=closures48932026/11/12 00:36:38 INFO Vacuumed table table=objects48942026/11/12 00:36:38 INFO Vacuumed table table=pending_closures48952026/11/12 00:36:38 INFO Received complete push request method=POST path=/48962026/11/12 00:36:38 INFO Vacuumed table table=pending_objects48972026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads48982026/11/12 00:36:38 INFO Vacuumed table table=closures48992026/11/12 00:36:38 INFO Completed push id=24 objects=5 duration=1.107051ms49002026/11/12 00:36:38 INFO Vacuumed table table=objects49012026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49022026/11/12 00:36:38 INFO Completed upload id=25 objects=2 duration=913.128µs49032026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49042026/11/12 00:36:38 INFO Completed upload id=26 objects=20 duration=945.749µs49052026/11/12 00:36:38 INFO Received complete push request method=POST path=/49062026/11/12 00:36:38 INFO Completed push id=16 objects=16 duration=3.342691ms49072026/11/12 00:36:38 INFO Received complete push request method=POST path=/49082026/11/12 00:36:38 INFO Completed push id=28 objects=7 duration=1.02981ms49092026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049102026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49112026/11/12 00:36:38 INFO Completed upload id=30 objects=6 duration=1.009549ms49122026/11/12 00:36:38 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."49132026/11/12 00:36:38 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."49142026/11/12 00:36:38 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."49152026/11/12 00:36:38 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."49162026/11/12 00:36:38 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."49172026/11/12 00:36:38 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."49182026/11/12 00:36:38 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."49192026/11/12 00:36:38 INFO Received complete push request method=POST path=/49202026/11/12 00:36:38 INFO Completed push id=31 objects=5 duration=967.489µs49212026/11/12 00:36:38 INFO Received uploads request method=POST path=/49222026/11/12 00:36:38 INFO Received uploads request method=POST path=/49232026/11/12 00:36:38 INFO Received complete push request method=POST path=/49242026/11/12 00:36:38 INFO Completed push id=32 objects=26 duration=1.436154ms49252026/11/12 00:36:38 INFO Received push request method=POST path=/49262026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49272026/11/12 00:36:38 INFO Completed upload id=33 objects=2 duration=829.188µs49282026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49292026/11/12 00:36:38 INFO Received uploads request method=POST path=/49302026/11/12 00:36:38 INFO Completed upload id=34 objects=5 duration=835.108µs49312026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49322026/11/12 00:36:38 INFO Completed upload id=35 objects=11 duration=934.809µs49332026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49342026/11/12 00:36:38 INFO Completed upload id=36 objects=3 duration=732.967µs49352026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049362026/11/12 00:36:38 WARN Force mode enabled - objects will be deleted immediately without grace period49372026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049382026/11/12 00:36:38 WARN Force mode enabled - objects will be deleted immediately without grace period49392026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049402026/11/12 00:36:38 INFO Vacuumed table table=pending_closures49412026/11/12 00:36:38 INFO Vacuumed table table=pending_objects49422026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads49432026/11/12 00:36:38 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."49442026/11/12 00:36:38 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."49452026/11/12 00:36:38 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."49462026/11/12 00:36:38 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."49472026/11/12 00:36:38 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."49482026/11/12 00:36:38 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."49492026/11/12 00:36:38 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."49502026/11/12 00:36:38 INFO Vacuumed table table=closures49512026/11/12 00:36:38 INFO Vacuumed table table=objects49522026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49532026/11/12 00:36:38 INFO Completed upload id=21 objects=3 duration=1.03623ms49542026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049552026/11/12 00:36:38 WARN Force mode enabled - objects will be deleted immediately without grace period49562026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=049572026/11/12 00:36:38 INFO Vacuumed table table=pending_closures49582026/11/12 00:36:38 INFO Vacuumed table table=pending_objects49592026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads49602026/11/12 00:36:38 INFO Vacuumed table table=closures49612026/11/12 00:36:38 INFO Vacuumed table table=objects49622026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049632026/11/12 00:36:38 WARN Force mode enabled - objects will be deleted immediately without grace period49642026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049652026/11/12 00:36:38 INFO Vacuumed table table=pending_closures49662026/11/12 00:36:38 INFO Vacuumed table table=pending_objects49672026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads49682026/11/12 00:36:38 INFO Vacuumed table table=closures49692026/11/12 00:36:38 INFO Vacuumed table table=objects49702026/11/12 00:36:38 INFO Received push request method=POST path=/49712026/11/12 00:36:38 INFO Received complete push request method=POST path=/49722026/11/12 00:36:38 INFO Completed push id=22 objects=3 duration=3.19879ms49732026/11/12 00:36:38 INFO Received uploads request method=POST path=/49742026/11/12 00:36:38 INFO Received push request method=POST path=/49752026/11/12 00:36:38 INFO Received complete push request method=POST path=/49762026/11/12 00:36:38 INFO Completed push id=24 objects=3 duration=1.03127ms49772026/11/12 00:36:38 INFO Received push request method=POST path=/49782026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049792026/11/12 00:36:38 WARN Force mode enabled - objects will be deleted immediately without grace period49802026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049812026/11/12 00:36:38 INFO Vacuumed table table=pending_closures49822026/11/12 00:36:38 INFO Vacuumed table table=pending_objects49832026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads49842026/11/12 00:36:38 INFO Vacuumed table table=closures49852026/11/12 00:36:38 INFO Vacuumed table table=objects49862026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049872026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=049882026/11/12 00:36:38 INFO Vacuumed table table=pending_closures49892026/11/12 00:36:38 INFO Vacuumed table table=pending_objects49902026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads49912026/11/12 00:36:38 INFO Vacuumed table table=closures49922026/11/12 00:36:38 INFO Vacuumed table table=objects49932026/11/12 00:36:38 INFO Received uploads request method=POST path=/49942026/11/12 00:36:38 INFO Received complete upload request method=POST path=/49952026/11/12 00:36:38 INFO Completed upload id=26 objects=2 duration=1.602255ms49962026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=049972026/11/12 00:36:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049982026/11/12 00:36:38 INFO Vacuumed table table=pending_closures49992026/11/12 00:36:38 INFO Vacuumed table table=pending_objects50002026/11/12 00:36:38 INFO Vacuumed table table=multipart_uploads50012026/11/12 00:36:38 INFO Vacuumed table table=closures50022026/11/12 00:36:38 INFO Vacuumed table table=objects50032026/11/12 00:36:38 INFO Received push request method=POST path=/50042026/11/12 00:36:38 INFO Aborted multipart uploads count=0 kept=050052026/11/12 00:36:38 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."50062026/11/12 00:36:38 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."50072026/11/12 00:36:38 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."50082026/11/12 00:36:38 INFO Received push request method=POST path=/50092026/11/12 00:36:38 INFO Received push request method=POST path=/50102026/11/12 00:36:38 INFO Received complete push request method=POST path=/50112026/11/12 00:36:38 INFO Completed push id=28 objects=11 duration=2.210741ms50122026/11/12 00:36:39 INFO Received complete push request method=POST path=/50132026/11/12 00:36:39 INFO Completed push id=29 objects=24 duration=1.455074ms50142026/11/12 00:36:39 INFO Aborted multipart uploads count=0 kept=050152026/11/12 00:36:39 WARN Force mode enabled - objects will be deleted immediately without grace period50162026/11/12 00:36:39 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050172026/11/12 00:36:39 INFO Vacuumed table table=pending_closures50182026/11/12 00:36:39 INFO Vacuumed table table=pending_objects50192026/11/12 00:36:39 INFO Vacuumed table table=multipart_uploads50202026/11/12 00:36:39 INFO Vacuumed table table=closures50212026/11/12 00:36:39 INFO Vacuumed table table=objects5022--- PASS: FuzzPushGC (0.01s)5023 --- PASS: FuzzPushGC/seed#3 (3.17s)5024 --- PASS: FuzzPushGC/seed#7 (3.87s)5025 --- PASS: FuzzPushGC/seed#1 (4.09s)5026 --- PASS: FuzzPushGC/seed#15 (4.28s)5027 --- PASS: FuzzPushGC/seed#13 (4.63s)5028 --- PASS: FuzzPushGC/seed#4 (4.76s)5029 --- PASS: FuzzPushGC/seed#12 (4.78s)5030 --- PASS: FuzzPushGC/seed#6 (4.90s)5031 --- PASS: FuzzPushGC/seed#9 (4.90s)5032 --- PASS: FuzzPushGC/seed#10 (4.93s)5033 --- PASS: FuzzPushGC/seed#14 (4.96s)5034 --- PASS: FuzzPushGC/seed#0 (4.98s)5035 --- PASS: FuzzPushGC/seed#5 (5.29s)5036 --- PASS: FuzzPushGC/seed#8 (5.33s)5037 --- PASS: FuzzPushGC/seed#11 (5.54s)5038 --- PASS: FuzzPushGC/seed#2 (6.09s)5039PASS5040{"timestamp":"2026-11-12T00:36:39.081478071Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49846","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(206)"}5041{"timestamp":"2026-11-12T00:36:39.081523512Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:45066","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(200)"}5042{"timestamp":"2026-11-12T00:36:39.081527032Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:48772","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(202)"}50432026-11-12 00:36:39.581 UTC [132] LOG: received smart shutdown request50442026-11-12 00:36:39.587 UTC [132] LOG: background worker "logical replication launcher" (PID 143) exited with exit code 150452026-11-12 00:36:40.511 UTC [138] LOG: checkpoint complete: wrote 3576 buffers (21.8%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=6.668 s, sync=0.641 s, total=7.489 s; sync files=42378, longest=0.004 s, average=0.001 s; distance=534051 kB, estimate=534051 kB; lsn=0/27015E38, redo lsn=0/220EE9F850462026-11-12 00:36:40.517 UTC [138] LOG: shutting down50472026-11-12 00:36:40.518 UTC [138] LOG: checkpoint starting: shutdown immediate50482026-11-12 00:36:41.073 UTC [138] LOG: checkpoint complete: wrote 8340 buffers (50.9%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.156 s, sync=0.390 s, total=0.557 s; sync files=2567, longest=0.003 s, average=0.001 s; distance=81053 kB, estimate=488751 kB; lsn=0/27015EB0, redo lsn=0/27015EB050492026-11-12 00:36:41.239 UTC [132] LOG: database system is shut down5050Running OIDC tests...5051=== RUN TestAudienceForIssuer5052=== PAUSE TestAudienceForIssuer5053=== RUN TestProviderClientGivesUpOnSilentIssuer5054=== PAUSE TestProviderClientGivesUpOnSilentIssuer5055=== RUN TestGlobMatch5056=== PAUSE TestGlobMatch5057=== RUN TestValidateToken_ValidToken5058=== PAUSE TestValidateToken_ValidToken5059=== RUN TestValidateToken_WrongAudience5060=== PAUSE TestValidateToken_WrongAudience5061=== RUN TestValidateToken_Expired5062=== PAUSE TestValidateToken_Expired5063=== RUN TestValidateToken_BoundClaimsMismatch5064=== PAUSE TestValidateToken_BoundClaimsMismatch5065=== RUN TestValidateToken_BoundSubjectMismatch5066=== PAUSE TestValidateToken_BoundSubjectMismatch5067=== RUN TestValidateToken_MultipleProviders5068=== PAUSE TestValidateToken_MultipleProviders5069=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5070=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5071=== RUN TestValidateToken_GoogleSchemelessIssuer5072=== PAUSE TestValidateToken_GoogleSchemelessIssuer5073=== RUN TestValidateToken_KubernetesServiceAccount5074=== PAUSE TestValidateToken_KubernetesServiceAccount5075=== RUN TestNewValidator_KubernetesRequiresCA5076=== PAUSE TestNewValidator_KubernetesRequiresCA5077=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5078=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5079=== RUN TestPins_ReservedForMatchingRule5080=== PAUSE TestPins_ReservedForMatchingRule5081=== RUN TestPins_TopLevelShorthand5082=== PAUSE TestPins_TopLevelShorthand5083=== RUN TestPins_ConfigValidation5084=== PAUSE TestPins_ConfigValidation5085=== RUN TestScopes_LegacyProviderDefaultsToWrite5086=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5087=== RUN TestScopes_Rules5088=== PAUSE TestScopes_Rules5089=== RUN TestScopes_ConfigValidation5090=== PAUSE TestScopes_ConfigValidation5091=== CONT TestAudienceForIssuer5092--- PASS: TestAudienceForIssuer (0.00s)5093=== CONT TestScopes_ConfigValidation5094--- PASS: TestScopes_ConfigValidation (0.00s)5095=== CONT TestScopes_Rules5096=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5097=== CONT TestValidateToken_Expired50982026/11/12 00:36:44 INFO OIDC provider initialized name=known issuer=https://known.example5099=== CONT TestValidateToken_WrongAudience5100=== CONT TestValidateToken_MultipleProviders5101=== CONT TestScopes_LegacyProviderDefaultsToWrite5102=== CONT TestValidateToken_BoundSubjectMismatch5103=== CONT TestPins_ConfigValidation5104--- PASS: TestPins_ConfigValidation (0.00s)5105=== CONT TestPins_TopLevelShorthand5106=== CONT TestValidateToken_BoundClaimsMismatch5107=== CONT TestPins_ReservedForMatchingRule5108=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5109=== CONT TestNewValidator_KubernetesRequiresCA5110=== CONT TestValidateToken_ValidToken5111=== CONT TestValidateToken_KubernetesServiceAccount5112=== CONT TestValidateToken_GoogleSchemelessIssuer5113=== CONT TestGlobMatch5114=== RUN TestGlobMatch/foo_foo5115=== PAUSE TestGlobMatch/foo_foo5116=== RUN TestGlobMatch/foo_bar5117=== PAUSE TestGlobMatch/foo_bar5118=== RUN TestGlobMatch/*_5119=== PAUSE TestGlobMatch/*_5120=== RUN TestGlobMatch/*_anything5121=== PAUSE TestGlobMatch/*_anything5122=== RUN TestGlobMatch/foo*_foo5123=== PAUSE TestGlobMatch/foo*_foo5124=== RUN TestGlobMatch/foo*_foobar5125=== PAUSE TestGlobMatch/foo*_foobar5126=== RUN TestGlobMatch/foo*_bar5127=== PAUSE TestGlobMatch/foo*_bar5128=== RUN TestGlobMatch/*bar_bar5129=== PAUSE TestGlobMatch/*bar_bar5130=== RUN TestGlobMatch/*bar_foobar5131=== CONT TestProviderClientGivesUpOnSilentIssuer5132=== PAUSE TestGlobMatch/*bar_foobar5133=== RUN TestGlobMatch/*bar_foo5134=== PAUSE TestGlobMatch/*bar_foo5135=== RUN TestGlobMatch/foo*bar_foobar5136=== PAUSE TestGlobMatch/foo*bar_foobar5137=== RUN TestGlobMatch/foo*bar_foo123bar5138=== PAUSE TestGlobMatch/foo*bar_foo123bar5139=== RUN TestGlobMatch/foo*bar_foobarbaz5140=== PAUSE TestGlobMatch/foo*bar_foobarbaz5141=== RUN TestGlobMatch/*/*_foo/bar5142=== PAUSE TestGlobMatch/*/*_foo/bar5143=== RUN TestGlobMatch/*/*_foo5144=== PAUSE TestGlobMatch/*/*_foo5145=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5146=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5147=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05148=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05149=== RUN TestGlobMatch/refs/*/main_refs/heads/main5150=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5151=== RUN TestGlobMatch/fo?_foo5152=== PAUSE TestGlobMatch/fo?_foo5153=== RUN TestGlobMatch/fo?_fo5154=== PAUSE TestGlobMatch/fo?_fo5155=== RUN TestGlobMatch/fo?_fooo5156=== PAUSE TestGlobMatch/fo?_fooo5157=== RUN TestGlobMatch/?oo_foo5158=== PAUSE TestGlobMatch/?oo_foo5159=== RUN TestGlobMatch/?oo_boo5160=== PAUSE TestGlobMatch/?oo_boo5161=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5162=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5163=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5164=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5165=== CONT TestGlobMatch/foo_foo5166=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5167=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5168=== CONT TestGlobMatch/?oo_boo5169=== CONT TestGlobMatch/?oo_foo5170=== CONT TestGlobMatch/fo?_fooo5171=== CONT TestGlobMatch/fo?_fo5172=== CONT TestGlobMatch/fo?_foo5173=== CONT TestGlobMatch/refs/*/main_refs/heads/main5174=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05175=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5176=== CONT TestGlobMatch/*/*_foo5177=== CONT TestGlobMatch/*/*_foo/bar5178=== CONT TestGlobMatch/foo*bar_foobarbaz5179=== CONT TestGlobMatch/foo*bar_foo123bar5180=== CONT TestGlobMatch/foo*bar_foobar5181=== CONT TestGlobMatch/*bar_foo5182=== CONT TestGlobMatch/*bar_foobar5183=== CONT TestGlobMatch/*bar_bar5184=== CONT TestGlobMatch/foo*_bar5185=== CONT TestGlobMatch/foo*_foobar5186=== CONT TestGlobMatch/foo*_foo5187=== CONT TestGlobMatch/*_anything5188=== CONT TestGlobMatch/*_5189=== CONT TestGlobMatch/foo_bar5190--- PASS: TestGlobMatch (0.00s)5191 --- PASS: TestGlobMatch/foo_foo (0.00s)5192 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5193 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5194 --- PASS: TestGlobMatch/?oo_boo (0.00s)5195 --- PASS: TestGlobMatch/?oo_foo (0.00s)5196 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5197 --- PASS: TestGlobMatch/fo?_fo (0.00s)5198 --- PASS: TestGlobMatch/fo?_foo (0.00s)5199 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5200 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5201 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5202 --- PASS: TestGlobMatch/*/*_foo (0.00s)5203 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5204 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5205 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5206 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5207 --- PASS: TestGlobMatch/*bar_foo (0.00s)5208 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5209 --- PASS: TestGlobMatch/*bar_bar (0.00s)5210 --- PASS: TestGlobMatch/foo*_bar (0.00s)5211 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5212 --- PASS: TestGlobMatch/foo*_foo (0.00s)5213 --- PASS: TestGlobMatch/*_anything (0.00s)5214 --- PASS: TestGlobMatch/*_ (0.00s)5215 --- PASS: TestGlobMatch/foo_bar (0.00s)5216--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)52172026/11/12 00:36:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42053/oidc52182026/11/12 00:36:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42267/oidc52192026/11/12 00:36:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36193/oidc5220--- PASS: TestValidateToken_BoundClaimsMismatch (0.83s)5221--- PASS: TestValidateToken_BoundSubjectMismatch (0.85s)5222--- PASS: TestPins_ReservedForMatchingRule (0.83s)52232026/11/12 00:36:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43829/oidc52242026/11/12 00:36:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42707/oidc5225--- PASS: TestScopes_Rules (3.33s)5226--- PASS: TestValidateToken_WrongAudience (3.32s)52272026/11/12 00:36:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38193/oidc5228--- PASS: TestValidateToken_Expired (3.36s)5229--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (3.39s)52302026/11/12 00:36:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43701/oidc5231--- PASS: TestScopes_LegacyProviderDefaultsToWrite (3.38s)52322026/11/12 00:36:48 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235233--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (3.34s)52342026/11/12 00:36:48 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5235--- PASS: TestValidateToken_GoogleSchemelessIssuer (2.55s)52362026/11/12 00:36:48 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:46061/oidc52372026/11/12 00:36:48 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:33819/oidc5238--- PASS: TestValidateToken_MultipleProviders (3.41s)52392026/11/12 00:36:48 http: TLS handshake error from 127.0.0.1:45062: remote error: tls: bad certificate5240--- PASS: TestNewValidator_KubernetesRequiresCA (3.33s)52412026/11/12 00:36:48 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:349155242--- PASS: TestValidateToken_KubernetesServiceAccount (2.64s)52432026/11/12 00:36:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35965/oidc5244--- PASS: TestPins_TopLevelShorthand (3.51s)52452026/11/12 00:36:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33577/oidc5246--- PASS: TestValidateToken_ValidToken (2.74s)5247PASS5248Running hook tests...5249=== RUN TestSendPathsEmpty5250=== PAUSE TestSendPathsEmpty5251=== RUN TestQueueEnqueueAndFetch5252=== PAUSE TestQueueEnqueueAndFetch5253=== RUN TestQueueDeduplication5254=== PAUSE TestQueueDeduplication5255=== RUN TestQueueRemove5256=== PAUSE TestQueueRemove5257=== RUN TestQueueFetchBatchLimit5258=== PAUSE TestQueueFetchBatchLimit5259=== RUN TestQueueRetryMovesToBack5260=== PAUSE TestQueueRetryMovesToBack5261=== RUN TestQueueFetchRemoveLifecycle5262=== PAUSE TestQueueFetchRemoveLifecycle5263=== RUN TestQueueConcurrentWriters5264=== PAUSE TestQueueConcurrentWriters5265=== RUN TestQueueRemoveLargeClosure5266=== PAUSE TestQueueRemoveLargeClosure5267=== RUN TestServerClientIntegration5268--- PASS: TestServerClientIntegration (0.01s)5269=== RUN TestServerQueueError52702026/11/12 00:36:49 ERROR Failed to queue paths error="permission denied" count=15271--- PASS: TestServerQueueError (0.00s)5272=== RUN TestGetListenerSocketActivation5273 server_test.go:220: === RUN TestGetListenerSocketActivation5274 --- PASS: TestGetListenerSocketActivation (0.00s)5275 PASS5276 5277--- PASS: TestGetListenerSocketActivation (0.23s)5278=== RUN TestServerStalledClientDoesNotBlockShutdown5279=== PAUSE TestServerStalledClientDoesNotBlockShutdown5280=== RUN TestServerBacksOffOnAcceptErrors5281=== PAUSE TestServerBacksOffOnAcceptErrors5282=== RUN TestDrainIsolatesPoisonPath5283=== PAUSE TestDrainIsolatesPoisonPath5284=== RUN TestRunNotBlockedByPoisonHead5285=== PAUSE TestRunNotBlockedByPoisonHead5286=== RUN TestDrainGivesUpWhenServerDown5287=== PAUSE TestDrainGivesUpWhenServerDown5288=== RUN TestFailedPathPrunedByLaterClosure5289=== PAUSE TestFailedPathPrunedByLaterClosure5290=== RUN TestWorkerUploadsAndRemoves5291=== PAUSE TestWorkerUploadsAndRemoves5292=== RUN TestWorkerSkipsGCdPaths5293=== PAUSE TestWorkerSkipsGCdPaths5294=== RUN TestWorkerPrunesClosureDeps5295=== PAUSE TestWorkerPrunesClosureDeps5296=== RUN TestDrainTimeout5297=== PAUSE TestDrainTimeout5298=== RUN TestWorkerRemoveFailureIsNotProgress5299=== PAUSE TestWorkerRemoveFailureIsNotProgress5300=== RUN TestWorkerKeepsPathItCannotStat5301=== PAUSE TestWorkerKeepsPathItCannotStat5302=== CONT TestSendPathsEmpty5303--- PASS: TestSendPathsEmpty (0.00s)5304=== CONT TestWorkerKeepsPathItCannotStat5305=== CONT TestServerBacksOffOnAcceptErrors5306=== CONT TestDrainGivesUpWhenServerDown53072000/01/01 00:00:00 ERROR Accept failed error="too many open files"5308=== CONT TestQueueRemove53092000/01/01 00:00:00 ERROR Accept failed error="too many open files"53102000/01/01 00:00:00 ERROR Accept failed error="too many open files"53112000/01/01 00:00:00 ERROR Accept failed error="too many open files"53122000/01/01 00:00:00 ERROR Accept failed error="too many open files"53132000/01/01 00:00:00 ERROR Accept failed error="too many open files"53142000/01/01 00:00:00 ERROR Accept failed error="too many open files"53152000/01/01 00:00:00 ERROR Accept failed error="too many open files"5316--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5317=== CONT TestQueueRetryMovesToBack5318=== CONT TestWorkerRemoveFailureIsNotProgress5319=== CONT TestDrainTimeout5320=== CONT TestWorkerPrunesClosureDeps5321=== CONT TestWorkerSkipsGCdPaths5322=== CONT TestQueueRemoveLargeClosure5323=== CONT TestServerStalledClientDoesNotBlockShutdown5324=== CONT TestQueueConcurrentWriters5325=== CONT TestQueueFetchRemoveLifecycle5326=== CONT TestQueueDeduplication5327=== CONT TestFailedPathPrunedByLaterClosure5328=== CONT TestQueueFetchBatchLimit5329=== CONT TestQueueEnqueueAndFetch5330=== CONT TestRunNotBlockedByPoisonHead5331=== CONT TestDrainIsolatesPoisonPath5332=== CONT TestWorkerUploadsAndRemoves53332026/11/12 00:36:49 INFO Upload queue status pending=253342026/11/12 00:36:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths3321576706/002/nonexistent53352026/11/12 00:36:49 INFO Uploading batch count=153362026/11/12 00:36:49 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2550096361/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2550096361/002/locked/aaa: permission denied"5337--- PASS: TestQueueRetryMovesToBack (0.02s)53382026/11/12 00:36:49 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2550096361/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2550096361/002/locked/aaa: permission denied"53392026/11/12 00:36:49 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2550096361/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2550096361/002/locked/aaa: permission denied"53402026/11/12 00:36:49 ERROR Drain finished with paths left in queue remaining=153412026/11/12 00:36:49 INFO Upload queue status pending=253422026/11/12 00:36:49 INFO Uploading batch count=253432000/01/01 00:00:00 INFO Uploading batch count=25344--- PASS: TestQueueEnqueueAndFetch (0.02s)53452000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253462000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45347--- PASS: TestQueueDeduplication (0.03s)5348--- PASS: TestDrainTimeout (0.03s)53492026/11/12 00:36:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3045483356/002/nonexistent53502026/11/12 00:36:49 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153512026/11/12 00:36:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3045483356/002/nonexistent53522026/11/12 00:36:49 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153532026/11/12 00:36:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress3045483356/002/nonexistent53542026/11/12 00:36:49 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153552026/11/12 00:36:49 ERROR Drain finished with paths left in queue remaining=15356--- PASS: TestWorkerRemoveFailureIsNotProgress (0.04s)53572026/11/12 00:36:49 INFO Uploading batch count=253582026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=253592026/11/12 00:36:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown501906004/002/a53602026/11/12 00:36:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown501906004/002/b53612026/11/12 00:36:49 INFO Uploading batch count=253622026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=253632026/11/12 00:36:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown501906004/002/c5364--- PASS: TestQueueRemove (0.04s)53652026/11/12 00:36:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown501906004/002/d53662026/11/12 00:36:49 INFO Uploading batch count=253672026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=253682026/11/12 00:36:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown501906004/002/e53692026/11/12 00:36:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown501906004/002/f53702026/11/12 00:36:49 ERROR Drain finished with paths left in queue remaining=105371--- PASS: TestDrainGivesUpWhenServerDown (0.05s)5372--- PASS: TestWorkerSkipsGCdPaths (0.05s)53732026/11/12 00:36:49 INFO Upload queue status pending=253742026/11/12 00:36:49 INFO Uploading batch count=453752026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=453762026/11/12 00:36:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath2963652367/002/bbb53772026/11/12 00:36:49 INFO Uploading batch count=153782026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=153792026/11/12 00:36:49 INFO Uploading batch count=153802026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=153812026/11/12 00:36:49 INFO Uploading batch count=153822026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=153832026/11/12 00:36:49 ERROR Drain finished with paths left in queue remaining=15384--- PASS: TestDrainIsolatesPoisonPath (0.05s)53852026/11/12 00:36:49 INFO Uploading batch count=153862026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=153872026/11/12 00:36:49 INFO Uploading batch count=153882026/11/12 00:36:49 INFO Uploading batch count=15389--- PASS: TestFailedPathPrunedByLaterClosure (0.05s)5390--- PASS: TestQueueFetchRemoveLifecycle (0.05s)5391--- PASS: TestWorkerKeepsPathItCannotStat (0.06s)53922026/11/12 00:36:49 INFO Upload queue status pending=353932026/11/12 00:36:49 INFO Uploading batch count=153942026/11/12 00:36:49 ERROR Upload failed error="upload failed" count=15395--- PASS: TestWorkerUploadsAndRemoves (0.07s)53962026/11/12 00:36:49 INFO Uploading batch count=15397--- PASS: TestQueueFetchBatchLimit (0.08s)53982026/11/12 00:36:51 ERROR Failed to decode request error="read unix /build/hook4006883132/test.sock->@: i/o timeout"53992026/11/12 00:36:51 ERROR Failed to write response error="write unix /build/hook4006883132/test.sock->@: i/o timeout"5400--- PASS: TestServerStalledClientDoesNotBlockShutdown (1.68s)54012026/11/12 00:36:51 INFO Uploading batch count=154022026/11/12 00:36:51 INFO Uploading batch count=15403--- PASS: TestWorkerPrunesClosureDeps (1.69s)54042026/11/12 00:36:51 INFO Uploading batch count=154052026/11/12 00:36:51 ERROR Upload failed error="upload failed" count=154062026/11/12 00:36:51 INFO Uploading batch count=154072026/11/12 00:36:51 ERROR Upload failed error="upload failed" count=154082026/11/12 00:36:51 INFO Uploading batch count=154092026/11/12 00:36:51 ERROR Upload failed error="upload failed" count=154102026/11/12 00:36:51 INFO Uploading batch count=154112026/11/12 00:36:51 ERROR Upload failed error="upload failed" count=154122026/11/12 00:36:51 ERROR Drain finished with paths left in queue remaining=15413--- PASS: TestRunNotBlockedByPoisonHead (1.71s)5414--- PASS: TestQueueConcurrentWriters (1.75s)5415--- PASS: TestQueueRemoveLargeClosure (1.78s)5416PASS5417Running signing tests...5418=== RUN TestExternalSigner5419=== PAUSE TestExternalSigner5420=== RUN TestExternalSignerWrongPublicKey5421=== PAUSE TestExternalSignerWrongPublicKey5422=== RUN TestExternalSignerErrors5423=== PAUSE TestExternalSignerErrors5424=== RUN TestExternalSignerRestartsAfterCrash5425=== PAUSE TestExternalSignerRestartsAfterCrash5426=== RUN TestExternalSignerProgramErrorKeepsProcess5427=== PAUSE TestExternalSignerProgramErrorKeepsProcess5428=== RUN TestExternalSignerClose5429=== PAUSE TestExternalSignerClose5430=== RUN TestExternalSignerRejectsInvalidPublicKey5431=== PAUSE TestExternalSignerRejectsInvalidPublicKey5432=== RUN TestExternalSignerCancellation5433=== PAUSE TestExternalSignerCancellation5434=== RUN TestGenerateFingerprint5435=== PAUSE TestGenerateFingerprint5436=== RUN TestParseSigningKey5437=== PAUSE TestParseSigningKey5438=== RUN TestSign5439=== PAUSE TestSign5440=== RUN TestSignBatch5441=== PAUSE TestSignBatch5442=== RUN TestNilKeyPublicKey5443=== PAUSE TestNilKeyPublicKey5444=== RUN TestSignErrors5445=== PAUSE TestSignErrors5446=== CONT TestExternalSigner5447=== CONT TestExternalSignerCancellation5448=== CONT TestExternalSignerProgramErrorKeepsProcess5449=== CONT TestExternalSignerRejectsInvalidPublicKey5450=== RUN TestExternalSignerRejectsInvalidPublicKey/#005451=== CONT TestSignErrors5452=== CONT TestNilKeyPublicKey5453--- PASS: TestNilKeyPublicKey (0.00s)5454=== CONT TestSignBatch5455=== CONT TestSign5456=== CONT TestParseSigningKey5457=== CONT TestGenerateFingerprint5458=== RUN TestGenerateFingerprint/basic_with_references5459=== CONT TestExternalSignerErrors5460=== RUN TestExternalSignerErrors/wrong_shape5461=== CONT TestExternalSignerRestartsAfterCrash5462=== PAUSE TestExternalSignerErrors/wrong_shape5463=== RUN TestExternalSignerErrors/too_few_signatures5464=== PAUSE TestExternalSignerErrors/too_few_signatures5465=== RUN TestExternalSignerErrors/missing_signatures5466=== PAUSE TestExternalSignerErrors/missing_signatures5467=== RUN TestExternalSignerErrors/unrequested_key5468=== PAUSE TestExternalSignerErrors/unrequested_key5469=== RUN TestExternalSignerErrors/invalid_signature_base645470=== PAUSE TestExternalSignerErrors/invalid_signature_base645471=== RUN TestExternalSignerErrors/wrong_signature_length5472=== PAUSE TestExternalSignerErrors/wrong_signature_length5473=== RUN TestExternalSignerErrors/exit_at_startup5474=== PAUSE TestExternalSignerErrors/exit_at_startup5475=== RUN TestExternalSignerErrors/wrong_id5476=== PAUSE TestExternalSignerErrors/wrong_id5477=== RUN TestExternalSignerErrors/null_signatures5478=== PAUSE TestExternalSignerErrors/null_signatures5479=== RUN TestExternalSignerErrors/missing_signature_name5480=== PAUSE TestExternalSignerErrors/missing_signature_name5481=== RUN TestExternalSignerErrors/wrong_signature_name5482=== PAUSE TestExternalSignerErrors/wrong_signature_name5483=== RUN TestExternalSignerErrors/program_error5484=== PAUSE TestExternalSignerErrors/program_error5485=== RUN TestExternalSignerErrors/exit_on_request5486=== PAUSE TestExternalSignerErrors/exit_on_request5487=== RUN TestExternalSignerErrors/invalid_JSON5488=== PAUSE TestExternalSignerErrors/invalid_JSON5489=== RUN TestExternalSignerErrors/null5490=== PAUSE TestExternalSignerErrors/null5491=== RUN TestExternalSignerErrors/wrong_signatures_shape5492=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5493=== CONT TestExternalSignerErrors/wrong_shape5494=== CONT TestExternalSignerErrors/null_signatures5495=== RUN TestSignErrors/nil_key5496=== PAUSE TestSignErrors/nil_key5497=== RUN TestSignErrors/nil_narinfo5498=== PAUSE TestSignErrors/nil_narinfo5499=== RUN TestSignErrors/invalid_narinfo5500=== CONT TestExternalSignerErrors/null5501=== CONT TestExternalSignerErrors/wrong_id5502=== PAUSE TestSignErrors/invalid_narinfo5503=== RUN TestSignErrors/canceled5504=== PAUSE TestSignErrors/canceled5505=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005506=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5507=== CONT TestExternalSignerWrongPublicKey5508=== CONT TestExternalSignerErrors/program_error5509=== PAUSE TestGenerateFingerprint/basic_with_references5510=== RUN TestGenerateFingerprint/no_references5511=== PAUSE TestGenerateFingerprint/no_references5512=== CONT TestExternalSignerErrors/exit_at_startup5513=== CONT TestExternalSignerErrors/wrong_signature_length5514=== CONT TestExternalSignerErrors/invalid_signature_base645515=== CONT TestExternalSignerErrors/unrequested_key5516=== CONT TestExternalSignerErrors/missing_signatures5517=== CONT TestExternalSignerErrors/too_few_signatures5518=== CONT TestExternalSignerErrors/exit_on_request5519=== CONT TestExternalSignerErrors/wrong_signatures_shape5520=== CONT TestExternalSignerClose5521=== CONT TestSignErrors/nil_key5522=== CONT TestExternalSignerErrors/invalid_JSON5523=== RUN TestParseSigningKey/valid_32-byte_key5524=== PAUSE TestParseSigningKey/valid_32-byte_key5525=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5526=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5527--- PASS: TestSignBatch (0.00s)5528=== CONT TestExternalSignerErrors/wrong_signature_name5529=== CONT TestExternalSignerErrors/missing_signature_name5530=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5531=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5532=== RUN TestParseSigningKey/no_colon5533=== PAUSE TestParseSigningKey/no_colon5534=== RUN TestParseSigningKey/empty_name5535=== PAUSE TestParseSigningKey/empty_name5536=== RUN TestParseSigningKey/invalid_base645537=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5538=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5539=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5540=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5541=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5542--- PASS: TestSign (0.00s)5543=== PAUSE TestParseSigningKey/invalid_base645544=== RUN TestParseSigningKey/wrong_length5545=== PAUSE TestParseSigningKey/wrong_length5546=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5547=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5548=== RUN TestGenerateFingerprint/invalid_nar_hash_length5549=== CONT TestSignErrors/invalid_narinfo5550--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5551=== CONT TestSignErrors/canceled5552=== CONT TestSignErrors/nil_narinfo5553=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5554=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5555=== CONT TestParseSigningKey/valid_32-byte_key5556--- PASS: TestSignErrors (0.00s)5557 --- PASS: TestSignErrors/nil_key (0.00s)5558 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5559 --- PASS: TestSignErrors/canceled (0.00s)5560 --- PASS: TestSignErrors/nil_narinfo (0.00s)5561=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5562=== RUN TestGenerateFingerprint/invalid_store_path_prefix5563=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5564=== RUN TestGenerateFingerprint/invalid_reference_prefix5565=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5566=== CONT TestParseSigningKey/wrong_length5567=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5568=== CONT TestParseSigningKey/invalid_base645569=== CONT TestParseSigningKey/empty_name5570=== CONT TestExternalSignerRejectsInvalidPublicKey/#005571=== CONT TestParseSigningKey/no_colon5572=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5573=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5574=== CONT TestGenerateFingerprint/basic_with_references5575=== CONT TestGenerateFingerprint/invalid_nar_hash_length5576=== CONT TestGenerateFingerprint/invalid_store_path_prefix5577--- PASS: TestParseSigningKey (0.00s)5578 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5579 --- PASS: TestParseSigningKey/wrong_length (0.00s)5580 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5581 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5582 --- PASS: TestParseSigningKey/empty_name (0.00s)5583 --- PASS: TestParseSigningKey/no_colon (0.00s)5584=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5585=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5586--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.01s)5587 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5588 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5589 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5590 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5591 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5592=== CONT TestGenerateFingerprint/invalid_reference_prefix5593=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5594=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5595=== CONT TestGenerateFingerprint/no_references5596--- PASS: TestGenerateFingerprint (0.01s)5597 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5598 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5599 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5600 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5601 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5602 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5603 --- PASS: TestGenerateFingerprint/no_references (0.00s)5604--- PASS: TestExternalSignerWrongPublicKey (0.01s)5605--- PASS: TestExternalSigner (0.02s)5606--- PASS: TestExternalSignerRestartsAfterCrash (0.02s)5607--- PASS: TestExternalSignerErrors (0.00s)5608 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5609 --- PASS: TestExternalSignerErrors/null (0.01s)5610 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5611 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5612 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5613 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5614 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5615 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5616 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5617 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5618 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5619 --- PASS: TestExternalSignerErrors/program_error (0.02s)5620 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5621 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5622 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5623 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5624--- PASS: TestExternalSignerClose (0.01s)5625--- PASS: TestExternalSignerCancellation (0.02s)5626PASS5627Running OpenBao signer tests...5628=== RUN TestSign5629=== PAUSE TestSign5630=== RUN TestSignErrors5631=== PAUSE TestSignErrors5632=== RUN TestServe5633=== PAUSE TestServe5634=== RUN TestServeRejectsGarbage5635=== PAUSE TestServeRejectsGarbage5636=== CONT TestSign5637=== CONT TestServeRejectsGarbage5638--- PASS: TestServeRejectsGarbage (0.00s)5639=== CONT TestServe5640--- PASS: TestServe (0.00s)5641=== CONT TestSignErrors5642=== RUN TestSignErrors/http_error5643=== PAUSE TestSignErrors/http_error5644=== RUN TestSignErrors/invalid_json5645=== PAUSE TestSignErrors/invalid_json5646=== RUN TestSignErrors/no_prefix5647=== PAUSE TestSignErrors/no_prefix5648=== RUN TestSignErrors/invalid_base645649=== PAUSE TestSignErrors/invalid_base645650=== CONT TestSignErrors/http_error5651=== CONT TestSignErrors/invalid_base645652=== CONT TestSignErrors/no_prefix5653=== CONT TestSignErrors/invalid_json5654--- PASS: TestSignErrors (0.00s)5655 --- PASS: TestSignErrors/http_error (0.83s)5656 --- PASS: TestSignErrors/invalid_base64 (0.83s)5657 --- PASS: TestSignErrors/invalid_json (0.83s)5658 --- PASS: TestSignErrors/no_prefix (0.83s)5659--- PASS: TestSign (0.84s)5660PASS