niks3-go-unit-tests log

succeeded `niks3-go-unit-tests` `checks.aarch64-linux.go-unit-tests` · build #497 · raw

1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (2.46s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestSetClientTLS119=== CONT TestRetryTimeoutOutlastsMaxRetries120=== CONT TestEncodeNixBase32121=== CONT TestScriptTokenNoExpiryRerunsEveryCall122=== CONT TestRegisterUploadedObject_OutlivesPush123=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer124=== CONT TestDoWithRetry_FinalResponseBodyReadable125=== CONT TestDumpPathWriterError126=== CONT TestStreamPushStopsWhenCanceled127--- PASS: TestStreamPushStopsWhenCanceled (0.00s)128=== CONT TestDumpPathMatchesNix129=== RUN TestEncodeNixBase32/test_string_hash130=== PAUSE TestEncodeNixBase32/test_string_hash131=== RUN TestEncodeNixBase32/empty_input132=== PAUSE TestEncodeNixBase32/empty_input133=== CONT TestStreamPushAnswersEveryLineWhenCanceled134=== CONT TestResolveStorePath135=== CONT TestPartSizeForNAR136=== RUN TestPartSizeForNAR/zero_stays_at_minimum137=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum138=== RUN TestPartSizeForNAR/small_stays_at_minimum139=== PAUSE TestPartSizeForNAR/small_stays_at_minimum140=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum141=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum142=== CONT TestClientSignaturesByStorePath143--- PASS: TestClientSignaturesByStorePath (0.00s)144=== CONT TestUploadMultipart_FailedProducerIsNotCompleted145=== CONT TestStreamPushReportsSignatures146=== CONT TestStreamPushRequestLine147=== CONT TestStreamPushGivesUpOnDeadServer148=== CONT TestStreamPushIsolatesFailures149=== CONT TestStreamPushBatchesUnderLoad150=== CONT TestStreamPushReportsEveryPath151=== CONT TestShellSplitErrors152=== CONT TestShellSplit153=== CONT TestDumpPathSingleFile154=== CONT TestDoWithRetry_BodyReplayedViaGetBody155=== CONT TestStreamPushReportsSkippedPaths156=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts157=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part1582026/11/12 00:22:32 ERROR Upload failed error="bad path" count=3159=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false160--- PASS: TestShellSplitErrors (0.00s)161--- PASS: TestShellSplit (0.00s)162=== CONT TestUploadMultipart_ReturnsAfterDroppedBody163=== CONT TestUploadMultipart_PartsInParallel164=== CONT TestFilterOversizedClosures165--- PASS: TestResolveStorePath (2.28s)1662026/11/12 00:22:32 ERROR Upload failed error="connection refused" count=201672026/11/12 00:22:32 ERROR Server seems unavailable, giving up on batch untried=17168--- PASS: TestStreamPushReportsSkippedPaths (0.00s)1692026/11/12 00:22:32 ERROR Upload failed error=boom count=1170--- PASS: TestStreamPushIsolatesFailures (0.01s)171=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun172=== RUN TestFilterOversizedClosures/no_limit_keeps_everything173=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything174=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped1752026/11/12 00:22:32 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:35683/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:35683/api/uploads/complete1762026/11/12 00:22:32 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"177--- PASS: TestStreamPushGivesUpOnDeadServer (0.01s)178=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica179=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts1802000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2181=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part182=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped1832026/11/12 00:22:32 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:35683/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:35683/api/uploads/complete1842026/11/12 00:22:32 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"185=== CONT TestCaseHackSuffix186--- PASS: TestStreamPushReportsEveryPath (0.01s)187--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (2.29s)188=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry1892026/11/12 00:22:32 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:35683/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:35683/api/uploads/complete190=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false1912026/11/12 00:22:32 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"192=== RUN TestFilterOversizedClosures/all_closures_skipped193=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary1942026/11/12 00:22:32 WARN Rate limiter enabled after throttle name=server-test rate=51952026/11/12 00:22:32 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:446151962026/11/12 00:22:32 ERROR Upload failed error=boom count=11972026/11/12 00:22:32 WARN Rate limiter enabled after throttle name=server-test rate=51982026/11/12 00:22:32 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:38225199=== CONT TestRegisterUploadedObjectReusesConnections2002026/11/12 00:22:32 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:382252012026/11/12 00:22:32 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:43631202--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (5.21s)203--- PASS: TestDoServerRequestAttachesToken (5.23s)204--- PASS: TestStreamPushReportsSignatures (0.04s)205--- PASS: TestRegisterUploadedObject_OutlivesPush (5.21s)206--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (5.22s)207=== CONT TestParsePathInfoJSON208=== RUN TestParsePathInfoJSON/Nix_format209=== PAUSE TestParsePathInfoJSON/Nix_format210=== RUN TestParsePathInfoJSON/Lix_format211=== PAUSE TestParsePathInfoJSON/Lix_format212=== RUN TestParsePathInfoJSON/empty_input213=== PAUSE TestParsePathInfoJSON/empty_input214=== RUN TestParsePathInfoJSON/whitespace_only215=== PAUSE TestParsePathInfoJSON/whitespace_only216=== RUN TestParsePathInfoJSON/invalid_JSON217=== PAUSE TestParsePathInfoJSON/invalid_JSON218=== CONT TestPathInfoHashCompatibility219=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)220=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)221=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon222=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon223=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI224=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI225=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512226=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512227=== CONT TestGetStorePathHash228=== RUN TestGetStorePathHash/valid_store_path229=== PAUSE TestGetStorePathHash/valid_store_path230=== RUN TestGetStorePathHash/basename_without_hyphen_should_error231=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error232=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error233=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error234=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error235=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error236=== CONT TestConvertHashToNix32237=== RUN TestConvertHashToNix32/SRI_format_to_Nix32238=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32239=== RUN TestConvertHashToNix32/already_Nix32_format240=== PAUSE TestConvertHashToNix32/already_Nix32_format241=== RUN TestConvertHashToNix32/invalid_format242=== PAUSE TestConvertHashToNix32/invalid_format243=== CONT TestEncodeNixBase32WithRealHash244--- PASS: TestEncodeNixBase32WithRealHash (0.00s)245=== CONT TestScriptTokenEmptyCommand246--- PASS: TestScriptTokenEmptyCommand (0.00s)247=== CONT TestScriptTokenScriptFails2482026/11/12 00:22:32 WARN Rate limiter backed off name=server-test rate=5249=== PAUSE TestFilterOversizedClosures/all_closures_skipped250--- PASS: TestDoWithRetry_FinalResponseBodyReadable (5.22s)2512026/11/12 00:22:32 INFO Garbage collection started252=== CONT TestScriptTokenBadJSON253=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess2542026/11/12 00:22:32 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:382252552026/11/12 00:22:32 WARN Rate limiter backed off name=server-test rate=52562026/11/12 00:22:32 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:446152572026/11/12 00:22:32 WARN Rate limiter enabled after throttle name=server-test rate=5258=== CONT TestScriptTokenEmptyToken259--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (2.06s)260=== CONT TestScriptTokenCachesUntilRefresh261=== CONT TestRateLimiterFeedback262=== RUN TestRateLimiterFeedback/429_enables_limiter263=== PAUSE TestRateLimiterFeedback/429_enables_limiter264=== RUN TestRateLimiterFeedback/503_enables_limiter265=== PAUSE TestRateLimiterFeedback/503_enables_limiter266=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter267=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter268=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter269=== CONT TestEncodeNixBase32/test_string_hash270=== CONT TestFileTokenEmpty271--- PASS: TestScriptTokenScriptFails (2.02s)272=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter273=== RUN TestPartSizeForNAR/1_TiB2742026/11/12 00:22:35 WARN Rate limiter enabled after throttle name=s3-test rate=5275=== CONT TestFileTokenReadsAndCaches276=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true277=== CONT TestParsePathInfoJSONMultiplePaths278=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary279=== CONT TestPathInfoCACompatibility280--- PASS: TestDumpPathSingleFile (2.94s)281=== CONT TestFileTokenMissing282=== PAUSE TestPartSizeForNAR/1_TiB283=== RUN TestPartSizeForNAR/5_TiB_S3_max_object284=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object285=== RUN TestPartSizeForNAR/capped_at_5_GiB286=== PAUSE TestPartSizeForNAR/capped_at_5_GiB287=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths288=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths289=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths290=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths291=== RUN TestPathInfoCACompatibility/null_ca_field292=== PAUSE TestPathInfoCACompatibility/null_ca_field293=== RUN TestPathInfoCACompatibility/old_string_format_-_text294=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text295=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive296=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive297=== RUN TestPathInfoCACompatibility/new_structured_format_-_text298=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text299=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method300=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method301=== CONT TestSetClientTLSErrors302--- PASS: TestFileTokenMissing (0.00s)303=== CONT TestParsePathInfoJSON/Nix_format304--- PASS: TestFileTokenEmpty (0.89s)305=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)306=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true307=== CONT TestStaticToken308=== CONT TestParsePathInfoJSON/whitespace_only3092026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:38225310=== CONT TestParsePathInfoJSON/empty_input311=== CONT TestParsePathInfoJSON/Lix_format312=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512313=== CONT TestConvertHashToNix32/SRI_format_to_Nix32314=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error315=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error316=== CONT TestGetStorePathHash/basename_without_hyphen_should_error317=== CONT TestConvertHashToNix32/invalid_format318=== CONT TestConvertHashToNix32/already_Nix32_format319=== CONT TestFilterOversizedClosures/no_limit_keeps_everything320=== CONT TestFilterOversizedClosures/all_closures_skipped3212026/11/12 00:22:35 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50322=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped323=== CONT TestRateLimiterFeedback/429_enables_limiter3242026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:38225325=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter326=== RUN TestSetClientTLSErrors/missing_cert_file327=== PAUSE TestSetClientTLSErrors/missing_cert_file328=== RUN TestSetClientTLSErrors/missing_key_file329=== PAUSE TestSetClientTLSErrors/missing_key_file330=== RUN TestSetClientTLSErrors/missing_ca_file331=== PAUSE TestSetClientTLSErrors/missing_ca_file332=== RUN TestSetClientTLSErrors/invalid_ca_file333=== PAUSE TestSetClientTLSErrors/invalid_ca_file334=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter335--- PASS: TestScriptTokenBadJSON (2.90s)336--- PASS: TestStaticToken (0.00s)337--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (2.95s)338=== CONT TestParsePathInfoJSON/invalid_JSON339=== CONT TestRateLimiterFeedback/503_enables_limiter340--- PASS: TestConvertHashToNix32 (0.00s)341 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)342 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)343 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)344--- PASS: TestFileTokenReadsAndCaches (0.00s)345--- PASS: TestParsePathInfoJSON (0.00s)346 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)347 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)348 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)349 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)350 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)351=== CONT TestSetClientTLSDoesNotMutateDefaultTransport3522026/11/12 00:22:35 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000353--- PASS: TestFilterOversizedClosures (0.04s)354 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)355 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)356 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)357=== CONT TestGetStorePathHash/valid_store_path358=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon359=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI360=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths361=== CONT TestPartSizeForNAR/5_TiB_S3_max_object3622026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:38225363=== CONT TestPartSizeForNAR/capped_at_5_GiB364=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts3652026/11/12 00:22:35 INFO Garbage collection started366=== CONT TestPartSizeForNAR/1_TiB3672026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:46327368=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary3692026/11/12 00:22:35 WARN Rate limiter enabled after throttle name=server-test rate=5370=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method3712026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:37495372=== RUN TestSetClientTLS/rejects_connection_without_client_cert373=== CONT TestPathInfoCACompatibility/new_structured_format_-_text374=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive375=== CONT TestPathInfoCACompatibility/old_string_format_-_text376=== CONT TestSetClientTLSErrors/missing_cert_file3772026/11/12 00:22:35 WARN Rate limiter enabled after throttle name=server-test rate=5378=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum3792026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:45197380=== CONT TestPartSizeForNAR/small_stays_at_minimum381=== CONT TestSetClientTLSErrors/invalid_ca_file3822026/11/12 00:22:35 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0383=== CONT TestSetClientTLSErrors/missing_ca_file3842026/11/12 00:22:35 WARN Rate limiter backed off name=server-test rate=5385=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part386=== CONT TestSetClientTLSErrors/missing_key_file387=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true388=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert389=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA390=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA391=== RUN TestSetClientTLS/preserves_debug_logging_transport392=== PAUSE TestSetClientTLS/preserves_debug_logging_transport393=== CONT TestSetClientTLS/rejects_connection_without_client_cert394=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false3952026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:38225396--- PASS: TestGetStorePathHash (0.00s)397 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)398 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)399 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)400 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)401--- PASS: TestPathInfoHashCompatibility (0.00s)402 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)403 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)404 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)405 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)406=== CONT TestSetClientTLS/preserves_debug_logging_transport407--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)408=== CONT TestEncodeNixBase32/empty_input409--- PASS: TestEncodeNixBase32 (2.90s)410 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)411 --- PASS: TestEncodeNixBase32/empty_input (0.00s)4122026/11/12 00:22:35 WARN Rate limiter backed off name=server-test rate=5413=== CONT TestPartSizeForNAR/zero_stays_at_minimum414=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA415=== CONT TestPathInfoCACompatibility/null_ca_field416=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths417--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)418 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)419 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)420--- PASS: TestScriptTokenEmptyToken (0.97s)421--- PASS: TestPathInfoCACompatibility (0.00s)422 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)423 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)424 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)425 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)426 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)427--- PASS: TestStreamPushBatchesUnderLoad (2.96s)428 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)429 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)430--- PASS: TestPartSizeForNAR (5.23s)431 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)432 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)433 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)434 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)435 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)436 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)437 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)438--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (2.95s)439--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (5.23s)440 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)441 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)442--- PASS: TestSetClientTLSErrors (0.00s)443 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)444 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)445 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)446 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)447--- PASS: TestCaseHackSuffix (2.95s)448--- PASS: TestRateLimiterFeedback (0.88s)449 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)450 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)451 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)452 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)453--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (2.95s)4542026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:382254552026/11/12 00:22:35 WARN Garbage collection finished on another replica. Its statistics are not available here456--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (2.95s)4572026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:38225458--- PASS: TestScriptTokenCachesUntilRefresh (0.90s)4592026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:382254602026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:382254612026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:382254622026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:382254632026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:382254642026/11/12 00:22:35 http: TLS handshake error from 127.0.0.1:37466: remote error: tls: bad certificate4652026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:38225466--- PASS: TestSetClientTLS (8.13s)467 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)468 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)469 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)4702026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:382254712026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:38225472--- PASS: TestStreamPushRequestLine (2.98s)4732026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:382254742026/11/12 00:22:35 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:38225475--- PASS: TestRegisterUploadedObjectReusesConnections (2.93s)476--- PASS: TestDumpPathWriterError (8.15s)477--- PASS: TestRetryTimeoutOutlastsMaxRetries (8.15s)478--- PASS: TestUploadMultipart_PartsInParallel (3.56s)479--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (3.72s)480--- PASS: TestDumpPathMatchesNix (9.12s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres364493440/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres364493440/data -l logfile start509510/build/postgres364493440:5432 - no response511/build/postgres364493440:5432 - no response512/build/postgres364493440:5432 - no response513/build/postgres364493440:5432 - no response514/build/postgres364493440:5432 - no response5152026-11-12 00:22:46.773 UTC [132] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5162026-11-12 00:22:46.774 UTC [132] LOG: listening on Unix socket "/build/postgres364493440/.s.PGSQL.5432"5172026-11-12 00:22:46.787 UTC [143] LOG: database system was shut down at 2026-11-12 00:22:45 UTC5182026-11-12 00:22:46.792 UTC [132] LOG: database system is ready to accept connections519/build/postgres364493440:5432 - accepting connections520{"timestamp":"2026-11-12T00:22:56.31002524Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c2a43569-b271-4814-9010-e374c82d3aa7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":1,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(208)"}521{"timestamp":"2026-11-12T00:22:56.419076269Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"4e898f07-1967-428c-a3d5-a72f854f6ca7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(200)"}522=== RUN TestAbortSucceeded523=== PAUSE TestAbortSucceeded524=== RUN TestService_AuthMiddleware525=== PAUSE TestService_AuthMiddleware526=== RUN TestService_AuthMiddleware_MTLSProxyHeader527=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader528=== RUN TestService_AuthMiddleware_MTLSBoundSubjects529=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects530=== RUN TestService_ReadAuthMiddleware531=== PAUSE TestService_ReadAuthMiddleware532=== RUN TestService_AuthMiddleware_OIDC533=== PAUSE TestService_AuthMiddleware_OIDC534=== RUN TestService_RequireScope_OIDC535=== PAUSE TestService_RequireScope_OIDC536=== RUN TestService_ReadScope_PublicByDefault537=== PAUSE TestService_ReadScope_PublicByDefault538=== RUN TestCacheConfigHandler539=== PAUSE TestCacheConfigHandler540=== RUN TestCacheStatsHandler541=== PAUSE TestCacheStatsHandler542=== RUN TestClientCADerivations543=== PAUSE TestClientCADerivations544=== RUN TestClientErrorHandling545=== PAUSE TestClientErrorHandling546=== RUN TestClientIntegration547=== PAUSE TestClientIntegration548=== RUN TestClientMultipleUploads549=== PAUSE TestClientMultipleUploads550=== RUN TestClientWithDependencies551=== PAUSE TestClientWithDependencies552=== RUN TestClientSharedPathCommittedMidPush553=== PAUSE TestClientSharedPathCommittedMidPush554=== RUN TestPinProtectsFromGC555=== PAUSE TestPinProtectsFromGC556=== RUN TestClientPushesUseOnePush557=== PAUSE TestClientPushesUseOnePush558=== RUN TestClientFallsBackToClosures559=== PAUSE TestClientFallsBackToClosures560=== RUN TestDeleteClosuresRacingPins561=== PAUSE TestDeleteClosuresRacingPins562=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock563=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock564=== RUN TestResolveDBConnectionString565=== PAUSE TestResolveDBConnectionString566=== RUN TestConnectWaitsForAPeerMigration567=== PAUSE TestConnectWaitsForAPeerMigration568=== RUN TestLeadElectsOneAndHandsOver569=== PAUSE TestLeadElectsOneAndHandsOver570=== RUN TestLeadIncumbentWinsAfterRestart5712026/11/12 00:22:56 INFO lead: acquired remote=192.0.2.1:1234 generation=15722026/11/12 00:22:57 INFO lead: released remote=192.0.2.1:1234 generation=15732026/11/12 00:22:57 INFO lead: acquired remote=192.0.2.1:1234 generation=25742026/11/12 00:22:58 INFO lead: released remote=192.0.2.1:1234 generation=2575--- PASS: TestLeadIncumbentWinsAfterRestart (2.15s)576=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent577=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent578=== RUN TestLeadIncumbentAnnouncesAtOnce579=== PAUSE TestLeadIncumbentAnnouncesAtOnce580=== RUN TestLeadGenerationGrowsWithEveryGrant581=== PAUSE TestLeadGenerationGrowsWithEveryGrant582=== RUN TestLeadEndsOnShutdown583=== PAUSE TestLeadEndsOnShutdown584=== RUN TestLeadEndsWhenItsConnectionHangs5852026/11/12 00:22:59 INFO lead: acquired remote=192.0.2.1:1234 generation=15862026-11-12 00:23:00.481 UTC [394] FATAL: terminating connection due to administrator command5872026/11/12 00:23:00 INFO lead: acquired remote=192.0.2.1:1234 generation=25882026/11/12 00:23:01 WARN lead: lock connection lost error="timeout: context deadline exceeded"5892026/11/12 00:23:01 INFO lead: released remote=192.0.2.1:1234 generation=15902026/11/12 00:23:02 INFO lead: released remote=192.0.2.1:1234 generation=2591--- PASS: TestLeadEndsWhenItsConnectionHangs (3.38s)592=== RUN TestGCAdvisoryLockBlocksConcurrentRun593=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun594=== RUN TestGCBugBareHashReferences595=== PAUSE TestGCBugBareHashReferences596=== RUN TestGCMetrics597=== PAUSE TestGCMetrics598=== RUN TestPushDedupSurvivesConcurrentGC599=== PAUSE TestPushDedupSurvivesConcurrentGC600=== RUN TestDeduplicatedObjectsRecordedAsPending601=== PAUSE TestDeduplicatedObjectsRecordedAsPending602=== RUN TestGCSweepSkipsPendingObjects603=== PAUSE TestGCSweepSkipsPendingObjects604=== RUN TestTombstonedObjectOfferedWithoutWaiting605=== PAUSE TestTombstonedObjectOfferedWithoutWaiting606=== RUN TestGCSweepDeliversEachKeyOnce607=== PAUSE TestGCSweepDeliversEachKeyOnce608=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection609=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection610=== RUN TestForceGCDuringPushOffersSweptObject611=== PAUSE TestForceGCDuringPushOffersSweptObject612=== RUN TestFailedS3DeleteKeepsTombstone613=== PAUSE TestFailedS3DeleteKeepsTombstone614=== RUN TestSweepSkipsRowsLockedByPush615=== PAUSE TestSweepSkipsRowsLockedByPush616=== RUN TestRegisterCompletedObjectNeedsPendingRow617=== PAUSE TestRegisterCompletedObjectNeedsPendingRow618=== RUN TestGetExistingObjectsLocksRows619=== PAUSE TestGetExistingObjectsLocksRows620=== RUN TestSweepRechecksPendingAfterLocking621=== PAUSE TestSweepRechecksPendingAfterLocking622=== RUN TestPushWaitsForSweepHoldingRow623=== PAUSE TestPushWaitsForSweepHoldingRow624=== RUN TestCommitRacingPendingCleanupKeepsObjects625=== PAUSE TestCommitRacingPendingCleanupKeepsObjects626=== RUN TestGCEndsOnShutdown627=== PAUSE TestGCEndsOnShutdown628=== RUN TestGCTaskStore_StartNew629=== PAUSE TestGCTaskStore_StartNew630=== RUN TestGCTaskStore_DeduplicateSameParams631=== PAUSE TestGCTaskStore_DeduplicateSameParams632=== RUN TestGCTaskStore_ConflictDifferentParams633=== PAUSE TestGCTaskStore_ConflictDifferentParams634=== RUN TestGCTaskStore_GetEmpty635=== PAUSE TestGCTaskStore_GetEmpty636=== RUN TestGCTaskStore_GetReturnsLatest637=== PAUSE TestGCTaskStore_GetReturnsLatest638=== RUN TestGCTaskStore_CompletedAllowsNewTask639=== PAUSE TestGCTaskStore_CompletedAllowsNewTask640=== RUN TestGCTaskStore_PhaseUpdates641=== PAUSE TestGCTaskStore_PhaseUpdates642=== RUN TestGCTaskStore_Fail643=== PAUSE TestGCTaskStore_Fail644=== RUN TestGracefulShutdownDrainsInflight645=== PAUSE TestGracefulShutdownDrainsInflight646=== RUN TestService_healthCheckHandler647=== PAUSE TestService_healthCheckHandler648=== RUN TestService_readinessHandler649=== PAUSE TestService_readinessHandler650=== RUN TestGenerateLandingPage651=== PAUSE TestGenerateLandingPage652=== RUN TestEnsureAbortUploadsRule653=== PAUSE TestEnsureAbortUploadsRule654=== RUN TestCacheConfigHandlerMaxNarSize655=== PAUSE TestCacheConfigHandlerMaxNarSize656=== RUN TestCreatePendingClosureRejectsOversizedNAR657=== PAUSE TestCreatePendingClosureRejectsOversizedNAR658=== RUN TestNARDeduplicationMetadataUploadBug659=== PAUSE TestNARDeduplicationMetadataUploadBug660=== RUN TestMetricsInventory661=== PAUSE TestMetricsInventory662=== RUN TestMetricsCommit663=== PAUSE TestMetricsCommit664=== RUN TestService_NativeMTLS665=== PAUSE TestService_NativeMTLS666=== RUN TestServerTLSConfig667=== PAUSE TestServerTLSConfig668=== RUN TestMultipartCleanup669=== PAUSE TestMultipartCleanup670=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded671=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded672=== RUN TestPendingCleanupUsesOneCutoff673=== PAUSE TestPendingCleanupUsesOneCutoff674=== RUN TestObjectStatsTrigger675=== PAUSE TestObjectStatsTrigger676=== RUN TestObjectStatsUpdatedOncePerCommit677=== PAUSE TestObjectStatsUpdatedOncePerCommit678=== RUN TestOrphanedObjectsGC679=== PAUSE TestOrphanedObjectsGC680=== RUN TestOrphanedObjectsGCStressTest681=== PAUSE TestOrphanedObjectsGCStressTest682=== RUN TestCreatePin_ReservedPins683=== PAUSE TestCreatePin_ReservedPins684=== RUN TestPresentReportsOnlyClosureRoots685=== PAUSE TestPresentReportsOnlyClosureRoots686=== RUN TestPresentRefreshesOnlyStaleClosures687=== PAUSE TestPresentRefreshesOnlyStaleClosures688=== RUN TestPresentNotReportedWhileGCDeletesClosure689=== PAUSE TestPresentNotReportedWhileGCDeletesClosure690=== RUN TestParseSingleRange691=== PAUSE TestParseSingleRange692=== RUN TestProxyHeadersOnlyTrustedOnSocket6932026/11/12 00:23:02 INFO Starting HTTP server address=proxy.sock6942026/11/12 00:23:02 INFO Starting HTTP server address=127.0.0.1:400816952026/11/12 00:23:02 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6962026/11/12 00:23:02 INFO Shutdown signal received, draining in-flight requests timeout=10s697--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.92s)698=== RUN TestIsValidCachePath699=== PAUSE TestIsValidCachePath700=== RUN TestReadProxyNarinfo701=== PAUSE TestReadProxyNarinfo702=== RUN TestReadProxyNarinfoAlreadyDecompressed703=== PAUSE TestReadProxyNarinfoAlreadyDecompressed704=== RUN TestReadProxyContentNegotiation705=== PAUSE TestReadProxyContentNegotiation706=== RUN TestReadProxyNarStreaming707=== PAUSE TestReadProxyNarStreaming708=== RUN TestReadProxy404709=== PAUSE TestReadProxy404710=== RUN TestReadProxyInvalidPath711=== PAUSE TestReadProxyInvalidPath712=== RUN TestReadProxyHead713=== PAUSE TestReadProxyHead714=== RUN TestReadProxyOutlastsServerWriteTimeout715=== PAUSE TestReadProxyOutlastsServerWriteTimeout716=== RUN TestReadProxyConditionalGet717=== PAUSE TestReadProxyConditionalGet718=== RUN TestReadProxyRootRedirectsToIndexHTML719=== PAUSE TestReadProxyRootRedirectsToIndexHTML720=== RUN TestReadProxyDisabled721=== PAUSE TestReadProxyDisabled722=== RUN TestReadRedirectNar723=== PAUSE TestReadRedirectNar724=== RUN TestReadRedirectKeepsNarinfoProxied725=== PAUSE TestReadRedirectKeepsNarinfoProxied726=== RUN TestReadProxyRangeRequest727=== PAUSE TestReadProxyRangeRequest728=== RUN TestReadRedirectUsesPublicS3URL729=== PAUSE TestReadRedirectUsesPublicS3URL730=== RUN TestPush_OverlappingRootsStoreOneRowPerKey731=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey732=== RUN TestPush_CompleteCommitsEveryRoot733=== PAUSE TestPush_CompleteCommitsEveryRoot734=== RUN TestPush_RepeatedRootCommitsOnce735=== PAUSE TestPush_RepeatedRootCommitsOnce736=== RUN TestPush_CompleteWaitsForCleanupOfThePush737=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush738=== RUN TestPush_RootsAreLockedInByteOrder739=== PAUSE TestPush_RootsAreLockedInByteOrder740=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit741=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit742=== RUN TestPush_RejectsBadRequests743=== PAUSE TestPush_RejectsBadRequests744=== RUN TestPush_SignsNarinfosOfItsPendingObjects745=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects746=== RUN TestSharedNarBothUploadsComplete747=== PAUSE TestSharedNarBothUploadsComplete748=== RUN TestCompleteMultipartUpload_ErrorButObjectExists749=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists750=== RUN TestCompletedNarNotReofferedAcrossClosures751=== PAUSE TestCompletedNarNotReofferedAcrossClosures752=== RUN TestPresignedUploadRegisteredBeforeCommit753=== PAUSE TestPresignedUploadRegisteredBeforeCommit754=== RUN TestService_Rustfstest755=== PAUSE TestService_Rustfstest756=== RUN TestParseSize757=== PAUSE TestParseSize758=== RUN TestSkippedUploadsHandler759=== PAUSE TestSkippedUploadsHandler760=== RUN TestSystemdListenerNotActivated761--- PASS: TestSystemdListenerNotActivated (0.00s)762=== RUN TestWatchdogBeatsWhenHealthy763--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)764=== RUN TestWatchdogSkipsWhenUnhealthy7652026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7702026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7712026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7722026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7732026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7742026/11/12 00:23:03 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"775--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)776=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle777=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle778=== RUN TestProxyWriteTimeout779=== PAUSE TestProxyWriteTimeout780=== RUN TestPendingClosureWriteTimeout781=== PAUSE TestPendingClosureWriteTimeout782=== RUN TestIsValidUploadKey783=== PAUSE TestIsValidUploadKey784=== RUN TestUploadHandlersRejectInvalidKeys785=== PAUSE TestUploadHandlersRejectInvalidKeys786=== RUN TestUploadHandlersRejectOversizedBody787=== PAUSE TestUploadHandlersRejectOversizedBody788=== RUN TestService_cleanupPendingClosuresHandler789=== PAUSE TestService_cleanupPendingClosuresHandler790=== RUN TestService_createPendingClosureHandler791=== PAUSE TestService_createPendingClosureHandler792=== RUN TestService_verifyS3Integrity793=== PAUSE TestService_verifyS3Integrity794=== RUN TestService_verifyS3ReuploadsStaleNarinfo795=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo796=== RUN TestCompleteMultipartUnregistered797=== PAUSE TestCompleteMultipartUnregistered798=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT799=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT800=== CONT TestPendingClosureWriteTimeout801=== CONT TestAbortSucceeded802=== CONT TestService_healthCheckHandler803=== CONT TestGracefulShutdownDrainsInflight8042026/11/12 00:23:03 INFO Starting HTTP server address=127.0.0.1:40589805=== CONT TestGCTaskStore_Fail806=== CONT TestSweepRechecksPendingAfterLocking807=== CONT TestGCTaskStore_PhaseUpdates808=== CONT TestPush_SignsNarinfosOfItsPendingObjects809=== CONT TestGCTaskStore_CompletedAllowsNewTask810=== CONT TestGetExistingObjectsLocksRows811=== CONT TestGCTaskStore_GetReturnsLatest812=== CONT TestPush_RejectsBadRequests813=== CONT TestService_readinessHandler814=== CONT TestGCTaskStore_GetEmpty8152026/11/12 00:23:03 INFO Shutdown signal received, draining in-flight requests timeout=10s816=== CONT TestProxyWriteTimeout817=== RUN TestProxyWriteTimeout/narinfo818=== PAUSE TestProxyWriteTimeout/narinfo819=== RUN TestProxyWriteTimeout/1_GiB_nar820=== PAUSE TestProxyWriteTimeout/1_GiB_nar821=== RUN TestProxyWriteTimeout/10_GiB_nar822=== PAUSE TestProxyWriteTimeout/10_GiB_nar823=== RUN TestProxyWriteTimeout/unknown_size824=== PAUSE TestProxyWriteTimeout/unknown_size825=== CONT TestGCTaskStore_ConflictDifferentParams826=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit827=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle828=== CONT TestGCTaskStore_DeduplicateSameParams829=== CONT TestPush_RootsAreLockedInByteOrder830=== CONT TestSkippedUploadsHandler831=== CONT TestRegisterCompletedObjectNeedsPendingRow8322026/11/12 00:23:03 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000833=== CONT TestGCTaskStore_StartNew834=== CONT TestFailedS3DeleteKeepsTombstone835=== CONT TestParseSize836=== CONT TestPush_CompleteWaitsForCleanupOfThePush837=== CONT TestGCEndsOnShutdown838=== CONT TestService_Rustfstest839=== RUN TestGCEndsOnShutdown/before_the_run840=== PAUSE TestGCEndsOnShutdown/before_the_run841=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete842=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete843=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark844=== CONT TestPresignedUploadRegisteredBeforeCommit845=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark846=== CONT TestForceGCDuringPushOffersSweptObject847=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows848=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows849=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check850=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check851=== CONT TestPush_RepeatedRootCommitsOnce852=== CONT TestCommitRacingPendingCleanupKeepsObjects853=== CONT TestCompletedNarNotReofferedAcrossClosures854=== CONT TestCompleteMultipartUpload_ErrorButObjectExists855=== CONT TestPushWaitsForSweepHoldingRow856=== RUN TestPendingClosureWriteTimeout/empty857--- PASS: TestAbortSucceeded (0.00s)858=== CONT TestSharedNarBothUploadsComplete859--- PASS: TestGCTaskStore_Fail (0.00s)860--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)861=== CONT TestSweepSkipsRowsLockedByPush862=== PAUSE TestPendingClosureWriteTimeout/empty863--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)864--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)865--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)866--- PASS: TestGCTaskStore_GetEmpty (0.00s)867--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)868--- PASS: TestGCTaskStore_StartNew (0.00s)869--- PASS: TestParseSize (0.00s)870=== RUN TestPendingClosureWriteTimeout/400_objects871=== PAUSE TestPendingClosureWriteTimeout/400_objects872=== RUN TestPendingClosureWriteTimeout/670k_objects873=== PAUSE TestPendingClosureWriteTimeout/670k_objects874=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline875=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline876=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection877--- PASS: TestSkippedUploadsHandler (1.35s)878=== CONT TestPush_CompleteCommitsEveryRoot879--- PASS: TestGracefulShutdownDrainsInflight (1.36s)880=== CONT TestGCSweepDeliversEachKeyOnce881--- PASS: TestService_healthCheckHandler (1.53s)882=== CONT TestPush_OverlappingRootsStoreOneRowPerKey8832026/11/12 00:23:04 INFO Received push request method=POST path=/api/pushes8842026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8852026/11/12 00:23:06 INFO Signed narinfos id=1 count=18862026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8872026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8882026/11/12 00:23:06 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"8892026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8902026/11/12 00:23:06 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"8912026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8922026/11/12 00:23:06 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=18932026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8942026/11/12 00:23:06 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo895--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (3.40s)896=== CONT TestTombstonedObjectOfferedWithoutWaiting8972026/11/12 00:23:06 WARN readiness check failed error="closed pool"898--- PASS: TestService_readinessHandler (3.42s)899=== CONT TestReadRedirectUsesPublicS3URL9002026/11/12 00:23:06 INFO Received uploads request method=POST path=/api/pending_closures9012026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes902--- PASS: TestRegisterCompletedObjectNeedsPendingRow (3.51s)903=== CONT TestGCSweepSkipsPendingObjects9042026/11/12 00:23:06 INFO Received uploads request method=POST path=/api/pending_closures9052026-11-12 00:23:06.764 UTC [502] ERROR: could not obtain lock on row in relation "closures"9062026-11-12 00:23:06.764 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9072026-11-12 00:23:06.765 UTC [502] ERROR: could not obtain lock on row in relation "closures"9082026-11-12 00:23:06.765 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9092026-11-12 00:23:06.765 UTC [502] ERROR: could not obtain lock on row in relation "closures"9102026-11-12 00:23:06.765 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9112026-11-12 00:23:06.765 UTC [502] ERROR: could not obtain lock on row in relation "closures"9122026-11-12 00:23:06.765 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9132026-11-12 00:23:06.765 UTC [502] ERROR: could not obtain lock on row in relation "closures"9142026-11-12 00:23:06.765 UTC [502] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9152026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes9162026/11/12 00:23:06 INFO Aborted multipart uploads count=0 kept=09172026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete918=== RUN TestPush_RejectsBadRequests/no_objects919=== PAUSE TestPush_RejectsBadRequests/no_objects920=== RUN TestPush_RejectsBadRequests/bad_root921=== PAUSE TestPush_RejectsBadRequests/bad_root922=== RUN TestPush_RejectsBadRequests/root_not_in_objects923=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects924=== RUN TestPush_RejectsBadRequests/no_roots925=== PAUSE TestPush_RejectsBadRequests/no_roots926=== CONT TestReadProxyRangeRequest9272026/11/12 00:23:06 INFO Aborted multipart uploads count=0 kept=09282026/11/12 00:23:06 INFO Received complete multipart upload request method=POST path=/api/multipart/complete929--- PASS: TestGetExistingObjectsLocksRows (3.68s)930=== CONT TestDeduplicatedObjectsRecordedAsPending9312026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes9322026/11/12 00:23:06 INFO Aborted multipart uploads count=0 kept=09332026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete9342026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures9352026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=0936--- PASS: TestService_Rustfstest (2.45s)937=== CONT TestReadRedirectKeepsNarinfoProxied9382026/11/12 00:23:07 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9392026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures9402026/11/12 00:23:07 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjllYTZhN2I3LTcwMTctNGExNC05NWM0LTFlZGY0NWQ0MDkwZngxNzk0NDQyOTg3MDEzNTY2MTk49412026/11/12 00:23:07 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjllYTZhN2I3LTcwMTctNGExNC05NWM0LTFlZGY0NWQ0MDkwZngxNzk0NDQyOTg3MDEzNTY2MTk4 parts=1942--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (2.49s)943=== CONT TestReadRedirectNar9442026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures9452026/11/12 00:23:07 INFO Received push request method=POST path=/api/pushes9462026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures9472026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=09482026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures9492026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures9502026/11/12 00:23:07 INFO Received push request method=POST path=/api/pushes951--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (2.52s)952=== CONT TestPushDedupSurvivesConcurrentGC9532026/11/12 00:23:07 INFO Received push request method=POST path=/api/pushes9542026/11/12 00:23:07 INFO Received complete push request method=POST path=/api/pushes/1/complete955--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (2.80s)956=== CONT TestReadProxyDisabled9572026-11-12 00:23:07.463 UTC [531] ERROR: Push does not exist: id=19582026-11-12 00:23:07.463 UTC [531] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE9592026-11-12 00:23:07.463 UTC [531] STATEMENT: SELECT commit_push($1)960--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (4.06s)961=== CONT TestGCMetrics962--- PASS: TestPushWaitsForSweepHoldingRow (2.98s)963=== CONT TestReadProxyRootRedirectsToIndexHTML9642026/11/12 00:23:07 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9652026/11/12 00:23:07 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLmQwMDk4ZGRhLTNlNjYtNGRlYy04YTg2LTU5ZmYzODI4N2NiNHgxNzk0NDQyOTg3MDkyMDgxNjQ0 parts=129662026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures967--- PASS: TestCompletedNarNotReofferedAcrossClosures (3.05s)968=== CONT TestGCBugBareHashReferences9692026/11/12 00:23:07 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9702026/11/12 00:23:07 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLmI4Njk3NjM3LTUwZmUtNDBiNy1iMDg2LTMxMGU2MmNlMzk4YngxNzk0NDQyOTg3MjAyNzg2ODg5 parts=129712026/11/12 00:23:08 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9722026/11/12 00:23:08 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjgxNzRhNTc0LWQ4YmUtNDdjNi1iMGVlLWQ5MmYyYTY2ZDA0NXgxNzk0NDQyOTg3MjEwMDIxMzU2 parts=12973--- PASS: TestSharedNarBothUploadsComplete (3.72s)974=== CONT TestReadProxyConditionalGet9752026/11/12 00:23:08 INFO Completed push id=1 objects=6 duration=1.410915275s976--- PASS: TestPush_CompleteCommitsEveryRoot (4.13s)977=== CONT TestReadProxyOutlastsServerWriteTimeout9782026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period9792026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period9802026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period9812026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period9822026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period9832026/11/12 00:23:08 INFO Completed push id=1 objects=4 duration=1.762289406s984--- PASS: TestPush_RepeatedRootCommitsOnce (4.15s)985=== CONT TestGCAdvisoryLockBlocksConcurrentRun9862026/11/12 00:23:08 INFO Completed push id=1 objects=4 duration=1.911628748s9872026/11/12 00:23:08 INFO Received push request method=POST path=/api/pushes9882026/11/12 00:23:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09892026/11/12 00:23:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09902026/11/12 00:23:08 INFO Vacuumed table table=pending_closures9912026/11/12 00:23:08 INFO Vacuumed table table=pending_objects9922026/11/12 00:23:08 INFO Vacuumed table table=multipart_uploads9932026/11/12 00:23:08 INFO Vacuumed table table=closures9942026/11/12 00:23:08 INFO Vacuumed table table=pending_closures9952026/11/12 00:23:08 INFO Vacuumed table table=pending_objects9962026/11/12 00:23:08 INFO Vacuumed table table=objects9972026/11/12 00:23:08 INFO Vacuumed table table=multipart_uploads9982026/11/12 00:23:08 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."9992026/11/12 00:23:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."10002026/11/12 00:23:08 INFO Vacuumed table table=closures10012026/11/12 00:23:08 INFO Vacuumed table table=objects1002--- PASS: TestFailedS3DeleteKeepsTombstone (5.52s)1003=== CONT TestReadProxyHead10042026/11/12 00:23:08 INFO Aborted multipart uploads count=0 kept=010052026/11/12 00:23:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01006--- PASS: TestCommitRacingPendingCleanupKeepsObjects (4.19s)1007=== CONT TestReadProxyInvalidPath10082026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period10092026/11/12 00:23:08 INFO Vacuumed table table=pending_closures10102026/11/12 00:23:08 INFO Vacuumed table table=pending_objects10112026/11/12 00:23:08 INFO Vacuumed table table=multipart_uploads10122026/11/12 00:23:08 INFO Vacuumed table table=closures10132026/11/12 00:23:08 INFO Vacuumed table table=objects1014--- PASS: TestSweepRechecksPendingAfterLocking (5.55s)1015=== CONT TestLeadEndsOnShutdown10162026/11/12 00:23:08 INFO Aborted multipart uploads count=0 kept=010172026/11/12 00:23:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=010182026/11/12 00:23:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=010192026/11/12 00:23:08 INFO Vacuumed table table=pending_closures10202026/11/12 00:23:08 INFO Vacuumed table table=pending_objects10212026/11/12 00:23:08 INFO Vacuumed table table=multipart_uploads10222026/11/12 00:23:08 INFO Vacuumed table table=closures10232026/11/12 00:23:08 INFO Vacuumed table table=pending_closures10242026/11/12 00:23:08 INFO Vacuumed table table=pending_objects10252026/11/12 00:23:08 INFO Vacuumed table table=multipart_uploads10262026/11/12 00:23:08 INFO Vacuumed table table=objects10272026/11/12 00:23:08 INFO Vacuumed table table=closures10282026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period10292026/11/12 00:23:08 INFO Vacuumed table table=objects1030--- PASS: TestGCSweepDeliversEachKeyOnce (4.23s)1031=== CONT TestReadProxy4041032--- PASS: TestSweepSkipsRowsLockedByPush (4.23s)1033=== CONT TestLeadGenerationGrowsWithEveryGrant1034--- PASS: TestReadRedirectKeepsNarinfoProxied (1.80s)1035=== CONT TestReadProxyNarStreaming10362026/11/12 00:23:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=010372026/11/12 00:23:08 INFO Vacuumed table table=pending_closures10382026/11/12 00:23:08 INFO Vacuumed table table=pending_objects10392026/11/12 00:23:08 INFO Vacuumed table table=multipart_uploads10402026/11/12 00:23:08 INFO Vacuumed table table=closures10412026/11/12 00:23:08 INFO Vacuumed table table=objects10422026/11/12 00:23:08 INFO Received complete push request method=POST path=/api/pushes/2/complete10432026/11/12 00:23:08 INFO Completed push id=2 objects=4 duration=27.941458ms1044--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (5.68s)1045=== CONT TestReadProxyContentNegotiation10462026/11/12 00:23:08 INFO Received uploads request method=POST path=/api/pending_closures10472026/11/12 00:23:08 INFO Received uploads request method=POST path=/api/pending_closures1048--- PASS: TestDeduplicatedObjectsRecordedAsPending (2.03s)1049=== CONT TestLeadIncumbentAnnouncesAtOnce1050--- PASS: TestReadProxyConditionalGet (0.64s)1051=== CONT TestReadProxyNarinfoAlreadyDecompressed1052--- PASS: TestReadRedirectUsesPublicS3URL (2.34s)1053=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent10542026/11/12 00:23:09 INFO Received uploads request method=POST path=/api/pending_closures10552026/11/12 00:23:09 INFO Aborted multipart uploads count=0 kept=01056--- PASS: TestTombstonedObjectOfferedWithoutWaiting (2.38s)1057=== CONT TestReadProxyNarinfo1058--- PASS: TestReadProxyRangeRequest (2.23s)1059=== CONT TestLeadElectsOneAndHandsOver10602026/11/12 00:23:09 INFO Received uploads request method=POST path=/api/pending_closures1061--- PASS: TestReadRedirectNar (2.02s)1062=== CONT TestIsValidCachePath1063=== RUN TestIsValidCachePath/narinfo1064=== PAUSE TestIsValidCachePath/narinfo1065=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1066=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1067=== RUN TestIsValidCachePath/nar_zst1068=== PAUSE TestIsValidCachePath/nar_zst1069=== RUN TestIsValidCachePath/nar_xz1070=== PAUSE TestIsValidCachePath/nar_xz1071=== RUN TestIsValidCachePath/nar_bz21072=== PAUSE TestIsValidCachePath/nar_bz21073=== RUN TestIsValidCachePath/nar_uncompressed1074=== PAUSE TestIsValidCachePath/nar_uncompressed1075=== RUN TestIsValidCachePath/ls1076=== PAUSE TestIsValidCachePath/ls1077=== RUN TestIsValidCachePath/log1078=== PAUSE TestIsValidCachePath/log1079=== RUN TestIsValidCachePath/realisation1080=== PAUSE TestIsValidCachePath/realisation1081=== RUN TestIsValidCachePath/nix-cache-info1082=== PAUSE TestIsValidCachePath/nix-cache-info1083=== RUN TestIsValidCachePath/index.html1084=== PAUSE TestIsValidCachePath/index.html1085=== RUN TestIsValidCachePath/traversal_parent1086=== PAUSE TestIsValidCachePath/traversal_parent1087=== RUN TestIsValidCachePath/traversal_in_middle1088=== PAUSE TestIsValidCachePath/traversal_in_middle1089=== RUN TestIsValidCachePath/invalid_char_e1090=== PAUSE TestIsValidCachePath/invalid_char_e1091=== RUN TestIsValidCachePath/invalid_char_u1092=== PAUSE TestIsValidCachePath/invalid_char_u1093=== RUN TestIsValidCachePath/random_path1094=== PAUSE TestIsValidCachePath/random_path1095=== RUN TestIsValidCachePath/empty1096=== PAUSE TestIsValidCachePath/empty1097=== RUN TestIsValidCachePath/leading_slash1098=== PAUSE TestIsValidCachePath/leading_slash1099=== RUN TestIsValidCachePath/wrong_extension1100=== PAUSE TestIsValidCachePath/wrong_extension1101=== RUN TestIsValidCachePath/short_hash1102=== PAUSE TestIsValidCachePath/short_hash1103=== CONT TestConnectWaitsForAPeerMigration11042026/11/12 00:23:09 INFO Aborted multipart uploads count=0 kept=011052026/11/12 00:23:10 WARN Rate limiter enabled after throttle name=s3-test rate=511062026/11/12 00:23:10 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1107=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1108 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101109 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001110--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (7.45s)1111=== CONT TestParseSingleRange1112=== RUN TestParseSingleRange/none1113=== PAUSE TestParseSingleRange/none1114=== RUN TestParseSingleRange/unknown_unit1115=== PAUSE TestParseSingleRange/unknown_unit1116=== RUN TestParseSingleRange/multi-range_ignored1117=== PAUSE TestParseSingleRange/multi-range_ignored1118=== RUN TestParseSingleRange/malformed_no_dash1119=== PAUSE TestParseSingleRange/malformed_no_dash1120=== RUN TestParseSingleRange/malformed_both_empty1121=== PAUSE TestParseSingleRange/malformed_both_empty1122=== RUN TestParseSingleRange/malformed_end_before_start1123=== PAUSE TestParseSingleRange/malformed_end_before_start1124=== RUN TestParseSingleRange/closed1125=== PAUSE TestParseSingleRange/closed1126=== RUN TestParseSingleRange/open-ended1127=== PAUSE TestParseSingleRange/open-ended1128=== RUN TestParseSingleRange/end_clamped_to_size1129=== PAUSE TestParseSingleRange/end_clamped_to_size1130=== RUN TestParseSingleRange/suffix1131=== PAUSE TestParseSingleRange/suffix1132=== RUN TestParseSingleRange/suffix_exceeds_size1133=== PAUSE TestParseSingleRange/suffix_exceeds_size1134=== RUN TestParseSingleRange/single_byte1135=== PAUSE TestParseSingleRange/single_byte1136=== RUN TestParseSingleRange/start_past_EOF1137=== PAUSE TestParseSingleRange/start_past_EOF1138=== RUN TestParseSingleRange/start_far_past_EOF1139=== PAUSE TestParseSingleRange/start_far_past_EOF1140=== CONT TestService_createPendingClosureHandler1141=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT1142--- PASS: TestPush_RootsAreLockedInByteOrder (7.84s)11432026/11/12 00:23:11 INFO lead: acquired remote=192.0.2.1:1234 generation=11144--- PASS: TestReadProxyDisabled (4.12s)1145=== CONT TestCompleteMultipartUnregistered11462026/11/12 00:23:11 INFO Aborted multipart uploads count=0 kept=01147--- PASS: TestReadProxyRootRedirectsToIndexHTML (3.99s)1148=== CONT TestService_verifyS3ReuploadsStaleNarinfo1149=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1150=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1151=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1152=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1153=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1154=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1155=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1156=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1157=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1158=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1159=== CONT TestService_verifyS3Integrity11602026/11/12 00:23:11 INFO Received uploads request method=POST path=/api/pending_closures11612026/11/12 00:23:11 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst11622026/11/12 00:23:11 INFO Received uploads request method=POST path=/api/pending_closures1163--- PASS: TestPresignedUploadRegisteredBeforeCommit (7.03s)1164=== CONT TestResolveDBConnectionString1165=== RUN TestResolveDBConnectionString/flag_wins1166=== PAUSE TestResolveDBConnectionString/flag_wins1167=== RUN TestResolveDBConnectionString/file_when_flag_empty1168=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1169=== RUN TestResolveDBConnectionString/missing_file_is_an_error1170=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1171=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1172=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1173=== RUN TestResolveDBConnectionString/nothing_configured1174=== PAUSE TestResolveDBConnectionString/nothing_configured1175=== CONT TestMultipartCleanup11762026/11/12 00:23:12 INFO lead: released remote=192.0.2.1:1234 generation=11177--- PASS: TestGCBugBareHashReferences (4.83s)1178=== CONT TestPresentRefreshesOnlyStaleClosures11792026/11/12 00:23:12 WARN Force mode enabled - objects will be deleted immediately without grace period11802026/11/12 00:23:12 WARN Force mode enabled - objects will be deleted immediately without grace period11812026/11/12 00:23:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011822026/11/12 00:23:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011832026/11/12 00:23:12 INFO Vacuumed table table=pending_closures11842026/11/12 00:23:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011852026/11/12 00:23:12 INFO Vacuumed table table=pending_closures11862026/11/12 00:23:12 INFO Vacuumed table table=pending_objects11872026/11/12 00:23:12 INFO Vacuumed table table=multipart_uploads11882026/11/12 00:23:12 INFO Vacuumed table table=closures11892026/11/12 00:23:12 INFO Vacuumed table table=objects11902026/11/12 00:23:12 INFO Vacuumed table table=pending_objects11912026/11/12 00:23:12 INFO Vacuumed table table=multipart_uploads11922026/11/12 00:23:12 INFO Vacuumed table table=closures11932026/11/12 00:23:12 INFO Vacuumed table table=pending_closures11942026/11/12 00:23:12 INFO Vacuumed table table=objects1195--- PASS: TestGCMetrics (5.03s)1196=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock11972026/11/12 00:23:12 INFO Vacuumed table table=pending_objects11982026/11/12 00:23:12 INFO lead: acquired remote=192.0.2.1:1234 generation=211992026/11/12 00:23:12 INFO Vacuumed table table=multipart_uploads12002026/11/12 00:23:12 INFO Vacuumed table table=closures12012026/11/12 00:23:12 INFO Vacuumed table table=objects1202--- PASS: TestGCSweepSkipsPendingObjects (5.77s)1203=== CONT TestPresentReportsOnlyClosureRoots12042026/11/12 00:23:12 INFO Aborted multipart uploads count=0 kept=012052026/11/12 00:23:12 WARN Force mode enabled - objects will be deleted immediately without grace period12062026/11/12 00:23:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012072026/11/12 00:23:12 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=212082026/11/12 00:23:12 INFO Vacuumed table table=pending_closures12092026/11/12 00:23:12 INFO lead: acquired remote=192.0.2.1:1234 generation=312102026/11/12 00:23:12 INFO Vacuumed table table=pending_objects12112026/11/12 00:23:12 INFO Vacuumed table table=multipart_uploads12122026/11/12 00:23:12 INFO Vacuumed table table=closures12132026/11/12 00:23:12 INFO Vacuumed table table=objects1214--- PASS: TestPushDedupSurvivesConcurrentGC (5.30s)1215=== CONT TestDeleteClosuresRacingPins12162026/11/12 00:23:12 INFO Received uploads request method=POST path=/api/pending_closures12172026/11/12 00:23:12 INFO Received uploads request method=POST path=/api/pending_closures12182026/11/12 00:23:12 INFO Received uploads request method=POST path=/api/pending_closures12192026/11/12 00:23:12 INFO lead: acquired remote=192.0.2.1:1234 generation=112202026/11/12 00:23:12 INFO Received uploads request method=POST path=/api/pending_closures1221--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (1.62s)1222=== CONT TestCreatePin_ReservedPins12232026/11/12 00:23:12 INFO Received uploads request method=POST path=/api/pending_closures12242026/11/12 00:23:12 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1225--- PASS: TestReadProxyNarinfo (3.74s)1226=== CONT TestPresentNotReportedWhileGCDeletesClosure1227=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1228=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1229=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1230=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1231=== CONT TestClientFallsBackToClosures1232--- PASS: TestReadProxyHead (4.00s)1233=== CONT TestOrphanedObjectsGCStressTest1234--- PASS: TestReadProxyNarStreaming (3.98s)1235=== CONT TestServerTLSConfig1236=== RUN TestServerTLSConfig/no_client_CA1237=== PAUSE TestServerTLSConfig/no_client_CA1238=== RUN TestServerTLSConfig/missing_CA_file1239=== PAUSE TestServerTLSConfig/missing_CA_file1240=== RUN TestServerTLSConfig/not_a_PEM_file1241=== PAUSE TestServerTLSConfig/not_a_PEM_file1242=== CONT TestClientPushesUseOnePush12432026/11/12 00:23:12 INFO Received cleanup request method=DELETE path=/api/pending_closures12442026/11/12 00:23:12 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket47/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""12452026/11/12 00:23:12 INFO Aborted multipart uploads count=0 kept=112462026/11/12 00:23:12 INFO Received cleanup request method=DELETE path=/api/pending_closures12472026/11/12 00:23:12 INFO Aborted multipart uploads count=1 kept=01248--- PASS: TestMultipartCleanup (1.25s)1249=== CONT TestOrphanedObjectsGC12502026/11/12 00:23:12 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40293/oidc12512026/11/12 00:23:12 INFO lead: acquired remote=192.0.2.1:1234 generation=112522026/11/12 00:23:12 INFO lead: released remote=192.0.2.1:1234 generation=11253--- PASS: TestLeadIncumbentAnnouncesAtOnce (4.01s)1254=== CONT TestService_NativeMTLS1255--- PASS: TestReadProxyNarinfoAlreadyDecompressed (4.03s)1256=== CONT TestPinProtectsFromGC1257--- PASS: TestReadProxy404 (4.15s)1258=== CONT TestObjectStatsUpdatedOncePerCommit12592026/11/12 00:23:13 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12602026/11/12 00:23:13 INFO lead: acquired remote=192.0.2.1:1234 generation=112612026/11/12 00:23:13 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLmVlMDM5NDFlLTNmM2QtNDE5Ny1iYmZlLTRmOWI1NmYwMGMwZngxNzk0NDQyOTkyNjE1OTQzNTAx parts=1012622026/11/12 00:23:13 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12632026/11/12 00:23:13 INFO Completed upload id=1 objects=2 duration=3.532612ms12642026/11/12 00:23:13 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000012652026/11/12 00:23:13 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1266--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (4.34s)1267=== CONT TestCacheConfigHandlerMaxNarSize12682026/11/12 00:23:13 INFO Received uploads request method=POST path=/api/pending_closures1269--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1270=== CONT TestClientSharedPathCommittedMidPush12712026/11/12 00:23:13 INFO Starting cleanup of old closures method=DELETE path=/api/closures1272--- PASS: TestReadProxyContentNegotiation (4.17s)1273=== CONT TestObjectStatsTrigger12742026/11/12 00:23:13 INFO Aborted multipart uploads count=0 kept=012752026/11/12 00:23:13 INFO lead: acquired remote=192.0.2.1:1234 generation=11276--- PASS: TestReadProxyInvalidPath (4.34s)1277=== CONT TestEnsureAbortUploadsRule1278--- PASS: TestPresentReportsOnlyClosureRoots (0.65s)1279=== CONT TestClientWithDependencies12802026/11/12 00:23:13 INFO Received uploads request method=POST path=/api/pending_closures12812026/11/12 00:23:13 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12822026/11/12 00:23:13 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1283--- PASS: TestCompleteMultipartUnregistered (1.72s)1284=== CONT TestPendingCleanupUsesOneCutoff1285--- PASS: TestPresentRefreshesOnlyStaleClosures (0.83s)1286=== CONT TestGenerateLandingPage1287--- PASS: TestGenerateLandingPage (0.01s)1288=== CONT TestClientMultipleUploads12892026/11/12 00:23:13 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12902026/11/12 00:23:13 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjE0YzFhMzA3LTUwZTEtNDU0NC05ZDQ5LTZkZjRmMjU3NGFmNngxNzk0NDQyOTkzMjAzNzk4NjQy parts=1012912026/11/12 00:23:13 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12922026/11/12 00:23:13 INFO Completed upload id=1 objects=2 duration=3.506792ms12932026/11/12 00:23:13 INFO Received uploads request method=POST path=/api/pending_closures12942026/11/12 00:23:13 INFO Received uploads request method=POST path=/api/pending_closures12952026/11/12 00:23:13 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo12962026/11/12 00:23:13 WARN Found objects in DB but missing from S3, will re-upload count=11297--- PASS: TestService_verifyS3Integrity (2.10s)1298=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded12992026/11/12 00:23:13 INFO lead: released remote=192.0.2.1:1234 generation=113002026/11/12 00:23:13 INFO lead: acquired remote=192.0.2.1:1234 generation=213012026/11/12 00:23:14 INFO lead: released remote=192.0.2.1:1234 generation=113022026/11/12 00:23:14 INFO lead: acquired remote=192.0.2.1:1234 generation=213032026/11/12 00:23:14 INFO lead: released remote=192.0.2.1:1234 generation=11304--- PASS: TestLeadEndsOnShutdown (5.41s)1305=== CONT TestClientIntegration13062026/11/12 00:23:14 INFO lead: released remote=192.0.2.1:1234 generation=313072026/11/12 00:23:14 ERROR lead error="context canceled"1308--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (5.51s)1309=== CONT TestUploadHandlersRejectOversizedBody1310=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1311=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1312=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1313=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1314=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1315=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1316=== CONT TestService_cleanupPendingClosuresHandler13172026/11/12 00:23:15 INFO lead: released remote=192.0.2.1:1234 generation=21318--- PASS: TestLeadElectsOneAndHandsOver (5.98s)1319=== CONT TestClientErrorHandling1320=== RUN TestClientErrorHandling/InvalidStorePath1321=== PAUSE TestClientErrorHandling/InvalidStorePath1322=== RUN TestClientErrorHandling/InvalidAuthToken1323=== PAUSE TestClientErrorHandling/InvalidAuthToken1324=== RUN TestClientErrorHandling/ServerNotAvailable1325=== PAUSE TestClientErrorHandling/ServerNotAvailable1326=== CONT TestMetricsInventory13272026/11/12 00:23:15 INFO Received uploads request method=POST path=/api/pending_closures1328--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (1.62s)1329=== CONT TestMetricsCommit13302026/11/12 00:23:15 INFO lead: released remote=192.0.2.1:1234 generation=213312026/11/12 00:23:15 INFO lead: acquired remote=192.0.2.1:1234 generation=313322026/11/12 00:23:15 INFO lead: released remote=192.0.2.1:1234 generation=31333--- PASS: TestLeadGenerationGrowsWithEveryGrant (6.48s)1334=== CONT TestClientCADerivations1335--- PASS: TestConnectWaitsForAPeerMigration (6.24s)1336=== CONT TestService_AuthMiddleware_OIDC13372026/11/12 00:23:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43475/oidc1338--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (2.94s)1339=== CONT TestNARDeduplicationMetadataUploadBug13402026/11/12 00:23:15 INFO Received cleanup request method=DELETE path=/api/pending_closures13412026/11/12 00:23:15 INFO Aborted multipart uploads count=0 kept=013422026/11/12 00:23:15 INFO Received uploads request method=POST path=/api/pending_closures13432026/11/12 00:23:15 INFO Received cleanup request method=DELETE path=/api/pending_closures13442026/11/12 00:23:15 INFO Aborted multipart uploads count=1 kept=013452026/11/12 00:23:15 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13462026-11-12 00:23:15.490 UTC [719] ERROR: Closure does not exist: id=113472026-11-12 00:23:15.490 UTC [719] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE13482026-11-12 00:23:15.490 UTC [719] STATEMENT: -- name: CommitPendingClosure :one1349 SELECT commit_pending_closure($1::bigint)::bigint1350 1351--- PASS: TestService_cleanupPendingClosuresHandler (0.80s)1352=== CONT TestService_ReadAuthMiddleware13532026/11/12 00:23:15 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket72 days=21354--- PASS: TestEnsureAbortUploadsRule (2.52s)1355=== CONT TestCreatePendingClosureRejectsOversizedNAR13562026/11/12 00:23:15 INFO Received uploads request method=POST path=/api/pending_closures1357--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1358=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1359--- PASS: TestObjectStatsTrigger (2.56s)1360=== CONT TestService_AuthMiddleware_MTLSProxyHeader13612026/11/12 00:23:15 WARN mTLS auth: subject not in bound subjects subject="CN=reader"13622026/11/12 00:23:15 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1363--- PASS: TestService_NativeMTLS (2.71s)1364=== CONT TestService_ReadScope_PublicByDefault1365--- PASS: TestMetricsInventory (0.67s)1366=== CONT TestService_AuthMiddleware13672026/11/12 00:23:15 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux13682026/11/12 00:23:15 WARN Refused reserved pin name=worker-x86_64-linux13692026/11/12 00:23:15 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux13702026/11/12 00:23:15 INFO Received create pin request method=POST path=/api/pins/my-app13712026/11/12 00:23:15 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1372--- PASS: TestCreatePin_ReservedPins (3.04s)1373=== CONT TestCacheStatsHandler1374--- PASS: TestObjectStatsUpdatedOncePerCommit (2.84s)1375=== CONT TestCacheConfigHandler1376=== RUN TestCacheConfigHandler/full_config,_no_issuer1377=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1378=== RUN TestCacheConfigHandler/no_cache_url_configured1379=== PAUSE TestCacheConfigHandler/no_cache_url_configured1380=== RUN TestCacheConfigHandler/no_signing_keys1381=== PAUSE TestCacheConfigHandler/no_signing_keys1382=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1383=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1384=== CONT TestService_RequireScope_OIDC13852026/11/12 00:23:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42759/oidc1386=== NAME TestOrphanedObjectsGCStressTest1387 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains13882026/11/12 00:23:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013892026/11/12 00:23:16 INFO Vacuumed table table=pending_closures1390 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion1391=== NAME TestClientMultipleUploads1392 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1442135502/001/store/x6mwby4xa49csjj2axxiwhv8b1xvfmy4-test-file-0.txt13932026/11/12 00:23:16 INFO Vacuumed table table=pending_objects1394=== NAME TestClientIntegration1395 client_integration_test.go:292: Created store path: /build/TestClientIntegration3731239629/002/store/hwdmjh9kgzw4y9d29cp08xnpla1gx8bp-test-file.txt13962026/11/12 00:23:16 INFO Vacuumed table table=multipart_uploads13972026/11/12 00:23:16 INFO Vacuumed table table=closures13982026/11/12 00:23:16 INFO Vacuumed table table=objects13992026/11/12 00:23:16 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001400--- PASS: TestService_createPendingClosureHandler (6.24s)1401=== CONT TestIsValidUploadKey1402=== RUN TestIsValidUploadKey/narinfo1403=== PAUSE TestIsValidUploadKey/narinfo1404=== RUN TestIsValidUploadKey/nar_zst1405=== PAUSE TestIsValidUploadKey/nar_zst1406=== RUN TestIsValidUploadKey/nar_xz1407=== PAUSE TestIsValidUploadKey/nar_xz1408=== RUN TestIsValidUploadKey/nar_plain1409=== PAUSE TestIsValidUploadKey/nar_plain1410=== RUN TestIsValidUploadKey/listing1411=== PAUSE TestIsValidUploadKey/listing1412=== RUN TestIsValidUploadKey/build_log1413=== PAUSE TestIsValidUploadKey/build_log1414=== RUN TestIsValidUploadKey/build_log_home-manager_file1415=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1416=== RUN TestIsValidUploadKey/build_log_plus_in_name1417=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1418=== RUN TestIsValidUploadKey/build_log_question_mark1419=== PAUSE TestIsValidUploadKey/build_log_question_mark1420=== RUN TestIsValidUploadKey/build_log_equals1421=== PAUSE TestIsValidUploadKey/build_log_equals1422=== RUN TestIsValidUploadKey/realisation1423=== PAUSE TestIsValidUploadKey/realisation1424=== RUN TestIsValidUploadKey/realisation_plus_in_output1425=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1426=== RUN TestIsValidUploadKey/nix-cache-info1427=== PAUSE TestIsValidUploadKey/nix-cache-info1428=== RUN TestIsValidUploadKey/index.html1429=== PAUSE TestIsValidUploadKey/index.html1430=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1431=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1432=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1433=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1434=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1435=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1436=== RUN TestIsValidUploadKey/traversal1437=== PAUSE TestIsValidUploadKey/traversal1438=== RUN TestIsValidUploadKey/traversal_nar1439=== PAUSE TestIsValidUploadKey/traversal_nar1440=== RUN TestIsValidUploadKey/absolute1441=== PAUSE TestIsValidUploadKey/absolute1442=== RUN TestIsValidUploadKey/empty_key1443=== PAUSE TestIsValidUploadKey/empty_key1444=== RUN TestIsValidUploadKey/unknown_type1445=== PAUSE TestIsValidUploadKey/unknown_type1446=== CONT TestUploadHandlersRejectInvalidKeys1447=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1448=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1449=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1450=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1451=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1452=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1453=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1454=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1455=== CONT TestProxyWriteTimeout/narinfo1456=== CONT TestProxyWriteTimeout/10_GiB_nar1457=== CONT TestProxyWriteTimeout/1_GiB_nar1458=== CONT TestProxyWriteTimeout/unknown_size1459--- PASS: TestProxyWriteTimeout (0.00s)1460 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1461 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1462 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1463 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1464=== CONT TestGCEndsOnShutdown/before_the_run14652026/11/12 00:23:16 INFO Received push request method=POST path=/api/pushes1466--- PASS: TestReadProxyOutlastsServerWriteTimeout (8.41s)1467=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows14682026/11/12 00:23:17 INFO Received complete push request method=POST path=/api/pushes/1/complete14692026/11/12 00:23:17 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"14702026/11/12 00:23:17 WARN mTLS auth: bound subjects configured but subject DN unavailable14712026/11/12 00:23:17 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1472--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (1.53s)1473=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1474=== NAME TestOrphanedObjectsGC1475 orphaned_objects_gc_test.go:287: GC Test Summary:1476 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1477 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1478 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1479 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1480 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1481--- PASS: TestOrphanedObjectsGC (4.31s)1482=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete14832026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures14842026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures14852026/11/12 00:23:17 INFO Received cleanup request method=DELETE path=/api/pending_closures1486--- PASS: TestCacheStatsHandler (1.50s)1487=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1488--- PASS: TestService_ReadScope_PublicByDefault (1.59s)1489=== CONT TestPendingClosureWriteTimeout/empty1490=== CONT TestPendingClosureWriteTimeout/670k_objects1491=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline14922026/11/12 00:23:17 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1493--- PASS: TestService_AuthMiddleware (1.57s)1494=== CONT TestPendingClosureWriteTimeout/400_objects1495=== CONT TestPush_RejectsBadRequests/no_objects14962026/11/12 00:23:17 INFO Received push request method=POST path=/api/pushes1497=== CONT TestPush_RejectsBadRequests/root_not_in_objects14982026/11/12 00:23:17 INFO Received push request method=POST path=/api/pushes1499=== CONT TestPush_RejectsBadRequests/no_roots15002026/11/12 00:23:17 INFO Received push request method=POST path=/api/pushes1501=== CONT TestPush_RejectsBadRequests/bad_root15022026/11/12 00:23:17 INFO Received push request method=POST path=/api/pushes1503=== CONT TestIsValidCachePath/narinfo1504--- PASS: TestPush_RejectsBadRequests (3.60s)1505 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1506 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1507 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1508 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1509=== CONT TestIsValidCachePath/wrong_extension1510=== CONT TestIsValidCachePath/leading_slash1511=== CONT TestIsValidCachePath/empty1512=== CONT TestIsValidCachePath/short_hash1513=== CONT TestIsValidCachePath/random_path1514=== CONT TestIsValidCachePath/log1515=== CONT TestIsValidCachePath/ls1516=== CONT TestIsValidCachePath/realisation1517=== CONT TestIsValidCachePath/nar_uncompressed1518=== CONT TestIsValidCachePath/nar_bz21519=== CONT TestIsValidCachePath/invalid_char_u1520=== CONT TestIsValidCachePath/nar_xz1521=== CONT TestIsValidCachePath/invalid_char_e1522=== CONT TestIsValidCachePath/nar_zst1523=== CONT TestIsValidCachePath/traversal_in_middle1524=== CONT TestIsValidCachePath/traversal_parent1525=== CONT TestIsValidCachePath/index.html1526=== CONT TestIsValidCachePath/nix-cache-info1527=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1528--- PASS: TestIsValidCachePath (0.00s)1529 --- PASS: TestIsValidCachePath/narinfo (0.00s)1530 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1531 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1532 --- PASS: TestIsValidCachePath/empty (0.00s)1533 --- PASS: TestIsValidCachePath/short_hash (0.00s)1534 --- PASS: TestIsValidCachePath/random_path (0.00s)1535 --- PASS: TestIsValidCachePath/log (0.00s)1536 --- PASS: TestIsValidCachePath/ls (0.00s)1537 --- PASS: TestIsValidCachePath/realisation (0.00s)1538 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1539 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1540 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1541 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1542 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1543 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1544 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1545 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1546 --- PASS: TestIsValidCachePath/index.html (0.00s)1547 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1548 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1549=== CONT TestParseSingleRange/none1550=== CONT TestParseSingleRange/open-ended1551=== CONT TestParseSingleRange/start_far_past_EOF1552=== CONT TestParseSingleRange/suffix1553=== CONT TestParseSingleRange/end_clamped_to_size1554=== CONT TestParseSingleRange/malformed_both_empty1555=== CONT TestParseSingleRange/closed1556=== CONT TestParseSingleRange/malformed_end_before_start1557=== CONT TestParseSingleRange/start_past_EOF1558=== CONT TestParseSingleRange/single_byte1559=== CONT TestParseSingleRange/suffix_exceeds_size1560=== CONT TestParseSingleRange/multi-range_ignored1561=== CONT TestParseSingleRange/unknown_unit1562=== CONT TestParseSingleRange/malformed_no_dash1563=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1564--- PASS: TestParseSingleRange (0.00s)1565 --- PASS: TestParseSingleRange/none (0.00s)1566 --- PASS: TestParseSingleRange/open-ended (0.00s)1567 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1568 --- PASS: TestParseSingleRange/suffix (0.00s)1569 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1570 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1571 --- PASS: TestParseSingleRange/closed (0.00s)1572 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1573 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1574 --- PASS: TestParseSingleRange/single_byte (0.00s)1575 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1576 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1577 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1578 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1579=== RUN TestService_RequireScope_OIDC/builder_may_write1580=== PAUSE TestService_RequireScope_OIDC/builder_may_write1581=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1582=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1583=== RUN TestService_RequireScope_OIDC/ops_may_admin1584=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1585=== RUN TestService_RequireScope_OIDC/ops_may_not_write1586=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1587=== RUN TestService_RequireScope_OIDC/reader_may_not_write1588=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1589=== RUN TestService_RequireScope_OIDC/static_token_may_admin1590=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1591=== RUN TestService_RequireScope_OIDC/static_token_may_write1592=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1593=== RUN TestService_RequireScope_OIDC/reader_may_read1594=== PAUSE TestService_RequireScope_OIDC/reader_may_read1595--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (1.70s)1596=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1597=== RUN TestService_RequireScope_OIDC/writer_implies_read1598=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1599=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1600=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1601=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1602=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1603=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1604=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1605=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1606=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1607=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1608=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1609=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1610--- PASS: TestService_ReadAuthMiddleware (1.89s)1611=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1612=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1613=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1614=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1615=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1616=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1617=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1618=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1619=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1620=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed16212026/11/12 00:23:17 INFO Completed push id=1 objects=4 duration=415.821349ms16222026/11/12 00:23:17 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete1623=== NAME TestPinProtectsFromGC1624 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC3444025990/001/store/5kh8xh7gi8gcvsgy80wsl5vcl4gyrg8q-pinned-file.txt1625 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC3444025990/001/store/yz7gmscdjvvsfxb305yva1ha6znbi59n-unpinned-file.txt16262026/11/12 00:23:17 INFO Completed upload id=2 objects=2 duration=2.385302ms1627--- PASS: TestMetricsCommit (2.30s)1628=== CONT TestResolveDBConnectionString/flag_wins1629=== CONT TestResolveDBConnectionString/nothing_configured1630=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1631=== CONT TestResolveDBConnectionString/missing_file_is_an_error1632=== CONT TestResolveDBConnectionString/file_when_flag_empty1633=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1634--- PASS: TestResolveDBConnectionString (0.00s)1635 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1636 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1637 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1638 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1639 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)16402026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures1641=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh16422026/11/12 00:23:17 INFO Aborted multipart uploads count=0 kept=016432026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures16442026/11/12 00:23:17 INFO Aborted multipart uploads count=0 kept=016452026/11/12 00:23:17 INFO Aborted multipart uploads count=0 kept=016462026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures16472026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures16482026/11/12 00:23:17 WARN Force mode enabled - objects will be deleted immediately without grace period16492026/11/12 00:23:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016502026/11/12 00:23:17 INFO Vacuumed table table=pending_closures1651=== NAME TestClientMultipleUploads1652 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1442135502/001/store/padhfll46k6i4c5czxhw6rmvgdjhilr3-test-file-1.txt16532026/11/12 00:23:17 INFO Vacuumed table table=pending_objects16542026/11/12 00:23:17 INFO Vacuumed table table=multipart_uploads16552026/11/12 00:23:17 INFO Vacuumed table table=closures16562026/11/12 00:23:17 INFO Vacuumed table table=objects1657=== NAME TestNARDeduplicationMetadataUploadBug1658 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug831978483/001/store/say0lrgppjq7zk0gyp96lm8yawbqw8gi-file1.txt1659=== CONT TestServerTLSConfig/no_client_CA1660=== CONT TestServerTLSConfig/missing_CA_file1661=== CONT TestServerTLSConfig/not_a_PEM_file1662--- PASS: TestServerTLSConfig (0.00s)1663 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1664 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1665 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1666=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts16672026/11/12 00:23:17 INFO Received request for more parts method=POST path=/16682026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures16692026/11/12 00:23:17 INFO Aborted multipart uploads count=0 kept=01670=== NAME TestClientWithDependencies1671 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies3538418694/001/store/25lnvsixsmnsmc730hq4ifmayj4fgcmh-test-script16722026/11/12 00:23:17 INFO Received uploads request method=POST path=/api/pending_closures1673=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart16742026/11/12 00:23:17 INFO Received complete multipart upload request method=POST path=/16752026/11/12 00:23:17 WARN Force mode enabled - objects will be deleted immediately without grace period16762026/11/12 00:23:17 WARN Force mode enabled - objects will be deleted immediately without grace period16772026/11/12 00:23:17 WARN Force mode enabled - objects will be deleted immediately without grace period1678=== NAME TestClientWithDependencies1679 client_integration_test.go:663: Found 1 dependencies (including self)1680=== NAME TestClientMultipleUploads1681 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1442135502/001/store/zjf6ygyxfhmrh8i18b97jybdndl31p60-test-file-2.txt1682=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure16832026/11/12 00:23:18 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Post \"http://localhost:46847/bucket98/?delete=\": context canceled"16842026/11/12 00:23:18 INFO Received uploads request method=POST path=/1685=== CONT TestClientErrorHandling/InvalidStorePath1686=== NAME TestClientCADerivations1687 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations629862571/001/store/c1vvnkk30b83i6jgl1g13z8mmx0q9jdv-ca-test1688--- PASS: TestGCEndsOnShutdown (0.00s)1689 --- PASS: TestGCEndsOnShutdown/before_the_run (0.75s)1690 --- PASS: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (0.84s)1691 --- PASS: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (0.86s)1692=== CONT TestClientErrorHandling/InvalidAuthToken16932026/11/12 00:23:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=1 objects-deleted-after-grace-period=1 objects-failed-to-delete=016942026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures16952026/11/12 00:23:18 INFO Vacuumed table table=pending_closures16962026/11/12 00:23:18 INFO Vacuumed table table=pending_objects16972026/11/12 00:23:18 INFO Vacuumed table table=multipart_uploads16982026/11/12 00:23:18 INFO Vacuumed table table=closures16992026/11/12 00:23:18 INFO Vacuumed table table=objects17002026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1701--- PASS: TestForceGCDuringPushOffersSweptObject (0.00s)1702 --- PASS: TestForceGCDuringPushOffersSweptObject/after_presence_check (0.63s)1703 --- PASS: TestForceGCDuringPushOffersSweptObject/before_pending_rows (0.95s)1704=== CONT TestClientErrorHandling/ServerNotAvailable17052026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures17062026/11/12 00:23:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLmM4MzM0ZjZkLWY0MDYtNGFhNy05ZTgzLTk3NGVmOGQ5YzhkZngxNzk0NDQyOTk3NjU3MjU5NDYx parts=101707=== NAME TestOrphanedObjectsGCStressTest1708 orphaned_objects_gc_test.go:503: Stress test completed successfully:1709 orphaned_objects_gc_test.go:504: - Active objects preserved: 201710 orphaned_objects_gc_test.go:505: - Objects deleted: 2101711 orphaned_objects_gc_test.go:506: - Total GC'd: 2101712--- PASS: TestOrphanedObjectsGCStressTest (5.37s)1713=== CONT TestCacheConfigHandler/full_config,_no_issuer1714=== CONT TestCacheConfigHandler/no_signing_keys1715=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1716=== CONT TestCacheConfigHandler/no_cache_url_configured1717--- PASS: TestCacheConfigHandler (0.00s)1718 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1719 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1720 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1721 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1722=== CONT TestIsValidUploadKey/narinfo1723=== CONT TestIsValidUploadKey/realisation_plus_in_output1724=== CONT TestIsValidUploadKey/unknown_type1725=== CONT TestIsValidUploadKey/empty_key1726=== CONT TestIsValidUploadKey/absolute1727=== CONT TestIsValidUploadKey/traversal_nar1728=== CONT TestIsValidUploadKey/traversal1729=== CONT TestIsValidUploadKey/build_log_home-manager_file1730=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1731=== CONT TestIsValidUploadKey/realisation1732=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1733=== CONT TestIsValidUploadKey/build_log_equals1734=== CONT TestIsValidUploadKey/build_log_question_mark1735=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1736=== CONT TestIsValidUploadKey/build_log_plus_in_name1737=== CONT TestIsValidUploadKey/index.html1738=== CONT TestIsValidUploadKey/nix-cache-info1739=== CONT TestIsValidUploadKey/build_log1740=== CONT TestIsValidUploadKey/nar_plain1741=== CONT TestIsValidUploadKey/listing1742=== CONT TestIsValidUploadKey/nar_xz1743=== CONT TestIsValidUploadKey/nar_zst1744--- PASS: TestIsValidUploadKey (0.00s)1745 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1746 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1747 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1748 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1749 --- PASS: TestIsValidUploadKey/absolute (0.00s)1750 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1751 --- PASS: TestIsValidUploadKey/traversal (0.00s)1752 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1753 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1754 --- PASS: TestIsValidUploadKey/realisation (0.00s)1755 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1756 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1757 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1758 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1759 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1760 --- PASS: TestIsValidUploadKey/index.html (0.00s)1761 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1762 --- PASS: TestIsValidUploadKey/build_log (0.00s)1763 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1764 --- PASS: TestIsValidUploadKey/listing (0.00s)1765 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1766 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1767=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info17682026/11/12 00:23:18 INFO Received uploads request method=POST path=/1769=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key17702026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/1771=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key17722026/11/12 00:23:18 INFO Received request for more parts method=POST path=/1773=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal17742026/11/12 00:23:18 INFO Received uploads request method=POST path=/1775--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1776 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1777 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1778 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1779 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1780=== CONT TestService_RequireScope_OIDC/builder_may_write1781=== NAME TestClientCADerivations1782 client_ca_test.go:139: Found 1 dependencies (including self)1783=== CONT TestService_RequireScope_OIDC/reader_may_read1784=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1785=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read17862026/11/12 00:23:18 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1787=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1788=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1789=== CONT TestService_RequireScope_OIDC/reader_may_not_write1790=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1791=== CONT TestService_RequireScope_OIDC/static_token_may_write1792=== CONT TestService_RequireScope_OIDC/writer_implies_read1793=== CONT TestService_RequireScope_OIDC/static_token_may_admin1794=== CONT TestService_RequireScope_OIDC/ops_may_admin1795=== CONT TestService_RequireScope_OIDC/ops_may_not_write1796=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1797=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1798--- PASS: TestService_RequireScope_OIDC (1.52s)1799 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1800 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1801 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1802 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1803 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1804 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1805 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1806 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1807 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1808 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1809 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1810 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1811 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1812 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1813=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected18142026/11/12 00:23:18 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1815=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1816=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected18172026/11/12 00:23:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18182026/11/12 00:23:18 WARN Authentication failed token_preview=eyJhbGciOi...JMV0_pS3qg token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]18192026/11/12 00:23:18 INFO Completed upload id=1 objects=3 duration=7.393508ms1820--- PASS: TestService_AuthMiddleware_OIDC (2.09s)1821 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1822 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1823 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1824 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)18252026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures18262026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18272026/11/12 00:23:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLmE4NWE4ZTIzLTBkNTAtNDAyZi05YzIzLTYzNDQ3ZWI2OGI1ZXgxNzk0NDQyOTk3Nzk1MTEyNDM4 parts=1018282026/11/12 00:23:18 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst18292026/11/12 00:23:18 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0001.narinfo18302026/11/12 00:23:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18312026/11/12 00:23:18 INFO Completed upload id=1 objects=3 duration=5.165507ms18322026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18332026/11/12 00:23:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000200000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjc2MDA2YmE4LTUyYjItNDVjNi1iZTBlLWQ0NjJjNmY2MDFjMXgxNzk0NDQyOTk3ODI0ODE1Mjky parts=1018342026/11/12 00:23:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18352026/11/12 00:23:18 INFO Completed upload id=1 objects=3 duration=7.51951ms18362026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures18372026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures18382026/11/12 00:23:18 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst18392026/11/12 00:23:18 WARN Found objects in DB but missing from S3, will re-upload count=118402026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18412026/11/12 00:23:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjE5Y2JiODQ5LTYzMWMtNGVmMS1iM2I0LTMyNjU1NjRlMTE4NXgxNzk0NDQyOTk3OTM0NTQ2MDg4 parts=1018422026/11/12 00:23:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18432026/11/12 00:23:18 INFO Completed upload id=1 objects=3 duration=2.850706ms18442026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures18452026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18462026/11/12 00:23:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLmY2NmZjY2IwLTkxMGQtNGYzNi1hNGQyLTgwNTkwNjg5Nzc3N3gxNzk0NDQyOTk4MDg4ODMzMDE2 parts=1018472026/11/12 00:23:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18482026/11/12 00:23:18 INFO Completed upload id=1 objects=3 duration=2.334462ms18492026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures1850--- PASS: TestPresentNotReportedWhileGCDeletesClosure (0.00s)1851 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/stale (0.93s)1852 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/fresh (0.97s)18532026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18542026/11/12 00:23:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100300000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjZkMjRhMGZhLTlmZjUtNDBhMi1iYjg5LWNiMThmOWE0NDQ3ZXgxNzk0NDQyOTk4MzU2Njg2ODE2 parts=1018552026/11/12 00:23:18 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18562026/11/12 00:23:18 INFO Completed upload id=2 objects=3 duration=1.516994ms18572026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures18582026/11/12 00:23:18 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst18592026/11/12 00:23:18 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0003.narinfo1860--- PASS: TestPendingClosureWriteTimeout (1.36s)1861 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)1862 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)1863 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)1864 --- PASS: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (1.60s)18652026/11/12 00:23:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18662026/11/12 00:23:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100400000000000000000000.nar.zst upload_id=NWYyZmFkOTUtOTc0My00YTM3LTg4MDYtMTBiMzk5OGYzOTAxLjZmNjQwZTE2LWQ1M2ItNDhhMS1iZmE4LTMxNTE1YzQ3ODc3OXgxNzk0NDQyOTk4NTIxMjY1Mjc5 parts=1018672026/11/12 00:23:18 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18682026/11/12 00:23:18 INFO Completed upload id=2 objects=3 duration=1.06567ms1869--- PASS: TestUploadHandlersRejectOversizedBody (0.20s)1870 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.06s)1871 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.08s)1872 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (0.96s)18732026/11/12 00:23:18 INFO Received uploads request method=POST path=/api/pending_closures18742026/11/12 00:23:18 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst18752026/11/12 00:23:18 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0004.narinfo1876--- PASS: TestService_verifyS3ReuploadsStaleNarinfo (0.00s)1877 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed (0.93s)1878 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed (0.88s)1879 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed (0.86s)1880 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost (1.50s)1881 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed (1.67s)18822026/11/12 00:23:19 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present18832026/11/12 00:23:19 INFO Received push request method=POST path=/api/pushes18842026/11/12 00:23:19 INFO Received push request method=POST path=/api/pushes18852026/11/12 00:23:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18862026/11/12 00:23:19 INFO Received push request method=POST path=/api/pushes18872026/11/12 00:23:19 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)18882026/11/12 00:23:19 INFO Uploading kvf4fw06s7vfksp6mnd8nh03lwnr6xy3-a (216B)18892026/11/12 00:23:19 INFO Uploading w1bxggkvqxzgcd8wqmqmxs57179d54j6-shared-dep (136B)18902026/11/12 00:23:19 INFO Uploading say0lrgppjq7zk0gyp96lm8yawbqw8gi-file1.txt (160B)18912026/11/12 00:23:19 INFO Received push request method=POST path=/api/pushes18922026/11/12 00:23:19 INFO Received uploads request method=POST path=/api/pending_closures18932026/11/12 00:23:19 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)18942026/11/12 00:23:19 INFO Uploading da42q96mlnvgmvp54khxxszz60m9gzai-shared-dep (136B)18952026/11/12 00:23:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18962026/11/12 00:23:19 INFO Uploading hwdmjh9kgzw4y9d29cp08xnpla1gx8bp-test-file.txt (152B)18972026/11/12 00:23:19 INFO Uploading mw85i13s4xb6x7wm9zk09gmvkscxx7y2-top (224B)18982026/11/12 00:23:19 WARN Failed to register uploaded object key=say0lrgppjq7zk0gyp96lm8yawbqw8gi.ls error="server returned 404: 404 page not found\n"18992026/11/12 00:23:19 WARN Failed to register uploaded object key=34ib6jpi93s194fhp6zl878b1ih3kqmg.ls error="server returned 404: 404 page not found\n"19002026/11/12 00:23:19 WARN Failed to register uploaded object key=kvf4fw06s7vfksp6mnd8nh03lwnr6xy3.ls error="server returned 404: 404 page not found\n"19012026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/1ddkhmkhaj3ifqgd0hkhra2j0sch279f2pvw4cj91marj8vjk4s8.nar.zst error="server returned 404: 404 page not found\n"19022026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19032026/11/12 00:23:19 INFO Received uploads request method=POST path=/api/pending_closures19042026/11/12 00:23:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19052026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"19062026/11/12 00:23:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19072026/11/12 00:23:19 WARN Failed to register uploaded object key=w1bxggkvqxzgcd8wqmqmxs57179d54j6.ls error="server returned 404: 404 page not found\n"19082026/11/12 00:23:19 INFO Signed narinfos id=1 count=119092026/11/12 00:23:19 INFO Uploading 1 narinfos19102026/11/12 00:23:19 INFO Signed narinfos id=1 count=319112026/11/12 00:23:19 INFO Uploading 3 narinfos19122026/11/12 00:23:19 WARN Failed to register uploaded object key=da42q96mlnvgmvp54khxxszz60m9gzai.ls error="server returned 404: 404 page not found\n"19132026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/0gll30ksm9qcl5yjkc2kn16m95rb44i7gcq169girm0ccz1q3pry.nar.zst error="server returned 404: 404 page not found\n"19142026/11/12 00:23:19 WARN Failed to register uploaded object key=hwdmjh9kgzw4y9d29cp08xnpla1gx8bp.ls error="server returned 404: 404 page not found\n"19152026/11/12 00:23:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19162026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"19172026/11/12 00:23:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19182026/11/12 00:23:19 WARN Failed to register uploaded object key=mw85i13s4xb6x7wm9zk09gmvkscxx7y2.ls error="server returned 404: 404 page not found\n"19192026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19202026/11/12 00:23:19 INFO Signed narinfos id=1 count=219212026/11/12 00:23:19 INFO Signed narinfos id=1 count=119222026/11/12 00:23:19 INFO Uploading 2 narinfos19232026/11/12 00:23:19 INFO Uploading 1 narinfos19242026/11/12 00:23:19 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)19252026/11/12 00:23:19 INFO Uploading 5s9gri1rnhkzfbqmw9vb6jvgifjljdwb-b (216B)19262026/11/12 00:23:19 INFO Uploading 9j75p6zf4hv5bd7qxhg7glpj391d8hvs-shared-dep (136B)19272026/11/12 00:23:19 INFO Received complete push request method=POST path=/api/pushes/1/complete19282026/11/12 00:23:19 WARN Failed to register uploaded object key=say0lrgppjq7zk0gyp96lm8yawbqw8gi.narinfo error="server returned 404: 404 page not found\n"19292026/11/12 00:23:19 WARN Failed to register uploaded object key=kvf4fw06s7vfksp6mnd8nh03lwnr6xy3.narinfo error="server returned 404: 404 page not found\n"19302026/11/12 00:23:19 WARN Failed to register uploaded object key=34ib6jpi93s194fhp6zl878b1ih3kqmg.narinfo error="server returned 404: 404 page not found\n"19312026/11/12 00:23:19 INFO Received complete push request method=POST path=/api/pushes/1/complete19322026/11/12 00:23:19 WARN Failed to register uploaded object key=w1bxggkvqxzgcd8wqmqmxs57179d54j6.narinfo error="server returned 404: 404 page not found\n"19332026/11/12 00:23:19 INFO Received complete push request method=POST path=/api/pushes/1/complete19342026/11/12 00:23:19 WARN Failed to register uploaded object key=hwdmjh9kgzw4y9d29cp08xnpla1gx8bp.narinfo error="server returned 404: 404 page not found\n"19352026/11/12 00:23:19 WARN Failed to register uploaded object key=mw85i13s4xb6x7wm9zk09gmvkscxx7y2.narinfo error="server returned 404: 404 page not found\n"19362026/11/12 00:23:19 INFO Received complete push request method=POST path=/api/pushes/1/complete19372026/11/12 00:23:19 WARN Failed to register uploaded object key=da42q96mlnvgmvp54khxxszz60m9gzai.narinfo error="server returned 404: 404 page not found\n"19382026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/07zsl2z4k4558n5zpshdcbyb16znarg72clf3fx526ij4laslfyh.nar.zst error="server returned 404: 404 page not found\n"19392026/11/12 00:23:19 WARN Failed to register uploaded object key=fqgwnq8zsfzy19rc2s8bhr886n1yrqph.ls error="server returned 404: 404 page not found\n"19402026/11/12 00:23:19 WARN Failed to register uploaded object key=5s9gri1rnhkzfbqmw9vb6jvgifjljdwb.ls error="server returned 404: 404 page not found\n"19412026/11/12 00:23:19 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19422026/11/12 00:23:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign19432026/11/12 00:23:19 WARN Failed to register uploaded object key=9j75p6zf4hv5bd7qxhg7glpj391d8hvs.ls error="server returned 404: 404 page not found\n"19442026/11/12 00:23:19 INFO Signed narinfos id=2 count=219452026/11/12 00:23:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign19462026/11/12 00:23:19 INFO Signed narinfos id=1 count=219472026/11/12 00:23:19 INFO Uploading 4 narinfos19482026/11/12 00:23:19 INFO Completed push id=1 objects=8 duration=5.757573ms19492026/11/12 00:23:19 INFO Upload complete. (1.254s)19502026/11/12 00:23:19 INFO Completed push id=1 objects=3 duration=9.049144ms19512026/11/12 00:23:19 INFO Upload complete. (1.257s)1952=== NAME TestNARDeduplicationMetadataUploadBug1953 metadata_upload_test.go:54: Retrieved narinfo from S3:1954 StorePath: /build/TestNARDeduplicationMetadataUploadBug831978483/001/store/say0lrgppjq7zk0gyp96lm8yawbqw8gi-file1.txt1955 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1956 Compression: zstd1957 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1958 NarSize: 1601959 References: 1960 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1961=== NAME TestClientPushesUseOnePush19622026/11/12 00:23:19 INFO Completed push id=1 objects=6 duration=8.402358ms1963 client_pushes_test.go:97: Retrieved narinfo from S3:1964 StorePath: /build/TestClientPushesUseOnePush1444231097/001/store/w1bxggkvqxzgcd8wqmqmxs57179d54j6-shared-dep1965 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1966 Compression: zstd1967 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821968 NarSize: 1361969 References: 1970 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n19712026/11/12 00:23:19 INFO Upload complete. (1.386s)1972=== NAME TestClientSharedPathCommittedMidPush19732026/11/12 00:23:19 INFO Completed push id=1 objects=3 duration=10.249255ms1974 client_integration_test.go:728: Retrieved narinfo from S3:1975 StorePath: /build/TestClientSharedPathCommittedMidPush2180796700/001/store/da42q96mlnvgmvp54khxxszz60m9gzai-shared-dep1976 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1977 Compression: zstd1978 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821979 NarSize: 1361980 References: 1981 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n19822026/11/12 00:23:19 INFO Upload complete. (1.543s)19832026/11/12 00:23:19 WARN Failed to register uploaded object key=fqgwnq8zsfzy19rc2s8bhr886n1yrqph.narinfo error="server returned 404: 404 page not found\n"19842026/11/12 00:23:19 WARN Failed to register uploaded object key=9j75p6zf4hv5bd7qxhg7glpj391d8hvs.narinfo error="server returned 404: 404 page not found\n"1985=== NAME TestNARDeduplicationMetadataUploadBug1986 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1987 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1988 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}19892026/11/12 00:23:19 WARN Failed to register uploaded object key=5s9gri1rnhkzfbqmw9vb6jvgifjljdwb.narinfo error="server returned 404: 404 page not found\n"1990=== NAME TestClientPushesUseOnePush1991 client_pushes_test.go:97: Retrieved narinfo from S3:1992 StorePath: /build/TestClientPushesUseOnePush1444231097/001/store/kvf4fw06s7vfksp6mnd8nh03lwnr6xy3-a1993 URL: nar/1ddkhmkhaj3ifqgd0hkhra2j0sch279f2pvw4cj91marj8vjk4s8.nar.zst1994 Compression: zstd1995 NarHash: sha256:1ddkhmkhaj3ifqgd0hkhra2j0sch279f2pvw4cj91marj8vjk4s81996 NarSize: 2161997 References: /build/TestClientPushesUseOnePush1444231097/001/store/w1bxggkvqxzgcd8wqmqmxs57179d54j6-shared-dep1998 CA: text:sha256:12qivfqdd9v4hh3ygwjxdfbd2jyqxsy0ymi5ynzrlxnyxc1bgmbq1999=== NAME TestClientSharedPathCommittedMidPush2000 client_integration_test.go:728: Retrieved narinfo from S3:

loading 643 hidden lines…

26442026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=026452026/11/12 00:23:34 INFO Received push request method=POST path=/26462026/11/12 00:23:34 INFO Vacuumed table table=pending_objects26472026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads26482026/11/12 00:23:34 INFO Vacuumed table table=pending_closures26492026/11/12 00:23:34 INFO Vacuumed table table=closures26502026/11/12 00:23:34 INFO Vacuumed table table=objects26512026/11/12 00:23:34 INFO Vacuumed table table=pending_objects26522026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads26532026/11/12 00:23:34 INFO Vacuumed table table=closures26542026/11/12 00:23:34 INFO Vacuumed table table=objects26552026/11/12 00:23:34 INFO Received uploads request method=POST path=/26562026/11/12 00:23:34 INFO Received uploads request method=POST path=/26572026/11/12 00:23:34 INFO Received uploads request method=POST path=/26582026/11/12 00:23:34 INFO Received push request method=POST path=/26592026/11/12 00:23:34 INFO Received complete upload request method=POST path=/26602026/11/12 00:23:34 INFO Completed upload id=5 objects=7 duration=2.600044ms26612026/11/12 00:23:34 INFO Received push request method=POST path=/26622026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=026632026/11/12 00:23:34 INFO Received complete upload request method=POST path=/26642026/11/12 00:23:34 INFO Received push request method=POST path=/26652026/11/12 00:23:34 INFO Completed upload id=3 objects=24 duration=2.046479ms26662026/11/12 00:23:34 INFO Received complete push request method=POST path=/26672026/11/12 00:23:34 INFO Received uploads request method=POST path=/26682026/11/12 00:23:34 INFO Completed push id=3 objects=5 duration=2.504543ms26692026/11/12 00:23:34 INFO Received complete push request method=POST path=/26702026/11/12 00:23:34 INFO Received push request method=POST path=/26712026/11/12 00:23:34 INFO Received push request method=POST path=/26722026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=026732026/11/12 00:23:34 INFO Completed push id=2 objects=25 duration=4.042038ms26742026/11/12 00:23:34 INFO Received complete upload request method=POST path=/26752026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period26762026/11/12 00:23:34 INFO Completed upload id=6 objects=9 duration=1.203031ms26772026/11/12 00:23:34 INFO Received push request method=POST path=/26782026/11/12 00:23:34 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."26792026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=026802026/11/12 00:23:34 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."26812026/11/12 00:23:34 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."26822026/11/12 00:23:34 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."26832026/11/12 00:23:34 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."26842026/11/12 00:23:34 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."26852026/11/12 00:23:34 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."26862026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period26872026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026882026/11/12 00:23:34 INFO Vacuumed table table=pending_closures26892026/11/12 00:23:34 INFO Vacuumed table table=pending_objects26902026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads26912026/11/12 00:23:34 INFO Received complete push request method=POST path=/26922026/11/12 00:23:34 INFO Vacuumed table table=closures26932026/11/12 00:23:34 INFO Vacuumed table table=objects26942026/11/12 00:23:34 INFO Completed push id=2 objects=7 duration=3.703934ms26952026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=026962026/11/12 00:23:34 INFO Received push request method=POST path=/26972026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period26982026/11/12 00:23:34 INFO Received complete upload request method=POST path=/26992026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027002026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027012026/11/12 00:23:34 INFO Received complete upload request method=POST path=/27022026/11/12 00:23:34 INFO Completed upload id=10 objects=23 duration=2.088079ms27032026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027042026/11/12 00:23:34 INFO Completed upload id=2 objects=2 duration=2.544964ms27052026/11/12 00:23:34 INFO Vacuumed table table=pending_closures27062026/11/12 00:23:34 INFO Received uploads request method=POST path=/27072026/11/12 00:23:34 INFO Vacuumed table table=pending_objects27082026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads27092026/11/12 00:23:34 INFO Received push request method=POST path=/27102026/11/12 00:23:34 INFO Received push request method=POST path=/27112026/11/12 00:23:34 INFO Received push request method=POST path=/27122026/11/12 00:23:34 INFO Vacuumed table table=closures27132026/11/12 00:23:34 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27142026/11/12 00:23:34 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27152026/11/12 00:23:34 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27162026/11/12 00:23:34 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27172026/11/12 00:23:34 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27182026/11/12 00:23:34 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27192026/11/12 00:23:34 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27202026/11/12 00:23:34 INFO Received uploads request method=POST path=/27212026/11/12 00:23:34 INFO Vacuumed table table=objects27222026/11/12 00:23:34 INFO Received uploads request method=POST path=/27232026/11/12 00:23:34 INFO Received complete push request method=POST path=/27242026/11/12 00:23:34 INFO Received uploads request method=POST path=/27252026/11/12 00:23:34 INFO Completed push id=6 objects=5 duration=1.718256ms27262026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027272026/11/12 00:23:34 INFO Received uploads request method=POST path=/27282026/11/12 00:23:34 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27292026/11/12 00:23:34 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27302026/11/12 00:23:34 INFO Received push request method=POST path=/27312026/11/12 00:23:34 INFO Received complete upload request method=POST path=/27322026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027332026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027342026/11/12 00:23:34 INFO Completed upload id=3 objects=5 duration=3.20653ms27352026/11/12 00:23:34 INFO Vacuumed table table=pending_closures27362026/11/12 00:23:34 INFO Received push request method=POST path=/27372026/11/12 00:23:34 INFO Vacuumed table table=pending_objects27382026/11/12 00:23:34 INFO Received complete push request method=POST path=/27392026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads27402026/11/12 00:23:34 INFO Received push request method=POST path=/27412026/11/12 00:23:34 INFO Vacuumed table table=closures27422026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period27432026/11/12 00:23:34 INFO Completed push id=4 objects=4 duration=3.074309ms27442026/11/12 00:23:34 INFO Vacuumed table table=objects27452026/11/12 00:23:34 INFO Received uploads request method=POST path=/27462026/11/12 00:23:34 INFO Received complete push request method=POST path=/27472026/11/12 00:23:34 INFO Received complete push request method=POST path=/27482026/11/12 00:23:34 INFO Completed push id=14 objects=3 duration=2.719045ms27492026/11/12 00:23:34 INFO Completed push id=7 objects=20 duration=3.021288ms27502026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027512026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027522026/11/12 00:23:34 INFO Received complete upload request method=POST path=/27532026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=027542026/11/12 00:23:34 INFO Received uploads request method=POST path=/27552026/11/12 00:23:34 INFO Completed upload id=11 objects=6 duration=1.566915ms27562026/11/12 00:23:34 INFO Vacuumed table table=pending_closures27572026/11/12 00:23:34 INFO Vacuumed table table=pending_objects27582026/11/12 00:23:34 INFO Received push request method=POST path=/27592026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads27602026/11/12 00:23:34 INFO Received push request method=POST path=/27612026/11/12 00:23:34 INFO Received push request method=POST path=/27622026/11/12 00:23:34 INFO Received complete upload request method=POST path=/27632026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027642026/11/12 00:23:34 INFO Received uploads request method=POST path=/27652026/11/12 00:23:34 INFO Vacuumed table table=closures27662026/11/12 00:23:34 INFO Received complete upload request method=POST path=/27672026/11/12 00:23:34 INFO Received push request method=POST path=/27682026/11/12 00:23:34 INFO Vacuumed table table=pending_closures27692026/11/12 00:23:34 INFO Completed upload id=3 objects=25 duration=1.961498ms27702026/11/12 00:23:34 INFO Vacuumed table table=objects27712026/11/12 00:23:34 INFO Vacuumed table table=pending_objects27722026/11/12 00:23:34 INFO Completed upload id=3 objects=8 duration=2.269081ms27732026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads27742026/11/12 00:23:34 INFO Vacuumed table table=closures27752026/11/12 00:23:34 INFO Vacuumed table table=objects27762026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027772026/11/12 00:23:34 INFO Received push request method=POST path=/27782026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027792026/11/12 00:23:34 INFO Received complete push request method=POST path=/27802026/11/12 00:23:34 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27812026/11/12 00:23:34 INFO Received push request method=POST path=/27822026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027832026/11/12 00:23:34 INFO Completed push id=5 objects=6 duration=1.807157ms27842026/11/12 00:23:34 INFO Received push request method=POST path=/27852026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period27862026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027872026/11/12 00:23:34 INFO Vacuumed table table=pending_closures27882026/11/12 00:23:34 INFO Vacuumed table table=pending_objects27892026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads27902026/11/12 00:23:34 INFO Vacuumed table table=closures27912026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027922026/11/12 00:23:34 INFO Received push request method=POST path=/27932026/11/12 00:23:34 INFO Vacuumed table table=objects27942026/11/12 00:23:34 INFO Received uploads request method=POST path=/27952026/11/12 00:23:34 INFO Received complete push request method=POST path=/27962026/11/12 00:23:34 INFO Received uploads request method=POST path=/27972026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=027982026/11/12 00:23:34 INFO Received push request method=POST path=/27992026/11/12 00:23:34 INFO Completed push id=12 objects=15 duration=2.17118ms28002026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=028012026/11/12 00:23:34 INFO Vacuumed table table=pending_closures28022026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period28032026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028042026/11/12 00:23:34 INFO Vacuumed table table=pending_objects28052026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads28062026/11/12 00:23:34 INFO Received push request method=POST path=/28072026/11/12 00:23:34 INFO Vacuumed table table=pending_closures28082026/11/12 00:23:34 INFO Vacuumed table table=closures28092026/11/12 00:23:34 INFO Vacuumed table table=objects28102026/11/12 00:23:34 INFO Vacuumed table table=pending_objects28112026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads28122026/11/12 00:23:34 INFO Vacuumed table table=closures28132026/11/12 00:23:34 INFO Vacuumed table table=objects28142026/11/12 00:23:34 INFO Received complete push request method=POST path=/28152026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=028162026/11/12 00:23:34 INFO Received push request method=POST path=/28172026/11/12 00:23:34 INFO Received complete upload request method=POST path=/28182026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=028192026/11/12 00:23:34 INFO Received complete push request method=POST path=/28202026/11/12 00:23:34 INFO Completed push id=2 objects=10 duration=3.787875ms28212026/11/12 00:23:34 INFO Completed upload id=10 objects=6 duration=2.861366ms28222026/11/12 00:23:34 INFO Completed push id=5 objects=3 duration=2.651565ms28232026/11/12 00:23:34 INFO Received push request method=POST path=/28242026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028252026/11/12 00:23:34 INFO Received complete push request method=POST path=/28262026/11/12 00:23:34 INFO Vacuumed table table=pending_closures28272026/11/12 00:23:34 INFO Vacuumed table table=pending_objects28282026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads28292026/11/12 00:23:34 INFO Completed push id=5 objects=3 duration=1.635435ms28302026/11/12 00:23:34 INFO Vacuumed table table=closures28312026/11/12 00:23:34 INFO Vacuumed table table=objects28322026/11/12 00:23:34 INFO Received uploads request method=POST path=/28332026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=028342026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period28352026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028362026/11/12 00:23:34 INFO Vacuumed table table=pending_closures28372026/11/12 00:23:34 INFO Vacuumed table table=pending_objects28382026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads28392026/11/12 00:23:34 INFO Vacuumed table table=closures28402026/11/12 00:23:34 INFO Vacuumed table table=objects28412026/11/12 00:23:34 INFO Received complete push request method=POST path=/28422026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=028432026/11/12 00:23:34 INFO Completed push id=4 objects=4 duration=2.830886ms28442026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028452026/11/12 00:23:34 INFO Vacuumed table table=pending_closures28462026/11/12 00:23:34 INFO Vacuumed table table=pending_objects28472026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads28482026/11/12 00:23:34 INFO Received push request method=POST path=/28492026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=028502026/11/12 00:23:34 INFO Vacuumed table table=closures28512026/11/12 00:23:34 INFO Vacuumed table table=objects28522026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028532026/11/12 00:23:34 INFO Vacuumed table table=pending_closures28542026/11/12 00:23:34 INFO Vacuumed table table=pending_objects28552026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads28562026/11/12 00:23:34 INFO Vacuumed table table=closures28572026/11/12 00:23:34 INFO Vacuumed table table=objects28582026/11/12 00:23:34 INFO Received push request method=POST path=/28592026/11/12 00:23:34 INFO Received push request method=POST path=/28602026/11/12 00:23:34 INFO Received complete push request method=POST path=/28612026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=028622026/11/12 00:23:34 INFO Completed push id=8 objects=7 duration=1.644515ms28632026/11/12 00:23:34 INFO Received uploads request method=POST path=/28642026/11/12 00:23:34 INFO Received complete push request method=POST path=/28652026/11/12 00:23:34 INFO Completed push id=6 objects=6 duration=1.843237ms28662026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=028672026/11/12 00:23:34 INFO Received complete push request method=POST path=/28682026/11/12 00:23:34 INFO Received push request method=POST path=/28692026/11/12 00:23:34 INFO Completed push id=8 objects=6 duration=1.498873ms28702026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028712026/11/12 00:23:34 INFO Received uploads request method=POST path=/28722026/11/12 00:23:34 INFO Vacuumed table table=pending_closures28732026/11/12 00:23:34 INFO Vacuumed table table=pending_objects28742026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads28752026/11/12 00:23:34 INFO Vacuumed table table=closures28762026/11/12 00:23:34 INFO Received push request method=POST path=/28772026/11/12 00:23:34 INFO Vacuumed table table=objects28782026/11/12 00:23:34 INFO Received push request method=POST path=/28792026/11/12 00:23:34 INFO Received push request method=POST path=/28802026/11/12 00:23:34 INFO Received uploads request method=POST path=/28812026/11/12 00:23:34 INFO Received uploads request method=POST path=/28822026/11/12 00:23:34 INFO Received push request method=POST path=/28832026/11/12 00:23:34 INFO Received complete upload request method=POST path=/28842026/11/12 00:23:34 INFO Completed upload id=9 objects=16 duration=1.567074ms28852026/11/12 00:23:34 INFO Received push request method=POST path=/28862026/11/12 00:23:34 INFO Received complete push request method=POST path=/28872026/11/12 00:23:34 INFO Completed push id=5 objects=9 duration=3.908756ms28882026/11/12 00:23:34 INFO Received uploads request method=POST path=/28892026/11/12 00:23:34 INFO Received uploads request method=POST path=/28902026/11/12 00:23:34 INFO Received uploads request method=POST path=/28912026/11/12 00:23:34 INFO Received push request method=POST path=/28922026/11/12 00:23:34 INFO Received complete push request method=POST path=/28932026/11/12 00:23:34 INFO Completed push id=7 objects=27 duration=1.808957ms28942026/11/12 00:23:34 INFO Received push request method=POST path=/28952026/11/12 00:23:34 INFO Received complete push request method=POST path=/28962026/11/12 00:23:34 INFO Received uploads request method=POST path=/28972026/11/12 00:23:34 INFO Completed push id=6 objects=26 duration=3.155889ms28982026/11/12 00:23:34 INFO Received complete upload request method=POST path=/28992026/11/12 00:23:34 INFO Completed upload id=11 objects=6 duration=2.104559ms29002026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029012026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period29022026/11/12 00:23:34 INFO Received push request method=POST path=/29032026/11/12 00:23:34 INFO Received push request method=POST path=/29042026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029052026/11/12 00:23:34 INFO Received push request method=POST path=/29062026/11/12 00:23:34 INFO Vacuumed table table=pending_closures29072026/11/12 00:23:34 INFO Received uploads request method=POST path=/29082026/11/12 00:23:34 INFO Vacuumed table table=pending_objects29092026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads29102026/11/12 00:23:34 INFO Vacuumed table table=closures29112026/11/12 00:23:34 INFO Vacuumed table table=objects29122026/11/12 00:23:34 INFO Received uploads request method=POST path=/29132026/11/12 00:23:34 INFO Received uploads request method=POST path=/29142026/11/12 00:23:34 INFO Received complete upload request method=POST path=/29152026/11/12 00:23:34 INFO Received push request method=POST path=/29162026/11/12 00:23:34 INFO Completed upload id=3 objects=9 duration=1.298732ms29172026/11/12 00:23:34 INFO Received complete push request method=POST path=/29182026/11/12 00:23:34 INFO Received uploads request method=POST path=/29192026/11/12 00:23:34 INFO Received push request method=POST path=/29202026/11/12 00:23:34 INFO Completed push id=10 objects=3 duration=3.418452ms29212026/11/12 00:23:34 INFO Received push request method=POST path=/29222026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029232026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period29242026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029252026/11/12 00:23:34 INFO Vacuumed table table=pending_closures29262026/11/12 00:23:34 INFO Vacuumed table table=pending_objects29272026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads29282026/11/12 00:23:34 INFO Vacuumed table table=closures29292026/11/12 00:23:34 INFO Vacuumed table table=objects29302026/11/12 00:23:34 INFO Received uploads request method=POST path=/29312026/11/12 00:23:34 INFO Received uploads request method=POST path=/29322026/11/12 00:23:34 INFO Received push request method=POST path=/29332026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029342026/11/12 00:23:34 INFO Received push request method=POST path=/29352026/11/12 00:23:34 INFO Received complete upload request method=POST path=/29362026/11/12 00:23:34 INFO Completed upload id=10 objects=17 duration=1.230952ms29372026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029382026/11/12 00:23:34 INFO Vacuumed table table=pending_closures29392026/11/12 00:23:34 INFO Vacuumed table table=pending_objects29402026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads29412026/11/12 00:23:34 INFO Vacuumed table table=closures29422026/11/12 00:23:34 INFO Received complete upload request method=POST path=/29432026/11/12 00:23:34 INFO Completed upload id=15 objects=6 duration=1.244951ms29442026/11/12 00:23:34 INFO Vacuumed table table=objects29452026/11/12 00:23:34 INFO Received uploads request method=POST path=/29462026/11/12 00:23:34 INFO Received push request method=POST path=/29472026/11/12 00:23:34 INFO Received complete push request method=POST path=/29482026/11/12 00:23:34 INFO Completed push id=5 objects=6 duration=1.609495ms29492026/11/12 00:23:34 INFO Received push request method=POST path=/29502026/11/12 00:23:34 INFO Received complete push request method=POST path=/29512026/11/12 00:23:34 INFO Completed push id=9 objects=5 duration=1.565174ms29522026/11/12 00:23:34 INFO Received push request method=POST path=/29532026/11/12 00:23:34 INFO Received push request method=POST path=/29542026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029552026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029562026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029572026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period29582026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029592026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period29602026/11/12 00:23:34 INFO Vacuumed table table=pending_closures29612026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period29622026/11/12 00:23:34 INFO Received push request method=POST path=/29632026/11/12 00:23:34 INFO Vacuumed table table=pending_objects29642026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029652026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029662026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads29672026/11/12 00:23:34 INFO Received push request method=POST path=/29682026/11/12 00:23:34 INFO Vacuumed table table=closures29692026/11/12 00:23:34 INFO Vacuumed table table=pending_closures29702026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period29712026/11/12 00:23:34 INFO Vacuumed table table=objects29722026/11/12 00:23:34 INFO Received uploads request method=POST path=/29732026/11/12 00:23:34 INFO Vacuumed table table=pending_objects29742026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029752026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads29762026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=029772026/11/12 00:23:34 INFO Vacuumed table table=closures29782026/11/12 00:23:34 INFO Vacuumed table table=objects29792026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period29802026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029812026/11/12 00:23:34 INFO Vacuumed table table=pending_closures29822026/11/12 00:23:34 INFO Vacuumed table table=pending_objects29832026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads29842026/11/12 00:23:34 INFO Vacuumed table table=closures29852026/11/12 00:23:34 INFO Vacuumed table table=objects29862026/11/12 00:23:34 INFO Received push request method=POST path=/29872026/11/12 00:23:34 INFO Received uploads request method=POST path=/29882026/11/12 00:23:34 INFO Received uploads request method=POST path=/29892026/11/12 00:23:34 INFO Received push request method=POST path=/29902026/11/12 00:23:34 INFO Received push request method=POST path=/29912026/11/12 00:23:34 INFO Received complete push request method=POST path=/29922026/11/12 00:23:34 INFO Received push request method=POST path=/29932026/11/12 00:23:34 INFO Completed push id=14 objects=6 duration=1.565095ms29942026/11/12 00:23:34 INFO Received complete push request method=POST path=/29952026/11/12 00:23:34 INFO Completed push id=16 objects=27 duration=1.882737ms29962026/11/12 00:23:34 INFO Received uploads request method=POST path=/29972026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=029982026/11/12 00:23:34 INFO Vacuumed table table=pending_closures29992026/11/12 00:23:34 INFO Vacuumed table table=pending_objects30002026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads30012026/11/12 00:23:34 INFO Vacuumed table table=closures30022026/11/12 00:23:34 INFO Vacuumed table table=objects30032026/11/12 00:23:34 INFO Received push request method=POST path=/30042026/11/12 00:23:34 INFO Received push request method=POST path=/30052026/11/12 00:23:34 INFO Received uploads request method=POST path=/30062026/11/12 00:23:34 INFO Received uploads request method=POST path=/30072026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=030082026/11/12 00:23:34 INFO Received push request method=POST path=/30092026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period30102026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=030112026/11/12 00:23:34 INFO Received push request method=POST path=/30122026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030132026/11/12 00:23:35 INFO Vacuumed table table=pending_closures30142026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030152026/11/12 00:23:35 INFO Vacuumed table table=pending_closures30162026/11/12 00:23:35 INFO Vacuumed table table=pending_objects30172026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads30182026/11/12 00:23:35 INFO Vacuumed table table=closures30192026/11/12 00:23:35 INFO Vacuumed table table=pending_objects30202026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads30212026/11/12 00:23:35 INFO Vacuumed table table=objects30222026/11/12 00:23:35 INFO Vacuumed table table=closures30232026/11/12 00:23:35 INFO Vacuumed table table=objects30242026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=030252026/11/12 00:23:35 INFO Received push request method=POST path=/30262026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=030272026/11/12 00:23:35 INFO Vacuumed table table=pending_closures30282026/11/12 00:23:35 INFO Vacuumed table table=pending_objects30292026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads30302026/11/12 00:23:35 INFO Vacuumed table table=closures30312026/11/12 00:23:35 INFO Vacuumed table table=objects30322026/11/12 00:23:35 INFO Received complete push request method=POST path=/30332026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=030342026/11/12 00:23:35 INFO Completed push id=18 objects=10 duration=1.755637ms30352026/11/12 00:23:35 INFO Received complete push request method=POST path=/30362026/11/12 00:23:35 INFO Completed push id=9 objects=7 duration=3.665214ms30372026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=030382026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period30392026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030402026/11/12 00:23:35 INFO Vacuumed table table=pending_closures30412026/11/12 00:23:35 INFO Vacuumed table table=pending_objects30422026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads30432026/11/12 00:23:35 INFO Vacuumed table table=closures30442026/11/12 00:23:35 INFO Vacuumed table table=objects30452026/11/12 00:23:35 INFO Received uploads request method=POST path=/30462026/11/12 00:23:35 INFO Received push request method=POST path=/30472026/11/12 00:23:35 INFO Received complete upload request method=POST path=/30482026/11/12 00:23:35 INFO Received push request method=POST path=/30492026/11/12 00:23:35 INFO Completed upload id=10 objects=7 duration=1.955138ms30502026/11/12 00:23:35 INFO Received uploads request method=POST path=/30512026/11/12 00:23:35 INFO Received uploads request method=POST path=/30522026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=030532026/11/12 00:23:35 INFO Received complete upload request method=POST path=/30542026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period30552026/11/12 00:23:35 INFO Completed upload id=17 objects=6 duration=2.315581ms30562026/11/12 00:23:35 INFO Received complete push request method=POST path=/30572026/11/12 00:23:35 INFO Received uploads request method=POST path=/30582026/11/12 00:23:35 INFO Completed push id=15 objects=24 duration=4.855766ms30592026/11/12 00:23:35 INFO Received complete push request method=POST path=/30602026/11/12 00:23:35 INFO Received uploads request method=POST path=/30612026/11/12 00:23:35 INFO Received uploads request method=POST path=/30622026/11/12 00:23:35 INFO Completed push id=12 objects=2 duration=2.754585ms30632026/11/12 00:23:35 INFO Received push request method=POST path=/30642026/11/12 00:23:35 INFO Received complete push request method=POST path=/30652026/11/12 00:23:35 INFO Received complete upload request method=POST path=/30662026/11/12 00:23:35 INFO Completed push id=3 objects=21 duration=3.889856ms30672026/11/12 00:23:35 INFO Received uploads request method=POST path=/30682026/11/12 00:23:35 INFO Completed upload id=18 objects=4 duration=1.991778ms30692026/11/12 00:23:35 INFO Received uploads request method=POST path=/30702026/11/12 00:23:35 INFO Received uploads request method=POST path=/30712026/11/12 00:23:35 INFO Received push request method=POST path=/30722026/11/12 00:23:35 INFO Received uploads request method=POST path=/30732026/11/12 00:23:35 INFO Received uploads request method=POST path=/30742026/11/12 00:23:35 INFO Received push request method=POST path=/30752026/11/12 00:23:35 INFO Received uploads request method=POST path=/30762026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=030772026/11/12 00:23:35 INFO Received complete upload request method=POST path=/30782026/11/12 00:23:35 INFO Completed upload id=6 objects=6 duration=1.586935ms30792026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030802026/11/12 00:23:35 INFO Vacuumed table table=pending_closures30812026/11/12 00:23:35 INFO Vacuumed table table=pending_objects30822026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads30832026/11/12 00:23:35 INFO Vacuumed table table=closures30842026/11/12 00:23:35 INFO Vacuumed table table=objects30852026/11/12 00:23:35 INFO Received uploads request method=POST path=/30862026/11/12 00:23:35 INFO Received uploads request method=POST path=/30872026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=030882026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period30892026/11/12 00:23:35 INFO Received push request method=POST path=/30902026/11/12 00:23:35 INFO Received push request method=POST path=/30912026/11/12 00:23:35 INFO Received uploads request method=POST path=/30922026/11/12 00:23:35 INFO Received uploads request method=POST path=/30932026/11/12 00:23:35 INFO Received complete push request method=POST path=/30942026/11/12 00:23:35 INFO Completed push id=15 objects=5 duration=2.14728ms30952026/11/12 00:23:35 INFO Received complete push request method=POST path=/30962026/11/12 00:23:35 INFO Completed push id=1 objects=11 duration=1.828877ms30972026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=030982026/11/12 00:23:35 INFO Received complete upload request method=POST path=/30992026/11/12 00:23:35 INFO Received uploads request method=POST path=/31002026/11/12 00:23:35 INFO Completed upload id=15 objects=9 duration=1.128251ms31012026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031022026/11/12 00:23:35 INFO Vacuumed table table=pending_closures31032026/11/12 00:23:35 INFO Vacuumed table table=pending_objects31042026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads31052026/11/12 00:23:35 INFO Vacuumed table table=closures31062026/11/12 00:23:35 INFO Vacuumed table table=objects31072026/11/12 00:23:35 INFO Received complete upload request method=POST path=/31082026/11/12 00:23:35 INFO Completed upload id=16 objects=3 duration=1.149731ms31092026/11/12 00:23:35 INFO Received complete push request method=POST path=/31102026/11/12 00:23:35 INFO Completed push id=22 objects=3 duration=2.323462ms31112026/11/12 00:23:35 INFO Received push request method=POST path=/31122026/11/12 00:23:35 INFO Received push request method=POST path=/31132026/11/12 00:23:35 INFO Received complete push request method=POST path=/31142026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031152026/11/12 00:23:35 INFO Completed push id=21 objects=17 duration=1.983139ms31162026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period31172026/11/12 00:23:35 INFO Received push request method=POST path=/31182026/11/12 00:23:35 INFO Received complete push request method=POST path=/31192026/11/12 00:23:35 INFO Completed push id=19 objects=7 duration=1.456834ms31202026/11/12 00:23:35 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31212026/11/12 00:23:35 INFO Received push request method=POST path=/31222026/11/12 00:23:35 INFO Received uploads request method=POST path=/31232026/11/12 00:23:35 INFO Received uploads request method=POST path=/31242026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031252026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period31262026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031272026/11/12 00:23:35 INFO Vacuumed table table=pending_closures31282026/11/12 00:23:35 INFO Vacuumed table table=pending_objects31292026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads31302026/11/12 00:23:35 INFO Vacuumed table table=closures31312026/11/12 00:23:35 INFO Vacuumed table table=objects31322026/11/12 00:23:35 INFO Received uploads request method=POST path=/31332026/11/12 00:23:35 INFO Received uploads request method=POST path=/31342026/11/12 00:23:35 INFO Received complete push request method=POST path=/31352026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031362026/11/12 00:23:35 INFO Completed push id=4 objects=6 duration=1.626555ms31372026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period31382026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031392026/11/12 00:23:35 INFO Vacuumed table table=pending_closures31402026/11/12 00:23:35 INFO Vacuumed table table=pending_objects31412026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads31422026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031432026/11/12 00:23:35 INFO Vacuumed table table=closures31442026/11/12 00:23:35 INFO Vacuumed table table=objects31452026/11/12 00:23:35 INFO Received uploads request method=POST path=/31462026/11/12 00:23:35 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31472026/11/12 00:23:35 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31482026/11/12 00:23:35 INFO Received complete upload request method=POST path=/31492026/11/12 00:23:35 INFO Completed upload id=12 objects=26 duration=3.824676ms31502026/11/12 00:23:35 INFO Received push request method=POST path=/31512026/11/12 00:23:35 INFO Received uploads request method=POST path=/31522026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031532026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031542026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period31552026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period31562026/11/12 00:23:35 INFO Received uploads request method=POST path=/31572026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=031582026/11/12 00:23:35 INFO Vacuumed table table=pending_closures31592026/11/12 00:23:35 INFO Vacuumed table table=pending_objects31602026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads31612026/11/12 00:23:35 INFO Vacuumed table table=closures31622026/11/12 00:23:35 INFO Vacuumed table table=objects31632026/11/12 00:23:35 INFO Received complete push request method=POST path=/31642026/11/12 00:23:35 INFO Completed push id=14 objects=3 duration=3.29337ms31652026/11/12 00:23:35 INFO Received complete push request method=POST path=/31662026/11/12 00:23:35 INFO Completed push id=16 objects=8 duration=2.0969ms31672026/11/12 00:23:35 INFO Received complete upload request method=POST path=/31682026/11/12 00:23:35 INFO Completed upload id=21 objects=7 duration=2.765025ms31692026/11/12 00:23:35 INFO Received complete push request method=POST path=/31702026/11/12 00:23:35 INFO Completed push id=21 objects=9 duration=2.227341ms31712026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031722026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period31732026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031742026/11/12 00:23:35 INFO Vacuumed table table=pending_closures31752026/11/12 00:23:35 INFO Vacuumed table table=pending_objects31762026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads31772026/11/12 00:23:35 INFO Vacuumed table table=closures31782026/11/12 00:23:35 INFO Vacuumed table table=objects31792026/11/12 00:23:35 INFO Received push request method=POST path=/31802026/11/12 00:23:35 INFO Received uploads request method=POST path=/31812026/11/12 00:23:35 INFO Received complete push request method=POST path=/31822026/11/12 00:23:35 INFO Completed push id=11 objects=6 duration=3.298491ms31832026/11/12 00:23:35 INFO Received uploads request method=POST path=/31842026/11/12 00:23:35 INFO Received complete push request method=POST path=/31852026/11/12 00:23:35 INFO Completed push id=18 objects=6 duration=1.13013ms31862026/11/12 00:23:35 INFO Received uploads request method=POST path=/31872026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031882026/11/12 00:23:35 INFO Received complete upload request method=POST path=/31892026/11/12 00:23:35 INFO Received push request method=POST path=/31902026/11/12 00:23:35 INFO Completed upload id=27 objects=4 duration=1.885458ms31912026/11/12 00:23:35 INFO Received complete push request method=POST path=/31922026/11/12 00:23:35 INFO Completed push id=13 objects=11 duration=1.776097ms31932026/11/12 00:23:35 INFO Received push request method=POST path=/31942026/11/12 00:23:35 INFO Received uploads request method=POST path=/31952026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=031962026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period31972026/11/12 00:23:35 INFO Received complete upload request method=POST path=/31982026/11/12 00:23:35 INFO Completed upload id=13 objects=8 duration=2.11074ms31992026/11/12 00:23:35 INFO Received uploads request method=POST path=/32002026/11/12 00:23:35 INFO Received uploads request method=POST path=/32012026/11/12 00:23:35 INFO Received push request method=POST path=/32022026/11/12 00:23:35 INFO Received complete push request method=POST path=/32032026/11/12 00:23:35 INFO Completed push id=5 objects=28 duration=2.381882ms32042026/11/12 00:23:35 INFO Received complete push request method=POST path=/32052026/11/12 00:23:35 INFO Completed push id=10 objects=2 duration=2.13308ms32062026/11/12 00:23:35 INFO Received complete upload request method=POST path=/32072026/11/12 00:23:35 INFO Completed upload id=17 objects=5 duration=1.027109ms32082026/11/12 00:23:35 INFO Received complete push request method=POST path=/32092026/11/12 00:23:35 INFO Completed push id=19 objects=3 duration=1.179091ms32102026/11/12 00:23:35 INFO Received complete push request method=POST path=/32112026/11/12 00:23:35 INFO Completed push id=20 objects=4 duration=1.177211ms32122026/11/12 00:23:35 INFO Received complete upload request method=POST path=/32132026/11/12 00:23:35 INFO Completed upload id=22 objects=8 duration=1.006149ms32142026/11/12 00:23:35 INFO Received complete push request method=POST path=/32152026/11/12 00:23:35 INFO Completed push id=23 objects=5 duration=1.139571ms32162026/11/12 00:23:35 INFO Received complete upload request method=POST path=/32172026/11/12 00:23:35 INFO Completed upload id=24 objects=8 duration=2.832586ms32182026/11/12 00:23:35 INFO Received complete upload request method=POST path=/32192026/11/12 00:23:35 INFO Completed upload id=28 objects=5 duration=1.217892ms32202026/11/12 00:23:35 INFO Received complete upload request method=POST path=/32212026/11/12 00:23:35 INFO Completed upload id=30 objects=5 duration=1.018929ms32222026/11/12 00:23:36 INFO Received complete push request method=POST path=/32232026/11/12 00:23:36 INFO Completed push id=31 objects=3 duration=1.17277ms32242026/11/12 00:23:36 INFO Received complete push request method=POST path=/32252026/11/12 00:23:36 INFO Completed push id=32 objects=6 duration=1.230911ms32262026/11/12 00:23:36 INFO Received complete upload request method=POST path=/32272026/11/12 00:23:36 INFO Completed upload id=33 objects=7 duration=1.013489ms32282026/11/12 00:23:36 INFO Received complete upload request method=POST path=/32292026/11/12 00:23:36 INFO Completed upload id=34 objects=3 duration=1.06407ms32302026/11/12 00:23:36 INFO Received complete push request method=POST path=/32312026/11/12 00:23:36 INFO Completed push id=35 objects=11 duration=1.283972ms32322026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=032332026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period32342026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032352026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032362026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032372026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032382026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032392026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032402026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32412026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32422026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32432026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32442026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32452026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032462026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32472026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32482026/11/12 00:23:36 INFO Vacuumed table table=closures32492026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32502026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32512026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32522026/11/12 00:23:36 INFO Vacuumed table table=objects32532026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32542026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32552026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32562026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32572026/11/12 00:23:36 INFO Vacuumed table table=closures32582026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32592026/11/12 00:23:36 INFO Vacuumed table table=closures32602026/11/12 00:23:36 INFO Vacuumed table table=closures32612026/11/12 00:23:36 INFO Vacuumed table table=objects32622026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32632026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32642026/11/12 00:23:36 INFO Vacuumed table table=objects32652026/11/12 00:23:36 INFO Vacuumed table table=objects32662026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32672026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32682026/11/12 00:23:36 INFO Vacuumed table table=closures32692026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32702026/11/12 00:23:36 INFO Vacuumed table table=closures32712026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period32722026/11/12 00:23:36 INFO Vacuumed table table=objects32732026/11/12 00:23:36 INFO Vacuumed table table=objects32742026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32752026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032762026/11/12 00:23:36 INFO Vacuumed table table=closures32772026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32782026/11/12 00:23:36 INFO Received push request method=POST path=/32792026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period32802026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32812026/11/12 00:23:36 INFO Vacuumed table table=objects32822026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32832026/11/12 00:23:36 INFO Vacuumed table table=closures32842026/11/12 00:23:36 INFO Vacuumed table table=objects32852026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32862026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=032872026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period32882026/11/12 00:23:36 INFO Received complete upload request method=POST path=/32892026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032902026/11/12 00:23:36 INFO Vacuumed table table=pending_closures32912026/11/12 00:23:36 INFO Vacuumed table table=pending_objects32922026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads32932026/11/12 00:23:36 INFO Completed upload id=20 objects=3 duration=2.500303ms32942026/11/12 00:23:36 INFO Vacuumed table table=closures32952026/11/12 00:23:36 INFO Received push request method=POST path=/32962026/11/12 00:23:36 INFO Vacuumed table table=objects32972026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=032982026/11/12 00:23:36 INFO Received uploads request method=POST path=/32992026/11/12 00:23:36 INFO Vacuumed table table=pending_closures33002026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=033012026/11/12 00:23:36 INFO Received complete push request method=POST path=/33022026/11/12 00:23:36 INFO Vacuumed table table=pending_objects33032026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads33042026/11/12 00:23:36 INFO Vacuumed table table=closures33052026/11/12 00:23:36 INFO Vacuumed table table=pending_closures33062026/11/12 00:23:36 INFO Completed push id=8 objects=6 duration=3.035348ms33072026/11/12 00:23:36 INFO Vacuumed table table=objects33082026/11/12 00:23:36 INFO Vacuumed table table=pending_objects33092026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads33102026/11/12 00:23:36 INFO Received uploads request method=POST path=/33112026/11/12 00:23:36 INFO Vacuumed table table=closures33122026/11/12 00:23:36 INFO Received push request method=POST path=/33132026/11/12 00:23:36 INFO Vacuumed table table=objects33142026/11/12 00:23:36 INFO Received uploads request method=POST path=/33152026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=033162026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period33172026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033182026/11/12 00:23:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33192026/11/12 00:23:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."33202026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33212026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33222026/11/12 00:23:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33232026/11/12 00:23:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33242026/11/12 00:23:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."33252026/11/12 00:23:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."33262026/11/12 00:23:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33272026/11/12 00:23:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."33282026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."33292026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33302026/11/12 00:23:36 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."33312026/11/12 00:23:36 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."33322026/11/12 00:23:36 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33332026/11/12 00:23:36 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."33342026/11/12 00:23:36 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."33352026/11/12 00:23:36 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33362026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33372026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33382026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33392026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33402026/11/12 00:23:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33412026/11/12 00:23:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33422026/11/12 00:23:36 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33432026/11/12 00:23:36 INFO Vacuumed table table=pending_closures33442026/11/12 00:23:36 INFO Vacuumed table table=pending_objects33452026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads33462026/11/12 00:23:36 INFO Vacuumed table table=closures33472026/11/12 00:23:36 INFO Vacuumed table table=objects33482026/11/12 00:23:36 INFO Received push request method=POST path=/33492026/11/12 00:23:36 INFO Received uploads request method=POST path=/33502026/11/12 00:23:36 INFO Received complete push request method=POST path=/33512026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33522026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33532026/11/12 00:23:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33542026/11/12 00:23:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33552026/11/12 00:23:36 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."33562026/11/12 00:23:36 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33572026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."33582026/11/12 00:23:36 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33592026/11/12 00:23:36 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33602026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33612026/11/12 00:23:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33622026/11/12 00:23:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33632026/11/12 00:23:36 INFO Completed push id=16 objects=14 duration=2.17352ms33642026/11/12 00:23:36 INFO Received push request method=POST path=/33652026/11/12 00:23:36 INFO Received uploads request method=POST path=/33662026/11/12 00:23:36 INFO Received uploads request method=POST path=/33672026/11/12 00:23:36 INFO Received complete push request method=POST path=/33682026/11/12 00:23:36 INFO Received complete upload request method=POST path=/33692026/11/12 00:23:36 INFO Received complete upload request method=POST path=/33702026/11/12 00:23:36 INFO Completed upload id=24 objects=7 duration=829.008µs33712026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=033722026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=033732026/11/12 00:23:36 INFO Completed upload id=19 objects=25 duration=2.22362ms33742026/11/12 00:23:36 INFO Received push request method=POST path=/33752026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period33762026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=033772026/11/12 00:23:36 INFO Completed push id=11 objects=11 duration=3.837595ms33782026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period33792026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033802026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period33812026/11/12 00:23:36 INFO Vacuumed table table=pending_closures33822026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033832026/11/12 00:23:36 INFO Vacuumed table table=pending_objects33842026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033852026/11/12 00:23:36 INFO Vacuumed table table=pending_closures33862026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads33872026/11/12 00:23:36 INFO Received push request method=POST path=/33882026/11/12 00:23:36 INFO Vacuumed table table=closures33892026/11/12 00:23:36 INFO Received push request method=POST path=/33902026/11/12 00:23:36 INFO Vacuumed table table=pending_closures33912026/11/12 00:23:36 INFO Vacuumed table table=pending_objects33922026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads33932026/11/12 00:23:36 INFO Vacuumed table table=objects33942026/11/12 00:23:36 INFO Vacuumed table table=pending_objects33952026/11/12 00:23:36 INFO Vacuumed table table=closures33962026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads33972026/11/12 00:23:36 INFO Vacuumed table table=objects33982026/11/12 00:23:36 INFO Received complete push request method=POST path=/33992026/11/12 00:23:36 INFO Vacuumed table table=closures34002026/11/12 00:23:36 INFO Vacuumed table table=objects34012026/11/12 00:23:36 INFO Received push request method=POST path=/34022026/11/12 00:23:36 INFO Completed push id=1 objects=5 duration=7.437149ms34032026/11/12 00:23:36 INFO Received uploads request method=POST path=/34042026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034052026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034062026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period34072026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034082026/11/12 00:23:36 INFO Vacuumed table table=pending_closures34092026/11/12 00:23:36 INFO Received uploads request method=POST path=/34102026/11/12 00:23:36 INFO Vacuumed table table=pending_objects34112026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads34122026/11/12 00:23:36 INFO Vacuumed table table=closures34132026/11/12 00:23:36 INFO Vacuumed table table=objects34142026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034152026/11/12 00:23:36 INFO Received push request method=POST path=/34162026/11/12 00:23:36 INFO Received push request method=POST path=/34172026/11/12 00:23:36 INFO Vacuumed table table=pending_closures34182026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034192026/11/12 00:23:36 INFO Vacuumed table table=pending_objects34202026/11/12 00:23:36 INFO Received push request method=POST path=/34212026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads34222026/11/12 00:23:36 INFO Vacuumed table table=closures34232026/11/12 00:23:36 INFO Vacuumed table table=objects34242026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034252026/11/12 00:23:36 INFO Received complete upload request method=POST path=/34262026/11/12 00:23:36 INFO Vacuumed table table=pending_closures34272026/11/12 00:23:36 INFO Vacuumed table table=pending_objects34282026/11/12 00:23:36 INFO Received push request method=POST path=/34292026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads34302026/11/12 00:23:36 INFO Vacuumed table table=closures34312026/11/12 00:23:36 INFO Vacuumed table table=objects34322026/11/12 00:23:36 INFO Completed upload id=17 objects=25 duration=2.462843ms34332026/11/12 00:23:36 INFO Received push request method=POST path=/34342026/11/12 00:23:36 INFO Received push request method=POST path=/34352026/11/12 00:23:36 INFO Received uploads request method=POST path=/34362026/11/12 00:23:36 INFO Received complete push request method=POST path=/34372026/11/12 00:23:36 INFO Received uploads request method=POST path=/34382026/11/12 00:23:36 INFO Received push request method=POST path=/34392026/11/12 00:23:36 INFO Completed push id=9 objects=24 duration=1.339593ms34402026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034412026/11/12 00:23:36 INFO Received complete push request method=POST path=/34422026/11/12 00:23:36 INFO Received uploads request method=POST path=/34432026/11/12 00:23:36 INFO Completed push id=4 objects=6 duration=2.1301ms34442026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034452026/11/12 00:23:36 INFO Received push request method=POST path=/34462026/11/12 00:23:36 INFO Received push request method=POST path=/34472026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034482026/11/12 00:23:36 INFO Vacuumed table table=pending_closures34492026/11/12 00:23:36 INFO Vacuumed table table=pending_objects34502026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads34512026/11/12 00:23:36 INFO Vacuumed table table=closures34522026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034532026/11/12 00:23:36 INFO Received complete push request method=POST path=/34542026/11/12 00:23:36 INFO Vacuumed table table=objects34552026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period34562026/11/12 00:23:36 INFO Completed push id=6 objects=8 duration=2.796006ms34572026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034582026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=034592026/11/12 00:23:36 INFO Received uploads request method=POST path=/34602026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=034612026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period34622026/11/12 00:23:36 INFO Vacuumed table table=pending_closures34632026/11/12 00:23:36 INFO Received push request method=POST path=/34642026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period34652026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034662026/11/12 00:23:36 INFO Vacuumed table table=pending_objects34672026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads34682026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034692026/11/12 00:23:36 INFO Vacuumed table table=pending_closures34702026/11/12 00:23:36 INFO Vacuumed table table=pending_closures34712026/11/12 00:23:36 INFO Vacuumed table table=closures34722026/11/12 00:23:36 INFO Vacuumed table table=pending_objects34732026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads34742026/11/12 00:23:36 INFO Vacuumed table table=pending_objects34752026/11/12 00:23:36 INFO Vacuumed table table=closures34762026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads34772026/11/12 00:23:36 INFO Received complete upload request method=POST path=/34782026/11/12 00:23:36 INFO Vacuumed table table=objects34792026/11/12 00:23:36 INFO Received uploads request method=POST path=/34802026/11/12 00:23:36 INFO Vacuumed table table=closures34812026/11/12 00:23:36 INFO Vacuumed table table=objects34822026/11/12 00:23:36 INFO Completed upload id=21 objects=2 duration=1.527574ms34832026/11/12 00:23:36 INFO Vacuumed table table=objects34842026/11/12 00:23:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34852026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34862026/11/12 00:23:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34872026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."34882026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."34892026/11/12 00:23:36 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."34902026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34912026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."34922026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34932026/11/12 00:23:36 INFO Received uploads request method=POST path=/34942026/11/12 00:23:36 INFO Received uploads request method=POST path=/34952026/11/12 00:23:36 INFO Received push request method=POST path=/34962026/11/12 00:23:36 INFO Received uploads request method=POST path=/34972026/11/12 00:23:36 INFO Received uploads request method=POST path=/34982026/11/12 00:23:36 INFO Received complete upload request method=POST path=/34992026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35002026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035012026/11/12 00:23:36 INFO Completed upload id=9 objects=2 duration=1.016749ms35022026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period35032026/11/12 00:23:36 INFO Completed upload id=5 objects=5 duration=1.630235ms35042026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35052026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035062026/11/12 00:23:36 INFO Completed upload id=25 objects=6 duration=1.425713ms35072026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period35082026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035092026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period35102026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035112026/11/12 00:23:36 INFO Received uploads request method=POST path=/35122026/11/12 00:23:36 INFO Vacuumed table table=pending_closures35132026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035142026/11/12 00:23:36 INFO Vacuumed table table=pending_objects35152026/11/12 00:23:36 INFO Vacuumed table table=pending_closures35162026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads35172026/11/12 00:23:36 INFO Vacuumed table table=closures35182026/11/12 00:23:36 INFO Vacuumed table table=pending_objects35192026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads35202026/11/12 00:23:36 INFO Received complete push request method=POST path=/35212026/11/12 00:23:36 INFO Vacuumed table table=objects35222026/11/12 00:23:36 INFO Received uploads request method=POST path=/35232026/11/12 00:23:36 INFO Vacuumed table table=closures35242026/11/12 00:23:36 INFO Received complete push request method=POST path=/35252026/11/12 00:23:36 INFO Vacuumed table table=objects35262026/11/12 00:23:36 INFO Completed push id=23 objects=28 duration=2.18416ms35272026/11/12 00:23:36 INFO Completed push id=8 objects=3 duration=1.546014ms35282026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35292026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=035302026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35312026/11/12 00:23:36 INFO Completed upload id=10 objects=10 duration=1.800197ms35322026/11/12 00:23:36 INFO Completed upload id=10 objects=2 duration=3.043868ms35332026/11/12 00:23:36 INFO Vacuumed table table=pending_closures35342026/11/12 00:23:36 INFO Vacuumed table table=pending_objects35352026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads35362026/11/12 00:23:36 INFO Vacuumed table table=closures35372026/11/12 00:23:36 INFO Vacuumed table table=objects35382026/11/12 00:23:36 INFO Received uploads request method=POST path=/35392026/11/12 00:23:36 INFO Received push request method=POST path=/35402026/11/12 00:23:36 INFO Received push request method=POST path=/35412026/11/12 00:23:36 INFO Received uploads request method=POST path=/35422026/11/12 00:23:36 INFO Received push request method=POST path=/35432026/11/12 00:23:36 INFO Received uploads request method=POST path=/35442026/11/12 00:23:36 INFO Received push request method=POST path=/35452026/11/12 00:23:36 INFO Received push request method=POST path=/35462026/11/12 00:23:36 INFO Received uploads request method=POST path=/35472026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035482026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35492026/11/12 00:23:36 INFO Completed upload id=7 objects=4 duration=1.383573ms35502026/11/12 00:23:36 INFO Received uploads request method=POST path=/35512026/11/12 00:23:36 INFO Received uploads request method=POST path=/35522026/11/12 00:23:36 INFO Received push request method=POST path=/35532026/11/12 00:23:36 INFO Received push request method=POST path=/35542026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35552026/11/12 00:23:36 INFO Received uploads request method=POST path=/35562026/11/12 00:23:36 INFO Completed upload id=27 objects=8 duration=1.315293ms35572026/11/12 00:23:36 INFO Received uploads request method=POST path=/35582026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=035592026/11/12 00:23:36 INFO Received uploads request method=POST path=/35602026/11/12 00:23:36 INFO Vacuumed table table=pending_closures35612026/11/12 00:23:36 INFO Received push request method=POST path=/35622026/11/12 00:23:36 INFO Vacuumed table table=pending_objects35632026/11/12 00:23:36 INFO Received push request method=POST path=/35642026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35652026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads35662026/11/12 00:23:36 INFO Completed upload id=13 objects=10 duration=1.107711ms35672026/11/12 00:23:36 INFO Vacuumed table table=closures35682026/11/12 00:23:36 INFO Received complete push request method=POST path=/35692026/11/12 00:23:36 INFO Vacuumed table table=objects35702026/11/12 00:23:36 INFO Completed push id=11 objects=9 duration=1.513034ms35712026/11/12 00:23:36 INFO Received push request method=POST path=/35722026/11/12 00:23:36 INFO Received push request method=POST path=/35732026/11/12 00:23:36 INFO Received uploads request method=POST path=/35742026/11/12 00:23:36 INFO Received uploads request method=POST path=/35752026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35762026/11/12 00:23:36 INFO Received push request method=POST path=/35772026/11/12 00:23:36 INFO Completed upload id=23 objects=16 duration=2.598504ms35782026/11/12 00:23:36 INFO Received complete push request method=POST path=/35792026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035802026/11/12 00:23:36 INFO Received uploads request method=POST path=/35812026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035822026/11/12 00:23:36 INFO Received push request method=POST path=/35832026/11/12 00:23:36 INFO Completed push id=7 objects=24 duration=7.296848ms35842026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period35852026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period35862026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035872026/11/12 00:23:36 INFO Received complete push request method=POST path=/35882026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035892026/11/12 00:23:36 INFO Received push request method=POST path=/35902026/11/12 00:23:36 INFO Vacuumed table table=pending_closures35912026/11/12 00:23:36 INFO Completed push id=15 objects=24 duration=1.473094ms35922026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035932026/11/12 00:23:36 INFO Vacuumed table table=pending_objects35942026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period35952026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads35962026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035972026/11/12 00:23:36 INFO Received complete upload request method=POST path=/35982026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=035992026/11/12 00:23:36 INFO Vacuumed table table=closures36002026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036012026/11/12 00:23:36 INFO Vacuumed table table=pending_closures36022026/11/12 00:23:36 INFO Vacuumed table table=pending_closures36032026/11/12 00:23:36 INFO Completed upload id=23 objects=18 duration=1.902998ms36042026/11/12 00:23:36 INFO Received push request method=POST path=/36052026/11/12 00:23:36 INFO Vacuumed table table=pending_objects36062026/11/12 00:23:36 INFO Vacuumed table table=objects36072026/11/12 00:23:36 INFO Vacuumed table table=pending_objects36082026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads36092026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads36102026/11/12 00:23:36 INFO Vacuumed table table=closures36112026/11/12 00:23:36 INFO Vacuumed table table=closures36122026/11/12 00:23:36 INFO Vacuumed table table=objects36132026/11/12 00:23:36 INFO Vacuumed table table=objects36142026/11/12 00:23:36 INFO Received uploads request method=POST path=/36152026/11/12 00:23:36 INFO Received uploads request method=POST path=/36162026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36172026/11/12 00:23:36 INFO Completed upload id=19 objects=2 duration=769.147µs36182026/11/12 00:23:36 INFO Received uploads request method=POST path=/36192026/11/12 00:23:36 INFO Received uploads request method=POST path=/36202026/11/12 00:23:36 INFO Received uploads request method=POST path=/36212026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=036222026/11/12 00:23:36 INFO Received uploads request method=POST path=/36232026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036242026/11/12 00:23:36 INFO Received complete push request method=POST path=/36252026/11/12 00:23:36 INFO Vacuumed table table=pending_closures36262026/11/12 00:23:36 INFO Received complete push request method=POST path=/36272026/11/12 00:23:36 INFO Vacuumed table table=pending_objects36282026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36292026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads36302026/11/12 00:23:36 INFO Completed push id=16 objects=4 duration=2.16592ms36312026/11/12 00:23:36 INFO Vacuumed table table=closures36322026/11/12 00:23:36 INFO Completed upload id=24 objects=7 duration=1.024829ms36332026/11/12 00:23:36 INFO Completed push id=11 objects=8 duration=2.440562ms36342026/11/12 00:23:36 INFO Vacuumed table table=objects36352026/11/12 00:23:36 INFO Received push request method=POST path=/36362026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=036372026/11/12 00:23:36 INFO Vacuumed table table=pending_closures36382026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36392026/11/12 00:23:36 INFO Received uploads request method=POST path=/36402026/11/12 00:23:36 INFO Vacuumed table table=pending_objects36412026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads36422026/11/12 00:23:36 INFO Completed upload id=9 objects=7 duration=2.439443ms36432026/11/12 00:23:36 INFO Vacuumed table table=closures36442026/11/12 00:23:36 INFO Vacuumed table table=objects36452026/11/12 00:23:36 INFO Received uploads request method=POST path=/36462026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=036472026/11/12 00:23:36 INFO Received uploads request method=POST path=/36482026/11/12 00:23:36 INFO Received complete push request method=POST path=/36492026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036502026/11/12 00:23:36 INFO Completed push id=26 objects=3 duration=994.969µs36512026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36522026/11/12 00:23:36 INFO Vacuumed table table=pending_closures36532026/11/12 00:23:36 INFO Vacuumed table table=pending_objects36542026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads36552026/11/12 00:23:36 INFO Completed upload id=25 objects=16 duration=1.547495ms36562026/11/12 00:23:36 INFO Vacuumed table table=closures36572026/11/12 00:23:36 INFO Vacuumed table table=objects36582026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36592026/11/12 00:23:36 INFO Completed upload id=22 objects=22 duration=1.265652ms36602026/11/12 00:23:36 INFO Received push request method=POST path=/36612026/11/12 00:23:36 INFO Received complete push request method=POST path=/36622026/11/12 00:23:36 INFO Completed push id=14 objects=3 duration=1.313152ms36632026/11/12 00:23:36 INFO Received push request method=POST path=/36642026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36652026/11/12 00:23:36 INFO Completed upload id=20 objects=5 duration=822.008µs36662026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36672026/11/12 00:23:36 INFO Received complete push request method=POST path=/36682026/11/12 00:23:36 INFO Completed upload id=28 objects=8 duration=969.849µs36692026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=036702026/11/12 00:23:36 INFO Completed push id=13 objects=12 duration=3.151249ms36712026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period36722026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036732026/11/12 00:23:36 INFO Received push request method=POST path=/36742026/11/12 00:23:36 INFO Vacuumed table table=pending_closures36752026/11/12 00:23:36 INFO Received push request method=POST path=/36762026/11/12 00:23:36 INFO Vacuumed table table=pending_objects36772026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads36782026/11/12 00:23:36 INFO Vacuumed table table=closures36792026/11/12 00:23:36 INFO Vacuumed table table=objects36802026/11/12 00:23:36 INFO Received push request method=POST path=/36812026/11/12 00:23:36 INFO Received uploads request method=POST path=/36822026/11/12 00:23:36 INFO Received uploads request method=POST path=/36832026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=036842026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36852026/11/12 00:23:36 INFO Received complete push request method=POST path=/36862026/11/12 00:23:36 INFO Completed upload id=29 objects=10 duration=815.948µs36872026/11/12 00:23:36 INFO Completed push id=13 objects=6 duration=2.572464ms36882026/11/12 00:23:36 INFO Received complete push request method=POST path=/36892026/11/12 00:23:36 INFO Received complete push request method=POST path=/36902026/11/12 00:23:36 INFO Received push request method=POST path=/36912026/11/12 00:23:36 INFO Completed push id=26 objects=5 duration=1.787296ms36922026/11/12 00:23:36 INFO Completed push id=14 objects=4 duration=2.395302ms36932026/11/12 00:23:36 INFO Received push request method=POST path=/36942026/11/12 00:23:36 INFO Received complete push request method=POST path=/36952026/11/12 00:23:36 INFO Completed push id=13 objects=2 duration=783.527µs36962026/11/12 00:23:36 INFO Received push request method=POST path=/36972026/11/12 00:23:36 INFO Received complete upload request method=POST path=/36982026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036992026/11/12 00:23:36 INFO Completed upload id=30 objects=20 duration=791.087µs37002026/11/12 00:23:36 INFO Vacuumed table table=pending_closures37012026/11/12 00:23:36 INFO Vacuumed table table=pending_objects37022026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads37032026/11/12 00:23:36 INFO Vacuumed table table=closures37042026/11/12 00:23:36 INFO Vacuumed table table=objects37052026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=037062026/11/12 00:23:36 INFO Received push request method=POST path=/37072026/11/12 00:23:36 INFO Received complete upload request method=POST path=/37082026/11/12 00:23:36 INFO Completed upload id=24 objects=14 duration=905.668µs37092026/11/12 00:23:36 INFO Received complete upload request method=POST path=/37102026/11/12 00:23:36 INFO Received push request method=POST path=/37112026/11/12 00:23:36 INFO Received push request method=POST path=/37122026/11/12 00:23:36 INFO Completed upload id=27 objects=13 duration=1.214351ms37132026/11/12 00:23:36 INFO Received push request method=POST path=/37142026/11/12 00:23:36 INFO Received complete push request method=POST path=/37152026/11/12 00:23:36 INFO Completed push id=31 objects=11 duration=1.09553ms37162026/11/12 00:23:36 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."37172026/11/12 00:23:36 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37182026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37192026/11/12 00:23:36 INFO Received push request method=POST path=/37202026/11/12 00:23:36 INFO Received complete push request method=POST path=/37212026/11/12 00:23:36 INFO Completed push id=7 objects=15 duration=2.364702ms37222026/11/12 00:23:36 INFO Received complete push request method=POST path=/37232026/11/12 00:23:36 INFO Received push request method=POST path=/37242026/11/12 00:23:36 INFO Completed push id=19 objects=3 duration=1.094151ms37252026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=037262026/11/12 00:23:36 INFO Received push request method=POST path=/37272026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=037282026/11/12 00:23:36 INFO Received complete upload request method=POST path=/37292026/11/12 00:23:36 INFO Completed upload id=32 objects=2 duration=815.307µs37302026/11/12 00:23:36 INFO Received uploads request method=POST path=/37312026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period37322026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037332026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037342026/11/12 00:23:36 INFO Vacuumed table table=pending_closures37352026/11/12 00:23:36 INFO Vacuumed table table=pending_closures37362026/11/12 00:23:36 INFO Vacuumed table table=pending_objects37372026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads37382026/11/12 00:23:36 INFO Vacuumed table table=pending_objects37392026/11/12 00:23:36 INFO Vacuumed table table=closures37402026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads37412026/11/12 00:23:36 INFO Vacuumed table table=closures37422026/11/12 00:23:36 INFO Vacuumed table table=objects37432026/11/12 00:23:36 INFO Vacuumed table table=objects37442026/11/12 00:23:36 INFO Received complete push request method=POST path=/37452026/11/12 00:23:36 INFO Received uploads request method=POST path=/37462026/11/12 00:23:36 INFO Completed push id=33 objects=3 duration=974.609µs37472026/11/12 00:23:36 INFO Received complete push request method=POST path=/37482026/11/12 00:23:36 INFO Completed push id=17 objects=6 duration=2.10176ms37492026/11/12 00:23:36 INFO Received push request method=POST path=/37502026/11/12 00:23:36 INFO Received uploads request method=POST path=/37512026/11/12 00:23:36 INFO Received push request method=POST path=/37522026/11/12 00:23:36 INFO Received uploads request method=POST path=/37532026/11/12 00:23:36 INFO Received uploads request method=POST path=/37542026/11/12 00:23:36 INFO Received complete upload request method=POST path=/37552026/11/12 00:23:36 INFO Completed upload id=34 objects=3 duration=714.446µs37562026/11/12 00:23:36 INFO Received complete push request method=POST path=/37572026/11/12 00:23:36 INFO Completed push id=10 objects=2 duration=1.574634ms37582026/11/12 00:23:36 INFO Received complete push request method=POST path=/37592026/11/12 00:23:36 INFO Received push request method=POST path=/37602026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=037612026/11/12 00:23:36 INFO Completed push id=19 objects=6 duration=3.24183ms37622026/11/12 00:23:36 INFO Received push request method=POST path=/37632026/11/12 00:23:36 INFO Received uploads request method=POST path=/37642026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period37652026/11/12 00:23:36 INFO Received complete upload request method=POST path=/37662026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=037672026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037682026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=037692026/11/12 00:23:36 INFO Completed upload id=22 objects=5 duration=2.107519ms37702026/11/12 00:23:36 INFO Vacuumed table table=pending_closures37712026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period37722026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period37732026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037742026/11/12 00:23:36 INFO Vacuumed table table=pending_objects37752026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads37762026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037772026/11/12 00:23:36 INFO Vacuumed table table=pending_closures37782026/11/12 00:23:36 INFO Vacuumed table table=closures37792026/11/12 00:23:36 INFO Vacuumed table table=objects37802026/11/12 00:23:36 INFO Vacuumed table table=pending_closures37812026/11/12 00:23:36 INFO Received complete upload request method=POST path=/37822026/11/12 00:23:36 INFO Vacuumed table table=pending_objects37832026/11/12 00:23:36 INFO Vacuumed table table=pending_objects37842026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads37852026/11/12 00:23:36 INFO Received complete push request method=POST path=/37862026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads37872026/11/12 00:23:36 INFO Completed upload id=12 objects=21 duration=1.534754ms37882026/11/12 00:23:36 INFO Vacuumed table table=closures37892026/11/12 00:23:36 INFO Vacuumed table table=closures37902026/11/12 00:23:36 INFO Vacuumed table table=objects37912026/11/12 00:23:36 INFO Vacuumed table table=objects37922026/11/12 00:23:36 INFO Completed push id=4 objects=7 duration=1.892537ms37932026/11/12 00:23:36 INFO Received push request method=POST path=/37942026/11/12 00:23:36 INFO Received uploads request method=POST path=/37952026/11/12 00:23:36 INFO Received uploads request method=POST path=/37962026/11/12 00:23:36 INFO Received complete push request method=POST path=/37972026/11/12 00:23:36 INFO Completed push id=28 objects=3 duration=1.556194ms37982026/11/12 00:23:36 INFO Received uploads request method=POST path=/37992026/11/12 00:23:36 INFO Received push request method=POST path=/38002026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038012026/11/12 00:23:36 INFO Received uploads request method=POST path=/38022026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038032026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038042026/11/12 00:23:36 INFO Received complete upload request method=POST path=/38052026/11/12 00:23:36 INFO Vacuumed table table=pending_closures38062026/11/12 00:23:36 INFO Completed upload id=25 objects=4 duration=1.050169ms38072026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period38082026/11/12 00:23:36 INFO Vacuumed table table=pending_objects38092026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038102026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads38112026/11/12 00:23:36 INFO Received complete push request method=POST path=/38122026/11/12 00:23:36 INFO Vacuumed table table=closures38132026/11/12 00:23:36 INFO Vacuumed table table=objects38142026/11/12 00:23:36 INFO Completed push id=29 objects=15 duration=1.11313ms38152026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038162026/11/12 00:23:36 INFO Received uploads request method=POST path=/38172026/11/12 00:23:36 INFO Vacuumed table table=pending_closures38182026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period38192026/11/12 00:23:36 INFO Vacuumed table table=pending_objects38202026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads38212026/11/12 00:23:36 INFO Vacuumed table table=closures38222026/11/12 00:23:36 INFO Vacuumed table table=objects38232026/11/12 00:23:36 INFO Received uploads request method=POST path=/38242026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."38252026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."38262026/11/12 00:23:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."38272026/11/12 00:23:36 INFO Received push request method=POST path=/38282026/11/12 00:23:36 INFO Received complete push request method=POST path=/38292026/11/12 00:23:36 INFO Completed push id=13 objects=23 duration=1.606495ms38302026/11/12 00:23:36 INFO Received push request method=POST path=/38312026/11/12 00:23:36 INFO Received complete push request method=POST path=/38322026/11/12 00:23:36 INFO Received complete push request method=POST path=/38332026/11/12 00:23:36 INFO Completed push id=30 objects=6 duration=1.08613ms38342026/11/12 00:23:36 INFO Completed push id=16 objects=16 duration=3.112389ms38352026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038362026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period38372026/11/12 00:23:36 INFO Received complete upload request method=POST path=/38382026/11/12 00:23:36 INFO Completed upload id=29 objects=4 duration=1.507473ms38392026/11/12 00:23:36 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."38402026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."38412026/11/12 00:23:36 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."38422026/11/12 00:23:36 INFO Received complete upload request method=POST path=/38432026/11/12 00:23:36 INFO Completed upload id=31 objects=3 duration=736.087µs38442026/11/12 00:23:36 INFO Received uploads request method=POST path=/38452026/11/12 00:23:36 INFO Received uploads request method=POST path=/38462026/11/12 00:23:36 INFO Received complete push request method=POST path=/38472026/11/12 00:23:36 INFO Received complete push request method=POST path=/38482026/11/12 00:23:36 INFO Completed push id=20 objects=8 duration=974.189µs38492026/11/12 00:23:36 INFO Completed push id=19 objects=14 duration=2.077239ms38502026/11/12 00:23:36 INFO Received push request method=POST path=/38512026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038522026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038532026/11/12 00:23:36 INFO Received complete push request method=POST path=/38542026/11/12 00:23:36 INFO Completed push id=7 objects=7 duration=1.247391ms38552026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period38562026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038572026/11/12 00:23:36 INFO Received uploads request method=POST path=/38582026/11/12 00:23:36 INFO Vacuumed table table=pending_closures38592026/11/12 00:23:36 INFO Received uploads request method=POST path=/38602026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038612026/11/12 00:23:36 INFO Received uploads request method=POST path=/38622026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038632026/11/12 00:23:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."38642026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."38652026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."38662026/11/12 00:23:36 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."38672026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."38682026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."38692026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38702026/11/12 00:23:36 INFO Vacuumed table table=pending_objects38712026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads38722026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period38732026/11/12 00:23:36 INFO Vacuumed table table=closures38742026/11/12 00:23:36 INFO Received uploads request method=POST path=/38752026/11/12 00:23:36 INFO Vacuumed table table=objects38762026/11/12 00:23:36 INFO Received uploads request method=POST path=/38772026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=038782026/11/12 00:23:36 INFO Vacuumed table table=pending_closures38792026/11/12 00:23:36 INFO Vacuumed table table=pending_objects38802026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads38812026/11/12 00:23:36 INFO Vacuumed table table=closures38822026/11/12 00:23:36 INFO Received uploads request method=POST path=/38832026/11/12 00:23:36 INFO Vacuumed table table=objects38842026/11/12 00:23:36 INFO Received push request method=POST path=/38852026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=038862026/11/12 00:23:36 INFO Received complete push request method=POST path=/38872026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period38882026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038892026/11/12 00:23:36 INFO Completed push id=11 objects=6 duration=1.736496ms38902026/11/12 00:23:36 INFO Received complete push request method=POST path=/38912026/11/12 00:23:36 INFO Vacuumed table table=pending_closures38922026/11/12 00:23:36 INFO Vacuumed table table=pending_objects38932026/11/12 00:23:36 INFO Completed push id=15 objects=7 duration=2.12604ms38942026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads38952026/11/12 00:23:36 INFO Vacuumed table table=closures38962026/11/12 00:23:36 INFO Vacuumed table table=objects38972026/11/12 00:23:36 INFO Received uploads request method=POST path=/38982026/11/12 00:23:36 INFO Received push request method=POST path=/38992026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=039002026/11/12 00:23:36 INFO Vacuumed table table=pending_closures39012026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39022026/11/12 00:23:36 INFO Vacuumed table table=pending_objects39032026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads39042026/11/12 00:23:36 INFO Received uploads request method=POST path=/39052026/11/12 00:23:36 INFO Vacuumed table table=closures39062026/11/12 00:23:36 INFO Completed upload id=18 objects=11 duration=1.435753ms39072026/11/12 00:23:36 INFO Vacuumed table table=objects39082026/11/12 00:23:36 INFO Received uploads request method=POST path=/39092026/11/12 00:23:36 INFO Received complete push request method=POST path=/39102026/11/12 00:23:36 INFO Received push request method=POST path=/39112026/11/12 00:23:36 INFO Completed push id=20 objects=13 duration=1.12057ms39122026/11/12 00:23:36 INFO Received complete push request method=POST path=/39132026/11/12 00:23:36 INFO Completed push id=27 objects=11 duration=934.809µs39142026/11/12 00:23:36 INFO Received uploads request method=POST path=/39152026/11/12 00:23:36 INFO Received uploads request method=POST path=/39162026/11/12 00:23:36 INFO Received push request method=POST path=/39172026/11/12 00:23:36 INFO Received push request method=POST path=/39182026/11/12 00:23:36 INFO Received complete push request method=POST path=/39192026/11/12 00:23:36 INFO Received uploads request method=POST path=/39202026/11/12 00:23:36 INFO Completed push id=22 objects=4 duration=1.07507ms39212026/11/12 00:23:36 INFO Received push request method=POST path=/39222026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39232026/11/12 00:23:36 INFO Completed upload id=29 objects=26 duration=913.708µs39242026/11/12 00:23:36 INFO Received uploads request method=POST path=/39252026/11/12 00:23:36 INFO Received uploads request method=POST path=/39262026/11/12 00:23:36 INFO Received push request method=POST path=/39272026/11/12 00:23:36 INFO Received uploads request method=POST path=/39282026/11/12 00:23:36 INFO Received complete push request method=POST path=/39292026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39302026/11/12 00:23:36 INFO Completed upload id=23 objects=7 duration=1.017949ms39312026/11/12 00:23:36 INFO Received complete push request method=POST path=/39322026/11/12 00:23:36 INFO Completed push id=30 objects=8 duration=1.023309ms39332026/11/12 00:23:36 INFO Completed push id=18 objects=6 duration=2.836446ms39342026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=039352026/11/12 00:23:36 INFO Received complete push request method=POST path=/39362026/11/12 00:23:36 INFO Received push request method=POST path=/39372026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039382026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39392026/11/12 00:23:36 INFO Vacuumed table table=pending_closures39402026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=039412026/11/12 00:23:36 INFO Completed push id=31 objects=14 duration=2.333601ms39422026/11/12 00:23:36 INFO Vacuumed table table=pending_objects39432026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads39442026/11/12 00:23:36 INFO Completed upload id=15 objects=4 duration=2.307762ms39452026/11/12 00:23:36 INFO Vacuumed table table=closures39462026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period39472026/11/12 00:23:36 INFO Vacuumed table table=objects39482026/11/12 00:23:36 INFO Received complete push request method=POST path=/39492026/11/12 00:23:36 INFO Completed push id=24 objects=5 duration=1.112971ms39502026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39512026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39522026/11/12 00:23:36 INFO Completed upload id=23 objects=9 duration=970.109µs39532026/11/12 00:23:36 INFO Completed upload id=31 objects=5 duration=820.428µs39542026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39552026/11/12 00:23:36 INFO Completed upload id=22 objects=2 duration=960.289µs39562026/11/12 00:23:36 INFO Received push request method=POST path=/39572026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39582026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=039592026/11/12 00:23:36 INFO Completed upload id=25 objects=2 duration=773.528µs39602026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period39612026/11/12 00:23:36 INFO Received complete push request method=POST path=/39622026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=039632026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."39642026/11/12 00:23:36 INFO Completed push id=32 objects=3 duration=976.969µs39652026/11/12 00:23:36 INFO Received uploads request method=POST path=/39662026/11/12 00:23:36 INFO Received uploads request method=POST path=/39672026/11/12 00:23:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."39682026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."39692026/11/12 00:23:36 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."39702026/11/12 00:23:36 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."39712026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."39722026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."39732026/11/12 00:23:36 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39742026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039752026/11/12 00:23:36 INFO Vacuumed table table=pending_closures39762026/11/12 00:23:36 INFO Vacuumed table table=pending_objects39772026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39782026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads39792026/11/12 00:23:36 INFO Vacuumed table table=closures39802026/11/12 00:23:36 INFO Completed upload id=26 objects=20 duration=884.768µs39812026/11/12 00:23:36 INFO Vacuumed table table=objects39822026/11/12 00:23:36 INFO Received push request method=POST path=/39832026/11/12 00:23:36 INFO Received uploads request method=POST path=/39842026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=039852026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period39862026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039872026/11/12 00:23:36 INFO Received complete upload request method=POST path=/39882026/11/12 00:23:36 INFO Completed upload id=16 objects=3 duration=856.208µs39892026/11/12 00:23:36 INFO Vacuumed table table=pending_closures39902026/11/12 00:23:36 INFO Received uploads request method=POST path=/39912026/11/12 00:23:36 INFO Vacuumed table table=pending_objects39922026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads39932026/11/12 00:23:36 INFO Vacuumed table table=closures39942026/11/12 00:23:36 INFO Vacuumed table table=objects39952026/11/12 00:23:36 INFO Received uploads request method=POST path=/39962026/11/12 00:23:36 INFO Received complete push request method=POST path=/39972026/11/12 00:23:36 INFO Received complete push request method=POST path=/39982026/11/12 00:23:36 INFO Completed push id=28 objects=7 duration=1.04343ms39992026/11/12 00:23:36 INFO Completed push id=15 objects=4 duration=1.06345ms40002026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40012026/11/12 00:23:36 INFO Completed upload id=21 objects=3 duration=850.428µs40022026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=040032026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40042026/11/12 00:23:36 INFO Completed upload id=13 objects=14 duration=2.643365ms40052026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040062026/11/12 00:23:36 INFO Vacuumed table table=pending_closures40072026/11/12 00:23:36 INFO Received push request method=POST path=/40082026/11/12 00:23:36 INFO Vacuumed table table=pending_objects40092026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads40102026/11/12 00:23:36 INFO Vacuumed table table=closures40112026/11/12 00:23:36 INFO Received complete push request method=POST path=/40122026/11/12 00:23:36 INFO Vacuumed table table=objects40132026/11/12 00:23:36 INFO Completed push id=7 objects=7 duration=1.421413ms40142026/11/12 00:23:36 INFO Received push request method=POST path=/40152026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=040162026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period40172026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40182026/11/12 00:23:36 INFO Completed upload id=30 objects=6 duration=1.277072ms40192026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=040202026/11/12 00:23:36 INFO Received uploads request method=POST path=/40212026/11/12 00:23:36 INFO Received uploads request method=POST path=/40222026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040232026/11/12 00:23:36 INFO Received push request method=POST path=/40242026/11/12 00:23:36 INFO Received push request method=POST path=/40252026/11/12 00:23:36 INFO Vacuumed table table=pending_closures40262026/11/12 00:23:36 INFO Received complete push request method=POST path=/40272026/11/12 00:23:36 INFO Vacuumed table table=pending_objects40282026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads40292026/11/12 00:23:36 INFO Completed push id=12 objects=6 duration=1.247872ms40302026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=040312026/11/12 00:23:36 INFO Vacuumed table table=closures40322026/11/12 00:23:36 INFO Vacuumed table table=pending_closures40332026/11/12 00:23:36 INFO Vacuumed table table=objects40342026/11/12 00:23:36 INFO Vacuumed table table=pending_objects40352026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads40362026/11/12 00:23:36 INFO Vacuumed table table=closures40372026/11/12 00:23:36 INFO Vacuumed table table=objects40382026/11/12 00:23:36 INFO Received complete push request method=POST path=/40392026/11/12 00:23:36 INFO Completed push id=31 objects=5 duration=857.948µs40402026/11/12 00:23:36 INFO Received push request method=POST path=/40412026/11/12 00:23:36 INFO Received complete push request method=POST path=/40422026/11/12 00:23:36 INFO Received complete push request method=POST path=/40432026/11/12 00:23:36 INFO Completed push id=31 objects=6 duration=3.405072ms40442026/11/12 00:23:36 INFO Completed push id=14 objects=3 duration=1.06699ms40452026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40462026/11/12 00:23:36 INFO Received complete push request method=POST path=/40472026/11/12 00:23:36 INFO Completed upload id=18 objects=4 duration=2.001799ms40482026/11/12 00:23:36 INFO Completed push id=16 objects=26 duration=1.245572ms40492026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=040502026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period40512026/11/12 00:23:36 INFO Received complete push request method=POST path=/40522026/11/12 00:23:36 INFO Completed push id=32 objects=26 duration=1.300653ms40532026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040542026/11/12 00:23:36 INFO Vacuumed table table=pending_closures40552026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=040562026/11/12 00:23:36 INFO Vacuumed table table=pending_objects40572026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads40582026/11/12 00:23:36 INFO Vacuumed table table=closures40592026/11/12 00:23:36 INFO Vacuumed table table=objects40602026/11/12 00:23:36 INFO Received uploads request method=POST path=/40612026/11/12 00:23:36 INFO Received complete push request method=POST path=/40622026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040632026/11/12 00:23:36 INFO Vacuumed table table=pending_closures40642026/11/12 00:23:36 INFO Completed push id=13 objects=22 duration=1.307812ms40652026/11/12 00:23:36 INFO Vacuumed table table=pending_objects40662026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads40672026/11/12 00:23:36 INFO Vacuumed table table=closures40682026/11/12 00:23:36 INFO Vacuumed table table=objects40692026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=040702026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period40712026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40722026/11/12 00:23:36 INFO Completed upload id=33 objects=2 duration=825.347µs40732026/11/12 00:23:36 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040742026/11/12 00:23:36 INFO Received push request method=POST path=/40752026/11/12 00:23:36 INFO Vacuumed table table=pending_closures40762026/11/12 00:23:36 INFO Vacuumed table table=pending_objects40772026/11/12 00:23:36 INFO Vacuumed table table=multipart_uploads40782026/11/12 00:23:36 INFO Vacuumed table table=closures40792026/11/12 00:23:36 INFO Vacuumed table table=objects40802026/11/12 00:23:36 INFO Received complete push request method=POST path=/40812026/11/12 00:23:36 INFO Completed push id=17 objects=6 duration=1.739277ms40822026/11/12 00:23:36 INFO Received complete push request method=POST path=/40832026/11/12 00:23:36 INFO Received push request method=POST path=/40842026/11/12 00:23:36 INFO Completed push id=21 objects=6 duration=1.164291ms40852026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40862026/11/12 00:23:36 INFO Completed upload id=20 objects=4 duration=952.529µs40872026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40882026/11/12 00:23:36 INFO Received complete push request method=POST path=/40892026/11/12 00:23:36 INFO Completed upload id=34 objects=5 duration=863.068µs40902026/11/12 00:23:36 INFO Completed push id=22 objects=3 duration=2.937767ms40912026/11/12 00:23:36 INFO Received complete push request method=POST path=/40922026/11/12 00:23:36 INFO Completed push id=27 objects=6 duration=2.588124ms40932026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40942026/11/12 00:23:36 INFO Completed upload id=22 objects=7 duration=1.01221ms40952026/11/12 00:23:36 INFO Received push request method=POST path=/40962026/11/12 00:23:36 INFO Received complete upload request method=POST path=/40972026/11/12 00:23:36 INFO Received uploads request method=POST path=/40982026/11/12 00:23:36 INFO Completed upload id=35 objects=11 duration=1.287332ms40992026/11/12 00:23:36 INFO Received push request method=POST path=/41002026/11/12 00:23:36 INFO Received complete push request method=POST path=/41012026/11/12 00:23:36 INFO Completed push id=24 objects=4 duration=907.368µs41022026/11/12 00:23:36 INFO Received complete push request method=POST path=/41032026/11/12 00:23:36 INFO Completed push id=16 objects=5 duration=1.04521ms41042026/11/12 00:23:36 INFO Received complete upload request method=POST path=/41052026/11/12 00:23:36 INFO Completed upload id=36 objects=3 duration=635.666µs41062026/11/12 00:23:37 INFO Received uploads request method=POST path=/41072026/11/12 00:23:37 INFO Received complete upload request method=POST path=/41082026/11/12 00:23:37 INFO Received complete push request method=POST path=/41092026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041102026/11/12 00:23:37 INFO Completed upload id=15 objects=2 duration=965.648µs41112026/11/12 00:23:37 INFO Received complete push request method=POST path=/41122026/11/12 00:23:37 INFO Completed push id=1 objects=9 duration=1.11711ms41132026/11/12 00:23:37 INFO Completed push id=25 objects=9 duration=960.809µs41142026/11/12 00:23:37 INFO Received uploads request method=POST path=/41152026/11/12 00:23:37 INFO Received push request method=POST path=/41162026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041172026/11/12 00:23:37 INFO Vacuumed table table=pending_closures41182026/11/12 00:23:37 INFO Vacuumed table table=pending_objects41192026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads41202026/11/12 00:23:37 INFO Vacuumed table table=closures41212026/11/12 00:23:37 INFO Vacuumed table table=objects41222026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041232026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period41242026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041252026/11/12 00:23:37 INFO Vacuumed table table=pending_closures41262026/11/12 00:23:37 INFO Received complete upload request method=POST path=/41272026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041282026/11/12 00:23:37 INFO Received complete push request method=POST path=/41292026/11/12 00:23:37 INFO Completed upload id=26 objects=3 duration=731.547µs41302026/11/12 00:23:37 INFO Completed push id=24 objects=3 duration=1.164731ms41312026/11/12 00:23:37 INFO Vacuumed table table=pending_objects41322026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads41332026/11/12 00:23:37 INFO Vacuumed table table=closures41342026/11/12 00:23:37 INFO Vacuumed table table=objects41352026/11/12 00:23:37 INFO Received uploads request method=POST path=/41362026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041372026/11/12 00:23:37 INFO Vacuumed table table=pending_closures41382026/11/12 00:23:37 INFO Vacuumed table table=pending_objects41392026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads41402026/11/12 00:23:37 INFO Received push request method=POST path=/41412026/11/12 00:23:37 INFO Vacuumed table table=closures41422026/11/12 00:23:37 INFO Vacuumed table table=objects41432026/11/12 00:23:37 INFO Received complete upload request method=POST path=/41442026/11/12 00:23:37 INFO Received push request method=POST path=/41452026/11/12 00:23:37 INFO Completed upload id=27 objects=7 duration=1.15463ms41462026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041472026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041482026/11/12 00:23:37 INFO Vacuumed table table=pending_closures41492026/11/12 00:23:37 INFO Vacuumed table table=pending_objects41502026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads41512026/11/12 00:23:37 INFO Vacuumed table table=closures41522026/11/12 00:23:37 INFO Vacuumed table table=objects41532026/11/12 00:23:37 INFO Received complete upload request method=POST path=/41542026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041552026/11/12 00:23:37 INFO Completed upload id=2 objects=7 duration=1.173851ms41562026/11/12 00:23:37 INFO Received complete upload request method=POST path=/41572026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period41582026/11/12 00:23:37 INFO Completed upload id=28 objects=3 duration=835.168µs41592026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041602026/11/12 00:23:37 INFO Vacuumed table table=pending_closures41612026/11/12 00:23:37 INFO Received push request method=POST path=/41622026/11/12 00:23:37 INFO Vacuumed table table=pending_objects41632026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads41642026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041652026/11/12 00:23:37 INFO Vacuumed table table=closures41662026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period41672026/11/12 00:23:37 INFO Vacuumed table table=objects41682026/11/12 00:23:37 INFO Received uploads request method=POST path=/41692026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041702026/11/12 00:23:37 INFO Vacuumed table table=pending_closures41712026/11/12 00:23:37 INFO Vacuumed table table=pending_objects41722026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads41732026/11/12 00:23:37 INFO Vacuumed table table=closures41742026/11/12 00:23:37 INFO Vacuumed table table=objects41752026/11/12 00:23:37 INFO Received complete push request method=POST path=/41762026/11/12 00:23:37 INFO Received uploads request method=POST path=/41772026/11/12 00:23:37 INFO Completed push id=29 objects=23 duration=1.391013ms41782026/11/12 00:23:37 INFO Received uploads request method=POST path=/41792026/11/12 00:23:37 INFO Received uploads request method=POST path=/41802026/11/12 00:23:37 INFO Received complete upload request method=POST path=/41812026/11/12 00:23:37 INFO Completed upload id=8 objects=6 duration=946.969µs41822026/11/12 00:23:37 INFO Received complete push request method=POST path=/41832026/11/12 00:23:37 INFO Received push request method=POST path=/41842026/11/12 00:23:37 INFO Received uploads request method=POST path=/41852026/11/12 00:23:37 INFO Completed push id=30 objects=6 duration=1.136551ms41862026/11/12 00:23:37 INFO Received push request method=POST path=/41872026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041882026/11/12 00:23:37 INFO Received complete upload request method=POST path=/41892026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041902026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041912026/11/12 00:23:37 INFO Vacuumed table table=pending_closures41922026/11/12 00:23:37 INFO Completed upload id=26 objects=8 duration=1.477354ms41932026/11/12 00:23:37 INFO Received complete push request method=POST path=/41942026/11/12 00:23:37 INFO Vacuumed table table=pending_objects41952026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=041962026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads41972026/11/12 00:23:37 INFO Completed push id=33 objects=3 duration=869.388µs41982026/11/12 00:23:37 INFO Vacuumed table table=closures41992026/11/12 00:23:37 INFO Vacuumed table table=objects42002026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042012026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42022026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42032026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42042026/11/12 00:23:37 INFO Vacuumed table table=closures42052026/11/12 00:23:37 INFO Vacuumed table table=objects42062026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=042072026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42082026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42092026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42102026/11/12 00:23:37 INFO Vacuumed table table=closures42112026/11/12 00:23:37 INFO Vacuumed table table=objects42122026/11/12 00:23:37 INFO Received push request method=POST path=/42132026/11/12 00:23:37 INFO Received complete push request method=POST path=/42142026/11/12 00:23:37 INFO Completed push id=28 objects=14 duration=1.028969ms42152026/11/12 00:23:37 INFO Received complete push request method=POST path=/42162026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=042172026/11/12 00:23:37 INFO Completed push id=34 objects=6 duration=1.03941ms42182026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042192026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42202026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42212026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42222026/11/12 00:23:37 INFO Vacuumed table table=closures42232026/11/12 00:23:37 INFO Vacuumed table table=objects42242026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42252026/11/12 00:23:37 INFO Completed upload id=12 objects=9 duration=1.03313ms42262026/11/12 00:23:37 INFO Received push request method=POST path=/42272026/11/12 00:23:37 INFO Received push request method=POST path=/42282026/11/12 00:23:37 INFO Received complete push request method=POST path=/42292026/11/12 00:23:37 INFO Completed push id=35 objects=8 duration=753.387µs42302026/11/12 00:23:37 INFO Received uploads request method=POST path=/42312026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42322026/11/12 00:23:37 INFO Completed upload id=33 objects=5 duration=2.229061ms42332026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=042342026/11/12 00:23:37 INFO Received push request method=POST path=/42352026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42362026/11/12 00:23:37 INFO Received uploads request method=POST path=/42372026/11/12 00:23:37 INFO Completed upload id=36 objects=9 duration=910.229µs42382026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042392026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42402026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42412026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42422026/11/12 00:23:37 INFO Completed upload id=26 objects=2 duration=1.459014ms42432026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42442026/11/12 00:23:37 INFO Vacuumed table table=closures42452026/11/12 00:23:37 INFO Vacuumed table table=objects42462026/11/12 00:23:37 INFO Received push request method=POST path=/42472026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=042482026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=042492026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period42502026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=042512026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042522026/11/12 00:23:37 INFO Received push request method=POST path=/42532026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042542026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period42552026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42562026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42572026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042582026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42592026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42602026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42612026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42622026/11/12 00:23:37 INFO Vacuumed table table=closures42632026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42642026/11/12 00:23:37 INFO Vacuumed table table=objects42652026/11/12 00:23:37 INFO Received uploads request method=POST path=/42662026/11/12 00:23:37 INFO Vacuumed table table=closures42672026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42682026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42692026/11/12 00:23:37 INFO Vacuumed table table=objects42702026/11/12 00:23:37 INFO Vacuumed table table=closures42712026/11/12 00:23:37 INFO Vacuumed table table=objects42722026/11/12 00:23:37 INFO Received uploads request method=POST path=/42732026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42742026/11/12 00:23:37 INFO Completed upload id=21 objects=25 duration=1.330752ms42752026/11/12 00:23:37 INFO Received push request method=POST path=/42762026/11/12 00:23:37 INFO Received uploads request method=POST path=/42772026/11/12 00:23:37 INFO Received push request method=POST path=/42782026/11/12 00:23:37 INFO Received push request method=POST path=/42792026/11/12 00:23:37 INFO Received uploads request method=POST path=/42802026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42812026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42822026/11/12 00:23:37 INFO Completed upload id=21 objects=9 duration=1.798916ms42832026/11/12 00:23:37 INFO Completed upload id=18 objects=22 duration=1.214211ms42842026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=042852026/11/12 00:23:37 INFO Received complete push request method=POST path=/42862026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042872026/11/12 00:23:37 INFO Vacuumed table table=pending_closures42882026/11/12 00:23:37 INFO Completed push id=26 objects=14 duration=2.411043ms42892026/11/12 00:23:37 INFO Vacuumed table table=pending_objects42902026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads42912026/11/12 00:23:37 INFO Vacuumed table table=closures42922026/11/12 00:23:37 INFO Vacuumed table table=objects42932026/11/12 00:23:37 INFO Received complete push request method=POST path=/42942026/11/12 00:23:37 INFO Received complete upload request method=POST path=/42952026/11/12 00:23:37 INFO Completed push id=25 objects=10 duration=1.636975ms42962026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=042972026/11/12 00:23:37 INFO Completed upload id=21 objects=8 duration=775.227µs42982026/11/12 00:23:37 INFO Received push request method=POST path=/42992026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=043002026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period43012026/11/12 00:23:37 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43022026/11/12 00:23:37 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."43032026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43042026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043052026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43062026/11/12 00:23:37 INFO Vacuumed table table=pending_closures43072026/11/12 00:23:37 INFO Vacuumed table table=pending_objects43082026/11/12 00:23:37 INFO Completed upload id=12 objects=6 duration=983.489µs43092026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads43102026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43112026/11/12 00:23:37 INFO Vacuumed table table=closures43122026/11/12 00:23:37 INFO Completed upload id=29 objects=5 duration=840.208µs43132026/11/12 00:23:37 INFO Vacuumed table table=objects43142026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43152026/11/12 00:23:37 INFO Completed upload id=22 objects=7 duration=917.729µs43162026/11/12 00:23:37 INFO Received push request method=POST path=/43172026/11/12 00:23:37 INFO Received uploads request method=POST path=/43182026/11/12 00:23:37 INFO Received complete push request method=POST path=/43192026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43202026/11/12 00:23:37 INFO Completed push id=17 objects=3 duration=1.701416ms43212026/11/12 00:23:37 INFO Completed upload id=19 objects=2 duration=1.946158ms43222026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43232026/11/12 00:23:37 INFO Completed upload id=30 objects=2 duration=889.268µs43242026/11/12 00:23:37 INFO Received complete push request method=POST path=/43252026/11/12 00:23:37 INFO Completed push id=24 objects=5 duration=859.248µs43262026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=043272026/11/12 00:23:37 INFO Received uploads request method=POST path=/43282026/11/12 00:23:37 INFO Received complete push request method=POST path=/43292026/11/12 00:23:37 INFO Completed push id=31 objects=20 duration=1.170971ms43302026/11/12 00:23:37 INFO Received complete push request method=POST path=/43312026/11/12 00:23:37 INFO Completed push id=23 objects=5 duration=1.193891ms43322026/11/12 00:23:37 INFO Received push request method=POST path=/43332026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43342026/11/12 00:23:37 INFO Completed upload id=25 objects=8 duration=978.049µs43352026/11/12 00:23:37 INFO Received complete push request method=POST path=/43362026/11/12 00:23:37 INFO Completed push id=33 objects=7 duration=1.837097ms43372026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43382026/11/12 00:23:37 INFO Completed upload id=32 objects=5 duration=894.509µs43392026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43402026/11/12 00:23:37 INFO Received complete push request method=POST path=/43412026/11/12 00:23:37 INFO Completed upload id=24 objects=3 duration=906.969µs43422026/11/12 00:23:37 INFO Completed push id=28 objects=11 duration=2.1908ms43432026/11/12 00:23:37 INFO Received complete push request method=POST path=/43442026/11/12 00:23:37 INFO Completed push id=26 objects=4 duration=1.14757ms43452026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43462026/11/12 00:23:37 INFO Completed upload id=26 objects=3 duration=1.596395ms43472026/11/12 00:23:37 INFO Received complete upload request method=POST path=/43482026/11/12 00:23:37 INFO Completed upload id=33 objects=3 duration=802.527µs43492026/11/12 00:23:37 INFO Received complete push request method=POST path=/43502026/11/12 00:23:37 INFO Completed push id=25 objects=4 duration=1.189351ms43512026/11/12 00:23:37 INFO Received complete push request method=POST path=/43522026/11/12 00:23:37 INFO Completed push id=30 objects=2 duration=977.41µs43532026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=043542026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period43552026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043562026/11/12 00:23:37 INFO Vacuumed table table=pending_closures43572026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=043582026/11/12 00:23:37 INFO Vacuumed table table=pending_objects43592026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads43602026/11/12 00:23:37 INFO Vacuumed table table=closures43612026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period43622026/11/12 00:23:37 INFO Vacuumed table table=objects43632026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043642026/11/12 00:23:37 INFO Vacuumed table table=pending_closures43652026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=043662026/11/12 00:23:37 INFO Vacuumed table table=pending_objects43672026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads43682026/11/12 00:23:37 INFO Vacuumed table table=closures43692026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=043702026/11/12 00:23:37 INFO Vacuumed table table=objects43712026/11/12 00:23:37 INFO Vacuumed table table=pending_closures43722026/11/12 00:23:37 INFO Vacuumed table table=pending_closures43732026/11/12 00:23:37 INFO Vacuumed table table=pending_objects43742026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043752026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads43762026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043772026/11/12 00:23:37 INFO Vacuumed table table=pending_objects43782026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads43792026/11/12 00:23:37 INFO Vacuumed table table=closures43802026/11/12 00:23:37 INFO Vacuumed table table=pending_closures43812026/11/12 00:23:37 INFO Vacuumed table table=objects43822026/11/12 00:23:37 INFO Vacuumed table table=closures43832026/11/12 00:23:37 INFO Received complete push request method=POST path=/43842026/11/12 00:23:37 INFO Vacuumed table table=pending_closures43852026/11/12 00:23:37 INFO Vacuumed table table=objects43862026/11/12 00:23:37 INFO Vacuumed table table=pending_objects43872026/11/12 00:23:37 INFO Completed push id=29 objects=24 duration=1.825137ms43882026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads43892026/11/12 00:23:37 INFO Vacuumed table table=pending_objects43902026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads43912026/11/12 00:23:37 INFO Vacuumed table table=closures43922026/11/12 00:23:37 INFO Vacuumed table table=closures43932026/11/12 00:23:37 INFO Vacuumed table table=objects43942026/11/12 00:23:37 INFO Vacuumed table table=objects43952026/11/12 00:23:37 INFO Received complete push request method=POST path=/43962026/11/12 00:23:37 INFO Completed push id=32 objects=7 duration=1.225251ms43972026/11/12 00:23:37 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."43982026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."43992026/11/12 00:23:37 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."44002026/11/12 00:23:37 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."44012026/11/12 00:23:37 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."44022026/11/12 00:23:37 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."44032026/11/12 00:23:37 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."44042026/11/12 00:23:37 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."44052026/11/12 00:23:37 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."44062026/11/12 00:23:37 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."44072026/11/12 00:23:37 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."44082026/11/12 00:23:37 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."44092026/11/12 00:23:37 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."44102026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."44112026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."44122026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."44132026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."44142026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."44152026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."44162026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."44172026/11/12 00:23:37 INFO Received uploads request method=POST path=/44182026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=044192026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period44202026/11/12 00:23:37 INFO Received complete push request method=POST path=/44212026/11/12 00:23:37 INFO Completed push id=26 objects=23 duration=1.447134ms44222026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044232026/11/12 00:23:37 INFO Vacuumed table table=pending_closures44242026/11/12 00:23:37 INFO Received push request method=POST path=/44252026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44262026/11/12 00:23:37 INFO Completed upload id=33 objects=6 duration=987.409µs44272026/11/12 00:23:37 INFO Received push request method=POST path=/44282026/11/12 00:23:37 INFO Vacuumed table table=pending_objects44292026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=044302026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads44312026/11/12 00:23:37 INFO Vacuumed table table=closures44322026/11/12 00:23:37 INFO Vacuumed table table=objects44332026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period44342026/11/12 00:23:37 INFO Received push request method=POST path=/44352026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044362026/11/12 00:23:37 INFO Received complete push request method=POST path=/44372026/11/12 00:23:37 INFO Vacuumed table table=pending_closures44382026/11/12 00:23:37 INFO Received uploads request method=POST path=/44392026/11/12 00:23:37 INFO Vacuumed table table=pending_objects44402026/11/12 00:23:37 INFO Completed push id=9 objects=9 duration=3.613833ms44412026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads44422026/11/12 00:23:37 INFO Vacuumed table table=closures44432026/11/12 00:23:37 INFO Vacuumed table table=objects44442026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44452026/11/12 00:23:37 INFO Completed upload id=34 objects=3 duration=750.467µs44462026/11/12 00:23:37 INFO Received push request method=POST path=/44472026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=044482026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period44492026/11/12 00:23:37 INFO Received complete push request method=POST path=/44502026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44512026/11/12 00:23:37 INFO Completed push id=27 objects=15 duration=1.09937ms44522026/11/12 00:23:37 INFO Completed upload id=21 objects=8 duration=1.219911ms44532026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=044542026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period44552026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044562026/11/12 00:23:37 INFO Vacuumed table table=pending_closures44572026/11/12 00:23:37 INFO Vacuumed table table=pending_objects44582026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads44592026/11/12 00:23:37 INFO Vacuumed table table=closures44602026/11/12 00:23:37 INFO Vacuumed table table=objects44612026/11/12 00:23:37 INFO Received uploads request method=POST path=/44622026/11/12 00:23:37 INFO Received push request method=POST path=/44632026/11/12 00:23:37 INFO Received complete push request method=POST path=/44642026/11/12 00:23:37 INFO Completed push id=22 objects=6 duration=1.289052ms44652026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=044662026/11/12 00:23:37 INFO Vacuumed table table=pending_closures44672026/11/12 00:23:37 INFO Vacuumed table table=pending_objects44682026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads44692026/11/12 00:23:37 INFO Vacuumed table table=closures44702026/11/12 00:23:37 INFO Vacuumed table table=objects44712026/11/12 00:23:37 INFO Received complete push request method=POST path=/44722026/11/12 00:23:37 INFO Completed push id=24 objects=8 duration=1.10505ms44732026/11/12 00:23:37 INFO Received push request method=POST path=/44742026/11/12 00:23:37 INFO Received complete push request method=POST path=/44752026/11/12 00:23:37 INFO Completed push id=28 objects=19 duration=1.341012ms44762026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44772026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44782026/11/12 00:23:37 INFO Completed upload id=25 objects=9 duration=856.747µs44792026/11/12 00:23:37 INFO Completed upload id=6 objects=24 duration=3.148749ms44802026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44812026/11/12 00:23:37 INFO Completed upload id=27 objects=3 duration=921.649µs44822026/11/12 00:23:37 INFO Received push request method=POST path=/44832026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44842026/11/12 00:23:37 INFO Completed upload id=4 objects=14 duration=2.804106ms44852026/11/12 00:23:37 INFO Received push request method=POST path=/44862026/11/12 00:23:37 INFO Received complete push request method=POST path=/44872026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44882026/11/12 00:23:37 INFO Completed push id=29 objects=10 duration=962.289µs44892026/11/12 00:23:37 INFO Completed upload id=28 objects=6 duration=701.886µs44902026/11/12 00:23:37 INFO Received push request method=POST path=/44912026/11/12 00:23:37 INFO Received push request method=POST path=/44922026/11/12 00:23:37 INFO Received complete push request method=POST path=/44932026/11/12 00:23:37 INFO Received complete push request method=POST path=/44942026/11/12 00:23:37 INFO Completed push id=30 objects=4 duration=878.288µs44952026/11/12 00:23:37 INFO Completed push id=8 objects=6 duration=2.10914ms44962026/11/12 00:23:37 INFO Received push request method=POST path=/44972026/11/12 00:23:37 INFO Received complete upload request method=POST path=/44982026/11/12 00:23:37 INFO Completed upload id=30 objects=6 duration=1.431153ms44992026/11/12 00:23:37 INFO Received uploads request method=POST path=/45002026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045012026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period45022026/11/12 00:23:37 INFO Received complete push request method=POST path=/45032026/11/12 00:23:37 INFO Received complete push request method=POST path=/45042026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045052026/11/12 00:23:37 INFO Completed push id=31 objects=7 duration=881.308µs45062026/11/12 00:23:37 INFO Vacuumed table table=pending_closures45072026/11/12 00:23:37 INFO Completed push id=27 objects=25 duration=2.577004ms45082026/11/12 00:23:37 INFO Vacuumed table table=pending_objects45092026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads45102026/11/12 00:23:37 INFO Vacuumed table table=closures45112026/11/12 00:23:37 INFO Received uploads request method=POST path=/45122026/11/12 00:23:37 INFO Received complete upload request method=POST path=/45132026/11/12 00:23:37 INFO Vacuumed table table=objects45142026/11/12 00:23:37 INFO Completed upload id=9 objects=3 duration=1.163851ms45152026/11/12 00:23:37 INFO Received complete upload request method=POST path=/45162026/11/12 00:23:37 INFO Received uploads request method=POST path=/45172026/11/12 00:23:37 INFO Completed upload id=31 objects=5 duration=683.787µs45182026/11/12 00:23:37 INFO Received complete upload request method=POST path=/45192026/11/12 00:23:37 INFO Completed upload id=32 objects=5 duration=654.446µs45202026/11/12 00:23:37 INFO Received push request method=POST path=/45212026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045222026/11/12 00:23:37 INFO Received push request method=POST path=/45232026/11/12 00:23:37 INFO Received uploads request method=POST path=/45242026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045252026/11/12 00:23:37 INFO Vacuumed table table=pending_closures45262026/11/12 00:23:37 INFO Received complete upload request method=POST path=/45272026/11/12 00:23:37 INFO Completed upload id=34 objects=3 duration=704.646µs45282026/11/12 00:23:37 INFO Vacuumed table table=pending_objects45292026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads45302026/11/12 00:23:37 INFO Vacuumed table table=closures45312026/11/12 00:23:37 INFO Vacuumed table table=objects45322026/11/12 00:23:37 INFO Received push request method=POST path=/45332026/11/12 00:23:37 INFO Received push request method=POST path=/45342026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045352026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045362026/11/12 00:23:37 INFO Received uploads request method=POST path=/45372026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period45382026/11/12 00:23:37 INFO Received complete push request method=POST path=/45392026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045402026/11/12 00:23:37 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."45412026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."45422026/11/12 00:23:37 INFO Vacuumed table table=pending_closures45432026/11/12 00:23:37 INFO Completed push id=24 objects=5 duration=2.853367ms45442026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045452026/11/12 00:23:37 INFO Vacuumed table table=pending_objects45462026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads45472026/11/12 00:23:37 INFO Vacuumed table table=closures45482026/11/12 00:23:37 INFO Vacuumed table table=objects45492026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045502026/11/12 00:23:37 INFO Vacuumed table table=pending_closures45512026/11/12 00:23:37 INFO Vacuumed table table=pending_objects45522026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads45532026/11/12 00:23:37 INFO Vacuumed table table=closures45542026/11/12 00:23:37 INFO Vacuumed table table=objects45552026/11/12 00:23:37 INFO Received uploads request method=POST path=/45562026/11/12 00:23:37 INFO Received push request method=POST path=/45572026/11/12 00:23:37 INFO Received complete upload request method=POST path=/45582026/11/12 00:23:37 INFO Completed upload id=34 objects=24 duration=1.492154ms45592026/11/12 00:23:37 INFO Received complete upload request method=POST path=/45602026/11/12 00:23:37 INFO Completed upload id=10 objects=6 duration=2.680024ms45612026/11/12 00:23:37 INFO Received uploads request method=POST path=/45622026/11/12 00:23:37 INFO Received complete push request method=POST path=/45632026/11/12 00:23:37 INFO Received uploads request method=POST path=/45642026/11/12 00:23:37 INFO Completed push id=5 objects=8 duration=3.495333ms45652026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045662026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period45672026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045682026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045692026/11/12 00:23:37 INFO Vacuumed table table=pending_closures45702026/11/12 00:23:37 INFO Vacuumed table table=pending_objects45712026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads45722026/11/12 00:23:37 INFO Vacuumed table table=closures45732026/11/12 00:23:37 INFO Vacuumed table table=objects45742026/11/12 00:23:37 INFO Received complete push request method=POST path=/45752026/11/12 00:23:37 INFO Received push request method=POST path=/45762026/11/12 00:23:37 INFO Completed push id=12 objects=3 duration=2.005979ms45772026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=045782026/11/12 00:23:37 INFO Vacuumed table table=pending_closures45792026/11/12 00:23:37 INFO Vacuumed table table=pending_objects45802026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads45812026/11/12 00:23:37 INFO Vacuumed table table=closures45822026/11/12 00:23:37 INFO Vacuumed table table=objects45832026/11/12 00:23:37 INFO Received uploads request method=POST path=/45842026/11/12 00:23:37 INFO Received uploads request method=POST path=/45852026/11/12 00:23:37 INFO Received complete push request method=POST path=/45862026/11/12 00:23:37 INFO Received push request method=POST path=/45872026/11/12 00:23:37 INFO Completed push id=21 objects=26 duration=1.334692ms45882026/11/12 00:23:37 INFO Received push request method=POST path=/45892026/11/12 00:23:37 INFO Received uploads request method=POST path=/45902026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=045912026/11/12 00:23:37 INFO Received push request method=POST path=/45922026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045932026/11/12 00:23:37 INFO Vacuumed table table=pending_closures45942026/11/12 00:23:37 INFO Received complete push request method=POST path=/45952026/11/12 00:23:37 INFO Vacuumed table table=pending_objects45962026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads45972026/11/12 00:23:37 INFO Vacuumed table table=closures45982026/11/12 00:23:37 INFO Vacuumed table table=objects45992026/11/12 00:23:37 INFO Completed push id=8 objects=6 duration=1.726456ms46002026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46012026/11/12 00:23:37 INFO Received uploads request method=POST path=/46022026/11/12 00:23:37 INFO Completed upload id=28 objects=17 duration=1.142771ms46032026/11/12 00:23:37 INFO Received uploads request method=POST path=/46042026/11/12 00:23:37 INFO Received complete push request method=POST path=/46052026/11/12 00:23:37 INFO Completed push id=12 objects=7 duration=1.484434ms46062026/11/12 00:23:37 INFO Received complete push request method=POST path=/46072026/11/12 00:23:37 INFO Completed push id=29 objects=2 duration=949.269µs46082026/11/12 00:23:37 INFO Received uploads request method=POST path=/46092026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=046102026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46112026/11/12 00:23:37 INFO Received complete push request method=POST path=/46122026/11/12 00:23:37 INFO Completed upload id=16 objects=23 duration=1.677615ms46132026/11/12 00:23:37 INFO Completed push id=31 objects=6 duration=985.749µs46142026/11/12 00:23:37 INFO Received uploads request method=POST path=/46152026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=046162026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=046172026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period46182026/11/12 00:23:37 INFO Vacuumed table table=pending_closures46192026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46202026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046212026/11/12 00:23:37 INFO Received complete push request method=POST path=/46222026/11/12 00:23:37 INFO Vacuumed table table=pending_objects46232026/11/12 00:23:37 INFO Completed upload id=15 objects=6 duration=1.08553ms46242026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads46252026/11/12 00:23:37 INFO Vacuumed table table=pending_closures46262026/11/12 00:23:37 INFO Vacuumed table table=closures46272026/11/12 00:23:37 INFO Received push request method=POST path=/46282026/11/12 00:23:37 INFO Completed push id=32 objects=9 duration=1.432173ms46292026/11/12 00:23:37 INFO Vacuumed table table=pending_objects46302026/11/12 00:23:37 INFO Vacuumed table table=objects46312026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads46322026/11/12 00:23:37 INFO Vacuumed table table=closures46332026/11/12 00:23:37 INFO Vacuumed table table=objects46342026/11/12 00:23:37 INFO Received uploads request method=POST path=/46352026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46362026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=046372026/11/12 00:23:37 INFO Completed upload id=8 objects=3 duration=2.672685ms46382026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46392026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046402026/11/12 00:23:37 INFO Vacuumed table table=pending_closures46412026/11/12 00:23:37 INFO Completed upload id=33 objects=9 duration=1.11635ms46422026/11/12 00:23:37 INFO Vacuumed table table=pending_objects46432026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads46442026/11/12 00:23:37 INFO Vacuumed table table=closures46452026/11/12 00:23:37 INFO Vacuumed table table=objects46462026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46472026/11/12 00:23:37 INFO Received uploads request method=POST path=/46482026/11/12 00:23:37 INFO Completed upload id=17 objects=7 duration=1.204471ms46492026/11/12 00:23:37 INFO Received push request method=POST path=/46502026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=046512026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period46522026/11/12 00:23:37 INFO Received uploads request method=POST path=/46532026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046542026/11/12 00:23:37 INFO Vacuumed table table=pending_closures46552026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=046562026/11/12 00:23:37 INFO Vacuumed table table=pending_objects46572026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads46582026/11/12 00:23:37 INFO Vacuumed table table=closures46592026/11/12 00:23:37 INFO Vacuumed table table=objects46602026/11/12 00:23:37 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."46612026/11/12 00:23:37 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."46622026/11/12 00:23:37 INFO Received uploads request method=POST path=/46632026/11/12 00:23:37 INFO Received complete push request method=POST path=/46642026/11/12 00:23:37 INFO Received uploads request method=POST path=/46652026/11/12 00:23:37 INFO Received complete push request method=POST path=/46662026/11/12 00:23:37 INFO Completed push id=6 objects=6 duration=1.770756ms46672026/11/12 00:23:37 INFO Completed push id=9 objects=4 duration=3.24419ms46682026/11/12 00:23:37 INFO Received push request method=POST path=/46692026/11/12 00:23:37 INFO Received uploads request method=POST path=/46702026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46712026/11/12 00:23:37 INFO Completed upload id=14 objects=6 duration=929.428µs46722026/11/12 00:23:37 INFO Received push request method=POST path=/46732026/11/12 00:23:37 INFO Received push request method=POST path=/46742026/11/12 00:23:37 INFO Received uploads request method=POST path=/46752026/11/12 00:23:37 INFO Received complete push request method=POST path=/46762026/11/12 00:23:37 INFO Received complete upload request method=POST path=/46772026/11/12 00:23:37 INFO Completed upload id=7 objects=3 duration=942.729µs46782026/11/12 00:23:37 INFO Completed push id=10 objects=15 duration=1.875518ms46792026/11/12 00:23:37 INFO Received push request method=POST path=/46802026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=046812026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period46822026/11/12 00:23:37 INFO Received complete push request method=POST path=/46832026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046842026/11/12 00:23:37 INFO Vacuumed table table=pending_closures46852026/11/12 00:23:37 INFO Completed push id=18 objects=23 duration=3.603254ms46862026/11/12 00:23:37 INFO Vacuumed table table=pending_objects46872026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads46882026/11/12 00:23:37 INFO Vacuumed table table=closures46892026/11/12 00:23:37 INFO Vacuumed table table=objects46902026/11/12 00:23:37 INFO Received uploads request method=POST path=/46912026/11/12 00:23:37 INFO Received complete push request method=POST path=/46922026/11/12 00:23:37 INFO Completed push id=19 objects=8 duration=3.843176ms46932026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=046942026/11/12 00:23:37 INFO Received complete push request method=POST path=/46952026/11/12 00:23:37 INFO Completed push id=11 objects=9 duration=1.712396ms46962026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period46972026/11/12 00:23:37 INFO Received complete push request method=POST path=/46982026/11/12 00:23:37 INFO Completed push id=17 objects=22 duration=2.052359ms46992026/11/12 00:23:37 INFO Received uploads request method=POST path=/47002026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047012026/11/12 00:23:37 INFO Vacuumed table table=pending_closures47022026/11/12 00:23:37 INFO Vacuumed table table=pending_objects47032026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads47042026/11/12 00:23:37 INFO Vacuumed table table=closures47052026/11/12 00:23:37 INFO Vacuumed table table=objects47062026/11/12 00:23:37 INFO Received push request method=POST path=/47072026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=047082026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047092026/11/12 00:23:37 INFO Received push request method=POST path=/47102026/11/12 00:23:37 INFO Vacuumed table table=pending_closures47112026/11/12 00:23:37 INFO Vacuumed table table=pending_objects47122026/11/12 00:23:37 INFO Received push request method=POST path=/47132026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads47142026/11/12 00:23:37 INFO Vacuumed table table=closures47152026/11/12 00:23:37 INFO Vacuumed table table=objects47162026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=047172026/11/12 00:23:37 INFO Received uploads request method=POST path=/47182026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period47192026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047202026/11/12 00:23:37 INFO Vacuumed table table=pending_closures47212026/11/12 00:23:37 INFO Vacuumed table table=pending_objects47222026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads47232026/11/12 00:23:37 INFO Vacuumed table table=closures47242026/11/12 00:23:37 INFO Vacuumed table table=objects47252026/11/12 00:23:37 INFO Received uploads request method=POST path=/47262026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=047272026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period47282026/11/12 00:23:37 INFO Received push request method=POST path=/47292026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047302026/11/12 00:23:37 INFO Vacuumed table table=pending_closures47312026/11/12 00:23:37 INFO Vacuumed table table=pending_objects47322026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads47332026/11/12 00:23:37 INFO Vacuumed table table=closures47342026/11/12 00:23:37 INFO Vacuumed table table=objects47352026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=047362026/11/12 00:23:37 INFO Received complete upload request method=POST path=/47372026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period47382026/11/12 00:23:37 INFO Completed upload id=7 objects=10 duration=2.018558ms47392026/11/12 00:23:37 INFO Received uploads request method=POST path=/47402026/11/12 00:23:37 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."47412026/11/12 00:23:37 INFO Received uploads request method=POST path=/47422026/11/12 00:23:37 INFO Received uploads request method=POST path=/47432026/11/12 00:23:37 INFO Received uploads request method=POST path=/47442026/11/12 00:23:37 INFO Received complete upload request method=POST path=/47452026/11/12 00:23:37 INFO Completed upload id=14 objects=8 duration=2.17894ms47462026/11/12 00:23:37 INFO Received push request method=POST path=/47472026/11/12 00:23:37 INFO Received push request method=POST path=/47482026/11/12 00:23:37 INFO Received complete upload request method=POST path=/47492026/11/12 00:23:37 INFO Completed upload id=23 objects=15 duration=2.318801ms47502026/11/12 00:23:37 INFO Received uploads request method=POST path=/47512026/11/12 00:23:37 INFO Received complete upload request method=POST path=/47522026/11/12 00:23:37 INFO Completed upload id=11 objects=7 duration=937.489µs47532026/11/12 00:23:37 INFO Received uploads request method=POST path=/47542026/11/12 00:23:37 INFO Received uploads request method=POST path=/47552026/11/12 00:23:37 INFO Received complete upload request method=POST path=/47562026/11/12 00:23:37 INFO Completed upload id=19 objects=6 duration=1.149091ms47572026/11/12 00:23:37 INFO Received uploads request method=POST path=/47582026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=047592026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period47602026/11/12 00:23:37 INFO Received complete upload request method=POST path=/47612026/11/12 00:23:37 INFO Completed upload id=22 objects=7 duration=981.229µs47622026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047632026/11/12 00:23:37 INFO Vacuumed table table=pending_closures47642026/11/12 00:23:37 INFO Vacuumed table table=pending_objects47652026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads47662026/11/12 00:23:37 INFO Received complete upload request method=POST path=/47672026/11/12 00:23:37 INFO Vacuumed table table=closures47682026/11/12 00:23:37 INFO Vacuumed table table=objects47692026/11/12 00:23:37 INFO Completed upload id=14 objects=16 duration=2.312701ms47702026/11/12 00:23:38 INFO Received uploads request method=POST path=/47712026/11/12 00:23:38 INFO Received complete push request method=POST path=/47722026/11/12 00:23:38 INFO Completed push id=24 objects=9 duration=1.897097ms47732026/11/12 00:23:38 INFO Received complete push request method=POST path=/47742026/11/12 00:23:38 INFO Completed push id=17 objects=11 duration=2.040979ms47752026/11/12 00:23:38 INFO Received push request method=POST path=/47762026/11/12 00:23:38 INFO Received complete upload request method=POST path=/47772026/11/12 00:23:38 INFO Received push request method=POST path=/47782026/11/12 00:23:38 INFO Completed upload id=16 objects=17 duration=2.107459ms47792026/11/12 00:23:38 INFO Received push request method=POST path=/47802026/11/12 00:23:38 INFO Received push request method=POST path=/47812026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=047822026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period47832026/11/12 00:23:38 INFO Received complete push request method=POST path=/47842026/11/12 00:23:38 INFO Completed push id=13 objects=15 duration=1.668436ms47852026/11/12 00:23:38 INFO Received uploads request method=POST path=/47862026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=047872026/11/12 00:23:38 INFO Received complete upload request method=POST path=/47882026/11/12 00:23:38 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."47892026/11/12 00:23:38 INFO Completed upload id=26 objects=6 duration=986.749µs47902026/11/12 00:23:38 INFO Received uploads request method=POST path=/47912026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=047922026/11/12 00:23:38 INFO Vacuumed table table=pending_closures47932026/11/12 00:23:38 INFO Vacuumed table table=pending_objects47942026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads47952026/11/12 00:23:38 INFO Vacuumed table table=closures47962026/11/12 00:23:38 INFO Vacuumed table table=objects47972026/11/12 00:23:38 INFO Received complete upload request method=POST path=/47982026/11/12 00:23:38 INFO Completed upload id=29 objects=5 duration=901.429µs47992026/11/12 00:23:38 INFO Received complete push request method=POST path=/48002026/11/12 00:23:38 INFO Completed push id=20 objects=3 duration=1.684456ms48012026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=048022026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period48032026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048042026/11/12 00:23:38 INFO Received complete push request method=POST path=/48052026/11/12 00:23:38 INFO Vacuumed table table=pending_closures48062026/11/12 00:23:38 INFO Completed push id=21 objects=3 duration=944.969µs48072026/11/12 00:23:38 INFO Vacuumed table table=pending_objects48082026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads48092026/11/12 00:23:38 INFO Vacuumed table table=closures48102026/11/12 00:23:38 INFO Vacuumed table table=objects48112026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=048122026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period48132026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048142026/11/12 00:23:38 INFO Vacuumed table table=pending_closures48152026/11/12 00:23:38 INFO Vacuumed table table=pending_objects48162026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads48172026/11/12 00:23:38 INFO Vacuumed table table=closures48182026/11/12 00:23:38 INFO Vacuumed table table=objects48192026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48202026/11/12 00:23:38 INFO Received uploads request method=POST path=/48212026/11/12 00:23:38 INFO Completed upload id=15 objects=3 duration=1.606394ms48222026/11/12 00:23:38 INFO Received push request method=POST path=/48232026/11/12 00:23:38 INFO Received push request method=POST path=/48242026/11/12 00:23:38 INFO Received push request method=POST path=/48252026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=048262026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048272026/11/12 00:23:38 INFO Received push request method=POST path=/48282026/11/12 00:23:38 INFO Vacuumed table table=pending_closures48292026/11/12 00:23:38 INFO Vacuumed table table=pending_objects48302026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads48312026/11/12 00:23:38 INFO Vacuumed table table=closures48322026/11/12 00:23:38 INFO Received uploads request method=POST path=/48332026/11/12 00:23:38 INFO Vacuumed table table=objects48342026/11/12 00:23:38 INFO Received push request method=POST path=/48352026/11/12 00:23:38 INFO Received push request method=POST path=/48362026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48372026/11/12 00:23:38 INFO Completed upload id=22 objects=6 duration=942.029µs48382026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48392026/11/12 00:23:38 INFO Received push request method=POST path=/48402026/11/12 00:23:38 INFO Completed upload id=31 objects=3 duration=3.844776ms48412026/11/12 00:23:38 INFO Received uploads request method=POST path=/48422026/11/12 00:23:38 INFO Received complete push request method=POST path=/48432026/11/12 00:23:38 INFO Completed push id=17 objects=22 duration=2.082599ms48442026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48452026/11/12 00:23:38 INFO Completed upload id=22 objects=2 duration=896.368µs48462026/11/12 00:23:38 INFO Received complete push request method=POST path=/48472026/11/12 00:23:38 INFO Completed push id=30 objects=4 duration=3.164709ms48482026/11/12 00:23:38 INFO Received uploads request method=POST path=/48492026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=048502026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period48512026/11/12 00:23:38 INFO Received uploads request method=POST path=/48522026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048532026/11/12 00:23:38 INFO Vacuumed table table=pending_closures48542026/11/12 00:23:38 INFO Vacuumed table table=pending_objects48552026/11/12 00:23:38 INFO Received uploads request method=POST path=/48562026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads48572026/11/12 00:23:38 INFO Vacuumed table table=closures48582026/11/12 00:23:38 INFO Vacuumed table table=objects48592026/11/12 00:23:38 INFO Received push request method=POST path=/48602026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48612026/11/12 00:23:38 INFO Received push request method=POST path=/48622026/11/12 00:23:38 INFO Completed upload id=18 objects=2 duration=2.376182ms48632026/11/12 00:23:38 INFO Received push request method=POST path=/48642026/11/12 00:23:38 INFO Received push request method=POST path=/48652026/11/12 00:23:38 INFO Received push request method=POST path=/48662026/11/12 00:23:38 INFO Received push request method=POST path=/48672026/11/12 00:23:38 INFO Received complete push request method=POST path=/48682026/11/12 00:23:38 INFO Completed push id=32 objects=7 duration=1.391753ms48692026/11/12 00:23:38 INFO Received uploads request method=POST path=/48702026/11/12 00:23:38 INFO Received complete push request method=POST path=/48712026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48722026/11/12 00:23:38 INFO Received complete push request method=POST path=/48732026/11/12 00:23:38 INFO Completed upload id=13 objects=7 duration=979.129µs48742026/11/12 00:23:38 INFO Completed push id=19 objects=14 duration=1.518494ms48752026/11/12 00:23:38 INFO Completed push id=26 objects=4 duration=1.07839ms48762026/11/12 00:23:38 INFO Received uploads request method=POST path=/48772026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48782026/11/12 00:23:38 INFO Received uploads request method=POST path=/48792026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48802026/11/12 00:23:38 INFO Completed upload id=20 objects=4 duration=1.01131ms48812026/11/12 00:23:38 INFO Received uploads request method=POST path=/48822026/11/12 00:23:38 INFO Completed upload id=28 objects=13 duration=1.352013ms48832026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48842026/11/12 00:23:38 INFO Completed upload id=23 objects=7 duration=976.589µs48852026/11/12 00:23:38 INFO Received uploads request method=POST path=/48862026/11/12 00:23:38 INFO Received complete push request method=POST path=/48872026/11/12 00:23:38 INFO Completed push id=29 objects=3 duration=1.202171ms48882026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48892026/11/12 00:23:38 INFO Completed upload id=20 objects=6 duration=1.08913ms48902026/11/12 00:23:38 INFO Received uploads request method=POST path=/48912026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48922026/11/12 00:23:38 INFO Completed upload id=31 objects=2 duration=993.009µs48932026/11/12 00:23:38 INFO Received push request method=POST path=/48942026/11/12 00:23:38 INFO Received complete push request method=POST path=/48952026/11/12 00:23:38 INFO Completed push id=33 objects=6 duration=923.529µs48962026/11/12 00:23:38 INFO Received uploads request method=POST path=/48972026/11/12 00:23:38 INFO Received complete upload request method=POST path=/48982026/11/12 00:23:38 INFO Completed upload id=24 objects=18 duration=1.04073ms48992026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=049002026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period49012026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049022026/11/12 00:23:38 INFO Vacuumed table table=pending_closures49032026/11/12 00:23:38 INFO Received complete push request method=POST path=/49042026/11/12 00:23:38 INFO Vacuumed table table=pending_objects49052026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads49062026/11/12 00:23:38 INFO Vacuumed table table=closures49072026/11/12 00:23:38 INFO Completed push id=25 objects=2 duration=2.643784ms49082026/11/12 00:23:38 INFO Vacuumed table table=objects49092026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49102026/11/12 00:23:38 INFO Completed upload id=21 objects=2 duration=933.989µs49112026/11/12 00:23:38 INFO Received complete push request method=POST path=/49122026/11/12 00:23:38 INFO Completed push id=27 objects=4 duration=911.189µs49132026/11/12 00:23:38 INFO Received uploads request method=POST path=/49142026/11/12 00:23:38 INFO Received complete push request method=POST path=/49152026/11/12 00:23:38 INFO Completed push id=16 objects=7 duration=1.800437ms49162026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=049172026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period49182026/11/12 00:23:38 INFO Received complete push request method=POST path=/49192026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049202026/11/12 00:23:38 INFO Completed push id=28 objects=26 duration=1.320812ms49212026/11/12 00:23:38 INFO Vacuumed table table=pending_closures49222026/11/12 00:23:38 INFO Vacuumed table table=pending_objects49232026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads49242026/11/12 00:23:38 INFO Vacuumed table table=closures49252026/11/12 00:23:38 INFO Vacuumed table table=objects49262026/11/12 00:23:38 INFO Received complete push request method=POST path=/49272026/11/12 00:23:38 INFO Completed push id=23 objects=6 duration=1.14215ms49282026/11/12 00:23:38 INFO Received complete push request method=POST path=/49292026/11/12 00:23:38 INFO Completed push id=30 objects=3 duration=912.389µs49302026/11/12 00:23:38 INFO Received complete push request method=POST path=/49312026/11/12 00:23:38 INFO Completed push id=24 objects=6 duration=757.887µs49322026/11/12 00:23:38 INFO Received complete push request method=POST path=/49332026/11/12 00:23:38 INFO Received uploads request method=POST path=/49342026/11/12 00:23:38 INFO Completed push id=32 objects=19 duration=902.049µs49352026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=049362026/11/12 00:23:38 INFO Received push request method=POST path=/49372026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period49382026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049392026/11/12 00:23:38 INFO Received complete push request method=POST path=/49402026/11/12 00:23:38 INFO Vacuumed table table=pending_closures49412026/11/12 00:23:38 INFO Completed push id=25 objects=7 duration=1.277332ms49422026/11/12 00:23:38 INFO Vacuumed table table=pending_objects49432026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads49442026/11/12 00:23:38 INFO Vacuumed table table=closures49452026/11/12 00:23:38 INFO Vacuumed table table=objects49462026/11/12 00:23:38 INFO Received complete push request method=POST path=/49472026/11/12 00:23:38 INFO Completed push id=24 objects=24 duration=2.909047ms49482026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=049492026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period49502026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049512026/11/12 00:23:38 INFO Vacuumed table table=pending_closures49522026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49532026/11/12 00:23:38 INFO Vacuumed table table=pending_objects49542026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads49552026/11/12 00:23:38 INFO Vacuumed table table=closures49562026/11/12 00:23:38 INFO Completed upload id=26 objects=24 duration=1.213492ms49572026/11/12 00:23:38 INFO Vacuumed table table=objects49582026/11/12 00:23:38 INFO Received uploads request method=POST path=/49592026/11/12 00:23:38 INFO Received uploads request method=POST path=/49602026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49612026/11/12 00:23:38 INFO Completed upload id=27 objects=26 duration=1.456633ms49622026/11/12 00:23:38 INFO Received complete push request method=POST path=/49632026/11/12 00:23:38 INFO Completed push id=28 objects=6 duration=990.97µs49642026/11/12 00:23:38 INFO Received push request method=POST path=/49652026/11/12 00:23:38 INFO Received complete push request method=POST path=/49662026/11/12 00:23:38 INFO Completed push id=29 objects=6 duration=946.569µs49672026/11/12 00:23:38 INFO Received uploads request method=POST path=/49682026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=049692026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period49702026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049712026/11/12 00:23:38 INFO Vacuumed table table=pending_closures49722026/11/12 00:23:38 INFO Vacuumed table table=pending_objects49732026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads49742026/11/12 00:23:38 INFO Vacuumed table table=closures49752026/11/12 00:23:38 INFO Vacuumed table table=objects49762026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49772026/11/12 00:23:38 INFO Completed upload id=30 objects=8 duration=1.034089ms49782026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49792026/11/12 00:23:38 INFO Completed upload id=31 objects=6 duration=897.289µs49802026/11/12 00:23:38 INFO Received push request method=POST path=/49812026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49822026/11/12 00:23:38 INFO Completed upload id=32 objects=4 duration=800.747µs49832026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49842026/11/12 00:23:38 INFO Completed upload id=23 objects=7 duration=3.18325ms49852026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49862026/11/12 00:23:38 INFO Completed upload id=33 objects=10 duration=887.928µs49872026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49882026/11/12 00:23:38 INFO Completed upload id=26 objects=5 duration=883.249µs49892026/11/12 00:23:38 INFO Received complete push request method=POST path=/49902026/11/12 00:23:38 INFO Completed push id=27 objects=3 duration=1.712636ms49912026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49922026/11/12 00:23:38 INFO Completed upload id=34 objects=23 duration=1.001529ms49932026/11/12 00:23:38 INFO Received complete upload request method=POST path=/49942026/11/12 00:23:38 INFO Completed upload id=28 objects=5 duration=1.09525ms49952026/11/12 00:23:38 INFO Received complete push request method=POST path=/49962026/11/12 00:23:38 INFO Completed push id=35 objects=6 duration=958.969µs49972026/11/12 00:23:38 INFO Received complete push request method=POST path=/49982026/11/12 00:23:38 INFO Completed push id=29 objects=17 duration=930.989µs49992026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=050002026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period50012026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=050022026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period50032026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050042026/11/12 00:23:38 INFO Vacuumed table table=pending_closures50052026/11/12 00:23:38 INFO Vacuumed table table=pending_objects50062026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads50072026/11/12 00:23:38 INFO Vacuumed table table=closures50082026/11/12 00:23:38 INFO Vacuumed table table=objects50092026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050102026/11/12 00:23:38 INFO Vacuumed table table=pending_closures50112026/11/12 00:23:38 INFO Vacuumed table table=pending_objects50122026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads50132026/11/12 00:23:38 INFO Vacuumed table table=closures50142026/11/12 00:23:38 INFO Vacuumed table table=objects5015--- PASS: FuzzPushGC (0.01s)5016 --- PASS: FuzzPushGC/seed#8 (4.43s)5017 --- PASS: FuzzPushGC/seed#5 (4.80s)5018 --- PASS: FuzzPushGC/seed#15 (4.88s)5019 --- PASS: FuzzPushGC/seed#3 (5.01s)5020 --- PASS: FuzzPushGC/seed#11 (5.16s)5021 --- PASS: FuzzPushGC/seed#4 (5.32s)5022 --- PASS: FuzzPushGC/seed#6 (5.46s)5023 --- PASS: FuzzPushGC/seed#2 (5.50s)5024 --- PASS: FuzzPushGC/seed#10 (5.53s)5025 --- PASS: FuzzPushGC/seed#13 (5.65s)5026 --- PASS: FuzzPushGC/seed#9 (5.68s)5027 --- PASS: FuzzPushGC/seed#0 (5.82s)5028 --- PASS: FuzzPushGC/seed#14 (6.52s)5029 --- PASS: FuzzPushGC/seed#7 (6.61s)5030 --- PASS: FuzzPushGC/seed#12 (6.85s)5031 --- PASS: FuzzPushGC/seed#1 (6.86s)5032PASS5033{"timestamp":"2026-11-12T00:23:38.74572568Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:46218","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(194)"}5034{"timestamp":"2026-11-12T00:23:38.7457231Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:34170","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(205)"}50352026-11-12 00:23:39.123 UTC [132] LOG: received smart shutdown request50362026-11-12 00:23:39.128 UTC [132] LOG: background worker "logical replication launcher" (PID 146) exited with exit code 150372026-11-12 00:23:42.173 UTC [141] LOG: checkpoint complete: wrote 4078 buffers (24.9%), wrote 7 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=7.333 s, sync=0.861 s, total=10.256 s; sync files=42363, longest=0.016 s, average=0.001 s; distance=536218 kB, estimate=536218 kB; lsn=0/26E3BCE8, redo lsn=0/2230C78050382026-11-12 00:23:42.179 UTC [141] LOG: shutting down50392026-11-12 00:23:42.180 UTC [141] LOG: checkpoint starting: shutdown immediate50402026-11-12 00:23:42.868 UTC [141] LOG: checkpoint complete: wrote 8244 buffers (50.3%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.165 s, sync=0.513 s, total=0.689 s; sync files=2566, longest=0.010 s, average=0.001 s; distance=76989 kB, estimate=490295 kB; lsn=0/26E3BD60, redo lsn=0/26E3BD6050412026-11-12 00:23:43.018 UTC [132] LOG: database system is shut down5042Running OIDC tests...5043=== RUN TestAudienceForIssuer5044=== PAUSE TestAudienceForIssuer5045=== RUN TestGlobMatch5046=== PAUSE TestGlobMatch5047=== RUN TestValidateToken_ValidToken5048=== PAUSE TestValidateToken_ValidToken5049=== RUN TestValidateToken_WrongAudience5050=== PAUSE TestValidateToken_WrongAudience5051=== RUN TestValidateToken_Expired5052=== PAUSE TestValidateToken_Expired5053=== RUN TestValidateToken_BoundClaimsMismatch5054=== PAUSE TestValidateToken_BoundClaimsMismatch5055=== RUN TestValidateToken_BoundSubjectMismatch5056=== PAUSE TestValidateToken_BoundSubjectMismatch5057=== RUN TestValidateToken_MultipleProviders5058=== PAUSE TestValidateToken_MultipleProviders5059=== RUN TestValidateToken_NoMatchingProvider5060=== PAUSE TestValidateToken_NoMatchingProvider5061=== RUN TestValidateToken_KubernetesServiceAccount5062=== PAUSE TestValidateToken_KubernetesServiceAccount5063=== RUN TestNewValidator_KubernetesRequiresCA5064=== PAUSE TestNewValidator_KubernetesRequiresCA5065=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5066=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5067=== RUN TestPins_ReservedForMatchingRule5068=== PAUSE TestPins_ReservedForMatchingRule5069=== RUN TestPins_TopLevelShorthand5070=== PAUSE TestPins_TopLevelShorthand5071=== RUN TestPins_ConfigValidation5072=== PAUSE TestPins_ConfigValidation5073=== RUN TestScopes_LegacyProviderDefaultsToWrite5074=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5075=== RUN TestScopes_Rules5076=== PAUSE TestScopes_Rules5077=== RUN TestScopes_ConfigValidation5078=== PAUSE TestScopes_ConfigValidation5079=== CONT TestAudienceForIssuer5080=== CONT TestValidateToken_BoundClaimsMismatch5081=== CONT TestValidateToken_KubernetesServiceAccount5082=== CONT TestPins_ConfigValidation5083=== CONT TestValidateToken_NoMatchingProvider5084=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5085=== CONT TestScopes_ConfigValidation5086=== CONT TestScopes_Rules5087=== CONT TestValidateToken_MultipleProviders5088=== CONT TestNewValidator_KubernetesRequiresCA5089=== CONT TestPins_TopLevelShorthand5090--- PASS: TestAudienceForIssuer (0.00s)5091=== CONT TestValidateToken_Expired5092=== CONT TestScopes_LegacyProviderDefaultsToWrite5093=== CONT TestValidateToken_ValidToken5094=== CONT TestGlobMatch5095=== RUN TestGlobMatch/foo_foo5096=== PAUSE TestGlobMatch/foo_foo5097=== RUN TestGlobMatch/foo_bar5098=== PAUSE TestGlobMatch/foo_bar5099=== RUN TestGlobMatch/*_5100=== PAUSE TestGlobMatch/*_5101=== RUN TestGlobMatch/*_anything5102=== PAUSE TestGlobMatch/*_anything5103=== RUN TestGlobMatch/foo*_foo5104=== PAUSE TestGlobMatch/foo*_foo5105=== RUN TestGlobMatch/foo*_foobar5106=== PAUSE TestGlobMatch/foo*_foobar5107=== RUN TestGlobMatch/foo*_bar5108=== PAUSE TestGlobMatch/foo*_bar5109=== RUN TestGlobMatch/*bar_bar5110=== PAUSE TestGlobMatch/*bar_bar5111=== RUN TestGlobMatch/*bar_foobar5112=== PAUSE TestGlobMatch/*bar_foobar5113=== RUN TestGlobMatch/*bar_foo5114=== PAUSE TestGlobMatch/*bar_foo5115=== RUN TestGlobMatch/foo*bar_foobar5116=== PAUSE TestGlobMatch/foo*bar_foobar5117=== RUN TestGlobMatch/foo*bar_foo123bar5118=== PAUSE TestGlobMatch/foo*bar_foo123bar5119=== RUN TestGlobMatch/foo*bar_foobarbaz5120=== PAUSE TestGlobMatch/foo*bar_foobarbaz5121=== RUN TestGlobMatch/*/*_foo/bar5122=== PAUSE TestGlobMatch/*/*_foo/bar5123=== RUN TestGlobMatch/*/*_foo5124=== PAUSE TestGlobMatch/*/*_foo5125=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5126=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5127=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05128=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05129=== RUN TestGlobMatch/refs/*/main_refs/heads/main5130=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5131=== RUN TestGlobMatch/fo?_foo5132=== PAUSE TestGlobMatch/fo?_foo5133=== RUN TestGlobMatch/fo?_fo5134=== PAUSE TestGlobMatch/fo?_fo5135=== RUN TestGlobMatch/fo?_fooo5136=== PAUSE TestGlobMatch/fo?_fooo5137=== RUN TestGlobMatch/?oo_foo5138=== PAUSE TestGlobMatch/?oo_foo5139=== RUN TestGlobMatch/?oo_boo5140=== PAUSE TestGlobMatch/?oo_boo5141=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5142=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5143=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5144=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5145=== CONT TestGlobMatch/foo_foo5146=== CONT TestPins_ReservedForMatchingRule5147=== CONT TestValidateToken_WrongAudience5148--- PASS: TestPins_ConfigValidation (0.00s)5149--- PASS: TestScopes_ConfigValidation (0.00s)5150=== CONT TestValidateToken_BoundSubjectMismatch5151=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5152=== CONT TestGlobMatch/*/*_foo5153=== CONT TestGlobMatch/*/*_foo/bar5154=== CONT TestGlobMatch/foo*bar_foobarbaz5155=== CONT TestGlobMatch/foo*bar_foo123bar5156=== CONT TestGlobMatch/foo*bar_foobar5157=== CONT TestGlobMatch/*bar_foo5158=== CONT TestGlobMatch/*bar_foobar5159=== CONT TestGlobMatch/*bar_bar5160=== CONT TestGlobMatch/foo*_bar5161=== CONT TestGlobMatch/foo*_foobar5162=== CONT TestGlobMatch/foo*_foo5163=== CONT TestGlobMatch/*_anything5164=== CONT TestGlobMatch/*_5165=== CONT TestGlobMatch/foo_bar5166=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5167=== CONT TestGlobMatch/?oo_boo5168=== CONT TestGlobMatch/?oo_foo5169=== CONT TestGlobMatch/fo?_fooo5170=== CONT TestGlobMatch/fo?_fo5171=== CONT TestGlobMatch/fo?_foo5172=== CONT TestGlobMatch/refs/*/main_refs/heads/main5173=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05174=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5175--- PASS: TestGlobMatch (0.00s)5176 --- PASS: TestGlobMatch/foo_foo (0.00s)5177 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5178 --- PASS: TestGlobMatch/*/*_foo (0.00s)5179 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5180 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5181 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5182 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5183 --- PASS: TestGlobMatch/*bar_foo (0.00s)5184 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5185 --- PASS: TestGlobMatch/*bar_bar (0.00s)5186 --- PASS: TestGlobMatch/foo*_bar (0.00s)5187 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5188 --- PASS: TestGlobMatch/foo*_foo (0.00s)5189 --- PASS: TestGlobMatch/*_anything (0.00s)5190 --- PASS: TestGlobMatch/*_ (0.00s)5191 --- PASS: TestGlobMatch/foo_bar (0.00s)5192 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5193 --- PASS: TestGlobMatch/?oo_boo (0.00s)5194 --- PASS: TestGlobMatch/?oo_foo (0.00s)5195 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5196 --- PASS: TestGlobMatch/fo?_fo (0.00s)5197 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5198 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5199 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5200 --- PASS: TestGlobMatch/fo?_foo (0.00s)52012026/11/12 00:23:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36847/oidc52022026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40475/oidc52032026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41583/oidc52042026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36517/oidc5205--- PASS: TestValidateToken_Expired (6.83s)5206--- PASS: TestValidateToken_ValidToken (6.81s)5207--- PASS: TestPins_TopLevelShorthand (6.84s)5208--- PASS: TestPins_ReservedForMatchingRule (6.80s)52092026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33773/oidc5210--- PASS: TestScopes_LegacyProviderDefaultsToWrite (6.83s)52112026/11/12 00:23:53 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352122026/11/12 00:23:53 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:452395213--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (6.89s)5214--- PASS: TestValidateToken_KubernetesServiceAccount (6.90s)52152026/11/12 00:23:53 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:42293/oidc52162026/11/12 00:23:53 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:41207/oidc52172026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45541/oidc5218--- PASS: TestValidateToken_MultipleProviders (6.93s)52192026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42791/oidc5220--- PASS: TestValidateToken_WrongAudience (4.20s)5221--- PASS: TestValidateToken_BoundSubjectMismatch (4.21s)52222026/11/12 00:23:53 http: TLS handshake error from 127.0.0.1:37516: remote error: tls: bad certificate5223--- PASS: TestNewValidator_KubernetesRequiresCA (6.94s)52242026/11/12 00:23:53 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:46083/oidc5225--- PASS: TestValidateToken_NoMatchingProvider (6.98s)52262026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44859/oidc5227--- PASS: TestScopes_Rules (7.00s)52282026/11/12 00:23:53 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41561/oidc5229--- PASS: TestValidateToken_BoundClaimsMismatch (7.08s)5230PASS5231Running hook tests...5232=== RUN TestSendPathsEmpty5233=== PAUSE TestSendPathsEmpty5234=== RUN TestQueueEnqueueAndFetch5235=== PAUSE TestQueueEnqueueAndFetch5236=== RUN TestQueueDeduplication5237=== PAUSE TestQueueDeduplication5238=== RUN TestQueueRemove5239=== PAUSE TestQueueRemove5240=== RUN TestQueueFetchBatchLimit5241=== PAUSE TestQueueFetchBatchLimit5242=== RUN TestQueueRetryMovesToBack5243=== PAUSE TestQueueRetryMovesToBack5244=== RUN TestQueueFetchRemoveLifecycle5245=== PAUSE TestQueueFetchRemoveLifecycle5246=== RUN TestQueueConcurrentWriters5247=== PAUSE TestQueueConcurrentWriters5248=== RUN TestQueueRemoveLargeClosure5249=== PAUSE TestQueueRemoveLargeClosure5250=== RUN TestServerClientIntegration5251--- PASS: TestServerClientIntegration (3.08s)5252=== RUN TestServerQueueError52532026/11/12 00:23:56 ERROR Failed to queue paths error="permission denied" count=15254--- PASS: TestServerQueueError (0.00s)5255=== RUN TestGetListenerSocketActivation5256 server_test.go:220: === RUN TestGetListenerSocketActivation5257 --- PASS: TestGetListenerSocketActivation (0.00s)5258 PASS5259 5260--- PASS: TestGetListenerSocketActivation (1.32s)5261=== RUN TestServerStalledClientDoesNotBlockShutdown5262=== PAUSE TestServerStalledClientDoesNotBlockShutdown5263=== RUN TestServerBacksOffOnAcceptErrors5264=== PAUSE TestServerBacksOffOnAcceptErrors5265=== RUN TestDrainIsolatesPoisonPath5266=== PAUSE TestDrainIsolatesPoisonPath5267=== RUN TestRunNotBlockedByPoisonHead5268=== PAUSE TestRunNotBlockedByPoisonHead5269=== RUN TestDrainGivesUpWhenServerDown5270=== PAUSE TestDrainGivesUpWhenServerDown5271=== RUN TestFailedPathPrunedByLaterClosure5272=== PAUSE TestFailedPathPrunedByLaterClosure5273=== RUN TestWorkerUploadsAndRemoves5274=== PAUSE TestWorkerUploadsAndRemoves5275=== RUN TestWorkerSkipsGCdPaths5276=== PAUSE TestWorkerSkipsGCdPaths5277=== RUN TestWorkerPrunesClosureDeps5278=== PAUSE TestWorkerPrunesClosureDeps5279=== RUN TestDrainTimeout5280=== PAUSE TestDrainTimeout5281=== RUN TestWorkerRemoveFailureIsNotProgress5282=== PAUSE TestWorkerRemoveFailureIsNotProgress5283=== RUN TestWorkerKeepsPathItCannotStat5284=== PAUSE TestWorkerKeepsPathItCannotStat5285=== CONT TestSendPathsEmpty5286--- PASS: TestSendPathsEmpty (0.00s)5287=== CONT TestWorkerRemoveFailureIsNotProgress5288=== CONT TestDrainTimeout5289=== CONT TestWorkerPrunesClosureDeps5290=== CONT TestWorkerSkipsGCdPaths5291=== CONT TestWorkerKeepsPathItCannotStat5292=== CONT TestWorkerUploadsAndRemoves5293=== CONT TestFailedPathPrunedByLaterClosure5294=== CONT TestQueueConcurrentWriters5295=== CONT TestDrainGivesUpWhenServerDown5296=== CONT TestRunNotBlockedByPoisonHead5297=== CONT TestDrainIsolatesPoisonPath5298=== CONT TestQueueFetchRemoveLifecycle5299=== CONT TestServerBacksOffOnAcceptErrors53002000/01/01 00:00:00 ERROR Accept failed error="too many open files"53012000/01/01 00:00:00 ERROR Accept failed error="too many open files"53022000/01/01 00:00:00 ERROR Accept failed error="too many open files"53032000/01/01 00:00:00 ERROR Accept failed error="too many open files"53042000/01/01 00:00:00 ERROR Accept failed error="too many open files"53052000/01/01 00:00:00 ERROR Accept failed error="too many open files"53062000/01/01 00:00:00 ERROR Accept failed error="too many open files"53072000/01/01 00:00:00 ERROR Accept failed error="too many open files"5308=== CONT TestQueueDeduplication5309--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5310=== CONT TestServerStalledClientDoesNotBlockShutdown5311=== CONT TestQueueEnqueueAndFetch5312=== CONT TestQueueRemoveLargeClosure5313=== CONT TestQueueRetryMovesToBack5314=== CONT TestQueueFetchBatchLimit5315=== CONT TestQueueRemove53162026/11/12 00:23:57 INFO Upload queue status pending=253172026/11/12 00:23:57 INFO Uploading batch count=253182000/01/01 00:00:00 INFO Uploading batch count=253192026/11/12 00:23:57 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2819382181/002/nonexistent53202026/11/12 00:23:57 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153212026/11/12 00:23:57 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2819382181/002/nonexistent53222026/11/12 00:23:57 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153232026/11/12 00:23:57 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2819382181/002/nonexistent53242026/11/12 00:23:57 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153252026/11/12 00:23:57 ERROR Drain finished with paths left in queue remaining=15326--- PASS: TestWorkerRemoveFailureIsNotProgress (0.04s)53272000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253282000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45329--- PASS: TestDrainTimeout (0.05s)53302026/11/12 00:23:57 INFO Upload queue status pending=353312026/11/12 00:23:57 INFO Uploading batch count=153322026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=153332026/11/12 00:23:57 INFO Uploading batch count=453342026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=453352026/11/12 00:23:57 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath3678587209/002/bbb53362026/11/12 00:23:57 INFO Uploading batch count=153372026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=153382026/11/12 00:23:57 INFO Uploading batch count=153392026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=153402026/11/12 00:23:57 INFO Uploading batch count=153412026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=153422026/11/12 00:23:57 ERROR Drain finished with paths left in queue remaining=15343--- PASS: TestDrainIsolatesPoisonPath (0.06s)5344--- PASS: TestQueueDeduplication (0.06s)53452026/11/12 00:23:57 INFO Upload queue status pending=253462026/11/12 00:23:57 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths607266687/002/nonexistent53472026/11/12 00:23:57 INFO Uploading batch count=153482026/11/12 00:23:57 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1793053162/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1793053162/002/locked/aaa: permission denied"53492026/11/12 00:23:57 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1793053162/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1793053162/002/locked/aaa: permission denied"53502026/11/12 00:23:57 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1793053162/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1793053162/002/locked/aaa: permission denied"53512026/11/12 00:23:57 ERROR Drain finished with paths left in queue remaining=153522026/11/12 00:23:57 INFO Upload queue status pending=253532026/11/12 00:23:57 INFO Uploading batch count=15354--- PASS: TestQueueRemove (0.07s)5355--- PASS: TestWorkerKeepsPathItCannotStat (0.08s)53562026/11/12 00:23:57 INFO Uploading batch count=253572026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=253582026/11/12 00:23:57 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown761711080/002/a53592026/11/12 00:23:57 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown761711080/002/b53602026/11/12 00:23:57 INFO Uploading batch count=253612026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=253622026/11/12 00:23:57 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown761711080/002/c53632026/11/12 00:23:57 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown761711080/002/d53642026/11/12 00:23:57 INFO Uploading batch count=253652026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=253662026/11/12 00:23:57 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown761711080/002/e53672026/11/12 00:23:57 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown761711080/002/f53682026/11/12 00:23:57 ERROR Drain finished with paths left in queue remaining=1053692026/11/12 00:23:57 INFO Uploading batch count=153702026/11/12 00:23:57 ERROR Upload failed error="upload failed" count=153712026/11/12 00:23:57 INFO Uploading batch count=153722026/11/12 00:23:57 INFO Uploading batch count=15373--- PASS: TestFailedPathPrunedByLaterClosure (0.09s)5374--- PASS: TestQueueFetchBatchLimit (0.09s)5375--- PASS: TestWorkerPrunesClosureDeps (0.10s)5376--- PASS: TestQueueRetryMovesToBack (0.09s)5377--- PASS: TestDrainGivesUpWhenServerDown (0.10s)5378--- PASS: TestQueueEnqueueAndFetch (0.09s)5379--- PASS: TestQueueFetchRemoveLifecycle (0.10s)5380--- PASS: TestWorkerUploadsAndRemoves (0.10s)5381--- PASS: TestWorkerSkipsGCdPaths (0.12s)53822026/11/12 00:24:00 ERROR Failed to decode request error="read unix /build/hook2888274748/test.sock->@: i/o timeout"53832026/11/12 00:24:00 ERROR Failed to write response error="write unix /build/hook2888274748/test.sock->@: i/o timeout"5384--- PASS: TestServerStalledClientDoesNotBlockShutdown (2.76s)53852026/11/12 00:24:00 INFO Uploading batch count=153862026/11/12 00:24:00 INFO Uploading batch count=153872026/11/12 00:24:00 INFO Uploading batch count=153882026/11/12 00:24:00 ERROR Upload failed error="upload failed" count=153892026/11/12 00:24:00 INFO Uploading batch count=153902026/11/12 00:24:00 ERROR Upload failed error="upload failed" count=153912026/11/12 00:24:00 INFO Uploading batch count=153922026/11/12 00:24:00 ERROR Upload failed error="upload failed" count=153932026/11/12 00:24:00 INFO Uploading batch count=153942026/11/12 00:24:00 ERROR Upload failed error="upload failed" count=153952026/11/12 00:24:00 ERROR Drain finished with paths left in queue remaining=15396--- PASS: TestRunNotBlockedByPoisonHead (2.79s)5397--- PASS: TestQueueConcurrentWriters (2.83s)5398--- PASS: TestQueueRemoveLargeClosure (4.72s)5399PASS5400Running signing tests...5401=== RUN TestExternalSigner5402=== PAUSE TestExternalSigner5403=== RUN TestExternalSignerWrongPublicKey5404=== PAUSE TestExternalSignerWrongPublicKey5405=== RUN TestExternalSignerErrors5406=== PAUSE TestExternalSignerErrors5407=== RUN TestExternalSignerRestartsAfterCrash5408=== PAUSE TestExternalSignerRestartsAfterCrash5409=== RUN TestExternalSignerProgramErrorKeepsProcess5410=== PAUSE TestExternalSignerProgramErrorKeepsProcess5411=== RUN TestExternalSignerClose5412=== PAUSE TestExternalSignerClose5413=== RUN TestExternalSignerRejectsInvalidPublicKey5414=== PAUSE TestExternalSignerRejectsInvalidPublicKey5415=== RUN TestExternalSignerCancellation5416=== PAUSE TestExternalSignerCancellation5417=== RUN TestGenerateFingerprint5418=== PAUSE TestGenerateFingerprint5419=== RUN TestParseSigningKey5420=== PAUSE TestParseSigningKey5421=== RUN TestSign5422=== PAUSE TestSign5423=== RUN TestSignBatch5424=== PAUSE TestSignBatch5425=== RUN TestNilKeyPublicKey5426=== PAUSE TestNilKeyPublicKey5427=== RUN TestSignErrors5428=== PAUSE TestSignErrors5429=== CONT TestExternalSigner5430=== CONT TestSignBatch5431=== CONT TestExternalSignerCancellation5432=== CONT TestParseSigningKey5433=== RUN TestParseSigningKey/valid_32-byte_key5434=== PAUSE TestParseSigningKey/valid_32-byte_key5435=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5436=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5437=== RUN TestParseSigningKey/no_colon5438=== PAUSE TestParseSigningKey/no_colon5439=== RUN TestParseSigningKey/empty_name5440=== PAUSE TestParseSigningKey/empty_name5441=== CONT TestSign5442=== CONT TestSignErrors5443=== CONT TestNilKeyPublicKey5444--- PASS: TestNilKeyPublicKey (0.00s)5445=== CONT TestExternalSignerProgramErrorKeepsProcess5446=== CONT TestExternalSignerRejectsInvalidPublicKey5447=== RUN TestExternalSignerRejectsInvalidPublicKey/#005448=== CONT TestExternalSignerClose5449=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005450=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5451=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5452=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5453=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5454=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5455=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5456=== CONT TestExternalSignerRestartsAfterCrash5457=== CONT TestExternalSignerWrongPublicKey5458=== CONT TestGenerateFingerprint5459=== RUN TestParseSigningKey/invalid_base645460=== PAUSE TestParseSigningKey/invalid_base645461=== RUN TestParseSigningKey/wrong_length5462=== PAUSE TestParseSigningKey/wrong_length5463=== CONT TestParseSigningKey/valid_32-byte_key5464=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5465=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5466=== CONT TestExternalSignerRejectsInvalidPublicKey/#005467=== CONT TestExternalSignerErrors5468=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5469=== RUN TestExternalSignerErrors/invalid_JSON5470=== PAUSE TestExternalSignerErrors/invalid_JSON5471=== RUN TestExternalSignerErrors/wrong_shape5472=== PAUSE TestExternalSignerErrors/wrong_shape5473=== RUN TestExternalSignerErrors/wrong_id5474=== PAUSE TestExternalSignerErrors/wrong_id5475=== RUN TestExternalSignerErrors/exit_at_startup5476=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5477=== PAUSE TestExternalSignerErrors/exit_at_startup5478=== RUN TestExternalSignerErrors/null_signatures5479=== PAUSE TestExternalSignerErrors/null_signatures5480=== RUN TestExternalSignerErrors/unrequested_key5481=== PAUSE TestExternalSignerErrors/unrequested_key5482=== RUN TestExternalSignerErrors/missing_signature_name5483=== PAUSE TestExternalSignerErrors/missing_signature_name5484=== RUN TestExternalSignerErrors/wrong_signature_name5485=== PAUSE TestExternalSignerErrors/wrong_signature_name5486=== RUN TestExternalSignerErrors/program_error5487=== PAUSE TestExternalSignerErrors/program_error5488=== RUN TestExternalSignerErrors/exit_on_request5489=== PAUSE TestExternalSignerErrors/exit_on_request5490=== RUN TestExternalSignerErrors/null5491=== PAUSE TestExternalSignerErrors/null5492=== RUN TestExternalSignerErrors/missing_signatures5493=== PAUSE TestExternalSignerErrors/missing_signatures5494=== RUN TestExternalSignerErrors/wrong_signatures_shape5495=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5496=== RUN TestExternalSignerErrors/too_few_signatures5497=== PAUSE TestExternalSignerErrors/too_few_signatures5498=== RUN TestExternalSignerErrors/invalid_signature_base645499=== PAUSE TestExternalSignerErrors/invalid_signature_base645500=== RUN TestExternalSignerErrors/wrong_signature_length5501=== PAUSE TestExternalSignerErrors/wrong_signature_length5502=== CONT TestExternalSignerErrors/invalid_JSON5503=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5504=== CONT TestExternalSignerErrors/too_few_signatures5505=== CONT TestExternalSignerErrors/program_error5506=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5507=== CONT TestParseSigningKey/wrong_length5508=== CONT TestExternalSignerErrors/wrong_signature_length5509=== RUN TestGenerateFingerprint/basic_with_references5510=== PAUSE TestGenerateFingerprint/basic_with_references5511=== RUN TestGenerateFingerprint/no_references5512=== PAUSE TestGenerateFingerprint/no_references5513=== CONT TestExternalSignerErrors/wrong_signatures_shape5514=== CONT TestExternalSignerErrors/invalid_signature_base645515=== CONT TestExternalSignerErrors/exit_on_request5516=== CONT TestExternalSignerErrors/null_signatures5517=== CONT TestParseSigningKey/no_colon5518=== CONT TestExternalSignerErrors/null5519=== RUN TestSignErrors/nil_key5520=== CONT TestExternalSignerErrors/missing_signature_name5521=== CONT TestExternalSignerErrors/wrong_id5522=== CONT TestExternalSignerErrors/wrong_shape5523=== CONT TestExternalSignerErrors/unrequested_key5524=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5525=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5526--- PASS: TestSign (0.00s)5527--- PASS: TestSignBatch (0.00s)5528=== CONT TestExternalSignerErrors/missing_signatures5529=== CONT TestExternalSignerErrors/wrong_signature_name5530=== CONT TestExternalSignerErrors/exit_at_startup5531=== PAUSE TestSignErrors/nil_key5532=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5533=== RUN TestSignErrors/nil_narinfo5534=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5535=== RUN TestGenerateFingerprint/invalid_nar_hash_length5536=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5537=== RUN TestGenerateFingerprint/invalid_store_path_prefix5538=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5539=== PAUSE TestSignErrors/nil_narinfo5540=== RUN TestSignErrors/invalid_narinfo5541=== PAUSE TestSignErrors/invalid_narinfo5542=== RUN TestSignErrors/canceled5543=== PAUSE TestSignErrors/canceled5544=== CONT TestParseSigningKey/invalid_base645545=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5546=== CONT TestParseSigningKey/empty_name5547=== CONT TestSignErrors/nil_key5548=== CONT TestSignErrors/invalid_narinfo5549--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5550 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5551 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5552 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5553 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5554 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5555=== CONT TestSignErrors/canceled5556=== RUN TestGenerateFingerprint/invalid_reference_prefix5557=== CONT TestSignErrors/nil_narinfo5558--- PASS: TestParseSigningKey (0.00s)5559 --- PASS: TestParseSigningKey/wrong_length (0.00s)5560 --- PASS: TestParseSigningKey/no_colon (0.00s)5561 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5562 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5563 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5564 --- PASS: TestParseSigningKey/empty_name (0.00s)5565=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5566=== CONT TestGenerateFingerprint/basic_with_references5567--- PASS: TestSignErrors (0.01s)5568 --- PASS: TestSignErrors/nil_key (0.00s)5569 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5570 --- PASS: TestSignErrors/canceled (0.00s)5571 --- PASS: TestSignErrors/nil_narinfo (0.00s)5572=== CONT TestGenerateFingerprint/invalid_nar_hash_length5573=== CONT TestGenerateFingerprint/invalid_store_path_prefix5574=== CONT TestGenerateFingerprint/invalid_reference_prefix5575=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5576=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5577--- PASS: TestExternalSignerWrongPublicKey (0.01s)5578=== CONT TestGenerateFingerprint/no_references5579--- PASS: TestGenerateFingerprint (0.00s)5580 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5581 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5582 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5583 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5584 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5585 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5586 --- PASS: TestGenerateFingerprint/no_references (0.00s)5587--- PASS: TestExternalSignerClose (0.02s)5588--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.02s)5589--- PASS: TestExternalSignerRestartsAfterCrash (0.02s)5590--- PASS: TestExternalSignerErrors (0.00s)5591 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5592 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5593 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.02s)5594 --- PASS: TestExternalSignerErrors/null (0.02s)5595 --- PASS: TestExternalSignerErrors/wrong_shape (0.02s)5596 --- PASS: TestExternalSignerErrors/exit_on_request (0.02s)5597 --- PASS: TestExternalSignerErrors/missing_signatures (0.02s)5598 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.02s)5599 --- PASS: TestExternalSignerErrors/null_signatures (0.02s)5600 --- PASS: TestExternalSignerErrors/missing_signature_name (0.02s)5601 --- PASS: TestExternalSignerErrors/exit_at_startup (0.02s)5602 --- PASS: TestExternalSignerErrors/too_few_signatures (0.02s)5603 --- PASS: TestExternalSignerErrors/unrequested_key (0.02s)5604 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.02s)5605 --- PASS: TestExternalSignerErrors/program_error (0.02s)5606 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.02s)5607--- PASS: TestExternalSigner (0.02s)5608--- PASS: TestExternalSignerCancellation (0.03s)5609PASS5610Running OpenBao signer tests...5611=== RUN TestSign5612=== PAUSE TestSign5613=== RUN TestSignErrors5614=== PAUSE TestSignErrors5615=== RUN TestServe5616=== PAUSE TestServe5617=== RUN TestServeRejectsGarbage5618=== PAUSE TestServeRejectsGarbage5619=== CONT TestSign5620=== CONT TestServe5621=== CONT TestServeRejectsGarbage5622--- PASS: TestServeRejectsGarbage (0.00s)5623=== CONT TestSignErrors5624=== RUN TestSignErrors/http_error5625=== PAUSE TestSignErrors/http_error5626=== RUN TestSignErrors/invalid_json5627=== PAUSE TestSignErrors/invalid_json5628=== RUN TestSignErrors/no_prefix5629=== PAUSE TestSignErrors/no_prefix5630=== RUN TestSignErrors/invalid_base645631=== PAUSE TestSignErrors/invalid_base645632=== CONT TestSignErrors/http_error5633=== CONT TestSignErrors/invalid_base645634=== CONT TestSignErrors/no_prefix5635=== CONT TestSignErrors/invalid_json5636--- PASS: TestServe (0.00s)5637--- PASS: TestSign (0.01s)5638--- PASS: TestSignErrors (0.00s)5639 --- PASS: TestSignErrors/invalid_json (0.01s)5640 --- PASS: TestSignErrors/http_error (0.01s)5641 --- PASS: TestSignErrors/no_prefix (0.01s)5642 --- PASS: TestSignErrors/invalid_base64 (0.01s)5643PASS

succeeded niks3-go-unit-tests checks.aarch64-linux.go-unit-tests · build #497 · raw

succeeded `niks3-go-unit-tests` `checks.aarch64-linux.go-unit-tests` · build #497 · raw