niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #496
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.34s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushAnswersEveryLineWhenCanceled86=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestDoServerRequestAttachesToken116=== CONT TestScriptTokenEmptyCommand117--- PASS: TestScriptTokenEmptyCommand (0.00s)118=== CONT TestScriptTokenScriptFails119=== CONT TestScriptTokenBadJSON120=== CONT TestScriptTokenEmptyToken121=== CONT TestScriptTokenCachesUntilRefresh122=== CONT TestScriptTokenNoExpiryRerunsEveryCall123=== CONT TestFileTokenEmpty124--- PASS: TestFileTokenEmpty (0.00s)125=== CONT TestFileTokenMissing126--- PASS: TestFileTokenMissing (0.00s)127=== CONT TestFileTokenReadsAndCaches128--- PASS: TestFileTokenReadsAndCaches (0.00s)129=== CONT TestStaticToken130--- PASS: TestStaticToken (0.00s)131=== CONT TestSetClientTLSErrors132--- PASS: TestScriptTokenCachesUntilRefresh (0.02s)133=== CONT TestPathInfoCACompatibility134=== CONT TestPartSizeForNAR135=== CONT TestParsePathInfoJSONMultiplePaths136=== CONT TestParsePathInfoJSON137=== CONT TestPathInfoHashCompatibility138=== CONT TestGetStorePathHash139=== CONT TestConvertHashToNix32140=== CONT TestEncodeNixBase32WithRealHash141=== CONT TestEncodeNixBase32142=== CONT TestDumpPathWriterError143=== CONT TestSetClientTLSDoesNotMutateDefaultTransport144=== CONT TestSetClientTLS145=== CONT TestStreamPushStopsWhenCanceled146=== CONT TestUploadMultipart_ReturnsAfterDroppedBody147=== RUN TestPathInfoCACompatibility/null_ca_field148=== CONT TestRegisterUploadedObjectReusesConnections149=== CONT TestUploadMultipart_PartsInParallel150--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.47s)151=== CONT TestStreamPushAnswersEveryLineWhenCanceled152=== CONT TestDumpPathSingleFile153--- PASS: TestStreamPushStopsWhenCanceled (0.30s)154=== CONT TestStreamPushReportsSkippedPaths155--- PASS: TestDoServerRequestAttachesToken (0.49s)156=== CONT TestStreamPushReportsSignatures157=== RUN TestSetClientTLSErrors/missing_cert_file158=== PAUSE TestSetClientTLSErrors/missing_cert_file159=== RUN TestSetClientTLSErrors/missing_key_file160=== CONT TestClientSignaturesByStorePath161=== RUN TestParsePathInfoJSON/Nix_format162=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)163=== PAUSE TestParsePathInfoJSON/Nix_format164=== RUN TestGetStorePathHash/valid_store_path165=== PAUSE TestGetStorePathHash/valid_store_path166=== RUN TestGetStorePathHash/basename_without_hyphen_should_error167=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error168=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)1692000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2170=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon171=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon172=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI173=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI174=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512175=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512176=== RUN TestEncodeNixBase32/test_string_hash177=== PAUSE TestEncodeNixBase32/test_string_hash178=== RUN TestEncodeNixBase32/empty_input179=== PAUSE TestEncodeNixBase32/empty_input180=== CONT TestStreamPushBatchesUnderLoad181=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false182=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false183=== CONT TestStreamPushReportsEveryPath184=== CONT TestStreamPushIsolatesFailures185=== RUN TestConvertHashToNix32/SRI_format_to_Nix32186=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32187=== RUN TestConvertHashToNix32/already_Nix32_format188=== PAUSE TestConvertHashToNix32/already_Nix32_format189=== RUN TestConvertHashToNix32/invalid_format190=== PAUSE TestConvertHashToNix32/invalid_format1912026/11/12 00:21:57 ERROR Upload failed error="bad path" count=3192=== CONT TestDoWithRetry_FinalResponseBodyReadable193=== CONT TestRetryTimeoutOutlastsMaxRetries194=== CONT TestShellSplitErrors195=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true196=== RUN TestParsePathInfoJSON/Lix_format197=== PAUSE TestParsePathInfoJSON/Lix_format198=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error199=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths200--- PASS: TestScriptTokenEmptyToken (0.58s)201=== CONT TestStreamPushRequestLine202=== CONT TestStreamPushGivesUpOnDeadServer203=== RUN TestPartSizeForNAR/zero_stays_at_minimum204=== CONT TestShellSplit205=== RUN TestParsePathInfoJSON/empty_input206--- PASS: TestScriptTokenBadJSON (0.58s)207--- PASS: TestScriptTokenScriptFails (0.58s)208=== CONT TestDoWithRetry_BodyReplayedViaGetBody209=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true210--- PASS: TestStreamPushReportsEveryPath (0.00s)211--- PASS: TestEncodeNixBase32WithRealHash (0.00s)212--- PASS: TestShellSplit (0.00s)213--- PASS: TestShellSplitErrors (0.00s)214--- PASS: TestStreamPushIsolatesFailures (0.00s)215--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)216=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer2172026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:329652182026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:329652192026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:32965220=== RUN TestSetClientTLS/rejects_connection_without_client_cert2212026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:329652222026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:329652232026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:329652242026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:329652252026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:329652262026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:329652272026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:329652282026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:329652292026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:329652302026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:329652312026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:329652322026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:329652332026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:329652342026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:329652352026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:329652362026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:32965237--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.04s)238=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess2392026/11/12 00:21:57 WARN Rate limiter enabled after throttle name=server-test rate=5240=== PAUSE TestParsePathInfoJSON/empty_input241=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error242=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error243--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.02s)244=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths245=== CONT TestResolveStorePath246=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths247=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry248=== PAUSE TestPathInfoCACompatibility/null_ca_field249=== RUN TestPathInfoCACompatibility/old_string_format_-_text250=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths251=== CONT TestRegisterUploadedObject_OutlivesPush252=== CONT TestRateLimiterFeedback253=== RUN TestRateLimiterFeedback/429_enables_limiter254=== PAUSE TestRateLimiterFeedback/429_enables_limiter255=== RUN TestRateLimiterFeedback/503_enables_limiter2562026/11/12 00:21:57 WARN Rate limiter enabled after throttle name=server-test rate=5257=== PAUSE TestRateLimiterFeedback/503_enables_limiter258=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter259=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter2602026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:40791261=== PAUSE TestSetClientTLSErrors/missing_key_file262=== RUN TestSetClientTLSErrors/missing_ca_file263=== PAUSE TestSetClientTLSErrors/missing_ca_file264=== RUN TestSetClientTLSErrors/invalid_ca_file265=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter266=== CONT TestDumpPathMatchesNix267--- PASS: TestStreamPushReportsSkippedPaths (0.00s)2682026/11/12 00:21:57 WARN Rate limiter enabled after throttle name=server-test rate=52692026/11/12 00:21:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:352452702026/11/12 00:21:58 ERROR Upload failed error=boom count=1271--- PASS: TestStreamPushReportsSignatures (0.62s)272=== CONT TestCaseHackSuffix273--- PASS: TestClientSignaturesByStorePath (0.70s)274=== CONT TestFilterOversizedClosures275=== PAUSE TestSetClientTLSErrors/invalid_ca_file2762026/11/12 00:22:00 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:35663/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:35663/api/uploads/complete2772026/11/12 00:22:00 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"278=== RUN TestParsePathInfoJSON/whitespace_only279=== PAUSE TestParsePathInfoJSON/whitespace_only280=== RUN TestFilterOversizedClosures/no_limit_keeps_everything281=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything282=== RUN TestParsePathInfoJSON/invalid_JSON283=== PAUSE TestParsePathInfoJSON/invalid_JSON284=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter2852026/11/12 00:22:02 WARN Rate limiter backed off name=server-test rate=5286=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped287=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica288=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)289--- PASS: TestDoWithRetry_FinalResponseBodyReadable (5.30s)2902026/11/12 00:22:02 WARN Rate limiter backed off name=server-test rate=5291=== CONT TestEncodeNixBase32/test_string_hash2922026/11/12 00:22:02 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:35245293=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon294--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (4.76s)295=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512296--- PASS: TestResolveStorePath (4.15s)297=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI298--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (5.26s)299=== CONT TestEncodeNixBase32/empty_input300=== CONT TestConvertHashToNix32/SRI_format_to_Nix32301=== CONT TestConvertHashToNix32/invalid_format302=== CONT TestConvertHashToNix32/already_Nix32_format303=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false304--- PASS: TestConvertHashToNix32 (0.00s)305 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)306 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)307 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)308=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true309=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun310=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert311=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA312=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA313=== RUN TestSetClientTLS/preserves_debug_logging_transport314=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped315=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum316=== RUN TestPartSizeForNAR/small_stays_at_minimum317=== PAUSE TestPartSizeForNAR/small_stays_at_minimum318--- PASS: TestEncodeNixBase32 (0.56s)319 --- PASS: TestEncodeNixBase32/empty_input (0.00s)320 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)321=== PAUSE TestSetClientTLS/preserves_debug_logging_transport322=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths3232026/11/12 00:22:02 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:35663/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:35663/api/uploads/complete324=== CONT TestSetClientTLSErrors/missing_cert_file3252026/11/12 00:22:02 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"326=== CONT TestSetClientTLSErrors/invalid_ca_file327=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error328=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum329=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum330=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths3312026/11/12 00:22:02 ERROR Upload failed error="connection refused" count=203322026/11/12 00:22:02 ERROR Server seems unavailable, giving up on batch untried=173332026/11/12 00:22:02 ERROR Upload failed error=boom count=1334--- PASS: TestPathInfoHashCompatibility (0.00s)335 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)336 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)337 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)338 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)339=== CONT TestSetClientTLSErrors/missing_ca_file340=== CONT TestParsePathInfoJSON/invalid_JSON341=== CONT TestParsePathInfoJSON/whitespace_only342=== CONT TestParsePathInfoJSON/empty_input343=== CONT TestParsePathInfoJSON/Lix_format344--- PASS: TestStreamPushBatchesUnderLoad (0.02s)345 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)346 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)347=== CONT TestRateLimiterFeedback/429_enables_limiter348=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text349--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)350=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter351=== CONT TestRateLimiterFeedback/503_enables_limiter352=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive353=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive354=== RUN TestPathInfoCACompatibility/new_structured_format_-_text355=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text356=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method357=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method358--- PASS: TestParsePathInfoJSONMultiplePaths (0.54s)359 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)360 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)361--- PASS: TestRegisterUploadedObject_OutlivesPush (4.77s)362=== CONT TestGetStorePathHash/valid_store_path363=== CONT TestSetClientTLS/rejects_connection_without_client_cert3642026/11/12 00:22:02 WARN Rate limiter enabled after throttle name=s3-test rate=5365=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error3662026/11/12 00:22:02 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:45815367=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error3682026/11/12 00:22:02 WARN Rate limiter enabled after throttle name=server-test rate=53692026/11/12 00:22:02 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:41621370=== CONT TestGetStorePathHash/basename_without_hyphen_should_error3712026/11/12 00:22:02 WARN Rate limiter enabled after throttle name=server-test rate=53722026/11/12 00:22:02 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:46503373=== CONT TestPathInfoCACompatibility/null_ca_field374=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method375=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3762026/11/12 00:22:02 WARN Rate limiter backed off name=server-test rate=5377=== CONT TestPathInfoCACompatibility/old_string_format_-_text378--- PASS: TestGetStorePathHash (5.30s)379 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)380 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)381 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)382 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)3832026/11/12 00:22:02 WARN Rate limiter backed off name=server-test rate=5384--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (4.77s)385=== RUN TestFilterOversizedClosures/all_closures_skipped386=== PAUSE TestFilterOversizedClosures/all_closures_skipped387=== CONT TestFilterOversizedClosures/no_limit_keeps_everything388=== CONT TestSetClientTLSErrors/missing_key_file3892026/11/12 00:22:02 INFO Garbage collection started390=== CONT TestFilterOversizedClosures/all_closures_skipped3912026/11/12 00:22:02 INFO Garbage collection started392=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3932026/11/12 00:22:02 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000394=== CONT TestParsePathInfoJSON/Nix_format395=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA396=== CONT TestSetClientTLS/preserves_debug_logging_transport397=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts398=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive399=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter400--- PASS: TestParsePathInfoJSON (5.86s)401 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)402 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)403 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)404 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)405 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)406=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts407--- PASS: TestPathInfoCACompatibility (5.86s)408 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)409 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)410 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)411 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)412 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)413--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (4.77s)414--- PASS: TestRateLimiterFeedback (4.76s)415 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)416 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)417 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)418 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)4192026/11/12 00:22:02 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50420--- PASS: TestFilterOversizedClosures (3.05s)421 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)422 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)423 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)424--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)425=== RUN TestPartSizeForNAR/1_TiB4262026/11/12 00:22:02 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0427=== PAUSE TestPartSizeForNAR/1_TiB428=== RUN TestPartSizeForNAR/5_TiB_S3_max_object429=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object430=== RUN TestPartSizeForNAR/capped_at_5_GiB431=== PAUSE TestPartSizeForNAR/capped_at_5_GiB432--- PASS: TestSetClientTLSErrors (2.81s)433 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)434 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)435 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)436 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)437=== CONT TestPartSizeForNAR/zero_stays_at_minimum438=== CONT TestPartSizeForNAR/capped_at_5_GiB439=== CONT TestPartSizeForNAR/5_TiB_S3_max_object440=== CONT TestPartSizeForNAR/1_TiB441=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts442=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum443=== CONT TestPartSizeForNAR/small_stays_at_minimum444--- PASS: TestPartSizeForNAR (5.43s)445 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)446 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)447 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)448 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)449 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)450 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)451 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)4522026/11/12 00:22:02 WARN Garbage collection finished on another replica. Its statistics are not available here453--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)4542026/11/12 00:22:02 http: TLS handshake error from 127.0.0.1:57114: remote error: tls: bad certificate455--- PASS: TestSetClientTLS (5.29s)456 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.02s)457 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.02s)458 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4592026/11/12 00:22:03 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:35663/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:35663/api/uploads/complete4602026/11/12 00:22:03 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"461--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (5.24s)462--- PASS: TestDumpPathWriterError (6.35s)463--- PASS: TestRegisterUploadedObjectReusesConnections (5.25s)464--- PASS: TestStreamPushRequestLine (0.49s)465--- PASS: TestCaseHackSuffix (4.65s)466--- PASS: TestDumpPathSingleFile (5.27s)467--- PASS: TestUploadMultipart_PartsInParallel (5.83s)468--- PASS: TestDumpPathMatchesNix (5.99s)469PASS470Running server tests...471The files belonging to this database system will be owned by user "nixbld".472This user must also own the server process.473474The database cluster will be initialized with locale "C".475The default database encoding has accordingly been set to "SQL_ASCII".476The default text search configuration will be set to "english".477478Data page checksums are enabled.479480creating directory /build/postgres1030177354/data ... ok481creating subdirectories ... ok482selecting dynamic shared memory implementation ... posix483selecting default "max_connections" ... 100484selecting default "shared_buffers" ... 128MB485selecting default time zone ... UTC486creating configuration files ... ok487running bootstrap script ... ok488performing post-bootstrap initialization ... ok489syncing data to disk ... ok490491initdb: warning: enabling "trust" authentication for local connections492initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.493494Success. You can now start the database server using:495496 pg_ctl -D /build/postgres1030177354/data -l logfile start497498/build/postgres1030177354:5432 - no response4992026-11-12 00:22:10.417 UTC [132] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5002026-11-12 00:22:10.418 UTC [132] LOG: listening on Unix socket "/build/postgres1030177354/.s.PGSQL.5432"5012026-11-12 00:22:10.427 UTC [139] LOG: database system was shut down at 2026-11-12 00:22:10 UTC5022026-11-12 00:22:10.432 UTC [132] LOG: database system is ready to accept connections503/build/postgres1030177354:5432 - accepting connections504{"timestamp":"2026-11-12T00:22:33.338357529Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3270d1c3-e450-41bb-bbd2-23507d0f3c1b","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":10,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(194)"}505{"timestamp":"2026-11-12T00:22:33.451126211Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"a98a57e1-e440-4f31-8f5e-ec5b362006f1","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(201)"}506=== RUN TestAbortSucceeded507=== PAUSE TestAbortSucceeded508=== RUN TestService_AuthMiddleware509=== PAUSE TestService_AuthMiddleware510=== RUN TestService_AuthMiddleware_MTLSProxyHeader511=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader512=== RUN TestService_AuthMiddleware_MTLSBoundSubjects513=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects514=== RUN TestService_ReadAuthMiddleware515=== PAUSE TestService_ReadAuthMiddleware516=== RUN TestService_AuthMiddleware_OIDC517=== PAUSE TestService_AuthMiddleware_OIDC518=== RUN TestService_RequireScope_OIDC519=== PAUSE TestService_RequireScope_OIDC520=== RUN TestService_ReadScope_PublicByDefault521=== PAUSE TestService_ReadScope_PublicByDefault522=== RUN TestCacheConfigHandler523=== PAUSE TestCacheConfigHandler524=== RUN TestCacheStatsHandler525=== PAUSE TestCacheStatsHandler526=== RUN TestClientCADerivations527=== PAUSE TestClientCADerivations528=== RUN TestClientErrorHandling529=== PAUSE TestClientErrorHandling530=== RUN TestClientIntegration531=== PAUSE TestClientIntegration532=== RUN TestClientMultipleUploads533=== PAUSE TestClientMultipleUploads534=== RUN TestClientWithDependencies535=== PAUSE TestClientWithDependencies536=== RUN TestClientSharedPathCommittedMidPush537=== PAUSE TestClientSharedPathCommittedMidPush538=== RUN TestPinProtectsFromGC539=== PAUSE TestPinProtectsFromGC540=== RUN TestClientPushesUseOnePush541=== PAUSE TestClientPushesUseOnePush542=== RUN TestClientFallsBackToClosures543=== PAUSE TestClientFallsBackToClosures544=== RUN TestDeleteClosuresRacingPins545=== PAUSE TestDeleteClosuresRacingPins546=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock547=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock548=== RUN TestResolveDBConnectionString549=== PAUSE TestResolveDBConnectionString550=== RUN TestConnectWaitsForAPeerMigration551=== PAUSE TestConnectWaitsForAPeerMigration552=== RUN TestLeadElectsOneAndHandsOver553=== PAUSE TestLeadElectsOneAndHandsOver554=== RUN TestLeadIncumbentWinsAfterRestart5552026/11/12 00:22:34 INFO lead: acquired remote=192.0.2.1:1234 generation=15562026/11/12 00:22:35 INFO lead: released remote=192.0.2.1:1234 generation=15572026/11/12 00:22:35 INFO lead: acquired remote=192.0.2.1:1234 generation=25582026/11/12 00:22:36 INFO lead: released remote=192.0.2.1:1234 generation=2559--- PASS: TestLeadIncumbentWinsAfterRestart (2.57s)560=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent561=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent562=== RUN TestLeadIncumbentAnnouncesAtOnce563=== PAUSE TestLeadIncumbentAnnouncesAtOnce564=== RUN TestLeadGenerationGrowsWithEveryGrant565=== PAUSE TestLeadGenerationGrowsWithEveryGrant566=== RUN TestLeadEndsOnShutdown567=== PAUSE TestLeadEndsOnShutdown568=== RUN TestLeadEndsWhenItsConnectionHangs5692026/11/12 00:22:36 INFO lead: acquired remote=192.0.2.1:1234 generation=15702026-11-12 00:22:38.339 UTC [389] FATAL: terminating connection due to administrator command5712026/11/12 00:22:38 INFO lead: acquired remote=192.0.2.1:1234 generation=25722026/11/12 00:22:39 WARN lead: lock connection lost error="timeout: context deadline exceeded"5732026/11/12 00:22:39 INFO lead: released remote=192.0.2.1:1234 generation=15742026/11/12 00:22:39 INFO lead: released remote=192.0.2.1:1234 generation=2575--- PASS: TestLeadEndsWhenItsConnectionHangs (3.76s)576=== RUN TestGCAdvisoryLockBlocksConcurrentRun577=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun578=== RUN TestGCBugBareHashReferences579=== PAUSE TestGCBugBareHashReferences580=== RUN TestGCMetrics581=== PAUSE TestGCMetrics582=== RUN TestPushDedupSurvivesConcurrentGC583=== PAUSE TestPushDedupSurvivesConcurrentGC584=== RUN TestDeduplicatedObjectsRecordedAsPending585=== PAUSE TestDeduplicatedObjectsRecordedAsPending586=== RUN TestGCSweepSkipsPendingObjects587=== PAUSE TestGCSweepSkipsPendingObjects588=== RUN TestTombstonedObjectOfferedWithoutWaiting589=== PAUSE TestTombstonedObjectOfferedWithoutWaiting590=== RUN TestGCSweepDeliversEachKeyOnce591=== PAUSE TestGCSweepDeliversEachKeyOnce592=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection593=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection594=== RUN TestForceGCDuringPushOffersSweptObject595=== PAUSE TestForceGCDuringPushOffersSweptObject596=== RUN TestFailedS3DeleteKeepsTombstone597=== PAUSE TestFailedS3DeleteKeepsTombstone598=== RUN TestSweepSkipsRowsLockedByPush599=== PAUSE TestSweepSkipsRowsLockedByPush600=== RUN TestRegisterCompletedObjectNeedsPendingRow601=== PAUSE TestRegisterCompletedObjectNeedsPendingRow602=== RUN TestGetExistingObjectsLocksRows603=== PAUSE TestGetExistingObjectsLocksRows604=== RUN TestSweepRechecksPendingAfterLocking605=== PAUSE TestSweepRechecksPendingAfterLocking606=== RUN TestPushWaitsForSweepHoldingRow607=== PAUSE TestPushWaitsForSweepHoldingRow608=== RUN TestCommitRacingPendingCleanupKeepsObjects609=== PAUSE TestCommitRacingPendingCleanupKeepsObjects610=== RUN TestGCEndsOnShutdown611=== PAUSE TestGCEndsOnShutdown612=== RUN TestGCTaskStore_StartNew613=== PAUSE TestGCTaskStore_StartNew614=== RUN TestGCTaskStore_DeduplicateSameParams615=== PAUSE TestGCTaskStore_DeduplicateSameParams616=== RUN TestGCTaskStore_ConflictDifferentParams617=== PAUSE TestGCTaskStore_ConflictDifferentParams618=== RUN TestGCTaskStore_GetEmpty619=== PAUSE TestGCTaskStore_GetEmpty620=== RUN TestGCTaskStore_GetReturnsLatest621=== PAUSE TestGCTaskStore_GetReturnsLatest622=== RUN TestGCTaskStore_CompletedAllowsNewTask623=== PAUSE TestGCTaskStore_CompletedAllowsNewTask624=== RUN TestGCTaskStore_PhaseUpdates625=== PAUSE TestGCTaskStore_PhaseUpdates626=== RUN TestGCTaskStore_Fail627=== PAUSE TestGCTaskStore_Fail628=== RUN TestGracefulShutdownDrainsInflight629=== PAUSE TestGracefulShutdownDrainsInflight630=== RUN TestService_healthCheckHandler631=== PAUSE TestService_healthCheckHandler632=== RUN TestService_readinessHandler633=== PAUSE TestService_readinessHandler634=== RUN TestGenerateLandingPage635=== PAUSE TestGenerateLandingPage636=== RUN TestEnsureAbortUploadsRule637=== PAUSE TestEnsureAbortUploadsRule638=== RUN TestCacheConfigHandlerMaxNarSize639=== PAUSE TestCacheConfigHandlerMaxNarSize640=== RUN TestCreatePendingClosureRejectsOversizedNAR641=== PAUSE TestCreatePendingClosureRejectsOversizedNAR642=== RUN TestNARDeduplicationMetadataUploadBug643=== PAUSE TestNARDeduplicationMetadataUploadBug644=== RUN TestMetricsInventory645=== PAUSE TestMetricsInventory646=== RUN TestMetricsCommit647=== PAUSE TestMetricsCommit648=== RUN TestService_NativeMTLS649=== PAUSE TestService_NativeMTLS650=== RUN TestServerTLSConfig651=== PAUSE TestServerTLSConfig652=== RUN TestMultipartCleanup653=== PAUSE TestMultipartCleanup654=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded655=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded656=== RUN TestPendingCleanupUsesOneCutoff657=== PAUSE TestPendingCleanupUsesOneCutoff658=== RUN TestObjectStatsTrigger659=== PAUSE TestObjectStatsTrigger660=== RUN TestObjectStatsUpdatedOncePerCommit661=== PAUSE TestObjectStatsUpdatedOncePerCommit662=== RUN TestOrphanedObjectsGC663=== PAUSE TestOrphanedObjectsGC664=== RUN TestOrphanedObjectsGCStressTest665=== PAUSE TestOrphanedObjectsGCStressTest666=== RUN TestCreatePin_ReservedPins667=== PAUSE TestCreatePin_ReservedPins668=== RUN TestPresentReportsOnlyClosureRoots669=== PAUSE TestPresentReportsOnlyClosureRoots670=== RUN TestPresentRefreshesOnlyStaleClosures671=== PAUSE TestPresentRefreshesOnlyStaleClosures672=== RUN TestPresentNotReportedWhileGCDeletesClosure673=== PAUSE TestPresentNotReportedWhileGCDeletesClosure674=== RUN TestParseSingleRange675=== PAUSE TestParseSingleRange676=== RUN TestProxyHeadersOnlyTrustedOnSocket6772026/11/12 00:22:40 INFO Starting HTTP server address=127.0.0.1:405596782026/11/12 00:22:40 INFO Starting HTTP server address=proxy.sock6792026/11/12 00:22:40 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6802026/11/12 00:22:40 INFO Shutdown signal received, draining in-flight requests timeout=10s681--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.79s)682=== RUN TestIsValidCachePath683=== PAUSE TestIsValidCachePath684=== RUN TestReadProxyNarinfo685=== PAUSE TestReadProxyNarinfo686=== RUN TestReadProxyNarinfoAlreadyDecompressed687=== PAUSE TestReadProxyNarinfoAlreadyDecompressed688=== RUN TestReadProxyContentNegotiation689=== PAUSE TestReadProxyContentNegotiation690=== RUN TestReadProxyNarStreaming691=== PAUSE TestReadProxyNarStreaming692=== RUN TestReadProxy404693=== PAUSE TestReadProxy404694=== RUN TestReadProxyInvalidPath695=== PAUSE TestReadProxyInvalidPath696=== RUN TestReadProxyHead697=== PAUSE TestReadProxyHead698=== RUN TestReadProxyOutlastsServerWriteTimeout699=== PAUSE TestReadProxyOutlastsServerWriteTimeout700=== RUN TestReadProxyConditionalGet701=== PAUSE TestReadProxyConditionalGet702=== RUN TestReadProxyRootRedirectsToIndexHTML703=== PAUSE TestReadProxyRootRedirectsToIndexHTML704=== RUN TestReadProxyDisabled705=== PAUSE TestReadProxyDisabled706=== RUN TestReadRedirectNar707=== PAUSE TestReadRedirectNar708=== RUN TestReadRedirectKeepsNarinfoProxied709=== PAUSE TestReadRedirectKeepsNarinfoProxied710=== RUN TestReadProxyRangeRequest711=== PAUSE TestReadProxyRangeRequest712=== RUN TestReadRedirectUsesPublicS3URL713=== PAUSE TestReadRedirectUsesPublicS3URL714=== RUN TestPush_OverlappingRootsStoreOneRowPerKey715=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey716=== RUN TestPush_CompleteCommitsEveryRoot717=== PAUSE TestPush_CompleteCommitsEveryRoot718=== RUN TestPush_RepeatedRootCommitsOnce719=== PAUSE TestPush_RepeatedRootCommitsOnce720=== RUN TestPush_CompleteWaitsForCleanupOfThePush721=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush722=== RUN TestPush_RootsAreLockedInByteOrder723=== PAUSE TestPush_RootsAreLockedInByteOrder724=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit725=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit726=== RUN TestPush_RejectsBadRequests727=== PAUSE TestPush_RejectsBadRequests728=== RUN TestPush_SignsNarinfosOfItsPendingObjects729=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects730=== RUN TestSharedNarBothUploadsComplete731=== PAUSE TestSharedNarBothUploadsComplete732=== RUN TestCompleteMultipartUpload_ErrorButObjectExists733=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists734=== RUN TestCompletedNarNotReofferedAcrossClosures735=== PAUSE TestCompletedNarNotReofferedAcrossClosures736=== RUN TestPresignedUploadRegisteredBeforeCommit737=== PAUSE TestPresignedUploadRegisteredBeforeCommit738=== RUN TestService_Rustfstest739=== PAUSE TestService_Rustfstest740=== RUN TestParseSize741=== PAUSE TestParseSize742=== RUN TestSkippedUploadsHandler743=== PAUSE TestSkippedUploadsHandler744=== RUN TestSystemdListenerNotActivated745--- PASS: TestSystemdListenerNotActivated (0.00s)746=== RUN TestWatchdogBeatsWhenHealthy747--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)748=== RUN TestWatchdogSkipsWhenUnhealthy7492026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7532026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7542026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7552026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7562026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7582026/11/12 00:22:40 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"759--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)760=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle761=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle762=== RUN TestProxyWriteTimeout763=== PAUSE TestProxyWriteTimeout764=== RUN TestPendingClosureWriteTimeout765=== PAUSE TestPendingClosureWriteTimeout766=== RUN TestIsValidUploadKey767=== PAUSE TestIsValidUploadKey768=== RUN TestUploadHandlersRejectInvalidKeys769=== PAUSE TestUploadHandlersRejectInvalidKeys770=== RUN TestUploadHandlersRejectOversizedBody771=== PAUSE TestUploadHandlersRejectOversizedBody772=== RUN TestService_cleanupPendingClosuresHandler773=== PAUSE TestService_cleanupPendingClosuresHandler774=== RUN TestService_createPendingClosureHandler775=== PAUSE TestService_createPendingClosureHandler776=== RUN TestService_verifyS3Integrity777=== PAUSE TestService_verifyS3Integrity778=== RUN TestService_verifyS3ReuploadsStaleNarinfo779=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo780=== RUN TestCompleteMultipartUnregistered781=== PAUSE TestCompleteMultipartUnregistered782=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT783=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT784=== CONT TestIsValidUploadKey785=== CONT TestAbortSucceeded786=== CONT TestService_verifyS3Integrity787=== CONT TestSkippedUploadsHandler788=== CONT TestPresentReportsOnlyClosureRoots789=== CONT TestService_createPendingClosureHandler790=== CONT TestService_readinessHandler791=== CONT TestPendingClosureWriteTimeout792=== CONT TestProxyWriteTimeout793=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle794=== CONT TestReadProxyNarStreaming795=== CONT TestReadProxyContentNegotiation796=== CONT TestReadProxyNarinfoAlreadyDecompressed797=== CONT TestReadProxyNarinfo798=== CONT TestIsValidCachePath799=== CONT TestParseSingleRange800=== CONT TestPresentNotReportedWhileGCDeletesClosure801=== CONT TestPresentRefreshesOnlyStaleClosures802=== CONT TestReadProxy404803=== CONT TestCreatePin_ReservedPins804=== CONT TestOrphanedObjectsGCStressTest805=== CONT TestOrphanedObjectsGC806=== CONT TestObjectStatsUpdatedOncePerCommit807=== CONT TestService_cleanupPendingClosuresHandler808--- PASS: TestAbortSucceeded (0.00s)809=== CONT TestObjectStatsTrigger810=== RUN TestIsValidUploadKey/narinfo811=== RUN TestPendingClosureWriteTimeout/empty812=== RUN TestProxyWriteTimeout/narinfo813=== PAUSE TestProxyWriteTimeout/narinfo8142026/11/12 00:22:40 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000815--- PASS: TestSkippedUploadsHandler (0.01s)816=== CONT TestPendingCleanupUsesOneCutoff817=== RUN TestIsValidCachePath/narinfo818=== PAUSE TestIsValidCachePath/narinfo819=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars820=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars821=== RUN TestIsValidCachePath/nar_zst822=== PAUSE TestIsValidCachePath/nar_zst823=== RUN TestIsValidCachePath/nar_xz824=== PAUSE TestIsValidCachePath/nar_xz825=== RUN TestIsValidCachePath/nar_bz2826=== PAUSE TestIsValidCachePath/nar_bz2827=== RUN TestIsValidCachePath/nar_uncompressed828=== PAUSE TestIsValidCachePath/nar_uncompressed829=== RUN TestIsValidCachePath/ls830=== PAUSE TestIsValidCachePath/ls831=== RUN TestIsValidCachePath/log832=== PAUSE TestIsValidCachePath/log833=== RUN TestIsValidCachePath/realisation834=== PAUSE TestIsValidCachePath/realisation835=== RUN TestIsValidCachePath/nix-cache-info836=== PAUSE TestIsValidCachePath/nix-cache-info837=== RUN TestIsValidCachePath/index.html838=== PAUSE TestIsValidCachePath/index.html839=== RUN TestIsValidCachePath/traversal_parent840=== PAUSE TestIsValidCachePath/traversal_parent841=== RUN TestIsValidCachePath/traversal_in_middle842=== PAUSE TestIsValidCachePath/traversal_in_middle843=== RUN TestIsValidCachePath/invalid_char_e844=== PAUSE TestIsValidCachePath/invalid_char_e845=== RUN TestIsValidCachePath/invalid_char_u846=== PAUSE TestIsValidCachePath/invalid_char_u847=== RUN TestIsValidCachePath/random_path848=== PAUSE TestIsValidCachePath/random_path849=== RUN TestIsValidCachePath/empty850=== PAUSE TestIsValidCachePath/empty851=== RUN TestIsValidCachePath/leading_slash852=== PAUSE TestIsValidCachePath/leading_slash853=== RUN TestIsValidCachePath/wrong_extension854=== PAUSE TestIsValidCachePath/wrong_extension855=== RUN TestIsValidCachePath/short_hash856=== PAUSE TestIsValidCachePath/short_hash857=== RUN TestParseSingleRange/none858=== PAUSE TestParseSingleRange/none859=== RUN TestParseSingleRange/unknown_unit860=== PAUSE TestParseSingleRange/unknown_unit861=== RUN TestParseSingleRange/multi-range_ignored862=== PAUSE TestParseSingleRange/multi-range_ignored863=== RUN TestParseSingleRange/malformed_no_dash864=== PAUSE TestParseSingleRange/malformed_no_dash865=== RUN TestParseSingleRange/malformed_both_empty866=== PAUSE TestParseSingleRange/malformed_both_empty867=== RUN TestParseSingleRange/malformed_end_before_start868=== PAUSE TestParseSingleRange/malformed_end_before_start869=== RUN TestParseSingleRange/closed870=== PAUSE TestParseSingleRange/closed871=== RUN TestParseSingleRange/open-ended872=== PAUSE TestParseSingleRange/open-ended873=== RUN TestParseSingleRange/end_clamped_to_size874=== PAUSE TestParseSingleRange/end_clamped_to_size875=== RUN TestParseSingleRange/suffix876=== PAUSE TestParseSingleRange/suffix877=== RUN TestParseSingleRange/suffix_exceeds_size878=== PAUSE TestParseSingleRange/suffix_exceeds_size879=== RUN TestParseSingleRange/single_byte880=== PAUSE TestParseSingleRange/single_byte881=== RUN TestParseSingleRange/start_past_EOF882=== PAUSE TestParseSingleRange/start_past_EOF883=== RUN TestParseSingleRange/start_far_past_EOF884=== PAUSE TestParseSingleRange/start_far_past_EOF885=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded886=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale887=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale888=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh889=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh890=== CONT TestMultipartCleanup891=== CONT TestServerTLSConfig892=== RUN TestServerTLSConfig/no_client_CA893=== PAUSE TestServerTLSConfig/no_client_CA894=== RUN TestServerTLSConfig/missing_CA_file895=== PAUSE TestServerTLSConfig/missing_CA_file896=== RUN TestServerTLSConfig/not_a_PEM_file897=== PAUSE TestServerTLSConfig/not_a_PEM_file898=== CONT TestService_NativeMTLS899=== RUN TestProxyWriteTimeout/1_GiB_nar900=== PAUSE TestProxyWriteTimeout/1_GiB_nar901=== RUN TestProxyWriteTimeout/10_GiB_nar902=== PAUSE TestProxyWriteTimeout/10_GiB_nar903=== RUN TestProxyWriteTimeout/unknown_size904=== PAUSE TestProxyWriteTimeout/unknown_size905=== PAUSE TestPendingClosureWriteTimeout/empty906=== CONT TestMetricsCommit907=== RUN TestPendingClosureWriteTimeout/400_objects908=== PAUSE TestPendingClosureWriteTimeout/400_objects909=== RUN TestPendingClosureWriteTimeout/670k_objects910=== PAUSE TestPendingClosureWriteTimeout/670k_objects911=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline912=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline913=== PAUSE TestIsValidUploadKey/narinfo914=== CONT TestMetricsInventory915=== RUN TestIsValidUploadKey/nar_zst916=== PAUSE TestIsValidUploadKey/nar_zst917=== RUN TestIsValidUploadKey/nar_xz918=== PAUSE TestIsValidUploadKey/nar_xz919=== RUN TestIsValidUploadKey/nar_plain920=== PAUSE TestIsValidUploadKey/nar_plain921=== RUN TestIsValidUploadKey/listing922=== PAUSE TestIsValidUploadKey/listing923=== RUN TestIsValidUploadKey/build_log924=== PAUSE TestIsValidUploadKey/build_log925=== RUN TestIsValidUploadKey/build_log_home-manager_file926=== PAUSE TestIsValidUploadKey/build_log_home-manager_file927=== RUN TestIsValidUploadKey/build_log_plus_in_name928=== PAUSE TestIsValidUploadKey/build_log_plus_in_name929=== RUN TestIsValidUploadKey/build_log_question_mark930=== PAUSE TestIsValidUploadKey/build_log_question_mark931=== RUN TestIsValidUploadKey/build_log_equals932=== PAUSE TestIsValidUploadKey/build_log_equals933=== RUN TestIsValidUploadKey/realisation934=== PAUSE TestIsValidUploadKey/realisation935=== RUN TestIsValidUploadKey/realisation_plus_in_output936=== PAUSE TestIsValidUploadKey/realisation_plus_in_output937=== RUN TestIsValidUploadKey/nix-cache-info938=== PAUSE TestIsValidUploadKey/nix-cache-info939=== RUN TestIsValidUploadKey/index.html940=== PAUSE TestIsValidUploadKey/index.html941=== RUN TestIsValidUploadKey/narinfo_key,_nar_type942=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type943=== RUN TestIsValidUploadKey/nar_key,_narinfo_type944=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type945=== RUN TestIsValidUploadKey/listing_key,_narinfo_type946=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type947=== RUN TestIsValidUploadKey/traversal948=== PAUSE TestIsValidUploadKey/traversal949=== RUN TestIsValidUploadKey/traversal_nar950=== PAUSE TestIsValidUploadKey/traversal_nar951=== RUN TestIsValidUploadKey/absolute952=== PAUSE TestIsValidUploadKey/absolute953=== RUN TestIsValidUploadKey/empty_key954=== PAUSE TestIsValidUploadKey/empty_key955=== RUN TestIsValidUploadKey/unknown_type956=== PAUSE TestIsValidUploadKey/unknown_type957=== CONT TestNARDeduplicationMetadataUploadBug9582026/11/12 00:22:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37437/oidc9592026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9602026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9612026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9622026/11/12 00:22:45 WARN mTLS auth: subject not in bound subjects subject="CN=reader"9632026/11/12 00:22:45 WARN mTLS auth: subject not in bound subjects subject="CN=reader"964=== CONT TestCreatePendingClosureRejectsOversizedNAR965--- PASS: TestService_NativeMTLS (4.20s)9662026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures967--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)968=== CONT TestCacheConfigHandlerMaxNarSize969--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)970=== CONT TestEnsureAbortUploadsRule971--- PASS: TestPresentRefreshesOnlyStaleClosures (4.24s)972=== CONT TestGenerateLandingPage973--- PASS: TestGenerateLandingPage (0.01s)974=== CONT TestSharedNarBothUploadsComplete975--- PASS: TestReadProxyNarinfoAlreadyDecompressed (4.50s)976=== CONT TestParseSize977--- PASS: TestParseSize (0.00s)978=== CONT TestService_Rustfstest9792026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9802026/11/12 00:22:45 WARN readiness check failed error="closed pool"981--- PASS: TestService_readinessHandler (4.56s)982=== CONT TestPresignedUploadRegisteredBeforeCommit983--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (4.54s)984=== CONT TestCompletedNarNotReofferedAcrossClosures985--- PASS: TestReadProxy404 (4.37s)986=== CONT TestCompleteMultipartUpload_ErrorButObjectExists987--- PASS: TestReadProxyContentNegotiation (4.60s)988=== CONT TestPush_RepeatedRootCommitsOnce9892026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9902026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9912026/11/12 00:22:45 INFO Received cleanup request method=DELETE path=/api/pending_closures992--- PASS: TestPresentReportsOnlyClosureRoots (4.70s)993=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit994--- PASS: TestReadProxyNarStreaming (4.72s)995=== CONT TestPush_RootsAreLockedInByteOrder9962026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9972026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures9982026/11/12 00:22:45 INFO Received cleanup request method=DELETE path=/api/pending_closures9992026/11/12 00:22:45 INFO Aborted multipart uploads count=0 kept=010002026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures10012026/11/12 00:22:45 INFO Received cleanup request method=DELETE path=/api/pending_closures10022026/11/12 00:22:45 INFO Aborted multipart uploads count=1 kept=010032026/11/12 00:22:45 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10042026-11-12 00:22:45.809 UTC [477] ERROR: Closure does not exist: id=110052026-11-12 00:22:45.809 UTC [477] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE10062026-11-12 00:22:45.809 UTC [477] STATEMENT: -- name: CommitPendingClosure :one1007 SELECT commit_pending_closure($1::bigint)::bigint1008 1009--- PASS: TestService_cleanupPendingClosuresHandler (0.71s)1010=== CONT TestPush_SignsNarinfosOfItsPendingObjects10112026/11/12 00:22:45 INFO Received cleanup request method=DELETE path=/api/pending_closures10122026/11/12 00:22:45 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket16/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""10132026/11/12 00:22:45 INFO Aborted multipart uploads count=0 kept=110142026/11/12 00:22:45 INFO Received cleanup request method=DELETE path=/api/pending_closures10152026/11/12 00:22:45 INFO Aborted multipart uploads count=1 kept=01016--- PASS: TestMultipartCleanup (4.70s)1017=== CONT TestPush_CompleteWaitsForCleanupOfThePush10182026/11/12 00:22:45 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1019--- PASS: TestReadProxyNarinfo (4.89s)1020=== CONT TestPush_RejectsBadRequests10212026/11/12 00:22:45 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10222026/11/12 00:22:45 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10232026/11/12 00:22:45 INFO Received push request method=POST path=/api/pushes10242026/11/12 00:22:45 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LjE3MjJiMmVmLWExZTYtNDZhMC1iZTVhLTU4YzhiMzM1ZmU0NHgxNzk0NDQyOTY1MzY5MDA4MTQ4 parts=1010252026/11/12 00:22:45 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10262026/11/12 00:22:45 INFO Completed upload id=1 objects=2 duration=8.057635ms10272026/11/12 00:22:45 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010282026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures10292026/11/12 00:22:45 INFO Starting cleanup of old closures method=DELETE path=/api/closures10302026/11/12 00:22:45 INFO Received complete push request method=POST path=/api/pushes/1/complete10312026/11/12 00:22:45 INFO Received uploads request method=POST path=/api/pending_closures10322026/11/12 00:22:45 INFO Aborted multipart uploads count=0 kept=01033--- PASS: TestObjectStatsUpdatedOncePerCommit (0.85s)1034=== CONT TestCompleteMultipartUnregistered1035--- PASS: TestObjectStatsTrigger (0.93s)1036=== CONT TestService_verifyS3ReuploadsStaleNarinfo1037=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1038=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1039=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1040=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1041=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1042=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1043=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1044=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1045=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1046=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1047=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT1048--- PASS: TestMetricsInventory (0.96s)1049=== CONT TestGCAdvisoryLockBlocksConcurrentRun10502026/11/12 00:22:46 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10512026/11/12 00:22:46 WARN Refused reserved pin name=worker-x86_64-linux10522026/11/12 00:22:46 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10532026/11/12 00:22:46 INFO Received create pin request method=POST path=/api/pins/my-app10542026/11/12 00:22:46 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1055--- PASS: TestCreatePin_ReservedPins (4.57s)1056=== CONT TestSweepRechecksPendingAfterLocking10572026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures10582026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures10592026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures10602026/11/12 00:22:46 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10612026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures1062--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.68s)1063=== CONT TestGetExistingObjectsLocksRows10642026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures10652026/11/12 00:22:46 INFO Received push request method=POST path=/api/pushes10662026/11/12 00:22:46 INFO Received complete push request method=POST path=/api/pushes/1/complete10672026/11/12 00:22:46 INFO Received push request method=POST path=/api/pushes10682026/11/12 00:22:46 INFO Received complete push request method=POST path=/api/pushes/1/complete10692026/11/12 00:22:46 INFO Completed push id=1 objects=4 duration=7.793872ms1070--- PASS: TestPush_RepeatedRootCommitsOnce (0.74s)1071=== CONT TestService_healthCheckHandler1072--- PASS: TestService_Rustfstest (0.86s)1073=== CONT TestRegisterCompletedObjectNeedsPendingRow10742026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures10752026/11/12 00:22:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10762026/11/12 00:22:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10772026/11/12 00:22:46 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LmM1NDdjNDdjLWNjZDktNDJlNy05MWZlLTFkN2UxNmIyMmEwN3gxNzk0NDQyOTY2MzY5MDA4NzEz10782026/11/12 00:22:46 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket35 days=210792026/11/12 00:22:46 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LmJkM2VhYmI5LTI2MDQtNDAxMi05MzRhLTkwN2EwYmMxYTY3YngxNzk0NDQyOTY1OTQ5NjM5NjE2 parts=1010802026/11/12 00:22:46 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1081=== CONT TestGracefulShutdownDrainsInflight10822026/11/12 00:22:46 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LmM1NDdjNDdjLWNjZDktNDJlNy05MWZlLTFkN2UxNmIyMmEwN3gxNzk0NDQyOTY2MzY5MDA4NzEz parts=11083--- PASS: TestEnsureAbortUploadsRule (1.08s)10842026/11/12 00:22:46 INFO Starting HTTP server address=127.0.0.1:3561510852026/11/12 00:22:46 INFO Shutdown signal received, draining in-flight requests timeout=10s1086--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.91s)10872026/11/12 00:22:46 INFO Completed upload id=1 objects=2 duration=3.075869ms1088=== CONT TestSweepSkipsRowsLockedByPush10892026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures10902026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures10912026/11/12 00:22:46 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10922026/11/12 00:22:46 WARN Found objects in DB but missing from S3, will re-upload count=11093--- PASS: TestService_verifyS3Integrity (5.51s)1094=== CONT TestGCTaskStore_Fail1095--- PASS: TestGCTaskStore_Fail (0.00s)1096=== CONT TestFailedS3DeleteKeepsTombstone1097--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1098=== CONT TestGCTaskStore_PhaseUpdates1099--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1100=== CONT TestForceGCDuringPushOffersSweptObject1101=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1102=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1103=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1104=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1105=== CONT TestGCTaskStore_CompletedAllowsNewTask1106--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1107=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection11082026/11/12 00:22:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11092026/11/12 00:22:46 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11102026/11/12 00:22:46 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LmVmZDZmMTIwLWNhNDAtNGM2MC04NTIxLTMyY2ZmM2Y1M2EwN3gxNzk0NDQyOTY2MTUyNDM0MDUx parts=1211112026/11/12 00:22:46 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LjdjMjJiY2IzLTAzZTgtNDRlZi1hNGFiLWQ5OTliY2M2ODUzYXgxNzk0NDQyOTY2MjA4NDIwMDQ4 parts=1211122026/11/12 00:22:46 INFO Received uploads request method=POST path=/api/pending_closures1113--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.20s)1114=== CONT TestGCTaskStore_GetReturnsLatest1115--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1116=== CONT TestGCSweepDeliversEachKeyOnce11172026/11/12 00:22:46 INFO Completed push id=1 objects=4 duration=832.487977ms11182026/11/12 00:22:47 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11192026/11/12 00:22:47 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LjgyYjQxYjllLWMxYWItNGQ2NC1hYjEzLWZiYzgyYzdiMWU1M3gxNzk0NDQyOTY2MTY2MDgwOTE3 parts=121120=== NAME TestNARDeduplicationMetadataUploadBug1121 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug4164701421/001/store/vyyf1lgn18j6fdvmq5gfw9gdfd316jvv-file1.txt1122--- PASS: TestSharedNarBothUploadsComplete (2.42s)1123=== CONT TestGCTaskStore_GetEmpty1124--- PASS: TestGCTaskStore_GetEmpty (0.00s)1125=== CONT TestTombstonedObjectOfferedWithoutWaiting11262026/11/12 00:22:47 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete11272026/11/12 00:22:47 INFO Completed upload id=2 objects=2 duration=2.851226ms11282026/11/12 00:22:47 INFO Completed push id=1 objects=4 duration=1.571954794s11292026/11/12 00:22:47 INFO Received push request method=POST path=/api/pushes1130--- PASS: TestMetricsCommit (2.74s)1131=== CONT TestGCTaskStore_ConflictDifferentParams1132--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1133=== CONT TestGCSweepSkipsPendingObjects11342026/11/12 00:22:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011352026/11/12 00:22:47 INFO Vacuumed table table=pending_closures11362026/11/12 00:22:47 INFO Vacuumed table table=pending_objects11372026/11/12 00:22:47 INFO Vacuumed table table=multipart_uploads11382026/11/12 00:22:47 INFO Vacuumed table table=closures11392026/11/12 00:22:47 INFO Vacuumed table table=objects11402026/11/12 00:22:47 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001141--- PASS: TestService_createPendingClosureHandler (6.96s)1142=== CONT TestGCTaskStore_DeduplicateSameParams1143--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1144=== CONT TestDeduplicatedObjectsRecordedAsPending11452026/11/12 00:22:47 INFO Aborted multipart uploads count=0 kept=011462026/11/12 00:22:47 WARN Force mode enabled - objects will be deleted immediately without grace period11472026/11/12 00:22:47 INFO Received uploads request method=POST path=/api/pending_closures1148=== NAME TestOrphanedObjectsGC1149 orphaned_objects_gc_test.go:287: GC Test Summary:1150 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1151 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1152 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1153 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1154 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1155--- PASS: TestOrphanedObjectsGC (2.87s)1156=== CONT TestGCTaskStore_StartNew1157--- PASS: TestGCTaskStore_StartNew (0.00s)1158=== CONT TestGCEndsOnShutdown1159=== RUN TestGCEndsOnShutdown/before_the_run1160=== PAUSE TestGCEndsOnShutdown/before_the_run1161=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1162=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1163=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1164=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1165=== CONT TestPushDedupSurvivesConcurrentGC11662026/11/12 00:22:47 INFO Received uploads request method=POST path=/api/pending_closures11672026/11/12 00:22:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=011682026/11/12 00:22:48 INFO Vacuumed table table=pending_closures11692026/11/12 00:22:48 INFO Vacuumed table table=pending_objects11702026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads11712026/11/12 00:22:48 INFO Vacuumed table table=closures11722026/11/12 00:22:48 INFO Vacuumed table table=objects1173--- PASS: TestRegisterCompletedObjectNeedsPendingRow (1.70s)1174=== CONT TestCommitRacingPendingCleanupKeepsObjects11752026/11/12 00:22:48 INFO Received complete push request method=POST path=/api/pushes/2/complete11762026/11/12 00:22:48 INFO Completed push id=2 objects=4 duration=5.596992ms1177--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (2.38s)1178=== CONT TestGCMetrics11792026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=01180--- PASS: TestService_healthCheckHandler (1.75s)1181=== CONT TestPushWaitsForSweepHoldingRow11822026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=01183--- PASS: TestGetExistingObjectsLocksRows (1.95s)1184=== CONT TestGCBugBareHashReferences11852026/11/12 00:22:48 WARN Force mode enabled - objects will be deleted immediately without grace period11862026/11/12 00:22:48 WARN Force mode enabled - objects will be deleted immediately without grace period11872026/11/12 00:22:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011882026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=011892026/11/12 00:22:48 INFO Vacuumed table table=pending_closures11902026/11/12 00:22:48 INFO Vacuumed table table=pending_objects11912026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads11922026/11/12 00:22:48 INFO Vacuumed table table=closures11932026/11/12 00:22:48 INFO Vacuumed table table=objects11942026/11/12 00:22:48 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"11952026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=01196--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (2.12s)1197=== CONT TestUploadHandlersRejectOversizedBody11982026/11/12 00:22:48 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."11992026/11/12 00:22:48 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1200--- PASS: TestFailedS3DeleteKeepsTombstone (1.73s)1201=== CONT TestReadRedirectUsesPublicS3URL12022026/11/12 00:22:48 INFO Received push request method=POST path=/api/pushes1203--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (1.68s)1204=== CONT TestReadProxyRangeRequest12052026/11/12 00:22:48 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12062026/11/12 00:22:48 INFO Signed narinfos id=1 count=112072026/11/12 00:22:48 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12082026/11/12 00:22:48 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12092026/11/12 00:22:48 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"12102026/11/12 00:22:48 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12112026/11/12 00:22:48 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"12122026/11/12 00:22:48 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12132026/11/12 00:22:48 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=112142026/11/12 00:22:48 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12152026/11/12 00:22:48 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1216--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (2.41s)1217=== CONT TestPush_CompleteCommitsEveryRoot12182026/11/12 00:22:48 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12192026/11/12 00:22:48 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1220--- PASS: TestCompleteMultipartUnregistered (2.30s)1221=== CONT TestUploadHandlersRejectInvalidKeys1222=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1223=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1224=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1225=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1226=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1227=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1228=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1229=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1230=== CONT TestPush_OverlappingRootsStoreOneRowPerKey12312026/11/12 00:22:48 INFO Received uploads request method=POST path=/api/pending_closures1232--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (2.25s)1233=== CONT TestReadRedirectKeepsNarinfoProxied12342026/11/12 00:22:48 INFO Received push request method=POST path=/api/pushes12352026/11/12 00:22:48 INFO Received push request method=POST path=/api/pushes1236=== RUN TestPush_RejectsBadRequests/no_roots1237=== PAUSE TestPush_RejectsBadRequests/no_roots1238=== RUN TestPush_RejectsBadRequests/no_objects1239=== PAUSE TestPush_RejectsBadRequests/no_objects1240=== RUN TestPush_RejectsBadRequests/bad_root1241=== PAUSE TestPush_RejectsBadRequests/bad_root1242=== RUN TestPush_RejectsBadRequests/root_not_in_objects1243=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1244=== CONT TestClientWithDependencies1245=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1246=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1247=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1248=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1249=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1250=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1251=== CONT TestService_ReadScope_PublicByDefault12522026/11/12 00:22:48 INFO Received uploads request method=POST path=/api/pending_closures12532026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=012542026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=012552026/11/12 00:22:48 INFO Received uploads request method=POST path=/api/pending_closures1256--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.63s)1257=== CONT TestReadRedirectNar12582026/11/12 00:22:48 INFO Received uploads request method=POST path=/api/pending_closures1259--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.57s)1260=== CONT TestClientMultipleUploads12612026/11/12 00:22:48 INFO Received uploads request method=POST path=/api/pending_closures12622026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=012632026/11/12 00:22:48 INFO Received uploads request method=POST path=/api/pending_closures12642026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=012652026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=012662026-11-12 00:22:48.647 UTC [680] ERROR: Push does not exist: id=112672026-11-12 00:22:48.647 UTC [680] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE12682026-11-12 00:22:48.647 UTC [680] STATEMENT: SELECT commit_push($1)1269--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (2.79s)1270=== CONT TestLeadEndsOnShutdown12712026-11-12 00:22:48.896 UTC [676] ERROR: could not obtain lock on row in relation "closures"12722026-11-12 00:22:48.896 UTC [676] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12732026/11/12 00:22:48 WARN Force mode enabled - objects will be deleted immediately without grace period12742026/11/12 00:22:48 WARN Force mode enabled - objects will be deleted immediately without grace period12752026-11-12 00:22:48.896 UTC [676] ERROR: could not obtain lock on row in relation "closures"12762026-11-12 00:22:48.896 UTC [676] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12772026/11/12 00:22:48 WARN Force mode enabled - objects will be deleted immediately without grace period12782026-11-12 00:22:48.897 UTC [676] ERROR: could not obtain lock on row in relation "closures"12792026-11-12 00:22:48.897 UTC [676] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12802026-11-12 00:22:48.898 UTC [676] ERROR: could not obtain lock on row in relation "closures"12812026-11-12 00:22:48.898 UTC [676] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12822026-11-12 00:22:48.899 UTC [676] ERROR: could not obtain lock on row in relation "closures"12832026-11-12 00:22:48.899 UTC [676] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12842026/11/12 00:22:48 WARN Force mode enabled - objects will be deleted immediately without grace period12852026/11/12 00:22:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012862026/11/12 00:22:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012872026/11/12 00:22:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012882026/11/12 00:22:48 INFO Vacuumed table table=pending_closures12892026/11/12 00:22:48 INFO Vacuumed table table=pending_closures12902026/11/12 00:22:48 INFO Vacuumed table table=pending_objects12912026/11/12 00:22:48 INFO Vacuumed table table=pending_closures12922026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads12932026/11/12 00:22:48 INFO Vacuumed table table=closures12942026/11/12 00:22:48 INFO Vacuumed table table=objects12952026/11/12 00:22:48 INFO Vacuumed table table=pending_objects12962026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads12972026/11/12 00:22:48 INFO Vacuumed table table=closures1298--- PASS: TestPush_RootsAreLockedInByteOrder (3.23s)1299=== CONT TestClientIntegration13002026/11/12 00:22:48 INFO Vacuumed table table=pending_objects13012026/11/12 00:22:48 INFO Vacuumed table table=objects13022026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads1303--- PASS: TestGCMetrics (0.88s)1304=== CONT TestLeadGenerationGrowsWithEveryGrant13052026/11/12 00:22:48 INFO Vacuumed table table=closures13062026/11/12 00:22:48 INFO Vacuumed table table=objects1307--- PASS: TestGCSweepSkipsPendingObjects (1.08s)1308=== CONT TestClientCADerivations13092026/11/12 00:22:48 INFO Aborted multipart uploads count=0 kept=013102026/11/12 00:22:48 WARN Force mode enabled - objects will be deleted immediately without grace period13112026/11/12 00:22:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013122026/11/12 00:22:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013132026/11/12 00:22:48 INFO Vacuumed table table=pending_closures13142026/11/12 00:22:48 INFO Vacuumed table table=pending_objects13152026/11/12 00:22:48 INFO Vacuumed table table=pending_closures13162026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads13172026/11/12 00:22:48 INFO Vacuumed table table=closures13182026/11/12 00:22:48 INFO Vacuumed table table=objects1319--- PASS: TestSweepRechecksPendingAfterLocking (2.86s)1320=== CONT TestCacheStatsHandler13212026/11/12 00:22:48 INFO Vacuumed table table=pending_objects13222026/11/12 00:22:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=013232026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads13242026/11/12 00:22:48 INFO Vacuumed table table=pending_closures13252026/11/12 00:22:48 INFO Vacuumed table table=pending_objects13262026/11/12 00:22:48 INFO Vacuumed table table=multipart_uploads13272026/11/12 00:22:48 INFO Vacuumed table table=closures13282026/11/12 00:22:48 INFO Vacuumed table table=closures13292026/11/12 00:22:48 INFO Vacuumed table table=objects1330--- PASS: TestPushDedupSurvivesConcurrentGC (1.02s)1331=== CONT TestLeadIncumbentAnnouncesAtOnce13322026/11/12 00:22:48 INFO Vacuumed table table=objects1333--- PASS: TestSweepSkipsRowsLockedByPush (2.54s)1334=== CONT TestCacheConfigHandler1335=== RUN TestCacheConfigHandler/full_config,_no_issuer1336=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1337=== RUN TestCacheConfigHandler/no_cache_url_configured1338=== PAUSE TestCacheConfigHandler/no_cache_url_configured1339=== RUN TestCacheConfigHandler/no_signing_keys1340=== PAUSE TestCacheConfigHandler/no_signing_keys1341=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1342=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1343=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent1344--- PASS: TestReadRedirectNar (0.59s)1345=== CONT TestClientErrorHandling1346=== RUN TestClientErrorHandling/InvalidStorePath1347=== PAUSE TestClientErrorHandling/InvalidStorePath1348=== RUN TestClientErrorHandling/InvalidAuthToken1349=== PAUSE TestClientErrorHandling/InvalidAuthToken1350=== RUN TestClientErrorHandling/ServerNotAvailable1351=== PAUSE TestClientErrorHandling/ServerNotAvailable1352=== CONT TestService_ReadAuthMiddleware1353--- PASS: TestPushWaitsForSweepHoldingRow (1.02s)1354=== CONT TestLeadElectsOneAndHandsOver13552026/11/12 00:22:49 INFO Received push request method=POST path=/api/pushes13562026/11/12 00:22:49 INFO Received complete push request method=POST path=/api/pushes/1/complete1357--- PASS: TestReadRedirectUsesPublicS3URL (0.94s)1358=== CONT TestReadProxyConditionalGet1359--- PASS: TestService_ReadScope_PublicByDefault (0.77s)1360=== CONT TestConnectWaitsForAPeerMigration1361--- PASS: TestReadProxyRangeRequest (0.99s)1362=== CONT TestService_RequireScope_OIDC1363=== NAME TestOrphanedObjectsGCStressTest1364 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains13652026/11/12 00:22:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:32847/oidc13662026/11/12 00:22:49 WARN Rate limiter enabled after throttle name=s3-test rate=513672026/11/12 00:22:49 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1368=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1369 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101370 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001371--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (8.79s)1372=== CONT TestReadProxyDisabled13732026/11/12 00:22:51 INFO Aborted multipart uploads count=1 kept=013742026/11/12 00:22:51 INFO Received cleanup request method=DELETE path=/api/pending_closures13752026/11/12 00:22:51 INFO Aborted multipart uploads count=1 kept=01376--- PASS: TestPendingCleanupUsesOneCutoff (10.74s)1377=== CONT TestResolveDBConnectionString1378=== RUN TestResolveDBConnectionString/flag_wins1379=== PAUSE TestResolveDBConnectionString/flag_wins1380=== RUN TestResolveDBConnectionString/file_when_flag_empty1381=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1382=== RUN TestResolveDBConnectionString/missing_file_is_an_error1383=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1384=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1385=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1386=== RUN TestResolveDBConnectionString/nothing_configured1387=== PAUSE TestResolveDBConnectionString/nothing_configured1388=== CONT TestService_AuthMiddleware_OIDC13892026/11/12 00:22:51 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41759/oidc13902026/11/12 00:22:51 WARN Force mode enabled - objects will be deleted immediately without grace period13912026/11/12 00:22:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013922026/11/12 00:22:52 WARN Force mode enabled - objects will be deleted immediately without grace period13932026/11/12 00:22:52 INFO Vacuumed table table=pending_closures13942026/11/12 00:22:52 INFO Vacuumed table table=pending_objects13952026/11/12 00:22:52 INFO Vacuumed table table=multipart_uploads13962026/11/12 00:22:52 INFO Received push request method=POST path=/api/pushes13972026/11/12 00:22:52 INFO Vacuumed table table=closures13982026/11/12 00:22:52 INFO Vacuumed table table=objects1399--- PASS: TestCommitRacingPendingCleanupKeepsObjects (4.03s)1400=== CONT TestReadProxyRootRedirectsToIndexHTML14012026/11/12 00:22:52 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14022026/11/12 00:22:52 INFO Uploading vyyf1lgn18j6fdvmq5gfw9gdfd316jvv-file1.txt (160B)14032026/11/12 00:22:52 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14042026/11/12 00:22:52 WARN Failed to register uploaded object key=vyyf1lgn18j6fdvmq5gfw9gdfd316jvv.ls error="server returned 404: 404 page not found\n"14052026/11/12 00:22:52 INFO Signed narinfos id=1 count=114062026/11/12 00:22:52 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"14072026/11/12 00:22:52 INFO Uploading 1 narinfos14082026/11/12 00:22:52 INFO Received complete push request method=POST path=/api/pushes/1/complete14092026/11/12 00:22:52 WARN Failed to register uploaded object key=vyyf1lgn18j6fdvmq5gfw9gdfd316jvv.narinfo error="server returned 404: 404 page not found\n"14102026/11/12 00:22:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014112026/11/12 00:22:52 INFO Vacuumed table table=pending_closures14122026/11/12 00:22:52 INFO Vacuumed table table=pending_objects14132026/11/12 00:22:52 INFO Vacuumed table table=multipart_uploads14142026/11/12 00:22:52 INFO Vacuumed table table=closures14152026/11/12 00:22:52 INFO Completed push id=1 objects=3 duration=12.88632ms14162026/11/12 00:22:52 INFO Upload complete. (4.044s)14172026/11/12 00:22:52 INFO Vacuumed table table=objects1418--- PASS: TestGCSweepDeliversEachKeyOnce (5.37s)1419=== CONT TestClientPushesUseOnePush1420=== NAME TestNARDeduplicationMetadataUploadBug1421 metadata_upload_test.go:54: Retrieved narinfo from S3:1422 StorePath: /build/TestNARDeduplicationMetadataUploadBug4164701421/001/store/vyyf1lgn18j6fdvmq5gfw9gdfd316jvv-file1.txt1423 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1424 Compression: zstd1425 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1426 NarSize: 1601427 References: 1428 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1429=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1430=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1431=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1432=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1433=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1434=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1435=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1436=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1437=== NAME TestNARDeduplicationMetadataUploadBug1438 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1439=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1440=== NAME TestNARDeduplicationMetadataUploadBug1441 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1442 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}14432026/11/12 00:22:52 INFO Received push request method=POST path=/api/pushes1444--- PASS: TestReadRedirectKeepsNarinfoProxied (3.89s)1445=== CONT TestService_AuthMiddleware_MTLSProxyHeader1446--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (3.98s)1447=== CONT TestPinProtectsFromGC1448=== CONT TestDeleteClosuresRacingPins1449--- PASS: TestGCBugBareHashReferences (7.84s)1450=== NAME TestOrphanedObjectsGCStressTest1451 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion14522026/11/12 00:22:56 INFO Completed push id=1 objects=6 duration=6.891120587s1453--- PASS: TestPush_CompleteCommitsEveryRoot (7.78s)1454=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1455=== NAME TestNARDeduplicationMetadataUploadBug1456 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug4164701421/001/store/ahjlln4zhdg05s0f9nsl3c9hnf3nwksi-file2.txt1457=== NAME TestClientMultipleUploads1458 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads193280977/001/store/s6cv8aqizfawz3s0n1gk8g489kiswc3i-test-file-0.txt1459--- PASS: TestReadProxyDisabled (6.46s)1460=== CONT TestClientSharedPathCommittedMidPush1461--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (4.10s)1462=== CONT TestService_AuthMiddleware1463--- PASS: TestReadProxyRootRedirectsToIndexHTML (4.38s)1464=== CONT TestReadProxyHead1465--- PASS: TestReadProxyConditionalGet (7.43s)1466=== CONT TestReadProxyInvalidPath1467--- PASS: TestCacheStatsHandler (7.67s)1468=== CONT TestReadProxyOutlastsServerWriteTimeout1469=== RUN TestService_RequireScope_OIDC/builder_may_write1470=== PAUSE TestService_RequireScope_OIDC/builder_may_write1471=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1472=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1473=== RUN TestService_RequireScope_OIDC/ops_may_admin1474=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1475=== RUN TestService_RequireScope_OIDC/ops_may_not_write1476=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1477=== RUN TestService_RequireScope_OIDC/reader_may_not_write1478=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1479=== RUN TestService_RequireScope_OIDC/static_token_may_admin1480=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1481=== RUN TestService_RequireScope_OIDC/static_token_may_write1482=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1483=== RUN TestService_RequireScope_OIDC/reader_may_read1484=== PAUSE TestService_RequireScope_OIDC/reader_may_read1485=== RUN TestService_RequireScope_OIDC/writer_implies_read1486=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1487=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1488=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1489=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1490=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1491=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1492=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1493=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1494=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1495=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1496=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1497=== CONT TestClientFallsBackToClosures14982026/11/12 00:22:56 INFO lead: acquired remote=192.0.2.1:1234 generation=114992026/11/12 00:22:56 INFO lead: acquired remote=192.0.2.1:1234 generation=11500=== NAME TestClientMultipleUploads1501 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads193280977/001/store/axb42f81ay72b9kx0akxbyszq0z0rdc1-test-file-1.txt1502--- PASS: TestService_ReadAuthMiddleware (7.80s)1503=== CONT TestIsValidCachePath/narinfo1504=== CONT TestParseSingleRange/none1505=== CONT TestIsValidCachePath/nar_xz1506=== CONT TestIsValidCachePath/short_hash1507=== CONT TestIsValidCachePath/wrong_extension1508=== CONT TestIsValidCachePath/leading_slash1509=== CONT TestIsValidCachePath/empty1510=== CONT TestIsValidCachePath/random_path1511=== CONT TestIsValidCachePath/invalid_char_u1512=== CONT TestIsValidCachePath/invalid_char_e1513=== CONT TestIsValidCachePath/traversal_in_middle1514=== CONT TestIsValidCachePath/traversal_parent1515=== CONT TestIsValidCachePath/index.html1516=== CONT TestIsValidCachePath/nix-cache-info1517=== CONT TestIsValidCachePath/realisation1518=== CONT TestIsValidCachePath/log1519=== CONT TestIsValidCachePath/ls1520=== CONT TestIsValidCachePath/nar_uncompressed1521=== CONT TestIsValidCachePath/nar_bz21522=== CONT TestParseSingleRange/suffix1523=== CONT TestIsValidCachePath/nar_zst1524=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1525--- PASS: TestIsValidCachePath (0.00s)1526 --- PASS: TestIsValidCachePath/narinfo (0.00s)1527 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1528 --- PASS: TestIsValidCachePath/short_hash (0.00s)1529 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1530 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1531 --- PASS: TestIsValidCachePath/empty (0.00s)1532 --- PASS: TestIsValidCachePath/random_path (0.00s)1533 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1534 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1535 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1536 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1537 --- PASS: TestIsValidCachePath/index.html (0.00s)1538 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1539 --- PASS: TestIsValidCachePath/realisation (0.00s)1540 --- PASS: TestIsValidCachePath/log (0.00s)1541 --- PASS: TestIsValidCachePath/ls (0.00s)1542 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1543 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1544 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1545 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1546=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1547=== NAME TestClientWithDependencies1548 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies2656679089/001/store/1nbza9svcd021ayrn8ndhi5wa807kznd-test-script1549=== NAME TestClientMultipleUploads1550 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads193280977/001/store/nw6qx2srnmd33746j00l625825480w5g-test-file-2.txt1551=== NAME TestPinProtectsFromGC1552 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC1790228392/001/store/vj8yqflgzk6p49nl4nh8pi333hdj61y4-pinned-file.txt1553 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC1790228392/001/store/y756jfxf07b43nrr34zpwagzs6211j07-unpinned-file.txt15542026/11/12 00:22:56 INFO Received push request method=POST path=/api/pushes1555--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (4.80s)1556=== CONT TestParseSingleRange/malformed_end_before_start1557=== CONT TestParseSingleRange/end_clamped_to_size1558=== CONT TestParseSingleRange/open-ended1559=== CONT TestParseSingleRange/closed1560=== CONT TestParseSingleRange/malformed_no_dash1561=== CONT TestParseSingleRange/malformed_both_empty1562=== CONT TestParseSingleRange/suffix_exceeds_size1563=== CONT TestParseSingleRange/start_far_past_EOF1564=== CONT TestParseSingleRange/start_past_EOF1565=== CONT TestParseSingleRange/single_byte1566=== CONT TestServerTLSConfig/no_client_CA1567=== CONT TestServerTLSConfig/not_a_PEM_file1568=== CONT TestServerTLSConfig/missing_CA_file1569=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1570--- PASS: TestServerTLSConfig (0.00s)1571 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1572 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1573 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)15742026/11/12 00:22:56 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1575=== NAME TestClientWithDependencies1576 client_integration_test.go:663: Found 1 dependencies (including self)15772026/11/12 00:22:56 WARN Failed to register uploaded object key=ahjlln4zhdg05s0f9nsl3c9hnf3nwksi.ls error="server returned 404: 404 page not found\n"15782026/11/12 00:22:56 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign15792026/11/12 00:22:56 INFO Signed narinfos id=2 count=115802026/11/12 00:22:56 INFO Uploading 1 narinfos15812026/11/12 00:22:56 INFO Received complete push request method=POST path=/api/pushes/2/complete15822026/11/12 00:22:56 WARN Failed to register uploaded object key=ahjlln4zhdg05s0f9nsl3c9hnf3nwksi.narinfo error="server returned 404: 404 page not found\n"15832026/11/12 00:22:56 INFO Completed push id=2 objects=3 duration=4.641563ms15842026/11/12 00:22:56 INFO Upload complete. (86ms)15852026/11/12 00:22:56 INFO lead: acquired remote=192.0.2.1:1234 generation=115862026/11/12 00:22:56 INFO lead: released remote=192.0.2.1:1234 generation=11587--- PASS: TestLeadIncumbentAnnouncesAtOnce (7.96s)1588=== CONT TestParseSingleRange/multi-range_ignored1589=== CONT TestParseSingleRange/unknown_unit1590--- PASS: TestParseSingleRange (0.00s)1591 --- PASS: TestParseSingleRange/none (0.00s)1592 --- PASS: TestParseSingleRange/suffix (0.00s)1593 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1594 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1595 --- PASS: TestParseSingleRange/open-ended (0.00s)1596 --- PASS: TestParseSingleRange/closed (0.00s)1597 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1598 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1599 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1600 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1601 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1602 --- PASS: TestParseSingleRange/single_byte (0.00s)1603 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1604 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1605=== CONT TestProxyWriteTimeout/narinfo1606=== CONT TestProxyWriteTimeout/10_GiB_nar1607=== CONT TestProxyWriteTimeout/1_GiB_nar1608=== CONT TestProxyWriteTimeout/unknown_size1609--- PASS: TestProxyWriteTimeout (4.15s)1610 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1611 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1612 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1613 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1614=== CONT TestPendingClosureWriteTimeout/empty1615=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1616=== NAME TestNARDeduplicationMetadataUploadBug1617 metadata_upload_test.go:76: Retrieved narinfo from S3:1618 StorePath: /build/TestNARDeduplicationMetadataUploadBug4164701421/001/store/ahjlln4zhdg05s0f9nsl3c9hnf3nwksi-file2.txt1619 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1620 Compression: zstd1621 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1622 NarSize: 1601623 References: 1624 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1625 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1626 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1627 {"version":1,"root":{"type":"regular","size":44}}1628--- PASS: TestNARDeduplicationMetadataUploadBug (12.01s)1629=== CONT TestPendingClosureWriteTimeout/670k_objects1630=== CONT TestPendingClosureWriteTimeout/400_objects1631=== CONT TestIsValidUploadKey/narinfo1632=== CONT TestIsValidUploadKey/realisation_plus_in_output1633=== CONT TestIsValidUploadKey/unknown_type1634=== CONT TestIsValidUploadKey/build_log_home-manager_file1635=== CONT TestIsValidUploadKey/empty_key1636=== CONT TestIsValidUploadKey/build_log_equals1637=== CONT TestIsValidUploadKey/build_log_question_mark1638=== CONT TestIsValidUploadKey/realisation1639=== CONT TestIsValidUploadKey/build_log_plus_in_name1640=== CONT TestIsValidUploadKey/absolute1641=== CONT TestIsValidUploadKey/traversal_nar1642=== CONT TestIsValidUploadKey/nar_plain1643=== CONT TestIsValidUploadKey/traversal1644=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1645=== CONT TestIsValidUploadKey/build_log1646=== CONT TestIsValidUploadKey/nix-cache-info1647=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1648=== CONT TestIsValidUploadKey/index.html1649=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1650=== CONT TestIsValidUploadKey/nar_xz1651=== CONT TestIsValidUploadKey/listing1652=== CONT TestIsValidUploadKey/nar_zst1653--- PASS: TestIsValidUploadKey (4.15s)1654 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1655 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1656 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1657 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1658 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1659 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1660 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1661 --- PASS: TestIsValidUploadKey/realisation (0.00s)1662 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1663 --- PASS: TestIsValidUploadKey/absolute (0.00s)1664 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1665 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1666 --- PASS: TestIsValidUploadKey/traversal (0.00s)1667 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1668 --- PASS: TestIsValidUploadKey/build_log (0.00s)1669 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1670 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1671 --- PASS: TestIsValidUploadKey/index.html (0.00s)1672 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1673 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1674 --- PASS: TestIsValidUploadKey/listing (0.00s)1675 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1676=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed16772026/11/12 00:22:57 INFO lead: acquired remote=192.0.2.1:1234 generation=116782026/11/12 00:22:57 INFO lead: acquired remote=192.0.2.1:1234 generation=116792026/11/12 00:22:57 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1680--- PASS: TestService_AuthMiddleware (0.93s)1681=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1682--- PASS: TestReadProxyInvalidPath (0.66s)1683=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1684--- PASS: TestReadProxyHead (0.88s)1685=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost16862026/11/12 00:22:57 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"16872026/11/12 00:22:57 WARN mTLS auth: bound subjects configured but subject DN unavailable16882026/11/12 00:22:57 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1689--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (1.33s)1690=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1691=== NAME TestOrphanedObjectsGCStressTest1692 orphaned_objects_gc_test.go:503: Stress test completed successfully:1693 orphaned_objects_gc_test.go:504: - Active objects preserved: 201694 orphaned_objects_gc_test.go:505: - Objects deleted: 2101695 orphaned_objects_gc_test.go:506: - Total GC'd: 2101696--- PASS: TestOrphanedObjectsGCStressTest (12.64s)1697=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows16982026/11/12 00:22:57 INFO lead: released remote=192.0.2.1:1234 generation=116992026/11/12 00:22:57 INFO lead: acquired remote=192.0.2.1:1234 generation=217002026/11/12 00:22:57 INFO lead: released remote=192.0.2.1:1234 generation=117012026/11/12 00:22:57 INFO lead: acquired remote=192.0.2.1:1234 generation=217022026/11/12 00:22:57 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=217032026/11/12 00:22:58 INFO lead: acquired remote=192.0.2.1:1234 generation=317042026/11/12 00:22:58 INFO lead: released remote=192.0.2.1:1234 generation=11705--- PASS: TestLeadEndsOnShutdown (9.60s)1706=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check17072026/11/12 00:22:58 INFO lead: released remote=192.0.2.1:1234 generation=117082026/11/12 00:22:58 INFO lead: acquired remote=192.0.2.1:1234 generation=21709=== NAME TestClientIntegration1710 client_integration_test.go:292: Created store path: /build/TestClientIntegration2848382039/002/store/cdcyzhjz2ccrlnk39agy8f3j2knpijd7-test-file.txt17112026/11/12 00:22:58 INFO lead: released remote=192.0.2.1:1234 generation=217122026/11/12 00:22:58 INFO lead: acquired remote=192.0.2.1:1234 generation=317132026/11/12 00:22:58 INFO lead: released remote=192.0.2.1:1234 generation=31714--- PASS: TestLeadGenerationGrowsWithEveryGrant (10.07s)1715=== CONT TestGCEndsOnShutdown/before_the_run17162026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17172026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17182026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17192026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17202026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17212026/11/12 00:22:59 INFO Aborted multipart uploads count=0 kept=017222026/11/12 00:22:59 INFO Aborted multipart uploads count=0 kept=017232026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17242026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17252026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17262026/11/12 00:22:59 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17272026/11/12 00:22:59 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17282026/11/12 00:22:59 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LmM1OTg0OGFlLWVmNjMtNDAxZC04NmIzLTU3MTIyYTZkZDcwZXgxNzk0NDQyOTc5MjEwOTI2MTMy parts=1017292026/11/12 00:22:59 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17302026/11/12 00:22:59 INFO Completed upload id=1 objects=3 duration=7.925373ms17312026/11/12 00:22:59 INFO Received uploads request method=POST path=/api/pending_closures17322026/11/12 00:22:59 INFO lead: released remote=192.0.2.1:1234 generation=31733=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1734--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (10.90s)1735=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark17362026/11/12 00:22:59 WARN Force mode enabled - objects will be deleted immediately without grace period17372026/11/12 00:22:59 WARN Force mode enabled - objects will be deleted immediately without grace period17382026/11/12 00:22:59 INFO lead: released remote=192.0.2.1:1234 generation=21739--- PASS: TestLeadElectsOneAndHandsOver (10.89s)1740=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info17412026/11/12 00:22:59 INFO Received uploads request method=POST path=/1742=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key17432026/11/12 00:22:59 INFO Received complete multipart upload request method=POST path=/1744=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal17452026/11/12 00:22:59 INFO Received uploads request method=POST path=/1746=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key17472026/11/12 00:22:59 INFO Received request for more parts method=POST path=/1748=== CONT TestPush_RejectsBadRequests/no_roots1749--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1750 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1751 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1752 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1753 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)17542026/11/12 00:22:59 INFO Received push request method=POST path=/api/pushes1755=== CONT TestPush_RejectsBadRequests/bad_root17562026/11/12 00:22:59 INFO Received push request method=POST path=/api/pushes1757=== CONT TestPush_RejectsBadRequests/root_not_in_objects17582026/11/12 00:22:59 INFO Received push request method=POST path=/api/pushes1759=== CONT TestPush_RejectsBadRequests/no_objects17602026/11/12 00:22:59 INFO Received push request method=POST path=/api/pushes1761=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure1762--- PASS: TestPush_RejectsBadRequests (2.50s)1763 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1764 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1765 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1766 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)17672026/11/12 00:22:59 INFO Received uploads request method=POST path=/17682026/11/12 00:22:59 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst17692026/11/12 00:22:59 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0001.narinfo17702026/11/12 00:22:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017712026/11/12 00:22:59 INFO Vacuumed table table=pending_closures1772=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart17732026/11/12 00:22:59 INFO Received complete multipart upload request method=POST path=/17742026/11/12 00:22:59 INFO Vacuumed table table=pending_objects17752026/11/12 00:22:59 INFO Vacuumed table table=multipart_uploads17762026/11/12 00:22:59 INFO Vacuumed table table=closures17772026/11/12 00:22:59 INFO Vacuumed table table=objects17782026/11/12 00:22:59 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LmMwNjg2ZGI5LTY1YzktNDE1MS1iOWQwLTZmNzM4NjIwODdhYngxNzk0NDQyOTc5MDgyNzY4MDQ3 parts=101779=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts1780--- PASS: TestPendingClosureWriteTimeout (4.15s)1781 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)1782 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)1783 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)1784 --- PASS: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (3.05s)17852026/11/12 00:22:59 INFO Received request for more parts method=POST path=/1786=== NAME TestClientCADerivations1787 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations3104144790/001/store/i3yqq0ilwndhhwh2kicpphc7s2ccbkgy-ca-test17882026/11/12 00:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17892026/11/12 00:23:00 INFO Completed upload id=1 objects=3 duration=5.256389ms1790=== CONT TestCacheConfigHandler/full_config,_no_issuer1791=== CONT TestCacheConfigHandler/no_signing_keys1792=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1793=== CONT TestCacheConfigHandler/no_cache_url_configured1794=== CONT TestClientErrorHandling/InvalidStorePath1795--- PASS: TestCacheConfigHandler (0.00s)1796 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1797 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1798 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1799 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)18002026/11/12 00:23:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=1 objects-deleted-after-grace-period=1 objects-failed-to-delete=018012026/11/12 00:23:00 INFO Vacuumed table table=pending_closures18022026/11/12 00:23:00 INFO Vacuumed table table=pending_objects18032026/11/12 00:23:00 INFO Vacuumed table table=multipart_uploads18042026/11/12 00:23:00 INFO Vacuumed table table=closures18052026/11/12 00:23:00 INFO Vacuumed table table=objects1806=== CONT TestClientErrorHandling/ServerNotAvailable18072026/11/12 00:23:00 INFO Received uploads request method=POST path=/api/pending_closures18082026/11/12 00:23:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18092026/11/12 00:23:00 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst18102026/11/12 00:23:00 WARN Found objects in DB but missing from S3, will re-upload count=11811=== CONT TestClientErrorHandling/InvalidAuthToken1812=== CONT TestResolveDBConnectionString/flag_wins1813=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1814=== CONT TestResolveDBConnectionString/missing_file_is_an_error1815=== CONT TestResolveDBConnectionString/file_when_flag_empty1816=== CONT TestResolveDBConnectionString/nothing_configured1817=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1818--- PASS: TestResolveDBConnectionString (0.00s)1819 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1820 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1821 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1822 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1823 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1824=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected18252026/11/12 00:23:00 WARN Authentication failed token_preview=eyJhbGciOi...HGg-Ok7UTA token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1826=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected18272026/11/12 00:23:00 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1828=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1829=== CONT TestService_RequireScope_OIDC/builder_may_write18302026/11/12 00:23:00 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LjBmNDM3MmM0LTJlOGMtNDg4My1hMmI5LTNjMjg3Y2FmYjU4ZHgxNzk0NDQyOTc5MDkxMDY1MjI0 parts=101831=== CONT TestService_RequireScope_OIDC/reader_may_read1832=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1833=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read18342026/11/12 00:23:00 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1835=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1836=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1837--- PASS: TestService_AuthMiddleware_OIDC (0.40s)1838 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.01s)1839 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)1840 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1841 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1842=== CONT TestService_RequireScope_OIDC/writer_implies_read1843=== CONT TestService_RequireScope_OIDC/reader_may_not_write1844=== CONT TestService_RequireScope_OIDC/static_token_may_write1845=== CONT TestService_RequireScope_OIDC/static_token_may_admin1846=== CONT TestService_RequireScope_OIDC/ops_may_admin1847=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1848=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1849=== CONT TestService_RequireScope_OIDC/ops_may_not_write1850--- PASS: TestService_RequireScope_OIDC (7.45s)1851 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1852 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1853 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1854 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1855 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1856 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1857 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1858 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1859 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1860 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1861 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1862 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1863 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1864 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)18652026/11/12 00:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18662026/11/12 00:23:00 INFO Completed upload id=1 objects=3 duration=12.109852ms18672026/11/12 00:23:00 INFO Received uploads request method=POST path=/api/pending_closures18682026/11/12 00:23:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18692026/11/12 00:23:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18702026/11/12 00:23:00 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000200000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LjA4NzIzYmE5LTEyY2ItNDNkNy05Nzg0LTgyMDZhYmM2NGY0MngxNzk0NDQyOTc5Mjk0NzUzNzA4 parts=1018712026/11/12 00:23:00 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LjVhZGZmMTI0LTk2MTAtNDAyNi1hYTVhLTMxY2UyNzFkYzE0ZngxNzk0NDQyOTc5OTYyNTA4NDQ1 parts=1018722026/11/12 00:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18732026/11/12 00:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete18742026/11/12 00:23:00 INFO Completed upload id=1 objects=3 duration=6.790603ms18752026/11/12 00:23:00 INFO Received uploads request method=POST path=/api/pending_closures18762026/11/12 00:23:00 INFO Completed upload id=1 objects=3 duration=8.598759ms18772026/11/12 00:23:00 INFO Received uploads request method=POST path=/api/pending_closures18782026/11/12 00:23:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18792026/11/12 00:23:00 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100400000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LjAzNTM3ZDk4LWZlNGUtNDhiZi1iYmI5LTM0MDU2ZGY1MGM1Y3gxNzk0NDQyOTgwMTQ4NDQ5NjY1 parts=1018802026/11/12 00:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18812026/11/12 00:23:00 INFO Completed upload id=2 objects=3 duration=4.948666ms1882--- PASS: TestPresentNotReportedWhileGCDeletesClosure (0.00s)1883 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/fresh (3.04s)1884 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/stale (3.68s)18852026/11/12 00:23:00 INFO Received uploads request method=POST path=/api/pending_closures18862026/11/12 00:23:00 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst18872026/11/12 00:23:00 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0004.narinfo18882026/11/12 00:23:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18892026/11/12 00:23:00 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100300000000000000000000.nar.zst upload_id=ZjQ4MzViNDAtNzhhMy00ZDY2LTg3NTgtZDIwYzgyZTljNDI2LmRkMTRlMTE0LWYyYmMtNDE5Ny05ZDYzLTFjZDc0MjA4MmY2NHgxNzk0NDQyOTgwNDk1NzY5MzE4 parts=1018902026/11/12 00:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18912026/11/12 00:23:00 INFO Completed upload id=2 objects=3 duration=1.388852ms18922026/11/12 00:23:00 INFO Received uploads request method=POST path=/api/pending_closures1893--- PASS: TestConnectWaitsForAPeerMigration (12.06s)1894--- PASS: TestForceGCDuringPushOffersSweptObject (0.00s)1895 --- PASS: TestForceGCDuringPushOffersSweptObject/before_pending_rows (3.47s)1896 --- PASS: TestForceGCDuringPushOffersSweptObject/after_presence_check (2.96s)18972026/11/12 00:23:01 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst18982026/11/12 00:23:01 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0003.narinfo1899=== NAME TestClientCADerivations1900 client_ca_test.go:139: Found 1 dependencies (including self)19012026/11/12 00:23:01 INFO Received push request method=POST path=/api/pushes19022026/11/12 00:23:01 INFO Received push request method=POST path=/api/pushes19032026/11/12 00:23:01 INFO Received push request method=POST path=/api/pushes19042026/11/12 00:23:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19052026/11/12 00:23:01 INFO Uploading 1nbza9svcd021ayrn8ndhi5wa807kznd-test-script (136B)19062026/11/12 00:23:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19072026/11/12 00:23:01 INFO Uploading vj8yqflgzk6p49nl4nh8pi333hdj61y4-pinned-file.txt (128B)19082026/11/12 00:23:01 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)19092026/11/12 00:23:01 INFO Uploading s6cv8aqizfawz3s0n1gk8g489kiswc3i-test-file-0.txt (160B)19102026/11/12 00:23:01 INFO Uploading axb42f81ay72b9kx0akxbyszq0z0rdc1-test-file-1.txt (160B)19112026/11/12 00:23:01 INFO Uploading nw6qx2srnmd33746j00l625825480w5g-test-file-2.txt (160B)1912--- PASS: TestService_verifyS3ReuploadsStaleNarinfo (0.00s)1913 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed (2.85s)1914 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed (2.87s)1915 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed (3.17s)1916 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed (3.39s)1917 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost (3.92s)19182026/11/12 00:23:01 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"19192026/11/12 00:23:01 WARN Failed to register uploaded object key=1nbza9svcd021ayrn8ndhi5wa807kznd.ls error="server returned 404: 404 page not found\n"19202026/11/12 00:23:01 WARN Failed to register uploaded object key=nw6qx2srnmd33746j00l625825480w5g.ls error="server returned 404: 404 page not found\n"19212026/11/12 00:23:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19222026/11/12 00:23:01 WARN Failed to register uploaded object key=s6cv8aqizfawz3s0n1gk8g489kiswc3i.ls error="server returned 404: 404 page not found\n"19232026/11/12 00:23:01 WARN Failed to register uploaded object key=log/xnp657pbncgxsm4qw3chpk1yk7fz84sx-test-script.drv error="server returned 404: 404 page not found\n"19242026/11/12 00:23:01 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"19252026/11/12 00:23:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19262026/11/12 00:23:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19272026/11/12 00:23:01 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"19282026/11/12 00:23:01 WARN Failed to register uploaded object key=axb42f81ay72b9kx0akxbyszq0z0rdc1.ls error="server returned 404: 404 page not found\n"19292026/11/12 00:23:01 WARN Failed to register uploaded object key=vj8yqflgzk6p49nl4nh8pi333hdj61y4.ls error="server returned 404: 404 page not found\n"19302026/11/12 00:23:01 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"19312026/11/12 00:23:01 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"19322026/11/12 00:23:01 INFO Signed narinfos id=1 count=119332026/11/12 00:23:01 INFO Signed narinfos id=1 count=319342026/11/12 00:23:01 INFO Uploading 1 narinfos19352026/11/12 00:23:01 INFO Uploading 3 narinfos19362026/11/12 00:23:01 INFO Signed narinfos id=1 count=119372026/11/12 00:23:01 INFO Uploading 1 narinfos19382026/11/12 00:23:01 INFO Received complete push request method=POST path=/api/pushes/1/complete19392026/11/12 00:23:01 WARN Failed to register uploaded object key=s6cv8aqizfawz3s0n1gk8g489kiswc3i.narinfo error="server returned 404: 404 page not found\n"19402026/11/12 00:23:01 WARN Failed to register uploaded object key=1nbza9svcd021ayrn8ndhi5wa807kznd.narinfo error="server returned 404: 404 page not found\n"19412026/11/12 00:23:01 INFO Received complete push request method=POST path=/api/pushes/1/complete19422026/11/12 00:23:01 WARN Failed to register uploaded object key=vj8yqflgzk6p49nl4nh8pi333hdj61y4.narinfo error="server returned 404: 404 page not found\n"19432026/11/12 00:23:01 WARN Failed to register uploaded object key=nw6qx2srnmd33746j00l625825480w5g.narinfo error="server returned 404: 404 page not found\n"19442026/11/12 00:23:01 INFO Received complete push request method=POST path=/api/pushes/1/complete19452026/11/12 00:23:01 WARN Failed to register uploaded object key=axb42f81ay72b9kx0akxbyszq0z0rdc1.narinfo error="server returned 404: 404 page not found\n"19462026/11/12 00:23:01 INFO Completed push id=1 objects=9 duration=10.905321ms19472026/11/12 00:23:01 INFO Upload complete. (2.32s)1948=== NAME TestClientMultipleUploads1949 client_integration_test.go:417: Uploaded 3 paths in 4.378662283s19502026/11/12 00:23:01 INFO Completed push id=1 objects=3 duration=11.416505ms19512026/11/12 00:23:01 INFO Upload complete. (2.32s)19522026/11/12 00:23:01 INFO Completed push id=1 objects=4 duration=13.073381ms19532026/11/12 00:23:01 INFO Upload complete. (2.32s)1954=== NAME TestClientWithDependencies1955 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies2656679089/001/store) requires matching store prefix1956--- PASS: TestClientWithDependencies (12.94s)1957--- PASS: TestClientMultipleUploads (12.81s)1958--- PASS: TestUploadHandlersRejectOversizedBody (0.18s)1959 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.05s)1960 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.06s)1961 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (1.41s)19622026/11/12 00:23:01 INFO Aborted multipart uploads count=0 kept=019632026/11/12 00:23:01 INFO Aborted multipart uploads count=0 kept=01964--- PASS: TestReadProxyOutlastsServerWriteTimeout (4.98s)19652026/11/12 00:23:02 WARN Force mode enabled - objects will be deleted immediately without grace period19662026/11/12 00:23:02 WARN Force mode enabled - objects will be deleted immediately without grace period19672026/11/12 00:23:02 INFO Received push request method=POST path=/api/pushes19682026/11/12 00:23:02 INFO Received push request method=POST path=/api/pushes19692026/11/12 00:23:02 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)19702026/11/12 00:23:02 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Post \"http://localhost:37747/bucket107/?delete=\": context canceled"19712026/11/12 00:23:02 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)19722026/11/12 00:23:02 INFO Uploading k9vbbc862px9rradpjf5ysq3crvag2qz-a (216B)19732026/11/12 00:23:02 INFO Uploading nxdf06n4jqh97fshfpaj9bp8qh6sw6xk-shared-dep (136B)19742026/11/12 00:23:02 INFO Uploading zdvp3b2c2pq9xcg4hhndy1v0nd9k8lig-top (224B)19752026/11/12 00:23:02 INFO Uploading 7gcp8il3036rzmycmlcg7lclvfyqjfr2-shared-dep (136B)1976--- PASS: TestGCEndsOnShutdown (0.00s)1977 --- PASS: TestGCEndsOnShutdown/before_the_run (1.11s)1978 --- PASS: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (2.98s)1979 --- PASS: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (3.04s)19802026/11/12 00:23:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19812026/11/12 00:23:02 WARN Failed to register uploaded object key=k9vbbc862px9rradpjf5ysq3crvag2qz.ls error="server returned 404: 404 page not found\n"19822026/11/12 00:23:02 WARN Failed to register uploaded object key=0jvzjbc7g4zsj4yji68jykv88vw6fzx0.ls error="server returned 404: 404 page not found\n"19832026/11/12 00:23:02 WARN Failed to register uploaded object key=nar/1ljcwp6im8h32qby7lkabbnr4ilqyrsfgk6x4s53ci4xd1hn2xjc.nar.zst error="server returned 404: 404 page not found\n"19842026/11/12 00:23:02 WARN Failed to register uploaded object key=zdvp3b2c2pq9xcg4hhndy1v0nd9k8lig.ls error="server returned 404: 404 page not found\n"19852026/11/12 00:23:02 WARN Failed to register uploaded object key=7gcp8il3036rzmycmlcg7lclvfyqjfr2.ls error="server returned 404: 404 page not found\n"19862026/11/12 00:23:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19872026/11/12 00:23:02 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19882026/11/12 00:23:02 WARN Failed to register uploaded object key=nxdf06n4jqh97fshfpaj9bp8qh6sw6xk.ls error="server returned 404: 404 page not found\n"19892026/11/12 00:23:02 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19902026/11/12 00:23:02 WARN Failed to register uploaded object key=nar/00i8ngxs30155w18xxlppk5a6l7mic0ip2vaky5bz6n1jsd23mva.nar.zst error="server returned 404: 404 page not found\n"19912026/11/12 00:23:02 INFO Signed narinfos id=1 count=319922026/11/12 00:23:02 INFO Uploading 3 narinfos19932026/11/12 00:23:02 INFO Signed narinfos id=1 count=219942026/11/12 00:23:02 INFO Uploading 2 narinfos19952026/11/12 00:23:02 WARN Failed to register uploaded object key=zdvp3b2c2pq9xcg4hhndy1v0nd9k8lig.narinfo error="server returned 404: 404 page not found\n"19962026/11/12 00:23:02 INFO Received complete push request method=POST path=/api/pushes/1/complete19972026/11/12 00:23:02 INFO Received complete push request method=POST path=/api/pushes/1/complete19982026/11/12 00:23:02 WARN Failed to register uploaded object key=0jvzjbc7g4zsj4yji68jykv88vw6fzx0.narinfo error="server returned 404: 404 page not found\n"19992026/11/12 00:23:02 WARN Failed to register uploaded object key=k9vbbc862px9rradpjf5ysq3crvag2qz.narinfo error="server returned 404: 404 page not found\n"20002026/11/12 00:23:02 WARN Failed to register uploaded object key=nxdf06n4jqh97fshfpaj9bp8qh6sw6xk.narinfo error="server returned 404: 404 page not found\n"
loading 628 hidden lines…
26292026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads26302026/11/12 00:23:23 INFO Received uploads request method=POST path=/26312026/11/12 00:23:23 INFO Received uploads request method=POST path=/26322026/11/12 00:23:23 INFO Received push request method=POST path=/26332026/11/12 00:23:23 INFO Vacuumed table table=closures26342026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=026352026/11/12 00:23:23 INFO Received complete upload request method=POST path=/26362026/11/12 00:23:23 INFO Vacuumed table table=objects26372026/11/12 00:23:23 INFO Completed upload id=2 objects=10 duration=2.621764ms26382026/11/12 00:23:23 INFO Received uploads request method=POST path=/26392026/11/12 00:23:23 INFO Received complete upload request method=POST path=/26402026/11/12 00:23:23 INFO Completed upload id=5 objects=5 duration=1.660216ms26412026/11/12 00:23:23 INFO Received push request method=POST path=/26422026/11/12 00:23:23 INFO Received push request method=POST path=/26432026/11/12 00:23:23 INFO Received uploads request method=POST path=/26442026/11/12 00:23:23 INFO Received push request method=POST path=/26452026/11/12 00:23:23 INFO Received complete push request method=POST path=/26462026/11/12 00:23:23 INFO Received complete upload request method=POST path=/26472026/11/12 00:23:23 INFO Received uploads request method=POST path=/26482026/11/12 00:23:23 INFO Completed upload id=7 objects=4 duration=2.17736ms26492026/11/12 00:23:23 INFO Completed push id=1 objects=5 duration=6.712801ms26502026/11/12 00:23:23 INFO Received complete upload request method=POST path=/26512026/11/12 00:23:23 INFO Received push request method=POST path=/26522026/11/12 00:23:23 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."26532026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."26542026/11/12 00:23:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."26552026/11/12 00:23:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."26562026/11/12 00:23:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."26572026/11/12 00:23:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."26582026/11/12 00:23:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."26592026/11/12 00:23:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."26602026/11/12 00:23:23 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."26612026/11/12 00:23:23 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."26622026/11/12 00:23:23 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."26632026/11/12 00:23:23 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."26642026/11/12 00:23:23 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."26652026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."26662026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."26672026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."26682026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."26692026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."26702026/11/12 00:23:23 INFO Completed upload id=5 objects=7 duration=2.732086ms26712026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."26722026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."26732026/11/12 00:23:23 INFO Received complete push request method=POST path=/26742026/11/12 00:23:23 INFO Received push request method=POST path=/26752026/11/12 00:23:23 INFO Received push request method=POST path=/26762026/11/12 00:23:23 INFO Received uploads request method=POST path=/26772026/11/12 00:23:23 INFO Received push request method=POST path=/26782026/11/12 00:23:23 INFO Received push request method=POST path=/26792026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=026802026/11/12 00:23:23 INFO Completed push id=3 objects=8 duration=10.254155ms26812026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period26822026/11/12 00:23:23 INFO Received uploads request method=POST path=/26832026/11/12 00:23:23 INFO Received complete upload request method=POST path=/26842026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=026852026/11/12 00:23:23 INFO Received uploads request method=POST path=/26862026/11/12 00:23:23 INFO Completed upload id=6 objects=9 duration=1.570694ms26872026/11/12 00:23:23 INFO Received push request method=POST path=/26882026/11/12 00:23:23 INFO Vacuumed table table=pending_closures26892026/11/12 00:23:23 INFO Received complete push request method=POST path=/26902026/11/12 00:23:23 INFO Vacuumed table table=pending_objects26912026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads26922026/11/12 00:23:23 INFO Received complete push request method=POST path=/26932026/11/12 00:23:23 INFO Vacuumed table table=closures26942026/11/12 00:23:23 INFO Vacuumed table table=objects26952026/11/12 00:23:23 INFO Received uploads request method=POST path=/26962026/11/12 00:23:23 INFO Completed push id=3 objects=5 duration=1.770356ms26972026/11/12 00:23:23 INFO Completed push id=2 objects=7 duration=4.149398ms26982026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=026992026/11/12 00:23:23 INFO Received push request method=POST path=/27002026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period27012026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027022026/11/12 00:23:23 INFO Received complete push request method=POST path=/27032026/11/12 00:23:23 INFO Completed push id=2 objects=4 duration=2.14286ms27042026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period27052026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027062026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027072026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027082026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period27092026/11/12 00:23:23 INFO Received uploads request method=POST path=/27102026/11/12 00:23:23 INFO Vacuumed table table=pending_closures27112026/11/12 00:23:23 INFO Received push request method=POST path=/27122026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027132026/11/12 00:23:23 INFO Vacuumed table table=pending_objects27142026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads27152026/11/12 00:23:23 INFO Vacuumed table table=pending_closures27162026/11/12 00:23:23 INFO Vacuumed table table=closures27172026/11/12 00:23:23 INFO Vacuumed table table=pending_objects27182026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads27192026/11/12 00:23:23 INFO Vacuumed table table=objects27202026/11/12 00:23:23 INFO Vacuumed table table=closures27212026/11/12 00:23:23 INFO Received uploads request method=POST path=/27222026/11/12 00:23:23 INFO Vacuumed table table=objects27232026/11/12 00:23:23 INFO Received complete push request method=POST path=/27242026/11/12 00:23:23 INFO Received uploads request method=POST path=/27252026/11/12 00:23:23 INFO Completed push id=4 objects=7 duration=1.958338ms27262026/11/12 00:23:23 INFO Received push request method=POST path=/27272026/11/12 00:23:23 INFO Received uploads request method=POST path=/27282026/11/12 00:23:23 INFO Received uploads request method=POST path=/27292026/11/12 00:23:23 INFO Received push request method=POST path=/27302026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=027312026/11/12 00:23:23 INFO Vacuumed table table=pending_closures27322026/11/12 00:23:23 INFO Received complete push request method=POST path=/27332026/11/12 00:23:23 INFO Received complete upload request method=POST path=/27342026/11/12 00:23:23 INFO Vacuumed table table=pending_objects27352026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads27362026/11/12 00:23:23 INFO Completed upload id=3 objects=5 duration=1.182471ms27372026/11/12 00:23:23 INFO Vacuumed table table=closures27382026/11/12 00:23:23 INFO Vacuumed table table=objects27392026/11/12 00:23:23 INFO Received uploads request method=POST path=/27402026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027412026/11/12 00:23:23 INFO Completed push id=1 objects=10 duration=8.946623ms27422026/11/12 00:23:23 INFO Received uploads request method=POST path=/27432026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period27442026/11/12 00:23:23 INFO Received push request method=POST path=/27452026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027462026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027472026/11/12 00:23:23 INFO Received uploads request method=POST path=/27482026/11/12 00:23:23 INFO Vacuumed table table=pending_closures27492026/11/12 00:23:23 INFO Received uploads request method=POST path=/27502026/11/12 00:23:23 INFO Vacuumed table table=pending_objects27512026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads27522026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027532026/11/12 00:23:23 INFO Vacuumed table table=closures27542026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027552026/11/12 00:23:23 INFO Received uploads request method=POST path=/27562026/11/12 00:23:23 INFO Vacuumed table table=objects27572026/11/12 00:23:23 INFO Received complete push request method=POST path=/27582026/11/12 00:23:23 INFO Completed push id=4 objects=4 duration=1.695675ms27592026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027602026/11/12 00:23:23 INFO Received uploads request method=POST path=/27612026/11/12 00:23:23 INFO Received push request method=POST path=/27622026/11/12 00:23:23 INFO Received uploads request method=POST path=/27632026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027642026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027652026/11/12 00:23:23 INFO Vacuumed table table=pending_closures27662026/11/12 00:23:23 INFO Vacuumed table table=pending_objects27672026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads27682026/11/12 00:23:23 INFO Vacuumed table table=closures27692026/11/12 00:23:23 INFO Vacuumed table table=objects27702026/11/12 00:23:23 INFO Received uploads request method=POST path=/27712026/11/12 00:23:23 INFO Received uploads request method=POST path=/27722026/11/12 00:23:23 INFO Received push request method=POST path=/27732026/11/12 00:23:23 INFO Received push request method=POST path=/27742026/11/12 00:23:23 INFO Received uploads request method=POST path=/27752026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027762026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period27772026/11/12 00:23:23 INFO Received push request method=POST path=/27782026/11/12 00:23:23 INFO Received complete upload request method=POST path=/27792026/11/12 00:23:23 INFO Completed upload id=3 objects=5 duration=1.209611ms27802026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=027812026/11/12 00:23:23 INFO Vacuumed table table=pending_closures27822026/11/12 00:23:23 INFO Received push request method=POST path=/27832026/11/12 00:23:23 INFO Vacuumed table table=pending_objects27842026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads27852026/11/12 00:23:23 INFO Received push request method=POST path=/27862026/11/12 00:23:23 INFO Vacuumed table table=closures27872026/11/12 00:23:23 INFO Vacuumed table table=objects27882026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027892026/11/12 00:23:23 INFO Received complete push request method=POST path=/27902026/11/12 00:23:23 INFO Received push request method=POST path=/27912026/11/12 00:23:23 INFO Completed push id=2 objects=10 duration=3.973537ms27922026/11/12 00:23:23 INFO Received complete upload request method=POST path=/27932026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027942026/11/12 00:23:23 INFO Completed upload id=10 objects=6 duration=3.735435ms27952026/11/12 00:23:23 INFO Received uploads request method=POST path=/27962026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=027972026/11/12 00:23:23 INFO Received complete push request method=POST path=/27982026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period27992026/11/12 00:23:23 INFO Completed push id=5 objects=3 duration=1.891958ms28002026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028012026/11/12 00:23:23 INFO Vacuumed table table=pending_closures28022026/11/12 00:23:23 INFO Received push request method=POST path=/28032026/11/12 00:23:23 INFO Vacuumed table table=pending_objects28042026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads28052026/11/12 00:23:23 INFO Vacuumed table table=closures28062026/11/12 00:23:23 INFO Vacuumed table table=objects28072026/11/12 00:23:23 INFO Received push request method=POST path=/28082026/11/12 00:23:23 INFO Received complete upload request method=POST path=/28092026/11/12 00:23:23 INFO Completed upload id=4 objects=14 duration=3.28757ms28102026/11/12 00:23:23 INFO Received push request method=POST path=/28112026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028122026/11/12 00:23:23 INFO Received complete push request method=POST path=/28132026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028142026/11/12 00:23:23 INFO Completed push id=4 objects=3 duration=1.742436ms28152026/11/12 00:23:23 INFO Vacuumed table table=pending_closures28162026/11/12 00:23:23 INFO Vacuumed table table=pending_objects28172026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads28182026/11/12 00:23:23 INFO Vacuumed table table=closures28192026/11/12 00:23:23 INFO Received complete push request method=POST path=/28202026/11/12 00:23:23 INFO Vacuumed table table=objects28212026/11/12 00:23:23 INFO Completed push id=8 objects=7 duration=2.048959ms28222026/11/12 00:23:23 INFO Received uploads request method=POST path=/28232026/11/12 00:23:23 INFO Received uploads request method=POST path=/28242026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028252026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028262026/11/12 00:23:23 INFO Received uploads request method=POST path=/28272026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period28282026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028292026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=028302026/11/12 00:23:23 INFO Vacuumed table table=pending_closures28312026/11/12 00:23:23 INFO Vacuumed table table=pending_closures28322026/11/12 00:23:23 INFO Vacuumed table table=pending_objects28332026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads28342026/11/12 00:23:23 INFO Vacuumed table table=closures28352026/11/12 00:23:23 INFO Vacuumed table table=pending_objects28362026/11/12 00:23:23 INFO Vacuumed table table=objects28372026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads28382026/11/12 00:23:23 INFO Received push request method=POST path=/28392026/11/12 00:23:23 INFO Vacuumed table table=closures28402026/11/12 00:23:23 INFO Received push request method=POST path=/28412026/11/12 00:23:23 INFO Vacuumed table table=objects28422026/11/12 00:23:23 INFO Received push request method=POST path=/28432026/11/12 00:23:23 INFO Received push request method=POST path=/28442026/11/12 00:23:23 INFO Received complete upload request method=POST path=/28452026/11/12 00:23:23 INFO Completed upload id=9 objects=16 duration=1.332152ms28462026/11/12 00:23:23 INFO Received uploads request method=POST path=/28472026/11/12 00:23:23 INFO Received push request method=POST path=/28482026/11/12 00:23:23 INFO Received push request method=POST path=/28492026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028502026/11/12 00:23:23 INFO Received push request method=POST path=/28512026/11/12 00:23:23 INFO Received push request method=POST path=/28522026/11/12 00:23:23 INFO Received uploads request method=POST path=/28532026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028542026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028552026/11/12 00:23:23 INFO Vacuumed table table=pending_closures28562026/11/12 00:23:23 INFO Vacuumed table table=pending_objects28572026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads28582026/11/12 00:23:23 INFO Vacuumed table table=closures28592026/11/12 00:23:23 INFO Vacuumed table table=objects28602026/11/12 00:23:23 INFO Received push request method=POST path=/28612026/11/12 00:23:23 INFO Received complete upload request method=POST path=/28622026/11/12 00:23:23 INFO Completed upload id=11 objects=6 duration=2.104599ms28632026/11/12 00:23:23 INFO Received push request method=POST path=/28642026/11/12 00:23:23 INFO Received uploads request method=POST path=/28652026/11/12 00:23:23 INFO Received complete push request method=POST path=/28662026/11/12 00:23:23 INFO Completed push id=10 objects=3 duration=1.609775ms28672026/11/12 00:23:23 INFO Received uploads request method=POST path=/28682026/11/12 00:23:23 INFO Received push request method=POST path=/28692026/11/12 00:23:23 INFO Received uploads request method=POST path=/28702026/11/12 00:23:23 INFO Received complete push request method=POST path=/28712026/11/12 00:23:23 INFO Received uploads request method=POST path=/28722026/11/12 00:23:23 INFO Completed push id=8 objects=13 duration=2.0595ms28732026/11/12 00:23:23 INFO Received uploads request method=POST path=/28742026/11/12 00:23:23 INFO Received uploads request method=POST path=/28752026/11/12 00:23:23 INFO Received push request method=POST path=/28762026/11/12 00:23:23 INFO Received push request method=POST path=/28772026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028782026/11/12 00:23:23 INFO Received complete push request method=POST path=/28792026/11/12 00:23:23 INFO Completed push id=9 objects=5 duration=1.962118ms28802026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028812026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period28822026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028832026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028842026/11/12 00:23:23 INFO Vacuumed table table=pending_closures28852026/11/12 00:23:23 INFO Vacuumed table table=pending_objects28862026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period28872026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads28882026/11/12 00:23:23 INFO Vacuumed table table=closures28892026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028902026/11/12 00:23:23 INFO Vacuumed table table=objects28912026/11/12 00:23:23 INFO Vacuumed table table=pending_closures28922026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=028932026/11/12 00:23:23 INFO Vacuumed table table=pending_objects28942026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads28952026/11/12 00:23:23 INFO Received push request method=POST path=/28962026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period28972026/11/12 00:23:23 INFO Vacuumed table table=closures28982026/11/12 00:23:23 INFO Vacuumed table table=objects28992026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029002026/11/12 00:23:23 INFO Vacuumed table table=pending_closures29012026/11/12 00:23:23 INFO Vacuumed table table=pending_objects29022026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads29032026/11/12 00:23:23 INFO Vacuumed table table=closures29042026/11/12 00:23:23 INFO Vacuumed table table=objects29052026/11/12 00:23:23 INFO Received uploads request method=POST path=/29062026/11/12 00:23:23 INFO Received uploads request method=POST path=/29072026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=029082026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029092026/11/12 00:23:23 INFO Vacuumed table table=pending_closures29102026/11/12 00:23:23 INFO Vacuumed table table=pending_objects29112026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads29122026/11/12 00:23:23 INFO Received push request method=POST path=/29132026/11/12 00:23:23 INFO Vacuumed table table=closures29142026/11/12 00:23:23 INFO Vacuumed table table=objects29152026/11/12 00:23:23 INFO Received uploads request method=POST path=/29162026/11/12 00:23:23 INFO Received push request method=POST path=/29172026/11/12 00:23:23 INFO Received uploads request method=POST path=/29182026/11/12 00:23:23 INFO Received uploads request method=POST path=/29192026/11/12 00:23:24 INFO Received complete push request method=POST path=/29202026/11/12 00:23:24 INFO Received push request method=POST path=/29212026/11/12 00:23:24 INFO Completed push id=9 objects=4 duration=3.358111ms29222026/11/12 00:23:24 INFO Received complete upload request method=POST path=/29232026/11/12 00:23:24 INFO Completed upload id=10 objects=6 duration=2.876647ms29242026/11/12 00:23:24 INFO Received uploads request method=POST path=/29252026/11/12 00:23:24 INFO Received push request method=POST path=/29262026/11/12 00:23:24 INFO Received complete upload request method=POST path=/29272026/11/12 00:23:24 INFO Received complete push request method=POST path=/29282026/11/12 00:23:24 INFO Completed upload id=17 objects=6 duration=2.618304ms29292026/11/12 00:23:24 INFO Completed push id=8 objects=6 duration=1.922638ms29302026/11/12 00:23:24 INFO Received complete push request method=POST path=/29312026/11/12 00:23:24 INFO Completed push id=15 objects=24 duration=3.26507ms29322026/11/12 00:23:24 INFO Received uploads request method=POST path=/29332026/11/12 00:23:24 INFO Received complete push request method=POST path=/29342026/11/12 00:23:24 INFO Completed push id=12 objects=7 duration=1.802037ms29352026/11/12 00:23:24 INFO Received push request method=POST path=/29362026/11/12 00:23:24 INFO Received uploads request method=POST path=/29372026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=029382026/11/12 00:23:24 INFO Received complete push request method=POST path=/29392026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=029402026/11/12 00:23:24 INFO Completed push id=17 objects=22 duration=3.953037ms29412026/11/12 00:23:24 INFO Vacuumed table table=pending_closures29422026/11/12 00:23:24 INFO Vacuumed table table=pending_objects29432026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads29442026/11/12 00:23:24 INFO Vacuumed table table=closures29452026/11/12 00:23:24 INFO Vacuumed table table=objects29462026/11/12 00:23:24 INFO Received uploads request method=POST path=/29472026/11/12 00:23:24 INFO Received push request method=POST path=/29482026/11/12 00:23:24 INFO Received complete upload request method=POST path=/29492026/11/12 00:23:24 INFO Completed upload id=6 objects=6 duration=1.318292ms29502026/11/12 00:23:24 INFO Received uploads request method=POST path=/29512026/11/12 00:23:24 INFO Received uploads request method=POST path=/29522026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=029532026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period29542026/11/12 00:23:24 INFO Received uploads request method=POST path=/29552026/11/12 00:23:24 INFO Received push request method=POST path=/29562026/11/12 00:23:24 INFO Received complete upload request method=POST path=/29572026/11/12 00:23:24 INFO Completed upload id=7 objects=10 duration=1.534694ms29582026/11/12 00:23:24 INFO Received uploads request method=POST path=/29592026/11/12 00:23:24 INFO Received uploads request method=POST path=/29602026/11/12 00:23:24 INFO Received uploads request method=POST path=/29612026/11/12 00:23:24 INFO Received complete upload request method=POST path=/29622026/11/12 00:23:24 INFO Completed upload id=15 objects=9 duration=5.037367ms29632026/11/12 00:23:24 INFO Received push request method=POST path=/29642026/11/12 00:23:24 INFO Received uploads request method=POST path=/29652026/11/12 00:23:24 INFO Received complete push request method=POST path=/29662026/11/12 00:23:24 INFO Completed push id=21 objects=17 duration=2.15094ms29672026/11/12 00:23:24 INFO Received push request method=POST path=/29682026/11/12 00:23:24 INFO Received complete upload request method=POST path=/29692026/11/12 00:23:24 INFO Completed upload id=11 objects=7 duration=1.274432ms29702026/11/12 00:23:24 INFO Received complete push request method=POST path=/29712026/11/12 00:23:24 INFO Completed push id=19 objects=7 duration=1.506854ms29722026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=029732026/11/12 00:23:24 INFO Received uploads request method=POST path=/29742026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period29752026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029762026/11/12 00:23:24 INFO Vacuumed table table=pending_closures29772026/11/12 00:23:24 INFO Received push request method=POST path=/29782026/11/12 00:23:24 INFO Vacuumed table table=pending_objects29792026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads29802026/11/12 00:23:24 INFO Vacuumed table table=closures29812026/11/12 00:23:24 INFO Vacuumed table table=objects29822026/11/12 00:23:24 INFO Received uploads request method=POST path=/29832026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=029842026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period29852026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029862026/11/12 00:23:24 INFO Vacuumed table table=pending_closures29872026/11/12 00:23:24 INFO Vacuumed table table=pending_objects29882026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads29892026/11/12 00:23:24 INFO Vacuumed table table=closures29902026/11/12 00:23:24 INFO Vacuumed table table=objects29912026/11/12 00:23:24 INFO Received complete push request method=POST path=/29922026/11/12 00:23:24 INFO Completed push id=19 objects=8 duration=3.388731ms29932026/11/12 00:23:24 INFO Received uploads request method=POST path=/29942026/11/12 00:23:24 INFO Received uploads request method=POST path=/29952026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=029962026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029972026/11/12 00:23:24 INFO Vacuumed table table=pending_closures29982026/11/12 00:23:24 INFO Vacuumed table table=pending_objects29992026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30002026/11/12 00:23:24 INFO Vacuumed table table=closures30012026/11/12 00:23:24 INFO Vacuumed table table=objects30022026/11/12 00:23:24 INFO Received complete upload request method=POST path=/30032026/11/12 00:23:24 INFO Completed upload id=12 objects=26 duration=4.119158ms30042026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=030052026/11/12 00:23:24 INFO Received complete push request method=POST path=/30062026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period30072026/11/12 00:23:24 INFO Completed push id=13 objects=15 duration=1.778457ms30082026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030092026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30102026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030112026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030122026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030132026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30142026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30152026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030162026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30172026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30182026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30192026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period30202026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period30212026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30222026/11/12 00:23:24 INFO Vacuumed table table=closures30232026/11/12 00:23:24 INFO Vacuumed table table=closures30242026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30252026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30262026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period30272026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30282026/11/12 00:23:24 INFO Vacuumed table table=objects30292026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30302026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30312026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30322026/11/12 00:23:24 INFO Vacuumed table table=objects30332026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30342026/11/12 00:23:24 INFO Vacuumed table table=closures30352026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30362026/11/12 00:23:24 INFO Vacuumed table table=closures30372026/11/12 00:23:24 INFO Vacuumed table table=closures30382026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030392026/11/12 00:23:24 INFO Vacuumed table table=objects30402026/11/12 00:23:24 INFO Vacuumed table table=objects30412026/11/12 00:23:24 INFO Vacuumed table table=objects30422026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030432026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30442026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030452026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30462026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=030472026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30482026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30492026/11/12 00:23:24 INFO Vacuumed table table=closures30502026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30512026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030522026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030532026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30542026/11/12 00:23:24 INFO Vacuumed table table=objects30552026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30562026/11/12 00:23:24 INFO Vacuumed table table=closures30572026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30582026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30592026/11/12 00:23:24 INFO Vacuumed table table=objects30602026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30612026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30622026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30632026/11/12 00:23:24 INFO Vacuumed table table=closures30642026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30652026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30662026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30672026/11/12 00:23:24 INFO Vacuumed table table=closures30682026/11/12 00:23:24 INFO Vacuumed table table=closures30692026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=030702026/11/12 00:23:24 INFO Received uploads request method=POST path=/30712026/11/12 00:23:24 INFO Vacuumed table table=objects30722026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30732026/11/12 00:23:24 INFO Received push request method=POST path=/30742026/11/12 00:23:24 INFO Vacuumed table table=objects30752026/11/12 00:23:24 INFO Received uploads request method=POST path=/30762026/11/12 00:23:24 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."30772026/11/12 00:23:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30782026/11/12 00:23:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."30792026/11/12 00:23:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30802026/11/12 00:23:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30812026/11/12 00:23:24 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."30822026/11/12 00:23:24 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30832026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30842026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30852026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30862026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."30872026/11/12 00:23:24 INFO Received complete push request method=POST path=/30882026/11/12 00:23:24 INFO Received push request method=POST path=/30892026/11/12 00:23:24 INFO Completed push id=5 objects=8 duration=3.351691ms30902026/11/12 00:23:24 INFO Vacuumed table table=objects30912026/11/12 00:23:24 INFO Received push request method=POST path=/30922026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period30932026/11/12 00:23:24 INFO Received push request method=POST path=/30942026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030952026/11/12 00:23:24 INFO Received push request method=POST path=/30962026/11/12 00:23:24 INFO Vacuumed table table=pending_closures30972026/11/12 00:23:24 INFO Vacuumed table table=pending_objects30982026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads30992026/11/12 00:23:24 INFO Vacuumed table table=closures31002026/11/12 00:23:24 INFO Received uploads request method=POST path=/31012026/11/12 00:23:24 INFO Vacuumed table table=objects31022026/11/12 00:23:24 INFO Received push request method=POST path=/31032026/11/12 00:23:24 INFO Received complete push request method=POST path=/31042026/11/12 00:23:24 INFO Received complete push request method=POST path=/31052026/11/12 00:23:24 INFO Received complete upload request method=POST path=/31062026/11/12 00:23:24 INFO Completed push id=8 objects=6 duration=3.002108ms31072026/11/12 00:23:24 INFO Completed push id=16 objects=8 duration=2.213581ms31082026/11/12 00:23:24 INFO Received push request method=POST path=/31092026/11/12 00:23:24 INFO Completed upload id=9 objects=7 duration=2.375142ms31102026/11/12 00:23:24 INFO Received uploads request method=POST path=/31112026/11/12 00:23:24 INFO Received uploads request method=POST path=/31122026/11/12 00:23:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31132026/11/12 00:23:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."31142026/11/12 00:23:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31152026/11/12 00:23:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31162026/11/12 00:23:24 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."31172026/11/12 00:23:24 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31182026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31192026/11/12 00:23:24 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31202026/11/12 00:23:24 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31212026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31222026/11/12 00:23:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31232026/11/12 00:23:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31242026/11/12 00:23:24 INFO Received complete push request method=POST path=/31252026/11/12 00:23:24 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."31262026/11/12 00:23:24 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."31272026/11/12 00:23:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31282026/11/12 00:23:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."31292026/11/12 00:23:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31302026/11/12 00:23:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31312026/11/12 00:23:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."31322026/11/12 00:23:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."31332026/11/12 00:23:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31342026/11/12 00:23:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31352026/11/12 00:23:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."31362026/11/12 00:23:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31372026/11/12 00:23:24 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."31382026/11/12 00:23:24 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."31392026/11/12 00:23:24 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31402026/11/12 00:23:24 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."31412026/11/12 00:23:24 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31422026/11/12 00:23:24 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31432026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31442026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31452026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31462026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31472026/11/12 00:23:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31482026/11/12 00:23:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31492026/11/12 00:23:24 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31502026/11/12 00:23:24 INFO Completed push id=8 objects=2 duration=2.221901ms31512026/11/12 00:23:24 INFO Received complete push request method=POST path=/31522026/11/12 00:23:24 INFO Completed push id=20 objects=3 duration=2.499203ms31532026/11/12 00:23:24 INFO Received uploads request method=POST path=/31542026/11/12 00:23:24 INFO Received uploads request method=POST path=/31552026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=031562026/11/12 00:23:24 INFO Received uploads request method=POST path=/31572026/11/12 00:23:24 INFO Received uploads request method=POST path=/31582026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period31592026/11/12 00:23:24 INFO Received complete upload request method=POST path=/31602026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031612026/11/12 00:23:24 INFO Vacuumed table table=pending_closures31622026/11/12 00:23:24 INFO Completed upload id=10 objects=4 duration=3.373671ms31632026/11/12 00:23:24 INFO Vacuumed table table=pending_objects31642026/11/12 00:23:24 INFO Received push request method=POST path=/31652026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads31662026/11/12 00:23:24 INFO Vacuumed table table=closures31672026/11/12 00:23:24 INFO Vacuumed table table=objects31682026/11/12 00:23:24 INFO Received push request method=POST path=/31692026/11/12 00:23:24 INFO Received uploads request method=POST path=/31702026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=031712026/11/12 00:23:24 INFO Received uploads request method=POST path=/31722026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031732026/11/12 00:23:24 INFO Received push request method=POST path=/31742026/11/12 00:23:24 INFO Vacuumed table table=pending_closures31752026/11/12 00:23:24 INFO Received uploads request method=POST path=/31762026/11/12 00:23:24 INFO Vacuumed table table=pending_objects31772026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads31782026/11/12 00:23:24 INFO Vacuumed table table=closures31792026/11/12 00:23:24 INFO Vacuumed table table=objects31802026/11/12 00:23:24 INFO Received complete push request method=POST path=/31812026/11/12 00:23:24 INFO Received complete upload request method=POST path=/31822026/11/12 00:23:24 INFO Completed upload id=6 objects=7 duration=4.020377ms31832026/11/12 00:23:24 INFO Completed push id=11 objects=11 duration=5.229448ms31842026/11/12 00:23:24 INFO Received push request method=POST path=/31852026/11/12 00:23:24 INFO Received push request method=POST path=/31862026/11/12 00:23:24 INFO Received push request method=POST path=/31872026/11/12 00:23:24 INFO Received push request method=POST path=/31882026/11/12 00:23:24 INFO Received complete push request method=POST path=/31892026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=031902026/11/12 00:23:24 INFO Completed push id=11 objects=6 duration=8.58308ms31912026/11/12 00:23:24 INFO Received push request method=POST path=/31922026/11/12 00:23:24 INFO Received push request method=POST path=/31932026/11/12 00:23:24 INFO Received uploads request method=POST path=/31942026/11/12 00:23:24 INFO Received uploads request method=POST path=/31952026/11/12 00:23:24 INFO Received uploads request method=POST path=/31962026/11/12 00:23:24 INFO Received push request method=POST path=/31972026/11/12 00:23:24 INFO Received uploads request method=POST path=/31982026/11/12 00:23:24 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."31992026/11/12 00:23:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32002026/11/12 00:23:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32012026/11/12 00:23:24 INFO Received push request method=POST path=/32022026/11/12 00:23:24 INFO Received complete push request method=POST path=/32032026/11/12 00:23:24 INFO Completed push id=18 objects=6 duration=1.514274ms32042026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=032052026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32062026/11/12 00:23:24 INFO Received push request method=POST path=/32072026/11/12 00:23:24 INFO Completed upload id=6 objects=24 duration=2.817786ms32082026/11/12 00:23:24 INFO Received uploads request method=POST path=/32092026/11/12 00:23:24 INFO Received uploads request method=POST path=/32102026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032112026/11/12 00:23:24 INFO Received push request method=POST path=/32122026/11/12 00:23:24 INFO Vacuumed table table=pending_closures32132026/11/12 00:23:24 INFO Vacuumed table table=pending_objects32142026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads32152026/11/12 00:23:24 INFO Vacuumed table table=closures32162026/11/12 00:23:24 INFO Received push request method=POST path=/32172026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=032182026/11/12 00:23:24 INFO Received complete push request method=POST path=/32192026/11/12 00:23:24 INFO Received complete push request method=POST path=/32202026/11/12 00:23:24 INFO Vacuumed table table=objects32212026/11/12 00:23:24 INFO Completed push id=4 objects=6 duration=2.090599ms32222026/11/12 00:23:24 INFO Completed push id=9 objects=24 duration=1.975858ms32232026/11/12 00:23:24 INFO Received push request method=POST path=/32242026/11/12 00:23:24 INFO Received push request method=POST path=/32252026/11/12 00:23:24 INFO Received uploads request method=POST path=/32262026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=032272026/11/12 00:23:24 INFO Received push request method=POST path=/32282026/11/12 00:23:24 INFO Received push request method=POST path=/32292026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period32302026/11/12 00:23:24 INFO Received uploads request method=POST path=/32312026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32322026/11/12 00:23:24 INFO Completed upload id=11 objects=14 duration=2.1748ms32332026/11/12 00:23:24 INFO Received complete push request method=POST path=/32342026/11/12 00:23:24 INFO Received complete push request method=POST path=/32352026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32362026/11/12 00:23:24 INFO Completed push id=8 objects=6 duration=2.853407ms32372026/11/12 00:23:24 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."32382026/11/12 00:23:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."32392026/11/12 00:23:24 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32402026/11/12 00:23:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."32412026/11/12 00:23:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."32422026/11/12 00:23:24 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."32432026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."32442026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."32452026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32462026/11/12 00:23:24 INFO Completed push id=3 objects=22 duration=6.689703ms32472026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32482026/11/12 00:23:24 INFO Received complete push request method=POST path=/32492026/11/12 00:23:24 INFO Completed upload id=8 objects=3 duration=2.620164ms32502026/11/12 00:23:24 INFO Completed upload id=22 objects=6 duration=969.988µs32512026/11/12 00:23:24 INFO Completed push id=10 objects=2 duration=1.917137ms32522026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32532026/11/12 00:23:24 INFO Received push request method=POST path=/32542026/11/12 00:23:24 INFO Completed upload id=10 objects=2 duration=2.307081ms32552026/11/12 00:23:24 INFO Received uploads request method=POST path=/32562026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32572026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32582026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=032592026/11/12 00:23:24 INFO Completed upload id=9 objects=2 duration=1.05667ms32602026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period32612026/11/12 00:23:24 INFO Received push request method=POST path=/32622026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32632026/11/12 00:23:24 INFO Completed upload id=2 objects=26 duration=5.724913ms32642026/11/12 00:23:24 INFO Received push request method=POST path=/32652026/11/12 00:23:24 INFO Received uploads request method=POST path=/32662026/11/12 00:23:24 INFO Completed upload id=14 objects=16 duration=2.067559ms32672026/11/12 00:23:24 INFO Received uploads request method=POST path=/32682026/11/12 00:23:24 INFO Received push request method=POST path=/32692026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32702026/11/12 00:23:24 INFO Completed upload id=9 objects=3 duration=1.706156ms32712026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32722026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32732026/11/12 00:23:24 INFO Received complete push request method=POST path=/32742026/11/12 00:23:24 INFO Completed upload id=12 objects=21 duration=1.970238ms32752026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=032762026/11/12 00:23:24 INFO Completed upload id=10 objects=10 duration=1.416853ms32772026/11/12 00:23:24 INFO Completed push id=8 objects=3 duration=2.511004ms32782026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32792026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=032802026/11/12 00:23:24 INFO Vacuumed table table=pending_closures32812026/11/12 00:23:24 INFO Completed upload id=10 objects=23 duration=2.16242ms32822026/11/12 00:23:24 INFO Vacuumed table table=pending_objects32832026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads32842026/11/12 00:23:24 INFO Received complete upload request method=POST path=/32852026/11/12 00:23:24 INFO Vacuumed table table=closures32862026/11/12 00:23:24 INFO Received push request method=POST path=/32872026/11/12 00:23:24 INFO Vacuumed table table=objects32882026/11/12 00:23:24 INFO Completed upload id=3 objects=24 duration=1.683116ms32892026/11/12 00:23:24 INFO Received push request method=POST path=/32902026/11/12 00:23:24 INFO Received push request method=POST path=/32912026/11/12 00:23:24 INFO Received push request method=POST path=/32922026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032932026/11/12 00:23:24 INFO Vacuumed table table=pending_closures32942026/11/12 00:23:24 INFO Vacuumed table table=pending_objects32952026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads32962026/11/12 00:23:24 INFO Received push request method=POST path=/32972026/11/12 00:23:24 INFO Vacuumed table table=closures32982026/11/12 00:23:24 INFO Vacuumed table table=objects32992026/11/12 00:23:24 INFO Received uploads request method=POST path=/33002026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=033012026/11/12 00:23:24 INFO Received uploads request method=POST path=/33022026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=033032026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period33042026/11/12 00:23:24 INFO Received uploads request method=POST path=/33052026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033062026/11/12 00:23:24 INFO Received push request method=POST path=/33072026/11/12 00:23:24 INFO Received complete push request method=POST path=/33082026/11/12 00:23:24 INFO Vacuumed table table=pending_closures33092026/11/12 00:23:24 INFO Received uploads request method=POST path=/33102026/11/12 00:23:24 INFO Vacuumed table table=pending_objects33112026/11/12 00:23:24 INFO Completed push id=6 objects=6 duration=2.899027ms33122026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads33132026/11/12 00:23:24 INFO Received uploads request method=POST path=/33142026/11/12 00:23:24 INFO Received complete push request method=POST path=/33152026/11/12 00:23:24 INFO Vacuumed table table=closures33162026/11/12 00:23:24 INFO Received complete push request method=POST path=/33172026/11/12 00:23:24 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33182026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33192026/11/12 00:23:24 INFO Vacuumed table table=objects33202026/11/12 00:23:24 INFO Completed push id=13 objects=23 duration=2.07176ms33212026/11/12 00:23:24 INFO Completed push id=17 objects=11 duration=2.13684ms33222026/11/12 00:23:24 INFO Received push request method=POST path=/33232026/11/12 00:23:24 INFO Received push request method=POST path=/33242026/11/12 00:23:24 INFO Received complete upload request method=POST path=/33252026/11/12 00:23:24 INFO Received push request method=POST path=/33262026/11/12 00:23:24 INFO Completed upload id=13 objects=10 duration=1.243432ms33272026/11/12 00:23:24 INFO Received push request method=POST path=/33282026/11/12 00:23:24 INFO Received push request method=POST path=/33292026/11/12 00:23:24 INFO Received uploads request method=POST path=/33302026/11/12 00:23:24 INFO Received complete upload request method=POST path=/33312026/11/12 00:23:24 INFO Received push request method=POST path=/33322026/11/12 00:23:24 INFO Received push request method=POST path=/33332026/11/12 00:23:24 INFO Completed upload id=11 objects=6 duration=1.536554ms33342026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=033352026/11/12 00:23:24 INFO Received push request method=POST path=/33362026/11/12 00:23:24 INFO Received complete upload request method=POST path=/33372026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period33382026/11/12 00:23:24 INFO Completed upload id=2 objects=2 duration=1.978338ms33392026/11/12 00:23:24 INFO Received uploads request method=POST path=/33402026/11/12 00:23:24 INFO Received push request method=POST path=/33412026/11/12 00:23:24 INFO Received uploads request method=POST path=/33422026/11/12 00:23:24 INFO Received uploads request method=POST path=/33432026/11/12 00:23:24 INFO Received push request method=POST path=/33442026/11/12 00:23:24 INFO Received push request method=POST path=/33452026/11/12 00:23:24 INFO Received complete push request method=POST path=/33462026/11/12 00:23:24 INFO Received complete push request method=POST path=/33472026/11/12 00:23:24 INFO Completed push id=7 objects=20 duration=2.317561ms33482026/11/12 00:23:24 INFO Received push request method=POST path=/33492026/11/12 00:23:24 INFO Completed push id=11 objects=8 duration=3.055548ms33502026/11/12 00:23:24 INFO Received complete push request method=POST path=/33512026/11/12 00:23:24 INFO Completed push id=6 objects=5 duration=2.359782ms33522026/11/12 00:23:24 INFO Received uploads request method=POST path=/33532026/11/12 00:23:24 INFO Received push request method=POST path=/33542026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=033552026/11/12 00:23:24 INFO Received complete push request method=POST path=/33562026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=033572026/11/12 00:23:24 INFO Completed push id=12 objects=3 duration=2.008119ms33582026/11/12 00:23:24 INFO Received complete push request method=POST path=/33592026/11/12 00:23:24 INFO Received complete push request method=POST path=/33602026/11/12 00:23:24 INFO Completed push id=10 objects=15 duration=1.815137ms33612026/11/12 00:23:24 INFO Completed push id=14 objects=3 duration=2.349762ms33622026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=033632026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033642026/11/12 00:23:24 INFO Vacuumed table table=pending_closures33652026/11/12 00:23:24 INFO Received push request method=POST path=/33662026/11/12 00:23:24 INFO Vacuumed table table=pending_closures33672026/11/12 00:23:24 INFO Received complete push request method=POST path=/33682026/11/12 00:23:24 INFO Vacuumed table table=pending_objects33692026/11/12 00:23:24 INFO Vacuumed table table=pending_objects33702026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads33712026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads33722026/11/12 00:23:24 INFO Received uploads request method=POST path=/33732026/11/12 00:23:24 INFO Vacuumed table table=closures33742026/11/12 00:23:24 INFO Completed push id=12 objects=15 duration=2.311901ms33752026/11/12 00:23:24 INFO Vacuumed table table=closures33762026/11/12 00:23:24 INFO Vacuumed table table=objects33772026/11/12 00:23:24 INFO Vacuumed table table=objects33782026/11/12 00:23:24 INFO Received push request method=POST path=/33792026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=033802026/11/12 00:23:24 INFO Vacuumed table table=pending_closures33812026/11/12 00:23:24 INFO Received uploads request method=POST path=/33822026/11/12 00:23:24 INFO Vacuumed table table=pending_objects33832026/11/12 00:23:24 INFO Received complete upload request method=POST path=/33842026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads33852026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=033862026/11/12 00:23:24 INFO Vacuumed table table=closures33872026/11/12 00:23:24 INFO Completed upload id=26 objects=6 duration=920.989µs33882026/11/12 00:23:24 INFO Vacuumed table table=objects33892026/11/12 00:23:24 INFO Received push request method=POST path=/33902026/11/12 00:23:24 INFO Received uploads request method=POST path=/33912026/11/12 00:23:24 INFO Received push request method=POST path=/33922026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033932026/11/12 00:23:24 INFO Vacuumed table table=pending_closures33942026/11/12 00:23:24 INFO Received push request method=POST path=/33952026/11/12 00:23:24 INFO Vacuumed table table=pending_objects33962026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads33972026/11/12 00:23:24 INFO Received uploads request method=POST path=/33982026/11/12 00:23:24 INFO Vacuumed table table=closures33992026/11/12 00:23:24 INFO Vacuumed table table=objects34002026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=034012026/11/12 00:23:24 INFO Received complete push request method=POST path=/34022026/11/12 00:23:24 INFO Received uploads request method=POST path=/34032026/11/12 00:23:24 INFO Completed push id=5 objects=6 duration=2.267501ms34042026/11/12 00:23:24 INFO Received push request method=POST path=/34052026/11/12 00:23:24 INFO Received uploads request method=POST path=/34062026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=034072026/11/12 00:23:24 INFO Received push request method=POST path=/34082026/11/12 00:23:24 INFO Vacuumed table table=pending_closures34092026/11/12 00:23:24 INFO Received complete push request method=POST path=/34102026/11/12 00:23:24 INFO Received complete push request method=POST path=/34112026/11/12 00:23:24 INFO Vacuumed table table=pending_objects34122026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads34132026/11/12 00:23:24 INFO Completed push id=14 objects=3 duration=1.435074ms34142026/11/12 00:23:24 INFO Completed push id=13 objects=6 duration=2.636384ms34152026/11/12 00:23:24 INFO Vacuumed table table=closures34162026/11/12 00:23:24 INFO Vacuumed table table=objects34172026/11/12 00:23:24 INFO Received complete push request method=POST path=/34182026/11/12 00:23:24 INFO Received uploads request method=POST path=/34192026/11/12 00:23:24 INFO Completed push id=13 objects=12 duration=2.759206ms34202026/11/12 00:23:24 INFO Received complete push request method=POST path=/34212026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=034222026/11/12 00:23:24 INFO Received push request method=POST path=/34232026/11/12 00:23:24 INFO Completed push id=5 objects=3 duration=2.673285ms34242026/11/12 00:23:24 INFO Received push request method=POST path=/34252026/11/12 00:23:24 INFO Received complete upload request method=POST path=/34262026/11/12 00:23:24 INFO Completed upload id=29 objects=5 duration=1.481714ms34272026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period34282026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034292026/11/12 00:23:24 INFO Received push request method=POST path=/34302026/11/12 00:23:24 INFO Vacuumed table table=pending_closures34312026/11/12 00:23:24 INFO Vacuumed table table=pending_objects34322026/11/12 00:23:24 INFO Received push request method=POST path=/34332026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=034342026/11/12 00:23:24 INFO Received uploads request method=POST path=/34352026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads34362026/11/12 00:23:24 INFO Vacuumed table table=closures34372026/11/12 00:23:24 INFO Vacuumed table table=objects34382026/11/12 00:23:24 INFO Received uploads request method=POST path=/34392026/11/12 00:23:24 INFO Received push request method=POST path=/34402026/11/12 00:23:24 INFO Received complete push request method=POST path=/34412026/11/12 00:23:24 INFO Completed push id=14 objects=4 duration=3.000568ms34422026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=034432026/11/12 00:23:24 INFO Vacuumed table table=pending_closures34442026/11/12 00:23:24 INFO Vacuumed table table=pending_objects34452026/11/12 00:23:24 INFO Received push request method=POST path=/34462026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=034472026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads34482026/11/12 00:23:24 INFO Received uploads request method=POST path=/34492026/11/12 00:23:24 INFO Vacuumed table table=closures34502026/11/12 00:23:24 INFO Vacuumed table table=objects34512026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period34522026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=034532026/11/12 00:23:24 INFO Received push request method=POST path=/34542026/11/12 00:23:24 INFO Received uploads request method=POST path=/34552026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period34562026/11/12 00:23:24 INFO Received complete upload request method=POST path=/34572026/11/12 00:23:24 INFO Completed upload id=16 objects=23 duration=1.159811ms34582026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034592026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034602026/11/12 00:23:24 INFO Received push request method=POST path=/34612026/11/12 00:23:24 INFO Vacuumed table table=pending_closures34622026/11/12 00:23:24 INFO Vacuumed table table=pending_objects34632026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads34642026/11/12 00:23:24 INFO Vacuumed table table=pending_closures34652026/11/12 00:23:24 INFO Vacuumed table table=closures34662026/11/12 00:23:24 INFO Received push request method=POST path=/34672026/11/12 00:23:24 INFO Vacuumed table table=pending_objects34682026/11/12 00:23:24 INFO Vacuumed table table=objects34692026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads34702026/11/12 00:23:24 INFO Received push request method=POST path=/34712026/11/12 00:23:24 INFO Vacuumed table table=closures34722026/11/12 00:23:24 INFO Received complete upload request method=POST path=/34732026/11/12 00:23:24 INFO Vacuumed table table=objects34742026/11/12 00:23:24 INFO Completed upload id=15 objects=4 duration=2.289361ms34752026/11/12 00:23:24 INFO Received push request method=POST path=/34762026/11/12 00:23:24 INFO Received complete upload request method=POST path=/34772026/11/12 00:23:24 INFO Received push request method=POST path=/34782026/11/12 00:23:24 INFO Completed upload id=15 objects=6 duration=1.00595ms34792026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=034802026/11/12 00:23:24 INFO Received complete push request method=POST path=/34812026/11/12 00:23:24 INFO Received push request method=POST path=/34822026/11/12 00:23:24 INFO Received complete push request method=POST path=/34832026/11/12 00:23:24 INFO Received push request method=POST path=/34842026/11/12 00:23:24 INFO Completed push id=17 objects=6 duration=1.861557ms34852026/11/12 00:23:24 INFO Completed push id=8 objects=6 duration=1.589735ms34862026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period34872026/11/12 00:23:24 INFO Received uploads request method=POST path=/34882026/11/12 00:23:24 INFO Received push request method=POST path=/34892026/11/12 00:23:24 INFO Received uploads request method=POST path=/34902026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034912026/11/12 00:23:24 INFO Received push request method=POST path=/34922026/11/12 00:23:24 INFO Vacuumed table table=pending_closures34932026/11/12 00:23:24 INFO Vacuumed table table=pending_objects34942026/11/12 00:23:24 INFO Received uploads request method=POST path=/34952026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads34962026/11/12 00:23:24 INFO Vacuumed table table=closures34972026/11/12 00:23:24 INFO Vacuumed table table=objects34982026/11/12 00:23:24 INFO Received uploads request method=POST path=/34992026/11/12 00:23:24 INFO Received uploads request method=POST path=/35002026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035012026/11/12 00:23:24 INFO Received complete push request method=POST path=/35022026/11/12 00:23:24 INFO Received push request method=POST path=/35032026/11/12 00:23:24 INFO Completed push id=16 objects=7 duration=1.938817ms35042026/11/12 00:23:24 INFO Received push request method=POST path=/35052026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035062026/11/12 00:23:24 INFO Vacuumed table table=pending_closures35072026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35082026/11/12 00:23:24 INFO Received complete push request method=POST path=/35092026/11/12 00:23:24 INFO Vacuumed table table=pending_objects35102026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads35112026/11/12 00:23:24 INFO Completed upload id=17 objects=7 duration=1.373133ms35122026/11/12 00:23:24 INFO Received uploads request method=POST path=/35132026/11/12 00:23:24 INFO Received uploads request method=POST path=/35142026/11/12 00:23:24 INFO Completed push id=4 objects=7 duration=3.084109ms35152026/11/12 00:23:24 INFO Vacuumed table table=closures35162026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35172026/11/12 00:23:24 INFO Received complete push request method=POST path=/35182026/11/12 00:23:24 INFO Vacuumed table table=objects35192026/11/12 00:23:24 INFO Completed upload id=16 objects=3 duration=1.11105ms35202026/11/12 00:23:24 INFO Completed push id=19 objects=6 duration=2.741905ms35212026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035222026/11/12 00:23:24 INFO Received uploads request method=POST path=/35232026/11/12 00:23:24 INFO Received uploads request method=POST path=/35242026/11/12 00:23:24 INFO Received push request method=POST path=/35252026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period35262026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035272026/11/12 00:23:24 INFO Received uploads request method=POST path=/35282026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035292026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035302026/11/12 00:23:24 INFO Vacuumed table table=pending_closures35312026/11/12 00:23:24 INFO Received complete push request method=POST path=/35322026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period35332026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035342026/11/12 00:23:24 INFO Vacuumed table table=pending_objects35352026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads35362026/11/12 00:23:24 INFO Completed push id=23 objects=6 duration=1.352152ms35372026/11/12 00:23:24 INFO Vacuumed table table=closures35382026/11/12 00:23:24 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35392026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35402026/11/12 00:23:24 INFO Vacuumed table table=objects35412026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035422026/11/12 00:23:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."35432026/11/12 00:23:24 INFO Vacuumed table table=pending_closures35442026/11/12 00:23:24 INFO Received uploads request method=POST path=/35452026/11/12 00:23:24 INFO Received push request method=POST path=/35462026/11/12 00:23:24 INFO Vacuumed table table=pending_objects35472026/11/12 00:23:24 INFO Received push request method=POST path=/35482026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads35492026/11/12 00:23:24 INFO Vacuumed table table=closures35502026/11/12 00:23:24 INFO Vacuumed table table=objects35512026/11/12 00:23:24 INFO Received uploads request method=POST path=/35522026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35532026/11/12 00:23:24 INFO Received uploads request method=POST path=/35542026/11/12 00:23:24 INFO Received complete push request method=POST path=/35552026/11/12 00:23:24 INFO Completed upload id=31 objects=3 duration=2.14934ms35562026/11/12 00:23:24 INFO Received complete push request method=POST path=/35572026/11/12 00:23:24 INFO Completed push id=24 objects=6 duration=943.829µs35582026/11/12 00:23:24 INFO Completed push id=6 objects=26 duration=3.607713ms35592026/11/12 00:23:24 INFO Received push request method=POST path=/35602026/11/12 00:23:24 INFO Received push request method=POST path=/35612026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35622026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035632026/11/12 00:23:24 INFO Completed upload id=15 objects=6 duration=3.29635ms35642026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35652026/11/12 00:23:24 INFO Received uploads request method=POST path=/35662026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035672026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035682026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period35692026/11/12 00:23:24 INFO Completed upload id=22 objects=2 duration=914.969µs35702026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period35712026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period35722026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035732026/11/12 00:23:24 INFO Vacuumed table table=pending_closures35742026/11/12 00:23:24 INFO Vacuumed table table=pending_objects35752026/11/12 00:23:24 INFO Received complete push request method=POST path=/35762026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads35772026/11/12 00:23:24 INFO Vacuumed table table=closures35782026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35792026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035802026/11/12 00:23:24 INFO Completed push id=25 objects=7 duration=1.288032ms35812026/11/12 00:23:24 INFO Vacuumed table table=objects35822026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35832026/11/12 00:23:24 INFO Completed upload id=14 objects=8 duration=2.294802ms35842026/11/12 00:23:24 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."35852026/11/12 00:23:24 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."35862026/11/12 00:23:24 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."35872026/11/12 00:23:24 INFO Vacuumed table table=pending_closures35882026/11/12 00:23:24 INFO Completed upload id=14 objects=6 duration=1.634595ms35892026/11/12 00:23:24 INFO Vacuumed table table=pending_objects35902026/11/12 00:23:24 INFO Received uploads request method=POST path=/35912026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads35922026/11/12 00:23:24 INFO Vacuumed table table=closures35932026/11/12 00:23:24 INFO Vacuumed table table=objects35942026/11/12 00:23:24 INFO Received complete push request method=POST path=/35952026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=035962026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period35972026/11/12 00:23:24 INFO Completed push id=16 objects=16 duration=3.272991ms35982026/11/12 00:23:24 INFO Received complete upload request method=POST path=/35992026/11/12 00:23:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036002026/11/12 00:23:24 INFO Vacuumed table table=pending_closures36012026/11/12 00:23:24 INFO Completed upload id=10 objects=17 duration=1.362132ms36022026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=036032026/11/12 00:23:24 WARN Force mode enabled - objects will be deleted immediately without grace period36042026/11/12 00:23:24 INFO Vacuumed table table=pending_objects36052026/11/12 00:23:24 INFO Vacuumed table table=multipart_uploads36062026/11/12 00:23:24 INFO Vacuumed table table=closures36072026/11/12 00:23:24 INFO Aborted multipart uploads count=0 kept=036082026/11/12 00:23:25 INFO Vacuumed table table=objects36092026/11/12 00:23:25 INFO Received push request method=POST path=/36102026/11/12 00:23:25 INFO Received push request method=POST path=/36112026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period36122026/11/12 00:23:25 INFO Received uploads request method=POST path=/36132026/11/12 00:23:25 INFO Received push request method=POST path=/36142026/11/12 00:23:25 INFO Received complete upload request method=POST path=/36152026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036162026/11/12 00:23:25 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36172026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36182026/11/12 00:23:25 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."36192026/11/12 00:23:25 INFO Received complete push request method=POST path=/36202026/11/12 00:23:25 INFO Completed upload id=18 objects=4 duration=1.609815ms36212026/11/12 00:23:25 INFO Vacuumed table table=pending_closures36222026/11/12 00:23:25 INFO Completed push id=7 objects=7 duration=1.018629ms36232026/11/12 00:23:25 INFO Vacuumed table table=pending_objects36242026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads36252026/11/12 00:23:25 INFO Vacuumed table table=closures36262026/11/12 00:23:25 INFO Vacuumed table table=objects36272026/11/12 00:23:25 INFO Received complete upload request method=POST path=/36282026/11/12 00:23:25 INFO Received uploads request method=POST path=/36292026/11/12 00:23:25 INFO Completed upload id=18 objects=2 duration=1.956738ms36302026/11/12 00:23:25 INFO Received complete upload request method=POST path=/36312026/11/12 00:23:25 INFO Received push request method=POST path=/36322026/11/12 00:23:25 INFO Received uploads request method=POST path=/36332026/11/12 00:23:25 INFO Completed upload id=26 objects=24 duration=1.12517ms36342026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=036352026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=036362026/11/12 00:23:25 INFO Received uploads request method=POST path=/36372026/11/12 00:23:25 INFO Received uploads request method=POST path=/36382026/11/12 00:23:25 INFO Received complete upload request method=POST path=/36392026/11/12 00:23:25 INFO Completed upload id=7 objects=3 duration=813.087µs36402026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036412026/11/12 00:23:25 INFO Vacuumed table table=pending_closures36422026/11/12 00:23:25 INFO Received push request method=POST path=/36432026/11/12 00:23:25 INFO Vacuumed table table=pending_objects36442026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads36452026/11/12 00:23:25 INFO Vacuumed table table=closures36462026/11/12 00:23:25 INFO Received push request method=POST path=/36472026/11/12 00:23:25 INFO Vacuumed table table=objects36482026/11/12 00:23:25 INFO Received uploads request method=POST path=/36492026/11/12 00:23:25 INFO Received push request method=POST path=/36502026/11/12 00:23:25 INFO Received complete push request method=POST path=/36512026/11/12 00:23:25 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."36522026/11/12 00:23:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."36532026/11/12 00:23:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."36542026/11/12 00:23:25 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."36552026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."36562026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."36572026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36582026/11/12 00:23:25 INFO Completed push id=17 objects=6 duration=2.14904ms36592026/11/12 00:23:25 INFO Received complete push request method=POST path=/36602026/11/12 00:23:25 INFO Completed push id=16 objects=27 duration=2.14266ms36612026/11/12 00:23:25 INFO Received push request method=POST path=/36622026/11/12 00:23:25 INFO Received push request method=POST path=/36632026/11/12 00:23:25 INFO Received uploads request method=POST path=/36642026/11/12 00:23:25 INFO Received complete push request method=POST path=/36652026/11/12 00:23:25 INFO Received complete upload request method=POST path=/36662026/11/12 00:23:25 INFO Completed push id=18 objects=23 duration=1.708116ms36672026/11/12 00:23:25 INFO Completed upload id=18 objects=11 duration=1.407153ms36682026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=036692026/11/12 00:23:25 INFO Received uploads request method=POST path=/36702026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period36712026/11/12 00:23:25 INFO Received uploads request method=POST path=/36722026/11/12 00:23:25 INFO Received complete upload request method=POST path=/36732026/11/12 00:23:25 INFO Received complete upload request method=POST path=/36742026/11/12 00:23:25 INFO Completed upload id=27 objects=26 duration=845.148µs36752026/11/12 00:23:25 INFO Completed upload id=13 objects=7 duration=1.019789ms36762026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=036772026/11/12 00:23:25 INFO Received complete push request method=POST path=/36782026/11/12 00:23:25 INFO Completed push id=14 objects=6 duration=1.321032ms36792026/11/12 00:23:25 INFO Received complete push request method=POST path=/36802026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036812026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period36822026/11/12 00:23:25 INFO Completed push id=15 objects=7 duration=1.803777ms36832026/11/12 00:23:25 INFO Vacuumed table table=pending_closures36842026/11/12 00:23:25 INFO Received uploads request method=POST path=/36852026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=036862026/11/12 00:23:25 INFO Received push request method=POST path=/36872026/11/12 00:23:25 INFO Received uploads request method=POST path=/36882026/11/12 00:23:25 INFO Vacuumed table table=pending_objects36892026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads36902026/11/12 00:23:25 INFO Received uploads request method=POST path=/36912026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036922026/11/12 00:23:25 INFO Vacuumed table table=closures36932026/11/12 00:23:25 INFO Received push request method=POST path=/36942026/11/12 00:23:25 INFO Vacuumed table table=pending_closures36952026/11/12 00:23:25 INFO Vacuumed table table=objects36962026/11/12 00:23:25 INFO Received push request method=POST path=/36972026/11/12 00:23:25 INFO Vacuumed table table=pending_objects36982026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads36992026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37002026/11/12 00:23:25 INFO Vacuumed table table=closures37012026/11/12 00:23:25 INFO Vacuumed table table=objects37022026/11/12 00:23:25 INFO Completed upload id=20 objects=4 duration=998.689µs37032026/11/12 00:23:25 INFO Received complete push request method=POST path=/37042026/11/12 00:23:25 INFO Received uploads request method=POST path=/37052026/11/12 00:23:25 INFO Completed push id=28 objects=6 duration=923.369µs37062026/11/12 00:23:25 INFO Received push request method=POST path=/37072026/11/12 00:23:25 INFO Received push request method=POST path=/37082026/11/12 00:23:25 INFO Received uploads request method=POST path=/37092026/11/12 00:23:25 INFO Received uploads request method=POST path=/37102026/11/12 00:23:25 INFO Received push request method=POST path=/37112026/11/12 00:23:25 INFO Received push request method=POST path=/37122026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037132026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37142026/11/12 00:23:25 INFO Completed upload id=23 objects=7 duration=962.609µs37152026/11/12 00:23:25 INFO Received uploads request method=POST path=/37162026/11/12 00:23:25 INFO Received complete push request method=POST path=/37172026/11/12 00:23:25 INFO Received complete push request method=POST path=/37182026/11/12 00:23:25 INFO Completed push id=29 objects=6 duration=961.748µs37192026/11/12 00:23:25 INFO Completed push id=11 objects=9 duration=1.557775ms37202026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037212026/11/12 00:23:25 INFO Received push request method=POST path=/37222026/11/12 00:23:25 INFO Vacuumed table table=pending_closures37232026/11/12 00:23:25 INFO Vacuumed table table=pending_objects37242026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads37252026/11/12 00:23:25 INFO Vacuumed table table=closures37262026/11/12 00:23:25 INFO Received complete push request method=POST path=/37272026/11/12 00:23:25 INFO Vacuumed table table=objects37282026/11/12 00:23:25 INFO Completed push id=18 objects=10 duration=1.471054ms37292026/11/12 00:23:25 INFO Received uploads request method=POST path=/37302026/11/12 00:23:25 INFO Received uploads request method=POST path=/37312026/11/12 00:23:25 INFO Received uploads request method=POST path=/37322026/11/12 00:23:25 INFO Received push request method=POST path=/37332026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037342026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037352026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period37362026/11/12 00:23:25 INFO Received uploads request method=POST path=/37372026/11/12 00:23:25 INFO Received complete push request method=POST path=/37382026/11/12 00:23:25 INFO Received push request method=POST path=/37392026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=037402026/11/12 00:23:25 INFO Completed push id=18 objects=6 duration=2.526204ms37412026/11/12 00:23:25 INFO Vacuumed table table=pending_closures37422026/11/12 00:23:25 INFO Vacuumed table table=pending_objects37432026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads37442026/11/12 00:23:25 INFO Vacuumed table table=closures37452026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37462026/11/12 00:23:25 INFO Vacuumed table table=objects37472026/11/12 00:23:25 INFO Completed upload id=30 objects=8 duration=980.649µs37482026/11/12 00:23:25 INFO Received push request method=POST path=/37492026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37502026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37512026/11/12 00:23:25 INFO Received push request method=POST path=/37522026/11/12 00:23:25 INFO Completed upload id=24 objects=18 duration=1.09497ms37532026/11/12 00:23:25 INFO Completed upload id=16 objects=17 duration=2.677965ms37542026/11/12 00:23:25 INFO Received push request method=POST path=/37552026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37562026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037572026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37582026/11/12 00:23:25 INFO Completed upload id=23 objects=9 duration=1.110471ms37592026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037602026/11/12 00:23:25 INFO Vacuumed table table=pending_closures37612026/11/12 00:23:25 INFO Vacuumed table table=pending_objects37622026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads37632026/11/12 00:23:25 INFO Vacuumed table table=closures37642026/11/12 00:23:25 INFO Vacuumed table table=objects37652026/11/12 00:23:25 INFO Received uploads request method=POST path=/37662026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37672026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37682026/11/12 00:23:25 INFO Received complete push request method=POST path=/37692026/11/12 00:23:25 INFO Completed upload id=22 objects=2 duration=1.284792ms37702026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037712026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037722026/11/12 00:23:25 INFO Completed upload id=31 objects=6 duration=1.067309ms37732026/11/12 00:23:25 INFO Completed push id=25 objects=2 duration=1.847777ms37742026/11/12 00:23:25 INFO Received uploads request method=POST path=/37752026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period37762026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037772026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period37782026/11/12 00:23:25 INFO Received push request method=POST path=/37792026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period37802026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037812026/11/12 00:23:25 INFO Vacuumed table table=pending_closures37822026/11/12 00:23:25 INFO Vacuumed table table=pending_objects37832026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads37842026/11/12 00:23:25 INFO Vacuumed table table=closures37852026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037862026/11/12 00:23:25 INFO Vacuumed table table=objects37872026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037882026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period37892026/11/12 00:23:25 INFO Received complete push request method=POST path=/37902026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=037912026/11/12 00:23:25 INFO Received complete upload request method=POST path=/37922026/11/12 00:23:25 INFO Completed push id=27 objects=4 duration=1.595394ms37932026/11/12 00:23:25 INFO Completed upload id=32 objects=4 duration=1.132431ms37942026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period37952026/11/12 00:23:25 INFO Received push request method=POST path=/37962026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037972026/11/12 00:23:25 INFO Vacuumed table table=pending_closures37982026/11/12 00:23:25 INFO Received complete push request method=POST path=/37992026/11/12 00:23:25 INFO Received complete push request method=POST path=/38002026/11/12 00:23:25 INFO Vacuumed table table=pending_objects38012026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038022026/11/12 00:23:25 INFO Completed push id=15 objects=4 duration=1.13919ms38032026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads38042026/11/12 00:23:25 INFO Received uploads request method=POST path=/38052026/11/12 00:23:25 INFO Vacuumed table table=closures38062026/11/12 00:23:25 INFO Vacuumed table table=pending_closures38072026/11/12 00:23:25 INFO Vacuumed table table=objects38082026/11/12 00:23:25 INFO Vacuumed table table=pending_objects38092026/11/12 00:23:25 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."38102026/11/12 00:23:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."38112026/11/12 00:23:25 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."38122026/11/12 00:23:25 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."38132026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."38142026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."38152026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38162026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads38172026/11/12 00:23:25 INFO Completed push id=9 objects=7 duration=3.761715ms38182026/11/12 00:23:25 INFO Vacuumed table table=closures38192026/11/12 00:23:25 INFO Vacuumed table table=objects38202026/11/12 00:23:25 INFO Received push request method=POST path=/38212026/11/12 00:23:25 INFO Received uploads request method=POST path=/38222026/11/12 00:23:25 INFO Received complete upload request method=POST path=/38232026/11/12 00:23:25 INFO Received uploads request method=POST path=/38242026/11/12 00:23:25 INFO Completed upload id=33 objects=10 duration=1.066729ms38252026/11/12 00:23:25 INFO Received push request method=POST path=/38262026/11/12 00:23:25 INFO Received uploads request method=POST path=/38272026/11/12 00:23:25 INFO Received uploads request method=POST path=/38282026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=038292026/11/12 00:23:25 INFO Received complete push request method=POST path=/38302026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period38312026/11/12 00:23:25 INFO Completed push id=28 objects=26 duration=1.378873ms38322026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038332026/11/12 00:23:25 INFO Received uploads request method=POST path=/38342026/11/12 00:23:25 INFO Vacuumed table table=pending_closures38352026/11/12 00:23:25 INFO Vacuumed table table=pending_objects38362026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads38372026/11/12 00:23:25 INFO Vacuumed table table=closures38382026/11/12 00:23:25 INFO Received complete upload request method=POST path=/38392026/11/12 00:23:25 INFO Vacuumed table table=objects38402026/11/12 00:23:25 INFO Received complete upload request method=POST path=/38412026/11/12 00:23:25 INFO Completed upload id=10 objects=7 duration=1.848057ms38422026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=038432026/11/12 00:23:25 INFO Completed upload id=21 objects=3 duration=988.649µs38442026/11/12 00:23:25 INFO Received complete push request method=POST path=/38452026/11/12 00:23:25 INFO Vacuumed table table=pending_closures38462026/11/12 00:23:25 INFO Vacuumed table table=pending_objects38472026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads38482026/11/12 00:23:25 INFO Vacuumed table table=closures38492026/11/12 00:23:25 INFO Completed push id=12 objects=2 duration=3.136329ms38502026/11/12 00:23:25 INFO Vacuumed table table=objects38512026/11/12 00:23:25 INFO Received complete push request method=POST path=/38522026/11/12 00:23:25 INFO Completed push id=30 objects=3 duration=964.509µs38532026/11/12 00:23:25 INFO Received complete upload request method=POST path=/38542026/11/12 00:23:25 INFO Completed upload id=34 objects=23 duration=921.749µs38552026/11/12 00:23:25 INFO Received push request method=POST path=/38562026/11/12 00:23:25 INFO Received uploads request method=POST path=/38572026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=038582026/11/12 00:23:25 INFO Received push request method=POST path=/38592026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period38602026/11/12 00:23:25 INFO Received push request method=POST path=/38612026/11/12 00:23:25 INFO Received push request method=POST path=/38622026/11/12 00:23:25 INFO Received uploads request method=POST path=/38632026/11/12 00:23:25 INFO Received push request method=POST path=/38642026/11/12 00:23:25 INFO Received complete push request method=POST path=/38652026/11/12 00:23:25 INFO Completed push id=32 objects=19 duration=1.00525ms38662026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=038672026/11/12 00:23:25 INFO Received complete push request method=POST path=/38682026/11/12 00:23:25 INFO Received push request method=POST path=/38692026/11/12 00:23:25 INFO Vacuumed table table=pending_closures38702026/11/12 00:23:25 INFO Completed push id=14 objects=3 duration=1.03493ms38712026/11/12 00:23:25 INFO Vacuumed table table=pending_objects38722026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads38732026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=038742026/11/12 00:23:25 INFO Vacuumed table table=closures38752026/11/12 00:23:25 INFO Received complete push request method=POST path=/38762026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period38772026/11/12 00:23:25 INFO Vacuumed table table=objects38782026/11/12 00:23:25 INFO Completed push id=35 objects=6 duration=1.14137ms38792026/11/12 00:23:25 INFO Received push request method=POST path=/38802026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038812026/11/12 00:23:25 INFO Vacuumed table table=pending_closures38822026/11/12 00:23:25 INFO Vacuumed table table=pending_objects38832026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads38842026/11/12 00:23:25 INFO Received complete upload request method=POST path=/38852026/11/12 00:23:25 INFO Vacuumed table table=closures38862026/11/12 00:23:25 INFO Vacuumed table table=objects38872026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=038882026/11/12 00:23:25 INFO Completed upload id=23 objects=15 duration=2.274042ms38892026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period38902026/11/12 00:23:25 INFO Received push request method=POST path=/38912026/11/12 00:23:25 INFO Received uploads request method=POST path=/38922026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038932026/11/12 00:23:25 INFO Vacuumed table table=pending_closures38942026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=038952026/11/12 00:23:25 INFO Vacuumed table table=pending_objects38962026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period38972026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads38982026/11/12 00:23:25 INFO Vacuumed table table=closures38992026/11/12 00:23:25 INFO Received complete push request method=POST path=/39002026/11/12 00:23:25 INFO Vacuumed table table=objects39012026/11/12 00:23:25 INFO Completed push id=16 objects=26 duration=1.170611ms39022026/11/12 00:23:25 INFO Received complete upload request method=POST path=/39032026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=039042026/11/12 00:23:25 INFO Received uploads request method=POST path=/39052026/11/12 00:23:25 INFO Completed upload id=15 objects=3 duration=1.841238ms39062026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039072026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039082026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39092026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39102026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39112026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=039122026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39132026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39142026/11/12 00:23:25 INFO Vacuumed table table=closures39152026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39162026/11/12 00:23:25 INFO Vacuumed table table=objects39172026/11/12 00:23:25 INFO Vacuumed table table=closures39182026/11/12 00:23:25 INFO Received complete upload request method=POST path=/39192026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=039202026/11/12 00:23:25 INFO Vacuumed table table=objects39212026/11/12 00:23:25 INFO Completed upload id=18 objects=4 duration=1.217151ms39222026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period39232026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039242026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39252026/11/12 00:23:25 INFO Received complete upload request method=POST path=/39262026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39272026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039282026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39292026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39302026/11/12 00:23:25 INFO Completed upload id=20 objects=4 duration=847.468µs39312026/11/12 00:23:25 INFO Vacuumed table table=closures39322026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39332026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39342026/11/12 00:23:25 INFO Vacuumed table table=objects39352026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=039362026/11/12 00:23:25 INFO Vacuumed table table=closures39372026/11/12 00:23:25 INFO Vacuumed table table=objects39382026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=039392026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period39402026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039412026/11/12 00:23:25 INFO Received complete push request method=POST path=/39422026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039432026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39442026/11/12 00:23:25 INFO Received complete upload request method=POST path=/39452026/11/12 00:23:25 INFO Received push request method=POST path=/39462026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39472026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39482026/11/12 00:23:25 INFO Completed upload id=19 objects=6 duration=1.213711ms39492026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39502026/11/12 00:23:25 INFO Received push request method=POST path=/39512026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39522026/11/12 00:23:25 INFO Vacuumed table table=closures39532026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39542026/11/12 00:23:25 INFO Vacuumed table table=objects39552026/11/12 00:23:25 INFO Vacuumed table table=closures39562026/11/12 00:23:25 INFO Completed push id=3 objects=21 duration=4.542142ms39572026/11/12 00:23:25 INFO Received push request method=POST path=/39582026/11/12 00:23:25 INFO Vacuumed table table=objects39592026/11/12 00:23:25 INFO Received push request method=POST path=/39602026/11/12 00:23:25 INFO Received uploads request method=POST path=/39612026/11/12 00:23:25 INFO Received push request method=POST path=/39622026/11/12 00:23:25 INFO Received complete push request method=POST path=/39632026/11/12 00:23:25 INFO Received uploads request method=POST path=/39642026/11/12 00:23:25 INFO Received push request method=POST path=/39652026/11/12 00:23:25 INFO Completed push id=16 objects=5 duration=1.015829ms39662026/11/12 00:23:25 INFO Received complete upload request method=POST path=/39672026/11/12 00:23:25 INFO Received complete push request method=POST path=/39682026/11/12 00:23:25 INFO Received complete push request method=POST path=/39692026/11/12 00:23:25 INFO Completed upload id=22 objects=7 duration=1.067769ms39702026/11/12 00:23:25 INFO Completed push id=27 objects=6 duration=2.510823ms39712026/11/12 00:23:25 INFO Completed push id=22 objects=3 duration=3.069008ms39722026/11/12 00:23:25 INFO Received uploads request method=POST path=/39732026/11/12 00:23:25 INFO Received uploads request method=POST path=/39742026/11/12 00:23:25 INFO Received push request method=POST path=/39752026/11/12 00:23:25 INFO Received push request method=POST path=/39762026/11/12 00:23:25 INFO Received uploads request method=POST path=/39772026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=039782026/11/12 00:23:25 INFO Received uploads request method=POST path=/39792026/11/12 00:23:25 INFO Received complete push request method=POST path=/39802026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=039812026/11/12 00:23:25 INFO Received complete push request method=POST path=/39822026/11/12 00:23:25 INFO Completed push id=26 objects=14 duration=2.413283ms39832026/11/12 00:23:25 INFO Received uploads request method=POST path=/39842026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039852026/11/12 00:23:25 INFO Completed push id=17 objects=22 duration=2.050459ms39862026/11/12 00:23:25 INFO Received complete upload request method=POST path=/39872026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39882026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039892026/11/12 00:23:25 INFO Completed upload id=15 objects=2 duration=1.150551ms39902026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39912026/11/12 00:23:25 INFO Vacuumed table table=pending_closures39922026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39932026/11/12 00:23:25 INFO Vacuumed table table=closures39942026/11/12 00:23:25 INFO Vacuumed table table=pending_objects39952026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads39962026/11/12 00:23:25 INFO Vacuumed table table=objects39972026/11/12 00:23:25 INFO Vacuumed table table=closures39982026/11/12 00:23:25 INFO Received push request method=POST path=/39992026/11/12 00:23:25 INFO Received uploads request method=POST path=/40002026/11/12 00:23:25 INFO Vacuumed table table=objects40012026/11/12 00:23:25 INFO Received complete push request method=POST path=/40022026/11/12 00:23:25 INFO Completed push id=24 objects=9 duration=1.622355ms40032026/11/12 00:23:25 INFO Received uploads request method=POST path=/40042026/11/12 00:23:25 INFO Received push request method=POST path=/40052026/11/12 00:23:25 INFO Received uploads request method=POST path=/40062026/11/12 00:23:25 INFO Received complete push request method=POST path=/40072026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=040082026/11/12 00:23:25 INFO Received uploads request method=POST path=/40092026/11/12 00:23:25 INFO Completed push id=22 objects=3 duration=2.219381ms40102026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040112026/11/12 00:23:25 INFO Received push request method=POST path=/40122026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40132026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40142026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40152026/11/12 00:23:25 INFO Received complete push request method=POST path=/40162026/11/12 00:23:25 INFO Vacuumed table table=closures40172026/11/12 00:23:25 INFO Completed push id=24 objects=3 duration=945.728µs40182026/11/12 00:23:25 INFO Vacuumed table table=objects40192026/11/12 00:23:25 INFO Received push request method=POST path=/40202026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=040212026/11/12 00:23:25 INFO Received complete push request method=POST path=/40222026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period40232026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=040242026/11/12 00:23:25 INFO Completed push id=1 objects=11 duration=1.932318ms40252026/11/12 00:23:25 INFO Received uploads request method=POST path=/40262026/11/12 00:23:25 INFO Received push request method=POST path=/40272026/11/12 00:23:25 INFO Received push request method=POST path=/40282026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040292026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=040302026/11/12 00:23:25 INFO Received complete push request method=POST path=/40312026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40322026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period40332026/11/12 00:23:25 INFO Completed push id=15 objects=5 duration=1.774056ms40342026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40352026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40362026/11/12 00:23:25 INFO Vacuumed table table=closures40372026/11/12 00:23:25 INFO Vacuumed table table=objects40382026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040392026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40402026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40412026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40422026/11/12 00:23:25 INFO Vacuumed table table=closures40432026/11/12 00:23:25 INFO Vacuumed table table=objects40442026/11/12 00:23:25 INFO Received uploads request method=POST path=/40452026/11/12 00:23:25 INFO Received push request method=POST path=/40462026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40472026/11/12 00:23:25 INFO Received push request method=POST path=/40482026/11/12 00:23:25 INFO Received uploads request method=POST path=/40492026/11/12 00:23:25 INFO Received complete upload request method=POST path=/40502026/11/12 00:23:25 INFO Completed upload id=16 objects=3 duration=920.968µs40512026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=040522026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040532026/11/12 00:23:25 INFO Received push request method=POST path=/40542026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40552026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period40562026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40572026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40582026/11/12 00:23:25 INFO Vacuumed table table=closures40592026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040602026/11/12 00:23:25 INFO Vacuumed table table=objects40612026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40622026/11/12 00:23:25 INFO Received uploads request method=POST path=/40632026/11/12 00:23:25 INFO Received uploads request method=POST path=/40642026/11/12 00:23:25 INFO Received complete push request method=POST path=/40652026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40662026/11/12 00:23:25 INFO Received uploads request method=POST path=/40672026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40682026/11/12 00:23:25 INFO Completed push id=19 objects=14 duration=1.507033ms40692026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040702026/11/12 00:23:25 INFO Received push request method=POST path=/40712026/11/12 00:23:25 INFO Vacuumed table table=closures40722026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040732026/11/12 00:23:25 INFO Received uploads request method=POST path=/40742026/11/12 00:23:25 INFO Received push request method=POST path=/40752026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40762026/11/12 00:23:25 INFO Vacuumed table table=objects40772026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40782026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40792026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40802026/11/12 00:23:25 INFO Vacuumed table table=closures40812026/11/12 00:23:25 INFO Vacuumed table table=objects40822026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40832026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40842026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=040852026/11/12 00:23:25 INFO Vacuumed table table=closures40862026/11/12 00:23:25 INFO Vacuumed table table=objects40872026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040882026/11/12 00:23:25 INFO Received complete upload request method=POST path=/40892026/11/12 00:23:25 INFO Vacuumed table table=pending_closures40902026/11/12 00:23:25 INFO Received uploads request method=POST path=/40912026/11/12 00:23:25 INFO Vacuumed table table=pending_objects40922026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads40932026/11/12 00:23:25 INFO Completed upload id=20 objects=3 duration=1.737976ms40942026/11/12 00:23:25 INFO Vacuumed table table=closures40952026/11/12 00:23:25 INFO Vacuumed table table=objects40962026/11/12 00:23:25 INFO Received uploads request method=POST path=/40972026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40982026/11/12 00:23:25 INFO Received push request method=POST path=/40992026/11/12 00:23:25 INFO Received push request method=POST path=/41002026/11/12 00:23:25 INFO Received push request method=POST path=/41012026/11/12 00:23:25 INFO Received push request method=POST path=/41022026/11/12 00:23:25 INFO Received uploads request method=POST path=/41032026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041042026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041052026/11/12 00:23:25 INFO Received push request method=POST path=/41062026/11/12 00:23:25 INFO Received complete upload request method=POST path=/41072026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041082026/11/12 00:23:25 INFO Completed upload id=26 objects=8 duration=1.031789ms41092026/11/12 00:23:25 INFO Received push request method=POST path=/41102026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041112026/11/12 00:23:25 INFO Vacuumed table table=pending_closures41122026/11/12 00:23:25 INFO Vacuumed table table=pending_objects41132026/11/12 00:23:25 INFO Vacuumed table table=pending_closures41142026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads41152026/11/12 00:23:25 INFO Vacuumed table table=closures41162026/11/12 00:23:25 INFO Vacuumed table table=objects41172026/11/12 00:23:25 INFO Received uploads request method=POST path=/41182026/11/12 00:23:25 INFO Vacuumed table table=pending_objects41192026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads41202026/11/12 00:23:25 INFO Vacuumed table table=closures41212026/11/12 00:23:25 INFO Vacuumed table table=objects41222026/11/12 00:23:25 INFO Received complete upload request method=POST path=/41232026/11/12 00:23:25 INFO Completed upload id=20 objects=6 duration=1.026409ms41242026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041252026/11/12 00:23:25 INFO Received complete push request method=POST path=/41262026/11/12 00:23:25 INFO Completed push id=28 objects=14 duration=1.13237ms41272026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=041282026/11/12 00:23:25 INFO Vacuumed table table=pending_closures41292026/11/12 00:23:25 INFO Vacuumed table table=pending_objects41302026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads41312026/11/12 00:23:25 INFO Vacuumed table table=closures41322026/11/12 00:23:25 INFO Received push request method=POST path=/41332026/11/12 00:23:25 INFO Vacuumed table table=objects41342026/11/12 00:23:25 INFO Received uploads request method=POST path=/41352026/11/12 00:23:25 INFO Received complete upload request method=POST path=/41362026/11/12 00:23:25 INFO Received complete upload request method=POST path=/41372026/11/12 00:23:25 INFO Received complete push request method=POST path=/41382026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041392026/11/12 00:23:25 INFO Received push request method=POST path=/41402026/11/12 00:23:25 INFO Completed upload id=24 objects=7 duration=1.09907ms41412026/11/12 00:23:25 INFO Completed upload id=19 objects=25 duration=3.022228ms41422026/11/12 00:23:25 INFO Completed push id=21 objects=3 duration=1.11941ms41432026/11/12 00:23:25 INFO Received uploads request method=POST path=/41442026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041452026/11/12 00:23:25 INFO Received push request method=POST path=/41462026/11/12 00:23:25 INFO Received complete upload request method=POST path=/41472026/11/12 00:23:25 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."41482026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."41492026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period41502026/11/12 00:23:25 INFO Completed upload id=33 objects=5 duration=2.323661ms41512026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041522026/11/12 00:23:25 INFO Received push request method=POST path=/41532026/11/12 00:23:25 INFO Vacuumed table table=pending_closures41542026/11/12 00:23:25 INFO Vacuumed table table=pending_objects41552026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads41562026/11/12 00:23:25 INFO Vacuumed table table=closures41572026/11/12 00:23:25 INFO Vacuumed table table=objects41582026/11/12 00:23:25 INFO Received uploads request method=POST path=/41592026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041602026/11/12 00:23:25 INFO Received push request method=POST path=/41612026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041622026/11/12 00:23:25 INFO Received uploads request method=POST path=/41632026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period41642026/11/12 00:23:25 INFO Received uploads request method=POST path=/41652026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041662026/11/12 00:23:25 INFO Vacuumed table table=pending_closures41672026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041682026/11/12 00:23:25 INFO Vacuumed table table=pending_objects41692026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads41702026/11/12 00:23:25 INFO Vacuumed table table=pending_closures41712026/11/12 00:23:25 INFO Vacuumed table table=closures41722026/11/12 00:23:25 INFO Vacuumed table table=objects41732026/11/12 00:23:25 INFO Vacuumed table table=pending_objects41742026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads41752026/11/12 00:23:25 INFO Vacuumed table table=closures41762026/11/12 00:23:25 INFO Received push request method=POST path=/41772026/11/12 00:23:25 INFO Received push request method=POST path=/41782026/11/12 00:23:25 INFO Vacuumed table table=objects41792026/11/12 00:23:25 INFO Received push request method=POST path=/41802026/11/12 00:23:25 INFO Received complete push request method=POST path=/41812026/11/12 00:23:25 INFO Received uploads request method=POST path=/41822026/11/12 00:23:25 INFO Completed push id=4 objects=6 duration=1.577854ms41832026/11/12 00:23:25 INFO Received complete upload request method=POST path=/41842026/11/12 00:23:25 INFO Completed upload id=26 objects=2 duration=1.680455ms41852026/11/12 00:23:25 INFO Received complete upload request method=POST path=/41862026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041872026/11/12 00:23:25 INFO Completed upload id=21 objects=2 duration=864.508µs41882026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period41892026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041902026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period41912026/11/12 00:23:25 INFO Received uploads request method=POST path=/41922026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041932026/11/12 00:23:25 INFO Received push request method=POST path=/41942026/11/12 00:23:25 INFO Vacuumed table table=pending_closures41952026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041962026/11/12 00:23:25 INFO Received uploads request method=POST path=/41972026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=041982026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=041992026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42002026/11/12 00:23:25 INFO Received uploads request method=POST path=/42012026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42022026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42032026/11/12 00:23:25 INFO Vacuumed table table=closures42042026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period42052026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42062026/11/12 00:23:25 INFO Vacuumed table table=objects42072026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42082026/11/12 00:23:25 INFO Vacuumed table table=closures42092026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042102026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042112026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042122026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042132026/11/12 00:23:25 INFO Received uploads request method=POST path=/42142026/11/12 00:23:25 INFO Vacuumed table table=objects42152026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42162026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42172026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42182026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042192026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42202026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42212026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42222026/11/12 00:23:25 INFO Vacuumed table table=closures42232026/11/12 00:23:25 INFO Received complete push request method=POST path=/42242026/11/12 00:23:25 INFO Received complete upload request method=POST path=/42252026/11/12 00:23:25 INFO Vacuumed table table=objects42262026/11/12 00:23:25 INFO Vacuumed table table=closures42272026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42282026/11/12 00:23:25 INFO Completed upload id=21 objects=25 duration=1.526994ms42292026/11/12 00:23:25 INFO Vacuumed table table=objects42302026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042312026/11/12 00:23:25 INFO Completed push id=27 objects=25 duration=3.315811ms42322026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42332026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42342026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42352026/11/12 00:23:25 INFO Vacuumed table table=closures42362026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042372026/11/12 00:23:25 INFO Vacuumed table table=objects42382026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42392026/11/12 00:23:25 INFO Received push request method=POST path=/42402026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042412026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42422026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period42432026/11/12 00:23:25 INFO Vacuumed table table=closures42442026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period42452026/11/12 00:23:25 INFO Received push request method=POST path=/42462026/11/12 00:23:25 INFO Vacuumed table table=objects42472026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042482026/11/12 00:23:25 INFO Received uploads request method=POST path=/42492026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42502026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042512026/11/12 00:23:25 INFO Received complete upload request method=POST path=/42522026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42532026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42542026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42552026/11/12 00:23:25 INFO Vacuumed table table=closures42562026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42572026/11/12 00:23:25 INFO Vacuumed table table=objects42582026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42592026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042602026/11/12 00:23:25 INFO Vacuumed table table=closures42612026/11/12 00:23:25 INFO Completed upload id=21 objects=7 duration=2.479383ms42622026/11/12 00:23:25 INFO Vacuumed table table=objects42632026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period42642026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042652026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42662026/11/12 00:23:25 INFO Received uploads request method=POST path=/42672026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042682026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42692026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42702026/11/12 00:23:25 INFO Received uploads request method=POST path=/42712026/11/12 00:23:25 INFO Vacuumed table table=closures42722026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042732026/11/12 00:23:25 INFO Vacuumed table table=objects42742026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042752026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42762026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042772026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42782026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42792026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042802026/11/12 00:23:25 INFO Vacuumed table table=closures42812026/11/12 00:23:25 INFO Received push request method=POST path=/42822026/11/12 00:23:25 INFO Received uploads request method=POST path=/42832026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42842026/11/12 00:23:25 INFO Vacuumed table table=objects42852026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period42862026/11/12 00:23:25 INFO Vacuumed table table=pending_objects42872026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads42882026/11/12 00:23:25 INFO Vacuumed table table=closures42892026/11/12 00:23:25 INFO Vacuumed table table=objects42902026/11/12 00:23:25 INFO Received complete upload request method=POST path=/42912026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=042922026/11/12 00:23:25 INFO Completed upload id=21 objects=9 duration=2.647505ms42932026/11/12 00:23:25 INFO Received push request method=POST path=/42942026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period42952026/11/12 00:23:25 INFO Received complete push request method=POST path=/42962026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042972026/11/12 00:23:25 INFO Vacuumed table table=pending_closures42982026/11/12 00:23:25 INFO Completed push id=23 objects=28 duration=2.538684ms42992026/11/12 00:23:25 INFO Received uploads request method=POST path=/43002026/11/12 00:23:25 INFO Vacuumed table table=pending_objects43012026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads43022026/11/12 00:23:25 INFO Vacuumed table table=closures43032026/11/12 00:23:25 INFO Vacuumed table table=objects43042026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=043052026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period43062026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043072026/11/12 00:23:25 INFO Received push request method=POST path=/43082026/11/12 00:23:25 INFO Vacuumed table table=pending_closures43092026/11/12 00:23:25 INFO Received complete push request method=POST path=/43102026/11/12 00:23:25 INFO Received push request method=POST path=/43112026/11/12 00:23:25 INFO Vacuumed table table=pending_objects43122026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads43132026/11/12 00:23:25 INFO Vacuumed table table=closures43142026/11/12 00:23:25 INFO Vacuumed table table=objects43152026/11/12 00:23:25 INFO Completed push id=24 objects=5 duration=2.439503ms43162026/11/12 00:23:25 INFO Received complete push request method=POST path=/43172026/11/12 00:23:25 INFO Completed push id=25 objects=10 duration=1.558454ms43182026/11/12 00:23:25 INFO Received push request method=POST path=/43192026/11/12 00:23:25 INFO Received uploads request method=POST path=/43202026/11/12 00:23:25 INFO Received uploads request method=POST path=/43212026/11/12 00:23:25 INFO Received uploads request method=POST path=/43222026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43232026/11/12 00:23:25 INFO Received complete push request method=POST path=/43242026/11/12 00:23:25 INFO Received complete push request method=POST path=/43252026/11/12 00:23:25 INFO Completed upload id=19 objects=2 duration=1.854498ms43262026/11/12 00:23:25 INFO Completed push id=14 objects=3 duration=3.20069ms43272026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=043282026/11/12 00:23:25 INFO Completed push id=30 objects=4 duration=2.845887ms43292026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43302026/11/12 00:23:25 INFO Completed upload id=12 objects=6 duration=1.190771ms43312026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43322026/11/12 00:23:25 INFO Completed upload id=29 objects=5 duration=873.548µs43332026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43342026/11/12 00:23:25 INFO Completed upload id=27 objects=8 duration=778.167µs43352026/11/12 00:23:25 INFO Received push request method=POST path=/43362026/11/12 00:23:25 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43372026/11/12 00:23:25 INFO Received uploads request method=POST path=/43382026/11/12 00:23:25 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."43392026/11/12 00:23:25 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43402026/11/12 00:23:25 INFO Received push request method=POST path=/43412026/11/12 00:23:25 INFO Received uploads request method=POST path=/43422026/11/12 00:23:25 INFO Received uploads request method=POST path=/43432026/11/12 00:23:25 INFO Received uploads request method=POST path=/43442026/11/12 00:23:25 INFO Received complete push request method=POST path=/43452026/11/12 00:23:25 INFO Completed push id=17 objects=3 duration=1.807036ms43462026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43472026/11/12 00:23:25 INFO Completed upload id=30 objects=2 duration=765.487µs43482026/11/12 00:23:25 INFO Received push request method=POST path=/43492026/11/12 00:23:25 INFO Received complete push request method=POST path=/43502026/11/12 00:23:25 INFO Completed push id=21 objects=9 duration=2.19838ms43512026/11/12 00:23:25 INFO Received push request method=POST path=/43522026/11/12 00:23:25 INFO Received complete push request method=POST path=/43532026/11/12 00:23:25 INFO Received push request method=POST path=/43542026/11/12 00:23:25 INFO Received complete push request method=POST path=/43552026/11/12 00:23:25 INFO Completed push id=33 objects=7 duration=1.782156ms43562026/11/12 00:23:25 INFO Received uploads request method=POST path=/43572026/11/12 00:23:25 INFO Received uploads request method=POST path=/43582026/11/12 00:23:25 INFO Received push request method=POST path=/43592026/11/12 00:23:25 INFO Received complete push request method=POST path=/43602026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43612026/11/12 00:23:25 INFO Completed push id=31 objects=20 duration=1.520854ms43622026/11/12 00:23:25 INFO Completed push id=24 objects=24 duration=7.975834ms43632026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=043642026/11/12 00:23:25 INFO Received complete push request method=POST path=/43652026/11/12 00:23:25 INFO Completed upload id=23 objects=16 duration=2.604704ms43662026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43672026/11/12 00:23:25 INFO Completed push id=23 objects=5 duration=1.401393ms43682026/11/12 00:23:25 INFO Completed upload id=26 objects=3 duration=811.287µs43692026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043702026/11/12 00:23:25 INFO Vacuumed table table=pending_closures43712026/11/12 00:23:25 INFO Vacuumed table table=pending_objects43722026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads43732026/11/12 00:23:25 INFO Vacuumed table table=closures43742026/11/12 00:23:25 INFO Received complete push request method=POST path=/43752026/11/12 00:23:25 INFO Vacuumed table table=objects43762026/11/12 00:23:25 INFO Completed push id=21 objects=26 duration=1.664956ms43772026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43782026/11/12 00:23:25 INFO Received push request method=POST path=/43792026/11/12 00:23:25 INFO Completed upload id=27 objects=4 duration=944.909µs43802026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43812026/11/12 00:23:25 INFO Completed upload id=32 objects=5 duration=767.427µs43822026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43832026/11/12 00:23:25 INFO Received uploads request method=POST path=/43842026/11/12 00:23:25 INFO Received push request method=POST path=/43852026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=043862026/11/12 00:23:25 INFO Completed upload id=23 objects=18 duration=2.1261ms43872026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=043882026/11/12 00:23:25 INFO Received complete upload request method=POST path=/43892026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period43902026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=043912026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period43922026/11/12 00:23:25 INFO Completed upload id=24 objects=3 duration=1.000869ms43932026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period43942026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043952026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043962026/11/12 00:23:25 INFO Vacuumed table table=pending_closures43972026/11/12 00:23:25 INFO Vacuumed table table=pending_closures43982026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043992026/11/12 00:23:25 INFO Vacuumed table table=pending_objects44002026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads44012026/11/12 00:23:25 INFO Vacuumed table table=pending_closures44022026/11/12 00:23:25 INFO Vacuumed table table=pending_objects44032026/11/12 00:23:25 INFO Vacuumed table table=closures44042026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads44052026/11/12 00:23:25 INFO Vacuumed table table=objects44062026/11/12 00:23:25 INFO Vacuumed table table=closures44072026/11/12 00:23:25 INFO Vacuumed table table=objects44082026/11/12 00:23:25 INFO Vacuumed table table=pending_objects44092026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads44102026/11/12 00:23:25 INFO Vacuumed table table=closures44112026/11/12 00:23:25 INFO Vacuumed table table=objects44122026/11/12 00:23:25 INFO Received uploads request method=POST path=/44132026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=044142026/11/12 00:23:25 INFO Received complete push request method=POST path=/44152026/11/12 00:23:25 INFO Received complete push request method=POST path=/44162026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044172026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44182026/11/12 00:23:25 INFO Vacuumed table table=pending_closures44192026/11/12 00:23:25 INFO Completed push id=28 objects=11 duration=2.0614ms44202026/11/12 00:23:25 INFO Completed upload id=33 objects=3 duration=1.04269ms44212026/11/12 00:23:25 INFO Completed push id=13 objects=11 duration=1.924278ms44222026/11/12 00:23:25 INFO Vacuumed table table=pending_objects44232026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads44242026/11/12 00:23:25 INFO Vacuumed table table=closures44252026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44262026/11/12 00:23:25 INFO Received uploads request method=POST path=/44272026/11/12 00:23:25 INFO Vacuumed table table=objects44282026/11/12 00:23:25 INFO Completed upload id=28 objects=17 duration=1.292933ms44292026/11/12 00:23:25 INFO Received complete push request method=POST path=/44302026/11/12 00:23:25 INFO Received complete push request method=POST path=/44312026/11/12 00:23:25 INFO Completed push id=25 objects=4 duration=861.268µs44322026/11/12 00:23:25 INFO Completed push id=32 objects=7 duration=1.07351ms44332026/11/12 00:23:25 INFO Received uploads request method=POST path=/44342026/11/12 00:23:25 INFO Received uploads request method=POST path=/44352026/11/12 00:23:25 INFO Received complete push request method=POST path=/44362026/11/12 00:23:25 INFO Completed push id=16 objects=4 duration=2.082859ms44372026/11/12 00:23:25 INFO Received uploads request method=POST path=/44382026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=044392026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44402026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period44412026/11/12 00:23:25 INFO Completed upload id=25 objects=16 duration=1.881777ms44422026/11/12 00:23:25 INFO Received push request method=POST path=/44432026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044442026/11/12 00:23:25 INFO Received complete push request method=POST path=/44452026/11/12 00:23:25 INFO Vacuumed table table=pending_closures44462026/11/12 00:23:25 INFO Received complete push request method=POST path=/44472026/11/12 00:23:25 INFO Completed push id=29 objects=2 duration=1.01451ms44482026/11/12 00:23:25 INFO Completed push id=26 objects=4 duration=968.769µs44492026/11/12 00:23:25 INFO Vacuumed table table=pending_objects44502026/11/12 00:23:25 INFO Received uploads request method=POST path=/44512026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads44522026/11/12 00:23:25 INFO Vacuumed table table=closures44532026/11/12 00:23:25 INFO Vacuumed table table=objects44542026/11/12 00:23:25 INFO Received uploads request method=POST path=/44552026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=044562026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=044572026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period44582026/11/12 00:23:25 INFO Received complete push request method=POST path=/44592026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044602026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period44612026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44622026/11/12 00:23:25 INFO Completed push id=29 objects=24 duration=1.543994ms44632026/11/12 00:23:25 INFO Received push request method=POST path=/44642026/11/12 00:23:25 INFO Vacuumed table table=pending_closures44652026/11/12 00:23:25 INFO Completed upload id=22 objects=22 duration=935.488µs44662026/11/12 00:23:25 INFO Received complete push request method=POST path=/44672026/11/12 00:23:25 INFO Vacuumed table table=pending_objects44682026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads44692026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44702026/11/12 00:23:25 INFO Completed push id=31 objects=6 duration=1.199352ms44712026/11/12 00:23:25 INFO Vacuumed table table=closures44722026/11/12 00:23:25 INFO Completed upload id=28 objects=13 duration=1.04643ms44732026/11/12 00:23:25 INFO Vacuumed table table=objects44742026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44752026/11/12 00:23:25 INFO Completed upload id=20 objects=5 duration=1.407793ms44762026/11/12 00:23:25 INFO Received complete push request method=POST path=/44772026/11/12 00:23:25 INFO Completed push id=9 objects=9 duration=2.537343ms44782026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=044792026/11/12 00:23:25 INFO Received complete push request method=POST path=/44802026/11/12 00:23:25 INFO Received push request method=POST path=/44812026/11/12 00:23:25 INFO Received complete push request method=POST path=/44822026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period44832026/11/12 00:23:25 INFO Completed push id=29 objects=3 duration=971.129µs44842026/11/12 00:23:25 INFO Completed push id=26 objects=23 duration=1.571055ms44852026/11/12 00:23:25 INFO Received push request method=POST path=/44862026/11/12 00:23:25 INFO Received complete push request method=POST path=/44872026/11/12 00:23:25 INFO Received push request method=POST path=/44882026/11/12 00:23:25 INFO Completed push id=32 objects=9 duration=988.589µs44892026/11/12 00:23:25 INFO Received push request method=POST path=/44902026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044912026/11/12 00:23:25 INFO Vacuumed table table=pending_closures44922026/11/12 00:23:25 INFO Vacuumed table table=pending_objects44932026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads44942026/11/12 00:23:25 INFO Vacuumed table table=closures44952026/11/12 00:23:25 INFO Received uploads request method=POST path=/44962026/11/12 00:23:25 INFO Vacuumed table table=objects44972026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44982026/11/12 00:23:25 INFO Received complete upload request method=POST path=/44992026/11/12 00:23:25 INFO Completed upload id=21 objects=8 duration=1.011029ms45002026/11/12 00:23:25 INFO Completed upload id=31 objects=2 duration=933.589µs45012026/11/12 00:23:25 INFO Received uploads request method=POST path=/45022026/11/12 00:23:25 INFO Received push request method=POST path=/45032026/11/12 00:23:25 INFO Received complete push request method=POST path=/45042026/11/12 00:23:25 INFO Completed push id=26 objects=5 duration=2.086899ms45052026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45062026/11/12 00:23:25 INFO Completed upload id=33 objects=9 duration=890.888µs45072026/11/12 00:23:25 INFO Received push request method=POST path=/45082026/11/12 00:23:25 INFO Received complete push request method=POST path=/45092026/11/12 00:23:25 INFO Received complete push request method=POST path=/45102026/11/12 00:23:25 INFO Completed push id=22 objects=6 duration=1.06975ms45112026/11/12 00:23:25 INFO Received complete push request method=POST path=/45122026/11/12 00:23:25 INFO Completed push id=27 objects=15 duration=1.09691ms45132026/11/12 00:23:25 INFO Completed push id=33 objects=6 duration=852.308µs45142026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=045152026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45162026/11/12 00:23:25 INFO Completed upload id=24 objects=14 duration=873.528µs45172026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=045182026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period45192026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period45202026/11/12 00:23:25 INFO Received complete push request method=POST path=/45212026/11/12 00:23:25 INFO Received push request method=POST path=/45222026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045232026/11/12 00:23:25 INFO Completed push id=24 objects=8 duration=967.949µs45242026/11/12 00:23:25 INFO Received complete push request method=POST path=/45252026/11/12 00:23:25 INFO Vacuumed table table=pending_closures45262026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045272026/11/12 00:23:25 INFO Vacuumed table table=pending_objects45282026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads45292026/11/12 00:23:25 INFO Vacuumed table table=closures45302026/11/12 00:23:25 INFO Completed push id=7 objects=15 duration=1.960698ms45312026/11/12 00:23:25 INFO Vacuumed table table=objects45322026/11/12 00:23:25 INFO Vacuumed table table=pending_closures45332026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=045342026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period45352026/11/12 00:23:25 INFO Vacuumed table table=pending_objects45362026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads45372026/11/12 00:23:25 INFO Vacuumed table table=closures45382026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045392026/11/12 00:23:25 INFO Vacuumed table table=objects45402026/11/12 00:23:25 INFO Vacuumed table table=pending_closures45412026/11/12 00:23:25 INFO Vacuumed table table=pending_objects45422026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads45432026/11/12 00:23:25 INFO Received complete push request method=POST path=/45442026/11/12 00:23:25 INFO Vacuumed table table=closures45452026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45462026/11/12 00:23:25 INFO Completed push id=19 objects=3 duration=1.04017ms45472026/11/12 00:23:25 INFO Vacuumed table table=objects45482026/11/12 00:23:25 INFO Completed upload id=25 objects=9 duration=1.263432ms45492026/11/12 00:23:25 INFO Received push request method=POST path=/45502026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45512026/11/12 00:23:25 INFO Completed upload id=27 objects=3 duration=882.668µs45522026/11/12 00:23:25 INFO Received uploads request method=POST path=/45532026/11/12 00:23:25 INFO Received complete push request method=POST path=/45542026/11/12 00:23:25 INFO Completed push id=28 objects=19 duration=1.333072ms45552026/11/12 00:23:25 INFO Received push request method=POST path=/45562026/11/12 00:23:25 INFO Received push request method=POST path=/45572026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45582026/11/12 00:23:25 INFO Completed upload id=28 objects=6 duration=824.087µs45592026/11/12 00:23:25 INFO Received uploads request method=POST path=/45602026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45612026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45622026/11/12 00:23:25 INFO Completed upload id=13 objects=8 duration=1.149691ms45632026/11/12 00:23:25 INFO Completed upload id=22 objects=5 duration=1.523514ms45642026/11/12 00:23:25 INFO Received push request method=POST path=/45652026/11/12 00:23:25 INFO Received complete push request method=POST path=/45662026/11/12 00:23:25 INFO Completed push id=30 objects=4 duration=856.927µs45672026/11/12 00:23:25 INFO Received complete push request method=POST path=/45682026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45692026/11/12 00:23:25 INFO Completed push id=29 objects=10 duration=1.08139ms45702026/11/12 00:23:25 INFO Received uploads request method=POST path=/45712026/11/12 00:23:25 INFO Completed upload id=23 objects=7 duration=2.380242ms45722026/11/12 00:23:25 INFO Received complete push request method=POST path=/45732026/11/12 00:23:25 INFO Completed push id=28 objects=3 duration=1.777056ms45742026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=045752026/11/12 00:23:25 INFO Received complete push request method=POST path=/45762026/11/12 00:23:25 INFO Completed push id=31 objects=7 duration=861.608µs45772026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period45782026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045792026/11/12 00:23:25 INFO Vacuumed table table=pending_closures45802026/11/12 00:23:25 INFO Received uploads request method=POST path=/45812026/11/12 00:23:25 INFO Vacuumed table table=pending_objects45822026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads45832026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45842026/11/12 00:23:25 INFO Vacuumed table table=closures45852026/11/12 00:23:25 INFO Completed upload id=26 objects=5 duration=936.168µs45862026/11/12 00:23:25 INFO Vacuumed table table=objects45872026/11/12 00:23:25 INFO Received uploads request method=POST path=/45882026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45892026/11/12 00:23:25 INFO Received complete push request method=POST path=/45902026/11/12 00:23:25 INFO Completed upload id=30 objects=6 duration=899.409µs45912026/11/12 00:23:25 INFO Completed push id=29 objects=15 duration=1.10555ms45922026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45932026/11/12 00:23:25 INFO Completed upload id=32 objects=5 duration=887.028µs45942026/11/12 00:23:25 INFO Received complete upload request method=POST path=/45952026/11/12 00:23:25 INFO Received complete push request method=POST path=/45962026/11/12 00:23:25 INFO Completed upload id=25 objects=4 duration=1.063489ms45972026/11/12 00:23:25 INFO Completed push id=27 objects=3 duration=2.017199ms45982026/11/12 00:23:25 INFO Received push request method=POST path=/45992026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=046002026/11/12 00:23:25 INFO Received complete upload request method=POST path=/46012026/11/12 00:23:25 INFO Received complete push request method=POST path=/46022026/11/12 00:23:25 INFO Completed upload id=31 objects=5 duration=1.002449ms46032026/11/12 00:23:25 INFO Completed push id=30 objects=6 duration=792.888µs46042026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046052026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46062026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46072026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46082026/11/12 00:23:25 INFO Vacuumed table table=closures46092026/11/12 00:23:25 INFO Vacuumed table table=objects46102026/11/12 00:23:25 INFO Received push request method=POST path=/46112026/11/12 00:23:25 INFO Received complete upload request method=POST path=/46122026/11/12 00:23:25 INFO Completed upload id=28 objects=5 duration=808.107µs46132026/11/12 00:23:25 INFO Received complete upload request method=POST path=/46142026/11/12 00:23:25 INFO Received complete upload request method=POST path=/46152026/11/12 00:23:25 INFO Completed upload id=34 objects=3 duration=705.047µs46162026/11/12 00:23:25 INFO Completed upload id=31 objects=3 duration=651.846µs46172026/11/12 00:23:25 INFO Received push request method=POST path=/46182026/11/12 00:23:25 INFO Received complete push request method=POST path=/46192026/11/12 00:23:25 INFO Completed push id=29 objects=17 duration=1.07301ms46202026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=046212026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=046222026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period46232026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period46242026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046252026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046262026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46272026/11/12 00:23:25 INFO Received complete upload request method=POST path=/46282026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46292026/11/12 00:23:25 INFO Received complete push request method=POST path=/46302026/11/12 00:23:25 INFO Received complete push request method=POST path=/46312026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46322026/11/12 00:23:25 INFO Completed upload id=34 objects=24 duration=1.469054ms46332026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46342026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46352026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46362026/11/12 00:23:25 INFO Vacuumed table table=closures46372026/11/12 00:23:25 INFO Completed push id=19 objects=14 duration=1.729496ms46382026/11/12 00:23:25 INFO Vacuumed table table=closures46392026/11/12 00:23:25 INFO Completed push id=5 objects=28 duration=2.446083ms46402026/11/12 00:23:25 INFO Vacuumed table table=objects46412026/11/12 00:23:25 INFO Vacuumed table table=objects46422026/11/12 00:23:25 INFO Received complete push request method=POST path=/46432026/11/12 00:23:25 INFO Completed push id=20 objects=8 duration=958.269µs46442026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=046452026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period46462026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046472026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46482026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46492026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46502026/11/12 00:23:25 INFO Vacuumed table table=closures46512026/11/12 00:23:25 INFO Received push request method=POST path=/46522026/11/12 00:23:25 INFO Vacuumed table table=objects46532026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=046542026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=046552026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period46562026/11/12 00:23:25 INFO Received complete push request method=POST path=/46572026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period46582026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046592026/11/12 00:23:25 INFO Received uploads request method=POST path=/46602026/11/12 00:23:25 INFO Completed push id=10 objects=2 duration=1.946778ms46612026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46622026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46632026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46642026/11/12 00:23:25 INFO Vacuumed table table=closures46652026/11/12 00:23:25 INFO Vacuumed table table=objects46662026/11/12 00:23:25 INFO Received push request method=POST path=/46672026/11/12 00:23:25 INFO Received complete upload request method=POST path=/46682026/11/12 00:23:25 INFO Completed upload id=17 objects=5 duration=1.165831ms46692026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=046702026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=046712026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period46722026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46732026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046742026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46752026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46762026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46772026/11/12 00:23:25 INFO Vacuumed table table=closures46782026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46792026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46802026/11/12 00:23:25 INFO Vacuumed table table=objects46812026/11/12 00:23:25 INFO Vacuumed table table=closures46822026/11/12 00:23:25 INFO Vacuumed table table=objects46832026/11/12 00:23:25 INFO Received complete push request method=POST path=/46842026/11/12 00:23:25 INFO Completed push id=19 objects=3 duration=1.145051ms46852026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046862026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46872026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46882026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46892026/11/12 00:23:25 INFO Vacuumed table table=closures46902026/11/12 00:23:25 INFO Vacuumed table table=objects46912026/11/12 00:23:25 INFO Received push request method=POST path=/46922026/11/12 00:23:25 INFO Received complete push request method=POST path=/46932026/11/12 00:23:25 INFO Completed push id=27 objects=11 duration=1.008369ms46942026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046952026/11/12 00:23:25 INFO Vacuumed table table=pending_closures46962026/11/12 00:23:25 INFO Vacuumed table table=pending_objects46972026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads46982026/11/12 00:23:25 INFO Received uploads request method=POST path=/46992026/11/12 00:23:25 INFO Vacuumed table table=closures47002026/11/12 00:23:25 INFO Vacuumed table table=objects47012026/11/12 00:23:25 INFO Received push request method=POST path=/47022026/11/12 00:23:25 INFO Received complete upload request method=POST path=/47032026/11/12 00:23:25 INFO Completed upload id=29 objects=26 duration=984.509µs47042026/11/12 00:23:25 INFO Received complete push request method=POST path=/47052026/11/12 00:23:25 INFO Completed push id=20 objects=4 duration=1.05143ms47062026/11/12 00:23:25 INFO Received uploads request method=POST path=/47072026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=047082026/11/12 00:23:25 INFO Received uploads request method=POST path=/47092026/11/12 00:23:25 INFO Received push request method=POST path=/47102026/11/12 00:23:25 INFO Received complete push request method=POST path=/47112026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047122026/11/12 00:23:25 INFO Completed push id=30 objects=8 duration=1.363492ms47132026/11/12 00:23:25 INFO Vacuumed table table=pending_closures47142026/11/12 00:23:25 INFO Vacuumed table table=pending_objects47152026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads47162026/11/12 00:23:25 INFO Vacuumed table table=closures47172026/11/12 00:23:25 INFO Vacuumed table table=objects47182026/11/12 00:23:25 INFO Received complete push request method=POST path=/47192026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=047202026/11/12 00:23:25 INFO Received complete upload request method=POST path=/47212026/11/12 00:23:25 INFO Received complete push request method=POST path=/47222026/11/12 00:23:25 INFO Completed push id=16 objects=14 duration=2.345822ms47232026/11/12 00:23:25 INFO Received complete upload request method=POST path=/47242026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period47252026/11/12 00:23:25 INFO Completed upload id=22 objects=8 duration=972.628µs47262026/11/12 00:23:25 INFO Completed upload id=31 objects=5 duration=704.787µs47272026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047282026/11/12 00:23:25 INFO Completed push id=31 objects=14 duration=2.403742ms47292026/11/12 00:23:25 INFO Vacuumed table table=pending_closures47302026/11/12 00:23:25 INFO Vacuumed table table=pending_objects47312026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads47322026/11/12 00:23:25 INFO Vacuumed table table=closures47332026/11/12 00:23:25 INFO Vacuumed table table=objects47342026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=047352026/11/12 00:23:25 INFO Received complete push request method=POST path=/47362026/11/12 00:23:25 INFO Received complete push request method=POST path=/47372026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period47382026/11/12 00:23:25 INFO Completed push id=32 objects=3 duration=946.929µs47392026/11/12 00:23:25 INFO Completed push id=23 objects=5 duration=1.102671ms47402026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047412026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=047422026/11/12 00:23:25 INFO Vacuumed table table=pending_closures47432026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period47442026/11/12 00:23:25 INFO Vacuumed table table=pending_objects47452026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads47462026/11/12 00:23:25 INFO Vacuumed table table=closures47472026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047482026/11/12 00:23:25 INFO Vacuumed table table=pending_closures47492026/11/12 00:23:25 INFO Vacuumed table table=objects47502026/11/12 00:23:25 INFO Vacuumed table table=pending_objects47512026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads47522026/11/12 00:23:25 INFO Vacuumed table table=closures47532026/11/12 00:23:25 INFO Vacuumed table table=objects47542026/11/12 00:23:25 INFO Received uploads request method=POST path=/47552026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=047562026/11/12 00:23:25 INFO Received complete upload request method=POST path=/47572026/11/12 00:23:25 INFO Completed upload id=24 objects=8 duration=797.147µs47582026/11/12 00:23:26 WARN Force mode enabled - objects will be deleted immediately without grace period47592026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=047602026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047612026/11/12 00:23:26 INFO Vacuumed table table=pending_closures47622026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047632026/11/12 00:23:26 INFO Vacuumed table table=pending_closures47642026/11/12 00:23:26 INFO Vacuumed table table=pending_objects47652026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads47662026/11/12 00:23:26 INFO Vacuumed table table=pending_objects47672026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads47682026/11/12 00:23:26 INFO Vacuumed table table=closures47692026/11/12 00:23:26 INFO Vacuumed table table=closures47702026/11/12 00:23:26 INFO Vacuumed table table=objects47712026/11/12 00:23:26 INFO Vacuumed table table=objects47722026/11/12 00:23:26 INFO Received complete upload request method=POST path=/47732026/11/12 00:23:26 INFO Received complete upload request method=POST path=/47742026/11/12 00:23:26 INFO Completed upload id=28 objects=5 duration=1.11707ms47752026/11/12 00:23:26 INFO Received complete upload request method=POST path=/47762026/11/12 00:23:26 INFO Completed upload id=17 objects=25 duration=2.553344ms47772026/11/12 00:23:26 INFO Completed upload id=13 objects=14 duration=2.331581ms47782026/11/12 00:23:26 INFO Received push request method=POST path=/47792026/11/12 00:23:26 INFO Received push request method=POST path=/47802026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=047812026/11/12 00:23:26 INFO Received complete upload request method=POST path=/47822026/11/12 00:23:26 INFO Completed upload id=30 objects=5 duration=734.386µs47832026/11/12 00:23:26 INFO Received uploads request method=POST path=/47842026/11/12 00:23:26 INFO Received complete push request method=POST path=/47852026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047862026/11/12 00:23:26 INFO Vacuumed table table=pending_closures47872026/11/12 00:23:26 INFO Completed push id=7 objects=7 duration=1.216951ms47882026/11/12 00:23:26 INFO Vacuumed table table=pending_objects47892026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads47902026/11/12 00:23:26 INFO Vacuumed table table=closures47912026/11/12 00:23:26 INFO Vacuumed table table=objects47922026/11/12 00:23:26 INFO Received complete upload request method=POST path=/47932026/11/12 00:23:26 INFO Completed upload id=21 objects=2 duration=923.509µs47942026/11/12 00:23:26 INFO Received complete upload request method=POST path=/47952026/11/12 00:23:26 INFO Completed upload id=25 objects=6 duration=1.334032ms47962026/11/12 00:23:26 INFO Received complete push request method=POST path=/47972026/11/12 00:23:26 INFO Completed push id=12 objects=6 duration=1.07759ms47982026/11/12 00:23:26 INFO Received complete push request method=POST path=/47992026/11/12 00:23:26 INFO Completed push id=31 objects=3 duration=836.468µs48002026/11/12 00:23:26 INFO Received complete push request method=POST path=/48012026/11/12 00:23:26 INFO Completed push id=31 objects=6 duration=2.852006ms48022026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=048032026/11/12 00:23:26 WARN Force mode enabled - objects will be deleted immediately without grace period48042026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048052026/11/12 00:23:26 INFO Vacuumed table table=pending_closures48062026/11/12 00:23:26 INFO Vacuumed table table=pending_objects48072026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads48082026/11/12 00:23:26 INFO Vacuumed table table=closures48092026/11/12 00:23:26 INFO Vacuumed table table=objects48102026/11/12 00:23:26 INFO Received uploads request method=POST path=/48112026/11/12 00:23:26 INFO Received push request method=POST path=/48122026/11/12 00:23:26 INFO Received complete push request method=POST path=/48132026/11/12 00:23:26 INFO Completed push id=32 objects=6 duration=965.069µs48142026/11/12 00:23:26 INFO Received uploads request method=POST path=/48152026/11/12 00:23:26 INFO Received complete push request method=POST path=/48162026/11/12 00:23:26 INFO Completed push id=13 objects=22 duration=1.280692ms48172026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48182026/11/12 00:23:26 INFO Completed upload id=33 objects=7 duration=702.346µs48192026/11/12 00:23:26 INFO Received complete push request method=POST path=/48202026/11/12 00:23:26 INFO Completed push id=11 objects=9 duration=1.660675ms48212026/11/12 00:23:26 INFO Received complete push request method=POST path=/48222026/11/12 00:23:26 INFO Completed push id=21 objects=6 duration=1.140531ms48232026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48242026/11/12 00:23:26 INFO Completed upload id=34 objects=3 duration=604.566µs48252026/11/12 00:23:26 INFO Received complete push request method=POST path=/48262026/11/12 00:23:26 INFO Completed push id=7 objects=24 duration=3.760635ms48272026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48282026/11/12 00:23:26 INFO Completed upload id=22 objects=7 duration=974.549µs48292026/11/12 00:23:26 INFO Received complete push request method=POST path=/48302026/11/12 00:23:26 INFO Received complete push request method=POST path=/48312026/11/12 00:23:26 INFO Completed push id=35 objects=11 duration=871.148µs48322026/11/12 00:23:26 INFO Completed push id=15 objects=24 duration=1.322392ms48332026/11/12 00:23:26 INFO Received complete push request method=POST path=/48342026/11/12 00:23:26 INFO Completed push id=1 objects=9 duration=1.025949ms48352026/11/12 00:23:26 INFO Received uploads request method=POST path=/48362026/11/12 00:23:26 INFO Received complete push request method=POST path=/48372026/11/12 00:23:26 INFO Completed push id=24 objects=4 duration=778.887µs48382026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48392026/11/12 00:23:26 INFO Completed upload id=19 objects=2 duration=893.448µs48402026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=048412026/11/12 00:23:26 WARN Force mode enabled - objects will be deleted immediately without grace period48422026/11/12 00:23:26 INFO Received uploads request method=POST path=/48432026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48442026/11/12 00:23:26 INFO Received complete push request method=POST path=/48452026/11/12 00:23:26 INFO Completed upload id=24 objects=7 duration=630.746µs48462026/11/12 00:23:26 INFO Completed push id=25 objects=9 duration=902.249µs48472026/11/12 00:23:26 INFO Received uploads request method=POST path=/48482026/11/12 00:23:26 INFO Received complete push request method=POST path=/48492026/11/12 00:23:26 INFO Completed push id=26 objects=3 duration=665.246µs48502026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48512026/11/12 00:23:26 INFO Completed upload id=26 objects=3 duration=664.346µs48522026/11/12 00:23:26 INFO Received push request method=POST path=/48532026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48542026/11/12 00:23:26 INFO Completed upload id=28 objects=8 duration=500.945µs48552026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48562026/11/12 00:23:26 INFO Completed upload id=2 objects=7 duration=906.809µs48572026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48582026/11/12 00:23:26 INFO Completed upload id=27 objects=7 duration=822.388µs48592026/11/12 00:23:26 INFO Received uploads request method=POST path=/48602026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48612026/11/12 00:23:26 INFO Completed upload id=29 objects=10 duration=804.287µs48622026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48632026/11/12 00:23:26 INFO Completed upload id=28 objects=3 duration=796.307µs48642026/11/12 00:23:26 INFO Received complete push request method=POST path=/48652026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48662026/11/12 00:23:26 INFO Completed push id=13 objects=2 duration=1.12161ms48672026/11/12 00:23:26 INFO Completed upload id=30 objects=20 duration=733.586µs48682026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48692026/11/12 00:23:26 INFO Completed upload id=8 objects=6 duration=1.04759ms48702026/11/12 00:23:26 INFO Received complete push request method=POST path=/48712026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48722026/11/12 00:23:26 INFO Completed push id=31 objects=11 duration=964.029µs48732026/11/12 00:23:26 INFO Completed upload id=27 objects=13 duration=1.145571ms48742026/11/12 00:23:26 INFO Received complete push request method=POST path=/48752026/11/12 00:23:26 INFO Completed push id=29 objects=23 duration=1.297351ms48762026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48772026/11/12 00:23:26 INFO Completed upload id=32 objects=2 duration=857.128µs48782026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=048792026/11/12 00:23:26 INFO Received complete push request method=POST path=/48802026/11/12 00:23:26 INFO Completed push id=30 objects=6 duration=1.892518ms48812026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048822026/11/12 00:23:26 INFO Received complete push request method=POST path=/48832026/11/12 00:23:26 INFO Vacuumed table table=pending_closures48842026/11/12 00:23:26 INFO Completed push id=33 objects=3 duration=4.800684ms48852026/11/12 00:23:26 INFO Vacuumed table table=pending_objects48862026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads48872026/11/12 00:23:26 INFO Vacuumed table table=closures48882026/11/12 00:23:26 INFO Received complete upload request method=POST path=/48892026/11/12 00:23:26 INFO Vacuumed table table=objects48902026/11/12 00:23:26 INFO Completed upload id=34 objects=3 duration=4.362021ms48912026/11/12 00:23:26 INFO Received complete push request method=POST path=/48922026/11/12 00:23:26 INFO Received uploads request method=POST path=/48932026/11/12 00:23:26 INFO Completed push id=33 objects=3 duration=3.084708ms48942026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=048952026/11/12 00:23:26 WARN Force mode enabled - objects will be deleted immediately without grace period48962026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048972026/11/12 00:23:26 INFO Vacuumed table table=pending_closures48982026/11/12 00:23:26 INFO Vacuumed table table=pending_objects48992026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads49002026/11/12 00:23:26 INFO Vacuumed table table=closures49012026/11/12 00:23:26 INFO Vacuumed table table=objects49022026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49032026/11/12 00:23:26 INFO Completed upload id=12 objects=9 duration=1.10305ms49042026/11/12 00:23:26 INFO Received complete push request method=POST path=/49052026/11/12 00:23:26 INFO Completed push id=34 objects=6 duration=969.969µs49062026/11/12 00:23:26 INFO Received push request method=POST path=/49072026/11/12 00:23:26 INFO Received complete push request method=POST path=/49082026/11/12 00:23:26 INFO Completed push id=35 objects=8 duration=867.728µs49092026/11/12 00:23:26 INFO Received push request method=POST path=/49102026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49112026/11/12 00:23:26 INFO Received uploads request method=POST path=/49122026/11/12 00:23:26 INFO Completed upload id=36 objects=9 duration=807.407µs49132026/11/12 00:23:26 INFO Received uploads request method=POST path=/49142026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=049152026/11/12 00:23:26 WARN Force mode enabled - objects will be deleted immediately without grace period49162026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049172026/11/12 00:23:26 INFO Vacuumed table table=pending_closures49182026/11/12 00:23:26 INFO Vacuumed table table=pending_objects49192026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads49202026/11/12 00:23:26 INFO Vacuumed table table=closures49212026/11/12 00:23:26 INFO Vacuumed table table=objects49222026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49232026/11/12 00:23:26 INFO Completed upload id=29 objects=4 duration=1.533495ms49242026/11/12 00:23:26 INFO Received uploads request method=POST path=/49252026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49262026/11/12 00:23:26 INFO Completed upload id=18 objects=22 duration=1.194131ms49272026/11/12 00:23:26 INFO Received uploads request method=POST path=/49282026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49292026/11/12 00:23:26 INFO Completed upload id=21 objects=8 duration=760.247µs49302026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49312026/11/12 00:23:26 INFO Received complete push request method=POST path=/49322026/11/12 00:23:26 INFO Completed upload id=22 objects=7 duration=678.626µs49332026/11/12 00:23:26 INFO Completed push id=11 objects=6 duration=1.628695ms49342026/11/12 00:23:26 INFO Received complete push request method=POST path=/49352026/11/12 00:23:26 INFO Completed push id=24 objects=5 duration=940.949µs49362026/11/12 00:23:26 INFO Received complete push request method=POST path=/49372026/11/12 00:23:26 INFO Completed push id=20 objects=13 duration=1.264952ms49382026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49392026/11/12 00:23:26 INFO Received complete push request method=POST path=/49402026/11/12 00:23:26 INFO Completed upload id=25 objects=8 duration=909.928µs49412026/11/12 00:23:26 INFO Completed push id=22 objects=4 duration=777.207µs49422026/11/12 00:23:26 INFO Received complete push request method=POST path=/49432026/11/12 00:23:26 INFO Completed push id=26 objects=4 duration=966.849µs49442026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49452026/11/12 00:23:26 INFO Completed upload id=23 objects=7 duration=864.488µs49462026/11/12 00:23:26 INFO Received complete push request method=POST path=/49472026/11/12 00:23:26 INFO Completed push id=30 objects=2 duration=813.028µs49482026/11/12 00:23:26 INFO Received complete push request method=POST path=/49492026/11/12 00:23:26 INFO Completed push id=24 objects=5 duration=987.469µs49502026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49512026/11/12 00:23:26 INFO Received complete push request method=POST path=/49522026/11/12 00:23:26 INFO Completed upload id=25 objects=2 duration=555.365µs49532026/11/12 00:23:26 INFO Completed push id=32 objects=7 duration=890.969µs49542026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49552026/11/12 00:23:26 INFO Completed upload id=26 objects=20 duration=619.746µs49562026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49572026/11/12 00:23:26 INFO Completed upload id=33 objects=6 duration=530.645µs49582026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49592026/11/12 00:23:26 INFO Received complete push request method=POST path=/49602026/11/12 00:23:26 INFO Completed upload id=34 objects=3 duration=470.704µs49612026/11/12 00:23:26 INFO Completed push id=28 objects=7 duration=604.105µs49622026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=049632026/11/12 00:23:26 WARN Force mode enabled - objects will be deleted immediately without grace period49642026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049652026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49662026/11/12 00:23:26 INFO Vacuumed table table=pending_closures49672026/11/12 00:23:26 INFO Completed upload id=30 objects=6 duration=841.128µs49682026/11/12 00:23:26 INFO Vacuumed table table=pending_objects49692026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads49702026/11/12 00:23:26 INFO Vacuumed table table=closures49712026/11/12 00:23:26 INFO Vacuumed table table=objects49722026/11/12 00:23:26 INFO Received complete push request method=POST path=/49732026/11/12 00:23:26 INFO Completed push id=31 objects=5 duration=670.566µs49742026/11/12 00:23:26 INFO Received complete push request method=POST path=/49752026/11/12 00:23:26 INFO Completed push id=32 objects=26 duration=1.264152ms49762026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49772026/11/12 00:23:26 INFO Completed upload id=33 objects=2 duration=518.485µs49782026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49792026/11/12 00:23:26 INFO Completed upload id=34 objects=5 duration=486.984µs49802026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49812026/11/12 00:23:26 INFO Completed upload id=35 objects=11 duration=458.384µs49822026/11/12 00:23:26 INFO Received complete upload request method=POST path=/49832026/11/12 00:23:26 INFO Completed upload id=36 objects=3 duration=747.787µs49842026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=049852026/11/12 00:23:26 WARN Force mode enabled - objects will be deleted immediately without grace period49862026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049872026/11/12 00:23:26 INFO Vacuumed table table=pending_closures49882026/11/12 00:23:26 INFO Vacuumed table table=pending_objects49892026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads49902026/11/12 00:23:26 INFO Vacuumed table table=closures49912026/11/12 00:23:26 INFO Vacuumed table table=objects49922026/11/12 00:23:29 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049932026/11/12 00:23:29 INFO Vacuumed table table=pending_closures49942026/11/12 00:23:29 INFO Vacuumed table table=pending_objects49952026/11/12 00:23:29 INFO Vacuumed table table=multipart_uploads49962026/11/12 00:23:29 INFO Vacuumed table table=closures49972026/11/12 00:23:29 INFO Vacuumed table table=objects4998--- PASS: FuzzPushGC (0.01s)4999 --- PASS: FuzzPushGC/seed#7 (3.04s)5000 --- PASS: FuzzPushGC/seed#12 (3.05s)5001 --- PASS: FuzzPushGC/seed#6 (3.48s)5002 --- PASS: FuzzPushGC/seed#2 (3.50s)5003 --- PASS: FuzzPushGC/seed#0 (3.54s)5004 --- PASS: FuzzPushGC/seed#14 (3.55s)5005 --- PASS: FuzzPushGC/seed#15 (3.66s)5006 --- PASS: FuzzPushGC/seed#13 (3.66s)5007 --- PASS: FuzzPushGC/seed#1 (3.67s)5008 --- PASS: FuzzPushGC/seed#9 (3.68s)5009 --- PASS: FuzzPushGC/seed#3 (3.81s)5010 --- PASS: FuzzPushGC/seed#5 (4.09s)5011 --- PASS: FuzzPushGC/seed#4 (4.15s)5012 --- PASS: FuzzPushGC/seed#10 (4.34s)5013 --- PASS: FuzzPushGC/seed#11 (4.45s)5014 --- PASS: FuzzPushGC/seed#8 (7.65s)5015PASS5016{"timestamp":"2026-11-12T00:23:29.864547002Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:58434","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(209)"}5017{"timestamp":"2026-11-12T00:23:29.864551642Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:58574","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(208)"}50182026-11-12 00:23:30.343 UTC [132] LOG: received smart shutdown request50192026-11-12 00:23:30.349 UTC [132] LOG: background worker "logical replication launcher" (PID 142) exited with exit code 150202026-11-12 00:23:34.183 UTC [137] LOG: checkpoint complete: wrote 3777 buffers (23.1%), wrote 7 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=8.267 s, sync=0.466 s, total=11.930 s; sync files=42371, longest=0.003 s, average=0.001 s; distance=536370 kB, estimate=536370 kB; lsn=0/2710A628, redo lsn=0/223324F050212026-11-12 00:23:34.189 UTC [137] LOG: shutting down50222026-11-12 00:23:34.189 UTC [137] LOG: checkpoint starting: shutdown immediate50232026-11-12 00:23:34.747 UTC [137] LOG: checkpoint complete: wrote 7849 buffers (47.9%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.170 s, sync=0.379 s, total=0.559 s; sync files=2389, longest=0.010 s, average=0.001 s; distance=79712 kB, estimate=490704 kB; lsn=0/2710A6A0, redo lsn=0/2710A6A050242026-11-12 00:23:34.971 UTC [132] LOG: database system is shut down5025Running OIDC tests...5026=== RUN TestAudienceForIssuer5027=== PAUSE TestAudienceForIssuer5028=== RUN TestGlobMatch5029=== PAUSE TestGlobMatch5030=== RUN TestValidateToken_ValidToken5031=== PAUSE TestValidateToken_ValidToken5032=== RUN TestValidateToken_WrongAudience5033=== PAUSE TestValidateToken_WrongAudience5034=== RUN TestValidateToken_Expired5035=== PAUSE TestValidateToken_Expired5036=== RUN TestValidateToken_BoundClaimsMismatch5037=== PAUSE TestValidateToken_BoundClaimsMismatch5038=== RUN TestValidateToken_BoundSubjectMismatch5039=== PAUSE TestValidateToken_BoundSubjectMismatch5040=== RUN TestValidateToken_MultipleProviders5041=== PAUSE TestValidateToken_MultipleProviders5042=== RUN TestValidateToken_NoMatchingProvider5043=== PAUSE TestValidateToken_NoMatchingProvider5044=== RUN TestValidateToken_KubernetesServiceAccount5045=== PAUSE TestValidateToken_KubernetesServiceAccount5046=== RUN TestNewValidator_KubernetesRequiresCA5047=== PAUSE TestNewValidator_KubernetesRequiresCA5048=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5049=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5050=== RUN TestPins_ReservedForMatchingRule5051=== PAUSE TestPins_ReservedForMatchingRule5052=== RUN TestPins_TopLevelShorthand5053=== PAUSE TestPins_TopLevelShorthand5054=== RUN TestPins_ConfigValidation5055=== PAUSE TestPins_ConfigValidation5056=== RUN TestScopes_LegacyProviderDefaultsToWrite5057=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5058=== RUN TestScopes_Rules5059=== PAUSE TestScopes_Rules5060=== RUN TestScopes_ConfigValidation5061=== PAUSE TestScopes_ConfigValidation5062=== CONT TestAudienceForIssuer5063=== CONT TestValidateToken_KubernetesServiceAccount5064=== CONT TestPins_ConfigValidation5065--- PASS: TestAudienceForIssuer (0.00s)5066=== CONT TestValidateToken_Expired5067=== CONT TestValidateToken_WrongAudience5068=== CONT TestValidateToken_ValidToken5069=== CONT TestGlobMatch5070=== RUN TestGlobMatch/foo_foo5071=== PAUSE TestGlobMatch/foo_foo5072=== RUN TestGlobMatch/foo_bar5073=== PAUSE TestGlobMatch/foo_bar5074=== RUN TestGlobMatch/*_5075=== PAUSE TestGlobMatch/*_5076=== RUN TestGlobMatch/*_anything5077=== PAUSE TestGlobMatch/*_anything5078=== RUN TestGlobMatch/foo*_foo5079=== PAUSE TestGlobMatch/foo*_foo5080=== RUN TestGlobMatch/foo*_foobar5081=== PAUSE TestGlobMatch/foo*_foobar5082=== RUN TestGlobMatch/foo*_bar5083=== PAUSE TestGlobMatch/foo*_bar5084=== RUN TestGlobMatch/*bar_bar5085=== PAUSE TestGlobMatch/*bar_bar5086=== RUN TestGlobMatch/*bar_foobar5087=== PAUSE TestGlobMatch/*bar_foobar5088=== RUN TestGlobMatch/*bar_foo5089=== PAUSE TestGlobMatch/*bar_foo5090=== RUN TestGlobMatch/foo*bar_foobar5091=== PAUSE TestGlobMatch/foo*bar_foobar5092=== RUN TestGlobMatch/foo*bar_foo123bar5093=== PAUSE TestGlobMatch/foo*bar_foo123bar5094=== RUN TestGlobMatch/foo*bar_foobarbaz5095=== PAUSE TestGlobMatch/foo*bar_foobarbaz5096=== RUN TestGlobMatch/*/*_foo/bar5097=== PAUSE TestGlobMatch/*/*_foo/bar5098=== RUN TestGlobMatch/*/*_foo5099=== PAUSE TestGlobMatch/*/*_foo5100=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5101=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5102=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05103=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05104=== RUN TestGlobMatch/refs/*/main_refs/heads/main5105=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5106=== RUN TestGlobMatch/fo?_foo5107=== PAUSE TestGlobMatch/fo?_foo5108=== RUN TestGlobMatch/fo?_fo5109=== PAUSE TestGlobMatch/fo?_fo5110=== RUN TestGlobMatch/fo?_fooo5111=== PAUSE TestGlobMatch/fo?_fooo5112=== RUN TestGlobMatch/?oo_foo5113=== PAUSE TestGlobMatch/?oo_foo5114=== RUN TestGlobMatch/?oo_boo5115=== PAUSE TestGlobMatch/?oo_boo5116=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5117=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5118=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5119=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5120=== CONT TestGlobMatch/foo_foo5121=== CONT TestPins_ReservedForMatchingRule5122=== CONT TestPins_TopLevelShorthand5123=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5124=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5125=== CONT TestValidateToken_MultipleProviders5126=== CONT TestValidateToken_NoMatchingProvider5127=== CONT TestScopes_Rules5128=== CONT TestScopes_ConfigValidation5129=== CONT TestScopes_LegacyProviderDefaultsToWrite5130=== CONT TestValidateToken_BoundSubjectMismatch5131=== CONT TestNewValidator_KubernetesRequiresCA5132=== CONT TestValidateToken_BoundClaimsMismatch5133--- PASS: TestPins_ConfigValidation (0.00s)5134=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5135=== CONT TestGlobMatch/?oo_boo5136=== CONT TestGlobMatch/?oo_foo5137=== CONT TestGlobMatch/fo?_fooo5138=== CONT TestGlobMatch/fo?_fo5139=== CONT TestGlobMatch/fo?_foo5140=== CONT TestGlobMatch/refs/*/main_refs/heads/main5141=== CONT TestGlobMatch/*/*_foo5142=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5143=== CONT TestGlobMatch/*bar_foobar5144=== CONT TestGlobMatch/foo*bar_foobarbaz5145=== CONT TestGlobMatch/*/*_foo/bar5146=== CONT TestGlobMatch/foo*bar_foo123bar5147=== CONT TestGlobMatch/foo*bar_foobar5148=== CONT TestGlobMatch/*bar_foo5149=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05150--- PASS: TestScopes_ConfigValidation (0.00s)5151=== CONT TestGlobMatch/foo*_foo5152=== CONT TestGlobMatch/*bar_bar5153=== CONT TestGlobMatch/foo*_bar5154=== CONT TestGlobMatch/foo*_foobar5155=== CONT TestGlobMatch/*_anything5156=== CONT TestGlobMatch/*_5157=== CONT TestGlobMatch/foo_bar5158--- PASS: TestGlobMatch (0.01s)5159 --- PASS: TestGlobMatch/foo_foo (0.00s)5160 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5161 --- PASS: TestGlobMatch/?oo_boo (0.00s)5162 --- PASS: TestGlobMatch/?oo_foo (0.00s)5163 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5164 --- PASS: TestGlobMatch/fo?_fo (0.00s)5165 --- PASS: TestGlobMatch/fo?_foo (0.00s)5166 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5167 --- PASS: TestGlobMatch/*/*_foo (0.00s)5168 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5169 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5170 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5171 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5172 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5173 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5174 --- PASS: TestGlobMatch/*bar_foo (0.00s)5175 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5176 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5177 --- PASS: TestGlobMatch/foo*_foo (0.00s)5178 --- PASS: TestGlobMatch/*bar_bar (0.00s)5179 --- PASS: TestGlobMatch/foo*_bar (0.00s)5180 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5181 --- PASS: TestGlobMatch/*_anything (0.00s)5182 --- PASS: TestGlobMatch/*_ (0.00s)5183 --- PASS: TestGlobMatch/foo_bar (0.00s)51842026/11/12 00:23:39 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42261/oidc5185--- PASS: TestValidateToken_ValidToken (0.78s)51862026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39039/oidc5187--- PASS: TestPins_ReservedForMatchingRule (3.13s)51882026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41349/oidc5189--- PASS: TestPins_TopLevelShorthand (3.14s)51902026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40327/oidc5191--- PASS: TestValidateToken_Expired (3.23s)51922026/11/12 00:23:42 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12351932026/11/12 00:23:42 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:340555194--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (3.16s)5195--- PASS: TestValidateToken_KubernetesServiceAccount (3.26s)51962026/11/12 00:23:42 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:36469/oidc5197--- PASS: TestValidateToken_NoMatchingProvider (2.46s)51982026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46627/oidc51992026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41653/oidc5200--- PASS: TestScopes_Rules (2.49s)5201--- PASS: TestValidateToken_WrongAudience (3.28s)52022026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34995/oidc5203--- PASS: TestScopes_LegacyProviderDefaultsToWrite (2.52s)52042026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34535/oidc5205--- PASS: TestValidateToken_BoundSubjectMismatch (2.52s)52062026/11/12 00:23:42 http: TLS handshake error from 127.0.0.1:51376: remote error: tls: bad certificate5207--- PASS: TestNewValidator_KubernetesRequiresCA (2.54s)52082026/11/12 00:23:42 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44977/oidc5209--- PASS: TestValidateToken_BoundClaimsMismatch (2.59s)52102026/11/12 00:23:42 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:37439/oidc52112026/11/12 00:23:42 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:45289/oidc5212--- PASS: TestValidateToken_MultipleProviders (2.75s)5213PASS5214Running hook tests...5215=== RUN TestSendPathsEmpty5216=== PAUSE TestSendPathsEmpty5217=== RUN TestQueueEnqueueAndFetch5218=== PAUSE TestQueueEnqueueAndFetch5219=== RUN TestQueueDeduplication5220=== PAUSE TestQueueDeduplication5221=== RUN TestQueueRemove5222=== PAUSE TestQueueRemove5223=== RUN TestQueueFetchBatchLimit5224=== PAUSE TestQueueFetchBatchLimit5225=== RUN TestQueueRetryMovesToBack5226=== PAUSE TestQueueRetryMovesToBack5227=== RUN TestQueueFetchRemoveLifecycle5228=== PAUSE TestQueueFetchRemoveLifecycle5229=== RUN TestQueueConcurrentWriters5230=== PAUSE TestQueueConcurrentWriters5231=== RUN TestQueueRemoveLargeClosure5232=== PAUSE TestQueueRemoveLargeClosure5233=== RUN TestServerClientIntegration5234--- PASS: TestServerClientIntegration (0.00s)5235=== RUN TestServerQueueError52362026/11/12 00:23:44 ERROR Failed to queue paths error="permission denied" count=15237--- PASS: TestServerQueueError (0.00s)5238=== RUN TestGetListenerSocketActivation5239 server_test.go:220: === RUN TestGetListenerSocketActivation5240 --- PASS: TestGetListenerSocketActivation (0.00s)5241 PASS5242 5243--- PASS: TestGetListenerSocketActivation (0.12s)5244=== RUN TestServerStalledClientDoesNotBlockShutdown5245=== PAUSE TestServerStalledClientDoesNotBlockShutdown5246=== RUN TestServerBacksOffOnAcceptErrors5247=== PAUSE TestServerBacksOffOnAcceptErrors5248=== RUN TestDrainIsolatesPoisonPath5249=== PAUSE TestDrainIsolatesPoisonPath5250=== RUN TestRunNotBlockedByPoisonHead5251=== PAUSE TestRunNotBlockedByPoisonHead5252=== RUN TestDrainGivesUpWhenServerDown5253=== PAUSE TestDrainGivesUpWhenServerDown5254=== RUN TestFailedPathPrunedByLaterClosure5255=== PAUSE TestFailedPathPrunedByLaterClosure5256=== RUN TestWorkerUploadsAndRemoves5257=== PAUSE TestWorkerUploadsAndRemoves5258=== RUN TestWorkerSkipsGCdPaths5259=== PAUSE TestWorkerSkipsGCdPaths5260=== RUN TestWorkerPrunesClosureDeps5261=== PAUSE TestWorkerPrunesClosureDeps5262=== RUN TestDrainTimeout5263=== PAUSE TestDrainTimeout5264=== RUN TestWorkerRemoveFailureIsNotProgress5265=== PAUSE TestWorkerRemoveFailureIsNotProgress5266=== RUN TestWorkerKeepsPathItCannotStat5267=== PAUSE TestWorkerKeepsPathItCannotStat5268=== CONT TestSendPathsEmpty5269--- PASS: TestSendPathsEmpty (0.00s)5270=== CONT TestServerStalledClientDoesNotBlockShutdown5271=== CONT TestWorkerPrunesClosureDeps5272=== CONT TestQueueRemoveLargeClosure5273=== CONT TestWorkerKeepsPathItCannotStat5274=== CONT TestDrainTimeout5275=== CONT TestQueueConcurrentWriters5276=== CONT TestQueueFetchRemoveLifecycle5277=== CONT TestWorkerRemoveFailureIsNotProgress5278=== CONT TestQueueDeduplication5279=== CONT TestQueueFetchBatchLimit5280=== CONT TestQueueRetryMovesToBack5281=== CONT TestQueueRemove5282=== CONT TestQueueEnqueueAndFetch5283=== CONT TestDrainGivesUpWhenServerDown5284=== CONT TestWorkerSkipsGCdPaths5285=== CONT TestWorkerUploadsAndRemoves5286=== CONT TestFailedPathPrunedByLaterClosure5287=== CONT TestDrainIsolatesPoisonPath5288=== CONT TestRunNotBlockedByPoisonHead5289=== CONT TestServerBacksOffOnAcceptErrors52902000/01/01 00:00:00 ERROR Accept failed error="too many open files"52912000/01/01 00:00:00 ERROR Accept failed error="too many open files"52922000/01/01 00:00:00 ERROR Accept failed error="too many open files"52932000/01/01 00:00:00 ERROR Accept failed error="too many open files"52942000/01/01 00:00:00 ERROR Accept failed error="too many open files"52952000/01/01 00:00:00 ERROR Accept failed error="too many open files"52962000/01/01 00:00:00 ERROR Accept failed error="too many open files"52972000/01/01 00:00:00 ERROR Accept failed error="too many open files"5298--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5299--- PASS: TestQueueRetryMovesToBack (0.02s)53002026/11/12 00:23:44 ERROR Failed to decode request error="read unix /build/hook3380849182/test.sock->@: i/o timeout"53012026/11/12 00:23:44 ERROR Failed to write response error="write unix /build/hook3380849182/test.sock->@: i/o timeout"5302--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.29s)53032026/11/12 00:23:44 INFO Upload queue status pending=253042026/11/12 00:23:44 INFO Uploading batch count=15305--- PASS: TestQueueEnqueueAndFetch (0.30s)53062026/11/12 00:23:44 INFO Uploading batch count=253072026/11/12 00:23:44 ERROR Upload failed error="upload failed" count=253082026/11/12 00:23:44 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3080746048/002/a5309--- PASS: TestQueueRemove (0.30s)53102026/11/12 00:23:46 INFO Uploading batch count=153112026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=153122026/11/12 00:23:46 INFO Upload queue status pending=353132026/11/12 00:23:46 INFO Uploading batch count=153142026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=153152026/11/12 00:23:46 INFO Uploading batch count=153162026/11/12 00:23:46 INFO Upload queue status pending=253172026/11/12 00:23:46 INFO Uploading batch count=153182026/11/12 00:23:46 INFO Uploading batch count=45319--- PASS: TestQueueFetchBatchLimit (1.94s)53202026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=453212026/11/12 00:23:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath2649864695/002/bbb5322--- PASS: TestFailedPathPrunedByLaterClosure (1.94s)53232000/01/01 00:00:00 INFO Uploading batch count=253242000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253252026/11/12 00:23:46 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3882734887/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3882734887/002/locked/aaa: permission denied"53262026/11/12 00:23:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths136121750/002/nonexistent5327--- PASS: TestWorkerPrunesClosureDeps (1.95s)53282000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453292026/11/12 00:23:46 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3882734887/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3882734887/002/locked/aaa: permission denied"5330--- PASS: TestQueueFetchRemoveLifecycle (1.94s)53312026/11/12 00:23:46 INFO Uploading batch count=153322026/11/12 00:23:46 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3882734887/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3882734887/002/locked/aaa: permission denied"53332026/11/12 00:23:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3080746048/002/b5334--- PASS: TestQueueDeduplication (1.95s)53352026/11/12 00:23:46 INFO Uploading batch count=153362026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=153372026/11/12 00:23:46 INFO Uploading batch count=153382026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=153392026/11/12 00:23:46 INFO Uploading batch count=253402026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=253412026/11/12 00:23:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3080746048/002/c53422026/11/12 00:23:46 ERROR Drain finished with paths left in queue remaining=153432026/11/12 00:23:46 INFO Upload queue status pending=253442026/11/12 00:23:46 INFO Uploading batch count=153452026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=153462026/11/12 00:23:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3080746048/002/d53472026/11/12 00:23:46 INFO Uploading batch count=25348--- PASS: TestDrainTimeout (1.95s)53492026/11/12 00:23:46 INFO Uploading batch count=253502026/11/12 00:23:46 ERROR Drain finished with paths left in queue remaining=15351--- PASS: TestWorkerKeepsPathItCannotStat (1.96s)53522026/11/12 00:23:46 ERROR Upload failed error="upload failed" count=253532026/11/12 00:23:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3080746048/002/e53542026/11/12 00:23:46 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3080746048/002/f53552026/11/12 00:23:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress979015146/002/nonexistent53562026/11/12 00:23:46 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153572026/11/12 00:23:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress979015146/002/nonexistent53582026/11/12 00:23:46 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153592026/11/12 00:23:46 ERROR Drain finished with paths left in queue remaining=1053602026/11/12 00:23:46 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress979015146/002/nonexistent5361--- PASS: TestDrainIsolatesPoisonPath (1.95s)53622026/11/12 00:23:46 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15363--- PASS: TestDrainGivesUpWhenServerDown (1.95s)53642026/11/12 00:23:46 ERROR Drain finished with paths left in queue remaining=15365--- PASS: TestWorkerRemoveFailureIsNotProgress (1.96s)5366--- PASS: TestWorkerSkipsGCdPaths (1.96s)53672026/11/12 00:23:47 INFO Uploading batch count=153682026/11/12 00:23:47 INFO Uploading batch count=153692026/11/12 00:23:47 INFO Uploading batch count=153702026/11/12 00:23:47 ERROR Upload failed error="upload failed" count=153712026/11/12 00:23:48 INFO Uploading batch count=153722026/11/12 00:23:48 ERROR Upload failed error="upload failed" count=153732026/11/12 00:23:48 INFO Uploading batch count=153742026/11/12 00:23:48 ERROR Upload failed error="upload failed" count=153752026/11/12 00:23:48 INFO Uploading batch count=15376--- PASS: TestWorkerUploadsAndRemoves (4.71s)53772026/11/12 00:23:48 ERROR Upload failed error="upload failed" count=153782026/11/12 00:23:48 INFO Uploading batch count=153792026/11/12 00:23:48 ERROR Upload failed error="upload failed" count=153802026/11/12 00:23:48 ERROR Drain finished with paths left in queue remaining=15381--- PASS: TestRunNotBlockedByPoisonHead (4.71s)5382--- PASS: TestQueueRemoveLargeClosure (4.79s)5383--- PASS: TestQueueConcurrentWriters (4.81s)5384PASS5385Running signing tests...5386=== RUN TestExternalSigner5387=== PAUSE TestExternalSigner5388=== RUN TestExternalSignerWrongPublicKey5389=== PAUSE TestExternalSignerWrongPublicKey5390=== RUN TestExternalSignerErrors5391=== PAUSE TestExternalSignerErrors5392=== RUN TestExternalSignerRestartsAfterCrash5393=== PAUSE TestExternalSignerRestartsAfterCrash5394=== RUN TestExternalSignerProgramErrorKeepsProcess5395=== PAUSE TestExternalSignerProgramErrorKeepsProcess5396=== RUN TestExternalSignerClose5397=== PAUSE TestExternalSignerClose5398=== RUN TestExternalSignerRejectsInvalidPublicKey5399=== PAUSE TestExternalSignerRejectsInvalidPublicKey5400=== RUN TestExternalSignerCancellation5401=== PAUSE TestExternalSignerCancellation5402=== RUN TestGenerateFingerprint5403=== PAUSE TestGenerateFingerprint5404=== RUN TestParseSigningKey5405=== PAUSE TestParseSigningKey5406=== RUN TestSign5407=== PAUSE TestSign5408=== RUN TestSignBatch5409=== PAUSE TestSignBatch5410=== RUN TestNilKeyPublicKey5411=== PAUSE TestNilKeyPublicKey5412=== RUN TestSignErrors5413=== PAUSE TestSignErrors5414=== CONT TestExternalSigner5415=== CONT TestSignBatch5416=== CONT TestSign5417=== CONT TestParseSigningKey5418=== RUN TestParseSigningKey/valid_32-byte_key5419=== CONT TestSignErrors5420=== PAUSE TestParseSigningKey/valid_32-byte_key5421=== CONT TestExternalSignerCancellation5422=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5423=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5424=== CONT TestExternalSignerProgramErrorKeepsProcess5425=== CONT TestExternalSignerRestartsAfterCrash5426=== CONT TestExternalSignerRejectsInvalidPublicKey5427=== RUN TestExternalSignerRejectsInvalidPublicKey/#005428=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005429=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5430=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5431=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5432=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5433=== CONT TestNilKeyPublicKey5434--- PASS: TestNilKeyPublicKey (0.00s)5435=== CONT TestExternalSignerWrongPublicKey5436=== CONT TestExternalSignerClose5437=== CONT TestGenerateFingerprint5438=== RUN TestGenerateFingerprint/basic_with_references5439=== CONT TestExternalSignerErrors5440=== RUN TestParseSigningKey/no_colon5441=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5442=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5443=== PAUSE TestParseSigningKey/no_colon5444=== RUN TestParseSigningKey/empty_name5445=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5446=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5447=== RUN TestExternalSignerErrors/wrong_shape5448=== PAUSE TestExternalSignerErrors/wrong_shape5449=== PAUSE TestParseSigningKey/empty_name5450=== CONT TestExternalSignerRejectsInvalidPublicKey/#005451=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5452=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5453=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5454=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5455--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5456 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5457 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5458 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5459 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5460 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5461=== RUN TestExternalSignerErrors/missing_signatures5462=== PAUSE TestExternalSignerErrors/missing_signatures5463=== RUN TestParseSigningKey/invalid_base645464=== RUN TestExternalSignerErrors/null_signatures5465=== PAUSE TestParseSigningKey/invalid_base645466=== PAUSE TestExternalSignerErrors/null_signatures5467=== RUN TestParseSigningKey/wrong_length5468=== PAUSE TestParseSigningKey/wrong_length5469=== CONT TestParseSigningKey/valid_32-byte_key5470=== RUN TestExternalSignerErrors/wrong_signatures_shape5471=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5472=== CONT TestParseSigningKey/no_colon5473=== CONT TestParseSigningKey/invalid_base645474=== CONT TestParseSigningKey/wrong_length5475=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5476=== CONT TestParseSigningKey/empty_name5477=== RUN TestExternalSignerErrors/missing_signature_name5478=== PAUSE TestExternalSignerErrors/missing_signature_name5479=== RUN TestExternalSignerErrors/exit_at_startup5480=== PAUSE TestExternalSignerErrors/exit_at_startup5481=== RUN TestExternalSignerErrors/null5482=== PAUSE TestExternalSignerErrors/null5483=== RUN TestExternalSignerErrors/unrequested_key5484=== PAUSE TestExternalSignerErrors/unrequested_key5485=== RUN TestExternalSignerErrors/too_few_signatures5486=== PAUSE TestExternalSignerErrors/too_few_signatures5487=== RUN TestExternalSignerErrors/invalid_signature_base645488=== PAUSE TestExternalSignerErrors/invalid_signature_base645489=== RUN TestExternalSignerErrors/wrong_signature_length5490=== PAUSE TestExternalSignerErrors/wrong_signature_length5491=== RUN TestExternalSignerErrors/program_error5492=== PAUSE TestExternalSignerErrors/program_error5493=== RUN TestExternalSignerErrors/wrong_id5494=== PAUSE TestGenerateFingerprint/basic_with_references5495=== RUN TestGenerateFingerprint/no_references5496=== PAUSE TestGenerateFingerprint/no_references5497=== PAUSE TestExternalSignerErrors/wrong_id5498=== RUN TestExternalSignerErrors/wrong_signature_name5499=== PAUSE TestExternalSignerErrors/wrong_signature_name5500=== RUN TestExternalSignerErrors/exit_on_request5501=== PAUSE TestExternalSignerErrors/exit_on_request5502=== RUN TestExternalSignerErrors/invalid_JSON5503=== PAUSE TestExternalSignerErrors/invalid_JSON5504=== CONT TestExternalSignerErrors/wrong_shape5505--- PASS: TestParseSigningKey (0.00s)5506 --- PASS: TestParseSigningKey/no_colon (0.00s)5507 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5508 --- PASS: TestParseSigningKey/wrong_length (0.00s)5509 --- PASS: TestParseSigningKey/empty_name (0.00s)5510 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5511 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5512=== CONT TestExternalSignerErrors/exit_at_startup5513=== RUN TestSignErrors/nil_key5514=== PAUSE TestSignErrors/nil_key5515=== RUN TestSignErrors/nil_narinfo5516=== PAUSE TestSignErrors/nil_narinfo5517=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5518=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5519--- PASS: TestSign (0.00s)5520=== CONT TestExternalSignerErrors/invalid_signature_base645521--- PASS: TestSignBatch (0.00s)5522=== CONT TestExternalSignerErrors/invalid_JSON5523=== CONT TestExternalSignerErrors/exit_on_request5524=== CONT TestExternalSignerErrors/wrong_signature_name5525=== CONT TestExternalSignerErrors/wrong_id5526=== CONT TestExternalSignerErrors/program_error5527=== CONT TestExternalSignerErrors/wrong_signature_length5528=== CONT TestExternalSignerErrors/unrequested_key5529=== CONT TestExternalSignerErrors/too_few_signatures5530=== CONT TestExternalSignerErrors/null5531=== CONT TestExternalSignerErrors/wrong_signatures_shape5532=== CONT TestExternalSignerErrors/missing_signature_name5533=== CONT TestExternalSignerErrors/null_signatures5534=== RUN TestSignErrors/invalid_narinfo5535=== PAUSE TestSignErrors/invalid_narinfo5536=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5537=== CONT TestExternalSignerErrors/missing_signatures5538=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5539=== RUN TestSignErrors/canceled5540--- PASS: TestExternalSignerWrongPublicKey (0.00s)5541=== RUN TestGenerateFingerprint/invalid_nar_hash_length5542--- PASS: TestExternalSignerClose (0.00s)5543=== PAUSE TestSignErrors/canceled5544=== CONT TestSignErrors/nil_key5545=== CONT TestSignErrors/canceled5546=== CONT TestSignErrors/invalid_narinfo5547=== CONT TestSignErrors/nil_narinfo5548--- PASS: TestSignErrors (0.01s)5549 --- PASS: TestSignErrors/nil_key (0.00s)5550 --- PASS: TestSignErrors/canceled (0.00s)5551 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5552 --- PASS: TestSignErrors/nil_narinfo (0.00s)5553=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5554=== RUN TestGenerateFingerprint/invalid_store_path_prefix5555=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5556=== RUN TestGenerateFingerprint/invalid_reference_prefix5557=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5558=== CONT TestGenerateFingerprint/basic_with_references5559=== CONT TestGenerateFingerprint/invalid_nar_hash_length5560=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5561=== CONT TestGenerateFingerprint/no_references5562=== CONT TestGenerateFingerprint/invalid_store_path_prefix5563=== CONT TestGenerateFingerprint/invalid_reference_prefix5564--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5565=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5566--- PASS: TestGenerateFingerprint (0.01s)5567 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5568 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5569 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5570 --- PASS: TestGenerateFingerprint/no_references (0.00s)5571 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5572 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5573 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5574--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5575--- PASS: TestExternalSigner (0.02s)5576--- PASS: TestExternalSignerErrors (0.00s)5577 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5578 --- PASS: TestExternalSignerErrors/null (0.00s)5579 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5580 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5581 --- PASS: TestExternalSignerErrors/program_error (0.01s)5582 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5583 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5584 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5585 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5586 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5587 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5588 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5589 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5590 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5591 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5592 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5593--- PASS: TestExternalSignerCancellation (0.02s)5594PASS5595Running OpenBao signer tests...5596=== RUN TestSign5597=== PAUSE TestSign5598=== RUN TestSignErrors5599=== PAUSE TestSignErrors5600=== RUN TestServe5601=== PAUSE TestServe5602=== RUN TestServeRejectsGarbage5603=== PAUSE TestServeRejectsGarbage5604=== CONT TestSign5605=== CONT TestSignErrors5606=== CONT TestServe5607=== RUN TestSignErrors/http_error5608=== PAUSE TestSignErrors/http_error5609=== RUN TestSignErrors/invalid_json5610=== PAUSE TestSignErrors/invalid_json5611=== RUN TestSignErrors/no_prefix5612=== PAUSE TestSignErrors/no_prefix5613=== RUN TestSignErrors/invalid_base645614=== PAUSE TestSignErrors/invalid_base645615=== CONT TestSignErrors/http_error5616=== CONT TestSignErrors/invalid_base645617=== CONT TestSignErrors/no_prefix5618=== CONT TestSignErrors/invalid_json5619=== CONT TestServeRejectsGarbage5620--- PASS: TestServeRejectsGarbage (0.00s)5621--- PASS: TestServe (0.00s)5622--- PASS: TestSign (3.53s)5623--- PASS: TestSignErrors (0.00s)5624 --- PASS: TestSignErrors/no_prefix (3.53s)5625 --- PASS: TestSignErrors/invalid_json (3.53s)5626 --- PASS: TestSignErrors/http_error (3.53s)5627 --- PASS: TestSignErrors/invalid_base64 (3.53s)5628PASS