niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #498
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (2.19s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushAnswersEveryLineWhenCanceled86=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestSetClientTLSErrors116=== CONT TestDoServerRequestAttachesToken117=== CONT TestSetClientTLSDoesNotMutateDefaultTransport118=== CONT TestSetClientTLS119=== CONT TestStreamPushStopsWhenCanceled120--- PASS: TestStreamPushStopsWhenCanceled (0.00s)121=== CONT TestRateLimiterFeedback122=== RUN TestRateLimiterFeedback/429_enables_limiter123=== PAUSE TestRateLimiterFeedback/429_enables_limiter124=== RUN TestRateLimiterFeedback/503_enables_limiter125=== PAUSE TestRateLimiterFeedback/503_enables_limiter126=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter127=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter128=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter129=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter130=== CONT TestStreamPushAnswersEveryLineWhenCanceled131=== CONT TestPathInfoCACompatibility132=== RUN TestPathInfoCACompatibility/null_ca_field133=== PAUSE TestPathInfoCACompatibility/null_ca_field134=== RUN TestPathInfoCACompatibility/old_string_format_-_text135=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text136=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive137=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive138=== RUN TestPathInfoCACompatibility/new_structured_format_-_text139=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text140=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method141=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method142=== CONT TestClientSignaturesByStorePath143--- PASS: TestClientSignaturesByStorePath (0.00s)144=== CONT TestParsePathInfoJSON145=== RUN TestParsePathInfoJSON/Nix_format146=== PAUSE TestParsePathInfoJSON/Nix_format147=== RUN TestParsePathInfoJSON/Lix_format148=== PAUSE TestParsePathInfoJSON/Lix_format149=== RUN TestParsePathInfoJSON/empty_input150=== PAUSE TestParsePathInfoJSON/empty_input151=== RUN TestParsePathInfoJSON/whitespace_only152=== PAUSE TestParsePathInfoJSON/whitespace_only153=== RUN TestParsePathInfoJSON/invalid_JSON154=== PAUSE TestParsePathInfoJSON/invalid_JSON155=== CONT TestParsePathInfoJSONMultiplePaths156=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths157=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths158=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths159=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths160=== CONT TestStreamPushReportsSkippedPaths161=== CONT TestPathInfoHashCompatibility162=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)163=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)164=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon165=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon166=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI167=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI168=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512169=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512170=== CONT TestGetStorePathHash171=== RUN TestGetStorePathHash/valid_store_path172=== PAUSE TestGetStorePathHash/valid_store_path173=== RUN TestGetStorePathHash/basename_without_hyphen_should_error174=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error175=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error176=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error177=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error178=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error179=== CONT TestStreamPushReportsSignatures180--- PASS: TestStreamPushReportsSkippedPaths (0.00s)181=== CONT TestEncodeNixBase32182=== RUN TestEncodeNixBase32/test_string_hash183=== PAUSE TestEncodeNixBase32/test_string_hash184=== RUN TestEncodeNixBase32/empty_input185=== PAUSE TestEncodeNixBase32/empty_input186=== CONT TestConvertHashToNix32187=== RUN TestConvertHashToNix32/SRI_format_to_Nix32188=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32189=== RUN TestConvertHashToNix32/already_Nix32_format190=== PAUSE TestConvertHashToNix32/already_Nix32_format191=== RUN TestConvertHashToNix32/invalid_format192=== PAUSE TestConvertHashToNix32/invalid_format1932000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2194=== CONT TestEncodeNixBase32WithRealHash195--- PASS: TestEncodeNixBase32WithRealHash (0.00s)196=== CONT TestDumpPathMatchesNix197=== RUN TestSetClientTLSErrors/missing_cert_file198=== PAUSE TestSetClientTLSErrors/missing_cert_file199=== RUN TestSetClientTLSErrors/missing_key_file200=== PAUSE TestSetClientTLSErrors/missing_key_file201=== RUN TestSetClientTLSErrors/missing_ca_file202=== PAUSE TestSetClientTLSErrors/missing_ca_file203=== RUN TestSetClientTLSErrors/invalid_ca_file204=== PAUSE TestSetClientTLSErrors/invalid_ca_file205=== CONT TestPartSizeForNAR206=== CONT TestStreamPushRequestLine207=== CONT TestStreamPushGivesUpOnDeadServer208=== CONT TestStreamPushIsolatesFailures209=== CONT TestStreamPushBatchesUnderLoad210=== CONT TestStreamPushReportsEveryPath211=== CONT TestShellSplitErrors212=== CONT TestShellSplit213=== CONT TestDoWithRetry_FinalResponseBodyReadable2142026/11/12 00:25:55 ERROR Upload failed error="bad path" count=3215=== CONT TestRetryTimeoutOutlastsMaxRetries216=== CONT TestDoWithRetry_BodyReplayedViaGetBody217=== CONT TestResolveStorePath218=== CONT TestRegisterUploadedObject_OutlivesPush219=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer220=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess221=== CONT TestScriptTokenCachesUntilRefresh222=== CONT TestDumpPathWriterError223=== CONT TestDumpPathSingleFile224=== CONT TestUploadMultipart_ReturnsAfterDroppedBody225--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (2.34s)226--- PASS: TestStreamPushIsolatesFailures (0.00s)227=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false228=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false229=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true230=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true231--- PASS: TestShellSplitErrors (0.00s)232=== CONT TestScriptTokenNoExpiryRerunsEveryCall233--- PASS: TestShellSplit (0.00s)234=== CONT TestFilterOversizedClosures2352026/11/12 00:25:55 ERROR Upload failed error="connection refused" count=112362026/11/12 00:25:55 ERROR Server seems unavailable, giving up on batch untried=8237=== RUN TestFilterOversizedClosures/no_limit_keeps_everything238=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything239=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped240=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped241=== RUN TestFilterOversizedClosures/all_closures_skipped2422026/11/12 00:25:55 ERROR Upload failed error="connection refused" count=92432026/11/12 00:25:55 ERROR Server seems unavailable, giving up on batch untried=6244=== PAUSE TestFilterOversizedClosures/all_closures_skipped2452026/11/12 00:25:55 ERROR Upload failed error=boom count=1246=== CONT TestFileTokenEmpty2472026/11/12 00:25:55 WARN Rate limiter enabled after throttle name=server-test rate=5248--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)249--- PASS: TestStreamPushReportsSignatures (2.34s)250=== CONT TestCaseHackSuffix2512026/11/12 00:25:55 ERROR Upload failed error=boom count=1252=== RUN TestPartSizeForNAR/zero_stays_at_minimum253=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum254=== RUN TestPartSizeForNAR/small_stays_at_minimum255=== PAUSE TestPartSizeForNAR/small_stays_at_minimum256=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum257=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum258=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts259=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts260=== CONT TestUploadMultipart_PartsInParallel261=== RUN TestPartSizeForNAR/1_TiB262=== PAUSE TestPartSizeForNAR/1_TiB263=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica264=== RUN TestPartSizeForNAR/5_TiB_S3_max_object265=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object266=== RUN TestPartSizeForNAR/capped_at_5_GiB267=== PAUSE TestPartSizeForNAR/capped_at_5_GiB268=== CONT TestScriptTokenEmptyCommand269=== CONT TestRegisterUploadedObjectReusesConnections2702026/11/12 00:25:58 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38041/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38041/api/uploads/complete2712026/11/12 00:25:58 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2722026/11/12 00:25:58 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38041/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38041/api/uploads/complete2732026/11/12 00:25:58 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2742026/11/12 00:25:58 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38041/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38041/api/uploads/complete2752026/11/12 00:25:58 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"2762026/11/12 00:25:58 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:34345277=== CONT TestScriptTokenBadJSON278=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry279=== CONT TestFileTokenReadsAndCaches2802026/11/12 00:25:58 WARN Rate limiter enabled after throttle name=server-test rate=5281=== CONT TestFileTokenMissing2822026/11/12 00:25:58 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:41853283=== CONT TestScriptTokenEmptyToken284--- PASS: TestResolveStorePath (0.00s)285--- PASS: TestFileTokenEmpty (0.00s)286--- PASS: TestScriptTokenEmptyCommand (0.00s)287--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (4.67s)2882026/11/12 00:25:58 WARN Rate limiter enabled after throttle name=server-test rate=5289--- PASS: TestStreamPushReportsEveryPath (2.33s)290--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (2.33s)291--- PASS: TestScriptTokenBadJSON (0.01s)2922026/11/12 00:25:58 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:38353293--- PASS: TestFileTokenReadsAndCaches (0.00s)294=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun295=== CONT TestRateLimiterFeedback/429_enables_limiter2962026/11/12 00:25:58 WARN Rate limiter backed off name=server-test rate=52972026/11/12 00:25:58 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:41853298=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter299--- PASS: TestRegisterUploadedObject_OutlivesPush (2.33s)300--- PASS: TestFileTokenMissing (0.00s)301--- PASS: TestDoServerRequestAttachesToken (4.67s)302=== CONT TestScriptTokenScriptFails303--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (2.33s)304=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter3052026/11/12 00:26:00 INFO Garbage collection started306--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (4.89s)307=== CONT TestRateLimiterFeedback/503_enables_limiter3082026/11/12 00:26:00 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0309=== CONT TestStaticToken310--- PASS: TestStaticToken (0.00s)311=== CONT TestParsePathInfoJSON/Nix_format3122026/11/12 00:26:00 WARN Rate limiter backed off name=server-test rate=5313--- PASS: TestDoWithRetry_FinalResponseBodyReadable (4.89s)314=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths315--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (4.89s)316=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method317--- PASS: TestScriptTokenEmptyToken (2.56s)318=== CONT TestPathInfoCACompatibility/new_structured_format_-_text319=== CONT TestPathInfoCACompatibility/old_string_format_-_text320=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)321=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive322=== CONT TestParsePathInfoJSON/whitespace_only323=== CONT TestParsePathInfoJSON/empty_input324=== CONT TestParsePathInfoJSON/Lix_format325=== CONT TestPathInfoCACompatibility/null_ca_field326--- PASS: TestPathInfoCACompatibility (0.00s)327 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)328 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)329 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)330 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)331 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)332=== CONT TestGetStorePathHash/valid_store_path333=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths334=== CONT TestParsePathInfoJSON/invalid_JSON335=== CONT TestEncodeNixBase32/test_string_hash336=== CONT TestConvertHashToNix32/SRI_format_to_Nix32337=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512338=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI339=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon340=== CONT TestSetClientTLSErrors/missing_cert_file341=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error342=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error343=== CONT TestGetStorePathHash/basename_without_hyphen_should_error344=== CONT TestEncodeNixBase32/empty_input345=== CONT TestConvertHashToNix32/invalid_format346=== CONT TestConvertHashToNix32/already_Nix32_format347=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false348=== RUN TestSetClientTLS/rejects_connection_without_client_cert349=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert350=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA351=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA352=== RUN TestSetClientTLS/preserves_debug_logging_transport353=== PAUSE TestSetClientTLS/preserves_debug_logging_transport354--- PASS: TestPathInfoHashCompatibility (0.00s)355 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)356 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)357 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)358 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)359=== CONT TestFilterOversizedClosures/no_limit_keeps_everything360=== CONT TestSetClientTLSErrors/invalid_ca_file361--- PASS: TestScriptTokenScriptFails (2.57s)3622026/11/12 00:26:00 WARN Garbage collection finished on another replica. Its statistics are not available here363--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)364 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)365 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)3662026/11/12 00:26:00 WARN Rate limiter enabled after throttle name=server-test rate=5367=== CONT TestPartSizeForNAR/capped_at_5_GiB3682026/11/12 00:26:00 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:46849369=== CONT TestPartSizeForNAR/1_TiB370=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum3712026/11/12 00:26:00 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:34345372=== CONT TestPartSizeForNAR/small_stays_at_minimum373=== CONT TestFilterOversizedClosures/all_closures_skipped3742026/11/12 00:26:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=503752026/11/12 00:26:00 WARN Rate limiter backed off name=server-test rate=5376=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3772026/11/12 00:26:00 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000378=== CONT TestSetClientTLS/rejects_connection_without_client_cert379=== CONT TestPartSizeForNAR/zero_stays_at_minimum380=== CONT TestSetClientTLS/preserves_debug_logging_transport3812026/11/12 00:26:00 WARN Rate limiter enabled after throttle name=server-test rate=53822026/11/12 00:26:01 WARN Rate limiter enabled after throttle name=s3-test rate=53832026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:462513842026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:34345385=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts386=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA387--- PASS: TestGetStorePathHash (0.00s)388 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)389 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)390 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)391 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)392--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (4.90s)393--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (5.12s)394=== CONT TestSetClientTLSErrors/missing_ca_file395=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true396=== CONT TestSetClientTLSErrors/missing_key_file397=== CONT TestPartSizeForNAR/5_TiB_S3_max_object398--- PASS: TestEncodeNixBase32 (0.00s)399 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)400 --- PASS: TestEncodeNixBase32/empty_input (0.00s)401--- PASS: TestFilterOversizedClosures (0.00s)402 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)403 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)404 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)405--- PASS: TestParsePathInfoJSON (0.00s)406 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)407 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)408 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)409 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)410 --- PASS: TestParsePathInfoJSON/Nix_format (0.01s)411--- PASS: TestConvertHashToNix32 (0.00s)412 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)413 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)414 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)415--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (2.79s)416--- PASS: TestPartSizeForNAR (0.00s)417 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)418 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)419 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)420 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)421 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)422 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)423 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)4242026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:34345425--- PASS: TestStreamPushBatchesUnderLoad (0.00s)426 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.22s)427 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)428--- PASS: TestSetClientTLSErrors (0.01s)429 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)430 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)431 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)432 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)4332026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:343454342026/11/12 00:26:01 INFO Garbage collection started4352026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:449694362026/11/12 00:26:01 WARN Rate limiter backed off name=server-test rate=5437--- PASS: TestRateLimiterFeedback (0.00s)438 --- PASS: TestRateLimiterFeedback/429_enables_limiter (2.57s)439 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (2.79s)440 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (2.79s)441 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.23s)4422026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:34345443--- PASS: TestScriptTokenCachesUntilRefresh (5.12s)444--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (2.79s)4452026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:34345446--- PASS: TestDumpPathSingleFile (5.12s)4472026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:34345448--- PASS: TestCaseHackSuffix (5.12s)4492026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:343454502026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:343454512026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:343454522026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:343454532026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:343454542026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:34345455--- PASS: TestRegisterUploadedObjectReusesConnections (5.13s)4562026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:34345457--- PASS: TestStreamPushRequestLine (5.14s)4582026/11/12 00:26:01 http: TLS handshake error from 127.0.0.1:48740: remote error: tls: bad certificate459--- PASS: TestSetClientTLS (7.24s)460 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)461 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.23s)462 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.24s)4632026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:343454642026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:343454652026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:34345466--- PASS: TestDumpPathWriterError (5.14s)4672026/11/12 00:26:01 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:34345468--- PASS: TestRetryTimeoutOutlastsMaxRetries (5.14s)469--- PASS: TestUploadMultipart_PartsInParallel (5.74s)470--- PASS: TestDumpPathMatchesNix (9.44s)471PASS472Running server tests...473The files belonging to this database system will be owned by user "nixbld".474This user must also own the server process.475476The database cluster will be initialized with locale "C".477The default database encoding has accordingly been set to "SQL_ASCII".478The default text search configuration will be set to "english".479480Data page checksums are enabled.481482creating directory /build/postgres1021783279/data ... ok483creating subdirectories ... ok484selecting dynamic shared memory implementation ... posix485selecting default "max_connections" ... 100486selecting default "shared_buffers" ... 128MB487selecting default time zone ... UTC488creating configuration files ... ok489running bootstrap script ... ok490performing post-bootstrap initialization ... ok491syncing data to disk ... ok492493initdb: warning: enabling "trust" authentication for local connections494initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.495496Success. You can now start the database server using:497498 pg_ctl -D /build/postgres1021783279/data -l logfile start499500/build/postgres1021783279:5432 - no response501/build/postgres1021783279:5432 - no response502/build/postgres1021783279:5432 - no response503/build/postgres1021783279:5432 - no response504/build/postgres1021783279:5432 - no response505/build/postgres1021783279:5432 - no response506/build/postgres1021783279:5432 - no response507/build/postgres1021783279:5432 - no response508/build/postgres1021783279:5432 - no response5092026-11-12 00:26:09.682 UTC [131] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5102026-11-12 00:26:09.682 UTC [131] LOG: listening on Unix socket "/build/postgres1021783279/.s.PGSQL.5432"5112026-11-12 00:26:09.689 UTC [146] LOG: database system was shut down at 2026-11-12 00:26:08 UTC5122026-11-12 00:26:09.695 UTC [131] LOG: database system is ready to accept connections513/build/postgres1021783279:5432 - accepting connections514{"timestamp":"2026-11-12T00:26:30.892434982Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"58909d41-db16-4750-b49a-f02b31ca0b4f","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":6,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(199)"}515{"timestamp":"2026-11-12T00:26:31.002078403Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"27b07844-e67b-49c6-ae92-677dc80534bd","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(204)"}516=== RUN TestAbortSucceeded517=== PAUSE TestAbortSucceeded518=== RUN TestService_AuthMiddleware519=== PAUSE TestService_AuthMiddleware520=== RUN TestService_AuthMiddleware_MTLSProxyHeader521=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader522=== RUN TestService_AuthMiddleware_MTLSBoundSubjects523=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects524=== RUN TestService_ReadAuthMiddleware525=== PAUSE TestService_ReadAuthMiddleware526=== RUN TestService_AuthMiddleware_OIDC527=== PAUSE TestService_AuthMiddleware_OIDC528=== RUN TestService_RequireScope_OIDC529=== PAUSE TestService_RequireScope_OIDC530=== RUN TestService_ReadScope_PublicByDefault531=== PAUSE TestService_ReadScope_PublicByDefault532=== RUN TestCacheConfigHandler533=== PAUSE TestCacheConfigHandler534=== RUN TestCacheStatsHandler535=== PAUSE TestCacheStatsHandler536=== RUN TestClientCADerivations537=== PAUSE TestClientCADerivations538=== RUN TestClientErrorHandling539=== PAUSE TestClientErrorHandling540=== RUN TestClientIntegration541=== PAUSE TestClientIntegration542=== RUN TestClientMultipleUploads543=== PAUSE TestClientMultipleUploads544=== RUN TestClientWithDependencies545=== PAUSE TestClientWithDependencies546=== RUN TestClientSharedPathCommittedMidPush547=== PAUSE TestClientSharedPathCommittedMidPush548=== RUN TestPinProtectsFromGC549=== PAUSE TestPinProtectsFromGC550=== RUN TestClientPushesUseOnePush551=== PAUSE TestClientPushesUseOnePush552=== RUN TestClientFallsBackToClosures553=== PAUSE TestClientFallsBackToClosures554=== RUN TestDeleteClosuresRacingPins555=== PAUSE TestDeleteClosuresRacingPins556=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock557=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock558=== RUN TestResolveDBConnectionString559=== PAUSE TestResolveDBConnectionString560=== RUN TestConnectWaitsForAPeerMigration561=== PAUSE TestConnectWaitsForAPeerMigration562=== RUN TestLeadElectsOneAndHandsOver563=== PAUSE TestLeadElectsOneAndHandsOver564=== RUN TestLeadIncumbentWinsAfterRestart5652026/11/12 00:26:33 INFO lead: acquired remote=192.0.2.1:1234 generation=15662026/11/12 00:26:33 INFO lead: released remote=192.0.2.1:1234 generation=15672026/11/12 00:26:34 INFO lead: acquired remote=192.0.2.1:1234 generation=25682026/11/12 00:26:35 INFO lead: released remote=192.0.2.1:1234 generation=2569--- PASS: TestLeadIncumbentWinsAfterRestart (4.01s)570=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent571=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent572=== RUN TestLeadIncumbentAnnouncesAtOnce573=== PAUSE TestLeadIncumbentAnnouncesAtOnce574=== RUN TestLeadGenerationGrowsWithEveryGrant575=== PAUSE TestLeadGenerationGrowsWithEveryGrant576=== RUN TestLeadEndsOnShutdown577=== PAUSE TestLeadEndsOnShutdown578=== RUN TestLeadEndsWhenItsConnectionHangs5792026/11/12 00:26:35 INFO lead: acquired remote=192.0.2.1:1234 generation=15802026-11-12 00:26:37.183 UTC [399] FATAL: terminating connection due to administrator command5812026/11/12 00:26:37 INFO lead: acquired remote=192.0.2.1:1234 generation=25822026/11/12 00:26:38 WARN lead: lock connection lost error="timeout: context deadline exceeded"5832026/11/12 00:26:38 INFO lead: released remote=192.0.2.1:1234 generation=15842026/11/12 00:26:38 INFO lead: released remote=192.0.2.1:1234 generation=2585--- PASS: TestLeadEndsWhenItsConnectionHangs (3.65s)586=== RUN TestGCAdvisoryLockBlocksConcurrentRun587=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun588=== RUN TestGCBugBareHashReferences589=== PAUSE TestGCBugBareHashReferences590=== RUN TestGCMetrics591=== PAUSE TestGCMetrics592=== RUN TestPushDedupSurvivesConcurrentGC593=== PAUSE TestPushDedupSurvivesConcurrentGC594=== RUN TestDeduplicatedObjectsRecordedAsPending595=== PAUSE TestDeduplicatedObjectsRecordedAsPending596=== RUN TestGCSweepSkipsPendingObjects597=== PAUSE TestGCSweepSkipsPendingObjects598=== RUN TestTombstonedObjectOfferedWithoutWaiting599=== PAUSE TestTombstonedObjectOfferedWithoutWaiting600=== RUN TestGCSweepDeliversEachKeyOnce601=== PAUSE TestGCSweepDeliversEachKeyOnce602=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection603=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection604=== RUN TestForceGCDuringPushOffersSweptObject605=== PAUSE TestForceGCDuringPushOffersSweptObject606=== RUN TestFailedS3DeleteKeepsTombstone607=== PAUSE TestFailedS3DeleteKeepsTombstone608=== RUN TestSweepSkipsRowsLockedByPush609=== PAUSE TestSweepSkipsRowsLockedByPush610=== RUN TestRegisterCompletedObjectNeedsPendingRow611=== PAUSE TestRegisterCompletedObjectNeedsPendingRow612=== RUN TestGetExistingObjectsLocksRows613=== PAUSE TestGetExistingObjectsLocksRows614=== RUN TestSweepRechecksPendingAfterLocking615=== PAUSE TestSweepRechecksPendingAfterLocking616=== RUN TestPushWaitsForSweepHoldingRow617=== PAUSE TestPushWaitsForSweepHoldingRow618=== RUN TestCommitRacingPendingCleanupKeepsObjects619=== PAUSE TestCommitRacingPendingCleanupKeepsObjects620=== RUN TestGCEndsOnShutdown621=== PAUSE TestGCEndsOnShutdown622=== RUN TestGCTaskStore_StartNew623=== PAUSE TestGCTaskStore_StartNew624=== RUN TestGCTaskStore_DeduplicateSameParams625=== PAUSE TestGCTaskStore_DeduplicateSameParams626=== RUN TestGCTaskStore_ConflictDifferentParams627=== PAUSE TestGCTaskStore_ConflictDifferentParams628=== RUN TestGCTaskStore_GetEmpty629=== PAUSE TestGCTaskStore_GetEmpty630=== RUN TestGCTaskStore_GetReturnsLatest631=== PAUSE TestGCTaskStore_GetReturnsLatest632=== RUN TestGCTaskStore_CompletedAllowsNewTask633=== PAUSE TestGCTaskStore_CompletedAllowsNewTask634=== RUN TestGCTaskStore_PhaseUpdates635=== PAUSE TestGCTaskStore_PhaseUpdates636=== RUN TestGCTaskStore_Fail637=== PAUSE TestGCTaskStore_Fail638=== RUN TestGracefulShutdownDrainsInflight639=== PAUSE TestGracefulShutdownDrainsInflight640=== RUN TestService_healthCheckHandler641=== PAUSE TestService_healthCheckHandler642=== RUN TestService_readinessHandler643=== PAUSE TestService_readinessHandler644=== RUN TestGenerateLandingPage645=== PAUSE TestGenerateLandingPage646=== RUN TestEnsureAbortUploadsRule647=== PAUSE TestEnsureAbortUploadsRule648=== RUN TestCacheConfigHandlerMaxNarSize649=== PAUSE TestCacheConfigHandlerMaxNarSize650=== RUN TestCreatePendingClosureRejectsOversizedNAR651=== PAUSE TestCreatePendingClosureRejectsOversizedNAR652=== RUN TestNARDeduplicationMetadataUploadBug653=== PAUSE TestNARDeduplicationMetadataUploadBug654=== RUN TestMetricsInventory655=== PAUSE TestMetricsInventory656=== RUN TestMetricsCommit657=== PAUSE TestMetricsCommit658=== RUN TestService_NativeMTLS659=== PAUSE TestService_NativeMTLS660=== RUN TestServerTLSConfig661=== PAUSE TestServerTLSConfig662=== RUN TestMultipartCleanup663=== PAUSE TestMultipartCleanup664=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded665=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded666=== RUN TestPendingCleanupUsesOneCutoff667=== PAUSE TestPendingCleanupUsesOneCutoff668=== RUN TestObjectStatsTrigger669=== PAUSE TestObjectStatsTrigger670=== RUN TestObjectStatsUpdatedOncePerCommit671=== PAUSE TestObjectStatsUpdatedOncePerCommit672=== RUN TestOrphanedObjectsGC673=== PAUSE TestOrphanedObjectsGC674=== RUN TestOrphanedObjectsGCStressTest675=== PAUSE TestOrphanedObjectsGCStressTest676=== RUN TestCreatePin_ReservedPins677=== PAUSE TestCreatePin_ReservedPins678=== RUN TestPresentReportsOnlyClosureRoots679=== PAUSE TestPresentReportsOnlyClosureRoots680=== RUN TestPresentRefreshesOnlyStaleClosures681=== PAUSE TestPresentRefreshesOnlyStaleClosures682=== RUN TestPresentNotReportedWhileGCDeletesClosure683=== PAUSE TestPresentNotReportedWhileGCDeletesClosure684=== RUN TestParseSingleRange685=== PAUSE TestParseSingleRange686=== RUN TestProxyHeadersOnlyTrustedOnSocket6872026/11/12 00:26:39 INFO Starting HTTP server address=127.0.0.1:342716882026/11/12 00:26:39 INFO Starting HTTP server address=proxy.sock6892026/11/12 00:26:39 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6902026/11/12 00:26:39 INFO Shutdown signal received, draining in-flight requests timeout=10s691--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.56s)692=== RUN TestIsValidCachePath693=== PAUSE TestIsValidCachePath694=== RUN TestReadProxyNarinfo695=== PAUSE TestReadProxyNarinfo696=== RUN TestReadProxyNarinfoAlreadyDecompressed697=== PAUSE TestReadProxyNarinfoAlreadyDecompressed698=== RUN TestReadProxyContentNegotiation699=== PAUSE TestReadProxyContentNegotiation700=== RUN TestReadProxyNarStreaming701=== PAUSE TestReadProxyNarStreaming702=== RUN TestReadProxy404703=== PAUSE TestReadProxy404704=== RUN TestReadProxyInvalidPath705=== PAUSE TestReadProxyInvalidPath706=== RUN TestReadProxyHead707=== PAUSE TestReadProxyHead708=== RUN TestReadProxyOutlastsServerWriteTimeout709=== PAUSE TestReadProxyOutlastsServerWriteTimeout710=== RUN TestReadProxyConditionalGet711=== PAUSE TestReadProxyConditionalGet712=== RUN TestReadProxyRootRedirectsToIndexHTML713=== PAUSE TestReadProxyRootRedirectsToIndexHTML714=== RUN TestReadProxyDisabled715=== PAUSE TestReadProxyDisabled716=== RUN TestReadRedirectNar717=== PAUSE TestReadRedirectNar718=== RUN TestReadRedirectKeepsNarinfoProxied719=== PAUSE TestReadRedirectKeepsNarinfoProxied720=== RUN TestReadProxyRangeRequest721=== PAUSE TestReadProxyRangeRequest722=== RUN TestReadRedirectUsesPublicS3URL723=== PAUSE TestReadRedirectUsesPublicS3URL724=== RUN TestPush_OverlappingRootsStoreOneRowPerKey725=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey726=== RUN TestPush_CompleteCommitsEveryRoot727=== PAUSE TestPush_CompleteCommitsEveryRoot728=== RUN TestPush_RepeatedRootCommitsOnce729=== PAUSE TestPush_RepeatedRootCommitsOnce730=== RUN TestPush_CompleteWaitsForCleanupOfThePush731=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush732=== RUN TestPush_RootsAreLockedInByteOrder733=== PAUSE TestPush_RootsAreLockedInByteOrder734=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit735=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit736=== RUN TestPush_RejectsBadRequests737=== PAUSE TestPush_RejectsBadRequests738=== RUN TestPush_SignsNarinfosOfItsPendingObjects739=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects740=== RUN TestSharedNarBothUploadsComplete741=== PAUSE TestSharedNarBothUploadsComplete742=== RUN TestCompleteMultipartUpload_ErrorButObjectExists743=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists744=== RUN TestCompletedNarNotReofferedAcrossClosures745=== PAUSE TestCompletedNarNotReofferedAcrossClosures746=== RUN TestPresignedUploadRegisteredBeforeCommit747=== PAUSE TestPresignedUploadRegisteredBeforeCommit748=== RUN TestService_Rustfstest749=== PAUSE TestService_Rustfstest750=== RUN TestParseSize751=== PAUSE TestParseSize752=== RUN TestSkippedUploadsHandler753=== PAUSE TestSkippedUploadsHandler754=== RUN TestSystemdListenerNotActivated755--- PASS: TestSystemdListenerNotActivated (0.00s)756=== RUN TestWatchdogBeatsWhenHealthy757--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)758=== RUN TestWatchdogSkipsWhenUnhealthy7592026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7602026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 00:26:39 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"768--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)769=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle770=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle771=== RUN TestProxyWriteTimeout772=== PAUSE TestProxyWriteTimeout773=== RUN TestPendingClosureWriteTimeout774=== PAUSE TestPendingClosureWriteTimeout775=== RUN TestIsValidUploadKey776=== PAUSE TestIsValidUploadKey777=== RUN TestUploadHandlersRejectInvalidKeys778=== PAUSE TestUploadHandlersRejectInvalidKeys779=== RUN TestUploadHandlersRejectOversizedBody780=== PAUSE TestUploadHandlersRejectOversizedBody781=== RUN TestService_cleanupPendingClosuresHandler782=== PAUSE TestService_cleanupPendingClosuresHandler783=== RUN TestService_createPendingClosureHandler784=== PAUSE TestService_createPendingClosureHandler785=== RUN TestService_verifyS3Integrity786=== PAUSE TestService_verifyS3Integrity787=== RUN TestService_verifyS3ReuploadsStaleNarinfo788=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo789=== RUN TestCompleteMultipartUnregistered790=== PAUSE TestCompleteMultipartUnregistered791=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT792=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT793=== CONT TestProxyWriteTimeout794=== CONT TestService_createPendingClosureHandler795=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT796=== CONT TestAbortSucceeded797=== CONT TestService_verifyS3Integrity798=== RUN TestProxyWriteTimeout/narinfo799=== PAUSE TestProxyWriteTimeout/narinfo800=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle801=== CONT TestSkippedUploadsHandler802=== CONT TestCompleteMultipartUnregistered803=== CONT TestService_verifyS3ReuploadsStaleNarinfo804=== CONT TestParseSize805=== CONT TestGracefulShutdownDrainsInflight806=== CONT TestLeadGenerationGrowsWithEveryGrant807=== CONT TestService_Rustfstest808=== CONT TestLeadIncumbentAnnouncesAtOnce809=== CONT TestLeadEndsOnShutdown810=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent811=== CONT TestGCTaskStore_Fail812=== CONT TestLeadElectsOneAndHandsOver813=== CONT TestGCTaskStore_PhaseUpdates814=== CONT TestConnectWaitsForAPeerMigration815=== CONT TestPresignedUploadRegisteredBeforeCommit816=== CONT TestGCTaskStore_CompletedAllowsNewTask817=== CONT TestResolveDBConnectionString818=== CONT TestCompletedNarNotReofferedAcrossClosures819=== RUN TestProxyWriteTimeout/1_GiB_nar820--- PASS: TestAbortSucceeded (0.00s)821=== CONT TestGCTaskStore_GetReturnsLatest822=== PAUSE TestProxyWriteTimeout/1_GiB_nar823=== RUN TestProxyWriteTimeout/10_GiB_nar824=== PAUSE TestProxyWriteTimeout/10_GiB_nar825=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed826=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed827=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed828=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed829=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost830=== RUN TestProxyWriteTimeout/unknown_size831--- PASS: TestParseSize (0.00s)832=== CONT TestCompleteMultipartUpload_ErrorButObjectExists833=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost834=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed835=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed836=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed837=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed838=== PAUSE TestProxyWriteTimeout/unknown_size839=== CONT TestGCTaskStore_GetEmpty840--- PASS: TestGCTaskStore_GetEmpty (0.00s)841=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock8422026/11/12 00:26:39 INFO Client skipped oversized paths paths=3 nar_bytes=50000000008432026/11/12 00:26:39 INFO Starting HTTP server address=127.0.0.1:40407844=== CONT TestGCTaskStore_ConflictDifferentParams845--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)846=== CONT TestGCTaskStore_DeduplicateSameParams847--- PASS: TestGCTaskStore_Fail (0.00s)848=== CONT TestSharedNarBothUploadsComplete849--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)850=== CONT TestGCTaskStore_StartNew851--- PASS: TestGCTaskStore_StartNew (0.00s)852=== CONT TestDeleteClosuresRacingPins8532026/11/12 00:26:39 INFO Shutdown signal received, draining in-flight requests timeout=10s854--- PASS: TestSkippedUploadsHandler (0.12s)855=== CONT TestGCEndsOnShutdown856=== RUN TestGCEndsOnShutdown/before_the_run857=== PAUSE TestGCEndsOnShutdown/before_the_run858=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete859=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete860=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark861=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark862=== CONT TestPush_SignsNarinfosOfItsPendingObjects863--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)864=== CONT TestCommitRacingPendingCleanupKeepsObjects865--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)866=== CONT TestClientFallsBackToClosures867--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)868=== CONT TestPush_RejectsBadRequests869=== RUN TestResolveDBConnectionString/flag_wins870=== PAUSE TestResolveDBConnectionString/flag_wins871=== RUN TestResolveDBConnectionString/file_when_flag_empty872=== PAUSE TestResolveDBConnectionString/file_when_flag_empty873=== RUN TestResolveDBConnectionString/missing_file_is_an_error874=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error875=== RUN TestResolveDBConnectionString/PGHOST_allows_empty876=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty877=== RUN TestResolveDBConnectionString/nothing_configured878=== PAUSE TestResolveDBConnectionString/nothing_configured879=== CONT TestPushWaitsForSweepHoldingRow880--- PASS: TestGracefulShutdownDrainsInflight (2.92s)881=== CONT TestClientPushesUseOnePush8822026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures8832026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures8842026/11/12 00:26:44 INFO lead: acquired remote=192.0.2.1:1234 generation=18852026/11/12 00:26:44 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst8862026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures8872026/11/12 00:26:44 INFO lead: acquired remote=192.0.2.1:1234 generation=1888--- PASS: TestPresignedUploadRegisteredBeforeCommit (4.51s)889=== CONT TestUploadHandlersRejectInvalidKeys890=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info891=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info892=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal893=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal894=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key895=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key896=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key897=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key898=== CONT TestPinProtectsFromGC8992026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9002026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9012026/11/12 00:26:44 INFO Received push request method=POST path=/api/pushes9022026/11/12 00:26:44 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9032026/11/12 00:26:44 INFO Signed narinfos id=1 count=19042026/11/12 00:26:44 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9052026/11/12 00:26:44 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9062026/11/12 00:26:44 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"9072026/11/12 00:26:44 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9082026/11/12 00:26:44 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"9092026/11/12 00:26:44 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9102026/11/12 00:26:44 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=19112026/11/12 00:26:44 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9122026/11/12 00:26:44 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo913--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (4.58s)914=== CONT TestSweepRechecksPendingAfterLocking9152026/11/12 00:26:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9162026/11/12 00:26:44 INFO lead: acquired remote=192.0.2.1:1234 generation=19172026/11/12 00:26:44 INFO lead: acquired remote=192.0.2.1:1234 generation=19182026/11/12 00:26:44 INFO lead: acquired remote=192.0.2.1:1234 generation=19192026/11/12 00:26:44 INFO lead: released remote=192.0.2.1:1234 generation=1920--- PASS: TestLeadIncumbentAnnouncesAtOnce (4.68s)921=== CONT TestClientWithDependencies9222026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9232026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9242026/11/12 00:26:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9252026/11/12 00:26:44 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=M2QxZmFjOTEtZWZhYy00ZDU2LTkyMDYtZTEwN2U0YTBkODQ4LjM2MWFkODQzLTNlMDUtNDVmNC04MzA5LWVlNWQ1YWYxYTQxNngxNzk0NDQzMjA0NDA1MjEyNzIx9262026/11/12 00:26:44 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=M2QxZmFjOTEtZWZhYy00ZDU2LTkyMDYtZTEwN2U0YTBkODQ4LjM2MWFkODQzLTNlMDUtNDVmNC04MzA5LWVlNWQ1YWYxYTQxNngxNzk0NDQzMjA0NDA1MjEyNzIx parts=19272026/11/12 00:26:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete928--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (4.92s)929=== CONT TestService_cleanupPendingClosuresHandler9302026/11/12 00:26:44 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst931--- PASS: TestCompleteMultipartUnregistered (4.92s)932=== CONT TestClientMultipleUploads9332026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9342026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9352026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9362026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures937--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (5.04s)938=== CONT TestGetExistingObjectsLocksRows939--- PASS: TestService_Rustfstest (5.05s)940=== CONT TestClientIntegration9412026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9422026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9432026/11/12 00:26:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9442026/11/12 00:26:44 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=M2QxZmFjOTEtZWZhYy00ZDU2LTkyMDYtZTEwN2U0YTBkODQ4LmZiOTgwZWZkLWE3ODQtNGY5MS1iYzRkLTQ3YmFkMmQzZWNkYngxNzk0NDQzMjA0MjM5MDU1NzE0 parts=12945=== RUN TestPush_RejectsBadRequests/root_not_in_objects946=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects947=== RUN TestPush_RejectsBadRequests/no_roots948=== PAUSE TestPush_RejectsBadRequests/no_roots949=== RUN TestPush_RejectsBadRequests/no_objects950=== PAUSE TestPush_RejectsBadRequests/no_objects951=== RUN TestPush_RejectsBadRequests/bad_root952=== PAUSE TestPush_RejectsBadRequests/bad_root953=== CONT TestRegisterCompletedObjectNeedsPendingRow9542026/11/12 00:26:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9552026/11/12 00:26:44 INFO Aborted multipart uploads count=0 kept=09562026/11/12 00:26:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=M2QxZmFjOTEtZWZhYy00ZDU2LTkyMDYtZTEwN2U0YTBkODQ4LjllMmM2Mzc2LTRmMjMtNGE4My1hNzdhLWM5MDFhMzIxYTE4NHgxNzk0NDQzMjA0NDUwMTY2NDgw parts=109572026/11/12 00:26:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete958--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (5.39s)959=== CONT TestClientErrorHandling960=== RUN TestClientErrorHandling/InvalidStorePath961=== PAUSE TestClientErrorHandling/InvalidStorePath962=== RUN TestClientErrorHandling/InvalidAuthToken963=== PAUSE TestClientErrorHandling/InvalidAuthToken964=== RUN TestClientErrorHandling/ServerNotAvailable965=== PAUSE TestClientErrorHandling/ServerNotAvailable966=== CONT TestSweepSkipsRowsLockedByPush9672026/11/12 00:26:44 INFO Completed upload id=1 objects=2 duration=7.627991ms9682026/11/12 00:26:44 INFO Received uploads request method=POST path=/api/pending_closures9692026/11/12 00:26:45 INFO Received uploads request method=POST path=/api/pending_closures9702026/11/12 00:26:45 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo9712026/11/12 00:26:45 WARN Found objects in DB but missing from S3, will re-upload count=1972--- PASS: TestService_verifyS3Integrity (5.50s)973=== CONT TestClientCADerivations9742026/11/12 00:26:45 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9752026/11/12 00:26:45 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=M2QxZmFjOTEtZWZhYy00ZDU2LTkyMDYtZTEwN2U0YTBkODQ4LmExZDU0MjdlLTEyZmQtNDZjOC1iZmJmLTlhZmIyMzk1YWNiZHgxNzk0NDQzMjA0NTY3OTYzMzk3 parts=109762026/11/12 00:26:45 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9772026/11/12 00:26:45 INFO Completed upload id=1 objects=2 duration=6.082796ms9782026/11/12 00:26:45 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000009792026/11/12 00:26:45 INFO Received uploads request method=POST path=/api/pending_closures9802026/11/12 00:26:45 INFO Starting cleanup of old closures method=DELETE path=/api/closures9812026/11/12 00:26:45 INFO Aborted multipart uploads count=0 kept=09822026/11/12 00:26:45 INFO Aborted multipart uploads count=0 kept=09832026/11/12 00:26:45 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9842026/11/12 00:26:45 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=M2QxZmFjOTEtZWZhYy00ZDU2LTkyMDYtZTEwN2U0YTBkODQ4LjFjY2NhOTlkLTU1ODctNGRkNC1hOGQ1LTUwODY2ZWQ2YzA0OHgxNzk0NDQzMjA0Njk2MTYxNzEx parts=129852026/11/12 00:26:45 INFO Received uploads request method=POST path=/api/pending_closures986--- PASS: TestCompletedNarNotReofferedAcrossClosures (2.75s)987=== CONT TestClientSharedPathCommittedMidPush988--- PASS: TestGetExistingObjectsLocksRows (0.66s)989=== CONT TestCacheStatsHandler9902026/11/12 00:26:45 INFO lead: released remote=192.0.2.1:1234 generation=1991--- PASS: TestLeadEndsOnShutdown (5.60s)992=== CONT TestFailedS3DeleteKeepsTombstone9932026/11/12 00:26:45 INFO Received cleanup request method=DELETE path=/api/pending_closures9942026/11/12 00:26:45 INFO Aborted multipart uploads count=0 kept=0995--- PASS: TestPushWaitsForSweepHoldingRow (2.80s)996=== CONT TestUploadHandlersRejectOversizedBody9972026/11/12 00:26:45 INFO Received uploads request method=POST path=/api/pending_closures9982026/11/12 00:26:45 INFO Received cleanup request method=DELETE path=/api/pending_closures9992026/11/12 00:26:45 INFO Aborted multipart uploads count=1 kept=010002026/11/12 00:26:45 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10012026-11-12 00:26:45.352 UTC [517] ERROR: Closure does not exist: id=110022026-11-12 00:26:45.352 UTC [517] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE10032026-11-12 00:26:45.352 UTC [517] STATEMENT: -- name: CommitPendingClosure :one1004 SELECT commit_pending_closure($1::bigint)::bigint1005 1006--- PASS: TestService_cleanupPendingClosuresHandler (0.85s)1007=== CONT TestCacheConfigHandler1008=== RUN TestCacheConfigHandler/full_config,_no_issuer1009=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1010=== RUN TestCacheConfigHandler/no_cache_url_configured1011=== PAUSE TestCacheConfigHandler/no_cache_url_configured1012=== RUN TestCacheConfigHandler/no_signing_keys1013=== PAUSE TestCacheConfigHandler/no_signing_keys1014=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1015=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1016=== CONT TestForceGCDuringPushOffersSweptObject1017=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1018=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1019=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1020=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1021=== CONT TestService_ReadScope_PublicByDefault10222026/11/12 00:26:45 INFO lead: released remote=192.0.2.1:1234 generation=110232026/11/12 00:26:45 INFO lead: acquired remote=192.0.2.1:1234 generation=210242026/11/12 00:26:45 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10252026/11/12 00:26:45 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=M2QxZmFjOTEtZWZhYy00ZDU2LTkyMDYtZTEwN2U0YTBkODQ4Ljg0ZDZmMjExLTMyYzMtNDM2My05MzdmLTdjMGU0ZmU3OTUyYngxNzk0NDQzMjA0MjQ5NjY4Mzky parts=121026--- PASS: TestSharedNarBothUploadsComplete (5.70s)1027=== CONT TestDeduplicatedObjectsRecordedAsPending10282026/11/12 00:26:45 INFO lead: released remote=192.0.2.1:1234 generation=11029=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1030=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1031=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1032=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1033=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1034=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1035=== CONT TestService_RequireScope_OIDC10362026/11/12 00:26:45 INFO lead: acquired remote=192.0.2.1:1234 generation=210372026/11/12 00:26:45 WARN Force mode enabled - objects will be deleted immediately without grace period10382026/11/12 00:26:45 WARN Force mode enabled - objects will be deleted immediately without grace period10392026/11/12 00:26:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010402026/11/12 00:26:45 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=210412026/11/12 00:26:45 INFO Vacuumed table table=pending_closures10422026/11/12 00:26:45 INFO Vacuumed table table=pending_objects10432026/11/12 00:26:45 INFO Vacuumed table table=multipart_uploads10442026/11/12 00:26:45 INFO Vacuumed table table=closures10452026/11/12 00:26:45 INFO Vacuumed table table=objects10462026/11/12 00:26:45 INFO lead: acquired remote=192.0.2.1:1234 generation=310472026/11/12 00:26:45 INFO lead: released remote=192.0.2.1:1234 generation=110482026/11/12 00:26:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010492026/11/12 00:26:45 INFO Vacuumed table table=pending_closures10502026/11/12 00:26:45 INFO Vacuumed table table=pending_objects1051=== NAME TestClientMultipleUploads10522026/11/12 00:26:45 INFO Vacuumed table table=multipart_uploads1053 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1962770990/001/store/y3hvg9cljcalic0sav1lx9skgivpbk7z-test-file-0.txt1054=== NAME TestClientIntegration1055 client_integration_test.go:292: Created store path: /build/TestClientIntegration2513944452/002/store/5g3a2223pa6ymsgig1k1bf2502dm399x-test-file.txt10562026/11/12 00:26:45 INFO Vacuumed table table=closures10572026/11/12 00:26:45 INFO Vacuumed table table=objects1058--- PASS: TestSweepRechecksPendingAfterLocking (1.27s)1059=== CONT TestTombstonedObjectOfferedWithoutWaiting10602026/11/12 00:26:45 INFO Received uploads request method=POST path=/api/pending_closures10612026/11/12 00:26:45 INFO lead: acquired remote=192.0.2.1:1234 generation=21062--- PASS: TestCacheStatsHandler (0.38s)1063=== CONT TestPushDedupSurvivesConcurrentGC10642026/11/12 00:26:45 INFO Aborted multipart uploads count=0 kept=010652026/11/12 00:26:45 INFO Aborted multipart uploads count=0 kept=010662026/11/12 00:26:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43321/oidc10672026/11/12 00:26:45 WARN Force mode enabled - objects will be deleted immediately without grace period10682026/11/12 00:26:45 WARN Force mode enabled - objects will be deleted immediately without grace period1069--- PASS: TestCommitRacingPendingCleanupKeepsObjects (5.46s)1070=== CONT TestGCSweepSkipsPendingObjects1071=== NAME TestClientMultipleUploads1072 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1962770990/001/store/cc6jzdisqfbbvjxx1xr4gqlvggi47qav-test-file-1.txt10732026/11/12 00:26:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010742026/11/12 00:26:45 INFO Vacuumed table table=pending_closures10752026/11/12 00:26:45 INFO Vacuumed table table=pending_objects10762026/11/12 00:26:45 INFO Vacuumed table table=multipart_uploads10772026/11/12 00:26:45 INFO Vacuumed table table=closures10782026/11/12 00:26:45 INFO Vacuumed table table=objects10792026/11/12 00:26:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010802026/11/12 00:26:45 INFO Aborted multipart uploads count=0 kept=010812026/11/12 00:26:45 INFO Vacuumed table table=pending_closures10822026/11/12 00:26:45 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."10832026/11/12 00:26:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."10842026/11/12 00:26:45 INFO Vacuumed table table=pending_objects1085--- PASS: TestFailedS3DeleteKeepsTombstone (0.55s)1086=== CONT TestService_AuthMiddleware_OIDC10872026/11/12 00:26:45 INFO Vacuumed table table=multipart_uploads10882026/11/12 00:26:45 INFO Vacuumed table table=closures10892026/11/12 00:26:45 INFO Vacuumed table table=objects10902026/11/12 00:26:45 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001091--- PASS: TestService_createPendingClosureHandler (6.33s)1092=== CONT TestGCMetrics10932026/11/12 00:26:45 WARN Force mode enabled - objects will be deleted immediately without grace period1094=== NAME TestClientMultipleUploads1095 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1962770990/001/store/pmdir6x438babhlhv9qrnjjrl0qci8is-test-file-2.txt1096=== NAME TestPinProtectsFromGC1097 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC3726109724/001/store/jvj7z6ihipxcsp8696grvnypb0gwl0ra-pinned-file.txt1098 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC3726109724/001/store/c2hc040g3ds2nmrpil1lwp1yjf08bkjj-unpinned-file.txt1099=== NAME TestClientCADerivations1100 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1487795790/001/store/k5vv2k0j5k45cf5di2awci8ns86fi3pj-ca-test1101=== NAME TestClientWithDependencies1102 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies3380290685/001/store/m5mnwbilj5w2xbid2iw652plqmq9zc0x-test-script11032026/11/12 00:26:45 INFO Received uploads request method=POST path=/api/pending_closures11042026/11/12 00:26:45 INFO Received uploads request method=POST path=/api/pending_closures1105--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.43s)1106=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit1107--- PASS: TestService_ReadScope_PublicByDefault (0.65s)1108=== CONT TestParseSingleRange1109=== RUN TestParseSingleRange/none1110=== PAUSE TestParseSingleRange/none1111=== RUN TestParseSingleRange/unknown_unit1112=== PAUSE TestParseSingleRange/unknown_unit1113=== RUN TestParseSingleRange/multi-range_ignored1114=== PAUSE TestParseSingleRange/multi-range_ignored1115=== RUN TestParseSingleRange/malformed_no_dash1116=== PAUSE TestParseSingleRange/malformed_no_dash1117=== RUN TestParseSingleRange/malformed_both_empty1118=== PAUSE TestParseSingleRange/malformed_both_empty1119=== RUN TestParseSingleRange/malformed_end_before_start1120=== PAUSE TestParseSingleRange/malformed_end_before_start1121=== RUN TestParseSingleRange/closed1122=== PAUSE TestParseSingleRange/closed1123=== RUN TestParseSingleRange/open-ended1124=== PAUSE TestParseSingleRange/open-ended1125=== RUN TestParseSingleRange/end_clamped_to_size1126=== PAUSE TestParseSingleRange/end_clamped_to_size1127=== RUN TestParseSingleRange/suffix1128=== PAUSE TestParseSingleRange/suffix1129=== RUN TestParseSingleRange/suffix_exceeds_size1130=== PAUSE TestParseSingleRange/suffix_exceeds_size1131=== RUN TestParseSingleRange/single_byte1132=== PAUSE TestParseSingleRange/single_byte1133=== RUN TestParseSingleRange/start_past_EOF1134=== PAUSE TestParseSingleRange/start_past_EOF1135=== RUN TestParseSingleRange/start_far_past_EOF1136=== PAUSE TestParseSingleRange/start_far_past_EOF1137=== CONT TestService_ReadAuthMiddleware11382026/11/12 00:26:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=011392026/11/12 00:26:46 INFO Vacuumed table table=pending_closures11402026/11/12 00:26:46 INFO Vacuumed table table=pending_objects11412026/11/12 00:26:46 INFO Vacuumed table table=multipart_uploads11422026/11/12 00:26:46 INFO Vacuumed table table=closures1143=== NAME TestClientWithDependencies1144 client_integration_test.go:663: Found 1 dependencies (including self)11452026/11/12 00:26:46 INFO Vacuumed table table=objects1146--- PASS: TestSweepSkipsRowsLockedByPush (1.04s)1147=== CONT TestGCBugBareHashReferences1148=== NAME TestClientCADerivations1149 client_ca_test.go:139: Found 1 dependencies (including self)1150--- PASS: TestRegisterCompletedObjectNeedsPendingRow (1.14s)1151=== CONT TestService_AuthMiddleware_MTLSBoundSubjects11522026/11/12 00:26:46 INFO Aborted multipart uploads count=0 kept=011532026/11/12 00:26:46 INFO Received uploads request method=POST path=/api/pending_closures1154--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.68s)1155=== CONT TestPresentNotReportedWhileGCDeletesClosure1156=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1157=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1158=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1159=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1160=== CONT TestPush_RootsAreLockedInByteOrder11612026/11/12 00:26:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34251/oidc1162=== RUN TestService_RequireScope_OIDC/builder_may_write1163=== PAUSE TestService_RequireScope_OIDC/builder_may_write1164=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1165=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1166=== RUN TestService_RequireScope_OIDC/ops_may_admin1167=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1168=== RUN TestService_RequireScope_OIDC/ops_may_not_write1169=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1170=== RUN TestService_RequireScope_OIDC/reader_may_not_write1171=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1172=== RUN TestService_RequireScope_OIDC/static_token_may_admin1173=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1174=== RUN TestService_RequireScope_OIDC/static_token_may_write1175=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1176=== RUN TestService_RequireScope_OIDC/reader_may_read1177=== PAUSE TestService_RequireScope_OIDC/reader_may_read1178=== RUN TestService_RequireScope_OIDC/writer_implies_read1179=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1180=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1181=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1182=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1183=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1184=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1185=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1186=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1187=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1188=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1189=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1190=== CONT TestService_AuthMiddleware_MTLSProxyHeader11912026/11/12 00:26:46 INFO Received uploads request method=POST path=/api/pending_closures11922026/11/12 00:26:46 INFO Aborted multipart uploads count=0 kept=011932026/11/12 00:26:46 WARN Force mode enabled - objects will be deleted immediately without grace period11942026/11/12 00:26:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011952026/11/12 00:26:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011962026/11/12 00:26:46 INFO Vacuumed table table=pending_closures11972026/11/12 00:26:46 INFO Vacuumed table table=pending_closures11982026/11/12 00:26:46 INFO Vacuumed table table=pending_objects11992026/11/12 00:26:46 INFO Vacuumed table table=multipart_uploads12002026/11/12 00:26:46 INFO Vacuumed table table=closures12012026/11/12 00:26:46 INFO Vacuumed table table=pending_objects12022026/11/12 00:26:46 INFO Vacuumed table table=multipart_uploads12032026/11/12 00:26:46 INFO Vacuumed table table=objects12042026/11/12 00:26:46 INFO Vacuumed table table=closures12052026/11/12 00:26:46 INFO Vacuumed table table=objects1206--- PASS: TestGCSweepSkipsPendingObjects (0.40s)1207=== CONT TestPresentRefreshesOnlyStaleClosures12082026/11/12 00:26:46 INFO Aborted multipart uploads count=0 kept=012092026/11/12 00:26:46 WARN Force mode enabled - objects will be deleted immediately without grace period12102026/11/12 00:26:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012112026/11/12 00:26:46 INFO Vacuumed table table=pending_closures12122026/11/12 00:26:46 INFO Vacuumed table table=pending_objects12132026/11/12 00:26:46 INFO Vacuumed table table=multipart_uploads12142026/11/12 00:26:46 INFO Vacuumed table table=closures12152026/11/12 00:26:46 INFO Vacuumed table table=objects1216--- PASS: TestPushDedupSurvivesConcurrentGC (0.58s)1217=== CONT TestPush_CompleteWaitsForCleanupOfThePush12182026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes1219--- PASS: TestService_ReadAuthMiddleware (0.25s)1220=== CONT TestService_AuthMiddleware12212026/11/12 00:26:46 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)12222026/11/12 00:26:46 INFO Uploading pmdir6x438babhlhv9qrnjjrl0qci8is-test-file-2.txt (160B)12232026/11/12 00:26:46 INFO Uploading cc6jzdisqfbbvjxx1xr4gqlvggi47qav-test-file-1.txt (160B)12242026/11/12 00:26:46 INFO Uploading y3hvg9cljcalic0sav1lx9skgivpbk7z-test-file-0.txt (160B)12252026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes12262026/11/12 00:26:46 WARN Failed to register uploaded object key=y3hvg9cljcalic0sav1lx9skgivpbk7z.ls error="server returned 404: 404 page not found\n"12272026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes12282026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"12292026/11/12 00:26:46 WARN Failed to register uploaded object key=cc6jzdisqfbbvjxx1xr4gqlvggi47qav.ls error="server returned 404: 404 page not found\n"12302026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"12312026/11/12 00:26:46 WARN Failed to register uploaded object key=pmdir6x438babhlhv9qrnjjrl0qci8is.ls error="server returned 404: 404 page not found\n"12322026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"12332026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12342026/11/12 00:26:46 INFO Signed narinfos id=1 count=312352026/11/12 00:26:46 INFO Uploading 3 narinfos12362026/11/12 00:26:46 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12372026/11/12 00:26:46 INFO Uploading 5g3a2223pa6ymsgig1k1bf2502dm399x-test-file.txt (152B)12382026/11/12 00:26:46 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12392026/11/12 00:26:46 INFO Uploading m5mnwbilj5w2xbid2iw652plqmq9zc0x-test-script (136B)12402026/11/12 00:26:46 WARN Failed to register uploaded object key=pmdir6x438babhlhv9qrnjjrl0qci8is.narinfo error="server returned 404: 404 page not found\n"12412026/11/12 00:26:46 WARN Failed to register uploaded object key=cc6jzdisqfbbvjxx1xr4gqlvggi47qav.narinfo error="server returned 404: 404 page not found\n"12422026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete12432026/11/12 00:26:46 WARN Failed to register uploaded object key=y3hvg9cljcalic0sav1lx9skgivpbk7z.narinfo error="server returned 404: 404 page not found\n"12442026/11/12 00:26:46 INFO Received uploads request method=POST path=/api/pending_closures12452026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"12462026/11/12 00:26:46 INFO Aborted multipart uploads count=0 kept=012472026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12482026/11/12 00:26:46 WARN Failed to register uploaded object key=m5mnwbilj5w2xbid2iw652plqmq9zc0x.ls error="server returned 404: 404 page not found\n"12492026/11/12 00:26:46 INFO Signed narinfos id=1 count=112502026/11/12 00:26:46 WARN Failed to register uploaded object key=5g3a2223pa6ymsgig1k1bf2502dm399x.ls error="server returned 404: 404 page not found\n"12512026/11/12 00:26:46 INFO Uploading 1 narinfos12522026/11/12 00:26:46 INFO Completed push id=1 objects=9 duration=10.667899ms12532026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes12542026/11/12 00:26:46 INFO Upload complete. (127ms)1255=== NAME TestClientMultipleUploads1256 client_integration_test.go:417: Uploaded 3 paths in 385.088207ms12572026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12582026/11/12 00:26:46 WARN Failed to register uploaded object key=log/8r20dq7giil3kaqpgah9232852agmhjz-test-script.drv error="server returned 404: 404 page not found\n"12592026/11/12 00:26:46 INFO Received uploads request method=POST path=/api/pending_closures12602026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete12612026/11/12 00:26:46 INFO Signed narinfos id=1 count=112622026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"12632026/11/12 00:26:46 WARN Failed to register uploaded object key=5g3a2223pa6ymsgig1k1bf2502dm399x.narinfo error="server returned 404: 404 page not found\n"12642026/11/12 00:26:46 INFO Uploading 1 narinfos12652026/11/12 00:26:46 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"12662026/11/12 00:26:46 WARN mTLS auth: bound subjects configured but subject DN unavailable12672026/11/12 00:26:46 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1268--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.28s)1269=== CONT TestPresentReportsOnlyClosureRoots12702026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete12712026/11/12 00:26:46 WARN Failed to register uploaded object key=m5mnwbilj5w2xbid2iw652plqmq9zc0x.narinfo error="server returned 404: 404 page not found\n"1272--- PASS: TestClientMultipleUploads (1.83s)1273=== CONT TestPush_RepeatedRootCommitsOnce12742026/11/12 00:26:46 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)12752026/11/12 00:26:46 INFO Uploading 1bkfi6r6a2niwsdg7fvn14sh4phkv1xc-shared-dep (136B)12762026/11/12 00:26:46 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12772026/11/12 00:26:46 INFO Uploading kbwyilhmpgv9ajz2xm2gyp8gc9xm4xf8-a (216B)12782026/11/12 00:26:46 INFO Uploading k5vv2k0j5k45cf5di2awci8ns86fi3pj-ca-test (144B)12792026/11/12 00:26:46 INFO Completed push id=1 objects=3 duration=13.132502ms12802026/11/12 00:26:46 INFO Upload complete. (166ms)12812026/11/12 00:26:46 INFO Completed push id=1 objects=4 duration=11.986751ms12822026/11/12 00:26:46 INFO Upload complete. (160ms)12832026/11/12 00:26:46 WARN Failed to register uploaded object key=kbwyilhmpgv9ajz2xm2gyp8gc9xm4xf8.ls error="server returned 404: 404 page not found\n"12842026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12852026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/11lmyapnq05m6x2sf8dh9mv655basnjk0zha8728imw0k3fi81r4.nar.zst error="server returned 404: 404 page not found\n"12862026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign12872026/11/12 00:26:46 WARN Failed to register uploaded object key=1bkfi6r6a2niwsdg7fvn14sh4phkv1xc.ls error="server returned 404: 404 page not found\n"12882026/11/12 00:26:46 INFO Signed narinfos id=1 count=212892026/11/12 00:26:46 WARN Failed to register uploaded object key=5ngqxs003aylyxq0nfkmx6yb6z2lgm88.ls error="server returned 404: 404 page not found\n"12902026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"12912026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign12922026/11/12 00:26:46 INFO Signed narinfos id=2 count=212932026/11/12 00:26:46 WARN Failed to register uploaded object key=log/57s4cr1948y8ijpq3nzy4701kjahclxd-ca-test.drv error="server returned 404: 404 page not found\n"1294=== NAME TestClientWithDependencies12952026/11/12 00:26:46 INFO Uploading 4 narinfos1296 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies3380290685/001/store) requires matching store prefix12972026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12982026/11/12 00:26:46 WARN Failed to register uploaded object key=k5vv2k0j5k45cf5di2awci8ns86fi3pj.ls error="server returned 404: 404 page not found\n"12992026/11/12 00:26:46 INFO Signed narinfos id=1 count=113002026/11/12 00:26:46 INFO Uploading 1 narinfos1301--- PASS: TestClientWithDependencies (1.97s)1302=== CONT TestCreatePin_ReservedPins13032026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete13042026/11/12 00:26:46 WARN Failed to register uploaded object key=5ngqxs003aylyxq0nfkmx6yb6z2lgm88.narinfo error="server returned 404: 404 page not found\n"13052026/11/12 00:26:46 WARN Failed to register uploaded object key=1bkfi6r6a2niwsdg7fvn14sh4phkv1xc.narinfo error="server returned 404: 404 page not found\n"13062026/11/12 00:26:46 WARN Failed to register uploaded object key=k5vv2k0j5k45cf5di2awci8ns86fi3pj.narinfo error="server returned 404: 404 page not found\n"13072026/11/12 00:26:46 WARN Failed to register uploaded object key=kbwyilhmpgv9ajz2xm2gyp8gc9xm4xf8.narinfo error="server returned 404: 404 page not found\n"13082026/11/12 00:26:46 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13092026/11/12 00:26:46 WARN Failed to register uploaded object key=1bkfi6r6a2niwsdg7fvn14sh4phkv1xc.narinfo error="server returned 404: 404 page not found\n"13102026/11/12 00:26:46 INFO Completed push id=1 objects=4 duration=12.018232ms13112026/11/12 00:26:46 INFO Upload complete. (182ms)13122026/11/12 00:26:46 INFO Completed upload id=1 objects=6 duration=9.105765ms13132026/11/12 00:26:46 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete1314=== NAME TestClientCADerivations1315 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1487795790/001/store/k5vv2k0j5k45cf5di2awci8ns86fi3pj-ca-test1316 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1317 Compression: zstd1318 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1319 NarSize: 1441320 References: 1321 Deriver: /build/TestClientCADerivations1487795790/001/store/57s4cr1948y8ijpq3nzy4701kjahclxd-ca-test.drv1322 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1323 client_ca_test.go:185: Checking for realisation files in S3...13242026/11/12 00:26:46 INFO Completed upload id=2 objects=6 duration=4.357361ms13252026/11/12 00:26:46 INFO Upload complete. (195ms)1326=== NAME TestClientFallsBackToClosures1327 client_pushes_test.go:112: Retrieved narinfo from S3:1328 StorePath: /build/TestClientFallsBackToClosures3959497828/001/store/1bkfi6r6a2niwsdg7fvn14sh4phkv1xc-shared-dep1329 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1330 Compression: zstd1331 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821332 NarSize: 1361333 References: 1334 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1335 client_pushes_test.go:112: Retrieved narinfo from S3:1336 StorePath: /build/TestClientFallsBackToClosures3959497828/001/store/kbwyilhmpgv9ajz2xm2gyp8gc9xm4xf8-a1337 URL: nar/11lmyapnq05m6x2sf8dh9mv655basnjk0zha8728imw0k3fi81r4.nar.zst1338 Compression: zstd1339 NarHash: sha256:11lmyapnq05m6x2sf8dh9mv655basnjk0zha8728imw0k3fi81r41340 NarSize: 2161341 References: /build/TestClientFallsBackToClosures3959497828/001/store/1bkfi6r6a2niwsdg7fvn14sh4phkv1xc-shared-dep1342 CA: text:sha256:0w61nlpjwznwri8lbvywspkp7dd23rplswvdys6n9qdhsww7z7fb1343--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.27s)1344=== CONT TestPush_CompleteCommitsEveryRoot1345=== NAME TestClientFallsBackToClosures1346 client_pushes_test.go:112: Retrieved narinfo from S3:1347 StorePath: /build/TestClientFallsBackToClosures3959497828/001/store/5ngqxs003aylyxq0nfkmx6yb6z2lgm88-b1348 URL: nar/11lmyapnq05m6x2sf8dh9mv655basnjk0zha8728imw0k3fi81r4.nar.zst1349 Compression: zstd1350 NarHash: sha256:11lmyapnq05m6x2sf8dh9mv655basnjk0zha8728imw0k3fi81r41351 NarSize: 2161352 References: /build/TestClientFallsBackToClosures3959497828/001/store/1bkfi6r6a2niwsdg7fvn14sh4phkv1xc-shared-dep1353 CA: text:sha256:0w61nlpjwznwri8lbvywspkp7dd23rplswvdys6n9qdhsww7z7fb1354--- PASS: TestClientFallsBackToClosures (3.89s)1355=== CONT TestOrphanedObjectsGCStressTest1356=== NAME TestClientCADerivations1357 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1358 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache13592026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes1360=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1361=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1362=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1363=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1364=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1365=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1366=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1367=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1368=== CONT TestPush_OverlappingRootsStoreOneRowPerKey13692026/11/12 00:26:46 INFO lead: released remote=192.0.2.1:1234 generation=213702026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes13712026/11/12 00:26:46 INFO lead: acquired remote=192.0.2.1:1234 generation=313722026/11/12 00:26:46 INFO lead: released remote=192.0.2.1:1234 generation=31373--- PASS: TestLeadGenerationGrowsWithEveryGrant (6.90s)1374=== CONT TestOrphanedObjectsGC13752026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete13762026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes13772026/11/12 00:26:46 INFO Completed push id=1 objects=4 duration=40.697999ms13782026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes13792026-11-12 00:26:46.540 UTC [1269] ERROR: could not obtain lock on row in relation "closures"13802026-11-12 00:26:46.540 UTC [1269] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT13812026/11/12 00:26:46 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13822026/11/12 00:26:46 INFO Uploading jvj7z6ihipxcsp8696grvnypb0gwl0ra-pinned-file.txt (128B)13832026-11-12 00:26:46.541 UTC [1269] ERROR: could not obtain lock on row in relation "closures"13842026-11-12 00:26:46.541 UTC [1269] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT13852026-11-12 00:26:46.542 UTC [1269] ERROR: could not obtain lock on row in relation "closures"13862026-11-12 00:26:46.542 UTC [1269] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT13872026-11-12 00:26:46.542 UTC [1269] ERROR: could not obtain lock on row in relation "closures"13882026-11-12 00:26:46.542 UTC [1269] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT13892026-11-12 00:26:46.542 UTC [1269] ERROR: could not obtain lock on row in relation "closures"13902026-11-12 00:26:46.542 UTC [1269] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1391--- PASS: TestPush_RootsAreLockedInByteOrder (0.47s)1392=== CONT TestGCSweepDeliversEachKeyOnce13932026/11/12 00:26:46 WARN Failed to register uploaded object key=jvj7z6ihipxcsp8696grvnypb0gwl0ra.ls error="server returned 404: 404 page not found\n"13942026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13952026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"13962026/11/12 00:26:46 INFO Signed narinfos id=1 count=113972026/11/12 00:26:46 INFO Uploading 1 narinfos13982026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete13992026/11/12 00:26:46 WARN Failed to register uploaded object key=jvj7z6ihipxcsp8696grvnypb0gwl0ra.narinfo error="server returned 404: 404 page not found\n"14002026/11/12 00:26:46 INFO Completed push id=1 objects=3 duration=12.928461ms14012026/11/12 00:26:46 INFO Upload complete. (329ms)14022026/11/12 00:26:46 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34777/oidc14032026/11/12 00:26:46 INFO Aborted multipart uploads count=0 kept=014042026/11/12 00:26:46 WARN Force mode enabled - objects will be deleted immediately without grace period14052026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes14062026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete14072026/11/12 00:26:46 INFO lead: released remote=192.0.2.1:1234 generation=21408--- PASS: TestLeadElectsOneAndHandsOver (7.00s)1409=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection14102026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes14112026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes14122026/11/12 00:26:46 INFO Completed push id=1 objects=4 duration=36.181096ms14132026/11/12 00:26:46 INFO All 1 paths already cached1414--- PASS: TestPush_RepeatedRootCommitsOnce (0.38s)1415=== CONT TestReadRedirectUsesPublicS3URL14162026/11/12 00:26:46 WARN Force mode enabled - objects will be deleted immediately without grace period14172026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes14182026/11/12 00:26:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014192026/11/12 00:26:46 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)14202026/11/12 00:26:46 INFO Uploading sgnr83gqg8ya15k8aavimmvm26kzrvi8-shared-dep (136B)14212026/11/12 00:26:46 INFO Uploading wkgz4myaay77yxrvgbq7d177cav4q4za-b (216B)1422=== NAME TestClientIntegration1423 client_integration_test.go:318: Retrieved narinfo from S3:1424 StorePath: /build/TestClientIntegration2513944452/002/store/5g3a2223pa6ymsgig1k1bf2502dm399x-test-file.txt1425 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1426 Compression: zstd1427 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11428 NarSize: 1521429 References: 1430 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk114312026/11/12 00:26:46 INFO Vacuumed table table=pending_closures14322026/11/12 00:26:46 INFO Vacuumed table table=pending_objects14332026/11/12 00:26:46 INFO Vacuumed table table=multipart_uploads14342026/11/12 00:26:46 INFO Vacuumed table table=closures14352026/11/12 00:26:46 INFO Vacuumed table table=objects14362026/11/12 00:26:46 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)14372026/11/12 00:26:46 INFO Uploading xvqn7jabp38ik4n4sqli3xm1zyd3dpd2-shared-dep (136B)14382026/11/12 00:26:46 INFO Uploading 1k6nmsgf2rr2nz40hsy0nhy6dzb8ryrr-top (224B)1439 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1440 client_integration_test.go:319: Decompressed .ls content (64 bytes):1441 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1442--- PASS: TestGCMetrics (0.83s)1443=== CONT TestEnsureAbortUploadsRule1444--- PASS: TestGCBugBareHashReferences (0.74s)1445=== CONT TestMetricsInventory14462026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/1a0w4kmz3npsrhny9yzv9dfgiq2szfj38n2fwcnr22791964wm1p.nar.zst error="server returned 404: 404 page not found\n"14472026/11/12 00:26:46 WARN Failed to register uploaded object key=sgnr83gqg8ya15k8aavimmvm26kzrvi8.ls error="server returned 404: 404 page not found\n"14482026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14492026/11/12 00:26:46 WARN Failed to register uploaded object key=wkgz4myaay77yxrvgbq7d177cav4q4za.ls error="server returned 404: 404 page not found\n"14502026/11/12 00:26:46 WARN Failed to register uploaded object key=dhz4k0i4psqbchbiyf8vv4c6yjyijr72.ls error="server returned 404: 404 page not found\n"14512026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14522026/11/12 00:26:46 INFO Signed narinfos id=1 count=314532026/11/12 00:26:46 INFO Uploading 3 narinfos14542026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14552026/11/12 00:26:46 WARN Failed to register uploaded object key=xvqn7jabp38ik4n4sqli3xm1zyd3dpd2.ls error="server returned 404: 404 page not found\n"14562026/11/12 00:26:46 WARN Failed to register uploaded object key=1k6nmsgf2rr2nz40hsy0nhy6dzb8ryrr.ls error="server returned 404: 404 page not found\n"14572026/11/12 00:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14582026/11/12 00:26:46 WARN Failed to register uploaded object key=nar/1wxsm42wg407vfq6ddb98kyphjn7n6lrdf1kwx6yp5iyz7iavf1r.nar.zst error="server returned 404: 404 page not found\n"14592026/11/12 00:26:46 INFO Signed narinfos id=1 count=214602026/11/12 00:26:46 INFO Uploading 2 narinfos14612026/11/12 00:26:46 WARN Failed to register uploaded object key=wkgz4myaay77yxrvgbq7d177cav4q4za.narinfo error="server returned 404: 404 page not found\n"14622026/11/12 00:26:46 INFO Received push request method=POST path=/api/pushes14632026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete14642026/11/12 00:26:46 WARN Failed to register uploaded object key=dhz4k0i4psqbchbiyf8vv4c6yjyijr72.narinfo error="server returned 404: 404 page not found\n"14652026/11/12 00:26:46 WARN Failed to register uploaded object key=sgnr83gqg8ya15k8aavimmvm26kzrvi8.narinfo error="server returned 404: 404 page not found\n"14662026/11/12 00:26:46 WARN Failed to register uploaded object key=1k6nmsgf2rr2nz40hsy0nhy6dzb8ryrr.narinfo error="server returned 404: 404 page not found\n"14672026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete14682026/11/12 00:26:46 WARN Failed to register uploaded object key=xvqn7jabp38ik4n4sqli3xm1zyd3dpd2.narinfo error="server returned 404: 404 page not found\n"14692026/11/12 00:26:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=014702026/11/12 00:26:46 INFO Vacuumed table table=pending_closures14712026/11/12 00:26:46 INFO Completed push id=1 objects=8 duration=12.376215ms14722026/11/12 00:26:46 INFO Completed push id=1 objects=6 duration=11.029022ms14732026/11/12 00:26:46 INFO Upload complete. (627ms)14742026/11/12 00:26:46 INFO Upload complete. (352ms)14752026/11/12 00:26:46 INFO Vacuumed table table=pending_objects1476=== NAME TestClientPushesUseOnePush1477 client_pushes_test.go:97: Retrieved narinfo from S3:1478 StorePath: /build/TestClientPushesUseOnePush3270469763/001/store/sgnr83gqg8ya15k8aavimmvm26kzrvi8-shared-dep1479 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1480 Compression: zstd1481 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821482 NarSize: 1361483 References: 1484 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1485=== NAME TestClientSharedPathCommittedMidPush1486 client_integration_test.go:728: Retrieved narinfo from S3:1487 StorePath: /build/TestClientSharedPathCommittedMidPush10807390/001/store/xvqn7jabp38ik4n4sqli3xm1zyd3dpd2-shared-dep1488 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1489 Compression: zstd1490 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821491 NarSize: 1361492 References: 1493 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n14942026/11/12 00:26:46 INFO Vacuumed table table=multipart_uploads14952026/11/12 00:26:46 INFO Vacuumed table table=closures1496 client_integration_test.go:728: Retrieved narinfo from S3:1497 StorePath: /build/TestClientSharedPathCommittedMidPush10807390/001/store/1k6nmsgf2rr2nz40hsy0nhy6dzb8ryrr-top1498 URL: nar/1wxsm42wg407vfq6ddb98kyphjn7n6lrdf1kwx6yp5iyz7iavf1r.nar.zst1499 Compression: zstd1500 NarHash: sha256:1wxsm42wg407vfq6ddb98kyphjn7n6lrdf1kwx6yp5iyz7iavf1r1501 NarSize: 2241502 References: /build/TestClientSharedPathCommittedMidPush10807390/001/store/xvqn7jabp38ik4n4sqli3xm1zyd3dpd2-shared-dep1503 CA: text:sha256:03n2lblxisigkbxj6ln481cdwzba61d937w52sd5msxw07dl9xs81504=== NAME TestClientPushesUseOnePush1505 client_pushes_test.go:97: Retrieved narinfo from S3:1506 StorePath: /build/TestClientPushesUseOnePush3270469763/001/store/dhz4k0i4psqbchbiyf8vv4c6yjyijr72-a1507 URL: nar/1a0w4kmz3npsrhny9yzv9dfgiq2szfj38n2fwcnr22791964wm1p.nar.zst1508 Compression: zstd1509 NarHash: sha256:1a0w4kmz3npsrhny9yzv9dfgiq2szfj38n2fwcnr22791964wm1p1510 NarSize: 2161511 References: /build/TestClientPushesUseOnePush3270469763/001/store/sgnr83gqg8ya15k8aavimmvm26kzrvi8-shared-dep1512 CA: text:sha256:12rm1sx756k8z38n76mjzn9g1ms7r1aah9qk3jwzrv1rf3k28yxf15132026/11/12 00:26:46 INFO Vacuumed table table=objects15142026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/1/complete1515 client_pushes_test.go:97: Retrieved narinfo from S3:1516 StorePath: /build/TestClientPushesUseOnePush3270469763/001/store/wkgz4myaay77yxrvgbq7d177cav4q4za-b1517 URL: nar/1a0w4kmz3npsrhny9yzv9dfgiq2szfj38n2fwcnr22791964wm1p.nar.zst1518 Compression: zstd1519 NarHash: sha256:1a0w4kmz3npsrhny9yzv9dfgiq2szfj38n2fwcnr22791964wm1p1520 NarSize: 2161521 References: /build/TestClientPushesUseOnePush3270469763/001/store/sgnr83gqg8ya15k8aavimmvm26kzrvi8-shared-dep1522 CA: text:sha256:12rm1sx756k8z38n76mjzn9g1ms7r1aah9qk3jwzrv1rf3k28yxf15232026/11/12 00:26:46 INFO Received complete push request method=POST path=/api/pushes/2/complete1524--- PASS: TestClientSharedPathCommittedMidPush (1.66s)1525=== CONT TestNARDeduplicationMetadataUploadBug15262026/11/12 00:26:46 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1527--- PASS: TestService_AuthMiddleware (0.64s)1528=== CONT TestReadProxyRangeRequest1529--- PASS: TestClientPushesUseOnePush (4.40s)1530=== CONT TestCreatePendingClosureRejectsOversizedNAR15312026/11/12 00:26:46 INFO Received uploads request method=POST path=/api/pending_closures1532=== CONT TestReadRedirectKeepsNarinfoProxied15332026/11/12 00:26:46 INFO Completed push id=2 objects=4 duration=4.23056ms1534--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1535--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.92s)1536=== CONT TestGCAdvisoryLockBlocksConcurrentRun1537--- PASS: TestPresentReportsOnlyClosureRoots (0.63s)1538=== CONT TestReadRedirectNar15392026/11/12 00:26:47 INFO Completed push id=1 objects=6 duration=119.848616ms1540--- PASS: TestPush_CompleteCommitsEveryRoot (0.64s)1541=== CONT TestObjectStatsTrigger1542--- PASS: TestPresentRefreshesOnlyStaleClosures (0.81s)1543=== CONT TestMultipartCleanup15442026-11-12 00:26:47.071 UTC [1385] ERROR: Push does not exist: id=115452026-11-12 00:26:47.071 UTC [1385] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE15462026-11-12 00:26:47.071 UTC [1385] STATEMENT: SELECT commit_push($1)1547--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.84s)1548=== CONT TestObjectStatsUpdatedOncePerCommit15492026/11/12 00:26:47 INFO Received push request method=POST path=/api/pushes1550--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.68s)1551=== CONT TestServerTLSConfig1552=== RUN TestServerTLSConfig/no_client_CA1553=== PAUSE TestServerTLSConfig/no_client_CA1554=== RUN TestServerTLSConfig/missing_CA_file1555=== PAUSE TestServerTLSConfig/missing_CA_file1556=== RUN TestServerTLSConfig/not_a_PEM_file1557=== PAUSE TestServerTLSConfig/not_a_PEM_file1558=== CONT TestGenerateLandingPage1559--- PASS: TestGenerateLandingPage (0.01s)1560=== CONT TestService_NativeMTLS15612026/11/12 00:26:47 INFO Received push request method=POST path=/api/pushes15622026/11/12 00:26:47 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15632026/11/12 00:26:47 INFO Uploading c2hc040g3ds2nmrpil1lwp1yjf08bkjj-unpinned-file.txt (128B)15642026/11/12 00:26:47 WARN Failed to register uploaded object key=c2hc040g3ds2nmrpil1lwp1yjf08bkjj.ls error="server returned 404: 404 page not found\n"15652026/11/12 00:26:47 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"15662026/11/12 00:26:47 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign15672026/11/12 00:26:47 INFO Signed narinfos id=2 count=115682026/11/12 00:26:47 INFO Uploading 1 narinfos15692026/11/12 00:26:47 INFO Received complete push request method=POST path=/api/pushes/2/complete15702026/11/12 00:26:47 WARN Failed to register uploaded object key=c2hc040g3ds2nmrpil1lwp1yjf08bkjj.narinfo error="server returned 404: 404 page not found\n"15712026/11/12 00:26:47 INFO Completed push id=2 objects=3 duration=4.854686ms15722026/11/12 00:26:47 INFO Upload complete. (377ms)15732026/11/12 00:26:47 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux15742026/11/12 00:26:47 WARN Refused reserved pin name=worker-x86_64-linux15752026/11/12 00:26:47 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux15762026/11/12 00:26:47 INFO Received create pin request method=POST path=/api/pins/my-app15772026/11/12 00:26:47 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1578--- PASS: TestCreatePin_ReservedPins (0.86s)1579=== CONT TestService_readinessHandler15802026/11/12 00:26:47 INFO Aborted multipart uploads count=0 kept=015812026/11/12 00:26:47 INFO Received uploads request method=POST path=/api/pending_closures15822026/11/12 00:26:47 INFO Received push request method=POST path=/api/pushes15832026/11/12 00:26:47 WARN Force mode enabled - objects will be deleted immediately without grace period15842026/11/12 00:26:47 INFO Object in database but missing from S3 key=5g3a2223pa6ymsgig1k1bf2502dm399x.ls15852026/11/12 00:26:47 WARN Found objects in DB but missing from S3, will re-upload count=115862026/11/12 00:26:47 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15872026/11/12 00:26:47 INFO Received complete push request method=POST path=/api/pushes/2/complete15882026/11/12 00:26:47 WARN Failed to register uploaded object key=5g3a2223pa6ymsgig1k1bf2502dm399x.ls error="server returned 404: 404 page not found\n"15892026/11/12 00:26:47 INFO Completed push id=2 objects=3 duration=9.277087ms15902026/11/12 00:26:47 INFO Upload complete. (145ms)1591=== NAME TestClientIntegration1592 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1593 client_integration_test.go:336: Decompressed .ls content (49 bytes):1594 {"version":1,"root":{"type":"regular","size":39}}1595--- PASS: TestReadRedirectUsesPublicS3URL (0.58s)1596=== CONT TestPendingCleanupUsesOneCutoff15972026/11/12 00:26:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=015982026/11/12 00:26:47 INFO Vacuumed table table=pending_closures15992026/11/12 00:26:47 INFO Vacuumed table table=pending_objects16002026/11/12 00:26:47 INFO Vacuumed table table=multipart_uploads16012026/11/12 00:26:47 INFO Vacuumed table table=closures16022026/11/12 00:26:47 INFO Vacuumed table table=objects1603--- PASS: TestGCSweepDeliversEachKeyOnce (0.77s)1604=== CONT TestService_healthCheckHandler16052026/11/12 00:26:47 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket67 days=21606--- PASS: TestEnsureAbortUploadsRule (0.62s)1607=== CONT TestMetricsCommit16082026/11/12 00:26:47 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1609--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.47s)1610=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1611--- PASS: TestMetricsInventory (0.67s)1612=== CONT TestIsValidUploadKey1613=== RUN TestIsValidUploadKey/narinfo1614=== PAUSE TestIsValidUploadKey/narinfo1615=== RUN TestIsValidUploadKey/nar_zst1616=== PAUSE TestIsValidUploadKey/nar_zst1617=== RUN TestIsValidUploadKey/nar_xz1618=== PAUSE TestIsValidUploadKey/nar_xz1619=== RUN TestIsValidUploadKey/nar_plain1620=== PAUSE TestIsValidUploadKey/nar_plain1621=== RUN TestIsValidUploadKey/listing1622=== PAUSE TestIsValidUploadKey/listing1623=== RUN TestIsValidUploadKey/build_log1624=== PAUSE TestIsValidUploadKey/build_log1625=== RUN TestIsValidUploadKey/build_log_home-manager_file1626=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1627=== RUN TestIsValidUploadKey/build_log_plus_in_name1628=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1629=== RUN TestIsValidUploadKey/build_log_question_mark1630=== PAUSE TestIsValidUploadKey/build_log_question_mark1631=== RUN TestIsValidUploadKey/build_log_equals1632=== PAUSE TestIsValidUploadKey/build_log_equals1633=== RUN TestIsValidUploadKey/realisation1634=== PAUSE TestIsValidUploadKey/realisation1635=== RUN TestIsValidUploadKey/realisation_plus_in_output1636=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1637=== RUN TestIsValidUploadKey/nix-cache-info1638=== PAUSE TestIsValidUploadKey/nix-cache-info1639=== RUN TestIsValidUploadKey/index.html1640=== PAUSE TestIsValidUploadKey/index.html1641=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1642=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1643=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1644=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1645=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1646=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1647=== RUN TestIsValidUploadKey/traversal1648=== PAUSE TestIsValidUploadKey/traversal1649=== RUN TestIsValidUploadKey/traversal_nar1650=== PAUSE TestIsValidUploadKey/traversal_nar1651=== RUN TestIsValidUploadKey/absolute1652=== PAUSE TestIsValidUploadKey/absolute1653=== RUN TestIsValidUploadKey/empty_key1654=== PAUSE TestIsValidUploadKey/empty_key1655=== RUN TestIsValidUploadKey/unknown_type1656=== PAUSE TestIsValidUploadKey/unknown_type1657=== CONT TestReadProxyInvalidPath16582026/11/12 00:26:47 INFO lead: released remote=192.0.2.1:1234 generation=31659--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (7.73s)1660=== CONT TestPendingClosureWriteTimeout1661=== RUN TestPendingClosureWriteTimeout/empty1662=== PAUSE TestPendingClosureWriteTimeout/empty1663=== RUN TestPendingClosureWriteTimeout/400_objects1664=== PAUSE TestPendingClosureWriteTimeout/400_objects1665=== RUN TestPendingClosureWriteTimeout/670k_objects1666=== PAUSE TestPendingClosureWriteTimeout/670k_objects1667=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1668=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1669=== CONT TestReadProxyContentNegotiation1670--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.75s)1671=== CONT TestReadProxyNarinfoAlreadyDecompressed1672--- PASS: TestReadProxyRangeRequest (0.56s)1673=== CONT TestReadProxy4041674--- PASS: TestReadRedirectKeepsNarinfoProxied (0.61s)1675=== CONT TestReadProxyDisabled16762026/11/12 00:26:47 INFO Starting cleanup of old closures method=DELETE path=/api/closures16772026/11/12 00:26:47 INFO Garbage collection started1678--- PASS: TestObjectStatsUpdatedOncePerCommit (0.45s)1679=== CONT TestReadProxyNarinfo16802026/11/12 00:26:47 INFO Aborted multipart uploads count=0 kept=01681--- PASS: TestObjectStatsTrigger (0.59s)1682=== CONT TestReadProxyConditionalGet16832026/11/12 00:26:47 INFO Received uploads request method=POST path=/api/pending_closures1684=== NAME TestOrphanedObjectsGC1685 orphaned_objects_gc_test.go:287: GC Test Summary:1686 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1687 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1688 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1689 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1690 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1691--- PASS: TestOrphanedObjectsGC (1.17s)1692=== CONT TestReadProxyNarStreaming1693--- PASS: TestReadRedirectNar (0.70s)1694=== CONT TestReadProxyRootRedirectsToIndexHTML1695--- PASS: TestConnectWaitsForAPeerMigration (7.96s)1696=== CONT TestReadProxyHead16972026/11/12 00:26:47 WARN readiness check failed error="closed pool"16982026/11/12 00:26:47 WARN Force mode enabled - objects will be deleted immediately without grace period1699--- PASS: TestService_readinessHandler (0.47s)1700=== CONT TestReadProxyOutlastsServerWriteTimeout1701=== NAME TestNARDeduplicationMetadataUploadBug1702 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3103893238/001/store/7i7j8bjxnkfh4m951bfx7fqjyiib0gpg-file1.txt1703--- PASS: TestService_healthCheckHandler (0.39s)1704=== CONT TestCacheConfigHandlerMaxNarSize1705--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1706=== CONT TestIsValidCachePath1707=== RUN TestIsValidCachePath/narinfo1708=== PAUSE TestIsValidCachePath/narinfo1709=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1710=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1711=== RUN TestIsValidCachePath/nar_zst1712=== PAUSE TestIsValidCachePath/nar_zst1713=== RUN TestIsValidCachePath/nar_xz1714=== PAUSE TestIsValidCachePath/nar_xz1715=== RUN TestIsValidCachePath/nar_bz21716=== PAUSE TestIsValidCachePath/nar_bz21717=== RUN TestIsValidCachePath/nar_uncompressed1718=== PAUSE TestIsValidCachePath/nar_uncompressed1719=== RUN TestIsValidCachePath/ls1720=== PAUSE TestIsValidCachePath/ls1721=== RUN TestIsValidCachePath/log1722=== PAUSE TestIsValidCachePath/log1723=== RUN TestIsValidCachePath/realisation1724=== PAUSE TestIsValidCachePath/realisation1725=== RUN TestIsValidCachePath/nix-cache-info1726=== PAUSE TestIsValidCachePath/nix-cache-info1727=== RUN TestIsValidCachePath/index.html1728=== PAUSE TestIsValidCachePath/index.html1729=== RUN TestIsValidCachePath/traversal_parent1730=== PAUSE TestIsValidCachePath/traversal_parent1731=== RUN TestIsValidCachePath/traversal_in_middle1732=== PAUSE TestIsValidCachePath/traversal_in_middle1733=== RUN TestIsValidCachePath/invalid_char_e1734=== PAUSE TestIsValidCachePath/invalid_char_e1735=== RUN TestIsValidCachePath/invalid_char_u1736=== PAUSE TestIsValidCachePath/invalid_char_u1737=== RUN TestIsValidCachePath/random_path1738=== PAUSE TestIsValidCachePath/random_path1739=== RUN TestIsValidCachePath/empty1740=== PAUSE TestIsValidCachePath/empty1741=== RUN TestIsValidCachePath/leading_slash1742=== PAUSE TestIsValidCachePath/leading_slash1743=== RUN TestIsValidCachePath/wrong_extension1744=== PAUSE TestIsValidCachePath/wrong_extension1745=== RUN TestIsValidCachePath/short_hash1746=== PAUSE TestIsValidCachePath/short_hash1747=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed17482026/11/12 00:26:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=017492026/11/12 00:26:47 INFO Received uploads request method=POST path=/api/pending_closures17502026/11/12 00:26:47 INFO Vacuumed table table=pending_closures17512026/11/12 00:26:47 INFO Received cleanup request method=DELETE path=/api/pending_closures17522026/11/12 00:26:47 INFO Vacuumed table table=pending_objects17532026/11/12 00:26:47 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket75/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""17542026/11/12 00:26:47 INFO Aborted multipart uploads count=0 kept=117552026/11/12 00:26:47 INFO Vacuumed table table=multipart_uploads17562026/11/12 00:26:47 INFO Vacuumed table table=closures17572026/11/12 00:26:47 INFO Received cleanup request method=DELETE path=/api/pending_closures17582026/11/12 00:26:47 INFO Received uploads request method=POST path=/api/pending_closures17592026/11/12 00:26:47 INFO Vacuumed table table=objects17602026/11/12 00:26:47 INFO Aborted multipart uploads count=1 kept=01761--- PASS: TestMultipartCleanup (0.74s)1762=== CONT TestProxyWriteTimeout/narinfo1763=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed17642026/11/12 00:26:47 INFO Received cleanup request method=DELETE path=/api/pending_closures17652026/11/12 00:26:47 INFO Received push request method=POST path=/api/pushes17662026/11/12 00:26:47 INFO Received complete push request method=POST path=/api/pushes/1/complete17672026/11/12 00:26:47 INFO Received uploads request method=POST path=/api/pending_closures1768--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.48s)1769=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed17702026/11/12 00:26:47 INFO Received push request method=POST path=/api/pushes17712026/11/12 00:26:47 INFO Completed push id=1 objects=4 duration=69.863111ms1772--- PASS: TestReadProxyInvalidPath (0.45s)1773=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost17742026/11/12 00:26:47 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17752026/11/12 00:26:47 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17762026/11/12 00:26:47 INFO Completed upload id=2 objects=2 duration=3.104849ms1777--- PASS: TestMetricsCommit (0.53s)1778=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed17792026/11/12 00:26:47 INFO Received complete push request method=POST path=/api/pushes/3/complete17802026/11/12 00:26:47 WARN Failed to register uploaded object key=5g3a2223pa6ymsgig1k1bf2502dm399x.ls error="server returned 404: 404 page not found\n"17812026/11/12 00:26:47 INFO Completed push id=3 objects=3 duration=4.155858ms17822026/11/12 00:26:47 INFO Upload complete. (221ms)1783=== NAME TestClientIntegration1784 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1785 client_integration_test.go:353: Decompressed .ls content (49 bytes):1786 {"version":1,"root":{"type":"regular","size":39}}1787 client_integration_test.go:364: Testing garbage collection...1788--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.48s)1789=== CONT TestProxyWriteTimeout/unknown_size1790=== CONT TestProxyWriteTimeout/10_GiB_nar1791=== CONT TestProxyWriteTimeout/1_GiB_nar1792--- PASS: TestProxyWriteTimeout (0.01s)1793 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1794 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1795 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1796 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1797=== CONT TestGCEndsOnShutdown/before_the_run1798--- PASS: TestReadProxyDisabled (0.64s)1799=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark18002026/11/12 00:26:48 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1801--- PASS: TestReadProxyNarinfo (0.64s)1802=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18032026/11/12 00:26:48 WARN mTLS auth: subject not in bound subjects subject="CN=reader"18042026/11/12 00:26:48 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1805--- PASS: TestService_NativeMTLS (1.15s)1806=== CONT TestResolveDBConnectionString/flag_wins1807=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1808=== CONT TestResolveDBConnectionString/nothing_configured1809=== CONT TestResolveDBConnectionString/missing_file_is_an_error1810=== CONT TestResolveDBConnectionString/file_when_flag_empty1811=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1812--- PASS: TestResolveDBConnectionString (1.36s)1813 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1814 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1815 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1816 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1817 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)18182026/11/12 00:26:48 INFO Received uploads request method=POST path=/1819=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key18202026/11/12 00:26:48 INFO Received request for more parts method=POST path=/1821=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key18222026/11/12 00:26:48 INFO Received complete multipart upload request method=POST path=/1823=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18242026/11/12 00:26:48 INFO Received uploads request method=POST path=/1825--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1826 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1827 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1828 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1829 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1830=== CONT TestPush_RejectsBadRequests/root_not_in_objects18312026/11/12 00:26:48 INFO Received push request method=POST path=/api/pushes1832=== CONT TestPush_RejectsBadRequests/no_objects18332026/11/12 00:26:48 INFO Received push request method=POST path=/api/pushes1834=== CONT TestPush_RejectsBadRequests/bad_root18352026/11/12 00:26:48 INFO Received push request method=POST path=/api/pushes1836=== CONT TestPush_RejectsBadRequests/no_roots18372026/11/12 00:26:48 INFO Received push request method=POST path=/api/pushes1838=== CONT TestClientErrorHandling/InvalidStorePath1839--- PASS: TestPush_RejectsBadRequests (2.41s)1840 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.01s)1841 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1842 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1843 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1844--- PASS: TestReadProxyConditionalGet (0.70s)1845=== CONT TestClientErrorHandling/ServerNotAvailable18462026/11/12 00:26:48 INFO Received push request method=POST path=/api/pushes18472026/11/12 00:26:48 INFO Starting cleanup of old closures method=DELETE path=/api/closures18482026/11/12 00:26:48 INFO Garbage collection started18492026/11/12 00:26:48 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18502026/11/12 00:26:48 INFO Uploading 7i7j8bjxnkfh4m951bfx7fqjyiib0gpg-file1.txt (160B)1851--- PASS: TestReadProxy404 (0.88s)1852=== CONT TestClientErrorHandling/InvalidAuthToken1853--- PASS: TestReadProxyNarStreaming (0.69s)1854=== CONT TestCacheConfigHandler/full_config,_no_issuer1855=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1856=== CONT TestCacheConfigHandler/no_signing_keys1857=== CONT TestCacheConfigHandler/no_cache_url_configured1858=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1859--- PASS: TestCacheConfigHandler (0.00s)1860 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1861 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1862 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1863 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)18642026/11/12 00:26:48 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign18652026/11/12 00:26:48 WARN Failed to register uploaded object key=7i7j8bjxnkfh4m951bfx7fqjyiib0gpg.ls error="server returned 404: 404 page not found\n"18662026/11/12 00:26:48 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"18672026/11/12 00:26:48 INFO Signed narinfos id=1 count=118682026/11/12 00:26:48 INFO Uploading 1 narinfos18692026/11/12 00:26:48 INFO Aborted multipart uploads count=0 kept=018702026/11/12 00:26:48 INFO Received complete push request method=POST path=/api/pushes/1/complete18712026/11/12 00:26:48 WARN Failed to register uploaded object key=7i7j8bjxnkfh4m951bfx7fqjyiib0gpg.narinfo error="server returned 404: 404 page not found\n"1872--- PASS: TestDeleteClosuresRacingPins (8.66s)1873=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1874--- PASS: TestReadProxyContentNegotiation (0.93s)1875=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart18762026/11/12 00:26:48 INFO Received complete multipart upload request method=POST path=/18772026/11/12 00:26:48 INFO Received uploads request method=POST path=/api/pending_closures18782026/11/12 00:26:48 INFO Completed push id=1 objects=3 duration=13.762768ms18792026/11/12 00:26:48 INFO Upload complete. (496ms)1880=== NAME TestNARDeduplicationMetadataUploadBug1881 metadata_upload_test.go:54: Retrieved narinfo from S3:1882 StorePath: /build/TestNARDeduplicationMetadataUploadBug3103893238/001/store/7i7j8bjxnkfh4m951bfx7fqjyiib0gpg-file1.txt1883 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1884 Compression: zstd1885 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1886 NarSize: 1601887 References: 1888 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1889 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1890 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1891 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}18922026/11/12 00:26:48 WARN Force mode enabled - objects will be deleted immediately without grace period1893=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure18942026/11/12 00:26:48 INFO Received uploads request method=POST path=/1895--- PASS: TestReadProxyHead (0.77s)1896=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts18972026/11/12 00:26:48 INFO Received request for more parts method=POST path=/18982026/11/12 00:26:48 INFO Received uploads request method=POST path=/api/pending_closures18992026/11/12 00:26:48 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=019002026/11/12 00:26:48 INFO Vacuumed table table=pending_closures19012026/11/12 00:26:48 INFO Vacuumed table table=pending_objects19022026/11/12 00:26:48 INFO Vacuumed table table=multipart_uploads19032026/11/12 00:26:48 INFO Vacuumed table table=closures19042026/11/12 00:26:48 INFO Vacuumed table table=objects1905=== CONT TestParseSingleRange/none1906=== CONT TestParseSingleRange/start_far_past_EOF1907=== CONT TestParseSingleRange/start_past_EOF1908=== CONT TestParseSingleRange/single_byte1909=== CONT TestParseSingleRange/suffix_exceeds_size1910=== CONT TestParseSingleRange/suffix1911=== CONT TestParseSingleRange/end_clamped_to_size1912=== CONT TestParseSingleRange/open-ended1913=== CONT TestParseSingleRange/closed1914=== CONT TestParseSingleRange/malformed_end_before_start1915=== CONT TestParseSingleRange/malformed_both_empty1916=== CONT TestParseSingleRange/malformed_no_dash1917=== CONT TestParseSingleRange/multi-range_ignored1918=== CONT TestParseSingleRange/unknown_unit1919--- PASS: TestParseSingleRange (0.00s)1920 --- PASS: TestParseSingleRange/none (0.00s)1921 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1922 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1923 --- PASS: TestParseSingleRange/single_byte (0.00s)1924 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1925 --- PASS: TestParseSingleRange/suffix (0.00s)1926 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1927 --- PASS: TestParseSingleRange/open-ended (0.00s)1928 --- PASS: TestParseSingleRange/closed (0.00s)1929 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1930 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1931 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1932 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1933 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1934=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale19352026/11/12 00:26:48 WARN Rate limiter enabled after throttle name=s3-test rate=519362026/11/12 00:26:48 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1937=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1938 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101939 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001940--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (9.01s)1941=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1942=== NAME TestNARDeduplicationMetadataUploadBug1943 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug3103893238/001/store/dxakli8dhwhhi7zlcbp8n7bmwq2b3l9y-file2.txt1944--- PASS: TestReadProxyRootRedirectsToIndexHTML (1.01s)19452026/11/12 00:26:48 INFO Received uploads request method=POST path=/api/pending_closures1946=== CONT TestService_RequireScope_OIDC/builder_may_write1947=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1948=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19492026/11/12 00:26:48 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1950=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1951=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1952=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1953=== CONT TestService_RequireScope_OIDC/writer_implies_read1954=== CONT TestService_RequireScope_OIDC/reader_may_read1955=== CONT TestService_RequireScope_OIDC/static_token_may_write1956=== CONT TestService_RequireScope_OIDC/static_token_may_admin1957=== CONT TestService_RequireScope_OIDC/reader_may_not_write1958=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1959=== CONT TestService_RequireScope_OIDC/ops_may_not_write1960=== CONT TestService_RequireScope_OIDC/ops_may_admin1961=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1962--- PASS: TestService_RequireScope_OIDC (0.64s)1963 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.01s)1964 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1965 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1966 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1967 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1968 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1969 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1970 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1971 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1972 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1973 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1974 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1975 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1976 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1977=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected19782026/11/12 00:26:48 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1979=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1980=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected19812026/11/12 00:26:48 WARN Authentication failed token_preview=eyJhbGciOi...sGNfeXpjwQ token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1982=== CONT TestServerTLSConfig/no_client_CA1983=== CONT TestServerTLSConfig/missing_CA_file1984=== CONT TestServerTLSConfig/not_a_PEM_file1985--- PASS: TestService_AuthMiddleware_OIDC (0.60s)1986 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1987 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1988 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1989 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)1990=== CONT TestIsValidUploadKey/narinfo1991--- PASS: TestServerTLSConfig (0.00s)1992 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1993 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1994 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1995=== CONT TestIsValidUploadKey/absolute1996=== CONT TestIsValidUploadKey/traversal_nar1997=== CONT TestIsValidUploadKey/traversal1998=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1999=== CONT TestIsValidUploadKey/nar_key,_narinfo_type2000=== CONT TestIsValidUploadKey/narinfo_key,_nar_type
loading 678 hidden lines…
26792026/11/12 00:27:04 INFO Received push request method=POST path=/26802026/11/12 00:27:04 INFO Received push request method=POST path=/26812026/11/12 00:27:04 INFO Received uploads request method=POST path=/26822026/11/12 00:27:04 INFO Received push request method=POST path=/26832026/11/12 00:27:04 INFO Received uploads request method=POST path=/26842026/11/12 00:27:04 INFO Aborted multipart uploads count=0 kept=026852026/11/12 00:27:04 INFO Aborted multipart uploads count=0 kept=026862026/11/12 00:27:04 INFO Received uploads request method=POST path=/26872026/11/12 00:27:04 INFO Received complete upload request method=POST path=/26882026/11/12 00:27:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026892026/11/12 00:27:04 INFO Completed upload id=11 objects=6 duration=1.269492ms26902026/11/12 00:27:04 INFO Received uploads request method=POST path=/26912026/11/12 00:27:04 INFO Received uploads request method=POST path=/26922026/11/12 00:27:04 INFO Vacuumed table table=pending_closures26932026/11/12 00:27:04 INFO Vacuumed table table=pending_objects26942026/11/12 00:27:04 INFO Vacuumed table table=multipart_uploads26952026/11/12 00:27:04 INFO Vacuumed table table=closures26962026/11/12 00:27:04 INFO Vacuumed table table=objects26972026/11/12 00:27:04 INFO Received complete push request method=POST path=/26982026/11/12 00:27:04 INFO Completed push id=14 objects=3 duration=2.355722ms26992026/11/12 00:27:04 INFO Received complete push request method=POST path=/27002026/11/12 00:27:04 INFO Received push request method=POST path=/27012026/11/12 00:27:04 INFO Received uploads request method=POST path=/27022026/11/12 00:27:04 INFO Received uploads request method=POST path=/27032026/11/12 00:27:04 INFO Completed push id=3 objects=22 duration=7.57755ms27042026/11/12 00:27:04 INFO Received uploads request method=POST path=/27052026/11/12 00:27:05 INFO Received complete push request method=POST path=/27062026/11/12 00:27:05 INFO Received push request method=POST path=/27072026/11/12 00:27:05 INFO Received uploads request method=POST path=/27082026/11/12 00:27:05 INFO Received complete push request method=POST path=/27092026/11/12 00:27:05 INFO Received complete push request method=POST path=/27102026/11/12 00:27:05 INFO Received uploads request method=POST path=/27112026/11/12 00:27:05 INFO Completed push id=5 objects=6 duration=3.914796ms27122026/11/12 00:27:05 INFO Completed push id=1 objects=8 duration=11.375186ms27132026/11/12 00:27:05 INFO Completed push id=1 objects=5 duration=8.586039ms27142026/11/12 00:27:05 INFO Received push request method=POST path=/27152026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27162026/11/12 00:27:05 INFO Received complete push request method=POST path=/27172026/11/12 00:27:05 INFO Completed push id=12 objects=15 duration=2.035179ms27182026/11/12 00:27:05 INFO Completed upload id=6 objects=7 duration=4.965066ms27192026/11/12 00:27:05 INFO Received uploads request method=POST path=/27202026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=027212026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period27222026/11/12 00:27:05 INFO Received uploads request method=POST path=/27232026/11/12 00:27:05 INFO Received push request method=POST path=/27242026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=027252026/11/12 00:27:05 INFO Received push request method=POST path=/27262026/11/12 00:27:05 INFO Received push request method=POST path=/27272026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period27282026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27292026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027302026/11/12 00:27:05 INFO Received push request method=POST path=/27312026/11/12 00:27:05 INFO Received push request method=POST path=/27322026/11/12 00:27:05 INFO Vacuumed table table=pending_closures27332026/11/12 00:27:05 INFO Completed upload id=5 objects=7 duration=3.030468ms27342026/11/12 00:27:05 INFO Received uploads request method=POST path=/27352026/11/12 00:27:05 INFO Vacuumed table table=pending_objects27362026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads27372026/11/12 00:27:05 INFO Received complete push request method=POST path=/27382026/11/12 00:27:05 INFO Vacuumed table table=closures27392026/11/12 00:27:05 INFO Vacuumed table table=objects27402026/11/12 00:27:05 INFO Received push request method=POST path=/27412026/11/12 00:27:05 INFO Completed push id=2 objects=25 duration=4.33164ms27422026/11/12 00:27:05 INFO Received push request method=POST path=/27432026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=027442026/11/12 00:27:05 INFO Received push request method=POST path=/27452026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27462026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=027472026/11/12 00:27:05 INFO Completed upload id=6 objects=9 duration=1.555974ms27482026/11/12 00:27:05 INFO Vacuumed table table=pending_closures27492026/11/12 00:27:05 INFO Vacuumed table table=pending_objects27502026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads27512026/11/12 00:27:05 INFO Vacuumed table table=closures27522026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=027532026/11/12 00:27:05 INFO Vacuumed table table=objects27542026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27552026/11/12 00:27:05 INFO Completed upload id=2 objects=2 duration=1.773337ms27562026/11/12 00:27:05 INFO Received push request method=POST path=/27572026/11/12 00:27:05 INFO Received uploads request method=POST path=/27582026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27592026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27602026/11/12 00:27:05 INFO Completed upload id=10 objects=6 duration=2.368002ms27612026/11/12 00:27:05 INFO Completed upload id=1 objects=15 duration=4.782725ms27622026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27632026/11/12 00:27:05 INFO Completed upload id=5 objects=5 duration=1.936298ms27642026/11/12 00:27:05 INFO Received complete push request method=POST path=/27652026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=027662026/11/12 00:27:05 INFO Completed push id=6 objects=5 duration=2.267081ms27672026/11/12 00:27:05 INFO Received uploads request method=POST path=/27682026/11/12 00:27:05 INFO Received uploads request method=POST path=/27692026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=027702026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027712026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period27722026/11/12 00:27:05 INFO Received uploads request method=POST path=/27732026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027742026/11/12 00:27:05 INFO Vacuumed table table=pending_closures27752026/11/12 00:27:05 INFO Received uploads request method=POST path=/27762026/11/12 00:27:05 INFO Vacuumed table table=pending_closures27772026/11/12 00:27:05 INFO Vacuumed table table=pending_objects27782026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads27792026/11/12 00:27:05 INFO Vacuumed table table=pending_objects27802026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads27812026/11/12 00:27:05 INFO Vacuumed table table=closures27822026/11/12 00:27:05 INFO Vacuumed table table=objects27832026/11/12 00:27:05 INFO Vacuumed table table=closures27842026/11/12 00:27:05 INFO Received push request method=POST path=/27852026/11/12 00:27:05 INFO Vacuumed table table=objects27862026/11/12 00:27:05 INFO Received uploads request method=POST path=/27872026/11/12 00:27:05 INFO Received complete upload request method=POST path=/27882026/11/12 00:27:05 INFO Completed upload id=3 objects=5 duration=1.589335ms27892026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=027902026/11/12 00:27:05 INFO Received uploads request method=POST path=/27912026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027922026/11/12 00:27:05 INFO Vacuumed table table=pending_closures27932026/11/12 00:27:05 INFO Vacuumed table table=pending_objects27942026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads27952026/11/12 00:27:05 INFO Vacuumed table table=closures27962026/11/12 00:27:05 INFO Vacuumed table table=objects27972026/11/12 00:27:05 INFO Received push request method=POST path=/27982026/11/12 00:27:05 INFO Received push request method=POST path=/27992026/11/12 00:27:05 INFO Received uploads request method=POST path=/28002026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028012026/11/12 00:27:05 INFO Received uploads request method=POST path=/28022026/11/12 00:27:05 INFO Received uploads request method=POST path=/28032026/11/12 00:27:05 INFO Received push request method=POST path=/28042026/11/12 00:27:05 INFO Received push request method=POST path=/28052026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028062026/11/12 00:27:05 INFO Received complete upload request method=POST path=/28072026/11/12 00:27:05 INFO Completed upload id=3 objects=24 duration=1.943018ms28082026/11/12 00:27:05 INFO Received uploads request method=POST path=/28092026/11/12 00:27:05 INFO Received push request method=POST path=/28102026/11/12 00:27:05 INFO Received push request method=POST path=/28112026/11/12 00:27:05 INFO Received complete upload request method=POST path=/28122026/11/12 00:27:05 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28132026/11/12 00:27:05 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28142026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28152026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28162026/11/12 00:27:05 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28172026/11/12 00:27:05 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28182026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28192026/11/12 00:27:05 INFO Received uploads request method=POST path=/28202026/11/12 00:27:05 INFO Completed upload id=10 objects=6 duration=3.861936ms28212026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028222026/11/12 00:27:05 INFO Vacuumed table table=pending_closures28232026/11/12 00:27:05 INFO Received complete push request method=POST path=/28242026/11/12 00:27:05 INFO Received uploads request method=POST path=/28252026/11/12 00:27:05 INFO Completed push id=6 objects=26 duration=4.617343ms28262026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028272026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028282026/11/12 00:27:05 INFO Vacuumed table table=pending_objects28292026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads28302026/11/12 00:27:05 INFO Vacuumed table table=closures28312026/11/12 00:27:05 INFO Vacuumed table table=objects28322026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028332026/11/12 00:27:05 INFO Received uploads request method=POST path=/28342026/11/12 00:27:05 INFO Vacuumed table table=pending_closures28352026/11/12 00:27:05 INFO Vacuumed table table=pending_objects28362026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads28372026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028382026/11/12 00:27:05 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28392026/11/12 00:27:05 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28402026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28412026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28422026/11/12 00:27:05 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28432026/11/12 00:27:05 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28442026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28452026/11/12 00:27:05 INFO Vacuumed table table=closures28462026/11/12 00:27:05 INFO Received complete upload request method=POST path=/28472026/11/12 00:27:05 INFO Vacuumed table table=objects28482026/11/12 00:27:05 INFO Completed upload id=3 objects=25 duration=2.369883ms28492026/11/12 00:27:05 INFO Received complete upload request method=POST path=/28502026/11/12 00:27:05 INFO Completed upload id=15 objects=6 duration=958.569µs28512026/11/12 00:27:05 INFO Received complete upload request method=POST path=/28522026/11/12 00:27:05 INFO Completed upload id=11 objects=6 duration=2.217061ms28532026/11/12 00:27:05 INFO Received complete push request method=POST path=/28542026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028552026/11/12 00:27:05 INFO Received push request method=POST path=/28562026/11/12 00:27:05 INFO Completed push id=4 objects=4 duration=3.683635ms28572026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period28582026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028592026/11/12 00:27:05 INFO Received uploads request method=POST path=/28602026/11/12 00:27:05 INFO Vacuumed table table=pending_closures28612026/11/12 00:27:05 INFO Received push request method=POST path=/28622026/11/12 00:27:05 INFO Vacuumed table table=pending_objects28632026/11/12 00:27:05 INFO Received push request method=POST path=/28642026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads28652026/11/12 00:27:05 INFO Vacuumed table table=closures28662026/11/12 00:27:05 INFO Received complete push request method=POST path=/28672026/11/12 00:27:05 INFO Vacuumed table table=objects28682026/11/12 00:27:05 INFO Received complete push request method=POST path=/28692026/11/12 00:27:05 INFO Completed push id=7 objects=20 duration=2.473883ms28702026/11/12 00:27:05 INFO Received complete push request method=POST path=/28712026/11/12 00:27:05 INFO Completed push id=2 objects=10 duration=2.480743ms28722026/11/12 00:27:05 INFO Completed push id=8 objects=6 duration=1.584755ms28732026/11/12 00:27:05 INFO Received push request method=POST path=/28742026/11/12 00:27:05 INFO Received uploads request method=POST path=/28752026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028762026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028772026/11/12 00:27:05 INFO Received complete push request method=POST path=/28782026/11/12 00:27:05 INFO Completed push id=6 objects=6 duration=1.811276ms28792026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028802026/11/12 00:27:05 INFO Received uploads request method=POST path=/28812026/11/12 00:27:05 INFO Vacuumed table table=pending_closures28822026/11/12 00:27:05 INFO Received push request method=POST path=/28832026/11/12 00:27:05 INFO Vacuumed table table=pending_objects28842026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads28852026/11/12 00:27:05 INFO Vacuumed table table=closures28862026/11/12 00:27:05 INFO Received complete push request method=POST path=/28872026/11/12 00:27:05 INFO Vacuumed table table=objects28882026/11/12 00:27:05 INFO Completed push id=12 objects=7 duration=2.312742ms28892026/11/12 00:27:05 INFO Received push request method=POST path=/28902026/11/12 00:27:05 INFO Received push request method=POST path=/28912026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028922026/11/12 00:27:05 INFO Received push request method=POST path=/28932026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=028942026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028952026/11/12 00:27:05 INFO Vacuumed table table=pending_closures28962026/11/12 00:27:05 INFO Vacuumed table table=pending_objects28972026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads28982026/11/12 00:27:05 INFO Vacuumed table table=closures28992026/11/12 00:27:05 INFO Vacuumed table table=objects29002026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=029012026/11/12 00:27:05 INFO Received complete push request method=POST path=/29022026/11/12 00:27:05 INFO Received complete push request method=POST path=/29032026/11/12 00:27:05 INFO Completed push id=9 objects=5 duration=3.087349ms29042026/11/12 00:27:05 INFO Received complete push request method=POST path=/29052026/11/12 00:27:05 INFO Completed push id=16 objects=27 duration=2.320582ms29062026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29072026/11/12 00:27:05 INFO Completed push id=8 objects=7 duration=2.307042ms29082026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=029092026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029102026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period29112026/11/12 00:27:05 INFO Received uploads request method=POST path=/29122026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29132026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=029142026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=029152026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029162026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period29172026/11/12 00:27:05 INFO Received uploads request method=POST path=/29182026/11/12 00:27:05 INFO Received uploads request method=POST path=/29192026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29202026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029212026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29222026/11/12 00:27:05 INFO Received uploads request method=POST path=/29232026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29242026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29252026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29262026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period29272026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29282026/11/12 00:27:05 INFO Received complete push request method=POST path=/29292026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29302026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period29312026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=029322026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029332026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29342026/11/12 00:27:05 INFO Vacuumed table table=closures29352026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29362026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029372026/11/12 00:27:05 INFO Completed push id=5 objects=3 duration=10.543158ms29382026/11/12 00:27:05 INFO Vacuumed table table=closures29392026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029402026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29412026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period29422026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29432026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029442026/11/12 00:27:05 INFO Vacuumed table table=objects29452026/11/12 00:27:05 INFO Vacuumed table table=closures29462026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29472026/11/12 00:27:05 INFO Vacuumed table table=closures29482026/11/12 00:27:05 INFO Vacuumed table table=objects29492026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29502026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29512026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29522026/11/12 00:27:05 INFO Vacuumed table table=objects29532026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29542026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29552026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29562026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29572026/11/12 00:27:05 INFO Vacuumed table table=objects29582026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29592026/11/12 00:27:05 INFO Vacuumed table table=closures29602026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29612026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=029622026/11/12 00:27:05 INFO Vacuumed table table=closures29632026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29642026/11/12 00:27:05 INFO Received uploads request method=POST path=/29652026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=029662026/11/12 00:27:05 INFO Vacuumed table table=closures29672026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29682026/11/12 00:27:05 INFO Vacuumed table table=objects29692026/11/12 00:27:05 INFO Vacuumed table table=objects29702026/11/12 00:27:05 INFO Vacuumed table table=objects29712026/11/12 00:27:05 INFO Received push request method=POST path=/29722026/11/12 00:27:05 INFO Vacuumed table table=closures29732026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29742026/11/12 00:27:05 INFO Received complete push request method=POST path=/29752026/11/12 00:27:05 INFO Vacuumed table table=objects29762026/11/12 00:27:05 INFO Completed push id=7 objects=27 duration=2.867607ms29772026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29782026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29792026/11/12 00:27:05 INFO Vacuumed table table=closures29802026/11/12 00:27:05 INFO Received complete upload request method=POST path=/29812026/11/12 00:27:05 INFO Vacuumed table table=objects29822026/11/12 00:27:05 INFO Completed upload id=9 objects=16 duration=1.734296ms29832026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=029842026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period29852026/11/12 00:27:05 INFO Received push request method=POST path=/29862026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029872026/11/12 00:27:05 INFO Received complete push request method=POST path=/29882026/11/12 00:27:05 INFO Vacuumed table table=pending_closures29892026/11/12 00:27:05 INFO Received push request method=POST path=/29902026/11/12 00:27:05 INFO Vacuumed table table=pending_objects29912026/11/12 00:27:05 INFO Completed push id=6 objects=5 duration=4.914506ms29922026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads29932026/11/12 00:27:05 INFO Vacuumed table table=closures29942026/11/12 00:27:05 INFO Vacuumed table table=objects29952026/11/12 00:27:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29962026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29972026/11/12 00:27:05 INFO Received push request method=POST path=/29982026/11/12 00:27:05 INFO Received uploads request method=POST path=/29992026/11/12 00:27:05 INFO Received complete push request method=POST path=/30002026/11/12 00:27:05 INFO Received push request method=POST path=/30012026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030022026/11/12 00:27:05 INFO Completed push id=18 objects=10 duration=2.097719ms30032026/11/12 00:27:05 INFO Received push request method=POST path=/30042026/11/12 00:27:05 INFO Vacuumed table table=pending_closures30052026/11/12 00:27:05 INFO Vacuumed table table=pending_objects30062026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads30072026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030082026/11/12 00:27:05 INFO Received complete upload request method=POST path=/30092026/11/12 00:27:05 INFO Vacuumed table table=closures30102026/11/12 00:27:05 INFO Completed upload id=3 objects=9 duration=2.421943ms30112026/11/12 00:27:05 INFO Received uploads request method=POST path=/30122026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030132026/11/12 00:27:05 INFO Vacuumed table table=objects30142026/11/12 00:27:05 INFO Vacuumed table table=pending_closures30152026/11/12 00:27:05 INFO Vacuumed table table=pending_objects30162026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads30172026/11/12 00:27:05 INFO Received push request method=POST path=/30182026/11/12 00:27:05 INFO Vacuumed table table=closures30192026/11/12 00:27:05 INFO Vacuumed table table=objects30202026/11/12 00:27:05 INFO Received push request method=POST path=/30212026/11/12 00:27:05 INFO Received push request method=POST path=/30222026/11/12 00:27:05 INFO Received complete push request method=POST path=/30232026/11/12 00:27:05 INFO Completed push id=8 objects=6 duration=1.757216ms30242026/11/12 00:27:05 INFO Received push request method=POST path=/30252026/11/12 00:27:05 INFO Received uploads request method=POST path=/30262026/11/12 00:27:05 INFO Received complete upload request method=POST path=/30272026/11/12 00:27:05 INFO Completed upload id=3 objects=8 duration=4.084098ms30282026/11/12 00:27:05 INFO Received complete push request method=POST path=/30292026/11/12 00:27:05 INFO Completed push id=10 objects=3 duration=4.890946ms30302026/11/12 00:27:05 INFO Received push request method=POST path=/30312026/11/12 00:27:05 INFO Received push request method=POST path=/30322026/11/12 00:27:05 INFO Received complete push request method=POST path=/30332026/11/12 00:27:05 INFO Received uploads request method=POST path=/30342026/11/12 00:27:05 INFO Received push request method=POST path=/30352026/11/12 00:27:05 INFO Completed push id=5 objects=6 duration=2.714825ms30362026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030372026/11/12 00:27:05 INFO Received uploads request method=POST path=/30382026/11/12 00:27:05 INFO Received uploads request method=POST path=/30392026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030402026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030412026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030422026/11/12 00:27:05 INFO Vacuumed table table=pending_closures30432026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period30442026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period30452026/11/12 00:27:05 INFO Vacuumed table table=pending_objects30462026/11/12 00:27:05 INFO Received uploads request method=POST path=/30472026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads30482026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030492026/11/12 00:27:05 INFO Vacuumed table table=closures30502026/11/12 00:27:05 INFO Received uploads request method=POST path=/30512026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period30522026/11/12 00:27:05 INFO Vacuumed table table=objects30532026/11/12 00:27:05 INFO Received uploads request method=POST path=/30542026/11/12 00:27:05 INFO Received complete push request method=POST path=/30552026/11/12 00:27:05 INFO Completed push id=6 objects=8 duration=4.755424ms30562026/11/12 00:27:05 INFO Received push request method=POST path=/30572026/11/12 00:27:05 INFO Received push request method=POST path=/30582026/11/12 00:27:05 INFO Received push request method=POST path=/30592026/11/12 00:27:05 INFO Received complete upload request method=POST path=/30602026/11/12 00:27:05 INFO Received push request method=POST path=/30612026/11/12 00:27:05 INFO Received push request method=POST path=/30622026/11/12 00:27:05 INFO Completed upload id=6 objects=24 duration=3.710415ms30632026/11/12 00:27:05 INFO Received uploads request method=POST path=/30642026/11/12 00:27:05 INFO Received complete push request method=POST path=/30652026/11/12 00:27:05 INFO Received complete upload request method=POST path=/30662026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030672026/11/12 00:27:05 INFO Completed upload id=4 objects=14 duration=3.336231ms30682026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period30692026/11/12 00:27:05 INFO Received uploads request method=POST path=/30702026/11/12 00:27:05 INFO Completed push id=5 objects=9 duration=12.81058ms30712026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030722026/11/12 00:27:05 INFO Vacuumed table table=pending_closures30732026/11/12 00:27:05 INFO Vacuumed table table=pending_objects30742026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads30752026/11/12 00:27:05 INFO Received uploads request method=POST path=/30762026/11/12 00:27:05 INFO Vacuumed table table=closures30772026/11/12 00:27:05 INFO Received push request method=POST path=/30782026/11/12 00:27:05 INFO Vacuumed table table=objects30792026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030802026/11/12 00:27:05 INFO Received push request method=POST path=/30812026/11/12 00:27:05 INFO Received complete push request method=POST path=/30822026/11/12 00:27:05 INFO Received uploads request method=POST path=/30832026/11/12 00:27:05 INFO Completed push id=6 objects=7 duration=4.041778ms30842026/11/12 00:27:05 INFO Received push request method=POST path=/30852026/11/12 00:27:05 INFO Received uploads request method=POST path=/30862026/11/12 00:27:05 INFO Received push request method=POST path=/30872026/11/12 00:27:05 INFO Received push request method=POST path=/30882026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=030892026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period30902026/11/12 00:27:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30912026/11/12 00:27:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."30922026/11/12 00:27:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30932026/11/12 00:27:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30942026/11/12 00:27:05 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."30952026/11/12 00:27:05 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30962026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30972026/11/12 00:27:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30982026/11/12 00:27:05 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30992026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31002026/11/12 00:27:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31012026/11/12 00:27:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31022026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031032026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=031042026/11/12 00:27:05 INFO Received complete push request method=POST path=/31052026/11/12 00:27:05 INFO Vacuumed table table=pending_closures31062026/11/12 00:27:05 INFO Completed push id=8 objects=6 duration=2.539544ms31072026/11/12 00:27:05 INFO Received uploads request method=POST path=/31082026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031092026/11/12 00:27:05 INFO Vacuumed table table=pending_closures31102026/11/12 00:27:05 INFO Vacuumed table table=pending_objects31112026/11/12 00:27:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31122026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31132026/11/12 00:27:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31142026/11/12 00:27:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31152026/11/12 00:27:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31162026/11/12 00:27:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31172026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31182026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31192026/11/12 00:27:05 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31202026/11/12 00:27:05 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31212026/11/12 00:27:05 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31222026/11/12 00:27:05 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31232026/11/12 00:27:05 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31242026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."31252026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31262026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31272026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31282026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31292026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."31302026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31312026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads31322026/11/12 00:27:05 INFO Vacuumed table table=pending_objects31332026/11/12 00:27:05 INFO Vacuumed table table=closures31342026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads31352026/11/12 00:27:05 INFO Vacuumed table table=closures31362026/11/12 00:27:05 INFO Vacuumed table table=objects31372026/11/12 00:27:05 INFO Received complete upload request method=POST path=/31382026/11/12 00:27:05 INFO Vacuumed table table=objects31392026/11/12 00:27:05 INFO Received uploads request method=POST path=/31402026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=031412026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=031422026/11/12 00:27:05 INFO Completed upload id=10 objects=17 duration=2.755826ms31432026/11/12 00:27:05 INFO Vacuumed table table=pending_closures31442026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period31452026/11/12 00:27:05 INFO Vacuumed table table=pending_objects31462026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads31472026/11/12 00:27:05 INFO Received uploads request method=POST path=/31482026/11/12 00:27:05 INFO Received uploads request method=POST path=/31492026/11/12 00:27:05 INFO Received complete upload request method=POST path=/31502026/11/12 00:27:05 INFO Vacuumed table table=closures31512026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031522026/11/12 00:27:05 INFO Vacuumed table table=objects31532026/11/12 00:27:05 INFO Vacuumed table table=pending_closures31542026/11/12 00:27:05 INFO Vacuumed table table=pending_objects31552026/11/12 00:27:05 INFO Received push request method=POST path=/31562026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads31572026/11/12 00:27:05 INFO Vacuumed table table=closures31582026/11/12 00:27:05 INFO Received complete push request method=POST path=/31592026/11/12 00:27:05 INFO Completed upload id=2 objects=26 duration=9.75839ms31602026/11/12 00:27:05 INFO Vacuumed table table=objects31612026/11/12 00:27:05 INFO Received complete upload request method=POST path=/31622026/11/12 00:27:05 INFO Received complete upload request method=POST path=/31632026/11/12 00:27:05 INFO Received uploads request method=POST path=/31642026/11/12 00:27:05 INFO Completed push id=15 objects=24 duration=5.649773ms31652026/11/12 00:27:05 INFO Completed upload id=9 objects=3 duration=2.502664ms31662026/11/12 00:27:05 INFO Completed upload id=7 objects=4 duration=2.345562ms31672026/11/12 00:27:05 INFO Received push request method=POST path=/31682026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031692026/11/12 00:27:05 INFO Received uploads request method=POST path=/31702026/11/12 00:27:05 INFO Received uploads request method=POST path=/31712026/11/12 00:27:05 INFO Received push request method=POST path=/31722026/11/12 00:27:05 INFO Received complete push request method=POST path=/31732026/11/12 00:27:05 INFO Vacuumed table table=pending_closures31742026/11/12 00:27:05 INFO Received uploads request method=POST path=/31752026/11/12 00:27:05 INFO Received push request method=POST path=/31762026/11/12 00:27:05 INFO Received push request method=POST path=/31772026/11/12 00:27:05 INFO Received push request method=POST path=/31782026/11/12 00:27:05 INFO Received uploads request method=POST path=/31792026/11/12 00:27:05 INFO Completed push id=9 objects=7 duration=6.447761ms31802026/11/12 00:27:05 INFO Received push request method=POST path=/31812026/11/12 00:27:05 INFO Vacuumed table table=pending_objects31822026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads31832026/11/12 00:27:05 INFO Vacuumed table table=closures31842026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=031852026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31862026/11/12 00:27:05 INFO Received push request method=POST path=/31872026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period31882026/11/12 00:27:05 INFO Received complete push request method=POST path=/31892026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=031902026/11/12 00:27:05 INFO Received uploads request method=POST path=/31912026/11/12 00:27:05 INFO Received push request method=POST path=/31922026/11/12 00:27:05 INFO Completed push id=19 objects=8 duration=6.364919ms31932026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031942026/11/12 00:27:05 INFO Vacuumed table table=objects31952026/11/12 00:27:05 INFO Received push request method=POST path=/31962026/11/12 00:27:05 INFO Received push request method=POST path=/31972026/11/12 00:27:05 INFO Received push request method=POST path=/31982026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=031992026/11/12 00:27:05 INFO Vacuumed table table=pending_closures32002026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032012026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period32022026/11/12 00:27:05 INFO Vacuumed table table=pending_objects32032026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads32042026/11/12 00:27:05 INFO Vacuumed table table=closures32052026/11/12 00:27:05 INFO Vacuumed table table=objects32062026/11/12 00:27:05 INFO Received uploads request method=POST path=/32072026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032082026/11/12 00:27:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32092026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32102026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032112026/11/12 00:27:05 INFO Received complete upload request method=POST path=/32122026/11/12 00:27:05 INFO Received uploads request method=POST path=/32132026/11/12 00:27:05 INFO Completed upload id=10 objects=7 duration=1.588935ms32142026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=032152026/11/12 00:27:05 INFO Vacuumed table table=pending_closures32162026/11/12 00:27:05 INFO Vacuumed table table=pending_objects32172026/11/12 00:27:05 INFO Received complete push request method=POST path=/32182026/11/12 00:27:05 INFO Received push request method=POST path=/32192026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads32202026/11/12 00:27:05 INFO Vacuumed table table=closures32212026/11/12 00:27:05 INFO Vacuumed table table=objects32222026/11/12 00:27:05 INFO Completed push id=14 objects=6 duration=5.568612ms32232026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032242026/11/12 00:27:05 INFO Received complete push request method=POST path=/32252026/11/12 00:27:05 INFO Received uploads request method=POST path=/32262026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032272026/11/12 00:27:05 INFO Completed push id=5 objects=8 duration=4.815725ms32282026/11/12 00:27:05 INFO Vacuumed table table=pending_closures32292026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032302026/11/12 00:27:05 INFO Received push request method=POST path=/32312026/11/12 00:27:05 INFO Received uploads request method=POST path=/32322026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032332026/11/12 00:27:05 INFO Received uploads request method=POST path=/32342026/11/12 00:27:05 INFO Received uploads request method=POST path=/32352026/11/12 00:27:05 INFO Vacuumed table table=pending_objects32362026/11/12 00:27:05 INFO Received push request method=POST path=/32372026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period32382026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads32392026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032402026/11/12 00:27:05 INFO Vacuumed table table=pending_closures32412026/11/12 00:27:05 INFO Received push request method=POST path=/32422026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032432026/11/12 00:27:05 INFO Vacuumed table table=pending_objects32442026/11/12 00:27:05 INFO Vacuumed table table=closures32452026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads32462026/11/12 00:27:05 INFO Vacuumed table table=pending_closures32472026/11/12 00:27:05 INFO Vacuumed table table=closures32482026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period32492026/11/12 00:27:05 INFO Vacuumed table table=objects32502026/11/12 00:27:05 INFO Vacuumed table table=pending_objects32512026/11/12 00:27:05 INFO Vacuumed table table=objects32522026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads32532026/11/12 00:27:05 INFO Vacuumed table table=closures32542026/11/12 00:27:05 INFO Vacuumed table table=objects32552026/11/12 00:27:05 INFO Received push request method=POST path=/32562026/11/12 00:27:05 INFO Received push request method=POST path=/32572026/11/12 00:27:05 INFO Received uploads request method=POST path=/32582026/11/12 00:27:05 INFO Received push request method=POST path=/32592026/11/12 00:27:05 INFO Received uploads request method=POST path=/32602026/11/12 00:27:05 INFO Received push request method=POST path=/32612026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032622026/11/12 00:27:05 INFO Received complete upload request method=POST path=/32632026/11/12 00:27:05 INFO Received uploads request method=POST path=/32642026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period32652026/11/12 00:27:05 INFO Completed upload id=19 objects=25 duration=2.902207ms32662026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032672026/11/12 00:27:05 INFO Received push request method=POST path=/32682026/11/12 00:27:05 INFO Received push request method=POST path=/32692026/11/12 00:27:05 INFO Vacuumed table table=pending_closures32702026/11/12 00:27:05 INFO Vacuumed table table=pending_objects32712026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads32722026/11/12 00:27:05 INFO Received complete upload request method=POST path=/32732026/11/12 00:27:05 INFO Vacuumed table table=closures32742026/11/12 00:27:05 INFO Vacuumed table table=objects32752026/11/12 00:27:05 INFO Completed upload id=6 objects=6 duration=2.073619ms32762026/11/12 00:27:05 INFO Received complete push request method=POST path=/32772026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032782026/11/12 00:27:05 INFO Received complete upload request method=POST path=/32792026/11/12 00:27:05 INFO Received push request method=POST path=/32802026/11/12 00:27:05 INFO Completed push id=12 objects=3 duration=1.525374ms32812026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032822026/11/12 00:27:05 INFO Completed upload id=10 objects=2 duration=3.280391ms32832026/11/12 00:27:05 INFO Received uploads request method=POST path=/32842026/11/12 00:27:05 INFO Vacuumed table table=pending_closures32852026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032862026/11/12 00:27:05 INFO Vacuumed table table=pending_objects32872026/11/12 00:27:05 INFO Received complete upload request method=POST path=/32882026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads32892026/11/12 00:27:05 INFO Completed upload id=17 objects=6 duration=3.996738ms32902026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032912026/11/12 00:27:05 INFO Vacuumed table table=closures32922026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period32932026/11/12 00:27:05 INFO Received complete upload request method=POST path=/32942026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=032952026/11/12 00:27:05 INFO Received push request method=POST path=/32962026/11/12 00:27:05 INFO Received push request method=POST path=/32972026/11/12 00:27:05 INFO Vacuumed table table=objects32982026/11/12 00:27:05 INFO Completed upload id=9 objects=7 duration=4.167439ms32992026/11/12 00:27:05 INFO Received push request method=POST path=/33002026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=033012026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=033022026/11/12 00:27:05 INFO Vacuumed table table=pending_closures33032026/11/12 00:27:05 INFO Received uploads request method=POST path=/33042026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033052026/11/12 00:27:05 WARN Force mode enabled - objects will be deleted immediately without grace period33062026/11/12 00:27:05 INFO Vacuumed table table=pending_objects33072026/11/12 00:27:05 INFO Vacuumed table table=pending_closures33082026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads33092026/11/12 00:27:05 INFO Vacuumed table table=pending_objects33102026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads33112026/11/12 00:27:05 INFO Vacuumed table table=closures33122026/11/12 00:27:05 INFO Vacuumed table table=closures33132026/11/12 00:27:05 INFO Vacuumed table table=objects33142026/11/12 00:27:05 INFO Vacuumed table table=objects33152026/11/12 00:27:05 INFO Received complete upload request method=POST path=/33162026/11/12 00:27:05 INFO Completed upload id=18 objects=4 duration=1.155691ms33172026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=033182026/11/12 00:27:05 INFO Received push request method=POST path=/33192026/11/12 00:27:05 INFO Received uploads request method=POST path=/33202026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=033212026/11/12 00:27:05 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33222026/11/12 00:27:05 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."33232026/11/12 00:27:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33242026/11/12 00:27:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33252026/11/12 00:27:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33262026/11/12 00:27:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33272026/11/12 00:27:05 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."33282026/11/12 00:27:05 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."33292026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33302026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."33312026/11/12 00:27:05 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."33322026/11/12 00:27:05 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33332026/11/12 00:27:05 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."33342026/11/12 00:27:05 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."33352026/11/12 00:27:05 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33362026/11/12 00:27:05 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."33372026/11/12 00:27:05 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."33382026/11/12 00:27:05 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33392026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33402026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33412026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33422026/11/12 00:27:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33432026/11/12 00:27:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33442026/11/12 00:27:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33452026/11/12 00:27:05 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33462026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033472026/11/12 00:27:05 INFO Received uploads request method=POST path=/33482026/11/12 00:27:05 INFO Received uploads request method=POST path=/33492026/11/12 00:27:05 INFO Vacuumed table table=pending_closures33502026/11/12 00:27:05 INFO Received uploads request method=POST path=/33512026/11/12 00:27:05 INFO Received uploads request method=POST path=/33522026/11/12 00:27:05 INFO Received push request method=POST path=/33532026/11/12 00:27:05 INFO Vacuumed table table=pending_objects33542026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads33552026/11/12 00:27:05 INFO Vacuumed table table=closures33562026/11/12 00:27:05 INFO Vacuumed table table=objects33572026/11/12 00:27:05 INFO Received push request method=POST path=/33582026/11/12 00:27:05 INFO Received push request method=POST path=/33592026/11/12 00:27:05 INFO Received uploads request method=POST path=/33602026/11/12 00:27:05 INFO Received uploads request method=POST path=/33612026/11/12 00:27:05 INFO Received uploads request method=POST path=/33622026/11/12 00:27:05 INFO Received push request method=POST path=/33632026/11/12 00:27:05 INFO Received uploads request method=POST path=/33642026/11/12 00:27:05 INFO Received push request method=POST path=/33652026/11/12 00:27:05 INFO Received push request method=POST path=/33662026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=033672026/11/12 00:27:05 INFO Received uploads request method=POST path=/33682026/11/12 00:27:05 INFO Received uploads request method=POST path=/33692026/11/12 00:27:05 INFO Received push request method=POST path=/33702026/11/12 00:27:05 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."33712026/11/12 00:27:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33722026/11/12 00:27:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33732026/11/12 00:27:05 INFO Received push request method=POST path=/33742026/11/12 00:27:05 INFO Received uploads request method=POST path=/33752026/11/12 00:27:05 INFO Received uploads request method=POST path=/33762026/11/12 00:27:05 INFO Received complete upload request method=POST path=/33772026/11/12 00:27:05 INFO Completed upload id=8 objects=3 duration=6.773464ms33782026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033792026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033802026/11/12 00:27:05 INFO Vacuumed table table=pending_closures33812026/11/12 00:27:05 INFO Vacuumed table table=pending_objects33822026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=033832026/11/12 00:27:05 INFO Received uploads request method=POST path=/33842026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads33852026/11/12 00:27:05 INFO Received uploads request method=POST path=/33862026/11/12 00:27:05 INFO Vacuumed table table=pending_closures33872026/11/12 00:27:05 INFO Vacuumed table table=closures33882026/11/12 00:27:05 INFO Received push request method=POST path=/33892026/11/12 00:27:05 INFO Vacuumed table table=pending_closures33902026/11/12 00:27:05 INFO Received complete upload request method=POST path=/33912026/11/12 00:27:05 INFO Vacuumed table table=pending_objects33922026/11/12 00:27:05 INFO Vacuumed table table=pending_objects33932026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads33942026/11/12 00:27:05 INFO Vacuumed table table=objects33952026/11/12 00:27:05 INFO Completed upload id=16 objects=23 duration=2.965588ms33962026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads33972026/11/12 00:27:05 INFO Received uploads request method=POST path=/33982026/11/12 00:27:05 INFO Received uploads request method=POST path=/33992026/11/12 00:27:05 INFO Vacuumed table table=closures34002026/11/12 00:27:05 INFO Received uploads request method=POST path=/34012026/11/12 00:27:05 INFO Vacuumed table table=closures34022026/11/12 00:27:05 INFO Vacuumed table table=objects34032026/11/12 00:27:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=034042026/11/12 00:27:05 INFO Vacuumed table table=objects34052026/11/12 00:27:05 INFO Vacuumed table table=pending_closures34062026/11/12 00:27:05 INFO Received push request method=POST path=/34072026/11/12 00:27:05 INFO Vacuumed table table=pending_objects34082026/11/12 00:27:05 INFO Vacuumed table table=multipart_uploads34092026/11/12 00:27:05 INFO Vacuumed table table=closures34102026/11/12 00:27:05 INFO Vacuumed table table=objects34112026/11/12 00:27:05 INFO Received uploads request method=POST path=/34122026/11/12 00:27:05 INFO Received complete upload request method=POST path=/34132026/11/12 00:27:05 INFO Received push request method=POST path=/34142026/11/12 00:27:05 INFO Received complete push request method=POST path=/34152026/11/12 00:27:05 INFO Completed upload id=15 objects=6 duration=4.736664ms34162026/11/12 00:27:05 INFO Received uploads request method=POST path=/34172026/11/12 00:27:05 INFO Received push request method=POST path=/34182026/11/12 00:27:05 INFO Completed push id=12 objects=2 duration=6.536901ms34192026/11/12 00:27:05 INFO Received push request method=POST path=/34202026/11/12 00:27:05 INFO Received push request method=POST path=/34212026/11/12 00:27:05 INFO Received push request method=POST path=/34222026/11/12 00:27:05 INFO Received push request method=POST path=/34232026/11/12 00:27:05 INFO Received uploads request method=POST path=/34242026/11/12 00:27:05 INFO Received push request method=POST path=/34252026/11/12 00:27:05 INFO Received uploads request method=POST path=/34262026/11/12 00:27:05 INFO Received complete push request method=POST path=/34272026/11/12 00:27:05 INFO Received push request method=POST path=/34282026/11/12 00:27:05 INFO Completed push id=10 objects=2 duration=2.14406ms34292026/11/12 00:27:05 INFO Aborted multipart uploads count=0 kept=034302026/11/12 00:27:05 INFO Received uploads request method=POST path=/34312026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period34322026/11/12 00:27:06 INFO Received complete push request method=POST path=/34332026/11/12 00:27:06 INFO Received push request method=POST path=/34342026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=034352026/11/12 00:27:06 INFO Received push request method=POST path=/34362026/11/12 00:27:06 INFO Received uploads request method=POST path=/34372026/11/12 00:27:06 INFO Completed push id=8 objects=6 duration=4.30422ms34382026/11/12 00:27:06 INFO Received complete push request method=POST path=/34392026/11/12 00:27:06 INFO Completed push id=6 objects=6 duration=5.493552ms34402026/11/12 00:27:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34412026/11/12 00:27:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34422026/11/12 00:27:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34432026/11/12 00:27:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."34442026/11/12 00:27:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."34452026/11/12 00:27:06 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."34462026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34472026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."34482026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34492026/11/12 00:27:06 INFO Received complete upload request method=POST path=/34502026/11/12 00:27:06 INFO Received uploads request method=POST path=/34512026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034522026/11/12 00:27:06 INFO Received push request method=POST path=/34532026/11/12 00:27:06 INFO Received complete upload request method=POST path=/34542026/11/12 00:27:06 INFO Completed upload id=17 objects=7 duration=2.843967ms34552026/11/12 00:27:06 INFO Vacuumed table table=pending_closures34562026/11/12 00:27:06 INFO Completed upload id=12 objects=21 duration=2.228441ms34572026/11/12 00:27:06 INFO Vacuumed table table=pending_objects34582026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads34592026/11/12 00:27:06 INFO Received complete upload request method=POST path=/34602026/11/12 00:27:06 INFO Vacuumed table table=closures34612026/11/12 00:27:06 INFO Completed upload id=23 objects=18 duration=2.07392ms34622026/11/12 00:27:06 INFO Vacuumed table table=objects34632026/11/12 00:27:06 INFO Received push request method=POST path=/34642026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=034652026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period34662026/11/12 00:27:06 INFO Received uploads request method=POST path=/34672026/11/12 00:27:06 INFO Received complete push request method=POST path=/34682026/11/12 00:27:06 INFO Received push request method=POST path=/34692026/11/12 00:27:06 INFO Completed push id=1 objects=11 duration=7.031305ms34702026/11/12 00:27:06 INFO Received complete push request method=POST path=/34712026/11/12 00:27:06 INFO Received push request method=POST path=/34722026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=034732026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=034742026/11/12 00:27:06 INFO Received uploads request method=POST path=/34752026/11/12 00:27:06 INFO Completed push id=11 objects=8 duration=2.916107ms34762026/11/12 00:27:06 INFO Received complete upload request method=POST path=/34772026/11/12 00:27:06 INFO Received uploads request method=POST path=/34782026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period34792026/11/12 00:27:06 INFO Completed upload id=15 objects=9 duration=1.473654ms34802026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034812026/11/12 00:27:06 INFO Received push request method=POST path=/34822026/11/12 00:27:06 INFO Vacuumed table table=pending_closures34832026/11/12 00:27:06 INFO Received complete upload request method=POST path=/34842026/11/12 00:27:06 INFO Vacuumed table table=pending_objects34852026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads34862026/11/12 00:27:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34872026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34882026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=034892026/11/12 00:27:06 INFO Completed upload id=14 objects=16 duration=1.646976ms34902026/11/12 00:27:06 INFO Vacuumed table table=closures34912026/11/12 00:27:06 INFO Vacuumed table table=objects34922026/11/12 00:27:06 INFO Vacuumed table table=pending_closures34932026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=034942026/11/12 00:27:06 INFO Vacuumed table table=pending_objects34952026/11/12 00:27:06 INFO Received push request method=POST path=/34962026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads34972026/11/12 00:27:06 INFO Vacuumed table table=closures34982026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034992026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=035002026/11/12 00:27:06 INFO Vacuumed table table=objects35012026/11/12 00:27:06 INFO Vacuumed table table=pending_closures35022026/11/12 00:27:06 INFO Vacuumed table table=pending_objects35032026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads35042026/11/12 00:27:06 INFO Received complete upload request method=POST path=/35052026/11/12 00:27:06 INFO Vacuumed table table=closures35062026/11/12 00:27:06 INFO Completed upload id=16 objects=3 duration=1.325092ms35072026/11/12 00:27:06 INFO Vacuumed table table=objects35082026/11/12 00:27:06 INFO Received complete push request method=POST path=/35092026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035102026/11/12 00:27:06 INFO Received push request method=POST path=/35112026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035122026/11/12 00:27:06 INFO Completed push id=16 objects=4 duration=4.796484ms35132026/11/12 00:27:06 INFO Vacuumed table table=pending_closures35142026/11/12 00:27:06 INFO Vacuumed table table=pending_closures35152026/11/12 00:27:06 INFO Vacuumed table table=pending_objects35162026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads35172026/11/12 00:27:06 INFO Vacuumed table table=pending_objects35182026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads35192026/11/12 00:27:06 INFO Vacuumed table table=closures35202026/11/12 00:27:06 INFO Vacuumed table table=closures35212026/11/12 00:27:06 INFO Received push request method=POST path=/35222026/11/12 00:27:06 INFO Vacuumed table table=objects35232026/11/12 00:27:06 INFO Vacuumed table table=objects35242026/11/12 00:27:06 INFO Received complete push request method=POST path=/35252026/11/12 00:27:06 INFO Received push request method=POST path=/35262026/11/12 00:27:06 INFO Completed push id=13 objects=23 duration=1.555515ms35272026/11/12 00:27:06 INFO Received uploads request method=POST path=/35282026/11/12 00:27:06 INFO Received complete push request method=POST path=/35292026/11/12 00:27:06 INFO Completed push id=10 objects=15 duration=3.124769ms35302026/11/12 00:27:06 INFO Received push request method=POST path=/35312026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=035322026/11/12 00:27:06 INFO Received uploads request method=POST path=/35332026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period35342026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035352026/11/12 00:27:06 INFO Received push request method=POST path=/35362026/11/12 00:27:06 INFO Received complete upload request method=POST path=/35372026/11/12 00:27:06 INFO Received uploads request method=POST path=/35382026/11/12 00:27:06 INFO Completed upload id=22 objects=22 duration=3.450512ms35392026/11/12 00:27:06 INFO Received uploads request method=POST path=/35402026/11/12 00:27:06 INFO Vacuumed table table=pending_closures35412026/11/12 00:27:06 INFO Received push request method=POST path=/35422026/11/12 00:27:06 INFO Received complete push request method=POST path=/35432026/11/12 00:27:06 INFO Received uploads request method=POST path=/35442026/11/12 00:27:06 INFO Vacuumed table table=pending_objects35452026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads35462026/11/12 00:27:06 INFO Received complete upload request method=POST path=/35472026/11/12 00:27:06 INFO Completed push id=21 objects=17 duration=2.680045ms35482026/11/12 00:27:06 INFO Vacuumed table table=closures35492026/11/12 00:27:06 INFO Completed upload id=14 objects=6 duration=1.686135ms35502026/11/12 00:27:06 INFO Vacuumed table table=objects35512026/11/12 00:27:06 INFO Received complete push request method=POST path=/35522026/11/12 00:27:06 INFO Received push request method=POST path=/35532026/11/12 00:27:06 INFO Completed push id=13 objects=6 duration=3.182829ms35542026/11/12 00:27:06 INFO Received push request method=POST path=/35552026/11/12 00:27:06 INFO Received complete push request method=POST path=/35562026/11/12 00:27:06 INFO Completed push id=17 objects=11 duration=2.026018ms35572026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=035582026/11/12 00:27:06 INFO Received push request method=POST path=/35592026/11/12 00:27:06 INFO Received push request method=POST path=/35602026/11/12 00:27:06 INFO Received uploads request method=POST path=/35612026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period35622026/11/12 00:27:06 INFO Received complete upload request method=POST path=/35632026/11/12 00:27:06 INFO Received complete push request method=POST path=/35642026/11/12 00:27:06 INFO Completed upload id=20 objects=5 duration=1.184771ms35652026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035662026/11/12 00:27:06 INFO Received complete push request method=POST path=/35672026/11/12 00:27:06 INFO Completed push id=9 objects=24 duration=2.059379ms35682026/11/12 00:27:06 INFO Vacuumed table table=pending_closures35692026/11/12 00:27:06 INFO Received push request method=POST path=/35702026/11/12 00:27:06 INFO Received push request method=POST path=/35712026/11/12 00:27:06 INFO Vacuumed table table=pending_objects35722026/11/12 00:27:06 INFO Completed push id=19 objects=7 duration=3.707115ms35732026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads35742026/11/12 00:27:06 INFO Vacuumed table table=closures35752026/11/12 00:27:06 INFO Vacuumed table table=objects35762026/11/12 00:27:06 INFO Received push request method=POST path=/35772026/11/12 00:27:06 INFO Received push request method=POST path=/35782026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=035792026/11/12 00:27:06 INFO Received complete push request method=POST path=/35802026/11/12 00:27:06 INFO Completed push id=26 objects=5 duration=1.593635ms35812026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=035822026/11/12 00:27:06 INFO Received push request method=POST path=/35832026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period35842026/11/12 00:27:06 INFO Received uploads request method=POST path=/35852026/11/12 00:27:06 INFO Received uploads request method=POST path=/35862026/11/12 00:27:06 INFO Received uploads request method=POST path=/35872026/11/12 00:27:06 INFO Received push request method=POST path=/35882026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035892026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=035902026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=035912026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=035922026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035932026/11/12 00:27:06 INFO Received uploads request method=POST path=/35942026/11/12 00:27:06 INFO Vacuumed table table=pending_closures35952026/11/12 00:27:06 INFO Vacuumed table table=pending_closures35962026/11/12 00:27:06 INFO Received complete upload request method=POST path=/35972026/11/12 00:27:06 INFO Received push request method=POST path=/35982026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period35992026/11/12 00:27:06 INFO Vacuumed table table=pending_objects36002026/11/12 00:27:06 INFO Vacuumed table table=pending_objects36012026/11/12 00:27:06 INFO Completed upload id=24 objects=14 duration=1.492874ms36022026/11/12 00:27:06 INFO Vacuumed table table=pending_closures36032026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads36042026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads36052026/11/12 00:27:06 INFO Vacuumed table table=closures36062026/11/12 00:27:06 INFO Vacuumed table table=closures36072026/11/12 00:27:06 INFO Received push request method=POST path=/36082026/11/12 00:27:06 INFO Received complete upload request method=POST path=/36092026/11/12 00:27:06 INFO Vacuumed table table=objects36102026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036112026/11/12 00:27:06 INFO Vacuumed table table=objects36122026/11/12 00:27:06 INFO Completed upload id=7 objects=3 duration=1.419013ms36132026/11/12 00:27:06 INFO Vacuumed table table=pending_objects36142026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads36152026/11/12 00:27:06 INFO Vacuumed table table=pending_closures36162026/11/12 00:27:06 INFO Vacuumed table table=closures36172026/11/12 00:27:06 INFO Vacuumed table table=pending_objects36182026/11/12 00:27:06 INFO Received complete push request method=POST path=/36192026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads36202026/11/12 00:27:06 INFO Vacuumed table table=objects36212026/11/12 00:27:06 INFO Vacuumed table table=closures36222026/11/12 00:27:06 INFO Vacuumed table table=objects36232026/11/12 00:27:06 INFO Completed push id=11 objects=11 duration=2.904727ms36242026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=036252026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period36262026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036272026/11/12 00:27:06 INFO Received push request method=POST path=/36282026/11/12 00:27:06 INFO Received uploads request method=POST path=/36292026/11/12 00:27:06 INFO Received complete push request method=POST path=/36302026/11/12 00:27:06 INFO Vacuumed table table=pending_closures36312026/11/12 00:27:06 INFO Received complete push request method=POST path=/36322026/11/12 00:27:06 INFO Received uploads request method=POST path=/36332026/11/12 00:27:06 INFO Completed push id=22 objects=3 duration=2.624285ms36342026/11/12 00:27:06 INFO Received push request method=POST path=/36352026/11/12 00:27:06 INFO Vacuumed table table=pending_objects36362026/11/12 00:27:06 INFO Completed push id=19 objects=3 duration=2.989428ms36372026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=036382026/11/12 00:27:06 INFO Received push request method=POST path=/36392026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads36402026/11/12 00:27:06 INFO Vacuumed table table=closures36412026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period36422026/11/12 00:27:06 INFO Vacuumed table table=objects36432026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=036442026/11/12 00:27:06 INFO Received complete push request method=POST path=/36452026/11/12 00:27:06 INFO Received complete upload request method=POST path=/36462026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036472026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036482026/11/12 00:27:06 INFO Completed upload id=26 objects=6 duration=2.470943ms36492026/11/12 00:27:06 INFO Completed push id=18 objects=23 duration=3.506093ms36502026/11/12 00:27:06 INFO Vacuumed table table=pending_closures36512026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=036522026/11/12 00:27:06 INFO Vacuumed table table=pending_closures36532026/11/12 00:27:06 INFO Vacuumed table table=pending_objects36542026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads36552026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period36562026/11/12 00:27:06 INFO Received push request method=POST path=/36572026/11/12 00:27:06 INFO Received push request method=POST path=/36582026/11/12 00:27:06 INFO Vacuumed table table=closures36592026/11/12 00:27:06 INFO Vacuumed table table=pending_objects36602026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads36612026/11/12 00:27:06 INFO Vacuumed table table=objects36622026/11/12 00:27:06 INFO Received complete upload request method=POST path=/36632026/11/12 00:27:06 INFO Vacuumed table table=closures36642026/11/12 00:27:06 INFO Completed upload id=10 objects=10 duration=2.257901ms36652026/11/12 00:27:06 INFO Received uploads request method=POST path=/36662026/11/12 00:27:06 INFO Vacuumed table table=objects36672026/11/12 00:27:06 INFO Received complete push request method=POST path=/36682026/11/12 00:27:06 INFO Received uploads request method=POST path=/36692026/11/12 00:27:06 INFO Completed push id=17 objects=6 duration=2.318422ms36702026/11/12 00:27:06 INFO Received uploads request method=POST path=/36712026/11/12 00:27:06 INFO Received push request method=POST path=/36722026/11/12 00:27:06 INFO Received push request method=POST path=/36732026/11/12 00:27:06 INFO Received uploads request method=POST path=/36742026/11/12 00:27:06 INFO Received uploads request method=POST path=/36752026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36762026/11/12 00:27:06 INFO Received complete push request method=POST path=/36772026/11/12 00:27:06 INFO Received complete push request method=POST path=/36782026/11/12 00:27:06 INFO Completed push id=4 objects=6 duration=4.724804ms36792026/11/12 00:27:06 INFO Completed push id=4 objects=6 duration=4.692663ms36802026/11/12 00:27:06 INFO Received complete upload request method=POST path=/36812026/11/12 00:27:06 INFO Received push request method=POST path=/36822026/11/12 00:27:06 INFO Received complete upload request method=POST path=/36832026/11/12 00:27:06 INFO Received uploads request method=POST path=/36842026/11/12 00:27:06 INFO Received push request method=POST path=/36852026/11/12 00:27:06 INFO Received uploads request method=POST path=/36862026/11/12 00:27:06 INFO Completed upload id=23 objects=16 duration=8.440818ms36872026/11/12 00:27:06 INFO Completed upload id=29 objects=5 duration=14.125672ms36882026/11/12 00:27:06 INFO Received uploads request method=POST path=/36892026/11/12 00:27:06 INFO Received push request method=POST path=/36902026/11/12 00:27:06 INFO Received complete push request method=POST path=/36912026/11/12 00:27:06 INFO Completed push id=11 objects=9 duration=8.253377ms36922026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=036932026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period36942026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036952026/11/12 00:27:06 INFO Received uploads request method=POST path=/36962026/11/12 00:27:06 INFO Vacuumed table table=pending_closures36972026/11/12 00:27:06 INFO Received complete upload request method=POST path=/36982026/11/12 00:27:06 INFO Received uploads request method=POST path=/36992026/11/12 00:27:06 INFO Received push request method=POST path=/37002026/11/12 00:27:06 INFO Vacuumed table table=pending_objects37012026/11/12 00:27:06 INFO Completed upload id=15 objects=4 duration=6.521261ms37022026/11/12 00:27:06 INFO Received push request method=POST path=/37032026/11/12 00:27:06 INFO Received complete upload request method=POST path=/37042026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads37052026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037062026/11/12 00:27:06 INFO Received uploads request method=POST path=/37072026/11/12 00:27:06 INFO Completed upload id=12 objects=26 duration=9.65707ms37082026/11/12 00:27:06 INFO Vacuumed table table=closures37092026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period37102026/11/12 00:27:06 INFO Received complete push request method=POST path=/37112026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037122026/11/12 00:27:06 INFO Vacuumed table table=objects37132026/11/12 00:27:06 INFO Completed push id=13 objects=12 duration=4.447742ms37142026/11/12 00:27:06 INFO Received push request method=POST path=/37152026/11/12 00:27:06 INFO Vacuumed table table=pending_closures37162026/11/12 00:27:06 INFO Received complete upload request method=POST path=/37172026/11/12 00:27:06 INFO Received complete push request method=POST path=/37182026/11/12 00:27:06 INFO Received uploads request method=POST path=/37192026/11/12 00:27:06 INFO Received uploads request method=POST path=/37202026/11/12 00:27:06 INFO Completed upload id=9 objects=2 duration=6.063857ms37212026/11/12 00:27:06 INFO Vacuumed table table=pending_objects37222026/11/12 00:27:06 INFO Completed push id=4 objects=7 duration=6.121637ms37232026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads37242026/11/12 00:27:06 INFO Received complete push request method=POST path=/37252026/11/12 00:27:06 INFO Vacuumed table table=closures37262026/11/12 00:27:06 INFO Vacuumed table table=objects37272026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037282026/11/12 00:27:06 INFO Received complete upload request method=POST path=/37292026/11/12 00:27:06 INFO Completed push id=9 objects=4 duration=7.662111ms37302026/11/12 00:27:06 INFO Completed upload id=25 objects=4 duration=2.399243ms37312026/11/12 00:27:06 INFO Received uploads request method=POST path=/37322026/11/12 00:27:06 INFO Received push request method=POST path=/37332026/11/12 00:27:06 INFO Received uploads request method=POST path=/37342026/11/12 00:27:06 INFO Received uploads request method=POST path=/37352026/11/12 00:27:06 INFO Received complete push request method=POST path=/37362026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037372026/11/12 00:27:06 INFO Completed push id=8 objects=3 duration=1.615135ms37382026/11/12 00:27:06 INFO Received push request method=POST path=/37392026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period37402026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037412026/11/12 00:27:06 INFO Received uploads request method=POST path=/37422026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037432026/11/12 00:27:06 INFO Received push request method=POST path=/37442026/11/12 00:27:06 INFO Received uploads request method=POST path=/37452026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period37462026/11/12 00:27:06 INFO Received push request method=POST path=/37472026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period37482026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037492026/11/12 00:27:06 INFO Vacuumed table table=pending_closures37502026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037512026/11/12 00:27:06 INFO Vacuumed table table=pending_objects37522026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads37532026/11/12 00:27:06 INFO Received push request method=POST path=/37542026/11/12 00:27:06 INFO Vacuumed table table=closures37552026/11/12 00:27:06 INFO Vacuumed table table=pending_closures37562026/11/12 00:27:06 INFO Vacuumed table table=objects37572026/11/12 00:27:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37582026/11/12 00:27:06 INFO Received uploads request method=POST path=/37592026/11/12 00:27:06 INFO Vacuumed table table=pending_objects37602026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads37612026/11/12 00:27:06 INFO Received uploads request method=POST path=/37622026/11/12 00:27:06 INFO Received push request method=POST path=/37632026/11/12 00:27:06 INFO Vacuumed table table=closures37642026/11/12 00:27:06 INFO Vacuumed table table=objects37652026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037662026/11/12 00:27:06 INFO Received push request method=POST path=/37672026/11/12 00:27:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37682026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37692026/11/12 00:27:06 INFO Received complete upload request method=POST path=/37702026/11/12 00:27:06 INFO Completed upload id=16 objects=3 duration=1.846037ms37712026/11/12 00:27:06 INFO Received complete push request method=POST path=/37722026/11/12 00:27:06 INFO Received push request method=POST path=/37732026/11/12 00:27:06 INFO Completed push id=16 objects=8 duration=2.401882ms37742026/11/12 00:27:06 INFO Received uploads request method=POST path=/37752026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037762026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037772026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period37782026/11/12 00:27:06 INFO Received push request method=POST path=/37792026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037802026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037812026/11/12 00:27:06 INFO Vacuumed table table=pending_closures37822026/11/12 00:27:06 INFO Vacuumed table table=pending_closures37832026/11/12 00:27:06 INFO Vacuumed table table=pending_objects37842026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads37852026/11/12 00:27:06 INFO Received uploads request method=POST path=/37862026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037872026/11/12 00:27:06 INFO Vacuumed table table=pending_objects37882026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads37892026/11/12 00:27:06 INFO Vacuumed table table=closures37902026/11/12 00:27:06 INFO Vacuumed table table=closures37912026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037922026/11/12 00:27:06 INFO Received complete push request method=POST path=/37932026/11/12 00:27:06 INFO Vacuumed table table=objects37942026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=037952026/11/12 00:27:06 INFO Vacuumed table table=objects37962026/11/12 00:27:06 INFO Completed push id=20 objects=8 duration=3.114309ms37972026/11/12 00:27:06 INFO Received complete upload request method=POST path=/37982026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period37992026/11/12 00:27:06 INFO Completed upload id=13 objects=10 duration=1.831377ms38002026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038012026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038022026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period38032026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38042026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38052026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038062026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38072026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38082026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38092026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38102026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38112026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=038122026/11/12 00:27:06 INFO Received complete upload request method=POST path=/38132026/11/12 00:27:06 INFO Received uploads request method=POST path=/38142026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period38152026/11/12 00:27:06 INFO Vacuumed table table=closures38162026/11/12 00:27:06 INFO Completed upload id=14 objects=8 duration=2.437963ms38172026/11/12 00:27:06 INFO Vacuumed table table=closures38182026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38192026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38202026/11/12 00:27:06 INFO Vacuumed table table=closures38212026/11/12 00:27:06 INFO Vacuumed table table=objects38222026/11/12 00:27:06 INFO Vacuumed table table=objects38232026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=038242026/11/12 00:27:06 INFO Vacuumed table table=objects38252026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period38262026/11/12 00:27:06 INFO Received push request method=POST path=/38272026/11/12 00:27:06 INFO Received uploads request method=POST path=/38282026/11/12 00:27:06 INFO Received uploads request method=POST path=/38292026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=038302026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38312026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38322026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38332026/11/12 00:27:06 INFO Vacuumed table table=closures38342026/11/12 00:27:06 INFO Vacuumed table table=objects38352026/11/12 00:27:06 INFO Received complete upload request method=POST path=/38362026/11/12 00:27:06 INFO Completed upload id=31 objects=3 duration=2.1169ms38372026/11/12 00:27:06 INFO Received complete push request method=POST path=/38382026/11/12 00:27:06 INFO Completed push id=14 objects=3 duration=1.07825ms38392026/11/12 00:27:06 INFO Received uploads request method=POST path=/38402026/11/12 00:27:06 INFO Received complete push request method=POST path=/38412026/11/12 00:27:06 INFO Received push request method=POST path=/38422026/11/12 00:27:06 INFO Completed push id=7 objects=7 duration=1.325152ms38432026/11/12 00:27:06 INFO Received complete push request method=POST path=/38442026/11/12 00:27:06 INFO Received complete upload request method=POST path=/38452026/11/12 00:27:06 INFO Completed push id=11 objects=6 duration=3.876956ms38462026/11/12 00:27:06 INFO Completed upload id=21 objects=7 duration=2.368102ms38472026/11/12 00:27:06 INFO Received complete push request method=POST path=/38482026/11/12 00:27:06 INFO Received complete upload request method=POST path=/38492026/11/12 00:27:06 INFO Received push request method=POST path=/38502026/11/12 00:27:06 INFO Received push request method=POST path=/38512026/11/12 00:27:06 INFO Completed upload id=22 objects=2 duration=2.14214ms38522026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=038532026/11/12 00:27:06 INFO Completed push id=19 objects=6 duration=3.916336ms38542026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period38552026/11/12 00:27:06 INFO Received uploads request method=POST path=/38562026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038572026/11/12 00:27:06 INFO Received push request method=POST path=/38582026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38592026/11/12 00:27:06 INFO Received uploads request method=POST path=/38602026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38612026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38622026/11/12 00:27:06 INFO Vacuumed table table=closures38632026/11/12 00:27:06 INFO Vacuumed table table=objects38642026/11/12 00:27:06 INFO Received complete push request method=POST path=/38652026/11/12 00:27:06 INFO Completed push id=7 objects=15 duration=3.148569ms38662026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=038672026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38682026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=038692026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38702026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38712026/11/12 00:27:06 INFO Received complete upload request method=POST path=/38722026/11/12 00:27:06 INFO Received complete push request method=POST path=/38732026/11/12 00:27:06 INFO Vacuumed table table=closures38742026/11/12 00:27:06 INFO Completed upload id=18 objects=4 duration=1.678176ms38752026/11/12 00:27:06 INFO Completed push id=18 objects=6 duration=2.271141ms38762026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period38772026/11/12 00:27:06 INFO Vacuumed table table=objects38782026/11/12 00:27:06 INFO Received push request method=POST path=/38792026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038802026/11/12 00:27:06 INFO Received complete push request method=POST path=/38812026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38822026/11/12 00:27:06 INFO Completed push id=27 objects=11 duration=1.09125ms38832026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38842026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38852026/11/12 00:27:06 INFO Received complete upload request method=POST path=/38862026/11/12 00:27:06 INFO Received push request method=POST path=/38872026/11/12 00:27:06 INFO Vacuumed table table=closures38882026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=038892026/11/12 00:27:06 INFO Completed upload id=23 objects=15 duration=4.889306ms38902026/11/12 00:27:06 INFO Vacuumed table table=objects38912026/11/12 00:27:06 INFO Received push request method=POST path=/38922026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period38932026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038942026/11/12 00:27:06 INFO Received push request method=POST path=/38952026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=038962026/11/12 00:27:06 INFO Vacuumed table table=pending_closures38972026/11/12 00:27:06 INFO Vacuumed table table=pending_objects38982026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads38992026/11/12 00:27:06 INFO Vacuumed table table=closures39002026/11/12 00:27:06 INFO Vacuumed table table=objects39012026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period39022026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39032026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=039042026/11/12 00:27:06 INFO Completed upload id=29 objects=26 duration=3.703614ms39052026/11/12 00:27:06 INFO Received push request method=POST path=/39062026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=039072026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period39082026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39092026/11/12 00:27:06 INFO Completed upload id=18 objects=11 duration=1.456174ms39102026/11/12 00:27:06 INFO Received complete push request method=POST path=/39112026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39122026/11/12 00:27:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39132026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39142026/11/12 00:27:06 INFO Completed push id=17 objects=22 duration=4.35354ms39152026/11/12 00:27:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39162026/11/12 00:27:06 INFO Received complete push request method=POST path=/39172026/11/12 00:27:06 INFO Completed upload id=18 objects=2 duration=1.961598ms39182026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039192026/11/12 00:27:06 INFO Completed push id=17 objects=6 duration=3.046048ms39202026/11/12 00:27:06 INFO Vacuumed table table=pending_closures39212026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=039222026/11/12 00:27:06 INFO Received push request method=POST path=/39232026/11/12 00:27:06 INFO Received uploads request method=POST path=/39242026/11/12 00:27:06 INFO Received complete push request method=POST path=/39252026/11/12 00:27:06 INFO Received uploads request method=POST path=/39262026/11/12 00:27:06 INFO Vacuumed table table=pending_objects39272026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads39282026/11/12 00:27:06 INFO Completed push id=30 objects=8 duration=2.789606ms39292026/11/12 00:27:06 INFO Vacuumed table table=closures39302026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period39312026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039322026/11/12 00:27:06 INFO Vacuumed table table=objects39332026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=039342026/11/12 00:27:06 INFO Vacuumed table table=pending_closures39352026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period39362026/11/12 00:27:06 INFO Received push request method=POST path=/39372026/11/12 00:27:06 INFO Vacuumed table table=pending_objects39382026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads39392026/11/12 00:27:06 INFO Vacuumed table table=closures39402026/11/12 00:27:06 INFO Vacuumed table table=objects39412026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39422026/11/12 00:27:06 INFO Completed upload id=19 objects=6 duration=2.943467ms39432026/11/12 00:27:06 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39442026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39452026/11/12 00:27:06 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39462026/11/12 00:27:06 INFO Received push request method=POST path=/39472026/11/12 00:27:06 INFO Received uploads request method=POST path=/39482026/11/12 00:27:06 INFO Received push request method=POST path=/39492026/11/12 00:27:06 INFO Received push request method=POST path=/39502026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39512026/11/12 00:27:06 INFO Received complete push request method=POST path=/39522026/11/12 00:27:06 INFO Received push request method=POST path=/39532026/11/12 00:27:06 INFO Received push request method=POST path=/39542026/11/12 00:27:06 INFO Completed push id=14 objects=4 duration=3.611334ms39552026/11/12 00:27:06 INFO Completed upload id=31 objects=5 duration=9.017684ms39562026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39572026/11/12 00:27:06 INFO Completed upload id=22 objects=7 duration=1.07969ms39582026/11/12 00:27:06 INFO Received push request method=POST path=/39592026/11/12 00:27:06 INFO Received uploads request method=POST path=/39602026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39612026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39622026/11/12 00:27:06 INFO Completed upload id=27 objects=4 duration=5.738333ms39632026/11/12 00:27:06 INFO Completed upload id=13 objects=7 duration=6.044177ms39642026/11/12 00:27:06 INFO Received uploads request method=POST path=/39652026/11/12 00:27:06 INFO Received complete push request method=POST path=/39662026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=039672026/11/12 00:27:06 INFO Received complete push request method=POST path=/39682026/11/12 00:27:06 INFO Completed push id=32 objects=3 duration=2.042899ms39692026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039702026/11/12 00:27:06 INFO Received complete push request method=POST path=/39712026/11/12 00:27:06 INFO Completed push id=3 objects=21 duration=8.403878ms39722026/11/12 00:27:06 INFO Vacuumed table table=pending_closures39732026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=039742026/11/12 00:27:06 INFO Received push request method=POST path=/39752026/11/12 00:27:06 INFO Completed push id=31 objects=14 duration=4.32814ms39762026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39772026/11/12 00:27:06 INFO Vacuumed table table=pending_objects39782026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads39792026/11/12 00:27:06 INFO Received complete push request method=POST path=/39802026/11/12 00:27:06 INFO Completed upload id=20 objects=4 duration=3.492953ms39812026/11/12 00:27:06 INFO Vacuumed table table=closures39822026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period39832026/11/12 00:27:06 INFO Completed push id=13 objects=11 duration=2.611824ms39842026/11/12 00:27:06 INFO Vacuumed table table=objects39852026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=039862026/11/12 00:27:06 INFO Received uploads request method=POST path=/39872026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period39882026/11/12 00:27:06 INFO Received complete upload request method=POST path=/39892026/11/12 00:27:06 INFO Received uploads request method=POST path=/39902026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039912026/11/12 00:27:06 INFO Completed upload id=16 objects=17 duration=2.433863ms39922026/11/12 00:27:06 INFO Vacuumed table table=pending_closures39932026/11/12 00:27:06 INFO Received uploads request method=POST path=/39942026/11/12 00:27:06 INFO Vacuumed table table=pending_objects39952026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads39962026/11/12 00:27:06 INFO Vacuumed table table=closures39972026/11/12 00:27:06 INFO Vacuumed table table=objects39982026/11/12 00:27:06 INFO Received uploads request method=POST path=/39992026/11/12 00:27:06 INFO Received complete push request method=POST path=/40002026/11/12 00:27:06 INFO Received complete upload request method=POST path=/40012026/11/12 00:27:06 INFO Completed push id=15 objects=7 duration=1.712096ms40022026/11/12 00:27:06 INFO Completed upload id=23 objects=7 duration=1.398593ms40032026/11/12 00:27:06 INFO Received complete upload request method=POST path=/40042026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."40052026/11/12 00:27:06 INFO Completed upload id=7 objects=10 duration=3.064589ms40062026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=040072026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period40082026/11/12 00:27:06 INFO Received complete push request method=POST path=/40092026/11/12 00:27:06 INFO Received uploads request method=POST path=/40102026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040112026/11/12 00:27:06 INFO Completed push id=24 objects=9 duration=1.797517ms40122026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=040132026/11/12 00:27:06 INFO Vacuumed table table=pending_closures40142026/11/12 00:27:06 INFO Received uploads request method=POST path=/40152026/11/12 00:27:06 INFO Received push request method=POST path=/40162026/11/12 00:27:06 INFO Vacuumed table table=pending_objects40172026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads40182026/11/12 00:27:06 INFO Vacuumed table table=closures40192026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period40202026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=040212026/11/12 00:27:06 INFO Vacuumed table table=objects40222026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period40232026/11/12 00:27:06 INFO Received uploads request method=POST path=/40242026/11/12 00:27:06 INFO Received push request method=POST path=/40252026/11/12 00:27:06 INFO Received uploads request method=POST path=/40262026/11/12 00:27:06 INFO Received uploads request method=POST path=/40272026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=040282026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period40292026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040302026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040312026/11/12 00:27:06 INFO Vacuumed table table=pending_closures40322026/11/12 00:27:06 INFO Vacuumed table table=pending_closures40332026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040342026/11/12 00:27:06 INFO Vacuumed table table=pending_objects40352026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads40362026/11/12 00:27:06 INFO Vacuumed table table=pending_closures40372026/11/12 00:27:06 INFO Vacuumed table table=pending_objects40382026/11/12 00:27:06 INFO Vacuumed table table=closures40392026/11/12 00:27:06 INFO Received push request method=POST path=/40402026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads40412026/11/12 00:27:06 INFO Vacuumed table table=objects40422026/11/12 00:27:06 INFO Vacuumed table table=closures40432026/11/12 00:27:06 INFO Vacuumed table table=pending_objects40442026/11/12 00:27:06 INFO Received complete push request method=POST path=/40452026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads40462026/11/12 00:27:06 INFO Vacuumed table table=objects40472026/11/12 00:27:06 INFO Received complete upload request method=POST path=/40482026/11/12 00:27:06 INFO Received complete push request method=POST path=/40492026/11/12 00:27:06 INFO Received uploads request method=POST path=/40502026/11/12 00:27:06 INFO Completed push id=15 objects=4 duration=3.067389ms40512026/11/12 00:27:06 INFO Completed upload id=24 objects=18 duration=2.062699ms40522026/11/12 00:27:06 INFO Vacuumed table table=closures40532026/11/12 00:27:06 INFO Completed push id=19 objects=14 duration=2.13658ms40542026/11/12 00:27:06 INFO Vacuumed table table=objects40552026/11/12 00:27:06 INFO Received uploads request method=POST path=/40562026/11/12 00:27:06 INFO Received complete push request method=POST path=/40572026/11/12 00:27:06 INFO Completed push id=7 objects=7 duration=1.265972ms40582026/11/12 00:27:06 INFO Received complete upload request method=POST path=/40592026/11/12 00:27:06 INFO Completed upload id=20 objects=3 duration=2.499223ms40602026/11/12 00:27:06 INFO Received uploads request method=POST path=/40612026/11/12 00:27:06 INFO Received complete push request method=POST path=/40622026/11/12 00:27:06 INFO Completed push id=25 objects=2 duration=1.466514ms40632026/11/12 00:27:06 INFO Received push request method=POST path=/40642026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=040652026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period40662026/11/12 00:27:06 INFO Received uploads request method=POST path=/40672026/11/12 00:27:06 INFO Received push request method=POST path=/40682026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=040692026/11/12 00:27:06 INFO Vacuumed table table=pending_closures40702026/11/12 00:27:06 INFO Received complete push request method=POST path=/40712026/11/12 00:27:06 INFO Vacuumed table table=pending_objects40722026/11/12 00:27:06 INFO Completed push id=12 objects=6 duration=1.03575ms40732026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads40742026/11/12 00:27:06 INFO Vacuumed table table=closures40752026/11/12 00:27:06 INFO Vacuumed table table=objects40762026/11/12 00:27:06 INFO Received complete push request method=POST path=/40772026/11/12 00:27:06 INFO Received push request method=POST path=/40782026/11/12 00:27:06 INFO Completed push id=27 objects=4 duration=1.06743ms40792026/11/12 00:27:06 INFO Received uploads request method=POST path=/40802026/11/12 00:27:06 INFO Received push request method=POST path=/40812026/11/12 00:27:06 INFO Received uploads request method=POST path=/40822026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=040832026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period40842026/11/12 00:27:06 INFO Received complete push request method=POST path=/40852026/11/12 00:27:06 INFO Received complete upload request method=POST path=/40862026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040872026/11/12 00:27:06 INFO Received complete upload request method=POST path=/40882026/11/12 00:27:06 INFO Vacuumed table table=pending_closures40892026/11/12 00:27:06 INFO Completed upload id=22 objects=2 duration=2.458283ms40902026/11/12 00:27:06 INFO Completed upload id=11 objects=7 duration=1.301152ms40912026/11/12 00:27:06 INFO Completed push id=15 objects=5 duration=2.579684ms40922026/11/12 00:27:06 INFO Vacuumed table table=pending_objects40932026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads40942026/11/12 00:27:06 INFO Vacuumed table table=closures40952026/11/12 00:27:06 INFO Vacuumed table table=objects40962026/11/12 00:27:06 INFO Received uploads request method=POST path=/40972026/11/12 00:27:06 INFO Received complete push request method=POST path=/40982026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=040992026/11/12 00:27:06 INFO Vacuumed table table=pending_closures41002026/11/12 00:27:06 INFO Completed push id=16 objects=16 duration=3.742895ms41012026/11/12 00:27:06 INFO Vacuumed table table=pending_objects41022026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads41032026/11/12 00:27:06 INFO Vacuumed table table=closures41042026/11/12 00:27:06 INFO Vacuumed table table=objects41052026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041062026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041072026/11/12 00:27:06 INFO Received uploads request method=POST path=/41082026/11/12 00:27:06 INFO Received uploads request method=POST path=/41092026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period41102026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041112026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041122026/11/12 00:27:06 INFO Received complete push request method=POST path=/41132026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period41142026/11/12 00:27:06 INFO Vacuumed table table=pending_closures41152026/11/12 00:27:06 INFO Vacuumed table table=pending_objects41162026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads41172026/11/12 00:27:06 INFO Completed push id=28 objects=26 duration=1.227471ms41182026/11/12 00:27:06 INFO Vacuumed table table=closures41192026/11/12 00:27:06 INFO Vacuumed table table=objects41202026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041212026/11/12 00:27:06 INFO Received complete upload request method=POST path=/41222026/11/12 00:27:06 INFO Received complete push request method=POST path=/41232026/11/12 00:27:06 INFO Vacuumed table table=pending_closures41242026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041252026/11/12 00:27:06 INFO Completed push id=14 objects=3 duration=1.335232ms41262026/11/12 00:27:06 INFO Completed upload id=24 objects=7 duration=1.494413ms41272026/11/12 00:27:06 INFO Received push request method=POST path=/41282026/11/12 00:27:06 INFO Vacuumed table table=pending_objects41292026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads41302026/11/12 00:27:06 INFO Vacuumed table table=pending_closures41312026/11/12 00:27:06 INFO Vacuumed table table=closures41322026/11/12 00:27:06 INFO Received push request method=POST path=/41332026/11/12 00:27:06 INFO Vacuumed table table=pending_objects41342026/11/12 00:27:06 INFO Vacuumed table table=objects41352026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads41362026/11/12 00:27:06 INFO Vacuumed table table=closures41372026/11/12 00:27:06 INFO Vacuumed table table=objects41382026/11/12 00:27:06 INFO Received uploads request method=POST path=/41392026/11/12 00:27:06 INFO Received complete push request method=POST path=/41402026/11/12 00:27:06 INFO Completed push id=13 objects=22 duration=1.418193ms41412026/11/12 00:27:06 INFO Received complete upload request method=POST path=/41422026/11/12 00:27:06 INFO Received uploads request method=POST path=/41432026/11/12 00:27:06 INFO Completed upload id=15 objects=3 duration=2.18026ms41442026/11/12 00:27:06 INFO Received complete push request method=POST path=/41452026/11/12 00:27:06 INFO Completed push id=30 objects=3 duration=1.15357ms41462026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041472026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041482026/11/12 00:27:06 INFO Received push request method=POST path=/41492026/11/12 00:27:06 INFO Received uploads request method=POST path=/41502026/11/12 00:27:06 INFO Received uploads request method=POST path=/41512026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041522026/11/12 00:27:06 INFO Vacuumed table table=pending_closures41532026/11/12 00:27:06 INFO Vacuumed table table=pending_objects41542026/11/12 00:27:06 INFO Received complete push request method=POST path=/41552026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads41562026/11/12 00:27:06 INFO Received push request method=POST path=/41572026/11/12 00:27:06 INFO Vacuumed table table=closures41582026/11/12 00:27:06 INFO Completed push id=21 objects=6 duration=1.531114ms41592026/11/12 00:27:06 INFO Received complete upload request method=POST path=/41602026/11/12 00:27:06 INFO Vacuumed table table=objects41612026/11/12 00:27:06 INFO Received complete push request method=POST path=/41622026/11/12 00:27:06 INFO Received complete push request method=POST path=/41632026/11/12 00:27:06 INFO Completed push id=32 objects=19 duration=1.632855ms41642026/11/12 00:27:06 INFO Completed upload id=17 objects=25 duration=3.622733ms41652026/11/12 00:27:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."41662026/11/12 00:27:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."41672026/11/12 00:27:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."41682026/11/12 00:27:06 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."41692026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41702026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."41712026/11/12 00:27:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41722026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041732026/11/12 00:27:06 INFO Completed push id=21 objects=3 duration=5.785354ms41742026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041752026/11/12 00:27:06 INFO Received push request method=POST path=/41762026/11/12 00:27:06 INFO Received complete upload request method=POST path=/41772026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041782026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041792026/11/12 00:27:06 INFO Vacuumed table table=pending_closures41802026/11/12 00:27:06 INFO Completed upload id=20 objects=4 duration=1.258292ms41812026/11/12 00:27:06 INFO Vacuumed table table=pending_closures41822026/11/12 00:27:06 INFO Vacuumed table table=pending_objects41832026/11/12 00:27:06 INFO Received complete upload request method=POST path=/41842026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads41852026/11/12 00:27:06 INFO Completed upload id=22 objects=7 duration=1.033189ms41862026/11/12 00:27:06 INFO Vacuumed table table=pending_objects41872026/11/12 00:27:06 INFO Vacuumed table table=closures41882026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads41892026/11/12 00:27:06 INFO Vacuumed table table=closures41902026/11/12 00:27:06 INFO Vacuumed table table=objects41912026/11/12 00:27:06 INFO Vacuumed table table=objects41922026/11/12 00:27:06 INFO Received complete push request method=POST path=/41932026/11/12 00:27:06 INFO Received uploads request method=POST path=/41942026/11/12 00:27:06 INFO Received uploads request method=POST path=/41952026/11/12 00:27:06 INFO Received push request method=POST path=/41962026/11/12 00:27:06 INFO Completed push id=18 objects=6 duration=2.558884ms41972026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=041982026/11/12 00:27:06 INFO Received push request method=POST path=/41992026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period42002026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042012026/11/12 00:27:06 INFO Received uploads request method=POST path=/42022026/11/12 00:27:06 INFO Vacuumed table table=pending_closures42032026/11/12 00:27:06 INFO Vacuumed table table=pending_objects42042026/11/12 00:27:06 INFO Received complete push request method=POST path=/42052026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads42062026/11/12 00:27:06 INFO Completed push id=24 objects=4 duration=1.263812ms42072026/11/12 00:27:06 INFO Vacuumed table table=closures42082026/11/12 00:27:06 INFO Vacuumed table table=objects42092026/11/12 00:27:06 INFO Received push request method=POST path=/42102026/11/12 00:27:06 INFO Received push request method=POST path=/42112026/11/12 00:27:06 INFO Received uploads request method=POST path=/42122026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042132026/11/12 00:27:06 INFO Received complete upload request method=POST path=/42142026/11/12 00:27:06 INFO Received complete upload request method=POST path=/42152026/11/12 00:27:06 INFO Completed upload id=21 objects=2 duration=913.768µs42162026/11/12 00:27:06 INFO Completed upload id=23 objects=9 duration=1.393653ms42172026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period42182026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042192026/11/12 00:27:06 INFO Vacuumed table table=pending_closures42202026/11/12 00:27:06 INFO Vacuumed table table=pending_objects42212026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads42222026/11/12 00:27:06 INFO Received complete push request method=POST path=/42232026/11/12 00:27:06 INFO Vacuumed table table=closures42242026/11/12 00:27:06 INFO Completed push id=25 objects=9 duration=981.629µs42252026/11/12 00:27:06 INFO Vacuumed table table=objects42262026/11/12 00:27:06 INFO Received uploads request method=POST path=/42272026/11/12 00:27:06 INFO Received complete push request method=POST path=/42282026/11/12 00:27:06 INFO Received uploads request method=POST path=/42292026/11/12 00:27:06 INFO Completed push id=16 objects=5 duration=1.09193ms42302026/11/12 00:27:06 INFO Received push request method=POST path=/42312026/11/12 00:27:06 INFO Received push request method=POST path=/42322026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042332026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period42342026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042352026/11/12 00:27:06 INFO Received complete push request method=POST path=/42362026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042372026/11/12 00:27:06 INFO Vacuumed table table=pending_closures42382026/11/12 00:27:06 INFO Received complete upload request method=POST path=/42392026/11/12 00:27:06 INFO Vacuumed table table=pending_objects42402026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads42412026/11/12 00:27:06 INFO Completed push id=17 objects=22 duration=2.365422ms42422026/11/12 00:27:06 INFO Completed upload id=26 objects=3 duration=854.368µs42432026/11/12 00:27:06 INFO Vacuumed table table=closures42442026/11/12 00:27:06 INFO Vacuumed table table=objects42452026/11/12 00:27:06 INFO Received uploads request method=POST path=/42462026/11/12 00:27:06 INFO Received push request method=POST path=/42472026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042482026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042492026/11/12 00:27:06 INFO Received complete push request method=POST path=/42502026/11/12 00:27:06 INFO Vacuumed table table=pending_closures42512026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042522026/11/12 00:27:06 INFO Completed push id=23 objects=28 duration=11.656168ms42532026/11/12 00:27:06 INFO Vacuumed table table=pending_closures42542026/11/12 00:27:06 INFO Vacuumed table table=pending_objects42552026/11/12 00:27:06 INFO Received push request method=POST path=/42562026/11/12 00:27:06 INFO Received uploads request method=POST path=/42572026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads42582026/11/12 00:27:06 INFO Vacuumed table table=pending_objects42592026/11/12 00:27:06 INFO Vacuumed table table=closures42602026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads42612026/11/12 00:27:06 INFO Vacuumed table table=closures42622026/11/12 00:27:06 INFO Vacuumed table table=objects42632026/11/12 00:27:06 INFO Vacuumed table table=objects42642026/11/12 00:27:06 INFO Received complete push request method=POST path=/42652026/11/12 00:27:06 INFO Received uploads request method=POST path=/42662026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042672026/11/12 00:27:06 INFO Completed push id=26 objects=14 duration=2.361222ms42682026/11/12 00:27:06 INFO Received complete upload request method=POST path=/42692026/11/12 00:27:06 INFO Received complete push request method=POST path=/42702026/11/12 00:27:06 INFO Received complete upload request method=POST path=/42712026/11/12 00:27:06 INFO Completed upload id=27 objects=7 duration=987.889µs42722026/11/12 00:27:06 INFO Completed push id=13 objects=15 duration=1.463514ms42732026/11/12 00:27:06 INFO Completed upload id=13 objects=14 duration=2.986488ms42742026/11/12 00:27:06 INFO Received push request method=POST path=/42752026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042762026/11/12 00:27:06 INFO Vacuumed table table=pending_closures42772026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042782026/11/12 00:27:06 INFO Vacuumed table table=pending_objects42792026/11/12 00:27:06 INFO Received uploads request method=POST path=/42802026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads42812026/11/12 00:27:06 INFO Received push request method=POST path=/42822026/11/12 00:27:06 INFO Vacuumed table table=closures42832026/11/12 00:27:06 INFO Vacuumed table table=objects42842026/11/12 00:27:06 INFO Received complete upload request method=POST path=/42852026/11/12 00:27:06 WARN Force mode enabled - objects will be deleted immediately without grace period42862026/11/12 00:27:06 INFO Completed upload id=28 objects=3 duration=2.011539ms42872026/11/12 00:27:06 INFO Received push request method=POST path=/42882026/11/12 00:27:06 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042892026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042902026/11/12 00:27:06 INFO Vacuumed table table=pending_closures42912026/11/12 00:27:06 INFO Received push request method=POST path=/42922026/11/12 00:27:06 INFO Received complete upload request method=POST path=/42932026/11/12 00:27:06 INFO Vacuumed table table=pending_objects42942026/11/12 00:27:06 INFO Aborted multipart uploads count=0 kept=042952026/11/12 00:27:06 INFO Vacuumed table table=multipart_uploads42962026/11/12 00:27:06 INFO Completed upload id=27 objects=8 duration=1.397773ms42972026/11/12 00:27:06 INFO Vacuumed table table=closures42982026/11/12 00:27:06 INFO Vacuumed table table=objects42992026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043002026/11/12 00:27:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43012026/11/12 00:27:07 INFO Vacuumed table table=pending_closures43022026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043032026/11/12 00:27:07 INFO Vacuumed table table=pending_objects43042026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads43052026/11/12 00:27:07 INFO Vacuumed table table=closures43062026/11/12 00:27:07 INFO Vacuumed table table=objects43072026/11/12 00:27:07 INFO Received push request method=POST path=/43082026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043092026/11/12 00:27:07 INFO Vacuumed table table=pending_closures43102026/11/12 00:27:07 INFO Vacuumed table table=pending_objects43112026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads43122026/11/12 00:27:07 INFO Vacuumed table table=closures43132026/11/12 00:27:07 INFO Received complete push request method=POST path=/43142026/11/12 00:27:07 INFO Received uploads request method=POST path=/43152026/11/12 00:27:07 INFO Vacuumed table table=objects43162026/11/12 00:27:07 INFO Received complete push request method=POST path=/43172026/11/12 00:27:07 INFO Received push request method=POST path=/43182026/11/12 00:27:07 INFO Completed push id=29 objects=23 duration=1.453254ms43192026/11/12 00:27:07 INFO Completed push id=7 objects=24 duration=3.903177ms43202026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043212026/11/12 00:27:07 INFO Received push request method=POST path=/43222026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period43232026/11/12 00:27:07 INFO Received uploads request method=POST path=/43242026/11/12 00:27:07 INFO Received complete push request method=POST path=/43252026/11/12 00:27:07 INFO Received push request method=POST path=/43262026/11/12 00:27:07 INFO Received uploads request method=POST path=/43272026/11/12 00:27:07 INFO Received complete push request method=POST path=/43282026/11/12 00:27:07 INFO Completed push id=30 objects=4 duration=3.241011ms43292026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043302026/11/12 00:27:07 INFO Received complete push request method=POST path=/43312026/11/12 00:27:07 INFO Completed push id=30 objects=6 duration=1.030269ms43322026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043332026/11/12 00:27:07 INFO Completed push id=31 objects=6 duration=2.914367ms43342026/11/12 00:27:07 INFO Received uploads request method=POST path=/43352026/11/12 00:27:07 INFO Received uploads request method=POST path=/43362026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043372026/11/12 00:27:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43382026/11/12 00:27:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."43392026/11/12 00:27:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."43402026/11/12 00:27:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."43412026/11/12 00:27:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43422026/11/12 00:27:07 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."43432026/11/12 00:27:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43442026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043452026/11/12 00:27:07 INFO Vacuumed table table=pending_closures43462026/11/12 00:27:07 INFO Vacuumed table table=pending_closures43472026/11/12 00:27:07 INFO Vacuumed table table=pending_objects43482026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads43492026/11/12 00:27:07 INFO Received complete push request method=POST path=/43502026/11/12 00:27:07 INFO Vacuumed table table=closures43512026/11/12 00:27:07 INFO Vacuumed table table=pending_objects43522026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads43532026/11/12 00:27:07 INFO Vacuumed table table=objects43542026/11/12 00:27:07 INFO Completed push id=20 objects=3 duration=2.275361ms43552026/11/12 00:27:07 INFO Vacuumed table table=closures43562026/11/12 00:27:07 INFO Vacuumed table table=objects43572026/11/12 00:27:07 INFO Received complete push request method=POST path=/43582026/11/12 00:27:07 INFO Received uploads request method=POST path=/43592026/11/12 00:27:07 INFO Received uploads request method=POST path=/43602026/11/12 00:27:07 INFO Completed push id=19 objects=14 duration=1.736636ms43612026/11/12 00:27:07 INFO Received uploads request method=POST path=/43622026/11/12 00:27:07 INFO Received uploads request method=POST path=/43632026/11/12 00:27:07 INFO Received uploads request method=POST path=/43642026/11/12 00:27:07 INFO Received complete push request method=POST path=/43652026/11/12 00:27:07 INFO Completed push id=33 objects=3 duration=1.009789ms43662026/11/12 00:27:07 INFO Received complete push request method=POST path=/43672026/11/12 00:27:07 INFO Received push request method=POST path=/43682026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043692026/11/12 00:27:07 INFO Received uploads request method=POST path=/43702026/11/12 00:27:07 INFO Completed push id=16 objects=26 duration=1.822458ms43712026/11/12 00:27:07 INFO Received push request method=POST path=/43722026/11/12 00:27:07 INFO Received complete push request method=POST path=/43732026/11/12 00:27:07 INFO Received push request method=POST path=/43742026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043752026/11/12 00:27:07 INFO Completed push id=14 objects=3 duration=3.151449ms43762026/11/12 00:27:07 INFO Vacuumed table table=pending_closures43772026/11/12 00:27:07 INFO Received complete upload request method=POST path=/43782026/11/12 00:27:07 INFO Vacuumed table table=pending_objects43792026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043802026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads43812026/11/12 00:27:07 INFO Completed upload id=21 objects=3 duration=1.648955ms43822026/11/12 00:27:07 INFO Received complete push request method=POST path=/43832026/11/12 00:27:07 INFO Received uploads request method=POST path=/43842026/11/12 00:27:07 INFO Vacuumed table table=closures43852026/11/12 00:27:07 INFO Vacuumed table table=pending_closures43862026/11/12 00:27:07 INFO Completed push id=34 objects=6 duration=2.443123ms43872026/11/12 00:27:07 INFO Received push request method=POST path=/43882026/11/12 00:27:07 INFO Vacuumed table table=objects43892026/11/12 00:27:07 INFO Vacuumed table table=pending_objects43902026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads43912026/11/12 00:27:07 INFO Vacuumed table table=closures43922026/11/12 00:27:07 INFO Vacuumed table table=objects43932026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043942026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043952026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043962026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=043972026/11/12 00:27:07 INFO Vacuumed table table=pending_closures43982026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period43992026/11/12 00:27:07 INFO Received push request method=POST path=/44002026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period44012026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44022026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=044032026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044042026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44052026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period44062026/11/12 00:27:07 INFO Vacuumed table table=closures44072026/11/12 00:27:07 INFO Vacuumed table table=pending_closures44082026/11/12 00:27:07 INFO Vacuumed table table=objects44092026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044102026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44112026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44122026/11/12 00:27:07 INFO Vacuumed table table=pending_closures44132026/11/12 00:27:07 INFO Vacuumed table table=closures44142026/11/12 00:27:07 INFO Vacuumed table table=objects44152026/11/12 00:27:07 INFO Received complete push request method=POST path=/44162026/11/12 00:27:07 INFO Received complete upload request method=POST path=/44172026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44182026/11/12 00:27:07 INFO Received push request method=POST path=/44192026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44202026/11/12 00:27:07 INFO Completed push id=21 objects=9 duration=2.307801ms44212026/11/12 00:27:07 INFO Completed upload id=25 objects=16 duration=2.089619ms44222026/11/12 00:27:07 INFO Vacuumed table table=closures44232026/11/12 00:27:07 INFO Received push request method=POST path=/44242026/11/12 00:27:07 INFO Vacuumed table table=objects44252026/11/12 00:27:07 INFO Received complete push request method=POST path=/44262026/11/12 00:27:07 INFO Completed push id=35 objects=8 duration=963.229µs44272026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=044282026/11/12 00:27:07 INFO Vacuumed table table=pending_closures44292026/11/12 00:27:07 INFO Received complete push request method=POST path=/44302026/11/12 00:27:07 INFO Received complete upload request method=POST path=/44312026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=044322026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44332026/11/12 00:27:07 INFO Completed push id=32 objects=7 duration=1.267392ms44342026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44352026/11/12 00:27:07 INFO Completed upload id=20 objects=6 duration=1.424834ms44362026/11/12 00:27:07 INFO Vacuumed table table=closures44372026/11/12 00:27:07 INFO Received push request method=POST path=/44382026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044392026/11/12 00:27:07 INFO Vacuumed table table=pending_closures44402026/11/12 00:27:07 INFO Vacuumed table table=objects44412026/11/12 00:27:07 INFO Received complete push request method=POST path=/44422026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44432026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44442026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=044452026/11/12 00:27:07 INFO Completed push id=27 objects=6 duration=3.389232ms44462026/11/12 00:27:07 INFO Vacuumed table table=closures44472026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period44482026/11/12 00:27:07 INFO Vacuumed table table=objects44492026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044502026/11/12 00:27:07 INFO Received uploads request method=POST path=/44512026/11/12 00:27:07 INFO Received push request method=POST path=/44522026/11/12 00:27:07 INFO Received uploads request method=POST path=/44532026/11/12 00:27:07 INFO Vacuumed table table=pending_closures44542026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44552026/11/12 00:27:07 INFO Received complete upload request method=POST path=/44562026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44572026/11/12 00:27:07 INFO Completed upload id=36 objects=9 duration=1.183371ms44582026/11/12 00:27:07 INFO Vacuumed table table=closures44592026/11/12 00:27:07 INFO Received complete push request method=POST path=/44602026/11/12 00:27:07 INFO Vacuumed table table=objects44612026/11/12 00:27:07 INFO Completed push id=26 objects=4 duration=1.00725ms44622026/11/12 00:27:07 INFO Received push request method=POST path=/44632026/11/12 00:27:07 INFO Received complete push request method=POST path=/44642026/11/12 00:27:07 INFO Completed push id=1 objects=9 duration=1.633816ms44652026/11/12 00:27:07 INFO Received uploads request method=POST path=/44662026/11/12 00:27:07 INFO Received uploads request method=POST path=/44672026/11/12 00:27:07 INFO Received uploads request method=POST path=/44682026/11/12 00:27:07 INFO Received complete push request method=POST path=/44692026/11/12 00:27:07 INFO Completed push id=13 objects=2 duration=995.289µs44702026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=044712026/11/12 00:27:07 INFO Received push request method=POST path=/44722026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=044732026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period44742026/11/12 00:27:07 INFO Received complete upload request method=POST path=/44752026/11/12 00:27:07 INFO Received complete push request method=POST path=/44762026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period44772026/11/12 00:27:07 INFO Received push request method=POST path=/44782026/11/12 00:27:07 INFO Completed upload id=28 objects=13 duration=1.036629ms44792026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044802026/11/12 00:27:07 INFO Received push request method=POST path=/44812026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044822026/11/12 00:27:07 INFO Completed push id=16 objects=14 duration=2.631004ms44832026/11/12 00:27:07 INFO Vacuumed table table=pending_closures44842026/11/12 00:27:07 INFO Vacuumed table table=pending_closures44852026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44862026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44872026/11/12 00:27:07 INFO Vacuumed table table=pending_objects44882026/11/12 00:27:07 INFO Vacuumed table table=closures44892026/11/12 00:27:07 INFO Received complete upload request method=POST path=/44902026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads44912026/11/12 00:27:07 INFO Vacuumed table table=objects44922026/11/12 00:27:07 INFO Completed upload id=15 objects=2 duration=1.05715ms44932026/11/12 00:27:07 INFO Vacuumed table table=closures44942026/11/12 00:27:07 INFO Vacuumed table table=objects44952026/11/12 00:27:07 INFO Received complete upload request method=POST path=/44962026/11/12 00:27:07 INFO Completed upload id=27 objects=13 duration=1.418553ms44972026/11/12 00:27:07 INFO Received push request method=POST path=/44982026/11/12 00:27:07 INFO Received push request method=POST path=/44992026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045002026/11/12 00:27:07 INFO Received complete upload request method=POST path=/45012026/11/12 00:27:07 INFO Received push request method=POST path=/45022026/11/12 00:27:07 INFO Received complete push request method=POST path=/45032026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045042026/11/12 00:27:07 INFO Completed upload id=22 objects=6 duration=883.209µs45052026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period45062026/11/12 00:27:07 INFO Completed push id=29 objects=3 duration=949.888µs45072026/11/12 00:27:07 INFO Received complete upload request method=POST path=/45082026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045092026/11/12 00:27:07 INFO Completed upload id=21 objects=2 duration=709.587µs45102026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045112026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45122026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45132026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45142026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45152026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045162026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45172026/11/12 00:27:07 INFO Vacuumed table table=closures45182026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45192026/11/12 00:27:07 INFO Vacuumed table table=objects45202026/11/12 00:27:07 INFO Vacuumed table table=closures45212026/11/12 00:27:07 INFO Vacuumed table table=objects45222026/11/12 00:27:07 INFO Received uploads request method=POST path=/45232026/11/12 00:27:07 INFO Received uploads request method=POST path=/45242026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045252026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45262026/11/12 00:27:07 INFO Received complete upload request method=POST path=/45272026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45282026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45292026/11/12 00:27:07 INFO Received complete push request method=POST path=/45302026/11/12 00:27:07 INFO Completed upload id=31 objects=2 duration=966.849µs45312026/11/12 00:27:07 INFO Vacuumed table table=closures45322026/11/12 00:27:07 INFO Vacuumed table table=objects45332026/11/12 00:27:07 INFO Received push request method=POST path=/45342026/11/12 00:27:07 INFO Completed push id=22 objects=3 duration=3.14287ms45352026/11/12 00:27:07 INFO Received push request method=POST path=/45362026/11/12 00:27:07 INFO Received uploads request method=POST path=/45372026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045382026/11/12 00:27:07 INFO Received push request method=POST path=/45392026/11/12 00:27:07 INFO Received complete upload request method=POST path=/45402026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045412026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45422026/11/12 00:27:07 INFO Completed upload id=2 objects=7 duration=1.303513ms45432026/11/12 00:27:07 INFO Received complete push request method=POST path=/45442026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45452026/11/12 00:27:07 INFO Received uploads request method=POST path=/45462026/11/12 00:27:07 INFO Completed push id=33 objects=6 duration=1.212791ms45472026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45482026/11/12 00:27:07 INFO Vacuumed table table=closures45492026/11/12 00:27:07 INFO Received uploads request method=POST path=/45502026/11/12 00:27:07 INFO Vacuumed table table=objects45512026/11/12 00:27:07 INFO Received push request method=POST path=/45522026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045532026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period45542026/11/12 00:27:07 INFO Received complete upload request method=POST path=/45552026/11/12 00:27:07 INFO Received complete push request method=POST path=/45562026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045572026/11/12 00:27:07 INFO Completed upload id=33 objects=5 duration=2.16138ms45582026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45592026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45602026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45612026/11/12 00:27:07 INFO Completed push id=27 objects=25 duration=2.738925ms45622026/11/12 00:27:07 INFO Vacuumed table table=closures45632026/11/12 00:27:07 INFO Vacuumed table table=objects45642026/11/12 00:27:07 INFO Received uploads request method=POST path=/45652026/11/12 00:27:07 INFO Received push request method=POST path=/45662026/11/12 00:27:07 INFO Received push request method=POST path=/45672026/11/12 00:27:07 INFO Received push request method=POST path=/45682026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045692026/11/12 00:27:07 INFO Received uploads request method=POST path=/45702026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period45712026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045722026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045732026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045742026/11/12 00:27:07 INFO Received push request method=POST path=/45752026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45762026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45772026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=045782026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45792026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45802026/11/12 00:27:07 INFO Received push request method=POST path=/45812026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45822026/11/12 00:27:07 INFO Vacuumed table table=closures45832026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45842026/11/12 00:27:07 INFO Vacuumed table table=objects45852026/11/12 00:27:07 INFO Vacuumed table table=closures45862026/11/12 00:27:07 INFO Vacuumed table table=objects45872026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045882026/11/12 00:27:07 INFO Vacuumed table table=pending_closures45892026/11/12 00:27:07 INFO Vacuumed table table=pending_objects45902026/11/12 00:27:07 INFO Received uploads request method=POST path=/45912026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads45922026/11/12 00:27:07 INFO Vacuumed table table=closures45932026/11/12 00:27:07 INFO Received push request method=POST path=/45942026/11/12 00:27:07 INFO Vacuumed table table=objects45952026/11/12 00:27:07 INFO Received complete upload request method=POST path=/45962026/11/12 00:27:07 INFO Received uploads request method=POST path=/45972026/11/12 00:27:07 INFO Received uploads request method=POST path=/45982026/11/12 00:27:07 INFO Completed upload id=22 objects=5 duration=2.266961ms45992026/11/12 00:27:07 INFO Received push request method=POST path=/46002026/11/12 00:27:07 INFO Received complete push request method=POST path=/46012026/11/12 00:27:07 INFO Completed push id=24 objects=3 duration=1.025129ms46022026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46032026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=046042026/11/12 00:27:07 INFO Completed upload id=8 objects=6 duration=1.06667ms46052026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period46062026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046072026/11/12 00:27:07 INFO Vacuumed table table=pending_closures46082026/11/12 00:27:07 INFO Vacuumed table table=pending_objects46092026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46102026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads46112026/11/12 00:27:07 INFO Received complete push request method=POST path=/46122026/11/12 00:27:07 INFO Vacuumed table table=closures46132026/11/12 00:27:07 INFO Received uploads request method=POST path=/46142026/11/12 00:27:07 INFO Vacuumed table table=objects46152026/11/12 00:27:07 INFO Completed upload id=25 objects=6 duration=1.763876ms46162026/11/12 00:27:07 INFO Received complete push request method=POST path=/46172026/11/12 00:27:07 INFO Received uploads request method=POST path=/46182026/11/12 00:27:07 INFO Received push request method=POST path=/46192026/11/12 00:27:07 INFO Completed push id=24 objects=5 duration=2.771265ms46202026/11/12 00:27:07 INFO Received push request method=POST path=/46212026/11/12 00:27:07 INFO Completed push id=28 objects=3 duration=1.714656ms46222026/11/12 00:27:07 INFO Received uploads request method=POST path=/46232026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46242026/11/12 00:27:07 INFO Completed upload id=21 objects=25 duration=1.558695ms46252026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46262026/11/12 00:27:07 INFO Completed upload id=19 objects=2 duration=1.991299ms46272026/11/12 00:27:07 INFO Received complete push request method=POST path=/46282026/11/12 00:27:07 INFO Received uploads request method=POST path=/46292026/11/12 00:27:07 INFO Received uploads request method=POST path=/46302026/11/12 00:27:07 INFO Received complete push request method=POST path=/46312026/11/12 00:27:07 INFO Completed push id=29 objects=15 duration=1.557374ms46322026/11/12 00:27:07 INFO Completed push id=24 objects=24 duration=3.553373ms46332026/11/12 00:27:07 INFO Received uploads request method=POST path=/46342026/11/12 00:27:07 INFO Received uploads request method=POST path=/46352026/11/12 00:27:07 INFO Received uploads request method=POST path=/46362026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46372026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46382026/11/12 00:27:07 INFO Received push request method=POST path=/46392026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=046402026/11/12 00:27:07 INFO Received complete push request method=POST path=/46412026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46422026/11/12 00:27:07 INFO Completed push id=30 objects=6 duration=2.626125ms46432026/11/12 00:27:07 INFO Completed upload id=12 objects=9 duration=3.621734ms46442026/11/12 00:27:07 INFO Completed upload id=26 objects=8 duration=1.832817ms46452026/11/12 00:27:07 INFO Completed upload id=29 objects=4 duration=3.769515ms46462026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period46472026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046482026/11/12 00:27:07 INFO Vacuumed table table=pending_closures46492026/11/12 00:27:07 INFO Received uploads request method=POST path=/46502026/11/12 00:27:07 INFO Received complete push request method=POST path=/46512026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=046522026/11/12 00:27:07 INFO Vacuumed table table=pending_objects46532026/11/12 00:27:07 INFO Completed push id=33 objects=7 duration=1.955838ms46542026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads46552026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period46562026/11/12 00:27:07 INFO Vacuumed table table=closures46572026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46582026/11/12 00:27:07 INFO Vacuumed table table=objects46592026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046602026/11/12 00:27:07 INFO Completed upload id=13 objects=8 duration=1.379433ms46612026/11/12 00:27:07 INFO Received uploads request method=POST path=/46622026/11/12 00:27:07 INFO Vacuumed table table=pending_closures46632026/11/12 00:27:07 INFO Received uploads request method=POST path=/46642026/11/12 00:27:07 INFO Received complete push request method=POST path=/46652026/11/12 00:27:07 INFO Received uploads request method=POST path=/46662026/11/12 00:27:07 INFO Vacuumed table table=pending_objects46672026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46682026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads46692026/11/12 00:27:07 INFO Completed push id=28 objects=14 duration=1.630396ms46702026/11/12 00:27:07 INFO Completed upload id=31 objects=3 duration=940.669µs46712026/11/12 00:27:07 INFO Vacuumed table table=closures46722026/11/12 00:27:07 INFO Vacuumed table table=objects46732026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46742026/11/12 00:27:07 INFO Completed upload id=26 objects=3 duration=1.029149ms46752026/11/12 00:27:07 INFO Received uploads request method=POST path=/46762026/11/12 00:27:07 INFO Received uploads request method=POST path=/46772026/11/12 00:27:07 INFO Received uploads request method=POST path=/46782026/11/12 00:27:07 INFO Received uploads request method=POST path=/46792026/11/12 00:27:07 INFO Received complete push request method=POST path=/46802026/11/12 00:27:07 INFO Completed push id=11 objects=9 duration=1.586115ms46812026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=046822026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period46832026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046842026/11/12 00:27:07 INFO Received complete upload request method=POST path=/46852026/11/12 00:27:07 INFO Vacuumed table table=pending_closures46862026/11/12 00:27:07 INFO Completed upload id=12 objects=6 duration=1.180831ms46872026/11/12 00:27:07 INFO Vacuumed table table=pending_objects46882026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads46892026/11/12 00:27:07 INFO Vacuumed table table=closures46902026/11/12 00:27:07 INFO Received complete push request method=POST path=/46912026/11/12 00:27:07 INFO Vacuumed table table=objects46922026/11/12 00:27:07 INFO Received uploads request method=POST path=/46932026/11/12 00:27:07 INFO Received uploads request method=POST path=/46942026/11/12 00:27:07 INFO Completed push id=21 objects=26 duration=1.632815ms46952026/11/12 00:27:07 INFO Received push request method=POST path=/46962026/11/12 00:27:07 INFO Received complete push request method=POST path=/46972026/11/12 00:27:07 INFO Received complete push request method=POST path=/46982026/11/12 00:27:07 INFO Completed push id=11 objects=6 duration=2.041719ms46992026/11/12 00:27:07 INFO Completed push id=17 objects=3 duration=2.288881ms47002026/11/12 00:27:07 INFO Received complete push request method=POST path=/47012026/11/12 00:27:07 INFO Completed push id=15 objects=24 duration=1.508014ms47022026/11/12 00:27:07 INFO Received push request method=POST path=/47032026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=047042026/11/12 00:27:07 INFO Received push request method=POST path=/47052026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47062026/11/12 00:27:07 INFO Completed upload id=28 objects=17 duration=1.497415ms47072026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period47082026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047092026/11/12 00:27:07 INFO Vacuumed table table=pending_closures47102026/11/12 00:27:07 INFO Vacuumed table table=pending_objects47112026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads47122026/11/12 00:27:07 INFO Vacuumed table table=closures47132026/11/12 00:27:07 INFO Vacuumed table table=objects47142026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47152026/11/12 00:27:07 INFO Completed upload id=19 objects=2 duration=1.08797ms47162026/11/12 00:27:07 INFO Received push request method=POST path=/47172026/11/12 00:27:07 INFO Received complete push request method=POST path=/47182026/11/12 00:27:07 INFO Received complete push request method=POST path=/47192026/11/12 00:27:07 INFO Completed push id=23 objects=5 duration=1.426393ms47202026/11/12 00:27:07 INFO Completed push id=20 objects=13 duration=1.471114ms47212026/11/12 00:27:07 INFO Received uploads request method=POST path=/47222026/11/12 00:27:07 INFO Received complete push request method=POST path=/47232026/11/12 00:27:07 INFO Completed push id=29 objects=2 duration=1.195171ms47242026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47252026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47262026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=047272026/11/12 00:27:07 INFO Completed upload id=18 objects=22 duration=1.378973ms47282026/11/12 00:27:07 INFO Completed upload id=24 objects=7 duration=1.474973ms47292026/11/12 00:27:07 INFO Received complete push request method=POST path=/47302026/11/12 00:27:07 INFO Received complete push request method=POST path=/47312026/11/12 00:27:07 INFO Received uploads request method=POST path=/47322026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47332026/11/12 00:27:07 INFO Received complete push request method=POST path=/47342026/11/12 00:27:07 INFO Completed push id=22 objects=4 duration=3.302491ms47352026/11/12 00:27:07 INFO Completed push id=31 objects=6 duration=1.479314ms47362026/11/12 00:27:07 INFO Completed push id=9 objects=9 duration=4.603364ms47372026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=047382026/11/12 00:27:07 INFO Completed upload id=24 objects=3 duration=3.613674ms47392026/11/12 00:27:07 INFO Received complete push request method=POST path=/47402026/11/12 00:27:07 INFO Completed push id=16 objects=7 duration=1.929438ms47412026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period47422026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047432026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=047442026/11/12 00:27:07 INFO Vacuumed table table=pending_closures47452026/11/12 00:27:07 INFO Vacuumed table table=pending_closures47462026/11/12 00:27:07 INFO Received complete push request method=POST path=/47472026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47482026/11/12 00:27:07 INFO Vacuumed table table=pending_objects47492026/11/12 00:27:07 INFO Vacuumed table table=pending_objects47502026/11/12 00:27:07 INFO Completed push id=26 objects=3 duration=1.04863ms47512026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads47522026/11/12 00:27:07 INFO Completed upload id=21 objects=8 duration=1.289652ms47532026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads47542026/11/12 00:27:07 INFO Vacuumed table table=closures47552026/11/12 00:27:07 INFO Vacuumed table table=closures47562026/11/12 00:27:07 INFO Vacuumed table table=objects47572026/11/12 00:27:07 INFO Received complete push request method=POST path=/47582026/11/12 00:27:07 INFO Completed push id=25 objects=4 duration=941.569µs47592026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=047602026/11/12 00:27:07 INFO Received complete push request method=POST path=/47612026/11/12 00:27:07 INFO Completed push id=32 objects=9 duration=1.09615ms47622026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period47632026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047642026/11/12 00:27:07 INFO Vacuumed table table=pending_closures47652026/11/12 00:27:07 INFO Vacuumed table table=pending_objects47662026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads47672026/11/12 00:27:07 INFO Vacuumed table table=closures47682026/11/12 00:27:07 INFO Vacuumed table table=objects47692026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47702026/11/12 00:27:07 INFO Completed upload id=28 objects=8 duration=793.348µs47712026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47722026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47732026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47742026/11/12 00:27:07 INFO Vacuumed table table=objects47752026/11/12 00:27:07 INFO Completed upload id=22 objects=7 duration=1.029129ms47762026/11/12 00:27:07 INFO Completed upload id=23 objects=7 duration=1.760056ms47772026/11/12 00:27:07 INFO Completed upload id=21 objects=8 duration=1.832017ms47782026/11/12 00:27:07 INFO Received uploads request method=POST path=/47792026/11/12 00:27:07 INFO Received complete push request method=POST path=/47802026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47812026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47822026/11/12 00:27:07 INFO Completed upload id=33 objects=9 duration=1.194551ms47832026/11/12 00:27:07 INFO Received complete push request method=POST path=/47842026/11/12 00:27:07 INFO Completed push id=5 objects=28 duration=2.321902ms47852026/11/12 00:27:07 INFO Completed upload id=29 objects=10 duration=945.628µs47862026/11/12 00:27:07 INFO Completed push id=23 objects=6 duration=1.237472ms47872026/11/12 00:27:07 INFO Received complete push request method=POST path=/47882026/11/12 00:27:07 INFO Received complete push request method=POST path=/47892026/11/12 00:27:07 INFO Completed push id=22 objects=6 duration=1.395053ms47902026/11/12 00:27:07 INFO Completed push id=24 objects=5 duration=1.027329ms47912026/11/12 00:27:07 INFO Received complete upload request method=POST path=/47922026/11/12 00:27:07 INFO Received complete push request method=POST path=/47932026/11/12 00:27:07 INFO Received uploads request method=POST path=/47942026/11/12 00:27:07 INFO Completed push id=24 objects=5 duration=1.14025ms47952026/11/12 00:27:07 INFO Completed upload id=26 objects=2 duration=1.918278ms47962026/11/12 00:27:07 INFO Received push request method=POST path=/47972026/11/12 00:27:07 INFO Received complete push request method=POST path=/47982026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=047992026/11/12 00:27:07 INFO Completed push id=10 objects=2 duration=1.701176ms48002026/11/12 00:27:07 INFO Received complete push request method=POST path=/48012026/11/12 00:27:07 INFO Completed push id=24 objects=6 duration=1.005689ms48022026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period48032026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048042026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48052026/11/12 00:27:07 INFO Received complete push request method=POST path=/48062026/11/12 00:27:07 INFO Completed upload id=30 objects=20 duration=777.807µs48072026/11/12 00:27:07 INFO Completed push id=26 objects=23 duration=1.315412ms48082026/11/12 00:27:07 INFO Vacuumed table table=pending_closures48092026/11/12 00:27:07 INFO Received complete push request method=POST path=/48102026/11/12 00:27:07 INFO Completed push id=24 objects=8 duration=975.429µs48112026/11/12 00:27:07 INFO Vacuumed table table=pending_objects48122026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads48132026/11/12 00:27:07 INFO Vacuumed table table=closures48142026/11/12 00:27:07 INFO Vacuumed table table=objects48152026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48162026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=048172026/11/12 00:27:07 INFO Completed upload id=25 objects=2 duration=959.709µs48182026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48192026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048202026/11/12 00:27:07 INFO Completed upload id=21 objects=9 duration=1.858898ms48212026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48222026/11/12 00:27:07 INFO Vacuumed table table=pending_closures48232026/11/12 00:27:07 INFO Completed upload id=25 objects=8 duration=982.509µs48242026/11/12 00:27:07 INFO Vacuumed table table=pending_objects48252026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads48262026/11/12 00:27:07 INFO Vacuumed table table=closures48272026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48282026/11/12 00:27:07 INFO Vacuumed table table=objects48292026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48302026/11/12 00:27:07 INFO Completed upload id=17 objects=5 duration=1.015149ms48312026/11/12 00:27:07 INFO Completed upload id=23 objects=7 duration=2.639965ms48322026/11/12 00:27:07 INFO Received complete push request method=POST path=/48332026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48342026/11/12 00:27:07 INFO Received complete push request method=POST path=/48352026/11/12 00:27:07 INFO Completed upload id=25 objects=9 duration=824.068µs48362026/11/12 00:27:07 INFO Completed push id=25 objects=7 duration=1.466353ms48372026/11/12 00:27:07 INFO Completed push id=31 objects=11 duration=1.230831ms48382026/11/12 00:27:07 INFO Received complete push request method=POST path=/48392026/11/12 00:27:07 INFO Received complete push request method=POST path=/48402026/11/12 00:27:07 INFO Completed push id=26 objects=4 duration=849.148µs48412026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48422026/11/12 00:27:07 INFO Completed push id=25 objects=10 duration=1.897417ms48432026/11/12 00:27:07 INFO Completed upload id=26 objects=20 duration=829.908µs48442026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48452026/11/12 00:27:07 INFO Completed upload id=26 objects=5 duration=1.04161ms48462026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48472026/11/12 00:27:07 INFO Completed upload id=27 objects=3 duration=1.10829ms48482026/11/12 00:27:07 INFO Received complete push request method=POST path=/48492026/11/12 00:27:07 INFO Completed push id=19 objects=3 duration=1.208271ms48502026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48512026/11/12 00:27:07 INFO Received push request method=POST path=/48522026/11/12 00:27:07 INFO Completed upload id=32 objects=2 duration=924.848µs48532026/11/12 00:27:07 INFO Received complete push request method=POST path=/48542026/11/12 00:27:07 INFO Received complete push request method=POST path=/48552026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48562026/11/12 00:27:07 INFO Completed push id=30 objects=2 duration=911.788µs48572026/11/12 00:27:07 INFO Completed upload id=29 objects=5 duration=1.008469ms48582026/11/12 00:27:07 INFO Completed push id=27 objects=15 duration=1.405013ms48592026/11/12 00:27:07 INFO Received complete push request method=POST path=/48602026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48612026/11/12 00:27:07 INFO Completed push id=27 objects=3 duration=1.702596ms48622026/11/12 00:27:07 INFO Completed upload id=28 objects=6 duration=847.068µs48632026/11/12 00:27:07 INFO Received complete push request method=POST path=/48642026/11/12 00:27:07 INFO Completed push id=28 objects=7 duration=1.149611ms48652026/11/12 00:27:07 INFO Received complete push request method=POST path=/48662026/11/12 00:27:07 INFO Completed push id=33 objects=3 duration=924.088µs48672026/11/12 00:27:07 INFO Received complete push request method=POST path=/48682026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48692026/11/12 00:27:07 INFO Completed push id=30 objects=4 duration=1.587095ms48702026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48712026/11/12 00:27:07 INFO Completed upload id=26 objects=24 duration=1.170691ms48722026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48732026/11/12 00:27:07 INFO Completed upload id=30 objects=2 duration=923.769µs48742026/11/12 00:27:07 INFO Completed upload id=28 objects=5 duration=899.769µs48752026/11/12 00:27:07 INFO Received complete push request method=POST path=/48762026/11/12 00:27:07 INFO Completed push id=20 objects=4 duration=1.315512ms48772026/11/12 00:27:07 INFO Received complete push request method=POST path=/48782026/11/12 00:27:07 INFO Completed push id=32 objects=7 duration=1.168011ms48792026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48802026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48812026/11/12 00:27:07 INFO Received complete push request method=POST path=/48822026/11/12 00:27:07 INFO Completed upload id=30 objects=6 duration=4.227579ms48832026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=048842026/11/12 00:27:07 INFO Completed upload id=34 objects=3 duration=789.527µs48852026/11/12 00:27:07 INFO Completed push id=31 objects=7 duration=1.307353ms48862026/11/12 00:27:07 INFO Received complete push request method=POST path=/48872026/11/12 00:27:07 INFO Received complete push request method=POST path=/48882026/11/12 00:27:07 INFO Completed push id=29 objects=17 duration=1.185831ms48892026/11/12 00:27:07 INFO Completed push id=31 objects=20 duration=1.248111ms48902026/11/12 00:27:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."48912026/11/12 00:27:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."48922026/11/12 00:27:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48932026/11/12 00:27:07 INFO Received complete push request method=POST path=/48942026/11/12 00:27:07 INFO Completed push id=28 objects=19 duration=1.03667ms48952026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48962026/11/12 00:27:07 INFO Completed upload id=33 objects=6 duration=919.569µs48972026/11/12 00:27:07 INFO Received complete upload request method=POST path=/48982026/11/12 00:27:07 INFO Received complete push request method=POST path=/48992026/11/12 00:27:07 INFO Completed push id=31 objects=5 duration=851.928µs49002026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49012026/11/12 00:27:07 INFO Completed upload id=22 objects=8 duration=925.929µs49022026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=049032026/11/12 00:27:07 INFO Completed upload id=32 objects=5 duration=6.589961ms49042026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period49052026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049062026/11/12 00:27:07 INFO Vacuumed table table=pending_closures49072026/11/12 00:27:07 INFO Vacuumed table table=pending_objects49082026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads49092026/11/12 00:27:07 INFO Vacuumed table table=closures49102026/11/12 00:27:07 INFO Vacuumed table table=objects49112026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49122026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=049132026/11/12 00:27:07 INFO Completed upload id=32 objects=5 duration=949.808µs49142026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period49152026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049162026/11/12 00:27:07 INFO Received push request method=POST path=/49172026/11/12 00:27:07 INFO Vacuumed table table=pending_closures49182026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49192026/11/12 00:27:07 INFO Vacuumed table table=pending_objects49202026/11/12 00:27:07 INFO Completed upload id=34 objects=3 duration=754.387µs49212026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads49222026/11/12 00:27:07 INFO Vacuumed table table=closures49232026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49242026/11/12 00:27:07 INFO Vacuumed table table=objects49252026/11/12 00:27:07 INFO Received complete push request method=POST path=/49262026/11/12 00:27:07 INFO Completed upload id=27 objects=26 duration=1.04429ms49272026/11/12 00:27:07 INFO Completed push id=23 objects=5 duration=1.113891ms49282026/11/12 00:27:07 INFO Received complete push request method=POST path=/49292026/11/12 00:27:07 INFO Completed push id=29 objects=10 duration=1.17265ms49302026/11/12 00:27:07 INFO Received complete push request method=POST path=/49312026/11/12 00:27:07 INFO Completed push id=32 objects=26 duration=1.369993ms49322026/11/12 00:27:07 INFO Received complete push request method=POST path=/49332026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49342026/11/12 00:27:07 INFO Completed upload id=24 objects=8 duration=1.051049ms49352026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49362026/11/12 00:27:07 INFO Completed upload id=33 objects=3 duration=843.788µs49372026/11/12 00:27:07 INFO Received push request method=POST path=/49382026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49392026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=049402026/11/12 00:27:07 INFO Completed push id=28 objects=6 duration=10.721339ms49412026/11/12 00:27:07 INFO Completed upload id=33 objects=2 duration=785.007µs49422026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period49432026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49442026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049452026/11/12 00:27:07 INFO Completed upload id=30 objects=6 duration=955.629µs49462026/11/12 00:27:07 INFO Vacuumed table table=pending_closures49472026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49482026/11/12 00:27:07 INFO Completed upload id=28 objects=5 duration=874.528µs49492026/11/12 00:27:07 INFO Vacuumed table table=pending_objects49502026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads49512026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=049522026/11/12 00:27:07 INFO Vacuumed table table=closures49532026/11/12 00:27:07 INFO Vacuumed table table=objects49542026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period49552026/11/12 00:27:07 INFO Received complete push request method=POST path=/49562026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049572026/11/12 00:27:07 INFO Completed push id=28 objects=11 duration=2.296401ms49582026/11/12 00:27:07 INFO Vacuumed table table=pending_closures49592026/11/12 00:27:07 INFO Vacuumed table table=pending_objects49602026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49612026/11/12 00:27:07 INFO Received complete push request method=POST path=/49622026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads49632026/11/12 00:27:07 INFO Vacuumed table table=closures49642026/11/12 00:27:07 INFO Completed push id=29 objects=6 duration=1.13511ms49652026/11/12 00:27:07 INFO Completed upload id=34 objects=24 duration=1.195671ms49662026/11/12 00:27:07 INFO Vacuumed table table=objects49672026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49682026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49692026/11/12 00:27:07 INFO Completed upload id=34 objects=5 duration=1.019909ms49702026/11/12 00:27:07 INFO Completed upload id=31 objects=5 duration=932.288µs49712026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49722026/11/12 00:27:07 INFO Completed upload id=30 objects=5 duration=972.889µs49732026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=049742026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period49752026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49762026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049772026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49782026/11/12 00:27:07 INFO Completed upload id=35 objects=11 duration=838.327µs49792026/11/12 00:27:07 INFO Completed upload id=34 objects=3 duration=859.148µs49802026/11/12 00:27:07 INFO Vacuumed table table=pending_closures49812026/11/12 00:27:07 INFO Vacuumed table table=pending_objects49822026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads49832026/11/12 00:27:07 INFO Vacuumed table table=closures49842026/11/12 00:27:07 INFO Vacuumed table table=objects49852026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49862026/11/12 00:27:07 INFO Completed upload id=30 objects=8 duration=859.948µs49872026/11/12 00:27:07 INFO Received complete push request method=POST path=/49882026/11/12 00:27:07 INFO Received complete upload request method=POST path=/49892026/11/12 00:27:07 INFO Completed push id=29 objects=24 duration=1.611135ms49902026/11/12 00:27:07 INFO Completed upload id=36 objects=3 duration=811.188µs49912026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=049922026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period49932026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049942026/11/12 00:27:07 INFO Vacuumed table table=pending_closures49952026/11/12 00:27:07 INFO Vacuumed table table=pending_objects49962026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads49972026/11/12 00:27:07 INFO Vacuumed table table=closures49982026/11/12 00:27:07 INFO Vacuumed table table=objects49992026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=050002026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period50012026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=050022026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period50032026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050042026/11/12 00:27:07 INFO Vacuumed table table=pending_closures50052026/11/12 00:27:07 INFO Vacuumed table table=pending_objects50062026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads50072026/11/12 00:27:07 INFO Vacuumed table table=closures50082026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050092026/11/12 00:27:07 INFO Vacuumed table table=objects50102026/11/12 00:27:07 INFO Vacuumed table table=pending_closures50112026/11/12 00:27:07 INFO Vacuumed table table=pending_objects50122026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads50132026/11/12 00:27:07 INFO Vacuumed table table=closures50142026/11/12 00:27:07 INFO Vacuumed table table=objects50152026/11/12 00:27:07 INFO Received complete upload request method=POST path=/50162026/11/12 00:27:07 INFO Completed upload id=31 objects=6 duration=1.05243ms50172026/11/12 00:27:07 INFO Received complete push request method=POST path=/50182026/11/12 00:27:07 INFO Completed push id=31 objects=3 duration=1.450393ms50192026/11/12 00:27:07 INFO Received complete upload request method=POST path=/50202026/11/12 00:27:07 INFO Completed upload id=32 objects=4 duration=877.448µs50212026/11/12 00:27:07 INFO Received complete upload request method=POST path=/50222026/11/12 00:27:07 INFO Completed upload id=33 objects=10 duration=982.029µs50232026/11/12 00:27:07 INFO Received complete push request method=POST path=/50242026/11/12 00:27:07 INFO Completed push id=32 objects=6 duration=7.838293ms50252026/11/12 00:27:07 INFO Received complete upload request method=POST path=/50262026/11/12 00:27:07 INFO Completed upload id=34 objects=23 duration=1.141131ms50272026/11/12 00:27:07 INFO Received complete upload request method=POST path=/50282026/11/12 00:27:07 INFO Completed upload id=33 objects=7 duration=835.528µs50292026/11/12 00:27:07 INFO Received complete upload request method=POST path=/50302026/11/12 00:27:07 INFO Completed upload id=34 objects=3 duration=882.628µs50312026/11/12 00:27:07 INFO Received complete push request method=POST path=/50322026/11/12 00:27:07 INFO Completed push id=35 objects=6 duration=1.07839ms50332026/11/12 00:27:07 INFO Received complete push request method=POST path=/50342026/11/12 00:27:07 INFO Completed push id=35 objects=11 duration=987.669µs50352026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=050362026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period50372026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=050382026/11/12 00:27:07 INFO Vacuumed table table=pending_closures50392026/11/12 00:27:07 INFO Vacuumed table table=pending_objects50402026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads50412026/11/12 00:27:07 INFO Vacuumed table table=closures50422026/11/12 00:27:07 INFO Vacuumed table table=objects50432026/11/12 00:27:07 INFO Aborted multipart uploads count=0 kept=050442026/11/12 00:27:07 WARN Force mode enabled - objects will be deleted immediately without grace period50452026/11/12 00:27:07 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050462026/11/12 00:27:07 INFO Vacuumed table table=pending_closures50472026/11/12 00:27:07 INFO Vacuumed table table=pending_objects50482026/11/12 00:27:07 INFO Vacuumed table table=multipart_uploads50492026/11/12 00:27:07 INFO Vacuumed table table=closures50502026/11/12 00:27:07 INFO Vacuumed table table=objects5051--- PASS: FuzzPushGC (0.01s)5052 --- PASS: FuzzPushGC/seed#3 (4.69s)5053 --- PASS: FuzzPushGC/seed#7 (4.89s)5054 --- PASS: FuzzPushGC/seed#4 (5.11s)5055 --- PASS: FuzzPushGC/seed#14 (5.21s)5056 --- PASS: FuzzPushGC/seed#15 (5.37s)5057 --- PASS: FuzzPushGC/seed#0 (5.47s)5058 --- PASS: FuzzPushGC/seed#5 (5.55s)5059 --- PASS: FuzzPushGC/seed#1 (5.56s)5060 --- PASS: FuzzPushGC/seed#10 (5.60s)5061 --- PASS: FuzzPushGC/seed#6 (5.61s)5062 --- PASS: FuzzPushGC/seed#9 (5.63s)5063 --- PASS: FuzzPushGC/seed#11 (5.67s)5064 --- PASS: FuzzPushGC/seed#13 (5.67s)5065 --- PASS: FuzzPushGC/seed#2 (5.67s)5066 --- PASS: FuzzPushGC/seed#12 (5.81s)5067 --- PASS: FuzzPushGC/seed#8 (5.83s)5068PASS5069{"timestamp":"2026-11-12T00:27:07.893279995Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:59760","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(203)"}50702026-11-12 00:27:08.571 UTC [131] LOG: received smart shutdown request50712026-11-12 00:27:08.588 UTC [131] LOG: background worker "logical replication launcher" (PID 149) exited with exit code 150722026-11-12 00:27:09.412 UTC [144] LOG: checkpoint complete: wrote 4376 buffers (26.7%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=6.587 s, sync=0.535 s, total=7.241 s; sync files=42364, longest=0.002 s, average=0.001 s; distance=536294 kB, estimate=536294 kB; lsn=0/26D92170, redo lsn=0/2231F52050732026-11-12 00:27:09.418 UTC [144] LOG: shutting down50742026-11-12 00:27:09.420 UTC [144] LOG: checkpoint starting: shutdown immediate50752026-11-12 00:27:09.822 UTC [144] LOG: checkpoint complete: wrote 9269 buffers (56.6%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.208 s, sync=0.177 s, total=0.404 s; sync files=2762, longest=0.006 s, average=0.001 s; distance=76235 kB, estimate=490288 kB; lsn=0/26D921E8, redo lsn=0/26D921E850762026-11-12 00:27:10.066 UTC [131] LOG: database system is shut down5077Running OIDC tests...5078=== RUN TestAudienceForIssuer5079=== PAUSE TestAudienceForIssuer5080=== RUN TestGlobMatch5081=== PAUSE TestGlobMatch5082=== RUN TestValidateToken_ValidToken5083=== PAUSE TestValidateToken_ValidToken5084=== RUN TestValidateToken_WrongAudience5085=== PAUSE TestValidateToken_WrongAudience5086=== RUN TestValidateToken_Expired5087=== PAUSE TestValidateToken_Expired5088=== RUN TestValidateToken_BoundClaimsMismatch5089=== PAUSE TestValidateToken_BoundClaimsMismatch5090=== RUN TestValidateToken_BoundSubjectMismatch5091=== PAUSE TestValidateToken_BoundSubjectMismatch5092=== RUN TestValidateToken_MultipleProviders5093=== PAUSE TestValidateToken_MultipleProviders5094=== RUN TestValidateToken_NoMatchingProvider5095=== PAUSE TestValidateToken_NoMatchingProvider5096=== RUN TestValidateToken_KubernetesServiceAccount5097=== PAUSE TestValidateToken_KubernetesServiceAccount5098=== RUN TestNewValidator_KubernetesRequiresCA5099=== PAUSE TestNewValidator_KubernetesRequiresCA5100=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5101=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5102=== RUN TestPins_ReservedForMatchingRule5103=== PAUSE TestPins_ReservedForMatchingRule5104=== RUN TestPins_TopLevelShorthand5105=== PAUSE TestPins_TopLevelShorthand5106=== RUN TestPins_ConfigValidation5107=== PAUSE TestPins_ConfigValidation5108=== RUN TestScopes_LegacyProviderDefaultsToWrite5109=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5110=== RUN TestScopes_Rules5111=== PAUSE TestScopes_Rules5112=== RUN TestScopes_ConfigValidation5113=== PAUSE TestScopes_ConfigValidation5114=== CONT TestAudienceForIssuer5115=== CONT TestValidateToken_KubernetesServiceAccount5116--- PASS: TestAudienceForIssuer (0.00s)5117=== CONT TestValidateToken_NoMatchingProvider5118=== CONT TestPins_ConfigValidation5119=== CONT TestValidateToken_Expired5120--- PASS: TestPins_ConfigValidation (0.00s)5121=== CONT TestScopes_ConfigValidation5122--- PASS: TestScopes_ConfigValidation (0.00s)5123=== CONT TestScopes_Rules5124=== CONT TestValidateToken_WrongAudience5125=== CONT TestScopes_LegacyProviderDefaultsToWrite5126=== CONT TestPins_TopLevelShorthand5127=== CONT TestValidateToken_MultipleProviders5128=== CONT TestPins_ReservedForMatchingRule5129=== CONT TestValidateToken_BoundSubjectMismatch5130=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5131=== CONT TestValidateToken_BoundClaimsMismatch5132=== CONT TestNewValidator_KubernetesRequiresCA5133=== CONT TestValidateToken_ValidToken5134=== CONT TestGlobMatch5135=== RUN TestGlobMatch/foo_foo5136=== PAUSE TestGlobMatch/foo_foo5137=== RUN TestGlobMatch/foo_bar5138=== PAUSE TestGlobMatch/foo_bar5139=== RUN TestGlobMatch/*_5140=== PAUSE TestGlobMatch/*_5141=== RUN TestGlobMatch/*_anything5142=== PAUSE TestGlobMatch/*_anything5143=== RUN TestGlobMatch/foo*_foo5144=== PAUSE TestGlobMatch/foo*_foo5145=== RUN TestGlobMatch/foo*_foobar5146=== PAUSE TestGlobMatch/foo*_foobar5147=== RUN TestGlobMatch/foo*_bar5148=== PAUSE TestGlobMatch/foo*_bar5149=== RUN TestGlobMatch/*bar_bar5150=== PAUSE TestGlobMatch/*bar_bar5151=== RUN TestGlobMatch/*bar_foobar5152=== PAUSE TestGlobMatch/*bar_foobar5153=== RUN TestGlobMatch/*bar_foo5154=== PAUSE TestGlobMatch/*bar_foo5155=== RUN TestGlobMatch/foo*bar_foobar5156=== PAUSE TestGlobMatch/foo*bar_foobar5157=== RUN TestGlobMatch/foo*bar_foo123bar5158=== PAUSE TestGlobMatch/foo*bar_foo123bar5159=== RUN TestGlobMatch/foo*bar_foobarbaz5160=== PAUSE TestGlobMatch/foo*bar_foobarbaz5161=== RUN TestGlobMatch/*/*_foo/bar5162=== PAUSE TestGlobMatch/*/*_foo/bar5163=== RUN TestGlobMatch/*/*_foo5164=== PAUSE TestGlobMatch/*/*_foo5165=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5166=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5167=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05168=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05169=== RUN TestGlobMatch/refs/*/main_refs/heads/main5170=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5171=== RUN TestGlobMatch/fo?_foo5172=== PAUSE TestGlobMatch/fo?_foo5173=== RUN TestGlobMatch/fo?_fo5174=== PAUSE TestGlobMatch/fo?_fo5175=== RUN TestGlobMatch/fo?_fooo5176=== PAUSE TestGlobMatch/fo?_fooo5177=== RUN TestGlobMatch/?oo_foo5178=== PAUSE TestGlobMatch/?oo_foo5179=== RUN TestGlobMatch/?oo_boo5180=== PAUSE TestGlobMatch/?oo_boo5181=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5182=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5183=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5184=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5185=== CONT TestGlobMatch/foo_foo5186=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5187=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5188=== CONT TestGlobMatch/?oo_boo5189=== CONT TestGlobMatch/?oo_foo5190=== CONT TestGlobMatch/fo?_fooo5191=== CONT TestGlobMatch/fo?_fo5192=== CONT TestGlobMatch/fo?_foo5193=== CONT TestGlobMatch/refs/*/main_refs/heads/main5194=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05195=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5196=== CONT TestGlobMatch/*/*_foo5197=== CONT TestGlobMatch/*/*_foo/bar5198=== CONT TestGlobMatch/foo*bar_foobarbaz5199=== CONT TestGlobMatch/foo*bar_foo123bar5200=== CONT TestGlobMatch/foo*bar_foobar5201=== CONT TestGlobMatch/*bar_foo5202=== CONT TestGlobMatch/*bar_foobar5203=== CONT TestGlobMatch/*bar_bar5204=== CONT TestGlobMatch/foo*_bar5205=== CONT TestGlobMatch/foo*_foobar5206=== CONT TestGlobMatch/foo*_foo5207=== CONT TestGlobMatch/*_anything5208=== CONT TestGlobMatch/*_5209=== CONT TestGlobMatch/foo_bar5210--- PASS: TestGlobMatch (0.00s)5211 --- PASS: TestGlobMatch/foo_foo (0.00s)5212 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5213 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5214 --- PASS: TestGlobMatch/?oo_boo (0.00s)5215 --- PASS: TestGlobMatch/?oo_foo (0.00s)5216 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5217 --- PASS: TestGlobMatch/fo?_fo (0.00s)5218 --- PASS: TestGlobMatch/fo?_foo (0.00s)5219 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5220 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5221 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5222 --- PASS: TestGlobMatch/*/*_foo (0.00s)5223 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5224 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5225 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5226 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5227 --- PASS: TestGlobMatch/*bar_foo (0.00s)5228 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5229 --- PASS: TestGlobMatch/*bar_bar (0.00s)5230 --- PASS: TestGlobMatch/foo*_bar (0.00s)5231 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5232 --- PASS: TestGlobMatch/foo*_foo (0.00s)5233 --- PASS: TestGlobMatch/*_anything (0.00s)5234 --- PASS: TestGlobMatch/*_ (0.00s)5235 --- PASS: TestGlobMatch/foo_bar (0.00s)52362026/11/12 00:27:14 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35653/oidc52372026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42081/oidc52382026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46367/oidc5239--- PASS: TestValidateToken_Expired (2.46s)52402026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37541/oidc5241--- PASS: TestScopes_LegacyProviderDefaultsToWrite (2.44s)52422026/11/12 00:27:16 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235243--- PASS: TestValidateToken_BoundSubjectMismatch (2.46s)5244--- PASS: TestValidateToken_BoundClaimsMismatch (2.45s)5245--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (2.47s)52462026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41805/oidc5247--- PASS: TestPins_ReservedForMatchingRule (2.52s)52482026/11/12 00:27:16 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:359575249--- PASS: TestValidateToken_KubernetesServiceAccount (2.57s)52502026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46815/oidc5251--- PASS: TestPins_TopLevelShorthand (2.57s)52522026/11/12 00:27:16 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:44455/oidc52532026/11/12 00:27:16 http: TLS handshake error from 127.0.0.1:33512: remote error: tls: bad certificate5254--- PASS: TestNewValidator_KubernetesRequiresCA (2.56s)52552026/11/12 00:27:16 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:38195/oidc5256--- PASS: TestValidateToken_MultipleProviders (2.60s)52572026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35535/oidc5258--- PASS: TestScopes_Rules (2.68s)52592026/11/12 00:27:16 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:36309/oidc5260--- PASS: TestValidateToken_NoMatchingProvider (2.70s)52612026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33889/oidc5262--- PASS: TestValidateToken_WrongAudience (2.69s)52632026/11/12 00:27:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44413/oidc5264--- PASS: TestValidateToken_ValidToken (1.97s)5265PASS5266Running hook tests...5267=== RUN TestSendPathsEmpty5268=== PAUSE TestSendPathsEmpty5269=== RUN TestQueueEnqueueAndFetch5270=== PAUSE TestQueueEnqueueAndFetch5271=== RUN TestQueueDeduplication5272=== PAUSE TestQueueDeduplication5273=== RUN TestQueueRemove5274=== PAUSE TestQueueRemove5275=== RUN TestQueueFetchBatchLimit5276=== PAUSE TestQueueFetchBatchLimit5277=== RUN TestQueueRetryMovesToBack5278=== PAUSE TestQueueRetryMovesToBack5279=== RUN TestQueueFetchRemoveLifecycle5280=== PAUSE TestQueueFetchRemoveLifecycle5281=== RUN TestQueueConcurrentWriters5282=== PAUSE TestQueueConcurrentWriters5283=== RUN TestQueueRemoveLargeClosure5284=== PAUSE TestQueueRemoveLargeClosure5285=== RUN TestServerClientIntegration5286--- PASS: TestServerClientIntegration (0.02s)5287=== RUN TestServerQueueError52882026/11/12 00:27:16 ERROR Failed to queue paths error="permission denied" count=15289--- PASS: TestServerQueueError (0.00s)5290=== RUN TestGetListenerSocketActivation5291 server_test.go:220: === RUN TestGetListenerSocketActivation5292 --- PASS: TestGetListenerSocketActivation (0.00s)5293 PASS5294 5295--- PASS: TestGetListenerSocketActivation (0.32s)5296=== RUN TestServerStalledClientDoesNotBlockShutdown5297=== PAUSE TestServerStalledClientDoesNotBlockShutdown5298=== RUN TestServerBacksOffOnAcceptErrors5299=== PAUSE TestServerBacksOffOnAcceptErrors5300=== RUN TestDrainIsolatesPoisonPath5301=== PAUSE TestDrainIsolatesPoisonPath5302=== RUN TestRunNotBlockedByPoisonHead5303=== PAUSE TestRunNotBlockedByPoisonHead5304=== RUN TestDrainGivesUpWhenServerDown5305=== PAUSE TestDrainGivesUpWhenServerDown5306=== RUN TestFailedPathPrunedByLaterClosure5307=== PAUSE TestFailedPathPrunedByLaterClosure5308=== RUN TestWorkerUploadsAndRemoves5309=== PAUSE TestWorkerUploadsAndRemoves5310=== RUN TestWorkerSkipsGCdPaths5311=== PAUSE TestWorkerSkipsGCdPaths5312=== RUN TestWorkerPrunesClosureDeps5313=== PAUSE TestWorkerPrunesClosureDeps5314=== RUN TestDrainTimeout5315=== PAUSE TestDrainTimeout5316=== RUN TestWorkerRemoveFailureIsNotProgress5317=== PAUSE TestWorkerRemoveFailureIsNotProgress5318=== RUN TestWorkerKeepsPathItCannotStat5319=== PAUSE TestWorkerKeepsPathItCannotStat5320=== CONT TestSendPathsEmpty5321--- PASS: TestSendPathsEmpty (0.00s)5322=== CONT TestQueueFetchRemoveLifecycle5323=== CONT TestWorkerKeepsPathItCannotStat5324=== CONT TestQueueRetryMovesToBack5325=== CONT TestServerStalledClientDoesNotBlockShutdown5326=== CONT TestQueueRemoveLargeClosure5327=== CONT TestQueueConcurrentWriters5328=== CONT TestServerBacksOffOnAcceptErrors5329=== CONT TestWorkerRemoveFailureIsNotProgress53302000/01/01 00:00:00 ERROR Accept failed error="too many open files"53312000/01/01 00:00:00 ERROR Accept failed error="too many open files"53322000/01/01 00:00:00 ERROR Accept failed error="too many open files"53332000/01/01 00:00:00 ERROR Accept failed error="too many open files"53342000/01/01 00:00:00 ERROR Accept failed error="too many open files"5335=== CONT TestQueueFetchBatchLimit53362000/01/01 00:00:00 ERROR Accept failed error="too many open files"53372000/01/01 00:00:00 ERROR Accept failed error="too many open files"53382000/01/01 00:00:00 ERROR Accept failed error="too many open files"5339--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5340=== CONT TestDrainTimeout5341=== CONT TestQueueRemove5342=== CONT TestWorkerPrunesClosureDeps5343=== CONT TestQueueDeduplication5344=== CONT TestWorkerSkipsGCdPaths5345=== CONT TestQueueEnqueueAndFetch5346=== CONT TestWorkerUploadsAndRemoves5347=== CONT TestFailedPathPrunedByLaterClosure5348=== CONT TestRunNotBlockedByPoisonHead5349=== CONT TestDrainIsolatesPoisonPath5350=== CONT TestDrainGivesUpWhenServerDown53512026/11/12 00:27:16 INFO Upload queue status pending=253522026/11/12 00:27:16 INFO Uploading batch count=25353--- PASS: TestQueueFetchBatchLimit (0.18s)53542026/11/12 00:27:17 INFO Uploading batch count=153552026/11/12 00:27:17 ERROR Upload failed error="upload failed" count=153562026/11/12 00:27:17 ERROR Failed to decode request error="read unix /build/hook3039270864/test.sock->@: i/o timeout"53572026/11/12 00:27:17 INFO Uploading batch count=253582026/11/12 00:27:17 ERROR Upload failed error="upload failed" count=253592026/11/12 00:27:17 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1108844414/002/a53602026/11/12 00:27:17 ERROR Failed to write response error="write unix /build/hook3039270864/test.sock->@: i/o timeout"5361--- PASS: TestServerStalledClientDoesNotBlockShutdown (1.07s)5362--- PASS: TestWorkerUploadsAndRemoves (1.07s)53632026/11/12 00:27:17 INFO Upload queue status pending=35364--- PASS: TestQueueDeduplication (1.07s)53652026/11/12 00:27:18 INFO Uploading batch count=153662026/11/12 00:27:18 ERROR Upload failed error="upload failed" count=153672026/11/12 00:27:18 INFO Upload queue status pending=253682026/11/12 00:27:18 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths3535748300/002/nonexistent53692026/11/12 00:27:18 INFO Uploading batch count=153702026/11/12 00:27:18 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat173586028/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat173586028/002/locked/aaa: permission denied"53712026/11/12 00:27:18 INFO Uploading batch count=453722026/11/12 00:27:18 ERROR Upload failed error="upload failed" count=45373--- PASS: TestQueueEnqueueAndFetch (1.08s)53742026/11/12 00:27:18 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath3239465677/002/bbb53752026/11/12 00:27:18 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat173586028/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat173586028/002/locked/aaa: permission denied"53762026/11/12 00:27:18 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1108844414/002/b53772026/11/12 00:27:18 INFO Uploading batch count=153782026/11/12 00:27:18 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress425930122/002/nonexistent53792026/11/12 00:27:18 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153802026/11/12 00:27:18 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress425930122/002/nonexistent53812026/11/12 00:27:18 INFO Upload queue status pending=253822026/11/12 00:27:18 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat173586028/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat173586028/002/locked/aaa: permission denied"53832026/11/12 00:27:18 INFO Uploading batch count=253842026/11/12 00:27:18 ERROR Upload failed error="upload failed" count=253852026/11/12 00:27:18 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153862026/11/12 00:27:18 INFO Uploading batch count=153872026/11/12 00:27:18 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1108844414/002/c53882000/01/01 00:00:00 INFO Uploading batch count=253892000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253902000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45391--- PASS: TestQueueRetryMovesToBack (1.80s)53922026/11/12 00:27:18 INFO Uploading batch count=153932026/11/12 00:27:18 ERROR Upload failed error="upload failed" count=153942026/11/12 00:27:18 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress425930122/002/nonexistent5395--- PASS: TestDrainTimeout (1.80s)53962026/11/12 00:27:18 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153972026/11/12 00:27:18 INFO Uploading batch count=153982026/11/12 00:27:18 ERROR Drain finished with paths left in queue remaining=15399--- PASS: TestWorkerRemoveFailureIsNotProgress (1.80s)54002026/11/12 00:27:18 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1108844414/002/d54012026/11/12 00:27:18 ERROR Upload failed error="upload failed" count=154022026/11/12 00:27:18 INFO Uploading batch count=15403--- PASS: TestQueueFetchRemoveLifecycle (1.81s)5404--- PASS: TestWorkerSkipsGCdPaths (1.80s)54052026/11/12 00:27:18 ERROR Drain finished with paths left in queue remaining=154062026/11/12 00:27:18 INFO Uploading batch count=254072026/11/12 00:27:18 ERROR Upload failed error="upload failed" count=254082026/11/12 00:27:18 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1108844414/002/e5409--- PASS: TestQueueRemove (1.80s)54102026/11/12 00:27:18 INFO Uploading batch count=154112026/11/12 00:27:18 ERROR Upload failed error="upload failed" count=15412--- PASS: TestWorkerPrunesClosureDeps (1.80s)5413--- PASS: TestWorkerKeepsPathItCannotStat (1.81s)54142026/11/12 00:27:18 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1108844414/002/f54152026/11/12 00:27:18 ERROR Drain finished with paths left in queue remaining=15416--- PASS: TestFailedPathPrunedByLaterClosure (1.80s)5417--- PASS: TestDrainIsolatesPoisonPath (1.80s)54182026/11/12 00:27:18 ERROR Drain finished with paths left in queue remaining=105419--- PASS: TestDrainGivesUpWhenServerDown (1.81s)5420--- PASS: TestQueueConcurrentWriters (1.92s)5421--- PASS: TestQueueRemoveLargeClosure (1.94s)54222026/11/12 00:27:19 INFO Uploading batch count=154232026/11/12 00:27:19 INFO Uploading batch count=154242026/11/12 00:27:19 INFO Uploading batch count=154252026/11/12 00:27:19 ERROR Upload failed error="upload failed" count=154262026/11/12 00:27:19 INFO Uploading batch count=154272026/11/12 00:27:19 ERROR Upload failed error="upload failed" count=154282026/11/12 00:27:19 INFO Uploading batch count=154292026/11/12 00:27:19 ERROR Upload failed error="upload failed" count=154302026/11/12 00:27:19 INFO Uploading batch count=154312026/11/12 00:27:19 ERROR Upload failed error="upload failed" count=154322026/11/12 00:27:19 ERROR Drain finished with paths left in queue remaining=15433--- PASS: TestRunNotBlockedByPoisonHead (2.08s)5434PASS5435Running signing tests...5436=== RUN TestExternalSigner5437=== PAUSE TestExternalSigner5438=== RUN TestExternalSignerWrongPublicKey5439=== PAUSE TestExternalSignerWrongPublicKey5440=== RUN TestExternalSignerErrors5441=== PAUSE TestExternalSignerErrors5442=== RUN TestExternalSignerRestartsAfterCrash5443=== PAUSE TestExternalSignerRestartsAfterCrash5444=== RUN TestExternalSignerProgramErrorKeepsProcess5445=== PAUSE TestExternalSignerProgramErrorKeepsProcess5446=== RUN TestExternalSignerClose5447=== PAUSE TestExternalSignerClose5448=== RUN TestExternalSignerRejectsInvalidPublicKey5449=== PAUSE TestExternalSignerRejectsInvalidPublicKey5450=== RUN TestExternalSignerCancellation5451=== PAUSE TestExternalSignerCancellation5452=== RUN TestGenerateFingerprint5453=== PAUSE TestGenerateFingerprint5454=== RUN TestParseSigningKey5455=== PAUSE TestParseSigningKey5456=== RUN TestSign5457=== PAUSE TestSign5458=== RUN TestSignBatch5459=== PAUSE TestSignBatch5460=== RUN TestNilKeyPublicKey5461=== PAUSE TestNilKeyPublicKey5462=== RUN TestSignErrors5463=== PAUSE TestSignErrors5464=== CONT TestExternalSigner5465=== CONT TestExternalSignerCancellation5466=== CONT TestExternalSignerProgramErrorKeepsProcess5467=== CONT TestSignErrors5468=== CONT TestNilKeyPublicKey5469--- PASS: TestNilKeyPublicKey (0.00s)5470=== CONT TestSignBatch5471=== CONT TestSign5472=== CONT TestParseSigningKey5473=== RUN TestParseSigningKey/valid_32-byte_key5474=== PAUSE TestParseSigningKey/valid_32-byte_key5475=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5476=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5477=== RUN TestParseSigningKey/no_colon5478=== PAUSE TestParseSigningKey/no_colon5479=== RUN TestParseSigningKey/empty_name5480=== PAUSE TestParseSigningKey/empty_name5481=== RUN TestParseSigningKey/invalid_base645482=== PAUSE TestParseSigningKey/invalid_base645483=== RUN TestParseSigningKey/wrong_length5484=== PAUSE TestParseSigningKey/wrong_length5485=== CONT TestParseSigningKey/valid_32-byte_key5486=== CONT TestGenerateFingerprint5487=== RUN TestGenerateFingerprint/basic_with_references5488=== PAUSE TestGenerateFingerprint/basic_with_references5489=== RUN TestGenerateFingerprint/no_references5490=== PAUSE TestGenerateFingerprint/no_references5491=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5492=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5493=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5494=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5495=== RUN TestGenerateFingerprint/invalid_nar_hash_length5496=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5497=== RUN TestGenerateFingerprint/invalid_store_path_prefix5498=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5499=== RUN TestGenerateFingerprint/invalid_reference_prefix5500=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5501=== CONT TestParseSigningKey/no_colon5502=== CONT TestParseSigningKey/wrong_length5503=== CONT TestExternalSignerErrors5504=== RUN TestExternalSignerErrors/invalid_JSON5505=== PAUSE TestExternalSignerErrors/invalid_JSON5506=== RUN TestExternalSignerErrors/null5507=== PAUSE TestExternalSignerErrors/null5508=== RUN TestExternalSignerErrors/wrong_id5509=== PAUSE TestExternalSignerErrors/wrong_id5510=== RUN TestExternalSignerErrors/null_signatures5511=== PAUSE TestExternalSignerErrors/null_signatures5512=== RUN TestExternalSignerErrors/wrong_signatures_shape5513=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5514=== RUN TestExternalSignerErrors/unrequested_key5515=== PAUSE TestExternalSignerErrors/unrequested_key5516=== RUN TestExternalSignerErrors/wrong_shape5517=== PAUSE TestExternalSignerErrors/wrong_shape5518=== RUN TestExternalSignerErrors/wrong_signature_name5519=== PAUSE TestExternalSignerErrors/wrong_signature_name5520=== RUN TestExternalSignerErrors/program_error5521=== PAUSE TestExternalSignerErrors/program_error5522=== RUN TestExternalSignerErrors/exit_at_startup5523=== CONT TestParseSigningKey/empty_name5524=== PAUSE TestExternalSignerErrors/exit_at_startup5525=== RUN TestExternalSignerErrors/missing_signatures5526=== PAUSE TestExternalSignerErrors/missing_signatures5527=== RUN TestExternalSignerErrors/missing_signature_name5528=== PAUSE TestExternalSignerErrors/missing_signature_name5529=== RUN TestExternalSignerErrors/wrong_signature_length5530=== PAUSE TestExternalSignerErrors/wrong_signature_length5531=== RUN TestExternalSignerErrors/exit_on_request5532=== PAUSE TestExternalSignerErrors/exit_on_request5533=== RUN TestExternalSignerErrors/too_few_signatures5534=== PAUSE TestExternalSignerErrors/too_few_signatures5535=== CONT TestParseSigningKey/invalid_base645536=== CONT TestExternalSignerClose5537=== RUN TestExternalSignerErrors/invalid_signature_base645538=== PAUSE TestExternalSignerErrors/invalid_signature_base645539=== CONT TestExternalSignerErrors/invalid_JSON5540=== CONT TestExternalSignerWrongPublicKey5541=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5542=== CONT TestExternalSignerErrors/wrong_signature_length5543=== CONT TestGenerateFingerprint/basic_with_references5544=== CONT TestExternalSignerErrors/missing_signatures5545=== CONT TestExternalSignerErrors/exit_at_startup5546=== CONT TestExternalSignerErrors/wrong_signatures_shape5547=== CONT TestGenerateFingerprint/invalid_reference_prefix5548=== CONT TestExternalSignerErrors/wrong_signature_name5549=== CONT TestGenerateFingerprint/invalid_store_path_prefix5550=== CONT TestGenerateFingerprint/invalid_nar_hash_length5551=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5552=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5553=== CONT TestGenerateFingerprint/no_references5554=== CONT TestExternalSignerRestartsAfterCrash5555=== CONT TestExternalSignerErrors/program_error5556=== CONT TestExternalSignerErrors/invalid_signature_base645557=== CONT TestExternalSignerErrors/too_few_signatures5558=== CONT TestExternalSignerErrors/exit_on_request5559=== RUN TestSignErrors/nil_key5560=== PAUSE TestSignErrors/nil_key5561=== RUN TestSignErrors/nil_narinfo5562=== PAUSE TestSignErrors/nil_narinfo5563=== RUN TestSignErrors/invalid_narinfo5564=== PAUSE TestSignErrors/invalid_narinfo5565=== RUN TestSignErrors/canceled5566=== PAUSE TestSignErrors/canceled5567--- PASS: TestSign (0.00s)5568--- PASS: TestParseSigningKey (0.00s)5569 --- PASS: TestParseSigningKey/no_colon (0.00s)5570 --- PASS: TestParseSigningKey/wrong_length (0.00s)5571 --- PASS: TestParseSigningKey/empty_name (0.00s)5572 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5573 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5574 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5575--- PASS: TestSignBatch (0.00s)5576=== CONT TestExternalSignerErrors/wrong_shape5577=== CONT TestExternalSignerErrors/unrequested_key5578=== CONT TestExternalSignerErrors/null_signatures5579=== CONT TestSignErrors/nil_key5580=== CONT TestSignErrors/nil_narinfo5581--- PASS: TestGenerateFingerprint (0.00s)5582 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5583 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5584 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5585 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5586 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5587 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5588 --- PASS: TestGenerateFingerprint/no_references (0.00s)5589=== CONT TestExternalSignerErrors/missing_signature_name5590=== CONT TestSignErrors/canceled5591=== CONT TestExternalSignerRejectsInvalidPublicKey5592=== RUN TestExternalSignerRejectsInvalidPublicKey/#005593=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005594=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5595=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5596=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5597=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5598=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5599=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5600=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5601=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5602=== CONT TestExternalSignerErrors/null5603=== CONT TestExternalSignerErrors/wrong_id5604=== CONT TestSignErrors/invalid_narinfo5605=== CONT TestExternalSignerRejectsInvalidPublicKey/#005606=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5607=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5608=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5609=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5610--- PASS: TestSignErrors (0.01s)5611 --- PASS: TestSignErrors/nil_key (0.00s)5612 --- PASS: TestSignErrors/nil_narinfo (0.00s)5613 --- PASS: TestSignErrors/canceled (0.00s)5614 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5615--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5616 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5617 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5618 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5619 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5620 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5621--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5622--- PASS: TestExternalSignerWrongPublicKey (0.01s)5623--- PASS: TestExternalSignerClose (0.01s)5624--- PASS: TestExternalSigner (0.02s)5625--- PASS: TestExternalSignerErrors (0.00s)5626 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5627 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5628 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5629 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5630 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5631 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5632 --- PASS: TestExternalSignerErrors/program_error (0.01s)5633 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5634 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5635 --- PASS: TestExternalSignerErrors/null (0.01s)5636 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5637 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5638 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5639 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5640 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5641 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5642--- PASS: TestExternalSignerRestartsAfterCrash (0.02s)5643--- PASS: TestExternalSignerCancellation (0.02s)5644PASS5645Running OpenBao signer tests...5646=== RUN TestSign5647=== PAUSE TestSign5648=== RUN TestSignErrors5649=== PAUSE TestSignErrors5650=== RUN TestServe5651=== PAUSE TestServe5652=== RUN TestServeRejectsGarbage5653=== PAUSE TestServeRejectsGarbage5654=== CONT TestSign5655=== CONT TestServeRejectsGarbage5656--- PASS: TestServeRejectsGarbage (0.00s)5657=== CONT TestServe5658--- PASS: TestServe (0.00s)5659=== CONT TestSignErrors5660=== RUN TestSignErrors/http_error5661=== PAUSE TestSignErrors/http_error5662=== RUN TestSignErrors/invalid_json5663=== PAUSE TestSignErrors/invalid_json5664=== RUN TestSignErrors/no_prefix5665=== PAUSE TestSignErrors/no_prefix5666=== RUN TestSignErrors/invalid_base645667=== PAUSE TestSignErrors/invalid_base645668=== CONT TestSignErrors/http_error5669=== CONT TestSignErrors/invalid_base645670=== CONT TestSignErrors/no_prefix5671=== CONT TestSignErrors/invalid_json5672--- PASS: TestSign (3.23s)5673--- PASS: TestSignErrors (0.00s)5674 --- PASS: TestSignErrors/http_error (3.22s)5675 --- PASS: TestSignErrors/invalid_base64 (3.22s)5676 --- PASS: TestSignErrors/no_prefix (3.22s)5677 --- PASS: TestSignErrors/invalid_json (3.22s)5678PASS