niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #498
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.07s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushAnswersEveryLineWhenCanceled86=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestDoServerRequestAttachesToken116=== CONT TestRetryTimeoutOutlastsMaxRetries117=== CONT TestSetClientTLS118=== CONT TestScriptTokenEmptyCommand119--- PASS: TestScriptTokenEmptyCommand (0.00s)120=== CONT TestEncodeNixBase32121=== RUN TestEncodeNixBase32/test_string_hash122=== CONT TestScriptTokenScriptFails123=== CONT TestScriptTokenBadJSON124=== CONT TestScriptTokenEmptyToken125=== CONT TestScriptTokenCachesUntilRefresh126=== CONT TestScriptTokenNoExpiryRerunsEveryCall127=== CONT TestFileTokenEmpty128=== CONT TestFileTokenMissing129=== CONT TestFileTokenReadsAndCaches130=== CONT TestStaticToken131=== CONT TestSetClientTLSErrors132=== CONT TestSetClientTLSDoesNotMutateDefaultTransport133=== CONT TestStreamPushGivesUpOnDeadServer134=== CONT TestStreamPushStopsWhenCanceled135=== CONT TestStreamPushAnswersEveryLineWhenCanceled136=== CONT TestClientSignaturesByStorePath137=== CONT TestStreamPushReportsSkippedPaths138=== CONT TestStreamPushReportsSignatures139=== CONT TestStreamPushRequestLine140=== CONT TestUploadMultipart_PartsInParallel141=== CONT TestEncodeNixBase32WithRealHash142=== PAUSE TestEncodeNixBase32/test_string_hash1432026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:50419144--- PASS: TestStreamPushStopsWhenCanceled (0.00s)145--- PASS: TestFileTokenEmpty (0.00s)146=== CONT TestDumpPathWriterError147=== CONT TestDoWithRetry_BodyReplayedViaGetBody148=== CONT TestRegisterUploadedObject_OutlivesPush149--- PASS: TestFileTokenMissing (0.00s)150--- PASS: TestStaticToken (0.00s)151=== CONT TestResolveStorePath1522026/11/12 00:23:13 ERROR Upload failed error="connection refused" count=201532026/11/12 00:23:13 ERROR Server seems unavailable, giving up on batch untried=17154--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)1552026/11/12 00:23:13 ERROR Upload failed error=boom count=1156--- PASS: TestClientSignaturesByStorePath (0.00s)157=== CONT TestDumpPathSingleFile158=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer159--- PASS: TestDoServerRequestAttachesToken (0.01s)160--- PASS: TestFileTokenReadsAndCaches (0.00s)161=== CONT TestDumpPathMatchesNix162--- PASS: TestEncodeNixBase32WithRealHash (0.00s)163=== RUN TestEncodeNixBase32/empty_input1642000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2165=== CONT TestPartSizeForNAR166=== RUN TestPartSizeForNAR/zero_stays_at_minimum167=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum168=== RUN TestPartSizeForNAR/small_stays_at_minimum169=== PAUSE TestPartSizeForNAR/small_stays_at_minimum170--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.00s)171=== CONT TestRateLimiterFeedback172--- PASS: TestResolveStorePath (0.00s)173=== RUN TestRateLimiterFeedback/429_enables_limiter174=== CONT TestUploadMultipart_ReturnsAfterDroppedBody175=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum176=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum177=== CONT TestPathInfoCACompatibility178=== RUN TestPathInfoCACompatibility/null_ca_field179--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)180=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts181=== PAUSE TestEncodeNixBase32/empty_input182=== PAUSE TestPathInfoCACompatibility/null_ca_field183=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts184=== CONT TestParsePathInfoJSONMultiplePaths185=== RUN TestPathInfoCACompatibility/old_string_format_-_text186=== RUN TestPartSizeForNAR/1_TiB187=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text188=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths189=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths190=== PAUSE TestPartSizeForNAR/1_TiB191=== PAUSE TestRateLimiterFeedback/429_enables_limiter192=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive193=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive194=== RUN TestPathInfoCACompatibility/new_structured_format_-_text195=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text196=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method197=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths198=== RUN TestPartSizeForNAR/5_TiB_S3_max_object199--- PASS: TestStreamPushReportsSkippedPaths (0.00s)200=== RUN TestRateLimiterFeedback/503_enables_limiter201=== PAUSE TestRateLimiterFeedback/503_enables_limiter202=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter203=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter204=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter205=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter206=== CONT TestPathInfoHashCompatibility207=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)208=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)209=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon210=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon2112026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:50419212=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI213--- PASS: TestScriptTokenScriptFails (0.01s)214=== CONT TestGetStorePathHash215=== RUN TestGetStorePathHash/valid_store_path216=== RUN TestSetClientTLSErrors/missing_cert_file217=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI218=== PAUSE TestGetStorePathHash/valid_store_path219=== PAUSE TestSetClientTLSErrors/missing_cert_file220--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)221=== CONT TestConvertHashToNix32222=== RUN TestConvertHashToNix32/SRI_format_to_Nix32223=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32224=== RUN TestConvertHashToNix32/already_Nix32_format2252026/11/12 00:23:13 ERROR Upload failed error=boom count=1226=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512227=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512228=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess229=== PAUSE TestConvertHashToNix32/already_Nix32_format230=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths231=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object2322026/11/12 00:23:13 WARN Rate limiter enabled after throttle name=server-test rate=5233=== RUN TestPartSizeForNAR/capped_at_5_GiB234=== CONT TestParsePathInfoJSON235=== RUN TestGetStorePathHash/basename_without_hyphen_should_error236=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method237=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error238--- PASS: TestStreamPushReportsSignatures (0.01s)239=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun240=== RUN TestSetClientTLSErrors/missing_key_file241=== PAUSE TestSetClientTLSErrors/missing_key_file242=== RUN TestSetClientTLSErrors/missing_ca_file243=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error244=== CONT TestCaseHackSuffix245=== CONT TestFilterOversizedClosures246=== CONT TestRegisterUploadedObjectReusesConnections247=== PAUSE TestSetClientTLSErrors/missing_ca_file248=== RUN TestConvertHashToNix32/invalid_format249=== PAUSE TestConvertHashToNix32/invalid_format250=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry251=== RUN TestParsePathInfoJSON/Nix_format252=== PAUSE TestParsePathInfoJSON/Nix_format253=== RUN TestSetClientTLS/rejects_connection_without_client_cert254=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert255=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA256=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA257=== RUN TestSetClientTLS/preserves_debug_logging_transport258=== PAUSE TestSetClientTLS/preserves_debug_logging_transport259=== RUN TestParsePathInfoJSON/Lix_format260=== PAUSE TestParsePathInfoJSON/Lix_format261=== RUN TestParsePathInfoJSON/empty_input262=== PAUSE TestParsePathInfoJSON/empty_input263=== RUN TestSetClientTLSErrors/invalid_ca_file264=== PAUSE TestSetClientTLSErrors/invalid_ca_file265=== PAUSE TestPartSizeForNAR/capped_at_5_GiB2662026/11/12 00:23:13 WARN Rate limiter enabled after throttle name=server-test rate=52672026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:50422268--- PASS: TestScriptTokenBadJSON (0.02s)269=== CONT TestStreamPushIsolatesFailures270=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica2712026/11/12 00:23:13 ERROR Upload failed error="bad path" count=3272=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error273--- PASS: TestStreamPushIsolatesFailures (0.00s)274=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error275=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error276=== CONT TestShellSplitErrors277--- PASS: TestShellSplitErrors (0.00s)278=== CONT TestDoWithRetry_FinalResponseBodyReadable279=== RUN TestFilterOversizedClosures/no_limit_keeps_everything280=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything281=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped282=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped283=== RUN TestFilterOversizedClosures/all_closures_skipped284=== PAUSE TestFilterOversizedClosures/all_closures_skipped285=== CONT TestStreamPushBatchesUnderLoad286=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false287=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false288=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true289=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true290=== CONT TestShellSplit291--- PASS: TestShellSplit (0.00s)292=== CONT TestRateLimiterFeedback/429_enables_limiter293=== CONT TestEncodeNixBase32/test_string_hash294=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter295=== CONT TestEncodeNixBase32/empty_input296=== CONT TestStreamPushReportsEveryPath297--- PASS: TestEncodeNixBase32 (0.01s)298 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)299 --- PASS: TestEncodeNixBase32/empty_input (0.00s)300=== CONT TestRateLimiterFeedback/503_enables_limiter301--- PASS: TestScriptTokenEmptyToken (0.02s)302=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter303=== RUN TestParsePathInfoJSON/whitespace_only304=== PAUSE TestParsePathInfoJSON/whitespace_only3052026/11/12 00:23:13 WARN Rate limiter backed off name=server-test rate=53062026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:504223072026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:504193082026/11/12 00:23:13 INFO Garbage collection started3092026/11/12 00:23:13 INFO Garbage collection started3102026/11/12 00:23:13 WARN Rate limiter enabled after throttle name=server-test rate=53112026/11/12 00:23:13 WARN Rate limiter enabled after throttle name=server-test rate=53122026/11/12 00:23:13 WARN Rate limiter enabled after throttle name=server-test rate=53132026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:504973142026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:505043152026/11/12 00:23:13 WARN Rate limiter backed off name=server-test rate=53162026/11/12 00:23:13 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=03172026/11/12 00:23:13 WARN Rate limiter backed off name=server-test rate=5318=== RUN TestParsePathInfoJSON/invalid_JSON319=== PAUSE TestParsePathInfoJSON/invalid_JSON320=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon3212026/11/12 00:23:13 WARN Garbage collection finished on another replica. Its statistics are not available here322=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths323=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths324=== CONT TestPathInfoCACompatibility/null_ca_field325=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive326=== CONT TestPathInfoCACompatibility/new_structured_format_-_text327=== CONT TestPathInfoCACompatibility/old_string_format_-_text328=== CONT TestSetClientTLS/rejects_connection_without_client_cert329=== CONT TestSetClientTLS/preserves_debug_logging_transport330--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.02s)331--- PASS: TestStreamPushReportsEveryPath (0.00s)332--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.00s)333=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)334=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512335=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA336=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI337=== CONT TestConvertHashToNix32/SRI_format_to_Nix32338=== CONT TestPartSizeForNAR/zero_stays_at_minimum339=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method340=== CONT TestConvertHashToNix32/invalid_format341=== CONT TestConvertHashToNix32/already_Nix32_format342=== CONT TestPartSizeForNAR/capped_at_5_GiB343=== CONT TestSetClientTLSErrors/missing_ca_file344--- PASS: TestPathInfoHashCompatibility (0.00s)345 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)346 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)347 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)348 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)349=== CONT TestSetClientTLSErrors/invalid_ca_file350=== CONT TestSetClientTLSErrors/missing_key_file351=== CONT TestGetStorePathHash/valid_store_path352=== CONT TestPartSizeForNAR/5_TiB_S3_max_object353=== CONT TestPartSizeForNAR/1_TiB354=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts355=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum356=== CONT TestPartSizeForNAR/small_stays_at_minimum357--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)358 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)359 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)360=== CONT TestFilterOversizedClosures/no_limit_keeps_everything361=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false362=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error363=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error364=== CONT TestGetStorePathHash/basename_without_hyphen_should_error365--- PASS: TestGetStorePathHash (0.01s)366 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)367 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)368 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)369 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)370--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)371--- PASS: TestPathInfoCACompatibility (0.00s)372 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)373 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)374 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)375 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)376 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)377=== CONT TestFilterOversizedClosures/all_closures_skipped378=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true379--- PASS: TestStreamPushBatchesUnderLoad (0.00s)380 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)381 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)382=== CONT TestParsePathInfoJSON/Nix_format383=== CONT TestParsePathInfoJSON/whitespace_only384=== CONT TestParsePathInfoJSON/empty_input385=== CONT TestParsePathInfoJSON/invalid_JSON386=== CONT TestParsePathInfoJSON/Lix_format387--- PASS: TestParsePathInfoJSON (0.01s)388 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)389 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)390 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)391 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)392 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)393=== CONT TestSetClientTLSErrors/missing_cert_file394=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped395--- PASS: TestPartSizeForNAR (0.01s)396 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)397 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)398 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)399 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)400 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)401 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)402 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)403--- PASS: TestConvertHashToNix32 (0.01s)404 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)405 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)406 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)4072026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:50482408--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.02s)4092026/11/12 00:23:13 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=504102026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:504194112026/11/12 00:23:13 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000412--- PASS: TestFilterOversizedClosures (0.00s)413 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)414 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)415 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)4162026/11/12 00:23:13 WARN Rate limiter backed off name=server-test rate=5417--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)4182026/11/12 00:23:13 WARN Rate limiter enabled after throttle name=s3-test rate=54192026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:50516420--- PASS: TestRateLimiterFeedback (0.00s)421 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)422 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)423 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.01s)424 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)4252026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:50419426--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)4272026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:504194282026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:50419429--- PASS: TestSetClientTLSErrors (0.03s)430 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)431 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)432 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)433 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)434--- PASS: TestStreamPushRequestLine (0.04s)4352026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:504194362026/11/12 00:23:13 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:504194372026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:504194382026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:504194392026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:504194402026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:504194412026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:504194422026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:504194432026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:504194442026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:50419445--- PASS: TestRegisterUploadedObjectReusesConnections (0.08s)4462026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:504194472026/11/12 00:23:14 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:50419448--- PASS: TestDumpPathWriterError (0.10s)4492026/11/12 00:23:14 http: TLS handshake error from 127.0.0.1:50518: read tcp 127.0.0.1:50432->127.0.0.1:50518: use of closed network connection450--- PASS: TestSetClientTLS (0.02s)451 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.02s)452 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.02s)453 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.08s)454--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.11s)455--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.11s)4562026/11/12 00:23:14 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:50426/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:50426/api/uploads/complete4572026/11/12 00:23:14 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4582026/11/12 00:23:14 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:50426/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:50426/api/uploads/complete4592026/11/12 00:23:14 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:50426/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:50426/api/uploads/complete4602026/11/12 00:23:14 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4612026/11/12 00:23:14 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"462--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.11s)463--- PASS: TestScriptTokenCachesUntilRefresh (0.12s)464--- PASS: TestDumpPathSingleFile (0.11s)465--- PASS: TestCaseHackSuffix (0.11s)466--- PASS: TestDumpPathMatchesNix (0.16s)467--- PASS: TestUploadMultipart_PartsInParallel (0.63s)468--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.01s)469PASS470Running server tests...471The files belonging to this database system will be owned by user "_tribuchetbld5".472This user must also own the server process.473474The database cluster will be initialized with locale "C".475The default database encoding has accordingly been set to "SQL_ASCII".476The default text search configuration will be set to "english".477478Data page checksums are enabled.479480creating directory /private/var/tb/5/scratch/75b857c59f8a960d/build/postgres3784755263/data ... ok481creating subdirectories ... ok482selecting dynamic shared memory implementation ... posix483selecting default "max_connections" ... 100484selecting default "shared_buffers" ... 128MB485selecting default time zone ... UTC486creating configuration files ... ok487running bootstrap script ... ok488performing post-bootstrap initialization ... ok489syncing data to disk ... ok490491initdb: warning: enabling "trust" authentication for local connections492initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.493494Success. You can now start the database server using:495496 pg_ctl -D /private/var/tb/5/scratch/75b857c59f8a960d/build/postgres3784755263/data -l logfile start497498/private/var/tb/5/scratch/75b857c59f8a960d/build/postgres3784755263:5432 - no response4992026-11-12 00:23:16.155 UTC [68144] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5002026-11-12 00:23:16.155 UTC [68144] LOG: listening on Unix socket "/private/var/tb/5/scratch/75b857c59f8a960d/build/postgres3784755263/.s.PGSQL.5432"5012026-11-12 00:23:16.168 UTC [68163] LOG: database system was shut down at 2026-11-12 00:23:16 UTC5022026-11-12 00:23:16.170 UTC [68144] LOG: database system is ready to accept connections503/private/var/tb/5/scratch/75b857c59f8a960d/build/postgres3784755263:5432 - accepting connections504{"timestamp":"2026-11-12T00:23:16.381474Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"65f5307e-880e-4483-b00a-5a6c2c3f7239","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}505=== RUN TestAbortSucceeded506=== PAUSE TestAbortSucceeded507=== RUN TestService_AuthMiddleware508=== PAUSE TestService_AuthMiddleware509=== RUN TestService_AuthMiddleware_MTLSProxyHeader510=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader511=== RUN TestService_AuthMiddleware_MTLSBoundSubjects512=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects513=== RUN TestService_ReadAuthMiddleware514=== PAUSE TestService_ReadAuthMiddleware515=== RUN TestService_AuthMiddleware_OIDC516=== PAUSE TestService_AuthMiddleware_OIDC517=== RUN TestService_RequireScope_OIDC518=== PAUSE TestService_RequireScope_OIDC519=== RUN TestService_ReadScope_PublicByDefault520=== PAUSE TestService_ReadScope_PublicByDefault521=== RUN TestCacheConfigHandler522=== PAUSE TestCacheConfigHandler523=== RUN TestCacheStatsHandler524=== PAUSE TestCacheStatsHandler525=== RUN TestClientCADerivations526=== PAUSE TestClientCADerivations527=== RUN TestClientErrorHandling528=== PAUSE TestClientErrorHandling529=== RUN TestClientIntegration530=== PAUSE TestClientIntegration531=== RUN TestClientMultipleUploads532=== PAUSE TestClientMultipleUploads533=== RUN TestClientWithDependencies534=== PAUSE TestClientWithDependencies535=== RUN TestClientSharedPathCommittedMidPush536=== PAUSE TestClientSharedPathCommittedMidPush537=== RUN TestPinProtectsFromGC538=== PAUSE TestPinProtectsFromGC539=== RUN TestClientPushesUseOnePush540=== PAUSE TestClientPushesUseOnePush541=== RUN TestClientFallsBackToClosures542=== PAUSE TestClientFallsBackToClosures543=== RUN TestDeleteClosuresRacingPins544=== PAUSE TestDeleteClosuresRacingPins545=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock546=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock547=== RUN TestResolveDBConnectionString548=== PAUSE TestResolveDBConnectionString549=== RUN TestConnectWaitsForAPeerMigration550=== PAUSE TestConnectWaitsForAPeerMigration551=== RUN TestLeadElectsOneAndHandsOver552=== PAUSE TestLeadElectsOneAndHandsOver553=== RUN TestLeadIncumbentWinsAfterRestart5542026/11/12 00:23:16 INFO lead: acquired remote=192.0.2.1:1234 generation=15552026/11/12 00:23:17 INFO lead: released remote=192.0.2.1:1234 generation=15562026/11/12 00:23:17 INFO lead: acquired remote=192.0.2.1:1234 generation=25572026/11/12 00:23:18 INFO lead: released remote=192.0.2.1:1234 generation=2558--- PASS: TestLeadIncumbentWinsAfterRestart (2.10s)559=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent560=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent561=== RUN TestLeadIncumbentAnnouncesAtOnce562=== PAUSE TestLeadIncumbentAnnouncesAtOnce563=== RUN TestLeadGenerationGrowsWithEveryGrant564=== PAUSE TestLeadGenerationGrowsWithEveryGrant565=== RUN TestLeadEndsOnShutdown566=== PAUSE TestLeadEndsOnShutdown567=== RUN TestLeadEndsWhenItsConnectionHangs5682026/11/12 00:23:18 INFO lead: acquired remote=192.0.2.1:1234 generation=15692026-11-12 00:23:20.201 UTC [68472] FATAL: terminating connection due to administrator command5702026/11/12 00:23:20 INFO lead: acquired remote=192.0.2.1:1234 generation=25712026/11/12 00:23:21 WARN lead: lock connection lost error="timeout: context deadline exceeded"5722026/11/12 00:23:21 INFO lead: released remote=192.0.2.1:1234 generation=15732026/11/12 00:23:21 INFO lead: released remote=192.0.2.1:1234 generation=2574--- PASS: TestLeadEndsWhenItsConnectionHangs (3.02s)575=== RUN TestGCAdvisoryLockBlocksConcurrentRun576=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun577=== RUN TestGCBugBareHashReferences578=== PAUSE TestGCBugBareHashReferences579=== RUN TestGCMetrics580=== PAUSE TestGCMetrics581=== RUN TestPushDedupSurvivesConcurrentGC582=== PAUSE TestPushDedupSurvivesConcurrentGC583=== RUN TestDeduplicatedObjectsRecordedAsPending584=== PAUSE TestDeduplicatedObjectsRecordedAsPending585=== RUN TestGCSweepSkipsPendingObjects586=== PAUSE TestGCSweepSkipsPendingObjects587=== RUN TestTombstonedObjectOfferedWithoutWaiting588=== PAUSE TestTombstonedObjectOfferedWithoutWaiting589=== RUN TestGCSweepDeliversEachKeyOnce590=== PAUSE TestGCSweepDeliversEachKeyOnce591=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection592=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection593=== RUN TestForceGCDuringPushOffersSweptObject594=== PAUSE TestForceGCDuringPushOffersSweptObject595=== RUN TestFailedS3DeleteKeepsTombstone596=== PAUSE TestFailedS3DeleteKeepsTombstone597=== RUN TestSweepSkipsRowsLockedByPush598=== PAUSE TestSweepSkipsRowsLockedByPush599=== RUN TestRegisterCompletedObjectNeedsPendingRow600=== PAUSE TestRegisterCompletedObjectNeedsPendingRow601=== RUN TestGetExistingObjectsLocksRows602=== PAUSE TestGetExistingObjectsLocksRows603=== RUN TestSweepRechecksPendingAfterLocking604=== PAUSE TestSweepRechecksPendingAfterLocking605=== RUN TestPushWaitsForSweepHoldingRow606=== PAUSE TestPushWaitsForSweepHoldingRow607=== RUN TestCommitRacingPendingCleanupKeepsObjects608=== PAUSE TestCommitRacingPendingCleanupKeepsObjects609=== RUN TestGCEndsOnShutdown610=== PAUSE TestGCEndsOnShutdown611=== RUN TestGCTaskStore_StartNew612=== PAUSE TestGCTaskStore_StartNew613=== RUN TestGCTaskStore_DeduplicateSameParams614=== PAUSE TestGCTaskStore_DeduplicateSameParams615=== RUN TestGCTaskStore_ConflictDifferentParams616=== PAUSE TestGCTaskStore_ConflictDifferentParams617=== RUN TestGCTaskStore_GetEmpty618=== PAUSE TestGCTaskStore_GetEmpty619=== RUN TestGCTaskStore_GetReturnsLatest620=== PAUSE TestGCTaskStore_GetReturnsLatest621=== RUN TestGCTaskStore_CompletedAllowsNewTask622=== PAUSE TestGCTaskStore_CompletedAllowsNewTask623=== RUN TestGCTaskStore_PhaseUpdates624=== PAUSE TestGCTaskStore_PhaseUpdates625=== RUN TestGCTaskStore_Fail626=== PAUSE TestGCTaskStore_Fail627=== RUN TestGracefulShutdownDrainsInflight628=== PAUSE TestGracefulShutdownDrainsInflight629=== RUN TestService_healthCheckHandler630=== PAUSE TestService_healthCheckHandler631=== RUN TestService_readinessHandler632=== PAUSE TestService_readinessHandler633=== RUN TestGenerateLandingPage634=== PAUSE TestGenerateLandingPage635=== RUN TestEnsureAbortUploadsRule636=== PAUSE TestEnsureAbortUploadsRule637=== RUN TestCacheConfigHandlerMaxNarSize638=== PAUSE TestCacheConfigHandlerMaxNarSize639=== RUN TestCreatePendingClosureRejectsOversizedNAR640=== PAUSE TestCreatePendingClosureRejectsOversizedNAR641=== RUN TestNARDeduplicationMetadataUploadBug642=== PAUSE TestNARDeduplicationMetadataUploadBug643=== RUN TestMetricsInventory644=== PAUSE TestMetricsInventory645=== RUN TestMetricsCommit646=== PAUSE TestMetricsCommit647=== RUN TestService_NativeMTLS648=== PAUSE TestService_NativeMTLS649=== RUN TestServerTLSConfig650=== PAUSE TestServerTLSConfig651=== RUN TestMultipartCleanup652=== PAUSE TestMultipartCleanup653=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded654=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded655=== RUN TestPendingCleanupUsesOneCutoff656=== PAUSE TestPendingCleanupUsesOneCutoff657=== RUN TestObjectStatsTrigger658=== PAUSE TestObjectStatsTrigger659=== RUN TestObjectStatsUpdatedOncePerCommit660=== PAUSE TestObjectStatsUpdatedOncePerCommit661=== RUN TestOrphanedObjectsGC662=== PAUSE TestOrphanedObjectsGC663=== RUN TestOrphanedObjectsGCStressTest664=== PAUSE TestOrphanedObjectsGCStressTest665=== RUN TestCreatePin_ReservedPins666=== PAUSE TestCreatePin_ReservedPins667=== RUN TestPresentReportsOnlyClosureRoots668=== PAUSE TestPresentReportsOnlyClosureRoots669=== RUN TestPresentRefreshesOnlyStaleClosures670=== PAUSE TestPresentRefreshesOnlyStaleClosures671=== RUN TestPresentNotReportedWhileGCDeletesClosure672=== PAUSE TestPresentNotReportedWhileGCDeletesClosure673=== RUN TestParseSingleRange674=== PAUSE TestParseSingleRange675=== RUN TestProxyHeadersOnlyTrustedOnSocket6762026/11/12 00:23:21 INFO Starting HTTP server address=proxy.sock6772026/11/12 00:23:21 INFO Starting HTTP server address=127.0.0.1:505516782026/11/12 00:23:21 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6792026/11/12 00:23:21 INFO Shutdown signal received, draining in-flight requests timeout=10s680--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.18s)681=== RUN TestIsValidCachePath682=== PAUSE TestIsValidCachePath683=== RUN TestReadProxyNarinfo684=== PAUSE TestReadProxyNarinfo685=== RUN TestReadProxyNarinfoAlreadyDecompressed686=== PAUSE TestReadProxyNarinfoAlreadyDecompressed687=== RUN TestReadProxyContentNegotiation688=== PAUSE TestReadProxyContentNegotiation689=== RUN TestReadProxyNarStreaming690=== PAUSE TestReadProxyNarStreaming691=== RUN TestReadProxy404692=== PAUSE TestReadProxy404693=== RUN TestReadProxyInvalidPath694=== PAUSE TestReadProxyInvalidPath695=== RUN TestReadProxyHead696=== PAUSE TestReadProxyHead697=== RUN TestReadProxyOutlastsServerWriteTimeout698=== PAUSE TestReadProxyOutlastsServerWriteTimeout699=== RUN TestReadProxyConditionalGet700=== PAUSE TestReadProxyConditionalGet701=== RUN TestReadProxyRootRedirectsToIndexHTML702=== PAUSE TestReadProxyRootRedirectsToIndexHTML703=== RUN TestReadProxyDisabled704=== PAUSE TestReadProxyDisabled705=== RUN TestReadRedirectNar706=== PAUSE TestReadRedirectNar707=== RUN TestReadRedirectKeepsNarinfoProxied708=== PAUSE TestReadRedirectKeepsNarinfoProxied709=== RUN TestReadProxyRangeRequest710=== PAUSE TestReadProxyRangeRequest711=== RUN TestReadRedirectUsesPublicS3URL712=== PAUSE TestReadRedirectUsesPublicS3URL713=== RUN TestPush_OverlappingRootsStoreOneRowPerKey714=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey715=== RUN TestPush_CompleteCommitsEveryRoot716=== PAUSE TestPush_CompleteCommitsEveryRoot717=== RUN TestPush_RepeatedRootCommitsOnce718=== PAUSE TestPush_RepeatedRootCommitsOnce719=== RUN TestPush_CompleteWaitsForCleanupOfThePush720=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush721=== RUN TestPush_RootsAreLockedInByteOrder722=== PAUSE TestPush_RootsAreLockedInByteOrder723=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit724=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit725=== RUN TestPush_RejectsBadRequests726=== PAUSE TestPush_RejectsBadRequests727=== RUN TestPush_SignsNarinfosOfItsPendingObjects728=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects729=== RUN TestSharedNarBothUploadsComplete730=== PAUSE TestSharedNarBothUploadsComplete731=== RUN TestCompleteMultipartUpload_ErrorButObjectExists732=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists733=== RUN TestCompletedNarNotReofferedAcrossClosures734=== PAUSE TestCompletedNarNotReofferedAcrossClosures735=== RUN TestPresignedUploadRegisteredBeforeCommit736=== PAUSE TestPresignedUploadRegisteredBeforeCommit737=== RUN TestService_Rustfstest738=== PAUSE TestService_Rustfstest739=== RUN TestParseSize740=== PAUSE TestParseSize741=== RUN TestSkippedUploadsHandler742=== PAUSE TestSkippedUploadsHandler743=== RUN TestSystemdListenerNotActivated744--- PASS: TestSystemdListenerNotActivated (0.00s)745=== RUN TestWatchdogBeatsWhenHealthy746--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)747=== RUN TestWatchdogSkipsWhenUnhealthy7482026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7492026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7532026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7542026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7552026/11/12 00:23:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7562026/11/12 00:23:22 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/12 00:23:22 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"758--- PASS: TestWatchdogSkipsWhenUnhealthy (0.21s)759=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle760=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle761=== RUN TestProxyWriteTimeout762=== PAUSE TestProxyWriteTimeout763=== RUN TestPendingClosureWriteTimeout764=== PAUSE TestPendingClosureWriteTimeout765=== RUN TestIsValidUploadKey766=== PAUSE TestIsValidUploadKey767=== RUN TestUploadHandlersRejectInvalidKeys768=== PAUSE TestUploadHandlersRejectInvalidKeys769=== RUN TestUploadHandlersRejectOversizedBody770=== PAUSE TestUploadHandlersRejectOversizedBody771=== RUN TestService_cleanupPendingClosuresHandler772=== PAUSE TestService_cleanupPendingClosuresHandler773=== RUN TestService_createPendingClosureHandler774=== PAUSE TestService_createPendingClosureHandler775=== RUN TestService_verifyS3Integrity776=== PAUSE TestService_verifyS3Integrity777=== RUN TestService_verifyS3ReuploadsStaleNarinfo778=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo779=== RUN TestCompleteMultipartUnregistered780=== PAUSE TestCompleteMultipartUnregistered781=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT782=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT783=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit784=== CONT TestService_createPendingClosureHandler785=== CONT TestReadProxyHead786=== CONT TestProxyWriteTimeout787=== RUN TestProxyWriteTimeout/narinfo788=== CONT TestPush_RootsAreLockedInByteOrder789=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT790=== CONT TestPush_CompleteWaitsForCleanupOfThePush791=== CONT TestCompleteMultipartUnregistered792=== CONT TestPush_RepeatedRootCommitsOnce793=== CONT TestPush_CompleteCommitsEveryRoot794=== CONT TestPush_OverlappingRootsStoreOneRowPerKey795=== CONT TestReadRedirectUsesPublicS3URL796=== CONT TestReadProxyRangeRequest797=== CONT TestReadRedirectKeepsNarinfoProxied798=== CONT TestReadRedirectNar799=== CONT TestReadProxyDisabled800=== CONT TestReadProxyRootRedirectsToIndexHTML801=== CONT TestReadProxyConditionalGet802=== CONT TestService_verifyS3ReuploadsStaleNarinfo803=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed804=== CONT TestReadProxyOutlastsServerWriteTimeout805=== CONT TestService_verifyS3Integrity806=== CONT TestReadProxyInvalidPath807=== CONT TestReadProxy404808=== CONT TestAbortSucceeded809=== PAUSE TestProxyWriteTimeout/narinfo810=== CONT TestReadProxyNarStreaming811=== RUN TestProxyWriteTimeout/1_GiB_nar812=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed813=== PAUSE TestProxyWriteTimeout/1_GiB_nar814--- PASS: TestAbortSucceeded (0.00s)815=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed816=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed817=== RUN TestProxyWriteTimeout/10_GiB_nar818=== PAUSE TestProxyWriteTimeout/10_GiB_nar819=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost820=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost821=== RUN TestProxyWriteTimeout/unknown_size822=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed823=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed824=== PAUSE TestProxyWriteTimeout/unknown_size825=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed826=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed827=== CONT TestReadProxyContentNegotiation828=== CONT TestReadProxyNarinfoAlreadyDecompressed829--- PASS: TestReadProxyNarinfoAlreadyDecompressed (1.10s)830=== CONT TestReadProxyNarinfo831--- PASS: TestReadProxyRangeRequest (1.41s)832=== CONT TestIsValidCachePath833=== RUN TestIsValidCachePath/narinfo834=== PAUSE TestIsValidCachePath/narinfo835=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars836=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars837=== RUN TestIsValidCachePath/nar_zst838=== PAUSE TestIsValidCachePath/nar_zst839=== RUN TestIsValidCachePath/nar_xz840=== PAUSE TestIsValidCachePath/nar_xz841=== RUN TestIsValidCachePath/nar_bz2842=== PAUSE TestIsValidCachePath/nar_bz2843=== RUN TestIsValidCachePath/nar_uncompressed844=== PAUSE TestIsValidCachePath/nar_uncompressed845=== RUN TestIsValidCachePath/ls846=== PAUSE TestIsValidCachePath/ls847=== RUN TestIsValidCachePath/log848=== PAUSE TestIsValidCachePath/log849=== RUN TestIsValidCachePath/realisation850=== PAUSE TestIsValidCachePath/realisation851=== RUN TestIsValidCachePath/nix-cache-info852=== PAUSE TestIsValidCachePath/nix-cache-info853=== RUN TestIsValidCachePath/index.html854=== PAUSE TestIsValidCachePath/index.html855=== RUN TestIsValidCachePath/traversal_parent856=== PAUSE TestIsValidCachePath/traversal_parent857=== RUN TestIsValidCachePath/traversal_in_middle858=== PAUSE TestIsValidCachePath/traversal_in_middle859=== RUN TestIsValidCachePath/invalid_char_e860=== PAUSE TestIsValidCachePath/invalid_char_e861=== RUN TestIsValidCachePath/invalid_char_u862=== PAUSE TestIsValidCachePath/invalid_char_u863=== RUN TestIsValidCachePath/random_path864=== PAUSE TestIsValidCachePath/random_path865=== RUN TestIsValidCachePath/empty866=== PAUSE TestIsValidCachePath/empty867=== RUN TestIsValidCachePath/leading_slash868=== PAUSE TestIsValidCachePath/leading_slash869=== RUN TestIsValidCachePath/wrong_extension870=== PAUSE TestIsValidCachePath/wrong_extension871=== RUN TestIsValidCachePath/short_hash872=== PAUSE TestIsValidCachePath/short_hash873=== CONT TestParseSingleRange874=== RUN TestParseSingleRange/none875=== PAUSE TestParseSingleRange/none876=== RUN TestParseSingleRange/unknown_unit877=== PAUSE TestParseSingleRange/unknown_unit878=== RUN TestParseSingleRange/multi-range_ignored879=== PAUSE TestParseSingleRange/multi-range_ignored880=== RUN TestParseSingleRange/malformed_no_dash881=== PAUSE TestParseSingleRange/malformed_no_dash882=== RUN TestParseSingleRange/malformed_both_empty883=== PAUSE TestParseSingleRange/malformed_both_empty884=== RUN TestParseSingleRange/malformed_end_before_start885=== PAUSE TestParseSingleRange/malformed_end_before_start886=== RUN TestParseSingleRange/closed887=== PAUSE TestParseSingleRange/closed888=== RUN TestParseSingleRange/open-ended889=== PAUSE TestParseSingleRange/open-ended890=== RUN TestParseSingleRange/end_clamped_to_size891=== PAUSE TestParseSingleRange/end_clamped_to_size892=== RUN TestParseSingleRange/suffix893=== PAUSE TestParseSingleRange/suffix894=== RUN TestParseSingleRange/suffix_exceeds_size895=== PAUSE TestParseSingleRange/suffix_exceeds_size896=== RUN TestParseSingleRange/single_byte897=== PAUSE TestParseSingleRange/single_byte898=== RUN TestParseSingleRange/start_past_EOF899=== PAUSE TestParseSingleRange/start_past_EOF900=== RUN TestParseSingleRange/start_far_past_EOF901=== PAUSE TestParseSingleRange/start_far_past_EOF902=== CONT TestPresentNotReportedWhileGCDeletesClosure903=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale904=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale905=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh906=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh907=== CONT TestPresentRefreshesOnlyStaleClosures908--- PASS: TestReadProxyConditionalGet (1.59s)909=== CONT TestPresentReportsOnlyClosureRoots9102026/11/12 00:23:23 INFO Received push request method=POST path=/api/pushes911--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (1.94s)912=== CONT TestCreatePin_ReservedPins9132026/11/12 00:23:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50566/oidc914--- PASS: TestReadProxyContentNegotiation (2.17s)915=== CONT TestOrphanedObjectsGCStressTest916--- PASS: TestReadProxyNarStreaming (2.36s)917=== CONT TestOrphanedObjectsGC918--- PASS: TestReadProxy404 (2.46s)919=== CONT TestObjectStatsUpdatedOncePerCommit9202026/11/12 00:23:24 INFO Received uploads request method=POST path=/api/pending_closures921--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (2.54s)922=== CONT TestObjectStatsTrigger9232026/11/12 00:23:24 INFO Received push request method=POST path=/api/pushes9242026/11/12 00:23:24 INFO Received complete push request method=POST path=/api/pushes/1/complete9252026/11/12 00:23:24 INFO Completed push id=1 objects=4 duration=18.258333ms926--- PASS: TestPush_RepeatedRootCommitsOnce (2.70s)927=== CONT TestPendingCleanupUsesOneCutoff928--- PASS: TestReadRedirectKeepsNarinfoProxied (2.79s)929=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded930--- PASS: TestReadProxyDisabled (2.90s)931=== CONT TestMultipartCleanup932--- PASS: TestReadProxyHead (3.06s)933=== CONT TestPresignedUploadRegisteredBeforeCommit9342026/11/12 00:23:25 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9352026/11/12 00:23:25 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst936--- PASS: TestCompleteMultipartUnregistered (3.16s)937=== CONT TestServerTLSConfig938=== RUN TestServerTLSConfig/no_client_CA939=== PAUSE TestServerTLSConfig/no_client_CA940=== RUN TestServerTLSConfig/missing_CA_file941=== PAUSE TestServerTLSConfig/missing_CA_file942=== RUN TestServerTLSConfig/not_a_PEM_file943=== PAUSE TestServerTLSConfig/not_a_PEM_file944=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle9452026/11/12 00:23:25 INFO Received push request method=POST path=/api/pushes9462026/11/12 00:23:25 INFO Received complete push request method=POST path=/api/pushes/1/complete9472026/11/12 00:23:25 INFO Completed push id=1 objects=6 duration=26.714833ms948--- PASS: TestPush_CompleteCommitsEveryRoot (3.37s)949=== CONT TestSkippedUploadsHandler9502026/11/12 00:23:25 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000951--- PASS: TestSkippedUploadsHandler (0.00s)952=== CONT TestService_NativeMTLS9532026/11/12 00:23:25 INFO Received push request method=POST path=/api/pushes9542026/11/12 00:23:25 INFO Received complete push request method=POST path=/api/pushes/1/complete9552026/11/12 00:23:25 INFO Completed push id=1 objects=4 duration=31.144625ms9562026/11/12 00:23:25 INFO Received push request method=POST path=/api/pushes9572026/11/12 00:23:25 INFO Aborted multipart uploads count=0 kept=09582026/11/12 00:23:25 WARN Force mode enabled - objects will be deleted immediately without grace period9592026/11/12 00:23:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=09602026/11/12 00:23:25 INFO Vacuumed table table=pending_closures9612026/11/12 00:23:25 INFO Vacuumed table table=pending_objects9622026/11/12 00:23:25 INFO Vacuumed table table=multipart_uploads9632026/11/12 00:23:25 INFO Vacuumed table table=closures9642026/11/12 00:23:25 INFO Vacuumed table table=objects9652026/11/12 00:23:25 INFO Received complete push request method=POST path=/api/pushes/2/complete9662026/11/12 00:23:25 INFO Completed push id=2 objects=4 duration=4.516417ms967--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (3.65s)968=== CONT TestParseSize969--- PASS: TestParseSize (0.00s)970=== CONT TestMetricsCommit971--- PASS: TestReadProxyRootRedirectsToIndexHTML (3.68s)972=== CONT TestMetricsInventory9732026/11/12 00:23:25 INFO Received uploads request method=POST path=/api/pending_closures9742026/11/12 00:23:25 INFO Received uploads request method=POST path=/api/pending_closures9752026/11/12 00:23:25 INFO Received uploads request method=POST path=/api/pending_closures976--- PASS: TestReadProxyInvalidPath (3.94s)977=== CONT TestNARDeduplicationMetadataUploadBug978--- PASS: TestReadRedirectUsesPublicS3URL (4.12s)979=== CONT TestService_Rustfstest980--- PASS: TestReadRedirectNar (4.25s)981=== CONT TestCreatePendingClosureRejectsOversizedNAR9822026/11/12 00:23:26 INFO Received uploads request method=POST path=/api/pending_closures983--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)984=== CONT TestUploadHandlersRejectInvalidKeys985=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info986=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info987=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal988=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal989=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key990=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key991=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key992=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key993=== CONT TestCacheConfigHandlerMaxNarSize994--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)995=== CONT TestService_cleanupPendingClosuresHandler9962026/11/12 00:23:26 INFO Received push request method=POST path=/api/pushes9972026-11-12 00:23:26.548 UTC [68809] ERROR: could not obtain lock on row in relation "closures"9982026-11-12 00:23:26.548 UTC [68809] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT9992026-11-12 00:23:26.548 UTC [68809] ERROR: could not obtain lock on row in relation "closures"10002026-11-12 00:23:26.548 UTC [68809] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10012026-11-12 00:23:26.558 UTC [68809] ERROR: could not obtain lock on row in relation "closures"10022026-11-12 00:23:26.558 UTC [68809] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10032026-11-12 00:23:26.559 UTC [68809] ERROR: could not obtain lock on row in relation "closures"10042026-11-12 00:23:26.559 UTC [68809] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10052026-11-12 00:23:26.563 UTC [68809] ERROR: could not obtain lock on row in relation "closures"10062026-11-12 00:23:26.563 UTC [68809] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1007--- PASS: TestPush_RootsAreLockedInByteOrder (4.55s)1008=== CONT TestUploadHandlersRejectOversizedBody10092026/11/12 00:23:26 INFO Received uploads request method=POST path=/api/pending_closures10102026/11/12 00:23:26 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1011=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1012=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1013=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1014=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1015=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1016=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1017=== CONT TestEnsureAbortUploadsRule10182026/11/12 00:23:26 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MDRhNTNkYTktY2ViNi00MWQ0LThkODctYTUxYzFiZmU4YjI3LmQ0ZTJhNTkyLTViMmEtNDRkNy1iZDgxLTQ3NDE2Y2Q0YTQ0OHgxNzk0NDQzMDA1ODQ5NTUzMDAw parts=1010192026/11/12 00:23:26 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10202026/11/12 00:23:26 INFO Completed upload id=1 objects=2 duration=11.52125ms10212026/11/12 00:23:26 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010222026/11/12 00:23:26 INFO Received uploads request method=POST path=/api/pending_closures10232026/11/12 00:23:26 INFO Starting cleanup of old closures method=DELETE path=/api/closures10242026/11/12 00:23:26 INFO Aborted multipart uploads count=0 kept=010252026/11/12 00:23:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010262026/11/12 00:23:26 INFO Vacuumed table table=pending_closures10272026/11/12 00:23:26 INFO Received push request method=POST path=/api/pushes10282026/11/12 00:23:26 INFO Vacuumed table table=pending_objects10292026/11/12 00:23:26 INFO Vacuumed table table=multipart_uploads10302026/11/12 00:23:26 INFO Vacuumed table table=closures10312026/11/12 00:23:26 INFO Vacuumed table table=objects10322026/11/12 00:23:26 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001033--- PASS: TestService_createPendingClosureHandler (4.79s)1034=== CONT TestGCBugBareHashReferences1035--- PASS: TestPresentRefreshesOnlyStaleClosures (3.53s)1036=== CONT TestGenerateLandingPage1037--- PASS: TestGenerateLandingPage (0.00s)1038=== CONT TestGCAdvisoryLockBlocksConcurrentRun10392026-11-12 00:23:27.099 UTC [68833] ERROR: Push does not exist: id=110402026-11-12 00:23:27.099 UTC [68833] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE10412026-11-12 00:23:27.099 UTC [68833] STATEMENT: SELECT commit_push($1)1042--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (5.08s)1043=== CONT TestService_readinessHandler10442026/11/12 00:23:27 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1045--- PASS: TestReadProxyNarinfo (4.05s)1046=== CONT TestLeadEndsOnShutdown1047--- PASS: TestPresentReportsOnlyClosureRoots (3.72s)1048=== CONT TestLeadGenerationGrowsWithEveryGrant10492026/11/12 00:23:27 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10502026/11/12 00:23:27 WARN Refused reserved pin name=worker-x86_64-linux10512026/11/12 00:23:27 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10522026/11/12 00:23:27 INFO Received create pin request method=POST path=/api/pins/my-app10532026/11/12 00:23:27 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1054--- PASS: TestCreatePin_ReservedPins (3.51s)1055=== CONT TestLeadIncumbentAnnouncesAtOnce10562026/11/12 00:23:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10572026/11/12 00:23:27 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MDRhNTNkYTktY2ViNi00MWQ0LThkODctYTUxYzFiZmU4YjI3LmU3NGQwNjI2LTIxMmItNDRhYi1iNjA2LTcyN2U1MDlmOGZkOHgxNzk0NDQzMDA2NjA5ODI2MDAw parts=1010582026/11/12 00:23:27 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10592026/11/12 00:23:27 INFO Completed upload id=1 objects=2 duration=21.81625ms10602026/11/12 00:23:27 INFO Received uploads request method=POST path=/api/pending_closures10612026/11/12 00:23:27 INFO Received uploads request method=POST path=/api/pending_closures10622026/11/12 00:23:27 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10632026/11/12 00:23:27 WARN Found objects in DB but missing from S3, will re-upload count=11064--- PASS: TestService_verifyS3Integrity (5.60s)1065=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent1066--- PASS: TestObjectStatsUpdatedOncePerCommit (3.64s)1067=== CONT TestLeadElectsOneAndHandsOver1068--- PASS: TestObjectStatsTrigger (3.66s)1069=== CONT TestService_healthCheckHandler10702026/11/12 00:23:28 INFO Received uploads request method=POST path=/api/pending_closures10712026/11/12 00:23:28 INFO Received uploads request method=POST path=/api/pending_closures10722026/11/12 00:23:28 INFO Received cleanup request method=DELETE path=/api/pending_closures10732026/11/12 00:23:28 INFO Received uploads request method=POST path=/api/pending_closures10742026/11/12 00:23:28 INFO Received cleanup request method=DELETE path=/api/pending_closures10752026/11/12 00:23:28 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket37/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""10762026/11/12 00:23:28 INFO Aborted multipart uploads count=0 kept=110772026/11/12 00:23:28 INFO Received cleanup request method=DELETE path=/api/pending_closures10782026/11/12 00:23:28 INFO Aborted multipart uploads count=1 kept=010792026/11/12 00:23:28 INFO Received uploads request method=POST path=/api/pending_closures1080--- PASS: TestMultipartCleanup (3.87s)1081=== CONT TestGracefulShutdownDrainsInflight10822026/11/12 00:23:28 INFO Starting HTTP server address=127.0.0.1:5065110832026/11/12 00:23:28 INFO Shutdown signal received, draining in-flight requests timeout=10s1084--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1085=== CONT TestGCTaskStore_Fail1086--- PASS: TestGCTaskStore_Fail (0.00s)1087--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (4.06s)1088=== CONT TestConnectWaitsForAPeerMigration1089=== CONT TestResolveDBConnectionString1090=== RUN TestResolveDBConnectionString/flag_wins1091=== PAUSE TestResolveDBConnectionString/flag_wins1092=== RUN TestResolveDBConnectionString/file_when_flag_empty1093=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1094=== RUN TestResolveDBConnectionString/missing_file_is_an_error1095=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1096=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1097=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1098=== RUN TestResolveDBConnectionString/nothing_configured1099=== PAUSE TestResolveDBConnectionString/nothing_configured1100=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1101=== NAME TestOrphanedObjectsGC1102 orphaned_objects_gc_test.go:287: GC Test Summary:1103 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A11042026/11/12 00:23:29 INFO Received uploads request method=POST path=/api/pending_closures1105 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1106 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1107 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1108 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1109--- PASS: TestOrphanedObjectsGC (4.75s)1110=== CONT TestDeleteClosuresRacingPins11112026/11/12 00:23:29 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst11122026/11/12 00:23:29 INFO Received uploads request method=POST path=/api/pending_closures1113--- PASS: TestPresignedUploadRegisteredBeforeCommit (4.15s)1114=== CONT TestClientFallsBackToClosures11152026/11/12 00:23:29 INFO Received uploads request method=POST path=/api/pending_closures11162026/11/12 00:23:29 WARN mTLS auth: subject not in bound subjects subject="CN=reader"11172026/11/12 00:23:29 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1118--- PASS: TestService_NativeMTLS (4.13s)1119=== CONT TestRegisterCompletedObjectNeedsPendingRow11202026/11/12 00:23:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1121--- PASS: TestMetricsInventory (4.08s)1122=== CONT TestClientPushesUseOnePush1123--- PASS: TestReadProxyOutlastsServerWriteTimeout (7.79s)1124=== CONT TestPinProtectsFromGC11252026/11/12 00:23:29 INFO Received push request method=POST path=/api/pushes11262026/11/12 00:23:30 INFO Received complete push request method=POST path=/api/pushes/1/complete11272026/11/12 00:23:30 INFO Completed push id=1 objects=4 duration=43.1155ms11282026/11/12 00:23:30 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete11292026/11/12 00:23:30 INFO Completed upload id=2 objects=2 duration=1.149042ms1130--- PASS: TestMetricsCommit (4.41s)1131=== CONT TestClientSharedPathCommittedMidPush1132--- PASS: TestService_Rustfstest (4.19s)1133=== CONT TestSweepSkipsRowsLockedByPush1134=== NAME TestNARDeduplicationMetadataUploadBug1135 metadata_upload_test.go:48: First store path: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestNARDeduplicationMetadataUploadBug2329342483/001/store/45j1pxr20vpr4hhdm29igrh8msaspvfa-file1.txt11362026/11/12 00:23:30 INFO Received push request method=POST path=/api/pushes11372026/11/12 00:23:30 INFO Received cleanup request method=DELETE path=/api/pending_closures11382026/11/12 00:23:30 INFO Aborted multipart uploads count=0 kept=011392026/11/12 00:23:30 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11402026/11/12 00:23:30 INFO Uploading 45j1pxr20vpr4hhdm29igrh8msaspvfa-file1.txt (160B)11412026/11/12 00:23:30 INFO Received uploads request method=POST path=/api/pending_closures11422026/11/12 00:23:30 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"11432026/11/12 00:23:30 WARN Failed to register uploaded object key=45j1pxr20vpr4hhdm29igrh8msaspvfa.ls error="server returned 404: 404 page not found\n"11442026/11/12 00:23:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11452026/11/12 00:23:30 INFO Signed narinfos id=1 count=111462026/11/12 00:23:30 INFO Uploading 1 narinfos11472026/11/12 00:23:30 INFO Received cleanup request method=DELETE path=/api/pending_closures11482026/11/12 00:23:30 INFO Aborted multipart uploads count=1 kept=011492026/11/12 00:23:30 INFO Received complete push request method=POST path=/api/pushes/1/complete11502026/11/12 00:23:30 WARN Failed to register uploaded object key=45j1pxr20vpr4hhdm29igrh8msaspvfa.narinfo error="server returned 404: 404 page not found\n"11512026/11/12 00:23:30 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11522026-11-12 00:23:30.734 UTC [68863] ERROR: Closure does not exist: id=111532026-11-12 00:23:30.734 UTC [68863] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE11542026-11-12 00:23:30.734 UTC [68863] STATEMENT: -- name: CommitPendingClosure :one1155 SELECT commit_pending_closure($1::bigint)::bigint1156 1157--- PASS: TestService_cleanupPendingClosuresHandler (4.45s)1158=== CONT TestClientWithDependencies11592026/11/12 00:23:30 INFO Completed push id=1 objects=3 duration=38.517709ms11602026/11/12 00:23:30 INFO Upload complete. (293ms)1161=== NAME TestNARDeduplicationMetadataUploadBug1162 metadata_upload_test.go:54: Retrieved narinfo from S3:1163 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestNARDeduplicationMetadataUploadBug2329342483/001/store/45j1pxr20vpr4hhdm29igrh8msaspvfa-file1.txt1164 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1165 Compression: zstd1166 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1167 NarSize: 1601168 References: 1169 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1170 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1171 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1172 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}11732026/11/12 00:23:30 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket47 days=21174--- PASS: TestEnsureAbortUploadsRule (4.26s)1175=== CONT TestGetExistingObjectsLocksRows1176=== NAME TestNARDeduplicationMetadataUploadBug1177 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/5/scratch/75b857c59f8a960d/build/TestNARDeduplicationMetadataUploadBug2329342483/001/store/0g18290y7xzyira8gknbzq9sir5vhxnl-file2.txt11782026/11/12 00:23:31 INFO Received push request method=POST path=/api/pushes11792026/11/12 00:23:31 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)11802026/11/12 00:23:31 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign11812026/11/12 00:23:31 WARN Failed to register uploaded object key=0g18290y7xzyira8gknbzq9sir5vhxnl.ls error="server returned 404: 404 page not found\n"11822026/11/12 00:23:31 INFO Signed narinfos id=2 count=111832026/11/12 00:23:31 INFO Uploading 1 narinfos11842026/11/12 00:23:31 INFO Received complete push request method=POST path=/api/pushes/2/complete11852026/11/12 00:23:31 WARN Failed to register uploaded object key=0g18290y7xzyira8gknbzq9sir5vhxnl.narinfo error="server returned 404: 404 page not found\n"11862026/11/12 00:23:31 INFO Completed push id=2 objects=3 duration=731.791µs11872026/11/12 00:23:31 INFO Upload complete. (213ms)1188 metadata_upload_test.go:76: Retrieved narinfo from S3:1189 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestNARDeduplicationMetadataUploadBug2329342483/001/store/0g18290y7xzyira8gknbzq9sir5vhxnl-file2.txt1190 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1191 Compression: zstd1192 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1193 NarSize: 1601194 References: 1195 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1196 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1197 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1198 {"version":1,"root":{"type":"regular","size":44}}1199--- PASS: TestNARDeduplicationMetadataUploadBug (5.33s)1200=== CONT TestClientMultipleUploads12012026/11/12 00:23:31 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld5 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1202--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (4.35s)1203=== CONT TestFailedS3DeleteKeepsTombstone1204--- PASS: TestGCBugBareHashReferences (4.53s)1205=== CONT TestGCTaskStore_PhaseUpdates1206--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1207=== CONT TestForceGCDuringPushOffersSweptObject1208=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1209=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1210=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1211=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1212=== CONT TestClientIntegration12132026/11/12 00:23:31 WARN readiness check failed error="closed pool"1214--- PASS: TestService_readinessHandler (4.40s)1215=== CONT TestClientErrorHandling1216=== RUN TestClientErrorHandling/InvalidStorePath1217=== PAUSE TestClientErrorHandling/InvalidStorePath1218=== RUN TestClientErrorHandling/InvalidAuthToken1219=== PAUSE TestClientErrorHandling/InvalidAuthToken1220=== RUN TestClientErrorHandling/ServerNotAvailable1221=== PAUSE TestClientErrorHandling/ServerNotAvailable1222=== CONT TestGCTaskStore_CompletedAllowsNewTask1223--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1224=== CONT TestGCTaskStore_GetReturnsLatest1225--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1226=== CONT TestGCTaskStore_GetEmpty1227--- PASS: TestGCTaskStore_GetEmpty (0.00s)1228=== CONT TestGCTaskStore_ConflictDifferentParams1229--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1230=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection12312026/11/12 00:23:31 INFO lead: acquired remote=192.0.2.1:1234 generation=112322026/11/12 00:23:31 INFO lead: acquired remote=192.0.2.1:1234 generation=112332026/11/12 00:23:32 INFO lead: acquired remote=192.0.2.1:1234 generation=112342026/11/12 00:23:32 INFO lead: released remote=192.0.2.1:1234 generation=11235--- PASS: TestLeadIncumbentAnnouncesAtOnce (4.70s)1236=== CONT TestGCTaskStore_DeduplicateSameParams1237--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1238=== CONT TestGCSweepDeliversEachKeyOnce12392026/11/12 00:23:32 INFO lead: acquired remote=192.0.2.1:1234 generation=112402026/11/12 00:23:32 INFO lead: acquired remote=192.0.2.1:1234 generation=11241--- PASS: TestService_healthCheckHandler (4.52s)1242=== CONT TestTombstonedObjectOfferedWithoutWaiting12432026/11/12 00:23:32 WARN Rate limiter enabled after throttle name=s3-test rate=512442026/11/12 00:23:32 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1245=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1246 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101247 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001248--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (7.61s)1249=== CONT TestGCSweepSkipsPendingObjects12502026/11/12 00:23:32 INFO lead: released remote=192.0.2.1:1234 generation=11251--- PASS: TestLeadEndsOnShutdown (5.66s)1252=== CONT TestDeduplicatedObjectsRecordedAsPending12532026/11/12 00:23:33 INFO lead: released remote=192.0.2.1:1234 generation=112542026/11/12 00:23:33 INFO lead: acquired remote=192.0.2.1:1234 generation=212552026/11/12 00:23:33 INFO lead: released remote=192.0.2.1:1234 generation=112562026/11/12 00:23:33 INFO lead: acquired remote=192.0.2.1:1234 generation=21257--- PASS: TestRegisterCompletedObjectNeedsPendingRow (4.03s)1258=== CONT TestGCTaskStore_StartNew1259--- PASS: TestGCTaskStore_StartNew (0.00s)1260=== CONT TestGCEndsOnShutdown1261=== RUN TestGCEndsOnShutdown/before_the_run1262=== PAUSE TestGCEndsOnShutdown/before_the_run1263=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1264=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1265=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1266=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1267=== CONT TestCommitRacingPendingCleanupKeepsObjects12682026/11/12 00:23:33 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=212692026/11/12 00:23:33 INFO lead: acquired remote=192.0.2.1:1234 generation=31270--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (4.87s)1271=== CONT TestPushWaitsForSweepHoldingRow12722026/11/12 00:23:33 INFO lead: released remote=192.0.2.1:1234 generation=112732026/11/12 00:23:33 INFO lead: acquired remote=192.0.2.1:1234 generation=212742026/11/12 00:23:33 INFO Received uploads request method=POST path=/api/pending_closures12752026/11/12 00:23:33 INFO Received uploads request method=POST path=/api/pending_closures12762026/11/12 00:23:33 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)12772026/11/12 00:23:33 INFO Uploading 146nk49gr34aypywb2qdd2sa1q1y4sjb-b (256B)12782026/11/12 00:23:33 INFO Uploading ipxjbqz484m1qbfzcdrwn4hh02dz4kg2-shared-dep (136B)12792026/11/12 00:23:33 WARN Failed to register uploaded object key=146nk49gr34aypywb2qdd2sa1q1y4sjb.ls error="server returned 404: 404 page not found\n"12802026/11/12 00:23:33 WARN Failed to register uploaded object key=4xivhqfxph5p2wfv4xili6ah34rc7xn3.ls error="server returned 404: 404 page not found\n"12812026/11/12 00:23:33 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12822026/11/12 00:23:33 WARN Failed to register uploaded object key=ipxjbqz484m1qbfzcdrwn4hh02dz4kg2.ls error="server returned 404: 404 page not found\n"12832026/11/12 00:23:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign12842026/11/12 00:23:33 WARN Failed to register uploaded object key=nar/1g6b3ass6j7j4yyzd7cyr1zika631xdmfnyxi99q4c93xbaqg4gz.nar.zst error="server returned 404: 404 page not found\n"12852026/11/12 00:23:34 INFO Signed narinfos id=1 count=212862026/11/12 00:23:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign12872026/11/12 00:23:34 INFO Signed narinfos id=2 count=212882026/11/12 00:23:34 INFO Uploading 4 narinfos12892026/11/12 00:23:34 WARN Failed to register uploaded object key=4xivhqfxph5p2wfv4xili6ah34rc7xn3.narinfo error="server returned 404: 404 page not found\n"12902026/11/12 00:23:34 WARN Failed to register uploaded object key=146nk49gr34aypywb2qdd2sa1q1y4sjb.narinfo error="server returned 404: 404 page not found\n"12912026/11/12 00:23:34 WARN Failed to register uploaded object key=ipxjbqz484m1qbfzcdrwn4hh02dz4kg2.narinfo error="server returned 404: 404 page not found\n"1292=== NAME TestPinProtectsFromGC1293 client_integration_test.go:779: Pinned store path: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestPinProtectsFromGC553048633/001/store/06ndhaz9wdvpm0qap2di5qjrbjxv3asn-pinned-file.txt1294 client_integration_test.go:780: Unpinned store path: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestPinProtectsFromGC553048633/001/store/xhf90vaj6rpr86whzlbgp8didnia3z1b-unpinned-file.txt12952026/11/12 00:23:34 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12962026/11/12 00:23:34 WARN Failed to register uploaded object key=ipxjbqz484m1qbfzcdrwn4hh02dz4kg2.narinfo error="server returned 404: 404 page not found\n"12972026/11/12 00:23:34 INFO Completed upload id=1 objects=6 duration=52.391666ms12982026/11/12 00:23:34 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12992026/11/12 00:23:34 INFO Completed upload id=2 objects=6 duration=2.794333ms13002026/11/12 00:23:34 INFO lead: released remote=192.0.2.1:1234 generation=213012026/11/12 00:23:34 INFO Upload complete. (398ms)1302=== NAME TestClientFallsBackToClosures13032026/11/12 00:23:34 INFO lead: acquired remote=192.0.2.1:1234 generation=31304 client_pushes_test.go:112: Retrieved narinfo from S3:13052026/11/12 00:23:34 INFO lead: released remote=192.0.2.1:1234 generation=31306 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientFallsBackToClosures2761900950/001/store/ipxjbqz484m1qbfzcdrwn4hh02dz4kg2-shared-dep1307 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1308 Compression: zstd1309 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821310 NarSize: 1361311 References: 1312--- PASS: TestLeadGenerationGrowsWithEveryGrant (6.86s)1313=== CONT TestSweepRechecksPendingAfterLocking1314=== NAME TestClientFallsBackToClosures1315 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1316 client_pushes_test.go:112: Retrieved narinfo from S3:1317 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientFallsBackToClosures2761900950/001/store/4xivhqfxph5p2wfv4xili6ah34rc7xn3-a1318 URL: nar/1g6b3ass6j7j4yyzd7cyr1zika631xdmfnyxi99q4c93xbaqg4gz.nar.zst1319 Compression: zstd1320 NarHash: sha256:1g6b3ass6j7j4yyzd7cyr1zika631xdmfnyxi99q4c93xbaqg4gz1321 NarSize: 2561322 References: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientFallsBackToClosures2761900950/001/store/ipxjbqz484m1qbfzcdrwn4hh02dz4kg2-shared-dep1323 CA: text:sha256:1l0wnwfvrhxw7x2hvqjgqmpbbkkvyvdj78k8a111s8c2v0zsph881324 client_pushes_test.go:112: Retrieved narinfo from S3:1325 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientFallsBackToClosures2761900950/001/store/146nk49gr34aypywb2qdd2sa1q1y4sjb-b1326 URL: nar/1g6b3ass6j7j4yyzd7cyr1zika631xdmfnyxi99q4c93xbaqg4gz.nar.zst1327 Compression: zstd1328 NarHash: sha256:1g6b3ass6j7j4yyzd7cyr1zika631xdmfnyxi99q4c93xbaqg4gz1329 NarSize: 2561330 References: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientFallsBackToClosures2761900950/001/store/ipxjbqz484m1qbfzcdrwn4hh02dz4kg2-shared-dep1331 CA: text:sha256:1l0wnwfvrhxw7x2hvqjgqmpbbkkvyvdj78k8a111s8c2v0zsph881332--- PASS: TestClientFallsBackToClosures (5.02s)1333=== CONT TestPushDedupSurvivesConcurrentGC13342026/11/12 00:23:34 INFO Received push request method=POST path=/api/pushes13352026/11/12 00:23:34 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13362026/11/12 00:23:34 INFO Uploading 06ndhaz9wdvpm0qap2di5qjrbjxv3asn-pinned-file.txt (128B)13372026/11/12 00:23:34 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"13382026/11/12 00:23:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13392026/11/12 00:23:34 WARN Failed to register uploaded object key=06ndhaz9wdvpm0qap2di5qjrbjxv3asn.ls error="server returned 404: 404 page not found\n"13402026/11/12 00:23:34 INFO Signed narinfos id=1 count=113412026/11/12 00:23:34 INFO Uploading 1 narinfos13422026/11/12 00:23:34 INFO Received complete push request method=POST path=/api/pushes/1/complete13432026/11/12 00:23:34 WARN Failed to register uploaded object key=06ndhaz9wdvpm0qap2di5qjrbjxv3asn.narinfo error="server returned 404: 404 page not found\n"13442026/11/12 00:23:34 INFO Completed push id=1 objects=3 duration=23.442625ms13452026/11/12 00:23:34 INFO Upload complete. (280ms)13462026/11/12 00:23:34 INFO Aborted multipart uploads count=1 kept=013472026/11/12 00:23:34 INFO Received cleanup request method=DELETE path=/api/pending_closures13482026/11/12 00:23:34 INFO Aborted multipart uploads count=1 kept=01349--- PASS: TestPendingCleanupUsesOneCutoff (9.80s)1350=== CONT TestGCMetrics13512026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=013522026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period13532026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013542026/11/12 00:23:34 INFO Vacuumed table table=pending_closures13552026/11/12 00:23:34 INFO Vacuumed table table=pending_objects13562026/11/12 00:23:34 INFO Received push request method=POST path=/api/pushes13572026/11/12 00:23:34 INFO Vacuumed table table=multipart_uploads13582026/11/12 00:23:34 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13592026/11/12 00:23:34 INFO Vacuumed table table=closures13602026/11/12 00:23:34 INFO Uploading xhf90vaj6rpr86whzlbgp8didnia3z1b-unpinned-file.txt (128B)13612026/11/12 00:23:34 INFO Vacuumed table table=objects13622026/11/12 00:23:34 INFO Received push request method=POST path=/api/pushes13632026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=013642026/11/12 00:23:34 WARN Failed to register uploaded object key=xhf90vaj6rpr86whzlbgp8didnia3z1b.ls error="server returned 404: 404 page not found\n"13652026/11/12 00:23:34 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13662026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period13672026/11/12 00:23:34 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"13682026/11/12 00:23:34 INFO Signed narinfos id=2 count=113692026/11/12 00:23:34 INFO Uploading 1 narinfos13702026/11/12 00:23:34 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)13712026/11/12 00:23:34 INFO Uploading jphrzm4yy6ckdk4y5980124r93x2w2i4-b (256B)13722026/11/12 00:23:34 INFO Uploading 65x04qx6nr2incmcz98j9r7hrniy4q58-shared-dep (136B)13732026/11/12 00:23:34 INFO Received complete push request method=POST path=/api/pushes/2/complete13742026/11/12 00:23:34 WARN Failed to register uploaded object key=xhf90vaj6rpr86whzlbgp8didnia3z1b.narinfo error="server returned 404: 404 page not found\n"13752026/11/12 00:23:34 WARN Failed to register uploaded object key=jphrzm4yy6ckdk4y5980124r93x2w2i4.ls error="server returned 404: 404 page not found\n"13762026/11/12 00:23:34 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13772026/11/12 00:23:34 WARN Failed to register uploaded object key=65x04qx6nr2incmcz98j9r7hrniy4q58.ls error="server returned 404: 404 page not found\n"13782026/11/12 00:23:34 WARN Failed to register uploaded object key=6ggrkqjq4q2781zw2nld19qnsi84cafy.ls error="server returned 404: 404 page not found\n"13792026/11/12 00:23:34 INFO Received push request method=POST path=/api/pushes13802026/11/12 00:23:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13812026/11/12 00:23:34 WARN Failed to register uploaded object key=nar/0p1gkk5snm1cswgikbrx279rilr0p0d3vq7xzp1d59v7v0i2y4kr.nar.zst error="server returned 404: 404 page not found\n"13822026/11/12 00:23:34 INFO Signed narinfos id=1 count=313832026/11/12 00:23:34 INFO Completed push id=2 objects=3 duration=41.594667ms13842026/11/12 00:23:34 INFO Uploading 3 narinfos13852026/11/12 00:23:34 INFO Upload complete. (241ms)13862026/11/12 00:23:34 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)13872026/11/12 00:23:34 INFO Uploading zvbss64il6x4jljzz6hlr0i7nn6mzsfp-top (264B)13882026/11/12 00:23:34 INFO Uploading cznc405mvv2w7qwpg1nhldb9vn43xlap-shared-dep (136B)13892026/11/12 00:23:34 WARN Failed to register uploaded object key=6ggrkqjq4q2781zw2nld19qnsi84cafy.narinfo error="server returned 404: 404 page not found\n"13902026/11/12 00:23:34 WARN Failed to register uploaded object key=jphrzm4yy6ckdk4y5980124r93x2w2i4.narinfo error="server returned 404: 404 page not found\n"13912026/11/12 00:23:34 INFO Received complete push request method=POST path=/api/pushes/1/complete13922026/11/12 00:23:34 WARN Failed to register uploaded object key=65x04qx6nr2incmcz98j9r7hrniy4q58.narinfo error="server returned 404: 404 page not found\n"13932026/11/12 00:23:34 WARN Failed to register uploaded object key=zvbss64il6x4jljzz6hlr0i7nn6mzsfp.ls error="server returned 404: 404 page not found\n"13942026/11/12 00:23:34 WARN Failed to register uploaded object key=cznc405mvv2w7qwpg1nhldb9vn43xlap.ls error="server returned 404: 404 page not found\n"13952026/11/12 00:23:34 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13962026/11/12 00:23:34 INFO Completed push id=1 objects=8 duration=35.651084ms13972026/11/12 00:23:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13982026/11/12 00:23:34 WARN Failed to register uploaded object key=nar/19fnxnzmz822is134az3jagqqvzqlpcbsgamv04maalc0bd6wzb7.nar.zst error="server returned 404: 404 page not found\n"13992026/11/12 00:23:34 INFO Upload complete. (341ms)14002026/11/12 00:23:34 INFO Signed narinfos id=1 count=214012026/11/12 00:23:34 INFO Uploading 2 narinfos1402=== NAME TestClientPushesUseOnePush14032026/11/12 00:23:34 INFO Starting cleanup of old closures method=DELETE path=/api/closures1404 client_pushes_test.go:97: Retrieved narinfo from S3:1405 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientPushesUseOnePush667723684/001/store/65x04qx6nr2incmcz98j9r7hrniy4q58-shared-dep1406 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1407 Compression: zstd1408 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y8214092026/11/12 00:23:34 INFO Garbage collection started1410 NarSize: 1361411 References: 1412 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1413 client_pushes_test.go:97: Retrieved narinfo from S3:1414 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientPushesUseOnePush667723684/001/store/6ggrkqjq4q2781zw2nld19qnsi84cafy-a1415 URL: nar/0p1gkk5snm1cswgikbrx279rilr0p0d3vq7xzp1d59v7v0i2y4kr.nar.zst1416 Compression: zstd1417 NarHash: sha256:0p1gkk5snm1cswgikbrx279rilr0p0d3vq7xzp1d59v7v0i2y4kr1418 NarSize: 2561419 References: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientPushesUseOnePush667723684/001/store/65x04qx6nr2incmcz98j9r7hrniy4q58-shared-dep1420 CA: text:sha256:1n5b7d5cmxyz8badjbn2rxvz1b6szkgxp20bjgvla1q37sjcsk731421 client_pushes_test.go:97: Retrieved narinfo from S3:1422 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientPushesUseOnePush667723684/001/store/jphrzm4yy6ckdk4y5980124r93x2w2i4-b1423 URL: nar/0p1gkk5snm1cswgikbrx279rilr0p0d3vq7xzp1d59v7v0i2y4kr.nar.zst1424 Compression: zstd1425 NarHash: sha256:0p1gkk5snm1cswgikbrx279rilr0p0d3vq7xzp1d59v7v0i2y4kr1426 NarSize: 2561427 References: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientPushesUseOnePush667723684/001/store/65x04qx6nr2incmcz98j9r7hrniy4q58-shared-dep1428 CA: text:sha256:1n5b7d5cmxyz8badjbn2rxvz1b6szkgxp20bjgvla1q37sjcsk7314292026/11/12 00:23:34 WARN Failed to register uploaded object key=cznc405mvv2w7qwpg1nhldb9vn43xlap.narinfo error="server returned 404: 404 page not found\n"14302026/11/12 00:23:34 INFO lead: released remote=192.0.2.1:1234 generation=21431--- PASS: TestLeadElectsOneAndHandsOver (6.84s)1432=== CONT TestClientCADerivations14332026/11/12 00:23:34 INFO Aborted multipart uploads count=0 kept=014342026/11/12 00:23:34 WARN Force mode enabled - objects will be deleted immediately without grace period14352026/11/12 00:23:34 INFO Received complete push request method=POST path=/api/pushes/1/complete14362026/11/12 00:23:34 WARN Failed to register uploaded object key=zvbss64il6x4jljzz6hlr0i7nn6mzsfp.narinfo error="server returned 404: 404 page not found\n"14372026/11/12 00:23:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=014382026/11/12 00:23:34 INFO Vacuumed table table=pending_closures14392026/11/12 00:23:34 INFO Vacuumed table table=pending_objects14402026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads14412026/11/12 00:23:35 INFO Vacuumed table table=closures14422026/11/12 00:23:35 INFO Completed push id=1 objects=6 duration=22.507167ms1443--- PASS: TestClientPushesUseOnePush (5.22s)1444=== CONT TestCacheConfigHandler14452026/11/12 00:23:35 INFO Upload complete. (332ms)1446=== RUN TestCacheConfigHandler/full_config,_no_issuer1447=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1448=== RUN TestCacheConfigHandler/no_cache_url_configured1449=== PAUSE TestCacheConfigHandler/no_cache_url_configured1450=== RUN TestCacheConfigHandler/no_signing_keys1451=== PAUSE TestCacheConfigHandler/no_signing_keys1452=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1453=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1454=== CONT TestCacheStatsHandler14552026/11/12 00:23:35 INFO Vacuumed table table=objects1456=== NAME TestClientSharedPathCommittedMidPush1457 client_integration_test.go:728: Retrieved narinfo from S3:1458 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientSharedPathCommittedMidPush3777147400/001/store/cznc405mvv2w7qwpg1nhldb9vn43xlap-shared-dep1459 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1460 Compression: zstd1461 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821462 NarSize: 1361463 References: 1464 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1465 client_integration_test.go:728: Retrieved narinfo from S3:1466 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientSharedPathCommittedMidPush3777147400/001/store/zvbss64il6x4jljzz6hlr0i7nn6mzsfp-top1467--- PASS: TestSweepSkipsRowsLockedByPush (4.70s)1468=== CONT TestSharedNarBothUploadsComplete1469=== NAME TestClientSharedPathCommittedMidPush1470 URL: nar/19fnxnzmz822is134az3jagqqvzqlpcbsgamv04maalc0bd6wzb7.nar.zst1471 Compression: zstd1472 NarHash: sha256:19fnxnzmz822is134az3jagqqvzqlpcbsgamv04maalc0bd6wzb71473 NarSize: 2641474 References: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientSharedPathCommittedMidPush3777147400/001/store/cznc405mvv2w7qwpg1nhldb9vn43xlap-shared-dep1475 CA: text:sha256:1m7cj4md2bxz3k4adz2dvlwnw49q74lkhccs9b7jknd6qpik5pw81476--- PASS: TestGetExistingObjectsLocksRows (4.18s)1477=== CONT TestCompletedNarNotReofferedAcrossClosures1478--- PASS: TestClientSharedPathCommittedMidPush (5.01s)1479=== CONT TestCompleteMultipartUpload_ErrorButObjectExists14802026/11/12 00:23:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01481=== NAME TestClientWithDependencies1482 client_integration_test.go:661: Built derivation: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientWithDependencies1636690297/001/store/46f0k9dg6135b84nc61rigx9bq4bbc1y-test-script14832026/11/12 00:23:35 INFO Received uploads request method=POST path=/api/pending_closures14842026/11/12 00:23:35 INFO Vacuumed table table=pending_closures14852026/11/12 00:23:35 INFO Vacuumed table table=pending_objects14862026/11/12 00:23:35 INFO Vacuumed table table=multipart_uploads14872026/11/12 00:23:35 INFO Vacuumed table table=closures1488 client_integration_test.go:663: Found 1 dependencies (including self)14892026/11/12 00:23:35 INFO Vacuumed table table=objects14902026/11/12 00:23:35 INFO lead: released remote=192.0.2.1:1234 generation=31491--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (7.84s)1492=== CONT TestService_AuthMiddleware_MTLSProxyHeader14932026/11/12 00:23:35 INFO Aborted multipart uploads count=0 kept=014942026/11/12 00:23:35 WARN Force mode enabled - objects will be deleted immediately without grace period14952026/11/12 00:23:35 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."14962026/11/12 00:23:35 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1497--- PASS: TestFailedS3DeleteKeepsTombstone (4.32s)1498=== CONT TestService_AuthMiddleware_MTLSBoundSubjects14992026/11/12 00:23:35 INFO Received push request method=POST path=/api/pushes15002026/11/12 00:23:35 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15012026/11/12 00:23:35 INFO Uploading 46f0k9dg6135b84nc61rigx9bq4bbc1y-test-script (136B)15022026/11/12 00:23:35 WARN Failed to register uploaded object key=log/x1mgyb3amy9ikms5h5lw9gfdi25i55vk-test-script.drv error="server returned 404: 404 page not found\n"15032026/11/12 00:23:35 WARN Failed to register uploaded object key=46f0k9dg6135b84nc61rigx9bq4bbc1y.ls error="server returned 404: 404 page not found\n"15042026/11/12 00:23:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15052026/11/12 00:23:35 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"15062026/11/12 00:23:35 INFO Signed narinfos id=1 count=115072026/11/12 00:23:35 INFO Uploading 1 narinfos15082026/11/12 00:23:35 INFO Received complete push request method=POST path=/api/pushes/1/complete15092026/11/12 00:23:35 WARN Failed to register uploaded object key=46f0k9dg6135b84nc61rigx9bq4bbc1y.narinfo error="server returned 404: 404 page not found\n"15102026/11/12 00:23:35 INFO Completed push id=1 objects=4 duration=45.022375ms15112026/11/12 00:23:35 INFO Upload complete. (310ms)1512=== NAME TestClientWithDependencies1513 client_integration_test.go:665: Skipping nix copy test - isolated store (/private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientWithDependencies1636690297/001/store) requires matching store prefix1514=== NAME TestClientMultipleUploads1515 client_integration_test.go:406: Created store path 0: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientMultipleUploads1694050125/001/store/i7bnvyqnij4qf0hlcbfjkj3wraq385dy-test-file-0.txt1516--- PASS: TestClientWithDependencies (5.17s)1517=== CONT TestService_AuthMiddleware1518=== NAME TestClientMultipleUploads1519 client_integration_test.go:406: Created store path 1: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientMultipleUploads1694050125/001/store/gi73g07drb8250vm5hip2v45w3nxz0z4-test-file-1.txt1520--- PASS: TestConnectWaitsForAPeerMigration (7.21s)1521=== CONT TestService_ReadScope_PublicByDefault1522=== NAME TestClientMultipleUploads1523 client_integration_test.go:406: Created store path 2: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientMultipleUploads1694050125/001/store/kkl78j9kvhws6y4863z54ckj7ji21acf-test-file-2.txt15242026/11/12 00:23:36 INFO Received uploads request method=POST path=/api/pending_closures1525=== NAME TestClientIntegration1526 client_integration_test.go:292: Created store path: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientIntegration2350307919/002/store/hkccpscqk67rmnw1ngzz7jhj3fa9ay2k-test-file.txt1527--- PASS: TestDeleteClosuresRacingPins (7.35s)1528=== CONT TestIsValidUploadKey1529=== RUN TestIsValidUploadKey/narinfo1530=== PAUSE TestIsValidUploadKey/narinfo1531=== RUN TestIsValidUploadKey/nar_zst1532=== PAUSE TestIsValidUploadKey/nar_zst1533=== RUN TestIsValidUploadKey/nar_xz1534=== PAUSE TestIsValidUploadKey/nar_xz1535=== RUN TestIsValidUploadKey/nar_plain1536=== PAUSE TestIsValidUploadKey/nar_plain1537=== RUN TestIsValidUploadKey/listing1538=== PAUSE TestIsValidUploadKey/listing1539=== RUN TestIsValidUploadKey/build_log1540=== PAUSE TestIsValidUploadKey/build_log1541=== RUN TestIsValidUploadKey/build_log_home-manager_file1542=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1543=== RUN TestIsValidUploadKey/build_log_plus_in_name1544=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1545=== RUN TestIsValidUploadKey/build_log_question_mark1546=== PAUSE TestIsValidUploadKey/build_log_question_mark1547=== RUN TestIsValidUploadKey/build_log_equals1548=== PAUSE TestIsValidUploadKey/build_log_equals1549=== RUN TestIsValidUploadKey/realisation1550=== PAUSE TestIsValidUploadKey/realisation1551=== RUN TestIsValidUploadKey/realisation_plus_in_output1552=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1553=== RUN TestIsValidUploadKey/nix-cache-info1554=== PAUSE TestIsValidUploadKey/nix-cache-info1555=== RUN TestIsValidUploadKey/index.html1556=== PAUSE TestIsValidUploadKey/index.html1557=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1558=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1559=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1560=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1561=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1562=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1563=== RUN TestIsValidUploadKey/traversal1564=== PAUSE TestIsValidUploadKey/traversal1565=== RUN TestIsValidUploadKey/traversal_nar1566=== PAUSE TestIsValidUploadKey/traversal_nar1567=== RUN TestIsValidUploadKey/absolute1568=== PAUSE TestIsValidUploadKey/absolute1569=== RUN TestIsValidUploadKey/empty_key1570=== PAUSE TestIsValidUploadKey/empty_key1571=== RUN TestIsValidUploadKey/unknown_type1572=== PAUSE TestIsValidUploadKey/unknown_type1573=== CONT TestPendingClosureWriteTimeout1574=== RUN TestPendingClosureWriteTimeout/empty1575=== PAUSE TestPendingClosureWriteTimeout/empty1576=== RUN TestPendingClosureWriteTimeout/400_objects1577=== PAUSE TestPendingClosureWriteTimeout/400_objects1578=== RUN TestPendingClosureWriteTimeout/670k_objects1579=== PAUSE TestPendingClosureWriteTimeout/670k_objects1580=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1581=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1582=== CONT TestPush_SignsNarinfosOfItsPendingObjects1583--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (5.00s)1584=== CONT TestService_RequireScope_OIDC15852026/11/12 00:23:36 INFO Received push request method=POST path=/api/pushes15862026/11/12 00:23:36 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)15872026/11/12 00:23:36 INFO Uploading i7bnvyqnij4qf0hlcbfjkj3wraq385dy-test-file-0.txt (160B)15882026/11/12 00:23:36 INFO Uploading kkl78j9kvhws6y4863z54ckj7ji21acf-test-file-2.txt (160B)15892026/11/12 00:23:36 INFO Uploading gi73g07drb8250vm5hip2v45w3nxz0z4-test-file-1.txt (160B)1590=== NAME TestOrphanedObjectsGCStressTest1591 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains15922026/11/12 00:23:36 WARN Failed to register uploaded object key=kkl78j9kvhws6y4863z54ckj7ji21acf.ls error="server returned 404: 404 page not found\n"15932026/11/12 00:23:36 WARN Failed to register uploaded object key=gi73g07drb8250vm5hip2v45w3nxz0z4.ls error="server returned 404: 404 page not found\n"15942026/11/12 00:23:36 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50763/oidc15952026/11/12 00:23:36 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"15962026/11/12 00:23:36 INFO Received push request method=POST path=/api/pushes15972026/11/12 00:23:36 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"15982026/11/12 00:23:36 WARN Failed to register uploaded object key=i7bnvyqnij4qf0hlcbfjkj3wraq385dy.ls error="server returned 404: 404 page not found\n"15992026/11/12 00:23:36 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16002026/11/12 00:23:36 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"16012026/11/12 00:23:36 INFO Signed narinfos id=1 count=316022026/11/12 00:23:36 INFO Uploading 3 narinfos1603 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion16042026/11/12 00:23:36 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16052026/11/12 00:23:36 INFO Uploading hkccpscqk67rmnw1ngzz7jhj3fa9ay2k-test-file.txt (152B)16062026/11/12 00:23:36 WARN Failed to register uploaded object key=i7bnvyqnij4qf0hlcbfjkj3wraq385dy.narinfo error="server returned 404: 404 page not found\n"16072026/11/12 00:23:36 WARN Failed to register uploaded object key=gi73g07drb8250vm5hip2v45w3nxz0z4.narinfo error="server returned 404: 404 page not found\n"16082026/11/12 00:23:36 INFO Received complete push request method=POST path=/api/pushes/1/complete16092026/11/12 00:23:36 WARN Failed to register uploaded object key=kkl78j9kvhws6y4863z54ckj7ji21acf.narinfo error="server returned 404: 404 page not found\n"16102026/11/12 00:23:36 WARN Failed to register uploaded object key=hkccpscqk67rmnw1ngzz7jhj3fa9ay2k.ls error="server returned 404: 404 page not found\n"16112026/11/12 00:23:36 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16122026/11/12 00:23:36 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"16132026/11/12 00:23:36 INFO Signed narinfos id=1 count=116142026/11/12 00:23:36 INFO Uploading 1 narinfos16152026/11/12 00:23:36 INFO Completed push id=1 objects=9 duration=36.774208ms16162026/11/12 00:23:36 INFO Upload complete. (564ms)1617=== NAME TestClientMultipleUploads1618 client_integration_test.go:417: Uploaded 3 paths in 687.6455ms16192026/11/12 00:23:36 INFO Aborted multipart uploads count=0 kept=016202026/11/12 00:23:36 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=016212026/11/12 00:23:36 INFO Received complete push request method=POST path=/api/pushes/1/complete16222026/11/12 00:23:36 WARN Failed to register uploaded object key=hkccpscqk67rmnw1ngzz7jhj3fa9ay2k.narinfo error="server returned 404: 404 page not found\n"16232026/11/12 00:23:36 INFO Received create pin request method=POST path=/api/pins/myapp16242026/11/12 00:23:36 WARN Force mode enabled - objects will be deleted immediately without grace period16252026/11/12 00:23:37 INFO Completed push id=1 objects=3 duration=62.868375ms16262026/11/12 00:23:37 INFO Upload complete. (581ms)16272026/11/12 00:23:37 INFO Created/updated pin name=myapp store_path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestPinProtectsFromGC553048633/001/store/06ndhaz9wdvpm0qap2di5qjrbjxv3asn-pinned-file.txt narinfo_key=06ndhaz9wdvpm0qap2di5qjrbjxv3asn.narinfo16282026/11/12 00:23:37 INFO Starting cleanup of old closures method=DELETE path=/api/closures16292026/11/12 00:23:37 INFO Garbage collection started16302026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=016312026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period16322026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01633--- PASS: TestClientMultipleUploads (5.79s)1634=== CONT TestService_AuthMiddleware_OIDC16352026/11/12 00:23:37 INFO Vacuumed table table=pending_closures16362026/11/12 00:23:37 INFO All 1 paths already cached16372026/11/12 00:23:37 INFO Vacuumed table table=pending_objects1638=== NAME TestClientIntegration1639 client_integration_test.go:318: Retrieved narinfo from S3:1640 StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientIntegration2350307919/002/store/hkccpscqk67rmnw1ngzz7jhj3fa9ay2k-test-file.txt1641 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1642 Compression: zstd1643 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11644 NarSize: 1521645 References: 1646 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk116472026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads16482026/11/12 00:23:37 INFO Vacuumed table table=closures1649 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1650 client_integration_test.go:319: Decompressed .ls content (64 bytes):16512026/11/12 00:23:37 INFO Vacuumed table table=objects1652 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}16532026/11/12 00:23:37 INFO Received uploads request method=POST path=/api/pending_closures1654--- PASS: TestDeduplicatedObjectsRecordedAsPending (4.36s)1655=== CONT TestPush_RejectsBadRequests16562026/11/12 00:23:37 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50771/oidc16572026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016582026/11/12 00:23:37 INFO Vacuumed table table=pending_closures16592026/11/12 00:23:37 INFO Vacuumed table table=pending_objects16602026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads16612026/11/12 00:23:37 INFO Vacuumed table table=closures16622026/11/12 00:23:37 INFO Received uploads request method=POST path=/api/pending_closures16632026/11/12 00:23:37 INFO Vacuumed table table=objects16642026/11/12 00:23:37 INFO Received push request method=POST path=/api/pushes1665--- PASS: TestGCSweepDeliversEachKeyOnce (5.44s)16662026/11/12 00:23:37 INFO Object in database but missing from S3 key=hkccpscqk67rmnw1ngzz7jhj3fa9ay2k.ls16672026/11/12 00:23:37 WARN Found objects in DB but missing from S3, will re-upload count=11668=== CONT TestService_ReadAuthMiddleware16692026/11/12 00:23:37 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)16702026/11/12 00:23:37 INFO Received complete push request method=POST path=/api/pushes/2/complete16712026/11/12 00:23:37 WARN Failed to register uploaded object key=hkccpscqk67rmnw1ngzz7jhj3fa9ay2k.ls error="server returned 404: 404 page not found\n"16722026/11/12 00:23:37 INFO Completed push id=2 objects=3 duration=1.2995ms16732026/11/12 00:23:37 INFO Upload complete. (158ms)1674=== NAME TestClientIntegration1675 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1676 client_integration_test.go:336: Decompressed .ls content (49 bytes):1677 {"version":1,"root":{"type":"regular","size":39}}16782026/11/12 00:23:37 INFO Aborted multipart uploads count=0 kept=016792026/11/12 00:23:37 WARN Force mode enabled - objects will be deleted immediately without grace period16802026/11/12 00:23:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016812026/11/12 00:23:37 INFO Vacuumed table table=pending_closures16822026/11/12 00:23:37 INFO Vacuumed table table=pending_objects16832026/11/12 00:23:37 INFO Vacuumed table table=multipart_uploads16842026/11/12 00:23:37 INFO Vacuumed table table=closures16852026/11/12 00:23:38 INFO Vacuumed table table=objects16862026/11/12 00:23:38 INFO Received uploads request method=POST path=/api/pending_closures1687--- PASS: TestGCSweepSkipsPendingObjects (5.26s)1688=== CONT TestProxyWriteTimeout/narinfo1689=== CONT TestProxyWriteTimeout/unknown_size1690=== CONT TestProxyWriteTimeout/10_GiB_nar1691=== CONT TestProxyWriteTimeout/1_GiB_nar1692--- PASS: TestProxyWriteTimeout (0.00s)1693 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1694 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1695 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1696 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1697=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1698--- PASS: TestTombstonedObjectOfferedWithoutWaiting (5.32s)1699=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed17002026/11/12 00:23:38 INFO Received push request method=POST path=/api/pushes17012026/11/12 00:23:38 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17022026/11/12 00:23:38 INFO Received complete push request method=POST path=/api/pushes/3/complete17032026/11/12 00:23:38 WARN Failed to register uploaded object key=hkccpscqk67rmnw1ngzz7jhj3fa9ay2k.ls error="server returned 404: 404 page not found\n"17042026/11/12 00:23:38 INFO Completed push id=3 objects=3 duration=1.837208ms17052026/11/12 00:23:38 INFO Upload complete. (170ms)1706=== NAME TestClientIntegration1707 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1708 client_integration_test.go:353: Decompressed .ls content (49 bytes):1709 {"version":1,"root":{"type":"regular","size":39}}1710 client_integration_test.go:364: Testing garbage collection...17112026/11/12 00:23:38 INFO Starting cleanup of old closures method=DELETE path=/api/closures17122026/11/12 00:23:38 INFO Garbage collection started17132026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=017142026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period17152026/11/12 00:23:38 INFO Aborted multipart uploads count=0 kept=017162026/11/12 00:23:38 WARN Force mode enabled - objects will be deleted immediately without grace period17172026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017182026/11/12 00:23:38 INFO Vacuumed table table=pending_closures17192026/11/12 00:23:38 INFO Vacuumed table table=pending_objects17202026/11/12 00:23:38 INFO Vacuumed table table=multipart_uploads17212026/11/12 00:23:38 INFO Vacuumed table table=closures17222026/11/12 00:23:38 INFO Vacuumed table table=objects17232026/11/12 00:23:38 INFO Received uploads request method=POST path=/api/pending_closures1724--- PASS: TestCommitRacingPendingCleanupKeepsObjects (5.24s)1725=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed17262026/11/12 00:23:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=017272026/11/12 00:23:39 INFO Vacuumed table table=pending_closures1728=== NAME TestPinProtectsFromGC17292026/11/12 00:23:39 INFO Vacuumed table table=pending_objects1730 client_integration_test.go:888: Pin successfully protected closure from garbage collection17312026/11/12 00:23:39 INFO Vacuumed table table=multipart_uploads17322026/11/12 00:23:39 INFO Vacuumed table table=closures17332026/11/12 00:23:39 INFO Vacuumed table table=objects1734--- PASS: TestPinProtectsFromGC (9.37s)1735=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1736--- PASS: TestPushWaitsForSweepHoldingRow (5.50s)1737=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed17382026/11/12 00:23:39 INFO Aborted multipart uploads count=0 kept=017392026/11/12 00:23:39 WARN Force mode enabled - objects will be deleted immediately without grace period17402026/11/12 00:23:39 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017412026/11/12 00:23:39 INFO Vacuumed table table=pending_closures17422026/11/12 00:23:39 INFO Vacuumed table table=pending_objects17432026/11/12 00:23:39 INFO Vacuumed table table=multipart_uploads17442026/11/12 00:23:39 INFO Vacuumed table table=closures17452026/11/12 00:23:39 INFO Vacuumed table table=objects1746--- PASS: TestSweepRechecksPendingAfterLocking (5.30s)1747=== CONT TestIsValidCachePath/narinfo1748=== CONT TestParseSingleRange/none1749=== CONT TestIsValidCachePath/short_hash1750=== CONT TestIsValidCachePath/wrong_extension1751=== CONT TestIsValidCachePath/leading_slash1752=== CONT TestIsValidCachePath/empty1753=== CONT TestIsValidCachePath/random_path1754=== CONT TestIsValidCachePath/invalid_char_u1755=== CONT TestIsValidCachePath/invalid_char_e1756=== CONT TestIsValidCachePath/traversal_in_middle1757=== CONT TestIsValidCachePath/traversal_parent1758=== CONT TestIsValidCachePath/index.html1759=== CONT TestIsValidCachePath/nix-cache-info1760=== CONT TestIsValidCachePath/realisation1761=== CONT TestIsValidCachePath/log1762=== CONT TestIsValidCachePath/ls1763=== CONT TestIsValidCachePath/nar_uncompressed1764=== CONT TestIsValidCachePath/nar_bz21765=== CONT TestIsValidCachePath/nar_xz1766=== CONT TestIsValidCachePath/nar_zst1767=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1768--- PASS: TestIsValidCachePath (0.00s)1769 --- PASS: TestIsValidCachePath/narinfo (0.00s)1770 --- PASS: TestIsValidCachePath/short_hash (0.00s)1771 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1772 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1773 --- PASS: TestIsValidCachePath/empty (0.00s)1774 --- PASS: TestIsValidCachePath/random_path (0.00s)1775 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1776 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1777 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1778 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1779 --- PASS: TestIsValidCachePath/index.html (0.00s)1780 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1781 --- PASS: TestIsValidCachePath/realisation (0.00s)1782 --- PASS: TestIsValidCachePath/log (0.00s)1783 --- PASS: TestIsValidCachePath/ls (0.00s)1784 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1785 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1786 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1787 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1788 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1789=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale17902026/11/12 00:23:39 INFO Received uploads request method=POST path=/api/pending_closures17912026/11/12 00:23:39 INFO Aborted multipart uploads count=0 kept=017922026/11/12 00:23:39 INFO Aborted multipart uploads count=0 kept=017932026/11/12 00:23:39 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017942026/11/12 00:23:39 INFO Vacuumed table table=pending_closures17952026/11/12 00:23:40 WARN Force mode enabled - objects will be deleted immediately without grace period17962026/11/12 00:23:40 INFO Vacuumed table table=pending_objects17972026/11/12 00:23:40 INFO Vacuumed table table=multipart_uploads17982026/11/12 00:23:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017992026/11/12 00:23:40 INFO Vacuumed table table=pending_closures18002026/11/12 00:23:40 INFO Vacuumed table table=pending_objects18012026/11/12 00:23:40 INFO Vacuumed table table=multipart_uploads18022026/11/12 00:23:40 INFO Vacuumed table table=closures18032026/11/12 00:23:40 INFO Vacuumed table table=objects18042026/11/12 00:23:40 INFO Vacuumed table table=closures1805--- PASS: TestGCMetrics (5.55s)1806=== CONT TestParseSingleRange/start_far_past_EOF1807=== CONT TestParseSingleRange/start_past_EOF1808=== CONT TestParseSingleRange/single_byte1809=== CONT TestParseSingleRange/suffix_exceeds_size1810=== CONT TestParseSingleRange/suffix1811=== CONT TestParseSingleRange/end_clamped_to_size1812=== CONT TestParseSingleRange/open-ended1813=== CONT TestParseSingleRange/closed1814=== CONT TestParseSingleRange/malformed_end_before_start1815=== CONT TestParseSingleRange/malformed_both_empty1816=== CONT TestParseSingleRange/malformed_no_dash1817=== CONT TestParseSingleRange/multi-range_ignored1818=== CONT TestParseSingleRange/unknown_unit1819--- PASS: TestParseSingleRange (0.00s)1820 --- PASS: TestParseSingleRange/none (0.00s)1821 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1822 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1823 --- PASS: TestParseSingleRange/single_byte (0.00s)1824 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1825 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1826 --- PASS: TestParseSingleRange/open-ended (0.00s)1827 --- PASS: TestParseSingleRange/suffix (0.00s)1828 --- PASS: TestParseSingleRange/closed (0.00s)1829 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1830 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1831 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1832 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1833 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1834=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh18352026/11/12 00:23:40 INFO Vacuumed table table=objects18362026/11/12 00:23:40 INFO Aborted multipart uploads count=0 kept=018372026/11/12 00:23:40 WARN Force mode enabled - objects will be deleted immediately without grace period18382026/11/12 00:23:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=018392026/11/12 00:23:40 INFO Vacuumed table table=pending_closures18402026/11/12 00:23:40 INFO Vacuumed table table=pending_objects18412026/11/12 00:23:40 INFO Vacuumed table table=multipart_uploads18422026/11/12 00:23:40 INFO Vacuumed table table=closures18432026/11/12 00:23:40 INFO Vacuumed table table=objects1844--- PASS: TestPushDedupSurvivesConcurrentGC (5.98s)1845=== CONT TestServerTLSConfig/no_client_CA1846=== CONT TestServerTLSConfig/not_a_PEM_file1847=== CONT TestServerTLSConfig/missing_CA_file1848--- PASS: TestServerTLSConfig (0.00s)1849 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1850 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.01s)1851 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1852=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info18532026/11/12 00:23:40 INFO Received uploads request method=POST path=/1854=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key18552026/11/12 00:23:40 INFO Received complete multipart upload request method=POST path=/1856=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key18572026/11/12 00:23:40 INFO Received request for more parts method=POST path=/1858=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18592026/11/12 00:23:40 INFO Received uploads request method=POST path=/1860--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1861 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1862 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1863 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1864 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1865=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts18662026/11/12 00:23:40 INFO Received request for more parts method=POST path=/1867=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart18682026/11/12 00:23:40 INFO Received complete multipart upload request method=POST path=/1869=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure18702026/11/12 00:23:40 INFO Received uploads request method=POST path=/18712026/11/12 00:23:40 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=01872=== NAME TestClientIntegration1873 client_integration_test.go:371: Objects in database after GC:1874 client_integration_test.go:371: Successfully deleted all objects with GC --force1875--- PASS: TestCacheStatsHandler (5.38s)1876=== CONT TestResolveDBConnectionString/flag_wins1877=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1878=== CONT TestResolveDBConnectionString/nothing_configured1879=== CONT TestResolveDBConnectionString/missing_file_is_an_error1880=== CONT TestResolveDBConnectionString/file_when_flag_empty1881=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1882--- PASS: TestResolveDBConnectionString (0.02s)1883 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1884 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1885 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1886 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1887 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1888--- PASS: TestClientIntegration (9.11s)1889=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1890--- PASS: TestUploadHandlersRejectOversizedBody (0.05s)1891 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.03s)1892 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.03s)1893 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (0.61s)1894=== CONT TestClientErrorHandling/InvalidStorePath18952026/11/12 00:23:41 INFO Received uploads request method=POST path=/api/pending_closures18962026/11/12 00:23:41 INFO Received uploads request method=POST path=/api/pending_closures1897=== NAME TestClientCADerivations1898 client_ca_test.go:136: Built CA derivation: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientCADerivations2432387618/001/store/l3dbg252149910msms9j49k928s2kxwv-ca-test1899 client_ca_test.go:139: Found 1 dependencies (including self)19002026/11/12 00:23:41 INFO Received uploads request method=POST path=/api/pending_closures19012026/11/12 00:23:41 INFO Received push request method=POST path=/api/pushes19022026/11/12 00:23:42 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19032026/11/12 00:23:42 INFO Uploading l3dbg252149910msms9j49k928s2kxwv-ca-test (144B)19042026/11/12 00:23:42 WARN Failed to register uploaded object key=l3dbg252149910msms9j49k928s2kxwv.ls error="server returned 404: 404 page not found\n"19052026/11/12 00:23:42 WARN Failed to register uploaded object key=log/mvjnv74n3a6w688d2s459b9qmz38czm3-ca-test.drv error="server returned 404: 404 page not found\n"19062026/11/12 00:23:42 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19072026/11/12 00:23:42 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"19082026/11/12 00:23:42 INFO Signed narinfos id=1 count=119092026/11/12 00:23:42 INFO Uploading 1 narinfos19102026/11/12 00:23:42 INFO Received uploads request method=POST path=/api/pending_closures19112026/11/12 00:23:42 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19122026/11/12 00:23:42 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MDRhNTNkYTktY2ViNi00MWQ0LThkODctYTUxYzFiZmU4YjI3LmUxZjdmYmExLTFmOWEtNDcyMi1hY2JmLTY2ODZmYjk4N2Y2NngxNzk0NDQzMDIxNzYwMjQ5MDAw19132026/11/12 00:23:42 INFO Received complete push request method=POST path=/api/pushes/1/complete19142026/11/12 00:23:42 WARN Failed to register uploaded object key=l3dbg252149910msms9j49k928s2kxwv.narinfo error="server returned 404: 404 page not found\n"19152026/11/12 00:23:42 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MDRhNTNkYTktY2ViNi00MWQ0LThkODctYTUxYzFiZmU4YjI3LmUxZjdmYmExLTFmOWEtNDcyMi1hY2JmLTY2ODZmYjk4N2Y2NngxNzk0NDQzMDIxNzYwMjQ5MDAw parts=11916--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (7.14s)1917=== CONT TestClientErrorHandling/InvalidAuthToken19182026/11/12 00:23:42 INFO Completed push id=1 objects=4 duration=48.221ms19192026/11/12 00:23:42 INFO Upload complete. (560ms)1920=== NAME TestClientCADerivations1921 client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientCADerivations2432387618/001/store/l3dbg252149910msms9j49k928s2kxwv-ca-test1922 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1923 Compression: zstd1924 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1925 NarSize: 1441926 References: 1927 Deriver: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientCADerivations2432387618/001/store/mvjnv74n3a6w688d2s459b9qmz38czm3-ca-test.drv1928 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1929 client_ca_test.go:185: Checking for realisation files in S3...1930 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1931 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1932 client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket81?endpoint=http://localhost:50528®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/5/scratch/75b857c59f8a960d/build/TestClientCADerivations2432387618/001/store'1933 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11934--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (7.11s)1935=== CONT TestClientErrorHandling/ServerNotAvailable1936--- PASS: TestClientCADerivations (7.68s)1937=== CONT TestGCEndsOnShutdown/before_the_run19382026/11/12 00:23:42 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"19392026/11/12 00:23:42 WARN mTLS auth: bound subjects configured but subject DN unavailable19402026/11/12 00:23:42 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1941--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (7.35s)1942=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark19432026/11/12 00:23:43 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19442026/11/12 00:23:43 WARN Request failed, retrying attempt=2 backoff=198.483829ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19452026/11/12 00:23:43 WARN Request failed, retrying attempt=3 backoff=363.842298ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19462026/11/12 00:23:43 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1947--- PASS: TestService_AuthMiddleware (7.59s)1948=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete19492026/11/12 00:23:43 WARN Request failed, retrying attempt=4 backoff=871.029323ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19502026/11/12 00:23:43 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1951--- PASS: TestService_ReadScope_PublicByDefault (7.83s)1952=== CONT TestCacheConfigHandler/full_config,_no_issuer1953=== CONT TestCacheConfigHandler/no_signing_keys1954=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1955=== CONT TestCacheConfigHandler/no_cache_url_configured1956--- PASS: TestCacheConfigHandler (0.01s)1957 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1958 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1959 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1960 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1961=== CONT TestIsValidUploadKey/narinfo1962=== CONT TestPendingClosureWriteTimeout/empty1963=== CONT TestIsValidUploadKey/unknown_type1964=== CONT TestIsValidUploadKey/empty_key1965=== CONT TestIsValidUploadKey/absolute1966=== CONT TestIsValidUploadKey/traversal_nar1967=== CONT TestIsValidUploadKey/traversal1968=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1969=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1970=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1971=== CONT TestIsValidUploadKey/index.html1972=== CONT TestIsValidUploadKey/nix-cache-info1973=== CONT TestIsValidUploadKey/realisation_plus_in_output1974=== CONT TestIsValidUploadKey/realisation1975=== CONT TestIsValidUploadKey/build_log_equals1976=== CONT TestIsValidUploadKey/build_log_question_mark1977=== CONT TestIsValidUploadKey/build_log_plus_in_name1978=== CONT TestIsValidUploadKey/build_log_home-manager_file1979=== CONT TestIsValidUploadKey/build_log1980=== CONT TestIsValidUploadKey/listing1981=== CONT TestIsValidUploadKey/nar_plain1982=== CONT TestIsValidUploadKey/nar_xz1983=== CONT TestIsValidUploadKey/nar_zst1984--- PASS: TestIsValidUploadKey (0.00s)1985 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1986 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1987 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1988 --- PASS: TestIsValidUploadKey/absolute (0.00s)1989 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1990 --- PASS: TestIsValidUploadKey/traversal (0.00s)1991 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1992 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1993 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1994 --- PASS: TestIsValidUploadKey/index.html (0.00s)1995 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1996 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1997 --- PASS: TestIsValidUploadKey/realisation (0.00s)1998 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1999 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)2000 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)
loading 660 hidden lines…
26612026/11/12 00:24:02 INFO Completed push id=2 objects=25 duration=6.8375ms26622026/11/12 00:24:02 INFO Received uploads request method=POST path=/26632026/11/12 00:24:02 INFO Received push request method=POST path=/26642026/11/12 00:24:02 INFO Received complete upload request method=POST path=/26652026/11/12 00:24:02 INFO Received push request method=POST path=/26662026/11/12 00:24:02 INFO Completed upload id=6 objects=7 duration=8.452ms26672026/11/12 00:24:02 INFO Received uploads request method=POST path=/26682026/11/12 00:24:02 INFO Received complete upload request method=POST path=/26692026/11/12 00:24:02 INFO Completed upload id=18 objects=4 duration=2.899125ms26702026/11/12 00:24:02 INFO Received complete upload request method=POST path=/26712026/11/12 00:24:02 INFO Received complete push request method=POST path=/26722026/11/12 00:24:02 INFO Completed upload id=9 objects=16 duration=4.965ms26732026/11/12 00:24:02 INFO Completed push id=8 objects=2 duration=2.145834ms26742026/11/12 00:24:02 INFO Received push request method=POST path=/26752026/11/12 00:24:02 INFO Received push request method=POST path=/26762026/11/12 00:24:02 INFO Aborted multipart uploads count=0 kept=026772026/11/12 00:24:02 INFO Received uploads request method=POST path=/26782026/11/12 00:24:02 WARN Force mode enabled - objects will be deleted immediately without grace period26792026/11/12 00:24:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026802026/11/12 00:24:02 INFO Received push request method=POST path=/26812026/11/12 00:24:02 INFO Aborted multipart uploads count=0 kept=026822026/11/12 00:24:02 INFO Vacuumed table table=pending_closures26832026/11/12 00:24:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026842026/11/12 00:24:02 INFO Vacuumed table table=pending_objects26852026/11/12 00:24:02 INFO Vacuumed table table=pending_closures26862026/11/12 00:24:02 INFO Vacuumed table table=multipart_uploads26872026/11/12 00:24:02 INFO Vacuumed table table=closures26882026/11/12 00:24:02 INFO Received complete upload request method=POST path=/26892026/11/12 00:24:02 INFO Vacuumed table table=objects26902026/11/12 00:24:02 INFO Vacuumed table table=pending_objects26912026/11/12 00:24:02 INFO Completed upload id=4 objects=14 duration=13.61975ms26922026/11/12 00:24:02 INFO Vacuumed table table=multipart_uploads26932026/11/12 00:24:02 INFO Vacuumed table table=closures26942026/11/12 00:24:02 INFO Aborted multipart uploads count=0 kept=026952026/11/12 00:24:02 INFO Received push request method=POST path=/26962026/11/12 00:24:02 WARN Force mode enabled - objects will be deleted immediately without grace period26972026/11/12 00:24:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026982026/11/12 00:24:02 INFO Vacuumed table table=pending_closures26992026/11/12 00:24:02 INFO Vacuumed table table=pending_objects27002026/11/12 00:24:02 INFO Vacuumed table table=objects27012026/11/12 00:24:02 INFO Vacuumed table table=multipart_uploads27022026/11/12 00:24:02 INFO Received complete push request method=POST path=/27032026/11/12 00:24:02 INFO Vacuumed table table=closures27042026/11/12 00:24:02 INFO Aborted multipart uploads count=0 kept=027052026/11/12 00:24:02 INFO Vacuumed table table=objects27062026/11/12 00:24:02 INFO Received complete push request method=POST path=/27072026/11/12 00:24:02 WARN Force mode enabled - objects will be deleted immediately without grace period27082026/11/12 00:24:02 INFO Completed push id=10 objects=3 duration=4.856208ms27092026/11/12 00:24:02 INFO Completed push id=1 objects=8 duration=28.1335ms27102026/11/12 00:24:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027112026/11/12 00:24:02 INFO Aborted multipart uploads count=0 kept=027122026/11/12 00:24:02 INFO Vacuumed table table=pending_closures27132026/11/12 00:24:02 INFO Received uploads request method=POST path=/27142026/11/12 00:24:02 INFO Received push request method=POST path=/27152026/11/12 00:24:02 WARN Force mode enabled - objects will be deleted immediately without grace period27162026/11/12 00:24:02 INFO Received uploads request method=POST path=/27172026/11/12 00:24:02 INFO Received uploads request method=POST path=/27182026/11/12 00:24:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027192026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27202026/11/12 00:24:03 INFO Vacuumed table table=pending_closures27212026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads27222026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27232026/11/12 00:24:03 INFO Vacuumed table table=closures27242026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads27252026/11/12 00:24:03 INFO Vacuumed table table=objects27262026/11/12 00:24:03 INFO Vacuumed table table=closures27272026/11/12 00:24:03 INFO Received push request method=POST path=/27282026/11/12 00:24:03 INFO Vacuumed table table=objects27292026/11/12 00:24:03 INFO Received push request method=POST path=/27302026/11/12 00:24:03 INFO Received push request method=POST path=/27312026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027322026/11/12 00:24:03 WARN Force mode enabled - objects will be deleted immediately without grace period27332026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027342026/11/12 00:24:03 INFO Vacuumed table table=pending_closures27352026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27362026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads27372026/11/12 00:24:03 INFO Received push request method=POST path=/27382026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027392026/11/12 00:24:03 INFO Vacuumed table table=closures27402026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027412026/11/12 00:24:03 INFO Vacuumed table table=objects27422026/11/12 00:24:03 INFO Vacuumed table table=pending_closures27432026/11/12 00:24:03 INFO Received uploads request method=POST path=/27442026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27452026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads27462026/11/12 00:24:03 INFO Vacuumed table table=closures27472026/11/12 00:24:03 INFO Vacuumed table table=objects27482026/11/12 00:24:03 INFO Received push request method=POST path=/27492026/11/12 00:24:03 INFO Received uploads request method=POST path=/27502026/11/12 00:24:03 INFO Received complete push request method=POST path=/27512026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027522026/11/12 00:24:03 INFO Completed push id=13 objects=12 duration=5.945958ms27532026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027542026/11/12 00:24:03 INFO Vacuumed table table=pending_closures27552026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27562026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads27572026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027582026/11/12 00:24:03 INFO Vacuumed table table=closures27592026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027602026/11/12 00:24:03 INFO Received complete push request method=POST path=/27612026/11/12 00:24:03 INFO Vacuumed table table=objects27622026/11/12 00:24:03 WARN Force mode enabled - objects will be deleted immediately without grace period27632026/11/12 00:24:03 INFO Completed push id=4 objects=4 duration=2.945375ms27642026/11/12 00:24:03 INFO Received uploads request method=POST path=/27652026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=027662026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=027672026/11/12 00:24:03 INFO Vacuumed table table=pending_closures27682026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27692026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads27702026/11/12 00:24:03 INFO Vacuumed table table=closures27712026/11/12 00:24:03 INFO Vacuumed table table=objects27722026/11/12 00:24:03 INFO Received complete upload request method=POST path=/27732026/11/12 00:24:03 INFO Vacuumed table table=pending_closures27742026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27752026/11/12 00:24:03 INFO Completed upload id=1 objects=15 duration=39.000542ms27762026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads27772026/11/12 00:24:03 INFO Received push request method=POST path=/27782026/11/12 00:24:03 INFO Vacuumed table table=closures27792026/11/12 00:24:03 INFO Received complete push request method=POST path=/27802026/11/12 00:24:03 INFO Received push request method=POST path=/27812026/11/12 00:24:03 INFO Completed push id=5 objects=8 duration=6.255959ms27822026/11/12 00:24:03 INFO Received uploads request method=POST path=/27832026/11/12 00:24:03 INFO Vacuumed table table=objects27842026/11/12 00:24:03 INFO Received complete push request method=POST path=/27852026/11/12 00:24:03 INFO Completed push id=6 objects=6 duration=2.36275ms27862026/11/12 00:24:03 INFO Received push request method=POST path=/27872026/11/12 00:24:03 INFO Received uploads request method=POST path=/27882026/11/12 00:24:03 INFO Received push request method=POST path=/27892026/11/12 00:24:03 INFO Received uploads request method=POST path=/27902026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027912026/11/12 00:24:03 INFO Received complete upload request method=POST path=/27922026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027932026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027942026/11/12 00:24:03 INFO Vacuumed table table=pending_closures27952026/11/12 00:24:03 INFO Completed upload id=2 objects=7 duration=29.663916ms27962026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=027972026/11/12 00:24:03 INFO Vacuumed table table=pending_objects27982026/11/12 00:24:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27992026/11/12 00:24:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28002026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads28012026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028022026/11/12 00:24:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28032026/11/12 00:24:03 INFO Vacuumed table table=closures28042026/11/12 00:24:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28052026/11/12 00:24:03 INFO Vacuumed table table=objects28062026/11/12 00:24:03 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28072026/11/12 00:24:03 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28082026/11/12 00:24:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28092026/11/12 00:24:03 INFO Vacuumed table table=pending_closures28102026/11/12 00:24:03 INFO Vacuumed table table=pending_objects28112026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads28122026/11/12 00:24:03 INFO Vacuumed table table=closures28132026/11/12 00:24:03 INFO Vacuumed table table=objects28142026/11/12 00:24:03 INFO Received complete upload request method=POST path=/28152026/11/12 00:24:03 INFO Completed upload id=17 objects=6 duration=2.264666ms28162026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=028172026/11/12 00:24:03 INFO Aborted multipart uploads count=0 kept=028182026/11/12 00:24:03 WARN Force mode enabled - objects will be deleted immediately without grace period28192026/11/12 00:24:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028202026/11/12 00:24:03 INFO Vacuumed table table=pending_closures28212026/11/12 00:24:03 INFO Vacuumed table table=pending_objects28222026/11/12 00:24:03 INFO Vacuumed table table=multipart_uploads28232026/11/12 00:24:03 INFO Vacuumed table table=closures28242026/11/12 00:24:03 INFO Vacuumed table table=objects28252026/11/12 00:24:03 INFO Received uploads request method=POST path=/28262026/11/12 00:24:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28272026/11/12 00:24:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28282026/11/12 00:24:03 INFO Received complete push request method=POST path=/28292026/11/12 00:24:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28302026/11/12 00:24:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28312026/11/12 00:24:03 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28322026/11/12 00:24:03 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28332026/11/12 00:24:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28342026/11/12 00:24:03 INFO Received push request method=POST path=/28352026/11/12 00:24:03 INFO Completed push id=1 objects=2 duration=8.533334ms28362026/11/12 00:24:03 INFO Received push request method=POST path=/28372026/11/12 00:24:03 INFO Received complete push request method=POST path=/28382026/11/12 00:24:03 INFO Received uploads request method=POST path=/28392026/11/12 00:24:03 INFO Received complete push request method=POST path=/28402026/11/12 00:24:03 INFO Completed push id=1 objects=5 duration=13.091166ms28412026/11/12 00:24:03 INFO Completed push id=19 objects=6 duration=3.844708ms28422026/11/12 00:24:04 INFO Received complete upload request method=POST path=/28432026/11/12 00:24:04 INFO Received complete push request method=POST path=/28442026/11/12 00:24:04 INFO Received complete push request method=POST path=/28452026/11/12 00:24:04 INFO Completed upload id=3 objects=24 duration=11.58875ms28462026/11/12 00:24:04 INFO Completed push id=7 objects=27 duration=2.058208ms28472026/11/12 00:24:04 INFO Completed push id=1 objects=11 duration=9.086083ms28482026/11/12 00:24:04 INFO Received push request method=POST path=/28492026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=028502026/11/12 00:24:04 INFO Received push request method=POST path=/28512026/11/12 00:24:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028522026/11/12 00:24:04 INFO Vacuumed table table=pending_closures28532026/11/12 00:24:04 INFO Vacuumed table table=pending_objects28542026/11/12 00:24:04 INFO Vacuumed table table=multipart_uploads28552026/11/12 00:24:04 INFO Vacuumed table table=closures28562026/11/12 00:24:04 INFO Received uploads request method=POST path=/28572026/11/12 00:24:04 INFO Vacuumed table table=objects28582026/11/12 00:24:04 INFO Received push request method=POST path=/28592026/11/12 00:24:04 INFO Received complete upload request method=POST path=/28602026/11/12 00:24:04 INFO Completed upload id=8 objects=3 duration=1.476333ms28612026/11/12 00:24:04 INFO Received push request method=POST path=/28622026/11/12 00:24:04 INFO Received uploads request method=POST path=/28632026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=028642026/11/12 00:24:04 INFO Received uploads request method=POST path=/28652026/11/12 00:24:04 INFO Received push request method=POST path=/28662026/11/12 00:24:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028672026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=028682026/11/12 00:24:04 INFO Received complete upload request method=POST path=/28692026/11/12 00:24:04 INFO Vacuumed table table=pending_closures28702026/11/12 00:24:04 INFO Received uploads request method=POST path=/28712026/11/12 00:24:04 INFO Vacuumed table table=pending_objects28722026/11/12 00:24:04 INFO Received complete upload request method=POST path=/28732026/11/12 00:24:04 INFO Completed upload id=10 objects=23 duration=8.555875ms28742026/11/12 00:24:04 INFO Vacuumed table table=multipart_uploads28752026/11/12 00:24:04 WARN Force mode enabled - objects will be deleted immediately without grace period28762026/11/12 00:24:04 INFO Completed upload id=16 objects=3 duration=8.968ms28772026/11/12 00:24:04 INFO Vacuumed table table=closures28782026/11/12 00:24:04 INFO Vacuumed table table=objects28792026/11/12 00:24:04 INFO Received complete upload request method=POST path=/28802026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=028812026/11/12 00:24:04 INFO Completed upload id=3 objects=9 duration=881.084µs28822026/11/12 00:24:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28832026/11/12 00:24:04 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28842026/11/12 00:24:04 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28852026/11/12 00:24:04 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28862026/11/12 00:24:04 INFO Received uploads request method=POST path=/28872026/11/12 00:24:04 INFO Received push request method=POST path=/28882026/11/12 00:24:04 INFO Received push request method=POST path=/28892026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=028902026/11/12 00:24:04 WARN Force mode enabled - objects will be deleted immediately without grace period28912026/11/12 00:24:04 INFO Received uploads request method=POST path=/28922026/11/12 00:24:04 INFO Received uploads request method=POST path=/28932026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=028942026/11/12 00:24:04 INFO Received push request method=POST path=/28952026/11/12 00:24:04 WARN Force mode enabled - objects will be deleted immediately without grace period28962026/11/12 00:24:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28972026/11/12 00:24:04 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28982026/11/12 00:24:04 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28992026/11/12 00:24:04 INFO Received complete push request method=POST path=/29002026/11/12 00:24:04 INFO Received uploads request method=POST path=/29012026/11/12 00:24:04 INFO Completed push id=6 objects=6 duration=1.680417ms29022026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=029032026/11/12 00:24:04 INFO Received push request method=POST path=/29042026/11/12 00:24:04 INFO Received complete push request method=POST path=/29052026/11/12 00:24:04 INFO Received uploads request method=POST path=/29062026/11/12 00:24:04 INFO Received complete upload request method=POST path=/29072026/11/12 00:24:04 INFO Received push request method=POST path=/29082026/11/12 00:24:04 INFO Received complete upload request method=POST path=/29092026/11/12 00:24:04 INFO Completed upload id=15 objects=9 duration=1.876875ms29102026/11/12 00:24:04 INFO Received complete push request method=POST path=/29112026/11/12 00:24:04 INFO Completed upload id=11 objects=6 duration=1.258125ms29122026/11/12 00:24:04 INFO Completed push id=5 objects=6 duration=1.1185ms29132026/11/12 00:24:04 INFO Completed push id=1 objects=7 duration=17.428459ms29142026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=029152026/11/12 00:24:04 INFO Received complete upload request method=POST path=/29162026/11/12 00:24:04 INFO Completed upload id=5 objects=5 duration=4.651709ms29172026/11/12 00:24:04 INFO Received push request method=POST path=/29182026/11/12 00:24:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029192026/11/12 00:24:04 INFO Received uploads request method=POST path=/29202026/11/12 00:24:04 INFO Received push request method=POST path=/29212026/11/12 00:24:04 INFO Vacuumed table table=pending_closures29222026/11/12 00:24:04 INFO Vacuumed table table=pending_objects29232026/11/12 00:24:04 INFO Vacuumed table table=multipart_uploads29242026/11/12 00:24:04 INFO Vacuumed table table=closures29252026/11/12 00:24:04 INFO Received uploads request method=POST path=/29262026/11/12 00:24:04 INFO Vacuumed table table=objects29272026/11/12 00:24:04 INFO Aborted multipart uploads count=0 kept=029282026/11/12 00:24:04 WARN Force mode enabled - objects will be deleted immediately without grace period29292026/11/12 00:24:04 INFO Received uploads request method=POST path=/29302026/11/12 00:24:04 INFO Received complete push request method=POST path=/29312026/11/12 00:24:04 INFO Completed push id=21 objects=17 duration=1.22425ms29322026/11/12 00:24:04 INFO Received push request method=POST path=/29332026/11/12 00:24:04 INFO Received complete push request method=POST path=/29342026/11/12 00:24:04 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29352026/11/12 00:24:04 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29362026/11/12 00:24:04 INFO Completed push id=10 objects=15 duration=2.931958ms29372026/11/12 00:24:04 INFO Received complete push request method=POST path=/29382026/11/12 00:24:04 INFO Received uploads request method=POST path=/29392026/11/12 00:24:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=029402026/11/12 00:24:04 INFO Completed push id=12 objects=15 duration=4.144959ms29412026/11/12 00:24:04 INFO Vacuumed table table=pending_closures29422026/11/12 00:24:04 INFO Vacuumed table table=pending_objects29432026/11/12 00:24:04 INFO Received complete push request method=POST path=/29442026/11/12 00:24:04 INFO Vacuumed table table=multipart_uploads29452026/11/12 00:24:04 INFO Completed push id=19 objects=7 duration=3.005625ms29462026/11/12 00:24:04 INFO Vacuumed table table=closures29472026/11/12 00:24:04 INFO Vacuumed table table=objects29482026/11/12 00:24:04 INFO Received uploads request method=POST path=/29492026/11/12 00:24:04 INFO Received push request method=POST path=/29502026/11/12 00:24:04 INFO Received complete push request method=POST path=/29512026/11/12 00:24:04 INFO Received push request method=POST path=/29522026/11/12 00:24:04 INFO Completed push id=7 objects=20 duration=2.746084ms29532026/11/12 00:24:04 INFO Received push request method=POST path=/29542026/11/12 00:24:04 INFO Received complete upload request method=POST path=/29552026/11/12 00:24:04 INFO Completed upload id=2 objects=10 duration=5.490042ms29562026/11/12 00:24:05 INFO Received push request method=POST path=/29572026/11/12 00:24:05 INFO Received complete push request method=POST path=/29582026/11/12 00:24:05 INFO Completed push id=15 objects=7 duration=3.672875ms29592026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=029602026/11/12 00:24:05 INFO Received complete upload request method=POST path=/29612026/11/12 00:24:05 INFO Completed upload id=3 objects=25 duration=1.146792ms29622026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029632026/11/12 00:24:05 INFO Vacuumed table table=pending_closures29642026/11/12 00:24:05 INFO Vacuumed table table=pending_objects29652026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads29662026/11/12 00:24:05 INFO Vacuumed table table=closures29672026/11/12 00:24:05 INFO Vacuumed table table=objects29682026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=029692026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period29702026/11/12 00:24:05 INFO Received complete upload request method=POST path=/29712026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029722026/11/12 00:24:05 INFO Received uploads request method=POST path=/29732026/11/12 00:24:05 INFO Completed upload id=3 objects=8 duration=1.486ms29742026/11/12 00:24:05 INFO Vacuumed table table=pending_closures29752026/11/12 00:24:05 INFO Received complete push request method=POST path=/29762026/11/12 00:24:05 INFO Vacuumed table table=pending_objects29772026/11/12 00:24:05 INFO Completed push id=4 objects=6 duration=4.827208ms29782026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads29792026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=029802026/11/12 00:24:05 INFO Vacuumed table table=closures29812026/11/12 00:24:05 INFO Vacuumed table table=objects29822026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029832026/11/12 00:24:05 INFO Vacuumed table table=pending_closures29842026/11/12 00:24:05 INFO Vacuumed table table=pending_objects29852026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads29862026/11/12 00:24:05 INFO Vacuumed table table=closures29872026/11/12 00:24:05 INFO Received push request method=POST path=/29882026/11/12 00:24:05 INFO Vacuumed table table=objects29892026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=029902026/11/12 00:24:05 INFO Received uploads request method=POST path=/29912026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=029922026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period29932026/11/12 00:24:05 INFO Received uploads request method=POST path=/29942026/11/12 00:24:05 INFO Received uploads request method=POST path=/29952026/11/12 00:24:05 INFO Received uploads request method=POST path=/29962026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period29972026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=029982026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029992026/11/12 00:24:05 INFO Vacuumed table table=pending_closures30002026/11/12 00:24:05 INFO Received push request method=POST path=/30012026/11/12 00:24:05 INFO Vacuumed table table=pending_objects30022026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads30032026/11/12 00:24:05 INFO Vacuumed table table=closures30042026/11/12 00:24:05 INFO Vacuumed table table=pending_closures30052026/11/12 00:24:05 INFO Vacuumed table table=objects30062026/11/12 00:24:05 INFO Received complete push request method=POST path=/30072026/11/12 00:24:05 INFO Received uploads request method=POST path=/30082026/11/12 00:24:05 INFO Completed push id=5 objects=3 duration=2.414833ms30092026/11/12 00:24:05 INFO Received push request method=POST path=/30102026/11/12 00:24:05 INFO Vacuumed table table=pending_objects30112026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=030122026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=030132026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads30142026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period30152026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030162026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period30172026/11/12 00:24:05 INFO Vacuumed table table=pending_closures30182026/11/12 00:24:05 INFO Vacuumed table table=closures30192026/11/12 00:24:05 INFO Vacuumed table table=pending_objects30202026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads30212026/11/12 00:24:05 INFO Vacuumed table table=closures30222026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030232026/11/12 00:24:05 INFO Vacuumed table table=objects30242026/11/12 00:24:05 INFO Vacuumed table table=pending_closures30252026/11/12 00:24:05 INFO Vacuumed table table=objects30262026/11/12 00:24:05 INFO Vacuumed table table=pending_objects30272026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads30282026/11/12 00:24:05 INFO Vacuumed table table=closures30292026/11/12 00:24:05 INFO Vacuumed table table=objects30302026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=030312026/11/12 00:24:05 INFO Received push request method=POST path=/30322026/11/12 00:24:05 INFO Received complete push request method=POST path=/30332026/11/12 00:24:05 INFO Received complete upload request method=POST path=/30342026/11/12 00:24:05 INFO Completed upload id=22 objects=2 duration=1.107292ms30352026/11/12 00:24:05 INFO Completed push id=1 objects=10 duration=30.184041ms30362026/11/12 00:24:05 INFO Received complete push request method=POST path=/30372026/11/12 00:24:05 INFO Completed push id=3 objects=5 duration=2.506291ms30382026/11/12 00:24:05 INFO Received uploads request method=POST path=/30392026/11/12 00:24:05 INFO Received uploads request method=POST path=/30402026/11/12 00:24:05 INFO Received complete push request method=POST path=/30412026/11/12 00:24:05 INFO Received uploads request method=POST path=/30422026/11/12 00:24:05 INFO Received complete upload request method=POST path=/30432026/11/12 00:24:05 INFO Completed upload id=12 objects=26 duration=1.967166ms30442026/11/12 00:24:05 INFO Completed push id=1 objects=7 duration=20.831458ms30452026/11/12 00:24:05 INFO Received push request method=POST path=/30462026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=030472026/11/12 00:24:05 INFO Received uploads request method=POST path=/30482026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=030492026/11/12 00:24:05 INFO Vacuumed table table=pending_closures30502026/11/12 00:24:05 INFO Vacuumed table table=pending_objects30512026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads30522026/11/12 00:24:05 INFO Vacuumed table table=closures30532026/11/12 00:24:05 INFO Vacuumed table table=objects30542026/11/12 00:24:05 INFO Received uploads request method=POST path=/30552026/11/12 00:24:05 INFO Received push request method=POST path=/30562026/11/12 00:24:05 INFO Received uploads request method=POST path=/30572026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=030582026/11/12 00:24:05 INFO Received push request method=POST path=/30592026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period30602026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030612026/11/12 00:24:05 INFO Vacuumed table table=pending_closures30622026/11/12 00:24:05 INFO Vacuumed table table=pending_objects30632026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads30642026/11/12 00:24:05 INFO Vacuumed table table=closures30652026/11/12 00:24:05 INFO Vacuumed table table=objects30662026/11/12 00:24:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30672026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30682026/11/12 00:24:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30692026/11/12 00:24:05 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30702026/11/12 00:24:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30712026/11/12 00:24:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30722026/11/12 00:24:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30732026/11/12 00:24:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30742026/11/12 00:24:05 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30752026/11/12 00:24:05 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30762026/11/12 00:24:05 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30772026/11/12 00:24:05 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30782026/11/12 00:24:05 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30792026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."30802026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."30812026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30822026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30832026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30842026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."30852026/11/12 00:24:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30862026/11/12 00:24:05 INFO Received push request method=POST path=/30872026/11/12 00:24:05 INFO Received uploads request method=POST path=/30882026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=030892026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period30902026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=030912026/11/12 00:24:05 INFO Vacuumed table table=pending_closures30922026/11/12 00:24:05 INFO Vacuumed table table=pending_objects30932026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads30942026/11/12 00:24:05 INFO Vacuumed table table=closures30952026/11/12 00:24:05 INFO Vacuumed table table=objects30962026/11/12 00:24:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30972026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=030982026/11/12 00:24:05 INFO Received push request method=POST path=/30992026/11/12 00:24:05 INFO Received push request method=POST path=/31002026/11/12 00:24:05 INFO Received push request method=POST path=/31012026/11/12 00:24:05 INFO Received complete push request method=POST path=/31022026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period31032026/11/12 00:24:05 INFO Received push request method=POST path=/31042026/11/12 00:24:05 INFO Completed push id=8 objects=6 duration=2.039167ms31052026/11/12 00:24:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031062026/11/12 00:24:05 INFO Vacuumed table table=pending_closures31072026/11/12 00:24:05 INFO Vacuumed table table=pending_objects31082026/11/12 00:24:05 INFO Vacuumed table table=multipart_uploads31092026/11/12 00:24:05 INFO Received complete push request method=POST path=/31102026/11/12 00:24:05 INFO Vacuumed table table=closures31112026/11/12 00:24:05 INFO Received uploads request method=POST path=/31122026/11/12 00:24:05 INFO Received uploads request method=POST path=/31132026/11/12 00:24:05 INFO Vacuumed table table=objects31142026/11/12 00:24:05 INFO Completed push id=5 objects=9 duration=7.537875ms31152026/11/12 00:24:05 INFO Received uploads request method=POST path=/31162026/11/12 00:24:05 INFO Received uploads request method=POST path=/31172026/11/12 00:24:05 INFO Received push request method=POST path=/31182026/11/12 00:24:05 INFO Received push request method=POST path=/31192026/11/12 00:24:05 INFO Aborted multipart uploads count=0 kept=031202026/11/12 00:24:05 WARN Force mode enabled - objects will be deleted immediately without grace period31212026/11/12 00:24:05 INFO Received complete push request method=POST path=/31222026/11/12 00:24:05 INFO Completed push id=16 objects=8 duration=8.777125ms31232026/11/12 00:24:05 INFO Received complete upload request method=POST path=/31242026/11/12 00:24:05 INFO Received push request method=POST path=/31252026/11/12 00:24:05 INFO Received complete upload request method=POST path=/31262026/11/12 00:24:05 INFO Completed upload id=15 objects=6 duration=542.667µs31272026/11/12 00:24:05 INFO Received push request method=POST path=/31282026/11/12 00:24:05 INFO Completed upload id=3 objects=5 duration=7.004ms31292026/11/12 00:24:05 INFO Received complete push request method=POST path=/31302026/11/12 00:24:05 INFO Received push request method=POST path=/31312026/11/12 00:24:05 INFO Completed push id=6 objects=8 duration=2.614292ms31322026/11/12 00:24:05 INFO Received push request method=POST path=/31332026/11/12 00:24:05 INFO Received complete upload request method=POST path=/31342026/11/12 00:24:05 INFO Completed upload id=14 objects=8 duration=2.125083ms31352026/11/12 00:24:06 INFO Received push request method=POST path=/31362026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=031372026/11/12 00:24:06 WARN Force mode enabled - objects will be deleted immediately without grace period31382026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031392026/11/12 00:24:06 INFO Vacuumed table table=pending_closures31402026/11/12 00:24:06 INFO Received push request method=POST path=/31412026/11/12 00:24:06 INFO Vacuumed table table=pending_objects31422026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads31432026/11/12 00:24:06 INFO Vacuumed table table=closures31442026/11/12 00:24:06 INFO Received complete push request method=POST path=/31452026/11/12 00:24:06 INFO Vacuumed table table=objects31462026/11/12 00:24:06 INFO Completed push id=4 objects=4 duration=2.590875ms31472026/11/12 00:24:06 INFO Received push request method=POST path=/31482026/11/12 00:24:06 INFO Received complete upload request method=POST path=/31492026/11/12 00:24:06 INFO Completed upload id=17 objects=25 duration=2.534792ms31502026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=031512026/11/12 00:24:06 INFO Received uploads request method=POST path=/31522026/11/12 00:24:06 INFO Vacuumed table table=pending_closures31532026/11/12 00:24:06 INFO Vacuumed table table=pending_objects31542026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads31552026/11/12 00:24:06 INFO Vacuumed table table=closures31562026/11/12 00:24:06 INFO Received uploads request method=POST path=/31572026/11/12 00:24:06 INFO Received push request method=POST path=/31582026/11/12 00:24:06 INFO Vacuumed table table=objects31592026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=031602026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031612026/11/12 00:24:06 INFO Vacuumed table table=pending_closures31622026/11/12 00:24:06 INFO Vacuumed table table=pending_objects31632026/11/12 00:24:06 INFO Received uploads request method=POST path=/31642026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads31652026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=031662026/11/12 00:24:06 INFO Vacuumed table table=closures31672026/11/12 00:24:06 INFO Vacuumed table table=objects31682026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031692026/11/12 00:24:06 INFO Vacuumed table table=pending_closures31702026/11/12 00:24:06 INFO Vacuumed table table=pending_objects31712026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads31722026/11/12 00:24:06 INFO Vacuumed table table=closures31732026/11/12 00:24:06 INFO Received push request method=POST path=/31742026/11/12 00:24:06 INFO Vacuumed table table=objects31752026/11/12 00:24:06 INFO Received push request method=POST path=/31762026/11/12 00:24:06 INFO Received complete push request method=POST path=/31772026/11/12 00:24:06 INFO Received push request method=POST path=/31782026/11/12 00:24:06 INFO Received push request method=POST path=/31792026/11/12 00:24:06 INFO Completed push id=3 objects=5 duration=12.550583ms31802026/11/12 00:24:06 INFO Received push request method=POST path=/31812026/11/12 00:24:06 INFO Received uploads request method=POST path=/31822026/11/12 00:24:06 INFO Received push request method=POST path=/31832026/11/12 00:24:06 INFO Received push request method=POST path=/31842026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=031852026/11/12 00:24:06 INFO Received complete upload request method=POST path=/31862026/11/12 00:24:06 INFO Completed upload id=21 objects=2 duration=754.167µs31872026/11/12 00:24:06 WARN Force mode enabled - objects will be deleted immediately without grace period31882026/11/12 00:24:06 INFO Received push request method=POST path=/31892026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031902026/11/12 00:24:06 INFO Vacuumed table table=pending_closures31912026/11/12 00:24:06 INFO Vacuumed table table=pending_objects31922026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads31932026/11/12 00:24:06 INFO Received complete push request method=POST path=/31942026/11/12 00:24:06 INFO Completed push id=8 objects=6 duration=5.576708ms31952026/11/12 00:24:06 INFO Received complete push request method=POST path=/31962026/11/12 00:24:06 INFO Vacuumed table table=closures31972026/11/12 00:24:06 INFO Completed push id=11 objects=6 duration=1.274541ms31982026/11/12 00:24:06 INFO Received uploads request method=POST path=/31992026/11/12 00:24:06 INFO Received uploads request method=POST path=/32002026/11/12 00:24:06 INFO Received uploads request method=POST path=/32012026/11/12 00:24:06 INFO Vacuumed table table=objects32022026/11/12 00:24:06 INFO Received push request method=POST path=/32032026/11/12 00:24:06 INFO Received push request method=POST path=/32042026/11/12 00:24:06 INFO Received push request method=POST path=/32052026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=032062026/11/12 00:24:06 INFO Received complete upload request method=POST path=/32072026/11/12 00:24:06 WARN Force mode enabled - objects will be deleted immediately without grace period32082026/11/12 00:24:06 INFO Completed upload id=6 objects=24 duration=3.1655ms32092026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032102026/11/12 00:24:06 INFO Vacuumed table table=pending_closures32112026/11/12 00:24:06 INFO Vacuumed table table=pending_objects32122026/11/12 00:24:06 INFO Received complete push request method=POST path=/32132026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads32142026/11/12 00:24:06 INFO Completed push id=4 objects=3 duration=703.5µs32152026/11/12 00:24:06 INFO Received complete upload request method=POST path=/32162026/11/12 00:24:06 INFO Vacuumed table table=closures32172026/11/12 00:24:06 INFO Received complete upload request method=POST path=/32182026/11/12 00:24:06 INFO Completed upload id=7 objects=4 duration=1.500083ms32192026/11/12 00:24:06 INFO Received push request method=POST path=/32202026/11/12 00:24:06 INFO Vacuumed table table=objects32212026/11/12 00:24:06 INFO Completed upload id=2 objects=26 duration=4.529792ms32222026/11/12 00:24:06 INFO Received push request method=POST path=/32232026/11/12 00:24:06 INFO Received complete push request method=POST path=/32242026/11/12 00:24:06 INFO Received push request method=POST path=/32252026/11/12 00:24:06 INFO Completed push id=6 objects=5 duration=3.231625ms32262026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=032272026/11/12 00:24:06 WARN Force mode enabled - objects will be deleted immediately without grace period32282026/11/12 00:24:06 INFO Received uploads request method=POST path=/32292026/11/12 00:24:06 INFO Received uploads request method=POST path=/32302026/11/12 00:24:06 INFO Received complete push request method=POST path=/32312026/11/12 00:24:06 INFO Completed push id=5 objects=3 duration=4.532333ms32322026/11/12 00:24:06 INFO Received complete push request method=POST path=/32332026/11/12 00:24:06 INFO Received push request method=POST path=/32342026/11/12 00:24:06 INFO Received push request method=POST path=/32352026/11/12 00:24:06 INFO Received push request method=POST path=/32362026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032372026/11/12 00:24:06 INFO Completed push id=18 objects=6 duration=633.917µs32382026/11/12 00:24:06 INFO Vacuumed table table=pending_closures32392026/11/12 00:24:06 INFO Vacuumed table table=pending_objects32402026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads32412026/11/12 00:24:06 INFO Vacuumed table table=closures32422026/11/12 00:24:06 INFO Vacuumed table table=objects32432026/11/12 00:24:06 INFO Received complete upload request method=POST path=/32442026/11/12 00:24:06 INFO Completed upload id=10 objects=17 duration=1.153083ms32452026/11/12 00:24:06 INFO Received complete push request method=POST path=/32462026/11/12 00:24:06 INFO Completed push id=16 objects=27 duration=1.131375ms32472026/11/12 00:24:06 INFO Received push request method=POST path=/32482026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=032492026/11/12 00:24:06 WARN Force mode enabled - objects will be deleted immediately without grace period32502026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032512026/11/12 00:24:06 INFO Vacuumed table table=pending_closures32522026/11/12 00:24:06 INFO Vacuumed table table=pending_objects32532026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads32542026/11/12 00:24:06 INFO Vacuumed table table=closures32552026/11/12 00:24:06 INFO Vacuumed table table=objects32562026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=032572026/11/12 00:24:06 WARN Force mode enabled - objects will be deleted immediately without grace period32582026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032592026/11/12 00:24:06 INFO Vacuumed table table=pending_closures32602026/11/12 00:24:06 INFO Vacuumed table table=pending_objects32612026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads32622026/11/12 00:24:06 INFO Vacuumed table table=closures32632026/11/12 00:24:06 INFO Received uploads request method=POST path=/32642026/11/12 00:24:06 INFO Received push request method=POST path=/32652026/11/12 00:24:06 INFO Vacuumed table table=objects32662026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=032672026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032682026/11/12 00:24:06 INFO Vacuumed table table=pending_closures32692026/11/12 00:24:06 INFO Vacuumed table table=pending_objects32702026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads32712026/11/12 00:24:06 INFO Vacuumed table table=closures32722026/11/12 00:24:06 INFO Vacuumed table table=objects32732026/11/12 00:24:06 INFO Received complete upload request method=POST path=/32742026/11/12 00:24:06 INFO Received push request method=POST path=/32752026/11/12 00:24:06 INFO Completed upload id=4 objects=7 duration=2.531958ms32762026/11/12 00:24:06 INFO Received complete push request method=POST path=/32772026/11/12 00:24:06 INFO Completed push id=8 objects=6 duration=4.77025ms32782026/11/12 00:24:06 INFO Received push request method=POST path=/32792026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=032802026/11/12 00:24:06 WARN Force mode enabled - objects will be deleted immediately without grace period32812026/11/12 00:24:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032822026/11/12 00:24:06 INFO Vacuumed table table=pending_closures32832026/11/12 00:24:06 INFO Vacuumed table table=pending_objects32842026/11/12 00:24:06 INFO Vacuumed table table=multipart_uploads32852026/11/12 00:24:06 INFO Vacuumed table table=closures32862026/11/12 00:24:06 INFO Vacuumed table table=objects32872026/11/12 00:24:06 INFO Received push request method=POST path=/32882026/11/12 00:24:06 INFO Received push request method=POST path=/32892026/11/12 00:24:06 INFO Received uploads request method=POST path=/32902026/11/12 00:24:06 INFO Received complete upload request method=POST path=/32912026/11/12 00:24:06 INFO Received uploads request method=POST path=/32922026/11/12 00:24:06 INFO Received uploads request method=POST path=/32932026/11/12 00:24:06 INFO Received uploads request method=POST path=/32942026/11/12 00:24:06 INFO Aborted multipart uploads count=0 kept=032952026/11/12 00:24:06 INFO Completed upload id=20 objects=3 duration=1.974292ms32962026/11/12 00:24:07 INFO Received push request method=POST path=/32972026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=032982026/11/12 00:24:07 INFO Vacuumed table table=pending_closures32992026/11/12 00:24:07 INFO Vacuumed table table=pending_objects33002026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads33012026/11/12 00:24:07 INFO Vacuumed table table=closures33022026/11/12 00:24:07 INFO Received push request method=POST path=/33032026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033042026/11/12 00:24:07 INFO Received push request method=POST path=/33052026/11/12 00:24:07 INFO Vacuumed table table=objects33062026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period33072026/11/12 00:24:07 INFO Received complete upload request method=POST path=/33082026/11/12 00:24:07 INFO Received uploads request method=POST path=/33092026/11/12 00:24:07 INFO Completed upload id=9 objects=7 duration=2.158708ms33102026/11/12 00:24:07 INFO Received push request method=POST path=/33112026/11/12 00:24:07 INFO Received complete push request method=POST path=/33122026/11/12 00:24:07 INFO Received push request method=POST path=/33132026/11/12 00:24:07 INFO Received push request method=POST path=/33142026/11/12 00:24:07 INFO Received complete upload request method=POST path=/33152026/11/12 00:24:07 INFO Completed push id=9 objects=24 duration=3.413917ms33162026/11/12 00:24:07 INFO Completed upload id=9 objects=3 duration=1.371416ms33172026/11/12 00:24:07 INFO Received uploads request method=POST path=/33182026/11/12 00:24:07 INFO Received push request method=POST path=/33192026/11/12 00:24:07 INFO Received push request method=POST path=/33202026/11/12 00:24:07 INFO Received complete push request method=POST path=/33212026/11/12 00:24:07 INFO Received uploads request method=POST path=/33222026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033232026/11/12 00:24:07 INFO Completed push id=18 objects=10 duration=1.287375ms33242026/11/12 00:24:07 INFO Received complete push request method=POST path=/33252026/11/12 00:24:07 INFO Received push request method=POST path=/33262026/11/12 00:24:07 INFO Received uploads request method=POST path=/33272026/11/12 00:24:07 INFO Received push request method=POST path=/33282026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period33292026/11/12 00:24:07 INFO Received complete push request method=POST path=/33302026/11/12 00:24:07 INFO Completed push id=7 objects=24 duration=8.498375ms33312026/11/12 00:24:07 INFO Completed push id=14 objects=6 duration=6.491ms33322026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033332026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033342026/11/12 00:24:07 INFO Received push request method=POST path=/33352026/11/12 00:24:07 INFO Vacuumed table table=pending_closures33362026/11/12 00:24:07 INFO Vacuumed table table=pending_objects33372026/11/12 00:24:07 INFO Received complete upload request method=POST path=/33382026/11/12 00:24:07 INFO Received push request method=POST path=/33392026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033402026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads33412026/11/12 00:24:07 INFO Completed upload id=16 objects=17 duration=5.418209ms33422026/11/12 00:24:07 INFO Received uploads request method=POST path=/33432026/11/12 00:24:07 INFO Received complete upload request method=POST path=/33442026/11/12 00:24:07 INFO Vacuumed table table=closures33452026/11/12 00:24:07 INFO Received uploads request method=POST path=/33462026/11/12 00:24:07 INFO Vacuumed table table=objects33472026/11/12 00:24:07 INFO Completed upload id=24 objects=7 duration=877.541µs33482026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033492026/11/12 00:24:07 INFO Vacuumed table table=pending_closures33502026/11/12 00:24:07 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33512026/11/12 00:24:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33522026/11/12 00:24:07 INFO Vacuumed table table=pending_objects33532026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads33542026/11/12 00:24:07 INFO Received push request method=POST path=/33552026/11/12 00:24:07 INFO Vacuumed table table=closures33562026/11/12 00:24:07 INFO Vacuumed table table=objects33572026/11/12 00:24:07 INFO Received uploads request method=POST path=/33582026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033592026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033602026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period33612026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033622026/11/12 00:24:07 INFO Received complete push request method=POST path=/33632026/11/12 00:24:07 INFO Received push request method=POST path=/33642026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033652026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033662026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033672026/11/12 00:24:07 INFO Completed push id=11 objects=11 duration=2.741708ms33682026/11/12 00:24:07 INFO Vacuumed table table=pending_closures33692026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period33702026/11/12 00:24:07 INFO Vacuumed table table=pending_objects33712026/11/12 00:24:07 INFO Vacuumed table table=pending_closures33722026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads33732026/11/12 00:24:07 INFO Received complete upload request method=POST path=/33742026/11/12 00:24:07 INFO Vacuumed table table=closures33752026/11/12 00:24:07 INFO Completed upload id=10 objects=10 duration=1.2455ms33762026/11/12 00:24:07 INFO Vacuumed table table=objects33772026/11/12 00:24:07 INFO Vacuumed table table=pending_objects33782026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads33792026/11/12 00:24:07 INFO Received uploads request method=POST path=/33802026/11/12 00:24:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."33812026/11/12 00:24:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33822026/11/12 00:24:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33832026/11/12 00:24:07 INFO Received push request method=POST path=/33842026/11/12 00:24:07 INFO Vacuumed table table=closures33852026/11/12 00:24:07 INFO Vacuumed table table=objects33862026/11/12 00:24:07 INFO Received uploads request method=POST path=/33872026/11/12 00:24:07 INFO Received uploads request method=POST path=/33882026/11/12 00:24:07 INFO Received uploads request method=POST path=/33892026/11/12 00:24:07 INFO Received push request method=POST path=/33902026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033912026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=033922026/11/12 00:24:07 INFO Received uploads request method=POST path=/33932026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=033942026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period33952026/11/12 00:24:07 INFO Vacuumed table table=pending_closures33962026/11/12 00:24:07 INFO Received uploads request method=POST path=/33972026/11/12 00:24:07 INFO Received push request method=POST path=/33982026/11/12 00:24:07 INFO Received uploads request method=POST path=/33992026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034002026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34012026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34022026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=034032026/11/12 00:24:07 INFO Received uploads request method=POST path=/34042026/11/12 00:24:07 INFO Vacuumed table table=closures34052026/11/12 00:24:07 INFO Vacuumed table table=pending_closures34062026/11/12 00:24:07 INFO Received push request method=POST path=/34072026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=034082026/11/12 00:24:07 INFO Vacuumed table table=objects34092026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=034102026/11/12 00:24:07 INFO Received uploads request method=POST path=/34112026/11/12 00:24:07 INFO Received complete push request method=POST path=/34122026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=034132026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period34142026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34152026/11/12 00:24:07 INFO Vacuumed table table=pending_closures34162026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34172026/11/12 00:24:07 INFO Completed push id=12 objects=3 duration=7.678334ms34182026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34192026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034202026/11/12 00:24:07 INFO Received uploads request method=POST path=/34212026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034222026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34232026/11/12 00:24:07 INFO Vacuumed table table=pending_closures34242026/11/12 00:24:07 INFO Vacuumed table table=pending_closures34252026/11/12 00:24:07 INFO Vacuumed table table=closures34262026/11/12 00:24:07 INFO Vacuumed table table=closures34272026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34282026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34292026/11/12 00:24:07 INFO Vacuumed table table=objects34302026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34312026/11/12 00:24:07 INFO Vacuumed table table=closures34322026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34332026/11/12 00:24:07 INFO Vacuumed table table=objects34342026/11/12 00:24:07 INFO Vacuumed table table=closures34352026/11/12 00:24:07 INFO Received complete push request method=POST path=/34362026/11/12 00:24:07 INFO Vacuumed table table=objects34372026/11/12 00:24:07 INFO Vacuumed table table=objects34382026/11/12 00:24:07 INFO Completed push id=8 objects=13 duration=817.083µs34392026/11/12 00:24:07 INFO Received push request method=POST path=/34402026/11/12 00:24:07 INFO Received uploads request method=POST path=/34412026/11/12 00:24:07 INFO Received uploads request method=POST path=/34422026/11/12 00:24:07 INFO Received uploads request method=POST path=/34432026/11/12 00:24:07 INFO Received uploads request method=POST path=/34442026/11/12 00:24:07 INFO Received push request method=POST path=/34452026/11/12 00:24:07 INFO Received complete push request method=POST path=/34462026/11/12 00:24:07 INFO Completed push id=10 objects=2 duration=7.079875ms34472026/11/12 00:24:07 INFO Received push request method=POST path=/34482026/11/12 00:24:07 INFO Received uploads request method=POST path=/34492026/11/12 00:24:07 INFO Received uploads request method=POST path=/34502026/11/12 00:24:07 INFO Received push request method=POST path=/34512026/11/12 00:24:07 INFO Received complete push request method=POST path=/34522026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=034532026/11/12 00:24:07 INFO Completed push id=13 objects=2 duration=3.03175ms34542026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034552026/11/12 00:24:07 INFO Received push request method=POST path=/34562026/11/12 00:24:07 INFO Vacuumed table table=pending_closures34572026/11/12 00:24:07 INFO Received complete push request method=POST path=/34582026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34592026/11/12 00:24:07 INFO Completed push id=23 objects=28 duration=3.520584ms34602026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=034612026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34622026/11/12 00:24:07 INFO Vacuumed table table=closures34632026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period34642026/11/12 00:24:07 INFO Vacuumed table table=objects34652026/11/12 00:24:07 INFO Received push request method=POST path=/34662026/11/12 00:24:07 INFO Received push request method=POST path=/34672026/11/12 00:24:07 INFO Received uploads request method=POST path=/34682026/11/12 00:24:07 INFO Received uploads request method=POST path=/34692026/11/12 00:24:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34702026/11/12 00:24:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."34712026/11/12 00:24:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."34722026/11/12 00:24:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."34732026/11/12 00:24:07 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."34742026/11/12 00:24:07 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."34752026/11/12 00:24:07 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."34762026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=034772026/11/12 00:24:07 INFO Received complete upload request method=POST path=/34782026/11/12 00:24:07 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34792026/11/12 00:24:07 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."34802026/11/12 00:24:07 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."34812026/11/12 00:24:07 INFO Completed upload id=27 objects=13 duration=3.180791ms34822026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=034832026/11/12 00:24:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."34842026/11/12 00:24:07 INFO Received uploads request method=POST path=/34852026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period34862026/11/12 00:24:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."34872026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034882026/11/12 00:24:07 INFO Vacuumed table table=pending_closures34892026/11/12 00:24:07 INFO Received uploads request method=POST path=/34902026/11/12 00:24:07 INFO Vacuumed table table=pending_closures34912026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34922026/11/12 00:24:07 INFO Vacuumed table table=pending_objects34932026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34942026/11/12 00:24:07 INFO Received complete upload request method=POST path=/34952026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads34962026/11/12 00:24:07 INFO Received complete upload request method=POST path=/34972026/11/12 00:24:07 INFO Completed upload id=12 objects=21 duration=3.827125ms34982026/11/12 00:24:07 INFO Vacuumed table table=closures34992026/11/12 00:24:07 INFO Received uploads request method=POST path=/35002026/11/12 00:24:07 INFO Vacuumed table table=closures35012026/11/12 00:24:07 INFO Completed upload id=33 objects=5 duration=5.852166ms35022026/11/12 00:24:07 INFO Received push request method=POST path=/35032026/11/12 00:24:07 INFO Vacuumed table table=objects35042026/11/12 00:24:07 INFO Vacuumed table table=objects35052026/11/12 00:24:07 INFO Received complete push request method=POST path=/35062026/11/12 00:24:07 INFO Received uploads request method=POST path=/35072026/11/12 00:24:07 INFO Received uploads request method=POST path=/35082026/11/12 00:24:07 INFO Completed push id=4 objects=6 duration=12.298417ms35092026/11/12 00:24:07 INFO Received complete upload request method=POST path=/35102026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=035112026/11/12 00:24:07 INFO Completed upload id=27 objects=8 duration=4.773834ms35122026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035132026/11/12 00:24:07 INFO Vacuumed table table=pending_closures35142026/11/12 00:24:07 INFO Vacuumed table table=pending_objects35152026/11/12 00:24:07 INFO Received uploads request method=POST path=/35162026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads35172026/11/12 00:24:07 INFO Vacuumed table table=closures35182026/11/12 00:24:07 INFO Vacuumed table table=objects35192026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=035202026/11/12 00:24:07 INFO Received complete upload request method=POST path=/35212026/11/12 00:24:07 INFO Received push request method=POST path=/35222026/11/12 00:24:07 INFO Completed upload id=10 objects=6 duration=2.930583ms35232026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035242026/11/12 00:24:07 INFO Vacuumed table table=pending_closures35252026/11/12 00:24:07 INFO Vacuumed table table=pending_objects35262026/11/12 00:24:07 INFO Vacuumed table table=multipart_uploads35272026/11/12 00:24:07 INFO Vacuumed table table=closures35282026/11/12 00:24:07 INFO Received complete push request method=POST path=/35292026/11/12 00:24:07 INFO Vacuumed table table=objects35302026/11/12 00:24:07 INFO Completed push id=12 objects=2 duration=2.995208ms35312026/11/12 00:24:07 INFO Received complete push request method=POST path=/35322026/11/12 00:24:07 INFO Received uploads request method=POST path=/35332026/11/12 00:24:07 INFO Completed push id=14 objects=3 duration=2.473875ms35342026/11/12 00:24:07 INFO Received push request method=POST path=/35352026/11/12 00:24:07 INFO Received complete upload request method=POST path=/35362026/11/12 00:24:07 INFO Completed upload id=10 objects=4 duration=5.43875ms35372026/11/12 00:24:07 INFO Aborted multipart uploads count=0 kept=035382026/11/12 00:24:07 WARN Force mode enabled - objects will be deleted immediately without grace period35392026/11/12 00:24:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035402026/11/12 00:24:07 INFO Vacuumed table table=pending_closures35412026/11/12 00:24:07 INFO Received push request method=POST path=/35422026/11/12 00:24:07 INFO Vacuumed table table=pending_objects35432026/11/12 00:24:08 INFO Received uploads request method=POST path=/35442026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads35452026/11/12 00:24:08 INFO Received push request method=POST path=/35462026/11/12 00:24:08 INFO Received push request method=POST path=/35472026/11/12 00:24:08 INFO Received push request method=POST path=/35482026/11/12 00:24:08 INFO Received push request method=POST path=/35492026/11/12 00:24:08 INFO Received uploads request method=POST path=/35502026/11/12 00:24:08 INFO Vacuumed table table=closures35512026/11/12 00:24:08 INFO Received push request method=POST path=/35522026/11/12 00:24:08 INFO Received push request method=POST path=/35532026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=035542026/11/12 00:24:08 INFO Vacuumed table table=objects35552026/11/12 00:24:08 INFO Received uploads request method=POST path=/35562026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035572026/11/12 00:24:08 INFO Vacuumed table table=pending_closures35582026/11/12 00:24:08 INFO Received complete upload request method=POST path=/35592026/11/12 00:24:08 INFO Vacuumed table table=pending_objects35602026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads35612026/11/12 00:24:08 INFO Completed upload id=16 objects=23 duration=5.156792ms35622026/11/12 00:24:08 INFO Vacuumed table table=closures35632026/11/12 00:24:08 INFO Vacuumed table table=objects35642026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=035652026/11/12 00:24:08 INFO Received push request method=POST path=/35662026/11/12 00:24:08 INFO Received push request method=POST path=/35672026/11/12 00:24:08 INFO Received complete push request method=POST path=/35682026/11/12 00:24:08 INFO Completed push id=13 objects=23 duration=481.125µs35692026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035702026/11/12 00:24:08 INFO Vacuumed table table=pending_closures35712026/11/12 00:24:08 INFO Received push request method=POST path=/35722026/11/12 00:24:08 INFO Received push request method=POST path=/35732026/11/12 00:24:08 INFO Vacuumed table table=pending_objects35742026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads35752026/11/12 00:24:08 INFO Received push request method=POST path=/35762026/11/12 00:24:08 INFO Received complete upload request method=POST path=/35772026/11/12 00:24:08 INFO Received push request method=POST path=/35782026/11/12 00:24:08 INFO Vacuumed table table=closures35792026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=035802026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=035812026/11/12 00:24:08 INFO Completed upload id=15 objects=6 duration=2.085333ms35822026/11/12 00:24:08 INFO Vacuumed table table=objects35832026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=035842026/11/12 00:24:08 INFO Received uploads request method=POST path=/35852026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035862026/11/12 00:24:08 INFO Received push request method=POST path=/35872026/11/12 00:24:08 INFO Received push request method=POST path=/35882026/11/12 00:24:08 INFO Vacuumed table table=pending_closures35892026/11/12 00:24:08 INFO Received push request method=POST path=/35902026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=035912026/11/12 00:24:08 INFO Vacuumed table table=pending_objects35922026/11/12 00:24:08 INFO Received uploads request method=POST path=/35932026/11/12 00:24:08 INFO Received uploads request method=POST path=/35942026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035952026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=035962026/11/12 00:24:08 INFO Vacuumed table table=pending_closures35972026/11/12 00:24:08 INFO Received complete upload request method=POST path=/35982026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads35992026/11/12 00:24:08 INFO Vacuumed table table=closures36002026/11/12 00:24:08 INFO Completed upload id=9 objects=2 duration=558.416µs36012026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36022026/11/12 00:24:08 WARN Force mode enabled - objects will be deleted immediately without grace period36032026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36042026/11/12 00:24:08 INFO Vacuumed table table=objects36052026/11/12 00:24:08 INFO Vacuumed table table=pending_closures36062026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036072026/11/12 00:24:08 INFO Vacuumed table table=closures36082026/11/12 00:24:08 INFO Vacuumed table table=pending_closures36092026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36102026/11/12 00:24:08 INFO Vacuumed table table=objects36112026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36122026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36132026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36142026/11/12 00:24:08 INFO Vacuumed table table=closures36152026/11/12 00:24:08 INFO Vacuumed table table=closures36162026/11/12 00:24:08 INFO Vacuumed table table=objects36172026/11/12 00:24:08 INFO Vacuumed table table=objects36182026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=036192026/11/12 00:24:08 INFO Received uploads request method=POST path=/36202026/11/12 00:24:08 INFO Vacuumed table table=pending_closures36212026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36222026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36232026/11/12 00:24:08 INFO Vacuumed table table=closures36242026/11/12 00:24:08 INFO Vacuumed table table=objects36252026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=036262026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036272026/11/12 00:24:08 INFO Vacuumed table table=pending_closures36282026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36292026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36302026/11/12 00:24:08 INFO Vacuumed table table=closures36312026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=036322026/11/12 00:24:08 INFO Vacuumed table table=objects36332026/11/12 00:24:08 INFO Received complete upload request method=POST path=/36342026/11/12 00:24:08 INFO Received push request method=POST path=/36352026/11/12 00:24:08 INFO Received complete upload request method=POST path=/36362026/11/12 00:24:08 INFO Received complete push request method=POST path=/36372026/11/12 00:24:08 INFO Completed push id=8 objects=3 duration=8.4345ms36382026/11/12 00:24:08 INFO Received uploads request method=POST path=/36392026/11/12 00:24:08 INFO Completed upload id=25 objects=16 duration=8.511667ms36402026/11/12 00:24:08 INFO Completed upload id=10 objects=6 duration=8.792333ms36412026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=036422026/11/12 00:24:08 INFO Received push request method=POST path=/36432026/11/12 00:24:08 INFO Received push request method=POST path=/36442026/11/12 00:24:08 INFO Received uploads request method=POST path=/36452026/11/12 00:24:08 INFO Vacuumed table table=pending_closures36462026/11/12 00:24:08 INFO Received push request method=POST path=/36472026/11/12 00:24:08 INFO Received uploads request method=POST path=/36482026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36492026/11/12 00:24:08 INFO Received complete upload request method=POST path=/36502026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36512026/11/12 00:24:08 INFO Completed upload id=21 objects=25 duration=2.25575ms36522026/11/12 00:24:08 INFO Vacuumed table table=closures36532026/11/12 00:24:08 INFO Vacuumed table table=objects36542026/11/12 00:24:08 INFO Received complete upload request method=POST path=/36552026/11/12 00:24:08 INFO Received complete push request method=POST path=/36562026/11/12 00:24:08 INFO Received uploads request method=POST path=/36572026/11/12 00:24:08 INFO Completed upload id=17 objects=7 duration=1.141083ms36582026/11/12 00:24:08 INFO Completed push id=9 objects=4 duration=4.751ms36592026/11/12 00:24:08 INFO Received uploads request method=POST path=/36602026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=036612026/11/12 00:24:08 INFO Received complete upload request method=POST path=/36622026/11/12 00:24:08 WARN Force mode enabled - objects will be deleted immediately without grace period36632026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=036642026/11/12 00:24:08 INFO Completed upload id=10 objects=2 duration=9.9395ms36652026/11/12 00:24:08 WARN Force mode enabled - objects will be deleted immediately without grace period36662026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036672026/11/12 00:24:08 INFO Vacuumed table table=pending_closures36682026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036692026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36702026/11/12 00:24:08 INFO Received uploads request method=POST path=/36712026/11/12 00:24:08 INFO Received push request method=POST path=/36722026/11/12 00:24:08 INFO Vacuumed table table=pending_closures36732026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36742026/11/12 00:24:08 INFO Vacuumed table table=closures36752026/11/12 00:24:08 INFO Vacuumed table table=pending_objects36762026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads36772026/11/12 00:24:08 INFO Vacuumed table table=objects36782026/11/12 00:24:08 INFO Vacuumed table table=closures36792026/11/12 00:24:08 INFO Vacuumed table table=objects36802026/11/12 00:24:08 INFO Received uploads request method=POST path=/36812026/11/12 00:24:08 INFO Received uploads request method=POST path=/36822026/11/12 00:24:08 INFO Received complete upload request method=POST path=/36832026/11/12 00:24:08 INFO Completed upload id=11 objects=14 duration=710.791µs36842026/11/12 00:24:08 INFO Received push request method=POST path=/36852026/11/12 00:24:08 INFO Received uploads request method=POST path=/36862026/11/12 00:24:08 INFO Received push request method=POST path=/36872026/11/12 00:24:08 INFO Received push request method=POST path=/36882026/11/12 00:24:08 INFO Received uploads request method=POST path=/36892026/11/12 00:24:08 INFO Received uploads request method=POST path=/36902026/11/12 00:24:08 INFO Received complete upload request method=POST path=/36912026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=036922026/11/12 00:24:08 INFO Completed upload id=15 objects=3 duration=980.459µs36932026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=036942026/11/12 00:24:08 INFO Received uploads request method=POST path=/36952026/11/12 00:24:08 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36962026/11/12 00:24:08 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36972026/11/12 00:24:08 INFO Received push request method=POST path=/36982026/11/12 00:24:08 INFO Received push request method=POST path=/36992026/11/12 00:24:08 INFO Received push request method=POST path=/37002026/11/12 00:24:08 INFO Received complete upload request method=POST path=/37012026/11/12 00:24:08 INFO Received complete upload request method=POST path=/37022026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=037032026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037042026/11/12 00:24:08 INFO Completed upload id=13 objects=10 duration=1.470792ms37052026/11/12 00:24:08 INFO Vacuumed table table=pending_closures37062026/11/12 00:24:08 INFO Completed upload id=29 objects=4 duration=1.986875ms37072026/11/12 00:24:08 INFO Vacuumed table table=pending_objects37082026/11/12 00:24:08 WARN Force mode enabled - objects will be deleted immediately without grace period37092026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads37102026/11/12 00:24:08 INFO Vacuumed table table=closures37112026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037122026/11/12 00:24:08 INFO Received push request method=POST path=/37132026/11/12 00:24:08 INFO Vacuumed table table=objects37142026/11/12 00:24:08 INFO Vacuumed table table=pending_closures37152026/11/12 00:24:08 INFO Vacuumed table table=pending_objects37162026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=037172026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads37182026/11/12 00:24:08 INFO Vacuumed table table=closures37192026/11/12 00:24:08 INFO Vacuumed table table=objects37202026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037212026/11/12 00:24:08 INFO Vacuumed table table=pending_closures37222026/11/12 00:24:08 INFO Vacuumed table table=pending_objects37232026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads37242026/11/12 00:24:08 INFO Received push request method=POST path=/37252026/11/12 00:24:08 INFO Vacuumed table table=closures37262026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=037272026/11/12 00:24:08 INFO Vacuumed table table=objects37282026/11/12 00:24:08 WARN Force mode enabled - objects will be deleted immediately without grace period37292026/11/12 00:24:08 INFO Received push request method=POST path=/37302026/11/12 00:24:08 INFO Received push request method=POST path=/37312026/11/12 00:24:08 INFO Received uploads request method=POST path=/37322026/11/12 00:24:08 INFO Received uploads request method=POST path=/37332026/11/12 00:24:08 INFO Received complete push request method=POST path=/37342026/11/12 00:24:08 INFO Completed push id=8 objects=6 duration=1.085083ms37352026/11/12 00:24:08 INFO Received push request method=POST path=/37362026/11/12 00:24:08 INFO Received complete upload request method=POST path=/37372026/11/12 00:24:08 INFO Completed upload id=11 objects=6 duration=983.167µs37382026/11/12 00:24:08 INFO Received push request method=POST path=/37392026/11/12 00:24:08 INFO Received uploads request method=POST path=/37402026/11/12 00:24:08 INFO Received complete push request method=POST path=/37412026/11/12 00:24:08 INFO Completed push id=22 objects=3 duration=2.909917ms37422026/11/12 00:24:08 INFO Received uploads request method=POST path=/37432026/11/12 00:24:08 INFO Received complete upload request method=POST path=/37442026/11/12 00:24:08 INFO Completed upload id=14 objects=6 duration=3.651458ms37452026/11/12 00:24:08 INFO Received uploads request method=POST path=/37462026/11/12 00:24:08 INFO Received push request method=POST path=/37472026/11/12 00:24:08 INFO Received uploads request method=POST path=/37482026/11/12 00:24:08 INFO Received uploads request method=POST path=/37492026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=037502026/11/12 00:24:08 WARN Force mode enabled - objects will be deleted immediately without grace period37512026/11/12 00:24:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037522026/11/12 00:24:08 INFO Vacuumed table table=pending_closures37532026/11/12 00:24:08 INFO Received complete push request method=POST path=/37542026/11/12 00:24:08 INFO Vacuumed table table=pending_objects37552026/11/12 00:24:08 INFO Completed push id=12 objects=7 duration=945.292µs37562026/11/12 00:24:08 INFO Vacuumed table table=multipart_uploads37572026/11/12 00:24:08 INFO Vacuumed table table=closures37582026/11/12 00:24:08 INFO Vacuumed table table=objects37592026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=037602026/11/12 00:24:08 WARN Force mode enabled - objects will be deleted immediately without grace period37612026/11/12 00:24:08 INFO Received complete upload request method=POST path=/37622026/11/12 00:24:08 INFO Received uploads request method=POST path=/37632026/11/12 00:24:08 INFO Received uploads request method=POST path=/37642026/11/12 00:24:08 INFO Completed upload id=23 objects=18 duration=2.628125ms37652026/11/12 00:24:08 INFO Received complete upload request method=POST path=/37662026/11/12 00:24:08 INFO Received uploads request method=POST path=/37672026/11/12 00:24:08 INFO Completed upload id=15 objects=4 duration=4.913583ms37682026/11/12 00:24:08 INFO Aborted multipart uploads count=0 kept=037692026/11/12 00:24:08 INFO Received push request method=POST path=/37702026/11/12 00:24:09 INFO Received complete upload request method=POST path=/37712026/11/12 00:24:09 INFO Completed upload id=12 objects=6 duration=1.263333ms37722026/11/12 00:24:09 INFO Received uploads request method=POST path=/37732026/11/12 00:24:09 INFO Received uploads request method=POST path=/37742026/11/12 00:24:09 INFO Received uploads request method=POST path=/37752026/11/12 00:24:09 INFO Received uploads request method=POST path=/37762026/11/12 00:24:09 INFO Received uploads request method=POST path=/37772026/11/12 00:24:09 INFO Received push request method=POST path=/37782026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037792026/11/12 00:24:09 INFO Received complete push request method=POST path=/37802026/11/12 00:24:09 INFO Completed push id=17 objects=3 duration=1.668292ms37812026/11/12 00:24:09 INFO Received push request method=POST path=/37822026/11/12 00:24:09 INFO Received push request method=POST path=/37832026/11/12 00:24:09 INFO Received complete push request method=POST path=/37842026/11/12 00:24:09 INFO Received complete upload request method=POST path=/37852026/11/12 00:24:09 INFO Received push request method=POST path=/37862026/11/12 00:24:09 INFO Received uploads request method=POST path=/37872026/11/12 00:24:09 INFO Completed push id=14 objects=3 duration=2.486416ms37882026/11/12 00:24:09 INFO Received complete push request method=POST path=/37892026/11/12 00:24:09 INFO Completed upload id=22 objects=5 duration=1.65275ms37902026/11/12 00:24:09 INFO Vacuumed table table=pending_closures37912026/11/12 00:24:09 INFO Received uploads request method=POST path=/37922026/11/12 00:24:09 INFO Received complete push request method=POST path=/37932026/11/12 00:24:09 INFO Completed push id=9 objects=5 duration=3.3135ms37942026/11/12 00:24:09 INFO Completed push id=11 objects=6 duration=4.937167ms37952026/11/12 00:24:09 INFO Received push request method=POST path=/37962026/11/12 00:24:09 INFO Received complete upload request method=POST path=/37972026/11/12 00:24:09 INFO Completed upload id=7 objects=3 duration=5.169959ms37982026/11/12 00:24:09 INFO Vacuumed table table=pending_objects37992026/11/12 00:24:09 INFO Received complete upload request method=POST path=/38002026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38012026/11/12 00:24:09 INFO Received complete upload request method=POST path=/38022026/11/12 00:24:09 INFO Completed upload id=22 objects=22 duration=4.113959ms38032026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=038042026/11/12 00:24:09 INFO Received complete push request method=POST path=/38052026/11/12 00:24:09 INFO Vacuumed table table=closures38062026/11/12 00:24:09 INFO Completed upload id=16 objects=3 duration=2.977ms38072026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=038082026/11/12 00:24:09 INFO Received complete push request method=POST path=/38092026/11/12 00:24:09 WARN Force mode enabled - objects will be deleted immediately without grace period38102026/11/12 00:24:09 INFO Completed push id=28 objects=3 duration=1.002542ms38112026/11/12 00:24:09 INFO Vacuumed table table=objects38122026/11/12 00:24:09 INFO Completed push id=5 objects=6 duration=1.737417ms38132026/11/12 00:24:09 WARN Force mode enabled - objects will be deleted immediately without grace period38142026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038152026/11/12 00:24:09 INFO Received complete push request method=POST path=/38162026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038172026/11/12 00:24:09 INFO Vacuumed table table=pending_closures38182026/11/12 00:24:09 INFO Completed push id=23 objects=5 duration=1.543042ms38192026/11/12 00:24:09 INFO Vacuumed table table=pending_closures38202026/11/12 00:24:09 INFO Received complete push request method=POST path=/38212026/11/12 00:24:09 INFO Vacuumed table table=pending_objects38222026/11/12 00:24:09 INFO Completed push id=18 objects=6 duration=3.496083ms38232026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38242026/11/12 00:24:09 INFO Vacuumed table table=pending_objects38252026/11/12 00:24:09 INFO Received complete push request method=POST path=/38262026/11/12 00:24:09 INFO Received push request method=POST path=/38272026/11/12 00:24:09 INFO Completed push id=11 objects=8 duration=9.482ms38282026/11/12 00:24:09 INFO Vacuumed table table=closures38292026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38302026/11/12 00:24:09 INFO Vacuumed table table=closures38312026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=038322026/11/12 00:24:09 INFO Vacuumed table table=objects38332026/11/12 00:24:09 INFO Vacuumed table table=objects38342026/11/12 00:24:09 INFO Received complete push request method=POST path=/38352026/11/12 00:24:09 WARN Force mode enabled - objects will be deleted immediately without grace period38362026/11/12 00:24:09 INFO Completed push id=20 objects=13 duration=3.795417ms38372026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038382026/11/12 00:24:09 INFO Received complete push request method=POST path=/38392026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=038402026/11/12 00:24:09 INFO Vacuumed table table=pending_closures38412026/11/12 00:24:09 INFO Completed push id=29 objects=15 duration=5.736042ms38422026/11/12 00:24:09 INFO Received complete upload request method=POST path=/38432026/11/12 00:24:09 INFO Vacuumed table table=pending_objects38442026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=038452026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038462026/11/12 00:24:09 INFO Completed upload id=24 objects=3 duration=1.540667ms38472026/11/12 00:24:09 INFO Received complete upload request method=POST path=/38482026/11/12 00:24:09 INFO Received push request method=POST path=/38492026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38502026/11/12 00:24:09 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38512026/11/12 00:24:09 INFO Received complete push request method=POST path=/38522026/11/12 00:24:09 INFO Received complete push request method=POST path=/38532026/11/12 00:24:09 INFO Completed upload id=23 objects=9 duration=709.708µs38542026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038552026/11/12 00:24:09 INFO Received complete push request method=POST path=/38562026/11/12 00:24:09 INFO Vacuumed table table=closures38572026/11/12 00:24:09 INFO Vacuumed table table=pending_closures38582026/11/12 00:24:09 INFO Completed push id=22 objects=4 duration=3.238417ms38592026/11/12 00:24:09 INFO Completed push id=26 objects=5 duration=2.066208ms38602026/11/12 00:24:09 INFO Received complete push request method=POST path=/38612026/11/12 00:24:09 INFO Completed push id=30 objects=6 duration=534.625µs38622026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=038632026/11/12 00:24:09 INFO Vacuumed table table=pending_closures38642026/11/12 00:24:09 INFO Received complete push request method=POST path=/38652026/11/12 00:24:09 INFO Vacuumed table table=pending_objects38662026/11/12 00:24:09 INFO Received uploads request method=POST path=/38672026/11/12 00:24:09 INFO Completed push id=18 objects=23 duration=7.088667ms38682026/11/12 00:24:09 INFO Completed push id=17 objects=22 duration=7.654792ms38692026/11/12 00:24:09 INFO Vacuumed table table=objects38702026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38712026/11/12 00:24:09 INFO Received complete push request method=POST path=/38722026/11/12 00:24:09 INFO Vacuumed table table=pending_objects38732026/11/12 00:24:09 WARN Force mode enabled - objects will be deleted immediately without grace period38742026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38752026/11/12 00:24:09 INFO Vacuumed table table=closures38762026/11/12 00:24:09 INFO Completed push id=25 objects=4 duration=6.995667ms38772026/11/12 00:24:09 INFO Vacuumed table table=closures38782026/11/12 00:24:09 INFO Vacuumed table table=objects38792026/11/12 00:24:09 INFO Vacuumed table table=objects38802026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038812026/11/12 00:24:09 INFO Vacuumed table table=pending_closures38822026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=038832026/11/12 00:24:09 INFO Vacuumed table table=pending_closures38842026/11/12 00:24:09 INFO Vacuumed table table=pending_objects38852026/11/12 00:24:09 INFO Vacuumed table table=pending_objects38862026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38872026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads38882026/11/12 00:24:09 INFO Received uploads request method=POST path=/38892026/11/12 00:24:09 INFO Vacuumed table table=closures38902026/11/12 00:24:09 INFO Vacuumed table table=closures38912026/11/12 00:24:09 INFO Vacuumed table table=objects38922026/11/12 00:24:09 INFO Vacuumed table table=objects38932026/11/12 00:24:09 INFO Received complete upload request method=POST path=/38942026/11/12 00:24:09 INFO Completed upload id=24 objects=14 duration=1.117084ms38952026/11/12 00:24:09 INFO Received complete upload request method=POST path=/38962026/11/12 00:24:09 INFO Completed upload id=31 objects=3 duration=1.084042ms38972026/11/12 00:24:09 INFO Received uploads request method=POST path=/38982026/11/12 00:24:09 INFO Received uploads request method=POST path=/38992026/11/12 00:24:09 INFO Received uploads request method=POST path=/39002026/11/12 00:24:09 INFO Received push request method=POST path=/39012026/11/12 00:24:09 INFO Received push request method=POST path=/39022026/11/12 00:24:09 INFO Received complete push request method=POST path=/39032026/11/12 00:24:09 INFO Received uploads request method=POST path=/39042026/11/12 00:24:09 INFO Completed push id=19 objects=3 duration=1.6615ms39052026/11/12 00:24:09 INFO Received push request method=POST path=/39062026/11/12 00:24:09 INFO Received uploads request method=POST path=/39072026/11/12 00:24:09 INFO Received push request method=POST path=/39082026/11/12 00:24:09 INFO Received uploads request method=POST path=/39092026/11/12 00:24:09 INFO Received complete upload request method=POST path=/39102026/11/12 00:24:09 INFO Completed upload id=23 objects=7 duration=970.125µs39112026/11/12 00:24:09 INFO Received complete upload request method=POST path=/39122026/11/12 00:24:09 INFO Completed upload id=18 objects=4 duration=1.793334ms39132026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=039142026/11/12 00:24:09 WARN Force mode enabled - objects will be deleted immediately without grace period39152026/11/12 00:24:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039162026/11/12 00:24:09 INFO Received uploads request method=POST path=/39172026/11/12 00:24:09 INFO Vacuumed table table=pending_closures39182026/11/12 00:24:09 INFO Vacuumed table table=pending_objects39192026/11/12 00:24:09 INFO Received uploads request method=POST path=/39202026/11/12 00:24:09 INFO Vacuumed table table=multipart_uploads39212026/11/12 00:24:09 INFO Received complete push request method=POST path=/39222026/11/12 00:24:09 INFO Vacuumed table table=closures39232026/11/12 00:24:09 INFO Completed push id=24 objects=5 duration=2.732ms39242026/11/12 00:24:09 INFO Vacuumed table table=objects39252026/11/12 00:24:09 INFO Received uploads request method=POST path=/39262026/11/12 00:24:09 INFO Received complete push request method=POST path=/39272026/11/12 00:24:09 INFO Completed push id=14 objects=4 duration=3.31475ms39282026/11/12 00:24:09 INFO Received complete push request method=POST path=/39292026/11/12 00:24:09 INFO Completed push id=11 objects=9 duration=970.125µs39302026/11/12 00:24:09 INFO Aborted multipart uploads count=0 kept=039312026/11/12 00:24:09 INFO Received push request method=POST path=/39322026/11/12 00:24:09 INFO Received push request method=POST path=/39332026/11/12 00:24:09 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."39342026/11/12 00:24:09 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."39352026/11/12 00:24:09 INFO Received complete push request method=POST path=/39362026/11/12 00:24:09 INFO Completed push id=13 objects=6 duration=2.608292ms39372026/11/12 00:24:09 INFO Received complete upload request method=POST path=/39382026/11/12 00:24:09 INFO Received complete push request method=POST path=/39392026/11/12 00:24:09 INFO Completed push id=17 objects=6 duration=1.743917ms39402026/11/12 00:24:09 INFO Completed upload id=25 objects=2 duration=4.864125ms39412026/11/12 00:24:09 INFO Received push request method=POST path=/39422026/11/12 00:24:09 INFO Received push request method=POST path=/39432026/11/12 00:24:09 INFO Received uploads request method=POST path=/39442026/11/12 00:24:09 INFO Received complete push request method=POST path=/39452026/11/12 00:24:09 INFO Received push request method=POST path=/39462026/11/12 00:24:09 INFO Completed push id=19 objects=14 duration=1.141125ms39472026/11/12 00:24:09 INFO Received push request method=POST path=/39482026/11/12 00:24:09 INFO Received complete upload request method=POST path=/39492026/11/12 00:24:09 INFO Received push request method=POST path=/39502026/11/12 00:24:10 INFO Received uploads request method=POST path=/39512026/11/12 00:24:10 INFO Completed upload id=26 objects=20 duration=3.239084ms39522026/11/12 00:24:10 INFO Received push request method=POST path=/39532026/11/12 00:24:10 INFO Received push request method=POST path=/39542026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=039552026/11/12 00:24:10 INFO Received uploads request method=POST path=/39562026/11/12 00:24:10 WARN Force mode enabled - objects will be deleted immediately without grace period39572026/11/12 00:24:10 INFO Received uploads request method=POST path=/39582026/11/12 00:24:10 INFO Received push request method=POST path=/39592026/11/12 00:24:10 INFO Received complete push request method=POST path=/39602026/11/12 00:24:10 INFO Completed push id=3 objects=21 duration=4.379042ms39612026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=039622026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=039632026/11/12 00:24:10 INFO Received complete upload request method=POST path=/39642026/11/12 00:24:10 INFO Completed upload id=25 objects=4 duration=2.255875ms39652026/11/12 00:24:10 INFO Vacuumed table table=pending_closures39662026/11/12 00:24:10 INFO Received uploads request method=POST path=/39672026/11/12 00:24:10 INFO Vacuumed table table=pending_objects39682026/11/12 00:24:10 WARN Force mode enabled - objects will be deleted immediately without grace period39692026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=039702026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads39712026/11/12 00:24:10 INFO Received uploads request method=POST path=/39722026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039732026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039742026/11/12 00:24:10 INFO Vacuumed table table=closures39752026/11/12 00:24:10 INFO Received complete push request method=POST path=/39762026/11/12 00:24:10 INFO Received push request method=POST path=/39772026/11/12 00:24:10 INFO Vacuumed table table=pending_closures39782026/11/12 00:24:10 INFO Received complete push request method=POST path=/39792026/11/12 00:24:10 INFO Completed push id=16 objects=26 duration=3.948208ms39802026/11/12 00:24:10 INFO Vacuumed table table=objects39812026/11/12 00:24:10 INFO Vacuumed table table=pending_objects39822026/11/12 00:24:10 INFO Completed push id=26 objects=23 duration=1.743291ms39832026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=039842026/11/12 00:24:10 INFO Vacuumed table table=pending_closures39852026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads39862026/11/12 00:24:10 INFO Received complete push request method=POST path=/39872026/11/12 00:24:10 INFO Received push request method=POST path=/39882026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=039892026/11/12 00:24:10 WARN Force mode enabled - objects will be deleted immediately without grace period39902026/11/12 00:24:10 INFO Vacuumed table table=closures39912026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=039922026/11/12 00:24:10 INFO Vacuumed table table=pending_objects39932026/11/12 00:24:10 INFO Completed push id=28 objects=7 duration=5.286042ms39942026/11/12 00:24:10 INFO Vacuumed table table=objects39952026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039962026/11/12 00:24:10 WARN Force mode enabled - objects will be deleted immediately without grace period39972026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads39982026/11/12 00:24:10 INFO Received complete upload request method=POST path=/39992026/11/12 00:24:10 WARN Force mode enabled - objects will be deleted immediately without grace period40002026/11/12 00:24:10 INFO Vacuumed table table=pending_closures40012026/11/12 00:24:10 INFO Completed upload id=21 objects=7 duration=6.586875ms40022026/11/12 00:24:10 INFO Received complete push request method=POST path=/40032026/11/12 00:24:10 INFO Vacuumed table table=closures40042026/11/12 00:24:10 INFO Vacuumed table table=pending_objects40052026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040062026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040072026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40082026/11/12 00:24:10 INFO Vacuumed table table=objects40092026/11/12 00:24:10 INFO Completed push id=17 objects=22 duration=6.784875ms40102026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads40112026/11/12 00:24:10 INFO Vacuumed table table=pending_closures40122026/11/12 00:24:10 INFO Received uploads request method=POST path=/40132026/11/12 00:24:10 INFO Received complete push request method=POST path=/40142026/11/12 00:24:10 INFO Vacuumed table table=pending_closures40152026/11/12 00:24:10 INFO Completed upload id=20 objects=6 duration=1.429416ms40162026/11/12 00:24:10 INFO Vacuumed table table=closures40172026/11/12 00:24:10 INFO Vacuumed table table=pending_objects40182026/11/12 00:24:10 INFO Completed push id=20 objects=8 duration=1.448917ms40192026/11/12 00:24:10 INFO Received push request method=POST path=/40202026/11/12 00:24:10 INFO Vacuumed table table=pending_objects40212026/11/12 00:24:10 INFO Vacuumed table table=objects40222026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads40232026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads40242026/11/12 00:24:10 INFO Received uploads request method=POST path=/40252026/11/12 00:24:10 INFO Vacuumed table table=closures40262026/11/12 00:24:10 INFO Vacuumed table table=closures40272026/11/12 00:24:10 INFO Vacuumed table table=objects40282026/11/12 00:24:10 INFO Vacuumed table table=objects40292026/11/12 00:24:10 INFO Received uploads request method=POST path=/40302026/11/12 00:24:10 INFO Received uploads request method=POST path=/40312026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40322026/11/12 00:24:10 INFO Completed upload id=30 objects=6 duration=523.958µs40332026/11/12 00:24:10 INFO Received complete push request method=POST path=/40342026/11/12 00:24:10 INFO Completed push id=27 objects=6 duration=5.984709ms40352026/11/12 00:24:10 INFO Received uploads request method=POST path=/40362026/11/12 00:24:10 INFO Received push request method=POST path=/40372026/11/12 00:24:10 INFO Received complete push request method=POST path=/40382026/11/12 00:24:10 INFO Completed push id=17 objects=6 duration=1.828125ms40392026/11/12 00:24:10 INFO Received complete push request method=POST path=/40402026/11/12 00:24:10 INFO Completed push id=27 objects=15 duration=13.388375ms40412026/11/12 00:24:10 INFO Received complete push request method=POST path=/40422026/11/12 00:24:10 INFO Received complete push request method=POST path=/40432026/11/12 00:24:10 INFO Received complete push request method=POST path=/40442026/11/12 00:24:10 INFO Completed push id=19 objects=8 duration=7.756916ms40452026/11/12 00:24:10 INFO Completed push id=15 objects=5 duration=6.270542ms40462026/11/12 00:24:10 INFO Completed push id=31 objects=5 duration=1.476958ms40472026/11/12 00:24:10 INFO Received uploads request method=POST path=/40482026/11/12 00:24:10 INFO Received uploads request method=POST path=/40492026/11/12 00:24:10 INFO Received uploads request method=POST path=/40502026/11/12 00:24:10 INFO Received push request method=POST path=/40512026/11/12 00:24:10 INFO Received push request method=POST path=/40522026/11/12 00:24:10 INFO Received complete push request method=POST path=/40532026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40542026/11/12 00:24:10 INFO Completed push id=27 objects=11 duration=2.223625ms40552026/11/12 00:24:10 INFO Completed upload id=21 objects=2 duration=847.084µs40562026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40572026/11/12 00:24:10 INFO Received uploads request method=POST path=/40582026/11/12 00:24:10 INFO Completed upload id=15 objects=2 duration=677.875µs40592026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=040602026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40612026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040622026/11/12 00:24:10 INFO Completed upload id=23 objects=15 duration=3.786583ms40632026/11/12 00:24:10 INFO Received uploads request method=POST path=/40642026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40652026/11/12 00:24:10 INFO Vacuumed table table=pending_closures40662026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40672026/11/12 00:24:10 INFO Completed upload id=29 objects=26 duration=2.32825ms40682026/11/12 00:24:10 INFO Vacuumed table table=pending_objects40692026/11/12 00:24:10 INFO Completed upload id=27 objects=4 duration=2.567708ms40702026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads40712026/11/12 00:24:10 INFO Received push request method=POST path=/40722026/11/12 00:24:10 INFO Received push request method=POST path=/40732026/11/12 00:24:10 INFO Vacuumed table table=closures40742026/11/12 00:24:10 INFO Vacuumed table table=objects40752026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=040762026/11/12 00:24:10 INFO Received complete push request method=POST path=/40772026/11/12 00:24:10 INFO Received complete push request method=POST path=/40782026/11/12 00:24:10 WARN Force mode enabled - objects will be deleted immediately without grace period40792026/11/12 00:24:10 INFO Completed push id=30 objects=8 duration=685.416µs40802026/11/12 00:24:10 INFO Completed push id=13 objects=11 duration=2.384208ms40812026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040822026/11/12 00:24:10 INFO Vacuumed table table=pending_closures40832026/11/12 00:24:10 INFO Received push request method=POST path=/40842026/11/12 00:24:10 INFO Vacuumed table table=pending_objects40852026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=040862026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads40872026/11/12 00:24:10 INFO Received complete push request method=POST path=/40882026/11/12 00:24:10 INFO Received uploads request method=POST path=/40892026/11/12 00:24:10 WARN Force mode enabled - objects will be deleted immediately without grace period40902026/11/12 00:24:10 INFO Vacuumed table table=closures40912026/11/12 00:24:10 INFO Completed push id=32 objects=26 duration=1.839583ms40922026/11/12 00:24:10 INFO Received push request method=POST path=/40932026/11/12 00:24:10 INFO Received complete upload request method=POST path=/40942026/11/12 00:24:10 INFO Vacuumed table table=objects40952026/11/12 00:24:10 INFO Completed upload id=31 objects=5 duration=2.544292ms40962026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040972026/11/12 00:24:10 INFO Vacuumed table table=pending_closures40982026/11/12 00:24:10 INFO Aborted multipart uploads count=0 kept=040992026/11/12 00:24:10 INFO Received complete upload request method=POST path=/41002026/11/12 00:24:10 INFO Vacuumed table table=pending_objects41012026/11/12 00:24:10 INFO Completed upload id=19 objects=6 duration=3.102334ms41022026/11/12 00:24:10 INFO Vacuumed table table=multipart_uploads41032026/11/12 00:24:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041042026/11/12 00:24:10 INFO Vacuumed table table=closures41052026/11/12 00:24:10 INFO Received complete push request method=POST path=/41062026/11/12 00:24:10 INFO Received uploads request method=POST path=/41072026/11/12 00:24:10 INFO Vacuumed table table=pending_closures41082026/11/12 00:24:10 INFO Received complete upload request method=POST path=/41092026/11/12 00:24:10 INFO Vacuumed table table=objects41102026/11/12 00:24:10 INFO Completed push id=4 objects=7 duration=4.365542ms41112026/11/12 00:24:11 INFO Received complete push request method=POST path=/41122026/11/12 00:24:11 INFO Received uploads request method=POST path=/41132026/11/12 00:24:11 INFO Completed upload id=33 objects=2 duration=4.506375ms41142026/11/12 00:24:11 INFO Received complete push request method=POST path=/41152026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=041162026/11/12 00:24:11 INFO Completed push id=16 objects=16 duration=5.676208ms41172026/11/12 00:24:11 INFO Completed push id=32 objects=3 duration=820.708µs41182026/11/12 00:24:11 INFO Vacuumed table table=pending_objects41192026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period41202026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads41212026/11/12 00:24:11 INFO Vacuumed table table=closures41222026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041232026/11/12 00:24:11 INFO Vacuumed table table=pending_closures41242026/11/12 00:24:11 INFO Vacuumed table table=objects41252026/11/12 00:24:11 INFO Vacuumed table table=pending_objects41262026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads41272026/11/12 00:24:11 INFO Vacuumed table table=closures41282026/11/12 00:24:11 INFO Vacuumed table table=objects41292026/11/12 00:24:11 INFO Received uploads request method=POST path=/41302026/11/12 00:24:11 INFO Received uploads request method=POST path=/41312026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=041322026/11/12 00:24:11 INFO Received push request method=POST path=/41332026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period41342026/11/12 00:24:11 INFO Received uploads request method=POST path=/41352026/11/12 00:24:11 INFO Received complete upload request method=POST path=/41362026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041372026/11/12 00:24:11 INFO Completed upload id=22 objects=7 duration=1.546416ms41382026/11/12 00:24:11 INFO Received complete upload request method=POST path=/41392026/11/12 00:24:11 INFO Vacuumed table table=pending_closures41402026/11/12 00:24:11 INFO Completed upload id=34 objects=5 duration=554.375µs41412026/11/12 00:24:11 INFO Vacuumed table table=pending_objects41422026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads41432026/11/12 00:24:11 INFO Received uploads request method=POST path=/41442026/11/12 00:24:11 INFO Vacuumed table table=closures41452026/11/12 00:24:11 INFO Vacuumed table table=objects41462026/11/12 00:24:11 INFO Received push request method=POST path=/41472026/11/12 00:24:11 INFO Received complete upload request method=POST path=/41482026/11/12 00:24:11 INFO Received uploads request method=POST path=/41492026/11/12 00:24:11 INFO Completed upload id=26 objects=8 duration=678.875µs41502026/11/12 00:24:11 INFO Received uploads request method=POST path=/41512026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=041522026/11/12 00:24:11 INFO Received complete push request method=POST path=/41532026/11/12 00:24:11 INFO Completed push id=28 objects=19 duration=4.910667ms41542026/11/12 00:24:11 INFO Received push request method=POST path=/41552026/11/12 00:24:11 INFO Received complete push request method=POST path=/41562026/11/12 00:24:11 INFO Received complete push request method=POST path=/41572026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=041582026/11/12 00:24:11 INFO Received uploads request method=POST path=/41592026/11/12 00:24:11 INFO Completed push id=24 objects=24 duration=5.535833ms41602026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period41612026/11/12 00:24:11 INFO Completed push id=15 objects=24 duration=6.959041ms41622026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041632026/11/12 00:24:11 INFO Vacuumed table table=pending_closures41642026/11/12 00:24:11 INFO Vacuumed table table=pending_objects41652026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads41662026/11/12 00:24:11 INFO Vacuumed table table=closures41672026/11/12 00:24:11 INFO Received complete upload request method=POST path=/41682026/11/12 00:24:11 INFO Received complete push request method=POST path=/41692026/11/12 00:24:11 INFO Vacuumed table table=objects41702026/11/12 00:24:11 INFO Completed upload id=35 objects=11 duration=463.292µs41712026/11/12 00:24:11 INFO Completed push id=28 objects=14 duration=616.5µs41722026/11/12 00:24:11 INFO Received complete push request method=POST path=/41732026/11/12 00:24:11 INFO Received uploads request method=POST path=/41742026/11/12 00:24:11 INFO Completed push id=16 objects=14 duration=1.955542ms41752026/11/12 00:24:11 INFO Received complete upload request method=POST path=/41762026/11/12 00:24:11 INFO Completed upload id=7 objects=10 duration=5.310417ms41772026/11/12 00:24:11 INFO Received uploads request method=POST path=/41782026/11/12 00:24:11 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."41792026/11/12 00:24:11 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."41802026/11/12 00:24:11 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."41812026/11/12 00:24:11 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."41822026/11/12 00:24:11 INFO Received complete push request method=POST path=/41832026/11/12 00:24:11 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41842026/11/12 00:24:11 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."41852026/11/12 00:24:11 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41862026/11/12 00:24:11 INFO Completed push id=24 objects=9 duration=1.085666ms41872026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=041882026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period41892026/11/12 00:24:11 INFO Received uploads request method=POST path=/41902026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041912026/11/12 00:24:11 INFO Vacuumed table table=pending_closures41922026/11/12 00:24:11 INFO Vacuumed table table=pending_objects41932026/11/12 00:24:11 INFO Received push request method=POST path=/41942026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads41952026/11/12 00:24:11 INFO Vacuumed table table=closures41962026/11/12 00:24:11 INFO Vacuumed table table=objects41972026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=041982026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period41992026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042002026/11/12 00:24:11 INFO Vacuumed table table=pending_closures42012026/11/12 00:24:11 INFO Received complete push request method=POST path=/42022026/11/12 00:24:11 INFO Vacuumed table table=pending_objects42032026/11/12 00:24:11 INFO Completed push id=6 objects=26 duration=3.0375ms42042026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads42052026/11/12 00:24:11 INFO Vacuumed table table=closures42062026/11/12 00:24:11 INFO Vacuumed table table=objects42072026/11/12 00:24:11 INFO Received complete upload request method=POST path=/42082026/11/12 00:24:11 INFO Completed upload id=36 objects=3 duration=714.042µs42092026/11/12 00:24:11 INFO Received uploads request method=POST path=/42102026/11/12 00:24:11 INFO Received push request method=POST path=/42112026/11/12 00:24:11 INFO Received push request method=POST path=/42122026/11/12 00:24:11 INFO Received uploads request method=POST path=/42132026/11/12 00:24:11 INFO Received uploads request method=POST path=/42142026/11/12 00:24:11 INFO Received push request method=POST path=/42152026/11/12 00:24:11 INFO Received complete push request method=POST path=/42162026/11/12 00:24:11 INFO Completed push id=29 objects=10 duration=2.235ms42172026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=042182026/11/12 00:24:11 INFO Received uploads request method=POST path=/42192026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=042202026/11/12 00:24:11 INFO Received uploads request method=POST path=/42212026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period42222026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042232026/11/12 00:24:11 INFO Received uploads request method=POST path=/42242026/11/12 00:24:11 INFO Vacuumed table table=pending_closures42252026/11/12 00:24:11 INFO Vacuumed table table=pending_objects42262026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042272026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads42282026/11/12 00:24:11 INFO Vacuumed table table=closures42292026/11/12 00:24:11 INFO Vacuumed table table=objects42302026/11/12 00:24:11 INFO Vacuumed table table=pending_closures42312026/11/12 00:24:11 INFO Vacuumed table table=pending_objects42322026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads42332026/11/12 00:24:11 INFO Vacuumed table table=closures42342026/11/12 00:24:11 INFO Received complete push request method=POST path=/42352026/11/12 00:24:11 INFO Received uploads request method=POST path=/42362026/11/12 00:24:11 INFO Vacuumed table table=objects42372026/11/12 00:24:11 INFO Completed push id=26 objects=14 duration=1.914875ms42382026/11/12 00:24:11 INFO Received uploads request method=POST path=/42392026/11/12 00:24:11 INFO Received push request method=POST path=/42402026/11/12 00:24:11 INFO Received push request method=POST path=/42412026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=042422026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period42432026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042442026/11/12 00:24:11 INFO Vacuumed table table=pending_closures42452026/11/12 00:24:11 INFO Vacuumed table table=pending_objects42462026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads42472026/11/12 00:24:11 INFO Vacuumed table table=closures42482026/11/12 00:24:11 INFO Vacuumed table table=objects42492026/11/12 00:24:11 INFO Received push request method=POST path=/42502026/11/12 00:24:11 INFO Received complete push request method=POST path=/42512026/11/12 00:24:11 INFO Completed push id=14 objects=3 duration=7.292416ms42522026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=042532026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period42542026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042552026/11/12 00:24:11 INFO Vacuumed table table=pending_closures42562026/11/12 00:24:11 INFO Received push request method=POST path=/42572026/11/12 00:24:11 INFO Vacuumed table table=pending_objects42582026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads42592026/11/12 00:24:11 INFO Received uploads request method=POST path=/42602026/11/12 00:24:11 INFO Vacuumed table table=closures42612026/11/12 00:24:11 INFO Vacuumed table table=objects42622026/11/12 00:24:11 INFO Received uploads request method=POST path=/42632026/11/12 00:24:11 INFO Received push request method=POST path=/42642026/11/12 00:24:11 INFO Received complete upload request method=POST path=/42652026/11/12 00:24:11 INFO Completed upload id=30 objects=6 duration=708.042µs42662026/11/12 00:24:11 INFO Received uploads request method=POST path=/42672026/11/12 00:24:11 INFO Received uploads request method=POST path=/42682026/11/12 00:24:11 INFO Received uploads request method=POST path=/42692026/11/12 00:24:11 INFO Received uploads request method=POST path=/42702026/11/12 00:24:11 INFO Received complete upload request method=POST path=/42712026/11/12 00:24:11 INFO Completed upload id=25 objects=6 duration=3.881417ms42722026/11/12 00:24:11 INFO Received complete push request method=POST path=/42732026/11/12 00:24:11 INFO Completed push id=7 objects=7 duration=4.173792ms42742026/11/12 00:24:11 INFO Received complete push request method=POST path=/42752026/11/12 00:24:11 INFO Received uploads request method=POST path=/42762026/11/12 00:24:11 INFO Completed push id=21 objects=9 duration=1.127125ms42772026/11/12 00:24:11 INFO Received complete upload request method=POST path=/42782026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=042792026/11/12 00:24:11 INFO Completed upload id=6 objects=6 duration=5.782916ms42802026/11/12 00:24:11 INFO Received complete upload request method=POST path=/42812026/11/12 00:24:11 INFO Completed upload id=31 objects=5 duration=2.3505ms42822026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042832026/11/12 00:24:11 INFO Vacuumed table table=pending_closures42842026/11/12 00:24:11 INFO Vacuumed table table=pending_objects42852026/11/12 00:24:11 INFO Vacuumed table table=multipart_uploads42862026/11/12 00:24:11 INFO Vacuumed table table=closures42872026/11/12 00:24:11 INFO Received uploads request method=POST path=/42882026/11/12 00:24:11 INFO Vacuumed table table=objects42892026/11/12 00:24:11 INFO Received complete upload request method=POST path=/42902026/11/12 00:24:11 INFO Completed upload id=21 objects=9 duration=6.170667ms42912026/11/12 00:24:11 INFO Received uploads request method=POST path=/42922026/11/12 00:24:11 INFO Aborted multipart uploads count=0 kept=042932026/11/12 00:24:11 WARN Force mode enabled - objects will be deleted immediately without grace period42942026/11/12 00:24:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042952026/11/12 00:24:11 INFO Received complete upload request method=POST path=/42962026/11/12 00:24:12 INFO Vacuumed table table=pending_closures42972026/11/12 00:24:12 INFO Completed upload id=34 objects=3 duration=932.208µs42982026/11/12 00:24:12 INFO Vacuumed table table=pending_objects42992026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads43002026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=043012026/11/12 00:24:12 INFO Vacuumed table table=closures43022026/11/12 00:24:12 INFO Vacuumed table table=objects43032026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period43042026/11/12 00:24:12 INFO Received push request method=POST path=/43052026/11/12 00:24:12 INFO Received complete push request method=POST path=/43062026/11/12 00:24:12 INFO Completed push id=25 objects=10 duration=1.372875ms43072026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=043082026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period43092026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043102026/11/12 00:24:12 INFO Vacuumed table table=pending_closures43112026/11/12 00:24:12 INFO Vacuumed table table=pending_objects43122026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads43132026/11/12 00:24:12 INFO Vacuumed table table=closures43142026/11/12 00:24:12 INFO Vacuumed table table=objects43152026/11/12 00:24:12 INFO Received uploads request method=POST path=/43162026/11/12 00:24:12 INFO Received push request method=POST path=/43172026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=043182026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period43192026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=043202026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period43212026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043222026/11/12 00:24:12 INFO Vacuumed table table=pending_closures43232026/11/12 00:24:12 INFO Vacuumed table table=pending_objects43242026/11/12 00:24:12 INFO Received uploads request method=POST path=/43252026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads43262026/11/12 00:24:12 INFO Vacuumed table table=closures43272026/11/12 00:24:12 INFO Received push request method=POST path=/43282026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43292026/11/12 00:24:12 INFO Received push request method=POST path=/43302026/11/12 00:24:12 INFO Vacuumed table table=objects43312026/11/12 00:24:12 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43322026/11/12 00:24:12 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."43332026/11/12 00:24:12 INFO Completed upload id=29 objects=5 duration=679.584µs43342026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43352026/11/12 00:24:12 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."43362026/11/12 00:24:12 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."43372026/11/12 00:24:12 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43382026/11/12 00:24:12 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."43392026/11/12 00:24:12 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43402026/11/12 00:24:12 INFO Completed upload id=11 objects=7 duration=551.083µs43412026/11/12 00:24:12 INFO Received push request method=POST path=/43422026/11/12 00:24:12 INFO Received push request method=POST path=/43432026/11/12 00:24:12 INFO Received complete push request method=POST path=/43442026/11/12 00:24:12 INFO Received uploads request method=POST path=/43452026/11/12 00:24:12 INFO Completed push id=11 objects=9 duration=1.564541ms43462026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43472026/11/12 00:24:12 INFO Completed upload id=30 objects=2 duration=2.346459ms43482026/11/12 00:24:12 INFO Received uploads request method=POST path=/43492026/11/12 00:24:12 INFO Received push request method=POST path=/43502026/11/12 00:24:12 INFO Received complete push request method=POST path=/43512026/11/12 00:24:12 INFO Completed push id=21 objects=3 duration=2.513833ms43522026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43532026/11/12 00:24:12 INFO Completed upload id=21 objects=3 duration=724.75µs43542026/11/12 00:24:12 INFO Received complete push request method=POST path=/43552026/11/12 00:24:12 INFO Completed push id=31 objects=20 duration=690.833µs43562026/11/12 00:24:12 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43572026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43582026/11/12 00:24:12 INFO Completed upload id=23 objects=7 duration=1.078417ms43592026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43602026/11/12 00:24:12 INFO Received push request method=POST path=/43612026/11/12 00:24:12 INFO Completed upload id=14 objects=16 duration=973.459µs43622026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=043632026/11/12 00:24:12 INFO Received complete push request method=POST path=/43642026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period43652026/11/12 00:24:12 INFO Completed push id=15 objects=24 duration=665.25µs43662026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043672026/11/12 00:24:12 INFO Vacuumed table table=pending_closures43682026/11/12 00:24:12 INFO Vacuumed table table=pending_objects43692026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads43702026/11/12 00:24:12 INFO Vacuumed table table=closures43712026/11/12 00:24:12 INFO Vacuumed table table=objects43722026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43732026/11/12 00:24:12 INFO Completed upload id=18 objects=11 duration=712.625µs43742026/11/12 00:24:12 INFO Received uploads request method=POST path=/43752026/11/12 00:24:12 INFO Received push request method=POST path=/43762026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43772026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43782026/11/12 00:24:12 INFO Completed upload id=32 objects=5 duration=9.469958ms43792026/11/12 00:24:12 INFO Received uploads request method=POST path=/43802026/11/12 00:24:12 INFO Completed upload id=19 objects=2 duration=5.415333ms43812026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43822026/11/12 00:24:12 INFO Received uploads request method=POST path=/43832026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=043842026/11/12 00:24:12 INFO Received push request method=POST path=/43852026/11/12 00:24:12 INFO Received uploads request method=POST path=/43862026/11/12 00:24:12 INFO Completed upload id=26 objects=5 duration=3.773042ms43872026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period43882026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43892026/11/12 00:24:12 INFO Received complete upload request method=POST path=/43902026/11/12 00:24:12 INFO Completed upload id=24 objects=7 duration=3.163792ms43912026/11/12 00:24:12 INFO Completed upload id=33 objects=3 duration=1.650584ms43922026/11/12 00:24:12 INFO Received complete push request method=POST path=/43932026/11/12 00:24:12 INFO Received push request method=POST path=/43942026/11/12 00:24:12 INFO Completed push id=27 objects=3 duration=1.564708ms43952026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=043962026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period43972026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043982026/11/12 00:24:12 INFO Vacuumed table table=pending_closures43992026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44002026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44012026/11/12 00:24:12 INFO Vacuumed table table=closures44022026/11/12 00:24:12 INFO Vacuumed table table=objects44032026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=044042026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044052026/11/12 00:24:12 INFO Vacuumed table table=pending_closures44062026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=044072026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44082026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44092026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period44102026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=044112026/11/12 00:24:12 INFO Vacuumed table table=closures44122026/11/12 00:24:12 INFO Vacuumed table table=pending_closures44132026/11/12 00:24:12 INFO Vacuumed table table=objects44142026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044152026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44162026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44172026/11/12 00:24:12 INFO Vacuumed table table=pending_closures44182026/11/12 00:24:12 INFO Vacuumed table table=closures44192026/11/12 00:24:12 INFO Vacuumed table table=objects44202026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44212026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44222026/11/12 00:24:12 INFO Vacuumed table table=closures44232026/11/12 00:24:12 INFO Vacuumed table table=objects44242026/11/12 00:24:12 INFO Received complete upload request method=POST path=/44252026/11/12 00:24:12 INFO Completed upload id=28 objects=5 duration=671.583µs44262026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=044272026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period44282026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044292026/11/12 00:24:12 INFO Vacuumed table table=pending_closures44302026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44312026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44322026/11/12 00:24:12 INFO Vacuumed table table=closures44332026/11/12 00:24:12 INFO Vacuumed table table=objects44342026/11/12 00:24:12 INFO Received complete push request method=POST path=/44352026/11/12 00:24:12 INFO Received push request method=POST path=/44362026/11/12 00:24:12 INFO Completed push id=29 objects=17 duration=774.167µs44372026/11/12 00:24:12 INFO Received complete push request method=POST path=/44382026/11/12 00:24:12 INFO Completed push id=26 objects=3 duration=3.878333ms44392026/11/12 00:24:12 INFO Received push request method=POST path=/44402026/11/12 00:24:12 INFO Received complete push request method=POST path=/44412026/11/12 00:24:12 INFO Received push request method=POST path=/44422026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=044432026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=044442026/11/12 00:24:12 INFO Completed push id=27 objects=25 duration=4.113875ms44452026/11/12 00:24:12 INFO Received complete push request method=POST path=/44462026/11/12 00:24:12 INFO Received complete upload request method=POST path=/44472026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period44482026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period44492026/11/12 00:24:12 INFO Completed push id=17 objects=11 duration=2.145417ms44502026/11/12 00:24:12 INFO Completed upload id=13 objects=8 duration=3.691208ms44512026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044522026/11/12 00:24:12 INFO Vacuumed table table=pending_closures44532026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044542026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44552026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44562026/11/12 00:24:12 INFO Vacuumed table table=pending_closures44572026/11/12 00:24:12 INFO Vacuumed table table=closures44582026/11/12 00:24:12 INFO Vacuumed table table=objects44592026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44602026/11/12 00:24:12 INFO Received uploads request method=POST path=/44612026/11/12 00:24:12 INFO Received complete upload request method=POST path=/44622026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=044632026/11/12 00:24:12 INFO Received complete push request method=POST path=/44642026/11/12 00:24:12 INFO Aborted multipart uploads count=0 kept=044652026/11/12 00:24:12 INFO Received push request method=POST path=/44662026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44672026/11/12 00:24:12 INFO Completed upload id=28 objects=8 duration=827.042µs44682026/11/12 00:24:12 INFO Vacuumed table table=closures44692026/11/12 00:24:12 INFO Completed push id=30 objects=4 duration=2.723458ms44702026/11/12 00:24:12 WARN Force mode enabled - objects will be deleted immediately without grace period44712026/11/12 00:24:12 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044722026/11/12 00:24:12 INFO Received push request method=POST path=/44732026/11/12 00:24:12 INFO Vacuumed table table=objects44742026/11/12 00:24:12 INFO Received uploads request method=POST path=/44752026/11/12 00:24:12 INFO Vacuumed table table=pending_closures44762026/11/12 00:24:12 INFO Received uploads request method=POST path=/44772026/11/12 00:24:12 INFO Received push request method=POST path=/44782026/11/12 00:24:12 INFO Vacuumed table table=pending_objects44792026/11/12 00:24:12 INFO Vacuumed table table=multipart_uploads44802026/11/12 00:24:12 INFO Vacuumed table table=closures44812026/11/12 00:24:12 INFO Vacuumed table table=objects44822026/11/12 00:24:12 INFO Received complete upload request method=POST path=/44832026/11/12 00:24:12 INFO Received push request method=POST path=/44842026/11/12 00:24:12 INFO Completed upload id=29 objects=10 duration=8.271375ms44852026/11/12 00:24:12 INFO Received complete push request method=POST path=/44862026/11/12 00:24:12 INFO Completed push id=22 objects=3 duration=4.346292ms44872026/11/12 00:24:12 INFO Received push request method=POST path=/44882026/11/12 00:24:12 INFO Received complete push request method=POST path=/44892026/11/12 00:24:12 INFO Received complete upload request method=POST path=/44902026/11/12 00:24:12 INFO Completed push id=24 objects=5 duration=1.398084ms44912026/11/12 00:24:12 INFO Completed upload id=30 objects=20 duration=1.195542ms44922026/11/12 00:24:12 INFO Received complete upload request method=POST path=/44932026/11/12 00:24:13 INFO Completed upload id=26 objects=6 duration=5.113458ms44942026/11/12 00:24:13 INFO Received push request method=POST path=/44952026/11/12 00:24:13 INFO Received complete push request method=POST path=/44962026/11/12 00:24:13 INFO Completed push id=31 objects=11 duration=2.955084ms44972026/11/12 00:24:13 INFO Received uploads request method=POST path=/44982026/11/12 00:24:13 INFO Received uploads request method=POST path=/44992026/11/12 00:24:13 INFO Received complete push request method=POST path=/45002026/11/12 00:24:13 INFO Completed push id=32 objects=7 duration=789.041µs45012026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45022026/11/12 00:24:13 INFO Received uploads request method=POST path=/45032026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45042026/11/12 00:24:13 INFO Completed upload id=23 objects=16 duration=9.805625ms45052026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45062026/11/12 00:24:13 INFO Completed upload id=32 objects=2 duration=890.334µs45072026/11/12 00:24:13 INFO Completed upload id=29 objects=5 duration=1.247542ms45082026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45092026/11/12 00:24:13 INFO Completed upload id=19 objects=25 duration=4.09125ms45102026/11/12 00:24:13 INFO Received complete push request method=POST path=/45112026/11/12 00:24:13 INFO Completed push id=26 objects=4 duration=977.875µs45122026/11/12 00:24:13 INFO Received push request method=POST path=/45132026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=045142026/11/12 00:24:13 WARN Force mode enabled - objects will be deleted immediately without grace period45152026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045162026/11/12 00:24:13 INFO Vacuumed table table=pending_closures45172026/11/12 00:24:13 INFO Vacuumed table table=pending_objects45182026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads45192026/11/12 00:24:13 INFO Vacuumed table table=closures45202026/11/12 00:24:13 INFO Vacuumed table table=objects45212026/11/12 00:24:13 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."45222026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=045232026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45242026/11/12 00:24:13 INFO Received complete push request method=POST path=/45252026/11/12 00:24:13 INFO Received complete push request method=POST path=/45262026/11/12 00:24:13 INFO Completed push id=33 objects=3 duration=1.703458ms45272026/11/12 00:24:13 INFO Completed push id=24 objects=3 duration=1.731792ms45282026/11/12 00:24:13 INFO Completed upload id=28 objects=13 duration=1.75625ms45292026/11/12 00:24:13 WARN Force mode enabled - objects will be deleted immediately without grace period45302026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=045312026/11/12 00:24:13 INFO Received uploads request method=POST path=/45322026/11/12 00:24:13 INFO Received uploads request method=POST path=/45332026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45342026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=045352026/11/12 00:24:13 WARN Force mode enabled - objects will be deleted immediately without grace period45362026/11/12 00:24:13 INFO Completed upload id=34 objects=3 duration=691.25µs45372026/11/12 00:24:13 INFO Received push request method=POST path=/45382026/11/12 00:24:13 INFO Vacuumed table table=pending_closures45392026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045402026/11/12 00:24:13 INFO Vacuumed table table=pending_objects45412026/11/12 00:24:13 INFO Vacuumed table table=pending_closures45422026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads45432026/11/12 00:24:13 INFO Vacuumed table table=closures45442026/11/12 00:24:13 INFO Vacuumed table table=pending_objects45452026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads45462026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=045472026/11/12 00:24:13 INFO Vacuumed table table=objects45482026/11/12 00:24:13 INFO Vacuumed table table=closures45492026/11/12 00:24:13 INFO Vacuumed table table=objects45502026/11/12 00:24:13 WARN Force mode enabled - objects will be deleted immediately without grace period45512026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045522026/11/12 00:24:13 INFO Received complete push request method=POST path=/45532026/11/12 00:24:13 INFO Vacuumed table table=pending_closures45542026/11/12 00:24:13 INFO Completed push id=29 objects=3 duration=2.523542ms45552026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=045562026/11/12 00:24:13 INFO Received push request method=POST path=/45572026/11/12 00:24:13 INFO Received push request method=POST path=/45582026/11/12 00:24:13 WARN Force mode enabled - objects will be deleted immediately without grace period45592026/11/12 00:24:13 INFO Received complete push request method=POST path=/45602026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45612026/11/12 00:24:13 INFO Vacuumed table table=pending_objects45622026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045632026/11/12 00:24:13 INFO Completed push id=15 objects=4 duration=1.282833ms45642026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads45652026/11/12 00:24:13 INFO Completed upload id=31 objects=2 duration=6.077ms45662026/11/12 00:24:13 INFO Vacuumed table table=closures45672026/11/12 00:24:13 INFO Vacuumed table table=objects45682026/11/12 00:24:13 INFO Received complete push request method=POST path=/45692026/11/12 00:24:13 INFO Vacuumed table table=pending_closures45702026/11/12 00:24:13 INFO Received uploads request method=POST path=/45712026/11/12 00:24:13 INFO Received complete push request method=POST path=/45722026/11/12 00:24:13 INFO Completed push id=5 objects=28 duration=1.153375ms45732026/11/12 00:24:13 INFO Received uploads request method=POST path=/45742026/11/12 00:24:13 INFO Completed push id=21 objects=26 duration=9.815292ms45752026/11/12 00:24:13 INFO Vacuumed table table=pending_objects45762026/11/12 00:24:13 INFO Received push request method=POST path=/45772026/11/12 00:24:13 INFO Received complete push request method=POST path=/45782026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads45792026/11/12 00:24:13 INFO Vacuumed table table=closures45802026/11/12 00:24:13 INFO Completed push id=33 objects=6 duration=3.232875ms45812026/11/12 00:24:13 INFO Vacuumed table table=objects45822026/11/12 00:24:13 INFO Received complete push request method=POST path=/45832026/11/12 00:24:13 INFO Completed push id=10 objects=2 duration=2.187459ms45842026/11/12 00:24:13 INFO Received uploads request method=POST path=/45852026/11/12 00:24:13 INFO Received uploads request method=POST path=/45862026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=045872026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45882026/11/12 00:24:13 INFO Completed upload id=31 objects=3 duration=3.881875ms45892026/11/12 00:24:13 WARN Force mode enabled - objects will be deleted immediately without grace period45902026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045912026/11/12 00:24:13 INFO Vacuumed table table=pending_closures45922026/11/12 00:24:13 INFO Vacuumed table table=pending_objects45932026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads45942026/11/12 00:24:13 INFO Received complete upload request method=POST path=/45952026/11/12 00:24:13 INFO Vacuumed table table=closures45962026/11/12 00:24:13 INFO Completed upload id=22 objects=2 duration=6.252792ms45972026/11/12 00:24:13 INFO Vacuumed table table=objects45982026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=045992026/11/12 00:24:13 INFO Received push request method=POST path=/46002026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=046012026/11/12 00:24:13 INFO Aborted multipart uploads count=0 kept=046022026/11/12 00:24:13 INFO Received push request method=POST path=/46032026/11/12 00:24:13 WARN Force mode enabled - objects will be deleted immediately without grace period46042026/11/12 00:24:13 INFO Vacuumed table table=pending_closures46052026/11/12 00:24:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046062026/11/12 00:24:13 INFO Vacuumed table table=pending_objects46072026/11/12 00:24:13 INFO Vacuumed table table=pending_closures46082026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads46092026/11/12 00:24:13 INFO Vacuumed table table=pending_objects46102026/11/12 00:24:13 INFO Vacuumed table table=closures46112026/11/12 00:24:13 INFO Vacuumed table table=multipart_uploads46122026/11/12 00:24:13 INFO Vacuumed table table=objects46132026/11/12 00:24:13 INFO Vacuumed table table=closures46142026/11/12 00:24:13 INFO Received complete upload request method=POST path=/46152026/11/12 00:24:13 INFO Received complete upload request method=POST path=/46162026/11/12 00:24:13 INFO Vacuumed table table=objects46172026/11/12 00:24:13 INFO Completed upload id=17 objects=5 duration=420.583µs46182026/11/12 00:24:13 INFO Received uploads request method=POST path=/46192026/11/12 00:24:13 INFO Completed upload id=28 objects=17 duration=1.332542ms46202026/11/12 00:24:13 INFO Received complete push request method=POST path=/46212026/11/12 00:24:13 INFO Received uploads request method=POST path=/46222026/11/12 00:24:13 INFO Received complete push request method=POST path=/46232026/11/12 00:24:13 INFO Received complete upload request method=POST path=/46242026/11/12 00:24:13 INFO Completed push id=13 objects=15 duration=22.804334ms46252026/11/12 00:24:13 INFO Received complete push request method=POST path=/46262026/11/12 00:24:13 INFO Completed push id=16 objects=4 duration=1.755834ms46272026/11/12 00:24:13 INFO Completed upload id=18 objects=2 duration=1.295709ms46282026/11/12 00:24:13 INFO Completed push id=29 objects=2 duration=667.583µs46292026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46302026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=046312026/11/12 00:24:14 INFO Completed upload id=26 objects=2 duration=9.33675ms46322026/11/12 00:24:14 INFO Received complete push request method=POST path=/46332026/11/12 00:24:14 INFO Completed push id=14 objects=3 duration=2.19825ms46342026/11/12 00:24:14 INFO Received complete push request method=POST path=/46352026/11/12 00:24:14 INFO Received uploads request method=POST path=/46362026/11/12 00:24:14 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46372026/11/12 00:24:14 INFO Received push request method=POST path=/46382026/11/12 00:24:14 INFO Completed push id=31 objects=6 duration=1.181ms46392026/11/12 00:24:14 INFO Received complete push request method=POST path=/46402026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=046412026/11/12 00:24:14 INFO Completed push id=19 objects=3 duration=377.583µs46422026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046432026/11/12 00:24:14 INFO Vacuumed table table=pending_closures46442026/11/12 00:24:14 INFO Vacuumed table table=pending_objects46452026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads46462026/11/12 00:24:14 INFO Vacuumed table table=closures46472026/11/12 00:24:14 INFO Vacuumed table table=objects46482026/11/12 00:24:14 INFO Received complete push request method=POST path=/46492026/11/12 00:24:14 INFO Completed push id=32 objects=9 duration=583.083µs46502026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46512026/11/12 00:24:14 INFO Completed upload id=20 objects=5 duration=7.138875ms46522026/11/12 00:24:14 INFO Received push request method=POST path=/46532026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46542026/11/12 00:24:14 INFO Received uploads request method=POST path=/46552026/11/12 00:24:14 INFO Completed upload id=20 objects=4 duration=662.458µs46562026/11/12 00:24:14 INFO Received complete push request method=POST path=/46572026/11/12 00:24:14 INFO Completed push id=7 objects=15 duration=3.2335ms46582026/11/12 00:24:14 INFO Received push request method=POST path=/46592026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=046602026/11/12 00:24:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."46612026/11/12 00:24:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."46622026/11/12 00:24:14 INFO Received push request method=POST path=/46632026/11/12 00:24:14 INFO Received push request method=POST path=/46642026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46652026/11/12 00:24:14 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46662026/11/12 00:24:14 INFO Completed upload id=33 objects=9 duration=2.522125ms46672026/11/12 00:24:14 INFO Received complete push request method=POST path=/46682026/11/12 00:24:14 INFO Completed push id=20 objects=4 duration=7.598167ms46692026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46702026/11/12 00:24:14 INFO Completed upload id=13 objects=7 duration=1.384834ms46712026/11/12 00:24:14 INFO Received push request method=POST path=/46722026/11/12 00:24:14 INFO Received push request method=POST path=/46732026/11/12 00:24:14 INFO Received push request method=POST path=/46742026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=046752026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46762026/11/12 00:24:14 INFO Received complete push request method=POST path=/46772026/11/12 00:24:14 WARN Force mode enabled - objects will be deleted immediately without grace period46782026/11/12 00:24:14 INFO Completed upload id=20 objects=4 duration=3.858292ms46792026/11/12 00:24:14 INFO Completed push id=20 objects=3 duration=1.839ms46802026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046812026/11/12 00:24:14 INFO Vacuumed table table=pending_closures46822026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46832026/11/12 00:24:14 INFO Vacuumed table table=pending_objects46842026/11/12 00:24:14 INFO Completed upload id=23 objects=7 duration=487.167µs46852026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads46862026/11/12 00:24:14 INFO Vacuumed table table=closures46872026/11/12 00:24:14 INFO Vacuumed table table=objects46882026/11/12 00:24:14 INFO Received complete push request method=POST path=/46892026/11/12 00:24:14 INFO Completed push id=16 objects=5 duration=826.417µs46902026/11/12 00:24:14 INFO Received complete upload request method=POST path=/46912026/11/12 00:24:14 INFO Completed upload id=22 objects=8 duration=7.450833ms46922026/11/12 00:24:14 INFO Received uploads request method=POST path=/46932026/11/12 00:24:14 INFO Received push request method=POST path=/46942026/11/12 00:24:14 INFO Received complete push request method=POST path=/46952026/11/12 00:24:14 INFO Completed push id=23 objects=5 duration=430.917µs46962026/11/12 00:24:14 INFO Received complete push request method=POST path=/46972026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=046982026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=046992026/11/12 00:24:14 INFO Completed push id=28 objects=11 duration=1.535583ms47002026/11/12 00:24:14 WARN Force mode enabled - objects will be deleted immediately without grace period47012026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047022026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047032026/11/12 00:24:14 INFO Received complete upload request method=POST path=/47042026/11/12 00:24:14 INFO Completed upload id=24 objects=8 duration=901.125µs47052026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47062026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47072026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47082026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47092026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47102026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47112026/11/12 00:24:14 INFO Vacuumed table table=closures47122026/11/12 00:24:14 INFO Vacuumed table table=closures47132026/11/12 00:24:14 INFO Vacuumed table table=objects47142026/11/12 00:24:14 INFO Vacuumed table table=objects47152026/11/12 00:24:14 INFO Received complete upload request method=POST path=/47162026/11/12 00:24:14 INFO Completed upload id=28 objects=5 duration=2.437042ms47172026/11/12 00:24:14 INFO Received push request method=POST path=/47182026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=047192026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=047202026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047212026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47222026/11/12 00:24:14 INFO Received push request method=POST path=/47232026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47242026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047252026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47262026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47272026/11/12 00:24:14 INFO Vacuumed table table=closures47282026/11/12 00:24:14 INFO Received push request method=POST path=/47292026/11/12 00:24:14 INFO Vacuumed table table=objects47302026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47312026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47322026/11/12 00:24:14 INFO Vacuumed table table=closures47332026/11/12 00:24:14 INFO Received uploads request method=POST path=/47342026/11/12 00:24:14 INFO Received complete upload request method=POST path=/47352026/11/12 00:24:14 INFO Vacuumed table table=objects47362026/11/12 00:24:14 INFO Completed upload id=22 objects=6 duration=1.00125ms47372026/11/12 00:24:14 INFO Received complete upload request method=POST path=/47382026/11/12 00:24:14 INFO Completed upload id=30 objects=5 duration=1.55625ms47392026/11/12 00:24:14 INFO Received push request method=POST path=/47402026/11/12 00:24:14 INFO Received uploads request method=POST path=/47412026/11/12 00:24:14 INFO Received uploads request method=POST path=/47422026/11/12 00:24:14 INFO Received push request method=POST path=/47432026/11/12 00:24:14 INFO Received complete upload request method=POST path=/47442026/11/12 00:24:14 INFO Completed upload id=24 objects=18 duration=2.192791ms47452026/11/12 00:24:14 INFO Received push request method=POST path=/47462026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=047472026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047482026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47492026/11/12 00:24:14 INFO Received complete push request method=POST path=/47502026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47512026/11/12 00:24:14 INFO Completed push id=31 objects=3 duration=8.978708ms47522026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47532026/11/12 00:24:14 INFO Vacuumed table table=closures47542026/11/12 00:24:14 INFO Received uploads request method=POST path=/47552026/11/12 00:24:14 INFO Vacuumed table table=objects47562026/11/12 00:24:14 INFO Received complete push request method=POST path=/47572026/11/12 00:24:14 INFO Completed push id=25 objects=2 duration=1.336875ms47582026/11/12 00:24:14 INFO Received complete push request method=POST path=/47592026/11/12 00:24:14 INFO Received push request method=POST path=/47602026/11/12 00:24:14 INFO Completed push id=29 objects=24 duration=1.398834ms47612026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=047622026/11/12 00:24:14 WARN Force mode enabled - objects will be deleted immediately without grace period47632026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=047642026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047652026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047662026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47672026/11/12 00:24:14 INFO Received complete push request method=POST path=/47682026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47692026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47702026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47712026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47722026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47732026/11/12 00:24:14 INFO Vacuumed table table=closures47742026/11/12 00:24:14 INFO Completed push id=27 objects=4 duration=1.17425ms47752026/11/12 00:24:14 INFO Vacuumed table table=closures47762026/11/12 00:24:14 INFO Vacuumed table table=objects47772026/11/12 00:24:14 INFO Vacuumed table table=objects47782026/11/12 00:24:14 INFO Aborted multipart uploads count=0 kept=047792026/11/12 00:24:14 WARN Force mode enabled - objects will be deleted immediately without grace period47802026/11/12 00:24:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047812026/11/12 00:24:14 INFO Received complete push request method=POST path=/47822026/11/12 00:24:14 INFO Vacuumed table table=pending_closures47832026/11/12 00:24:14 INFO Completed push id=32 objects=6 duration=1.351542ms47842026/11/12 00:24:14 INFO Vacuumed table table=pending_objects47852026/11/12 00:24:14 INFO Vacuumed table table=multipart_uploads47862026/11/12 00:24:14 INFO Received complete push request method=POST path=/47872026/11/12 00:24:14 INFO Vacuumed table table=closures47882026/11/12 00:24:14 INFO Completed push id=19 objects=14 duration=1.053ms47892026/11/12 00:24:14 INFO Vacuumed table table=objects47902026/11/12 00:24:14 INFO Received push request method=POST path=/47912026/11/12 00:24:14 INFO Received complete upload request method=POST path=/47922026/11/12 00:24:14 INFO Completed upload id=33 objects=7 duration=527.291µs47932026/11/12 00:24:15 INFO Received push request method=POST path=/47942026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=047952026/11/12 00:24:15 WARN Force mode enabled - objects will be deleted immediately without grace period47962026/11/12 00:24:15 INFO Received complete upload request method=POST path=/47972026/11/12 00:24:15 INFO Received push request method=POST path=/47982026/11/12 00:24:15 INFO Received push request method=POST path=/47992026/11/12 00:24:15 INFO Completed upload id=34 objects=3 duration=674.292µs48002026/11/12 00:24:15 INFO Received uploads request method=POST path=/48012026/11/12 00:24:15 INFO Received complete push request method=POST path=/48022026/11/12 00:24:15 INFO Received push request method=POST path=/48032026/11/12 00:24:15 INFO Received complete push request method=POST path=/48042026/11/12 00:24:15 INFO Completed push id=35 objects=11 duration=1.052667ms48052026/11/12 00:24:15 INFO Completed push id=31 objects=14 duration=5.584541ms48062026/11/12 00:24:15 INFO Received uploads request method=POST path=/48072026/11/12 00:24:15 INFO Received complete push request method=POST path=/48082026/11/12 00:24:15 INFO Completed push id=28 objects=26 duration=1.324209ms48092026/11/12 00:24:15 INFO Received uploads request method=POST path=/48102026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=048112026/11/12 00:24:15 INFO Received uploads request method=POST path=/48122026/11/12 00:24:15 WARN Force mode enabled - objects will be deleted immediately without grace period48132026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048142026/11/12 00:24:15 INFO Vacuumed table table=pending_closures48152026/11/12 00:24:15 INFO Vacuumed table table=pending_objects48162026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads48172026/11/12 00:24:15 INFO Vacuumed table table=closures48182026/11/12 00:24:15 INFO Vacuumed table table=objects48192026/11/12 00:24:15 INFO Received complete push request method=POST path=/48202026/11/12 00:24:15 INFO Received uploads request method=POST path=/48212026/11/12 00:24:15 INFO Received push request method=POST path=/48222026/11/12 00:24:15 INFO Completed push id=30 objects=3 duration=674.25µs48232026/11/12 00:24:15 INFO Received uploads request method=POST path=/48242026/11/12 00:24:15 INFO Received complete push request method=POST path=/48252026/11/12 00:24:15 INFO Completed push id=32 objects=19 duration=4.960334ms48262026/11/12 00:24:15 INFO Received uploads request method=POST path=/48272026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=048282026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=048292026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048302026/11/12 00:24:15 INFO Received push request method=POST path=/48312026/11/12 00:24:15 INFO Vacuumed table table=pending_closures48322026/11/12 00:24:15 WARN Force mode enabled - objects will be deleted immediately without grace period48332026/11/12 00:24:15 INFO Vacuumed table table=pending_objects48342026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048352026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads48362026/11/12 00:24:15 INFO Vacuumed table table=pending_closures48372026/11/12 00:24:15 INFO Vacuumed table table=closures48382026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=048392026/11/12 00:24:15 INFO Vacuumed table table=objects48402026/11/12 00:24:15 INFO Vacuumed table table=pending_closures48412026/11/12 00:24:15 INFO Vacuumed table table=pending_objects48422026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads48432026/11/12 00:24:15 INFO Vacuumed table table=pending_objects48442026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads48452026/11/12 00:24:15 INFO Vacuumed table table=closures48462026/11/12 00:24:15 INFO Vacuumed table table=closures48472026/11/12 00:24:15 INFO Vacuumed table table=objects48482026/11/12 00:24:15 INFO Vacuumed table table=objects48492026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=048502026/11/12 00:24:15 WARN Force mode enabled - objects will be deleted immediately without grace period48512026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048522026/11/12 00:24:15 INFO Vacuumed table table=pending_closures48532026/11/12 00:24:15 INFO Vacuumed table table=pending_objects48542026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads48552026/11/12 00:24:15 INFO Vacuumed table table=closures48562026/11/12 00:24:15 INFO Received push request method=POST path=/48572026/11/12 00:24:15 INFO Vacuumed table table=objects48582026/11/12 00:24:15 INFO Received complete upload request method=POST path=/48592026/11/12 00:24:15 INFO Completed upload id=19 objects=2 duration=1.595875ms48602026/11/12 00:24:15 INFO Received uploads request method=POST path=/48612026/11/12 00:24:15 INFO Received complete push request method=POST path=/48622026/11/12 00:24:15 INFO Completed push id=7 objects=7 duration=12.763ms48632026/11/12 00:24:15 INFO Received uploads request method=POST path=/48642026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=048652026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048662026/11/12 00:24:15 INFO Vacuumed table table=pending_closures48672026/11/12 00:24:15 INFO Received complete push request method=POST path=/48682026/11/12 00:24:15 INFO Vacuumed table table=pending_objects48692026/11/12 00:24:15 INFO Completed push id=12 objects=6 duration=5.763333ms48702026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads48712026/11/12 00:24:15 INFO Received complete push request method=POST path=/48722026/11/12 00:24:15 INFO Vacuumed table table=closures48732026/11/12 00:24:15 INFO Completed push id=33 objects=7 duration=3.070875ms48742026/11/12 00:24:15 INFO Vacuumed table table=objects48752026/11/12 00:24:15 INFO Received complete upload request method=POST path=/48762026/11/12 00:24:15 INFO Completed upload id=26 objects=3 duration=958.208µs48772026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=048782026/11/12 00:24:15 WARN Force mode enabled - objects will be deleted immediately without grace period48792026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048802026/11/12 00:24:15 INFO Vacuumed table table=pending_closures48812026/11/12 00:24:15 INFO Vacuumed table table=pending_objects48822026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads48832026/11/12 00:24:15 INFO Vacuumed table table=closures48842026/11/12 00:24:15 INFO Vacuumed table table=objects48852026/11/12 00:24:15 INFO Received complete push request method=POST path=/48862026/11/12 00:24:15 INFO Completed push id=16 objects=7 duration=2.0175ms48872026/11/12 00:24:15 INFO Received complete push request method=POST path=/48882026/11/12 00:24:15 INFO Completed push id=23 objects=6 duration=764.25µs48892026/11/12 00:24:15 INFO Received complete push request method=POST path=/48902026/11/12 00:24:15 INFO Completed push id=24 objects=6 duration=670.333µs48912026/11/12 00:24:15 INFO Received complete push request method=POST path=/48922026/11/12 00:24:15 INFO Completed push id=13 objects=22 duration=1.254417ms48932026/11/12 00:24:15 INFO Received complete push request method=POST path=/48942026/11/12 00:24:15 INFO Completed push id=21 objects=6 duration=495.416µs48952026/11/12 00:24:15 INFO Received complete upload request method=POST path=/48962026/11/12 00:24:15 INFO Completed upload id=22 objects=7 duration=416.25µs48972026/11/12 00:24:15 INFO Received complete push request method=POST path=/48982026/11/12 00:24:15 INFO Completed push id=24 objects=4 duration=704.208µs48992026/11/12 00:24:15 INFO Received complete push request method=POST path=/49002026/11/12 00:24:15 INFO Completed push id=25 objects=9 duration=632.458µs49012026/11/12 00:24:15 INFO Received complete push request method=POST path=/49022026/11/12 00:24:15 INFO Completed push id=25 objects=7 duration=726.458µs49032026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49042026/11/12 00:24:15 INFO Completed upload id=26 objects=3 duration=476.25µs49052026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49062026/11/12 00:24:15 INFO Completed upload id=27 objects=7 duration=456.5µs49072026/11/12 00:24:15 INFO Received complete push request method=POST path=/49082026/11/12 00:24:15 INFO Completed push id=9 objects=9 duration=2.190292ms49092026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49102026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49112026/11/12 00:24:15 INFO Completed upload id=28 objects=3 duration=499.333µs49122026/11/12 00:24:15 INFO Completed upload id=13 objects=14 duration=2.370166ms49132026/11/12 00:24:15 INFO Aborted multipart uploads count=0 kept=049142026/11/12 00:24:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049152026/11/12 00:24:15 INFO Vacuumed table table=pending_closures49162026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49172026/11/12 00:24:15 INFO Vacuumed table table=pending_objects49182026/11/12 00:24:15 INFO Vacuumed table table=multipart_uploads49192026/11/12 00:24:15 INFO Completed upload id=21 objects=8 duration=2.31475ms49202026/11/12 00:24:15 INFO Vacuumed table table=closures49212026/11/12 00:24:15 INFO Vacuumed table table=objects49222026/11/12 00:24:15 INFO Received complete push request method=POST path=/49232026/11/12 00:24:15 INFO Completed push id=29 objects=23 duration=689.917µs49242026/11/12 00:24:15 INFO Received complete push request method=POST path=/49252026/11/12 00:24:15 INFO Completed push id=22 objects=6 duration=730.25µs49262026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49272026/11/12 00:24:15 INFO Completed upload id=26 objects=24 duration=880.291µs49282026/11/12 00:24:15 INFO Received complete push request method=POST path=/49292026/11/12 00:24:15 INFO Received complete push request method=POST path=/49302026/11/12 00:24:15 INFO Completed push id=30 objects=6 duration=608.333µs49312026/11/12 00:24:15 INFO Completed push id=24 objects=8 duration=572.708µs49322026/11/12 00:24:15 INFO Received complete push request method=POST path=/49332026/11/12 00:24:15 INFO Completed push id=31 objects=6 duration=2.742375ms49342026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49352026/11/12 00:24:15 INFO Completed upload id=25 objects=9 duration=525.375µs49362026/11/12 00:24:15 INFO Received uploads request method=POST path=/49372026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49382026/11/12 00:24:15 INFO Completed upload id=27 objects=3 duration=933.5µs49392026/11/12 00:24:15 INFO Received complete push request method=POST path=/49402026/11/12 00:24:15 INFO Completed push id=33 objects=3 duration=1.217209ms49412026/11/12 00:24:15 INFO Received complete upload request method=POST path=/49422026/11/12 00:24:15 INFO Completed upload id=28 objects=6 duration=1.87875ms49432026/11/12 00:24:15 INFO Received complete push request method=POST path=/49442026/11/12 00:24:15 INFO Received complete push request method=POST path=/49452026/11/12 00:24:15 INFO Completed push id=34 objects=6 duration=793.583µs49462026/11/12 00:24:15 INFO Completed push id=30 objects=4 duration=612.083µs49472026/11/12 00:24:16 INFO Received complete push request method=POST path=/49482026/11/12 00:24:16 INFO Completed push id=35 objects=8 duration=515.458µs49492026/11/12 00:24:16 INFO Received complete push request method=POST path=/49502026/11/12 00:24:16 INFO Completed push id=31 objects=7 duration=398.083µs49512026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49522026/11/12 00:24:16 INFO Completed upload id=36 objects=9 duration=442.625µs49532026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49542026/11/12 00:24:16 INFO Completed upload id=32 objects=5 duration=475.042µs49552026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49562026/11/12 00:24:16 INFO Completed upload id=27 objects=26 duration=3.476875ms49572026/11/12 00:24:16 INFO Aborted multipart uploads count=0 kept=049582026/11/12 00:24:16 WARN Force mode enabled - objects will be deleted immediately without grace period49592026/11/12 00:24:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049602026/11/12 00:24:16 INFO Vacuumed table table=pending_closures49612026/11/12 00:24:16 INFO Vacuumed table table=pending_objects49622026/11/12 00:24:16 INFO Vacuumed table table=multipart_uploads49632026/11/12 00:24:16 INFO Received complete push request method=POST path=/49642026/11/12 00:24:16 INFO Vacuumed table table=closures49652026/11/12 00:24:16 INFO Completed push id=28 objects=6 duration=1.081459ms49662026/11/12 00:24:16 INFO Vacuumed table table=objects49672026/11/12 00:24:16 INFO Received complete push request method=POST path=/49682026/11/12 00:24:16 INFO Completed push id=29 objects=6 duration=735.334µs49692026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49702026/11/12 00:24:16 INFO Completed upload id=30 objects=8 duration=727.125µs49712026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49722026/11/12 00:24:16 INFO Completed upload id=34 objects=24 duration=579.958µs49732026/11/12 00:24:16 INFO Aborted multipart uploads count=0 kept=049742026/11/12 00:24:16 WARN Force mode enabled - objects will be deleted immediately without grace period49752026/11/12 00:24:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049762026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49772026/11/12 00:24:16 INFO Completed upload id=31 objects=6 duration=546.459µs49782026/11/12 00:24:16 INFO Received complete push request method=POST path=/49792026/11/12 00:24:16 INFO Vacuumed table table=pending_closures49802026/11/12 00:24:16 INFO Completed push id=1 objects=9 duration=721.5µs49812026/11/12 00:24:16 INFO Vacuumed table table=pending_objects49822026/11/12 00:24:16 INFO Vacuumed table table=multipart_uploads49832026/11/12 00:24:16 INFO Vacuumed table table=closures49842026/11/12 00:24:16 INFO Vacuumed table table=objects49852026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49862026/11/12 00:24:16 INFO Completed upload id=32 objects=4 duration=501µs49872026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49882026/11/12 00:24:16 INFO Completed upload id=33 objects=10 duration=575.875µs49892026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49902026/11/12 00:24:16 INFO Completed upload id=34 objects=23 duration=481.167µs49912026/11/12 00:24:16 INFO Received complete push request method=POST path=/49922026/11/12 00:24:16 INFO Completed push id=35 objects=6 duration=409.209µs49932026/11/12 00:24:16 INFO Aborted multipart uploads count=0 kept=049942026/11/12 00:24:16 WARN Force mode enabled - objects will be deleted immediately without grace period49952026/11/12 00:24:16 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049962026/11/12 00:24:16 INFO Received complete upload request method=POST path=/49972026/11/12 00:24:16 INFO Vacuumed table table=pending_closures49982026/11/12 00:24:16 INFO Completed upload id=2 objects=7 duration=630µs49992026/11/12 00:24:16 INFO Vacuumed table table=pending_objects50002026/11/12 00:24:16 INFO Vacuumed table table=multipart_uploads50012026/11/12 00:24:16 INFO Vacuumed table table=closures50022026/11/12 00:24:16 INFO Vacuumed table table=objects50032026/11/12 00:24:16 INFO Received complete upload request method=POST path=/50042026/11/12 00:24:16 INFO Completed upload id=8 objects=6 duration=382.459µs50052026/11/12 00:24:16 INFO Received complete upload request method=POST path=/50062026/11/12 00:24:16 INFO Completed upload id=12 objects=9 duration=638µs50072026/11/12 00:24:17 INFO Received complete upload request method=POST path=/50082026/11/12 00:24:17 INFO Completed upload id=18 objects=22 duration=641.042µs50092026/11/12 00:24:17 INFO Received complete upload request method=POST path=/50102026/11/12 00:24:17 INFO Completed upload id=21 objects=8 duration=487.458µs50112026/11/12 00:24:17 INFO Received complete upload request method=POST path=/50122026/11/12 00:24:17 INFO Completed upload id=22 objects=7 duration=627.25µs50132026/11/12 00:24:17 INFO Received complete push request method=POST path=/50142026/11/12 00:24:17 INFO Completed push id=24 objects=5 duration=520.459µs50152026/11/12 00:24:17 INFO Received complete upload request method=POST path=/50162026/11/12 00:24:17 INFO Completed upload id=25 objects=8 duration=551.375µs50172026/11/12 00:24:17 INFO Received complete push request method=POST path=/50182026/11/12 00:24:17 INFO Completed push id=26 objects=4 duration=262.875µs50192026/11/12 00:24:17 INFO Received complete push request method=POST path=/50202026/11/12 00:24:17 INFO Completed push id=30 objects=2 duration=221.5µs50212026/11/12 00:24:17 INFO Received complete push request method=POST path=/50222026/11/12 00:24:17 INFO Completed push id=32 objects=7 duration=680.542µs50232026/11/12 00:24:17 INFO Received complete upload request method=POST path=/50242026/11/12 00:24:17 INFO Completed upload id=33 objects=6 duration=471.834µs50252026/11/12 00:24:17 INFO Received complete upload request method=POST path=/50262026/11/12 00:24:17 INFO Completed upload id=34 objects=3 duration=417.333µs50272026/11/12 00:24:17 INFO Aborted multipart uploads count=0 kept=050282026/11/12 00:24:17 WARN Force mode enabled - objects will be deleted immediately without grace period50292026/11/12 00:24:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050302026/11/12 00:24:17 INFO Vacuumed table table=pending_closures50312026/11/12 00:24:17 INFO Vacuumed table table=pending_objects50322026/11/12 00:24:17 INFO Vacuumed table table=multipart_uploads50332026/11/12 00:24:17 INFO Vacuumed table table=closures50342026/11/12 00:24:17 INFO Vacuumed table table=objects5035--- PASS: FuzzPushGC (0.00s)5036 --- PASS: FuzzPushGC/seed#15 (12.71s)5037 --- PASS: FuzzPushGC/seed#3 (14.03s)5038 --- PASS: FuzzPushGC/seed#11 (14.48s)5039 --- PASS: FuzzPushGC/seed#13 (15.07s)5040 --- PASS: FuzzPushGC/seed#6 (15.53s)5041 --- PASS: FuzzPushGC/seed#1 (15.68s)5042 --- PASS: FuzzPushGC/seed#5 (16.43s)5043 --- PASS: FuzzPushGC/seed#14 (16.61s)5044 --- PASS: FuzzPushGC/seed#0 (17.37s)5045 --- PASS: FuzzPushGC/seed#2 (17.82s)5046 --- PASS: FuzzPushGC/seed#8 (18.06s)5047 --- PASS: FuzzPushGC/seed#7 (18.24s)5048 --- PASS: FuzzPushGC/seed#4 (18.93s)5049 --- PASS: FuzzPushGC/seed#9 (19.19s)5050 --- PASS: FuzzPushGC/seed#12 (19.42s)5051 --- PASS: FuzzPushGC/seed#10 (20.00s)5052PASS5053{"timestamp":"2026-11-12T00:24:17.195748Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:50637","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}50542026-11-12 00:24:17.415 UTC [68144] LOG: received smart shutdown request50552026-11-12 00:24:17.416 UTC [68144] LOG: background worker "logical replication launcher" (PID 68168) exited with exit code 150562026-11-12 00:24:18.997 UTC [68158] LOG: checkpoint complete: wrote 3300 buffers (20.1%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=31.967 s, sync=1.389 s, total=33.521 s; sync files=42365, longest=0.006 s, average=0.001 s; distance=533141 kB, estimate=533141 kB; lsn=0/278A6880, redo lsn=0/220325B050572026-11-12 00:24:19.001 UTC [68158] LOG: shutting down50582026-11-12 00:24:19.001 UTC [68158] LOG: checkpoint starting: shutdown immediate50592026-11-12 00:24:19.832 UTC [68158] LOG: checkpoint complete: wrote 8833 buffers (53.9%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.724 s, sync=0.099 s, total=0.832 s; sync files=2874, longest=0.001 s, average=0.001 s; distance=90576 kB, estimate=488885 kB; lsn=0/278A68F8, redo lsn=0/278A68F850602026-11-12 00:24:19.850 UTC [68144] LOG: database system is shut down5061Running OIDC tests...5062=== RUN TestAudienceForIssuer5063=== PAUSE TestAudienceForIssuer5064=== RUN TestGlobMatch5065=== PAUSE TestGlobMatch5066=== RUN TestValidateToken_ValidToken5067=== PAUSE TestValidateToken_ValidToken5068=== RUN TestValidateToken_WrongAudience5069=== PAUSE TestValidateToken_WrongAudience5070=== RUN TestValidateToken_Expired5071=== PAUSE TestValidateToken_Expired5072=== RUN TestValidateToken_BoundClaimsMismatch5073=== PAUSE TestValidateToken_BoundClaimsMismatch5074=== RUN TestValidateToken_BoundSubjectMismatch5075=== PAUSE TestValidateToken_BoundSubjectMismatch5076=== RUN TestValidateToken_MultipleProviders5077=== PAUSE TestValidateToken_MultipleProviders5078=== RUN TestValidateToken_NoMatchingProvider5079=== PAUSE TestValidateToken_NoMatchingProvider5080=== RUN TestValidateToken_KubernetesServiceAccount5081=== PAUSE TestValidateToken_KubernetesServiceAccount5082=== RUN TestNewValidator_KubernetesRequiresCA5083=== PAUSE TestNewValidator_KubernetesRequiresCA5084=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5085=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5086=== RUN TestPins_ReservedForMatchingRule5087=== PAUSE TestPins_ReservedForMatchingRule5088=== RUN TestPins_TopLevelShorthand5089=== PAUSE TestPins_TopLevelShorthand5090=== RUN TestPins_ConfigValidation5091=== PAUSE TestPins_ConfigValidation5092=== RUN TestScopes_LegacyProviderDefaultsToWrite5093=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5094=== RUN TestScopes_Rules5095=== PAUSE TestScopes_Rules5096=== RUN TestScopes_ConfigValidation5097=== PAUSE TestScopes_ConfigValidation5098=== CONT TestAudienceForIssuer5099--- PASS: TestAudienceForIssuer (0.00s)5100=== CONT TestScopes_LegacyProviderDefaultsToWrite5101=== CONT TestValidateToken_MultipleProviders5102=== CONT TestValidateToken_BoundSubjectMismatch5103=== CONT TestScopes_ConfigValidation5104=== CONT TestValidateToken_BoundClaimsMismatch5105=== CONT TestValidateToken_Expired5106=== CONT TestValidateToken_WrongAudience5107=== CONT TestValidateToken_ValidToken5108--- PASS: TestScopes_ConfigValidation (0.00s)5109=== CONT TestGlobMatch5110=== RUN TestGlobMatch/foo_foo5111=== PAUSE TestGlobMatch/foo_foo5112=== RUN TestGlobMatch/foo_bar5113=== PAUSE TestGlobMatch/foo_bar5114=== RUN TestGlobMatch/*_5115=== PAUSE TestGlobMatch/*_5116=== RUN TestGlobMatch/*_anything5117=== PAUSE TestGlobMatch/*_anything5118=== RUN TestGlobMatch/foo*_foo5119=== PAUSE TestGlobMatch/foo*_foo5120=== RUN TestGlobMatch/foo*_foobar5121=== PAUSE TestGlobMatch/foo*_foobar5122=== RUN TestGlobMatch/foo*_bar5123=== PAUSE TestGlobMatch/foo*_bar5124=== RUN TestGlobMatch/*bar_bar5125=== PAUSE TestGlobMatch/*bar_bar5126=== RUN TestGlobMatch/*bar_foobar5127=== PAUSE TestGlobMatch/*bar_foobar5128=== RUN TestGlobMatch/*bar_foo5129=== PAUSE TestGlobMatch/*bar_foo5130=== RUN TestGlobMatch/foo*bar_foobar5131=== PAUSE TestGlobMatch/foo*bar_foobar5132=== RUN TestGlobMatch/foo*bar_foo123bar5133=== PAUSE TestGlobMatch/foo*bar_foo123bar5134=== RUN TestGlobMatch/foo*bar_foobarbaz5135=== PAUSE TestGlobMatch/foo*bar_foobarbaz5136=== RUN TestGlobMatch/*/*_foo/bar5137=== PAUSE TestGlobMatch/*/*_foo/bar5138=== RUN TestGlobMatch/*/*_foo5139=== PAUSE TestGlobMatch/*/*_foo5140=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5141=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5142=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05143=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05144=== RUN TestGlobMatch/refs/*/main_refs/heads/main5145=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5146=== RUN TestGlobMatch/fo?_foo5147=== PAUSE TestGlobMatch/fo?_foo5148=== RUN TestGlobMatch/fo?_fo5149=== PAUSE TestGlobMatch/fo?_fo5150=== RUN TestGlobMatch/fo?_fooo5151=== PAUSE TestGlobMatch/fo?_fooo5152=== RUN TestGlobMatch/?oo_foo5153=== PAUSE TestGlobMatch/?oo_foo5154=== RUN TestGlobMatch/?oo_boo5155=== PAUSE TestGlobMatch/?oo_boo5156=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5157=== CONT TestScopes_Rules5158=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5159=== CONT TestPins_TopLevelShorthand5160=== CONT TestPins_ReservedForMatchingRule5161=== CONT TestPins_ConfigValidation5162=== CONT TestValidateToken_KubernetesServiceAccount5163=== CONT TestNewValidator_KubernetesRequiresCA5164=== CONT TestValidateToken_NoMatchingProvider5165=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5166=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5167=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5168=== CONT TestGlobMatch/foo_foo5169=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5170=== CONT TestGlobMatch/*bar_foobar5171=== CONT TestGlobMatch/*/*_foo5172=== CONT TestGlobMatch/*/*_foo/bar5173=== CONT TestGlobMatch/foo*bar_foobarbaz5174=== CONT TestGlobMatch/foo*bar_foo123bar5175=== CONT TestGlobMatch/foo*bar_foobar5176=== CONT TestGlobMatch/*bar_foo5177=== CONT TestGlobMatch/?oo_boo5178--- PASS: TestPins_ConfigValidation (0.00s)5179=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5180=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5181=== CONT TestGlobMatch/foo*_foo5182=== CONT TestGlobMatch/*bar_bar5183=== CONT TestGlobMatch/foo*_bar5184=== CONT TestGlobMatch/foo*_foobar5185=== CONT TestGlobMatch/*_5186=== CONT TestGlobMatch/*_anything5187=== CONT TestGlobMatch/foo_bar5188=== CONT TestGlobMatch/fo?_fo5189=== CONT TestGlobMatch/?oo_foo5190=== CONT TestGlobMatch/fo?_fooo5191=== CONT TestGlobMatch/refs/*/main_refs/heads/main5192=== CONT TestGlobMatch/fo?_foo5193=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05194--- PASS: TestGlobMatch (0.00s)5195 --- PASS: TestGlobMatch/foo_foo (0.00s)5196 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5197 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5198 --- PASS: TestGlobMatch/*/*_foo (0.00s)5199 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5200 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5201 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5202 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5203 --- PASS: TestGlobMatch/*bar_foo (0.00s)5204 --- PASS: TestGlobMatch/?oo_boo (0.00s)5205 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5206 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5207 --- PASS: TestGlobMatch/foo*_foo (0.00s)5208 --- PASS: TestGlobMatch/*bar_bar (0.00s)5209 --- PASS: TestGlobMatch/foo*_bar (0.00s)5210 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5211 --- PASS: TestGlobMatch/*_ (0.00s)5212 --- PASS: TestGlobMatch/*_anything (0.00s)5213 --- PASS: TestGlobMatch/foo_bar (0.00s)5214 --- PASS: TestGlobMatch/fo?_fo (0.00s)5215 --- PASS: TestGlobMatch/?oo_foo (0.00s)5216 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5217 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5218 --- PASS: TestGlobMatch/fo?_foo (0.00s)5219 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)52202026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51607/oidc52212026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51605/oidc5222--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.10s)5223--- PASS: TestValidateToken_WrongAudience (0.10s)52242026/11/12 00:24:23 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:5160952252026/11/12 00:24:23 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352262026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51611/oidc52272026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51614/oidc5228--- PASS: TestPins_ReservedForMatchingRule (0.22s)5229--- PASS: TestValidateToken_Expired (0.23s)5230--- PASS: TestValidateToken_KubernetesServiceAccount (0.22s)52312026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51617/oidc5232--- PASS: TestPins_TopLevelShorthand (0.24s)52332026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51622/oidc52342026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51621/oidc52352026/11/12 00:24:23 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:51604/oidc52362026/11/12 00:24:23 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:51616/oidc5237--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.27s)52382026/11/12 00:24:23 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:51623/oidc5239--- PASS: TestValidateToken_ValidToken (0.27s)5240--- PASS: TestValidateToken_NoMatchingProvider (0.27s)5241--- PASS: TestValidateToken_MultipleProviders (0.29s)52422026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51631/oidc5243--- PASS: TestScopes_Rules (0.28s)5244--- PASS: TestValidateToken_BoundSubjectMismatch (0.29s)52452026/11/12 00:24:23 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:51633/oidc5246--- PASS: TestValidateToken_BoundClaimsMismatch (0.31s)52472026/11/12 00:24:23 http: TLS handshake error from 127.0.0.1:51636: remote error: tls: bad certificate5248--- PASS: TestNewValidator_KubernetesRequiresCA (0.34s)5249PASS5250Running hook tests...5251=== RUN TestSendPathsEmpty5252=== PAUSE TestSendPathsEmpty5253=== RUN TestQueueEnqueueAndFetch5254=== PAUSE TestQueueEnqueueAndFetch5255=== RUN TestQueueDeduplication5256=== PAUSE TestQueueDeduplication5257=== RUN TestQueueRemove5258=== PAUSE TestQueueRemove5259=== RUN TestQueueFetchBatchLimit5260=== PAUSE TestQueueFetchBatchLimit5261=== RUN TestQueueRetryMovesToBack5262=== PAUSE TestQueueRetryMovesToBack5263=== RUN TestQueueFetchRemoveLifecycle5264=== PAUSE TestQueueFetchRemoveLifecycle5265=== RUN TestQueueConcurrentWriters5266=== PAUSE TestQueueConcurrentWriters5267=== RUN TestQueueRemoveLargeClosure5268=== PAUSE TestQueueRemoveLargeClosure5269=== RUN TestServerClientIntegration5270--- PASS: TestServerClientIntegration (0.00s)5271=== RUN TestServerQueueError52722026/11/12 00:24:23 ERROR Failed to queue paths error="permission denied" count=15273--- PASS: TestServerQueueError (0.00s)5274=== RUN TestGetListenerSocketActivation5275 server_test.go:147: systemd socket activation does not exist on darwin5276--- SKIP: TestGetListenerSocketActivation (0.00s)5277=== RUN TestServerStalledClientDoesNotBlockShutdown5278=== PAUSE TestServerStalledClientDoesNotBlockShutdown5279=== RUN TestServerBacksOffOnAcceptErrors5280=== PAUSE TestServerBacksOffOnAcceptErrors5281=== RUN TestDrainIsolatesPoisonPath5282=== PAUSE TestDrainIsolatesPoisonPath5283=== RUN TestRunNotBlockedByPoisonHead5284=== PAUSE TestRunNotBlockedByPoisonHead5285=== RUN TestDrainGivesUpWhenServerDown5286=== PAUSE TestDrainGivesUpWhenServerDown5287=== RUN TestFailedPathPrunedByLaterClosure5288=== PAUSE TestFailedPathPrunedByLaterClosure5289=== RUN TestWorkerUploadsAndRemoves5290=== PAUSE TestWorkerUploadsAndRemoves5291=== RUN TestWorkerSkipsGCdPaths5292=== PAUSE TestWorkerSkipsGCdPaths5293=== RUN TestWorkerPrunesClosureDeps5294=== PAUSE TestWorkerPrunesClosureDeps5295=== RUN TestDrainTimeout5296=== PAUSE TestDrainTimeout5297=== RUN TestWorkerRemoveFailureIsNotProgress5298=== PAUSE TestWorkerRemoveFailureIsNotProgress5299=== RUN TestWorkerKeepsPathItCannotStat5300=== PAUSE TestWorkerKeepsPathItCannotStat5301=== CONT TestSendPathsEmpty5302=== CONT TestDrainIsolatesPoisonPath5303--- PASS: TestSendPathsEmpty (0.00s)5304=== CONT TestServerBacksOffOnAcceptErrors5305=== CONT TestServerStalledClientDoesNotBlockShutdown5306=== CONT TestQueueRemoveLargeClosure53072000/01/01 00:00:00 ERROR Accept failed error="too many open files"53082000/01/01 00:00:00 ERROR Accept failed error="too many open files"5309=== CONT TestQueueConcurrentWriters53102000/01/01 00:00:00 ERROR Accept failed error="too many open files"53112000/01/01 00:00:00 ERROR Accept failed error="too many open files"53122000/01/01 00:00:00 ERROR Accept failed error="too many open files"53132000/01/01 00:00:00 ERROR Accept failed error="too many open files"5314=== CONT TestQueueFetchRemoveLifecycle53152000/01/01 00:00:00 ERROR Accept failed error="too many open files"53162000/01/01 00:00:00 ERROR Accept failed error="too many open files"5317--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5318=== CONT TestWorkerSkipsGCdPaths5319=== CONT TestQueueRetryMovesToBack5320=== CONT TestWorkerKeepsPathItCannotStat5321=== CONT TestQueueFetchBatchLimit5322=== CONT TestWorkerRemoveFailureIsNotProgress5323=== CONT TestDrainTimeout5324=== CONT TestQueueRemove5325=== CONT TestWorkerPrunesClosureDeps5326=== CONT TestQueueDeduplication5327=== CONT TestFailedPathPrunedByLaterClosure5328=== CONT TestQueueEnqueueAndFetch5329=== CONT TestWorkerUploadsAndRemoves5330=== CONT TestRunNotBlockedByPoisonHead5331=== CONT TestDrainGivesUpWhenServerDown53322000/01/01 00:00:00 INFO Uploading batch count=253332000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253342000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453352026/11/12 00:24:23 INFO Upload queue status pending=253362026/11/12 00:24:23 INFO Uploading batch count=253372026/11/12 00:24:23 INFO Uploading batch count=453382026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=45339--- PASS: TestQueueRemove (0.02s)53402026/11/12 00:24:23 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestDrainIsolatesPoisonPath2239783410/002/bbb5341--- PASS: TestQueueRetryMovesToBack (0.02s)53422026/11/12 00:24:23 INFO Upload queue status pending=253432026/11/12 00:24:23 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerSkipsGCdPaths1313029704/002/nonexistent53442026/11/12 00:24:23 INFO Uploading batch count=153452026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=15346--- PASS: TestDrainTimeout (0.02s)53472026/11/12 00:24:23 INFO Uploading batch count=15348--- PASS: TestQueueFetchBatchLimit (0.02s)53492026/11/12 00:24:23 INFO Uploading batch count=153502026/11/12 00:24:23 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerKeepsPathItCannotStat1262568596/002/locked/aaa error="stat /private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerKeepsPathItCannotStat1262568596/002/locked/aaa: permission denied"53512026/11/12 00:24:23 INFO Uploading batch count=153522026/11/12 00:24:23 INFO Upload queue status pending=253532026/11/12 00:24:23 INFO Uploading batch count=153542026/11/12 00:24:23 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerKeepsPathItCannotStat1262568596/002/locked/aaa error="stat /private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerKeepsPathItCannotStat1262568596/002/locked/aaa: permission denied"53552026/11/12 00:24:23 INFO Uploading batch count=253562026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=253572026/11/12 00:24:23 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestDrainGivesUpWhenServerDown1576494305/002/a53582026/11/12 00:24:23 INFO Upload queue status pending=35359--- PASS: TestQueueEnqueueAndFetch (0.03s)53602026/11/12 00:24:23 INFO Uploading batch count=153612026/11/12 00:24:23 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerRemoveFailureIsNotProgress2700115395/002/nonexistent53622026/11/12 00:24:23 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153632026/11/12 00:24:23 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerRemoveFailureIsNotProgress2700115395/002/nonexistent53642026/11/12 00:24:23 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153652026/11/12 00:24:23 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerRemoveFailureIsNotProgress2700115395/002/nonexistent53662026/11/12 00:24:23 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153672026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=15368--- PASS: TestQueueDeduplication (0.04s)5369--- PASS: TestQueueFetchRemoveLifecycle (0.04s)53702026/11/12 00:24:23 INFO Uploading batch count=153712026/11/12 00:24:23 WARN Cannot stat store path, will retry later path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerKeepsPathItCannotStat1262568596/002/locked/aaa error="stat /private/var/tb/5/scratch/75b857c59f8a960d/build/TestWorkerKeepsPathItCannotStat1262568596/002/locked/aaa: permission denied"53722026/11/12 00:24:23 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestDrainGivesUpWhenServerDown1576494305/002/b53732026/11/12 00:24:23 ERROR Drain finished with paths left in queue remaining=153742026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=153752026/11/12 00:24:23 INFO Uploading batch count=15376--- PASS: TestFailedPathPrunedByLaterClosure (0.04s)53772026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=153782026/11/12 00:24:23 INFO Uploading batch count=253792026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=253802026/11/12 00:24:23 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestDrainGivesUpWhenServerDown1576494305/002/c53812026/11/12 00:24:23 ERROR Drain finished with paths left in queue remaining=153822026/11/12 00:24:23 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestDrainGivesUpWhenServerDown1576494305/002/d53832026/11/12 00:24:23 INFO Uploading batch count=153842026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=15385--- PASS: TestWorkerRemoveFailureIsNotProgress (0.04s)53862026/11/12 00:24:23 INFO Uploading batch count=253872026/11/12 00:24:23 ERROR Upload failed error="upload failed" count=253882026/11/12 00:24:23 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestDrainGivesUpWhenServerDown1576494305/002/e53892026/11/12 00:24:23 ERROR Drain finished with paths left in queue remaining=153902026/11/12 00:24:23 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/5/scratch/75b857c59f8a960d/build/TestDrainGivesUpWhenServerDown1576494305/002/f5391--- PASS: TestWorkerUploadsAndRemoves (0.04s)53922026/11/12 00:24:23 ERROR Drain finished with paths left in queue remaining=105393--- PASS: TestWorkerKeepsPathItCannotStat (0.04s)5394--- PASS: TestDrainIsolatesPoisonPath (0.05s)5395--- PASS: TestWorkerSkipsGCdPaths (0.05s)5396--- PASS: TestDrainGivesUpWhenServerDown (0.04s)5397--- PASS: TestWorkerPrunesClosureDeps (0.05s)5398--- PASS: TestQueueRemoveLargeClosure (0.09s)5399--- PASS: TestQueueConcurrentWriters (0.10s)54002026/11/12 00:24:23 ERROR Failed to decode request error="read unix /private/var/tb/5/scratch/75b857c59f8a960d/build/hook1465491313/test.sock->: i/o timeout"54012026/11/12 00:24:23 ERROR Failed to write response error="write unix /private/var/tb/5/scratch/75b857c59f8a960d/build/hook1465491313/test.sock->: i/o timeout"5402--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)54032026/11/12 00:24:24 INFO Uploading batch count=154042026/11/12 00:24:24 INFO Uploading batch count=154052026/11/12 00:24:24 INFO Uploading batch count=154062026/11/12 00:24:24 ERROR Upload failed error="upload failed" count=154072026/11/12 00:24:24 INFO Uploading batch count=154082026/11/12 00:24:24 ERROR Upload failed error="upload failed" count=154092026/11/12 00:24:24 INFO Uploading batch count=154102026/11/12 00:24:24 ERROR Upload failed error="upload failed" count=154112026/11/12 00:24:24 INFO Uploading batch count=154122026/11/12 00:24:24 ERROR Upload failed error="upload failed" count=154132026/11/12 00:24:24 ERROR Drain finished with paths left in queue remaining=15414--- PASS: TestRunNotBlockedByPoisonHead (1.05s)5415PASS5416Running signing tests...5417=== RUN TestExternalSigner5418=== PAUSE TestExternalSigner5419=== RUN TestExternalSignerWrongPublicKey5420=== PAUSE TestExternalSignerWrongPublicKey5421=== RUN TestExternalSignerErrors5422=== PAUSE TestExternalSignerErrors5423=== RUN TestExternalSignerRestartsAfterCrash5424=== PAUSE TestExternalSignerRestartsAfterCrash5425=== RUN TestExternalSignerProgramErrorKeepsProcess5426=== PAUSE TestExternalSignerProgramErrorKeepsProcess5427=== RUN TestExternalSignerClose5428=== PAUSE TestExternalSignerClose5429=== RUN TestExternalSignerRejectsInvalidPublicKey5430=== PAUSE TestExternalSignerRejectsInvalidPublicKey5431=== RUN TestExternalSignerCancellation5432=== PAUSE TestExternalSignerCancellation5433=== RUN TestGenerateFingerprint5434=== PAUSE TestGenerateFingerprint5435=== RUN TestParseSigningKey5436=== PAUSE TestParseSigningKey5437=== RUN TestSign5438=== PAUSE TestSign5439=== RUN TestSignBatch5440=== PAUSE TestSignBatch5441=== RUN TestNilKeyPublicKey5442=== PAUSE TestNilKeyPublicKey5443=== RUN TestSignErrors5444=== PAUSE TestSignErrors5445=== CONT TestExternalSigner5446=== CONT TestSignBatch5447=== CONT TestExternalSignerCancellation5448=== CONT TestGenerateFingerprint5449=== RUN TestGenerateFingerprint/basic_with_references5450=== CONT TestSignErrors5451=== CONT TestNilKeyPublicKey5452--- PASS: TestNilKeyPublicKey (0.00s)5453=== CONT TestSign5454=== CONT TestExternalSignerProgramErrorKeepsProcess5455=== CONT TestExternalSignerRejectsInvalidPublicKey5456=== RUN TestExternalSignerRejectsInvalidPublicKey/#005457=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005458=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5459=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5460=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5461=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5462=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5463=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5464=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5465=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5466=== CONT TestExternalSignerRejectsInvalidPublicKey/#005467=== CONT TestExternalSignerClose5468=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5469=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5470=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5471=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5472--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5473 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5474 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5475 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5476 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5477 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5478=== CONT TestExternalSignerErrors5479=== RUN TestExternalSignerErrors/invalid_JSON5480=== CONT TestExternalSignerRestartsAfterCrash5481=== CONT TestExternalSignerWrongPublicKey5482=== CONT TestParseSigningKey5483=== RUN TestParseSigningKey/valid_32-byte_key5484=== PAUSE TestGenerateFingerprint/basic_with_references5485=== PAUSE TestExternalSignerErrors/invalid_JSON5486=== RUN TestGenerateFingerprint/no_references5487=== PAUSE TestGenerateFingerprint/no_references5488=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5489=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5490=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5491=== RUN TestExternalSignerErrors/wrong_shape5492=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5493=== RUN TestGenerateFingerprint/invalid_nar_hash_length5494=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5495=== PAUSE TestParseSigningKey/valid_32-byte_key5496=== RUN TestGenerateFingerprint/invalid_store_path_prefix5497=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5498=== PAUSE TestExternalSignerErrors/wrong_shape5499=== RUN TestExternalSignerErrors/missing_signatures5500=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5501=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5502=== RUN TestGenerateFingerprint/invalid_reference_prefix5503=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5504=== PAUSE TestExternalSignerErrors/missing_signatures5505=== CONT TestGenerateFingerprint/no_references5506=== RUN TestExternalSignerErrors/wrong_signatures_shape5507=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5508=== RUN TestExternalSignerErrors/missing_signature_name5509=== PAUSE TestExternalSignerErrors/missing_signature_name5510=== RUN TestExternalSignerErrors/wrong_signature_length5511=== PAUSE TestExternalSignerErrors/wrong_signature_length5512=== RUN TestExternalSignerErrors/wrong_id5513=== PAUSE TestExternalSignerErrors/wrong_id5514=== RUN TestParseSigningKey/no_colon5515=== RUN TestExternalSignerErrors/unrequested_key5516=== PAUSE TestExternalSignerErrors/unrequested_key5517=== RUN TestExternalSignerErrors/null_signatures5518=== PAUSE TestParseSigningKey/no_colon5519=== CONT TestGenerateFingerprint/invalid_reference_prefix5520=== RUN TestParseSigningKey/empty_name5521=== PAUSE TestParseSigningKey/empty_name5522=== CONT TestGenerateFingerprint/invalid_store_path_prefix5523=== CONT TestGenerateFingerprint/invalid_nar_hash_length5524=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5525=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5526=== CONT TestGenerateFingerprint/basic_with_references5527--- PASS: TestGenerateFingerprint (0.00s)5528 --- PASS: TestGenerateFingerprint/no_references (0.00s)5529 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5530 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5531 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5532 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5533 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5534 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5535=== PAUSE TestExternalSignerErrors/null_signatures5536=== RUN TestExternalSignerErrors/invalid_signature_base645537=== RUN TestParseSigningKey/invalid_base645538=== PAUSE TestExternalSignerErrors/invalid_signature_base645539=== RUN TestExternalSignerErrors/program_error5540=== PAUSE TestParseSigningKey/invalid_base645541=== PAUSE TestExternalSignerErrors/program_error5542=== RUN TestParseSigningKey/wrong_length5543=== PAUSE TestParseSigningKey/wrong_length5544=== CONT TestParseSigningKey/valid_32-byte_key5545=== RUN TestExternalSignerErrors/exit_on_request5546=== CONT TestParseSigningKey/empty_name5547=== PAUSE TestExternalSignerErrors/exit_on_request5548=== RUN TestExternalSignerErrors/null5549=== PAUSE TestExternalSignerErrors/null5550=== CONT TestParseSigningKey/no_colon5551=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5552=== CONT TestParseSigningKey/invalid_base645553=== CONT TestParseSigningKey/wrong_length5554=== RUN TestExternalSignerErrors/too_few_signatures5555=== PAUSE TestExternalSignerErrors/too_few_signatures5556=== RUN TestExternalSignerErrors/wrong_signature_name5557=== PAUSE TestExternalSignerErrors/wrong_signature_name5558=== RUN TestExternalSignerErrors/exit_at_startup5559=== PAUSE TestExternalSignerErrors/exit_at_startup5560=== CONT TestExternalSignerErrors/invalid_JSON5561=== CONT TestExternalSignerErrors/wrong_signature_length5562--- PASS: TestParseSigningKey (0.00s)5563 --- PASS: TestParseSigningKey/empty_name (0.00s)5564 --- PASS: TestParseSigningKey/no_colon (0.00s)5565 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5566 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5567 --- PASS: TestParseSigningKey/wrong_length (0.00s)5568 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5569=== CONT TestExternalSignerErrors/wrong_signatures_shape5570=== RUN TestSignErrors/nil_key5571=== PAUSE TestSignErrors/nil_key5572=== RUN TestSignErrors/nil_narinfo5573=== PAUSE TestSignErrors/nil_narinfo5574=== RUN TestSignErrors/invalid_narinfo5575=== PAUSE TestSignErrors/invalid_narinfo5576=== CONT TestExternalSignerErrors/wrong_id5577--- PASS: TestSign (0.00s)5578=== CONT TestExternalSignerErrors/null_signatures5579--- PASS: TestSignBatch (0.00s)5580=== CONT TestExternalSignerErrors/exit_at_startup5581=== CONT TestExternalSignerErrors/exit_on_request5582=== CONT TestExternalSignerErrors/wrong_signature_name5583=== CONT TestExternalSignerErrors/program_error5584=== CONT TestExternalSignerErrors/too_few_signatures5585=== CONT TestExternalSignerErrors/invalid_signature_base645586=== CONT TestExternalSignerErrors/missing_signature_name5587=== CONT TestExternalSignerErrors/unrequested_key5588=== CONT TestExternalSignerErrors/missing_signatures5589=== CONT TestExternalSignerErrors/wrong_shape5590=== CONT TestExternalSignerErrors/null5591=== RUN TestSignErrors/canceled5592=== PAUSE TestSignErrors/canceled5593=== CONT TestSignErrors/nil_key5594=== CONT TestSignErrors/invalid_narinfo5595=== CONT TestSignErrors/nil_narinfo5596=== CONT TestSignErrors/canceled5597--- PASS: TestSignErrors (0.00s)5598 --- PASS: TestSignErrors/nil_key (0.00s)5599 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5600 --- PASS: TestSignErrors/nil_narinfo (0.00s)5601 --- PASS: TestSignErrors/canceled (0.00s)5602--- PASS: TestExternalSigner (0.12s)5603--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.19s)5604--- PASS: TestExternalSignerRestartsAfterCrash (0.30s)5605/private/var/tb/5/scratch/75b857c59f8a960d/build/TestExternalSignerRestartsAfterCrash3023533693/001/sign program: line 7: /private/var/tb/5/scratch/75b857c59f8a960d/build/TestExternalSignerRestartsAfterCrash3023533693/001/sign program.starts: No such file or directory5606--- PASS: TestExternalSignerCancellation (0.99s)5607--- PASS: TestExternalSignerClose (1.05s)5608--- PASS: TestExternalSignerWrongPublicKey (1.22s)5609--- PASS: TestExternalSignerErrors (0.00s)5610 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.36s)5611 --- PASS: TestExternalSignerErrors/null (0.45s)5612 --- PASS: TestExternalSignerErrors/too_few_signatures (0.53s)5613 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.62s)5614 --- PASS: TestExternalSignerErrors/unrequested_key (0.71s)5615 --- PASS: TestExternalSignerErrors/program_error (0.80s)5616 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.88s)5617 --- PASS: TestExternalSignerErrors/invalid_JSON (1.14s)5618 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (1.30s)5619 --- PASS: TestExternalSignerErrors/null_signatures (1.38s)5620 --- PASS: TestExternalSignerErrors/exit_at_startup (1.46s)5621 --- PASS: TestExternalSignerErrors/missing_signature_name (1.54s)5622 --- PASS: TestExternalSignerErrors/wrong_shape (1.62s)5623 --- PASS: TestExternalSignerErrors/wrong_id (1.70s)5624 --- PASS: TestExternalSignerErrors/exit_on_request (1.78s)5625 --- PASS: TestExternalSignerErrors/missing_signatures (1.87s)5626PASS5627Running OpenBao signer tests...5628=== RUN TestSign5629=== PAUSE TestSign5630=== RUN TestSignErrors5631=== PAUSE TestSignErrors5632=== RUN TestServe5633=== PAUSE TestServe5634=== RUN TestServeRejectsGarbage5635=== PAUSE TestServeRejectsGarbage5636=== CONT TestSign5637=== CONT TestServe5638=== CONT TestSignErrors5639=== RUN TestSignErrors/http_error5640=== CONT TestServeRejectsGarbage5641--- PASS: TestServeRejectsGarbage (0.00s)5642=== PAUSE TestSignErrors/http_error5643=== RUN TestSignErrors/invalid_json5644=== PAUSE TestSignErrors/invalid_json5645=== RUN TestSignErrors/no_prefix5646=== PAUSE TestSignErrors/no_prefix5647=== RUN TestSignErrors/invalid_base645648=== PAUSE TestSignErrors/invalid_base645649=== CONT TestSignErrors/http_error5650=== CONT TestSignErrors/invalid_json5651--- PASS: TestServe (0.00s)5652=== CONT TestSignErrors/no_prefix5653=== CONT TestSignErrors/invalid_base645654--- PASS: TestSign (0.00s)5655--- PASS: TestSignErrors (0.00s)5656 --- PASS: TestSignErrors/no_prefix (0.00s)5657 --- PASS: TestSignErrors/http_error (0.00s)5658 --- PASS: TestSignErrors/invalid_base64 (0.00s)5659 --- PASS: TestSignErrors/invalid_json (0.00s)5660PASS