niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #491
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestDoServerRequestAttachesToken114=== CONT TestStreamPushReportsEveryPath115=== CONT TestSetClientTLSErrors116--- PASS: TestStreamPushReportsEveryPath (0.00s)117=== CONT TestDoWithRetry_BodyReplayedViaGetBody118=== CONT TestScriptTokenNoExpiryRerunsEveryCall119=== CONT TestFileTokenEmpty120=== CONT TestFileTokenMissing121=== CONT TestFileTokenReadsAndCaches122=== CONT TestStreamPushReportsSkippedPaths123=== CONT TestSetClientTLSDoesNotMutateDefaultTransport124=== CONT TestStaticToken125=== CONT TestDoWithRetry_FinalResponseBodyReadable126=== CONT TestSetClientTLS127=== CONT TestScriptTokenCachesUntilRefresh128=== CONT TestStreamPushStopsWhenCanceled1292026/11/11 23:37:16 WARN Rate limiter enabled after throttle name=server-test rate=51302026/11/11 23:37:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:33801131=== CONT TestClientSignaturesByStorePath132=== CONT TestParsePathInfoJSONMultiplePaths133=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths134=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths135=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths136=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths137=== CONT TestUploadMultipart_ReturnsAfterDroppedBody138=== CONT TestGetStorePathHash139=== RUN TestGetStorePathHash/valid_store_path140=== CONT TestParsePathInfoJSON141=== CONT TestRegisterUploadedObject_OutlivesPush142=== RUN TestParsePathInfoJSON/Nix_format143=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer1442026/11/11 23:37:16 WARN Rate limiter backed off name=server-test rate=51452026/11/11 23:37:16 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:33801146=== CONT TestShellSplitErrors147=== CONT TestStreamPushRequestLine148=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess1492026/11/11 23:37:16 WARN Rate limiter enabled after throttle name=server-test rate=5150=== CONT TestScriptTokenEmptyToken151=== CONT TestShellSplit1522026/11/11 23:37:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:354491532026/11/11 23:37:16 WARN Rate limiter enabled after throttle name=server-test rate=5154=== CONT TestRateLimiterFeedback155--- PASS: TestFileTokenEmpty (0.00s)156--- PASS: TestStreamPushReportsSkippedPaths (0.00s)157--- PASS: TestFileTokenMissing (0.00s)158=== CONT TestRetryTimeoutOutlastsMaxRetries159=== CONT TestResolveStorePath1602026/11/11 23:37:16 WARN Rate limiter backed off name=server-test rate=5161=== CONT TestPathInfoCACompatibility1622026/11/11 23:37:16 ERROR Upload failed error=boom count=1163=== RUN TestPathInfoCACompatibility/null_ca_field164=== PAUSE TestPathInfoCACompatibility/null_ca_field165=== CONT TestStreamPushGivesUpOnDeadServer166=== RUN TestSetClientTLSErrors/missing_cert_file167=== CONT TestScriptTokenEmptyCommand168=== CONT TestStreamPushReportsSignatures169=== CONT TestScriptTokenScriptFails170=== PAUSE TestGetStorePathHash/valid_store_path171=== PAUSE TestParsePathInfoJSON/Nix_format172=== CONT TestPathInfoHashCompatibility173=== RUN TestRateLimiterFeedback/429_enables_limiter174=== CONT TestConvertHashToNix321752026/11/11 23:37:16 ERROR Upload failed error="connection refused" count=20176--- PASS: TestStaticToken (0.00s)1772026/11/11 23:37:16 ERROR Server seems unavailable, giving up on batch untried=17178=== PAUSE TestRateLimiterFeedback/429_enables_limiter179=== RUN TestSetClientTLS/rejects_connection_without_client_cert1802026/11/11 23:37:16 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:38611181=== RUN TestConvertHashToNix32/SRI_format_to_Nix32182=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix321832026/11/11 23:37:16 ERROR Upload failed error=boom count=1184--- PASS: TestFileTokenReadsAndCaches (0.00s)185=== CONT TestDumpPathMatchesNix186=== RUN TestPathInfoCACompatibility/old_string_format_-_text187=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text188=== PAUSE TestSetClientTLSErrors/missing_cert_file189=== RUN TestSetClientTLSErrors/missing_key_file190=== CONT TestStreamPushIsolatesFailures191=== PAUSE TestSetClientTLSErrors/missing_key_file192=== RUN TestSetClientTLSErrors/missing_ca_file193=== CONT TestDumpPathSingleFile194=== CONT TestEncodeNixBase32195=== RUN TestGetStorePathHash/basename_without_hyphen_should_error196=== CONT TestEncodeNixBase32WithRealHash1972026/11/11 23:37:16 ERROR Upload failed error="bad path" count=3198=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun199=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error200=== RUN TestParsePathInfoJSON/Lix_format201=== PAUSE TestParsePathInfoJSON/Lix_format202=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert203=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA204=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA205=== RUN TestSetClientTLS/preserves_debug_logging_transport206=== RUN TestRateLimiterFeedback/503_enables_limiter207=== PAUSE TestRateLimiterFeedback/503_enables_limiter208=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter209=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter210=== CONT TestStreamPushBatchesUnderLoad211=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false212=== RUN TestConvertHashToNix32/already_Nix32_format213=== CONT TestPartSizeForNAR214=== RUN TestPartSizeForNAR/zero_stays_at_minimum215=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum216--- PASS: TestDoServerRequestAttachesToken (0.01s)217=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive2182026/11/11 23:37:16 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:38611219=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive220=== CONT TestDumpPathWriterError221=== PAUSE TestSetClientTLSErrors/missing_ca_file222=== RUN TestPathInfoCACompatibility/new_structured_format_-_text223=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)224=== RUN TestEncodeNixBase32/test_string_hash225=== RUN TestSetClientTLSErrors/invalid_ca_file226=== PAUSE TestSetClientTLSErrors/invalid_ca_file227=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error228=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error2292026/11/11 23:37:16 INFO Garbage collection started230=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error231=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error232=== RUN TestParsePathInfoJSON/empty_input233=== PAUSE TestParsePathInfoJSON/empty_input234=== RUN TestParsePathInfoJSON/whitespace_only235=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica236=== PAUSE TestParsePathInfoJSON/whitespace_only237=== RUN TestParsePathInfoJSON/invalid_JSON238=== PAUSE TestParsePathInfoJSON/invalid_JSON239=== CONT TestRegisterUploadedObjectReusesConnections240=== CONT TestFilterOversizedClosures241=== RUN TestFilterOversizedClosures/no_limit_keeps_everything242=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything243=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped244=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped245=== RUN TestFilterOversizedClosures/all_closures_skipped246=== PAUSE TestFilterOversizedClosures/all_closures_skipped247=== PAUSE TestSetClientTLS/preserves_debug_logging_transport248=== CONT TestCaseHackSuffix249=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter250=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths251=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false252=== PAUSE TestConvertHashToNix32/already_Nix32_format253=== RUN TestPartSizeForNAR/small_stays_at_minimum254--- PASS: TestClientSignaturesByStorePath (0.00s)255=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text256=== PAUSE TestEncodeNixBase32/test_string_hash257=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)258=== CONT TestUploadMultipart_PartsInParallel259=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry260=== CONT TestScriptTokenBadJSON261--- PASS: TestStreamPushStopsWhenCanceled (0.00s)262--- PASS: TestShellSplitErrors (0.00s)263--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)264=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths265--- PASS: TestShellSplit (0.00s)266--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)267--- PASS: TestScriptTokenEmptyCommand (0.00s)268--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)269--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)270--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)271--- PASS: TestStreamPushReportsSignatures (0.00s)272=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter273--- PASS: TestResolveStorePath (0.00s)274--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)275--- PASS: TestEncodeNixBase32WithRealHash (0.00s)276--- PASS: TestStreamPushIsolatesFailures (0.00s)277--- PASS: TestScriptTokenEmptyToken (0.00s)278--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)279--- PASS: TestScriptTokenScriptFails (0.00s)280--- PASS: TestScriptTokenCachesUntilRefresh (0.01s)281=== CONT TestSetClientTLSErrors/missing_ca_file282=== CONT TestSetClientTLSErrors/missing_key_file283=== CONT TestGetStorePathHash/valid_store_path284=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error285=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error286=== CONT TestGetStorePathHash/basename_without_hyphen_should_error287--- PASS: TestGetStorePathHash (0.01s)288 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)289 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)290 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)291 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)292=== CONT TestParsePathInfoJSON/Nix_format293=== CONT TestParsePathInfoJSON/whitespace_only294=== CONT TestParsePathInfoJSON/empty_input295=== CONT TestParsePathInfoJSON/Lix_format296=== CONT TestParsePathInfoJSON/invalid_JSON297--- PASS: TestParsePathInfoJSON (0.01s)298 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)299 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)300 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)301 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)302 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)303=== CONT TestFilterOversizedClosures/no_limit_keeps_everything304=== CONT TestFilterOversizedClosures/all_closures_skipped3052026/11/11 23:37:16 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=50306=== CONT TestSetClientTLS/rejects_connection_without_client_cert307=== RUN TestConvertHashToNix32/invalid_format308=== PAUSE TestConvertHashToNix32/invalid_format3092026/11/11 23:37:16 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:38611310=== RUN TestEncodeNixBase32/empty_input311=== PAUSE TestEncodeNixBase32/empty_input312=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3132026/11/11 23:37:16 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000314=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method315=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method316=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true317=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter318--- PASS: TestFilterOversizedClosures (0.00s)319 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)320 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)321 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)322=== CONT TestSetClientTLSErrors/missing_cert_file323--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)324 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)325 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.01s)326=== CONT TestSetClientTLS/preserves_debug_logging_transport327=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA328=== CONT TestRateLimiterFeedback/429_enables_limiter329=== CONT TestRateLimiterFeedback/503_enables_limiter330=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true3312026/11/11 23:37:16 WARN Rate limiter enabled after throttle name=server-test rate=5332=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter3332026/11/11 23:37:16 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:45143334=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon3352026/11/11 23:37:16 WARN Rate limiter backed off name=server-test rate=5336=== CONT TestSetClientTLSErrors/invalid_ca_file337=== CONT TestConvertHashToNix32/SRI_format_to_Nix32338--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)339--- PASS: TestScriptTokenBadJSON (0.01s)340=== CONT TestEncodeNixBase32/test_string_hash341=== CONT TestConvertHashToNix32/already_Nix32_format342=== CONT TestConvertHashToNix32/invalid_format343=== CONT TestPathInfoCACompatibility/null_ca_field344=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method3452026/11/11 23:37:17 WARN Rate limiter enabled after throttle name=server-test rate=5346=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive3472026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:43081348=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true349=== PAUSE TestPartSizeForNAR/small_stays_at_minimum350=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum351=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum352=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts353=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts354=== RUN TestPartSizeForNAR/1_TiB355=== PAUSE TestPartSizeForNAR/1_TiB356=== RUN TestPartSizeForNAR/5_TiB_S3_max_object357=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object3582026/11/11 23:37:17 WARN Rate limiter backed off name=server-test rate=5359=== CONT TestPathInfoCACompatibility/old_string_format_-_text3602026/11/11 23:37:17 WARN Rate limiter enabled after throttle name=s3-test rate=5361=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3622026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:37069363--- PASS: TestConvertHashToNix32 (0.01s)364 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)365 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)366 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)367=== CONT TestEncodeNixBase32/empty_input368=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false3692026/11/11 23:37:17 INFO Garbage collection started370=== RUN TestPartSizeForNAR/capped_at_5_GiB371--- PASS: TestRateLimiterFeedback (0.01s)372 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.03s)373 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.03s)374 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.03s)375 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)376=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon377=== PAUSE TestPartSizeForNAR/capped_at_5_GiB378--- PASS: TestPathInfoCACompatibility (0.01s)379 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)380 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)381 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)382 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)383 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)384--- PASS: TestStreamPushBatchesUnderLoad (0.01s)385 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)386 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)387--- PASS: TestEncodeNixBase32 (0.01s)388 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)389 --- PASS: TestEncodeNixBase32/empty_input (0.00s)390--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.04s)391=== CONT TestPartSizeForNAR/zero_stays_at_minimum392=== CONT TestPartSizeForNAR/capped_at_5_GiB393=== CONT TestPartSizeForNAR/5_TiB_S3_max_object3942026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:38611395=== CONT TestPartSizeForNAR/1_TiB396=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI397=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts398=== CONT TestPartSizeForNAR/small_stays_at_minimum399=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum400--- PASS: TestSetClientTLSErrors (0.01s)401 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)402 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)403 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)404 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.01s)4052026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:386114062026/11/11 23:37:17 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0407--- PASS: TestDumpPathSingleFile (0.05s)408=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI409=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512410--- PASS: TestPartSizeForNAR (0.05s)411 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)412 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)413 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)414 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)415 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)416 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)417 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)418=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512419=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)420=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon4212026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:38611422=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI4232026/11/11 23:37:17 WARN Garbage collection finished on another replica. Its statistics are not available here424=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512425--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.05s)426--- PASS: TestPathInfoHashCompatibility (0.05s)427 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)428 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)429 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)430 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)4312026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:386114322026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:386114332026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:386114342026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:38611435--- PASS: TestStreamPushRequestLine (0.06s)4362026/11/11 23:37:17 http: TLS handshake error from 127.0.0.1:36562: remote error: tls: bad certificate4372026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:38611438--- PASS: TestSetClientTLS (0.01s)439 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)440 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.03s)441 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.05s)4422026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:386114432026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:386114442026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:386114452026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:386114462026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:386114472026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:386114482026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:386114492026/11/11 23:37:17 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:38611450--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.07s)451--- PASS: TestRegisterUploadedObjectReusesConnections (0.07s)452--- PASS: TestCaseHackSuffix (0.08s)453--- PASS: TestDumpPathWriterError (0.08s)4542026/11/11 23:37:17 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:44869/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:44869/api/uploads/complete4552026/11/11 23:37:17 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4562026/11/11 23:37:17 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:44869/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:44869/api/uploads/complete4572026/11/11 23:37:17 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4582026/11/11 23:37:17 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:44869/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:44869/api/uploads/complete4592026/11/11 23:37:17 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"460--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)461--- PASS: TestDumpPathMatchesNix (0.10s)462--- PASS: TestUploadMultipart_PartsInParallel (0.65s)463--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "nixbld".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /build/postgres3243578602/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /build/postgres3243578602/data -l logfile start492493/build/postgres3243578602:5432 - no response4942026-11-11 23:37:18.766 UTC [129] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit4952026-11-11 23:37:18.767 UTC [129] LOG: listening on Unix socket "/build/postgres3243578602/.s.PGSQL.5432"4962026-11-11 23:37:18.772 UTC [136] LOG: database system was shut down at 2026-11-11 23:37:18 UTC4972026-11-11 23:37:18.775 UTC [129] LOG: database system is ready to accept connections498/build/postgres3243578602:5432 - accepting connections499{"timestamp":"2026-11-11T23:37:19.06862152Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"644dbd75-001a-4d8b-8599-f589ba43a1e9","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(388)"}500=== RUN TestAbortSucceeded501=== PAUSE TestAbortSucceeded502=== RUN TestService_AuthMiddleware503=== PAUSE TestService_AuthMiddleware504=== RUN TestService_AuthMiddleware_MTLSProxyHeader505=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader506=== RUN TestService_AuthMiddleware_MTLSBoundSubjects507=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects508=== RUN TestService_ReadAuthMiddleware509=== PAUSE TestService_ReadAuthMiddleware510=== RUN TestService_AuthMiddleware_OIDC511=== PAUSE TestService_AuthMiddleware_OIDC512=== RUN TestService_RequireScope_OIDC513=== PAUSE TestService_RequireScope_OIDC514=== RUN TestService_ReadScope_PublicByDefault515=== PAUSE TestService_ReadScope_PublicByDefault516=== RUN TestCacheConfigHandler517=== PAUSE TestCacheConfigHandler518=== RUN TestCacheStatsHandler519=== PAUSE TestCacheStatsHandler520=== RUN TestClientCADerivations521=== PAUSE TestClientCADerivations522=== RUN TestClientErrorHandling523=== PAUSE TestClientErrorHandling524=== RUN TestClientIntegration525=== PAUSE TestClientIntegration526=== RUN TestClientMultipleUploads527=== PAUSE TestClientMultipleUploads528=== RUN TestClientWithDependencies529=== PAUSE TestClientWithDependencies530=== RUN TestClientSharedPathCommittedMidPush531=== PAUSE TestClientSharedPathCommittedMidPush532=== RUN TestPinProtectsFromGC533=== PAUSE TestPinProtectsFromGC534=== RUN TestClientPushesUseOnePush535=== PAUSE TestClientPushesUseOnePush536=== RUN TestClientFallsBackToClosures537=== PAUSE TestClientFallsBackToClosures538=== RUN TestDeleteClosuresRacingPins539=== PAUSE TestDeleteClosuresRacingPins540=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock541=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock542=== RUN TestResolveDBConnectionString543=== PAUSE TestResolveDBConnectionString544=== RUN TestConnectWaitsForAPeerMigration545=== PAUSE TestConnectWaitsForAPeerMigration546=== RUN TestLeadElectsOneAndHandsOver547=== PAUSE TestLeadElectsOneAndHandsOver548=== RUN TestLeadIncumbentWinsAfterRestart5492026/11/11 23:37:19 INFO lead: acquired remote=192.0.2.1:1234 generation=15502026/11/11 23:37:19 INFO lead: released remote=192.0.2.1:1234 generation=15512026/11/11 23:37:19 INFO lead: acquired remote=192.0.2.1:1234 generation=25522026/11/11 23:37:21 INFO lead: released remote=192.0.2.1:1234 generation=2553--- PASS: TestLeadIncumbentWinsAfterRestart (1.90s)554=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent555=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent556=== RUN TestLeadIncumbentAnnouncesAtOnce557=== PAUSE TestLeadIncumbentAnnouncesAtOnce558=== RUN TestLeadGenerationGrowsWithEveryGrant559=== PAUSE TestLeadGenerationGrowsWithEveryGrant560=== RUN TestLeadEndsOnShutdown561=== PAUSE TestLeadEndsOnShutdown562=== RUN TestLeadEndsWhenItsConnectionHangs5632026/11/11 23:37:21 INFO lead: acquired remote=192.0.2.1:1234 generation=15642026-11-11 23:37:22.583 UTC [576] FATAL: terminating connection due to administrator command5652026/11/11 23:37:22 INFO lead: acquired remote=192.0.2.1:1234 generation=25662026/11/11 23:37:23 WARN lead: lock connection lost error="timeout: context deadline exceeded"5672026/11/11 23:37:23 INFO lead: released remote=192.0.2.1:1234 generation=15682026/11/11 23:37:24 INFO lead: released remote=192.0.2.1:1234 generation=2569--- PASS: TestLeadEndsWhenItsConnectionHangs (3.11s)570=== RUN TestGCAdvisoryLockBlocksConcurrentRun571=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun572=== RUN TestGCBugBareHashReferences573=== PAUSE TestGCBugBareHashReferences574=== RUN TestGCMetrics575=== PAUSE TestGCMetrics576=== RUN TestPushDedupSurvivesConcurrentGC577=== PAUSE TestPushDedupSurvivesConcurrentGC578=== RUN TestDeduplicatedObjectsRecordedAsPending579=== PAUSE TestDeduplicatedObjectsRecordedAsPending580=== RUN TestGCSweepSkipsPendingObjects581=== PAUSE TestGCSweepSkipsPendingObjects582=== RUN TestTombstonedObjectOfferedWithoutWaiting583=== PAUSE TestTombstonedObjectOfferedWithoutWaiting584=== RUN TestGCSweepDeliversEachKeyOnce585=== PAUSE TestGCSweepDeliversEachKeyOnce586=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection587=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection588=== RUN TestForceGCDuringPushOffersSweptObject589=== PAUSE TestForceGCDuringPushOffersSweptObject590=== RUN TestFailedS3DeleteKeepsTombstone591=== PAUSE TestFailedS3DeleteKeepsTombstone592=== RUN TestSweepSkipsRowsLockedByPush593=== PAUSE TestSweepSkipsRowsLockedByPush594=== RUN TestRegisterCompletedObjectNeedsPendingRow595=== PAUSE TestRegisterCompletedObjectNeedsPendingRow596=== RUN TestGetExistingObjectsLocksRows597=== PAUSE TestGetExistingObjectsLocksRows598=== RUN TestSweepRechecksPendingAfterLocking599=== PAUSE TestSweepRechecksPendingAfterLocking600=== RUN TestPushWaitsForSweepHoldingRow601=== PAUSE TestPushWaitsForSweepHoldingRow602=== RUN TestCommitRacingPendingCleanupKeepsObjects603=== PAUSE TestCommitRacingPendingCleanupKeepsObjects604=== RUN TestGCEndsOnShutdown605=== PAUSE TestGCEndsOnShutdown606=== RUN TestGCTaskStore_StartNew607=== PAUSE TestGCTaskStore_StartNew608=== RUN TestGCTaskStore_DeduplicateSameParams609=== PAUSE TestGCTaskStore_DeduplicateSameParams610=== RUN TestGCTaskStore_ConflictDifferentParams611=== PAUSE TestGCTaskStore_ConflictDifferentParams612=== RUN TestGCTaskStore_GetEmpty613=== PAUSE TestGCTaskStore_GetEmpty614=== RUN TestGCTaskStore_GetReturnsLatest615=== PAUSE TestGCTaskStore_GetReturnsLatest616=== RUN TestGCTaskStore_CompletedAllowsNewTask617=== PAUSE TestGCTaskStore_CompletedAllowsNewTask618=== RUN TestGCTaskStore_PhaseUpdates619=== PAUSE TestGCTaskStore_PhaseUpdates620=== RUN TestGCTaskStore_Fail621=== PAUSE TestGCTaskStore_Fail622=== RUN TestGracefulShutdownDrainsInflight623=== PAUSE TestGracefulShutdownDrainsInflight624=== RUN TestService_healthCheckHandler625=== PAUSE TestService_healthCheckHandler626=== RUN TestService_readinessHandler627=== PAUSE TestService_readinessHandler628=== RUN TestGenerateLandingPage629=== PAUSE TestGenerateLandingPage630=== RUN TestEnsureAbortUploadsRule631=== PAUSE TestEnsureAbortUploadsRule632=== RUN TestCacheConfigHandlerMaxNarSize633=== PAUSE TestCacheConfigHandlerMaxNarSize634=== RUN TestCreatePendingClosureRejectsOversizedNAR635=== PAUSE TestCreatePendingClosureRejectsOversizedNAR636=== RUN TestNARDeduplicationMetadataUploadBug637=== PAUSE TestNARDeduplicationMetadataUploadBug638=== RUN TestMetricsInventory639=== PAUSE TestMetricsInventory640=== RUN TestMetricsCommit641=== PAUSE TestMetricsCommit642=== RUN TestService_NativeMTLS643=== PAUSE TestService_NativeMTLS644=== RUN TestServerTLSConfig645=== PAUSE TestServerTLSConfig646=== RUN TestMultipartCleanup647=== PAUSE TestMultipartCleanup648=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded649=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded650=== RUN TestPendingCleanupUsesOneCutoff651=== PAUSE TestPendingCleanupUsesOneCutoff652=== RUN TestObjectStatsTrigger653=== PAUSE TestObjectStatsTrigger654=== RUN TestObjectStatsUpdatedOncePerCommit655=== PAUSE TestObjectStatsUpdatedOncePerCommit656=== RUN TestOrphanedObjectsGC657=== PAUSE TestOrphanedObjectsGC658=== RUN TestOrphanedObjectsGCStressTest659=== PAUSE TestOrphanedObjectsGCStressTest660=== RUN TestCreatePin_ReservedPins661=== PAUSE TestCreatePin_ReservedPins662=== RUN TestPresentReportsOnlyClosureRoots663=== PAUSE TestPresentReportsOnlyClosureRoots664=== RUN TestPresentRefreshesOnlyStaleClosures665=== PAUSE TestPresentRefreshesOnlyStaleClosures666=== RUN TestPresentNotReportedWhileGCDeletesClosure667=== PAUSE TestPresentNotReportedWhileGCDeletesClosure668=== RUN TestParseSingleRange669=== PAUSE TestParseSingleRange670=== RUN TestProxyHeadersOnlyTrustedOnSocket6712026/11/11 23:37:24 INFO Starting HTTP server address=127.0.0.1:429516722026/11/11 23:37:24 INFO Starting HTTP server address=proxy.sock6732026/11/11 23:37:24 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6742026/11/11 23:37:24 INFO Shutdown signal received, draining in-flight requests timeout=10s675--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)676=== RUN TestIsValidCachePath677=== PAUSE TestIsValidCachePath678=== RUN TestReadProxyNarinfo679=== PAUSE TestReadProxyNarinfo680=== RUN TestReadProxyNarinfoAlreadyDecompressed681=== PAUSE TestReadProxyNarinfoAlreadyDecompressed682=== RUN TestReadProxyContentNegotiation683=== PAUSE TestReadProxyContentNegotiation684=== RUN TestReadProxyNarStreaming685=== PAUSE TestReadProxyNarStreaming686=== RUN TestReadProxy404687=== PAUSE TestReadProxy404688=== RUN TestReadProxyInvalidPath689=== PAUSE TestReadProxyInvalidPath690=== RUN TestReadProxyHead691=== PAUSE TestReadProxyHead692=== RUN TestReadProxyOutlastsServerWriteTimeout693=== PAUSE TestReadProxyOutlastsServerWriteTimeout694=== RUN TestReadProxyConditionalGet695=== PAUSE TestReadProxyConditionalGet696=== RUN TestReadProxyRootRedirectsToIndexHTML697=== PAUSE TestReadProxyRootRedirectsToIndexHTML698=== RUN TestReadProxyDisabled699=== PAUSE TestReadProxyDisabled700=== RUN TestReadRedirectNar701=== PAUSE TestReadRedirectNar702=== RUN TestReadRedirectKeepsNarinfoProxied703=== PAUSE TestReadRedirectKeepsNarinfoProxied704=== RUN TestReadProxyRangeRequest705=== PAUSE TestReadProxyRangeRequest706=== RUN TestReadRedirectUsesPublicS3URL707=== PAUSE TestReadRedirectUsesPublicS3URL708=== RUN TestPush_OverlappingRootsStoreOneRowPerKey709=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey710=== RUN TestPush_CompleteCommitsEveryRoot711=== PAUSE TestPush_CompleteCommitsEveryRoot712=== RUN TestPush_RepeatedRootCommitsOnce713=== PAUSE TestPush_RepeatedRootCommitsOnce714=== RUN TestPush_CompleteWaitsForCleanupOfThePush715=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush716=== RUN TestPush_RootsAreLockedInByteOrder717=== PAUSE TestPush_RootsAreLockedInByteOrder718=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit719=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit720=== RUN TestPush_RejectsBadRequests721=== PAUSE TestPush_RejectsBadRequests722=== RUN TestPush_SignsNarinfosOfItsPendingObjects723=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects724=== RUN TestSharedNarBothUploadsComplete725=== PAUSE TestSharedNarBothUploadsComplete726=== RUN TestCompleteMultipartUpload_ErrorButObjectExists727=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists728=== RUN TestCompletedNarNotReofferedAcrossClosures729=== PAUSE TestCompletedNarNotReofferedAcrossClosures730=== RUN TestPresignedUploadRegisteredBeforeCommit731=== PAUSE TestPresignedUploadRegisteredBeforeCommit732=== RUN TestService_Rustfstest733=== PAUSE TestService_Rustfstest734=== RUN TestParseSize735=== PAUSE TestParseSize736=== RUN TestSkippedUploadsHandler737=== PAUSE TestSkippedUploadsHandler738=== RUN TestSystemdListenerNotActivated739--- PASS: TestSystemdListenerNotActivated (0.00s)740=== RUN TestWatchdogBeatsWhenHealthy741--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)742=== RUN TestWatchdogSkipsWhenUnhealthy7432026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7442026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7452026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7462026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7472026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7482026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7492026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/11 23:37:24 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"753--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)754=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle755=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle756=== RUN TestProxyWriteTimeout757=== PAUSE TestProxyWriteTimeout758=== RUN TestPendingClosureWriteTimeout759=== PAUSE TestPendingClosureWriteTimeout760=== RUN TestIsValidUploadKey761=== PAUSE TestIsValidUploadKey762=== RUN TestUploadHandlersRejectInvalidKeys763=== PAUSE TestUploadHandlersRejectInvalidKeys764=== RUN TestUploadHandlersRejectOversizedBody765=== PAUSE TestUploadHandlersRejectOversizedBody766=== RUN TestService_cleanupPendingClosuresHandler767=== PAUSE TestService_cleanupPendingClosuresHandler768=== RUN TestService_createPendingClosureHandler769=== PAUSE TestService_createPendingClosureHandler770=== RUN TestService_verifyS3Integrity771=== PAUSE TestService_verifyS3Integrity772=== RUN TestService_verifyS3ReuploadsStaleNarinfo773=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo774=== RUN TestCompleteMultipartUnregistered775=== PAUSE TestCompleteMultipartUnregistered776=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT777=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT778=== CONT TestPush_CompleteCommitsEveryRoot779=== CONT TestPresentReportsOnlyClosureRoots780=== CONT TestForceGCDuringPushOffersSweptObject781=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows782=== CONT TestAbortSucceeded783--- PASS: TestAbortSucceeded (0.00s)784=== CONT TestDeleteClosuresRacingPins785=== CONT TestPush_OverlappingRootsStoreOneRowPerKey786=== CONT TestReadRedirectUsesPublicS3URL787=== CONT TestReadProxyRangeRequest788=== CONT TestReadRedirectKeepsNarinfoProxied789=== CONT TestReadRedirectNar790=== CONT TestReadProxyDisabled791=== CONT TestReadProxyRootRedirectsToIndexHTML792=== CONT TestReadProxyConditionalGet793=== CONT TestReadProxyOutlastsServerWriteTimeout794=== CONT TestReadProxyHead795=== CONT TestReadProxyInvalidPath796=== CONT TestReadProxy404797=== CONT TestReadProxyNarStreaming798=== CONT TestReadProxyContentNegotiation799=== CONT TestReadProxyNarinfoAlreadyDecompressed800=== CONT TestReadProxyNarinfo801=== CONT TestIsValidCachePath802=== CONT TestParseSingleRange803=== CONT TestPresentNotReportedWhileGCDeletesClosure804=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale805=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale806=== CONT TestPresentRefreshesOnlyStaleClosures807=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows808=== RUN TestIsValidCachePath/narinfo809=== PAUSE TestIsValidCachePath/narinfo810=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars811=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars812=== RUN TestIsValidCachePath/nar_zst813=== PAUSE TestIsValidCachePath/nar_zst814=== RUN TestIsValidCachePath/nar_xz815=== PAUSE TestIsValidCachePath/nar_xz816=== RUN TestIsValidCachePath/nar_bz2817=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh818=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh819=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check820=== RUN TestParseSingleRange/none821=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check822=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection823=== PAUSE TestIsValidCachePath/nar_bz2824=== PAUSE TestParseSingleRange/none825=== CONT TestGCSweepDeliversEachKeyOnce826=== RUN TestIsValidCachePath/nar_uncompressed827=== PAUSE TestIsValidCachePath/nar_uncompressed828=== RUN TestParseSingleRange/unknown_unit829=== PAUSE TestParseSingleRange/unknown_unit830=== RUN TestIsValidCachePath/ls831=== PAUSE TestIsValidCachePath/ls832=== RUN TestParseSingleRange/multi-range_ignored833=== PAUSE TestParseSingleRange/multi-range_ignored834=== RUN TestParseSingleRange/malformed_no_dash835=== RUN TestIsValidCachePath/log836=== PAUSE TestIsValidCachePath/log837=== RUN TestIsValidCachePath/realisation838=== PAUSE TestIsValidCachePath/realisation839=== RUN TestIsValidCachePath/nix-cache-info840=== PAUSE TestIsValidCachePath/nix-cache-info841=== RUN TestIsValidCachePath/index.html842=== PAUSE TestIsValidCachePath/index.html843=== RUN TestIsValidCachePath/traversal_parent844=== PAUSE TestIsValidCachePath/traversal_parent845=== RUN TestIsValidCachePath/traversal_in_middle846=== PAUSE TestIsValidCachePath/traversal_in_middle847=== PAUSE TestParseSingleRange/malformed_no_dash848=== RUN TestParseSingleRange/malformed_both_empty849=== RUN TestIsValidCachePath/invalid_char_e850=== PAUSE TestIsValidCachePath/invalid_char_e851=== RUN TestIsValidCachePath/invalid_char_u852=== PAUSE TestIsValidCachePath/invalid_char_u853=== RUN TestIsValidCachePath/random_path854=== PAUSE TestIsValidCachePath/random_path855=== RUN TestIsValidCachePath/empty856=== PAUSE TestParseSingleRange/malformed_both_empty857=== RUN TestParseSingleRange/malformed_end_before_start858=== PAUSE TestParseSingleRange/malformed_end_before_start859=== RUN TestParseSingleRange/closed860=== PAUSE TestParseSingleRange/closed861=== PAUSE TestIsValidCachePath/empty862=== RUN TestIsValidCachePath/leading_slash863=== PAUSE TestIsValidCachePath/leading_slash864=== RUN TestParseSingleRange/open-ended865=== PAUSE TestParseSingleRange/open-ended866=== RUN TestParseSingleRange/end_clamped_to_size867=== RUN TestIsValidCachePath/wrong_extension868=== PAUSE TestIsValidCachePath/wrong_extension869=== RUN TestIsValidCachePath/short_hash870=== PAUSE TestIsValidCachePath/short_hash871=== PAUSE TestParseSingleRange/end_clamped_to_size872=== CONT TestTombstonedObjectOfferedWithoutWaiting873=== RUN TestParseSingleRange/suffix874=== PAUSE TestParseSingleRange/suffix875=== RUN TestParseSingleRange/suffix_exceeds_size876=== PAUSE TestParseSingleRange/suffix_exceeds_size877=== RUN TestParseSingleRange/single_byte878=== PAUSE TestParseSingleRange/single_byte879=== RUN TestParseSingleRange/start_past_EOF880=== PAUSE TestParseSingleRange/start_past_EOF881=== RUN TestParseSingleRange/start_far_past_EOF882=== PAUSE TestParseSingleRange/start_far_past_EOF883=== CONT TestGCSweepSkipsPendingObjects884--- PASS: TestReadRedirectKeepsNarinfoProxied (0.38s)885=== CONT TestDeduplicatedObjectsRecordedAsPending886--- PASS: TestPresentReportsOnlyClosureRoots (0.42s)887=== CONT TestPushDedupSurvivesConcurrentGC8882026/11/11 23:37:24 INFO Received push request method=POST path=/api/pushes889--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.45s)890=== CONT TestGCMetrics891--- PASS: TestReadRedirectUsesPublicS3URL (0.47s)892=== CONT TestGCBugBareHashReferences893--- PASS: TestReadProxyDisabled (0.48s)894=== CONT TestGCAdvisoryLockBlocksConcurrentRun895--- PASS: TestReadProxyRangeRequest (0.52s)896=== CONT TestLeadEndsOnShutdown8972026/11/11 23:37:25 INFO Received push request method=POST path=/api/pushes8982026/11/11 23:37:25 INFO Received complete push request method=POST path=/api/pushes/1/complete8992026/11/11 23:37:25 INFO Completed push id=1 objects=6 duration=7.654706ms900--- PASS: TestPush_CompleteCommitsEveryRoot (0.57s)901=== CONT TestLeadGenerationGrowsWithEveryGrant902--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.63s)903=== CONT TestLeadIncumbentAnnouncesAtOnce904--- PASS: TestReadProxyInvalidPath (0.66s)905=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent906--- PASS: TestReadProxyHead (0.70s)9072026/11/11 23:37:25 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>908=== CONT TestLeadElectsOneAndHandsOver909--- PASS: TestReadProxyNarinfo (0.70s)910=== CONT TestConnectWaitsForAPeerMigration911--- PASS: TestReadProxyConditionalGet (0.76s)912=== CONT TestResolveDBConnectionString913=== RUN TestResolveDBConnectionString/flag_wins914=== PAUSE TestResolveDBConnectionString/flag_wins915=== RUN TestResolveDBConnectionString/file_when_flag_empty916=== PAUSE TestResolveDBConnectionString/file_when_flag_empty917=== RUN TestResolveDBConnectionString/missing_file_is_an_error918=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error919=== RUN TestResolveDBConnectionString/PGHOST_allows_empty920=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty921=== RUN TestResolveDBConnectionString/nothing_configured922=== PAUSE TestResolveDBConnectionString/nothing_configured923=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock924--- PASS: TestReadRedirectNar (0.79s)925=== CONT TestSkippedUploadsHandler9262026/11/11 23:37:25 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000927--- PASS: TestSkippedUploadsHandler (0.01s)928=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT929--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.81s)930=== CONT TestCompleteMultipartUnregistered931--- PASS: TestReadProxy404 (0.91s)932=== CONT TestService_verifyS3ReuploadsStaleNarinfo933=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed934=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed935=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed936=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed937=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost938=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost939=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed940=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed941=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed942=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed943=== CONT TestService_verifyS3Integrity9442026/11/11 23:37:25 INFO Received uploads request method=POST path=/api/pending_closures9452026/11/11 23:37:25 INFO Aborted multipart uploads count=0 kept=09462026/11/11 23:37:25 WARN Force mode enabled - objects will be deleted immediately without grace period947--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.93s)948=== CONT TestService_createPendingClosureHandler949--- PASS: TestReadProxyContentNegotiation (0.95s)950=== CONT TestService_cleanupPendingClosuresHandler951--- PASS: TestPresentRefreshesOnlyStaleClosures (0.97s)952=== CONT TestUploadHandlersRejectOversizedBody9532026/11/11 23:37:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=09542026/11/11 23:37:25 INFO Vacuumed table table=pending_closures9552026/11/11 23:37:25 INFO Vacuumed table table=pending_objects9562026/11/11 23:37:25 INFO Vacuumed table table=multipart_uploads9572026/11/11 23:37:25 INFO Vacuumed table table=closures9582026/11/11 23:37:25 INFO Received uploads request method=POST path=/api/pending_closures9592026/11/11 23:37:25 INFO Vacuumed table table=objects9602026/11/11 23:37:25 INFO Received uploads request method=POST path=/api/pending_closures961--- PASS: TestGCSweepDeliversEachKeyOnce (1.05s)962=== CONT TestUploadHandlersRejectInvalidKeys963=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info964=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info965=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal966=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal967=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key968=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key969=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key970=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key971=== CONT TestIsValidUploadKey972=== RUN TestIsValidUploadKey/narinfo973=== PAUSE TestIsValidUploadKey/narinfo974=== RUN TestIsValidUploadKey/nar_zst975=== PAUSE TestIsValidUploadKey/nar_zst976=== RUN TestIsValidUploadKey/nar_xz977=== PAUSE TestIsValidUploadKey/nar_xz978=== RUN TestIsValidUploadKey/nar_plain979=== PAUSE TestIsValidUploadKey/nar_plain980=== RUN TestIsValidUploadKey/listing981=== PAUSE TestIsValidUploadKey/listing982=== RUN TestIsValidUploadKey/build_log983=== PAUSE TestIsValidUploadKey/build_log984=== RUN TestIsValidUploadKey/build_log_home-manager_file985=== PAUSE TestIsValidUploadKey/build_log_home-manager_file986=== RUN TestIsValidUploadKey/build_log_plus_in_name987=== PAUSE TestIsValidUploadKey/build_log_plus_in_name988=== RUN TestIsValidUploadKey/build_log_question_mark989=== PAUSE TestIsValidUploadKey/build_log_question_mark990=== RUN TestIsValidUploadKey/build_log_equals991=== PAUSE TestIsValidUploadKey/build_log_equals992=== RUN TestIsValidUploadKey/realisation993=== PAUSE TestIsValidUploadKey/realisation994=== RUN TestIsValidUploadKey/realisation_plus_in_output995=== PAUSE TestIsValidUploadKey/realisation_plus_in_output996=== RUN TestIsValidUploadKey/nix-cache-info997=== PAUSE TestIsValidUploadKey/nix-cache-info998=== RUN TestIsValidUploadKey/index.html999=== PAUSE TestIsValidUploadKey/index.html1000=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1001=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1002=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1003=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1004=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1005=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1006=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1007=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1008=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1009=== RUN TestIsValidUploadKey/traversal1010=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1011=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1012=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1013=== PAUSE TestIsValidUploadKey/traversal1014=== CONT TestPendingClosureWriteTimeout1015=== RUN TestPendingClosureWriteTimeout/empty1016=== RUN TestIsValidUploadKey/traversal_nar1017=== PAUSE TestIsValidUploadKey/traversal_nar1018=== RUN TestIsValidUploadKey/absolute1019=== PAUSE TestIsValidUploadKey/absolute1020=== RUN TestIsValidUploadKey/empty_key1021=== PAUSE TestIsValidUploadKey/empty_key1022=== RUN TestIsValidUploadKey/unknown_type1023=== PAUSE TestIsValidUploadKey/unknown_type1024=== PAUSE TestPendingClosureWriteTimeout/empty1025=== CONT TestProxyWriteTimeout1026=== RUN TestProxyWriteTimeout/narinfo1027=== RUN TestPendingClosureWriteTimeout/400_objects1028=== PAUSE TestPendingClosureWriteTimeout/400_objects1029=== PAUSE TestProxyWriteTimeout/narinfo1030=== RUN TestProxyWriteTimeout/1_GiB_nar1031=== PAUSE TestProxyWriteTimeout/1_GiB_nar1032--- PASS: TestReadProxyNarStreaming (1.23s)1033=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1034=== RUN TestPendingClosureWriteTimeout/670k_objects1035=== PAUSE TestPendingClosureWriteTimeout/670k_objects1036=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1037=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1038=== RUN TestProxyWriteTimeout/10_GiB_nar1039=== PAUSE TestProxyWriteTimeout/10_GiB_nar1040=== CONT TestClientCADerivations1041=== RUN TestProxyWriteTimeout/unknown_size1042=== PAUSE TestProxyWriteTimeout/unknown_size1043=== CONT TestClientFallsBackToClosures10442026/11/11 23:37:25 INFO Aborted multipart uploads count=0 kept=010452026/11/11 23:37:25 WARN Force mode enabled - objects will be deleted immediately without grace period10462026/11/11 23:37:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010472026/11/11 23:37:25 INFO Vacuumed table table=pending_closures10482026/11/11 23:37:25 INFO Vacuumed table table=pending_objects10492026/11/11 23:37:25 INFO Vacuumed table table=multipart_uploads10502026/11/11 23:37:25 INFO Vacuumed table table=closures10512026/11/11 23:37:25 INFO Vacuumed table table=objects1052--- PASS: TestGCSweepSkipsPendingObjects (1.27s)1053=== CONT TestClientPushesUseOnePush10542026/11/11 23:37:25 INFO Received uploads request method=POST path=/api/pending_closures1055--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (1.30s)1056=== CONT TestPinProtectsFromGC1057--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.93s)1058=== CONT TestClientSharedPathCommittedMidPush10592026/11/11 23:37:25 INFO Received uploads request method=POST path=/api/pending_closures10602026/11/11 23:37:25 INFO Aborted multipart uploads count=0 kept=010612026/11/11 23:37:25 WARN Force mode enabled - objects will be deleted immediately without grace period10622026/11/11 23:37:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010632026/11/11 23:37:25 INFO Vacuumed table table=pending_closures10642026/11/11 23:37:25 INFO Vacuumed table table=pending_objects10652026/11/11 23:37:25 INFO Vacuumed table table=multipart_uploads10662026/11/11 23:37:25 INFO Vacuumed table table=closures10672026/11/11 23:37:25 INFO Vacuumed table table=objects1068--- PASS: TestGCMetrics (0.92s)1069=== CONT TestClientWithDependencies10702026/11/11 23:37:25 INFO lead: acquired remote=192.0.2.1:1234 generation=110712026/11/11 23:37:25 INFO Aborted multipart uploads count=0 kept=010722026/11/11 23:37:25 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1073--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.93s)1074=== CONT TestClientMultipleUploads10752026/11/11 23:37:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010762026/11/11 23:37:25 INFO Vacuumed table table=pending_closures10772026/11/11 23:37:25 INFO Vacuumed table table=pending_objects10782026/11/11 23:37:25 INFO Vacuumed table table=multipart_uploads10792026/11/11 23:37:25 INFO Vacuumed table table=closures10802026/11/11 23:37:25 INFO Vacuumed table table=objects10812026/11/11 23:37:25 INFO Aborted multipart uploads count=0 kept=010822026/11/11 23:37:25 WARN Force mode enabled - objects will be deleted immediately without grace period10832026/11/11 23:37:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010842026/11/11 23:37:25 INFO Vacuumed table table=pending_closures10852026/11/11 23:37:25 INFO Vacuumed table table=pending_objects10862026/11/11 23:37:25 INFO Vacuumed table table=multipart_uploads10872026/11/11 23:37:25 INFO Vacuumed table table=closures10882026/11/11 23:37:25 INFO Vacuumed table table=objects1089--- PASS: TestPushDedupSurvivesConcurrentGC (1.04s)1090=== CONT TestClientIntegration10912026/11/11 23:37:25 INFO lead: acquired remote=192.0.2.1:1234 generation=11092--- PASS: TestDeleteClosuresRacingPins (1.47s)1093=== CONT TestClientErrorHandling1094=== RUN TestClientErrorHandling/InvalidStorePath1095=== PAUSE TestClientErrorHandling/InvalidStorePath1096=== RUN TestClientErrorHandling/InvalidAuthToken1097=== PAUSE TestClientErrorHandling/InvalidAuthToken1098=== RUN TestClientErrorHandling/ServerNotAvailable1099=== PAUSE TestClientErrorHandling/ServerNotAvailable1100=== CONT TestService_AuthMiddleware_OIDC11012026/11/11 23:37:26 INFO lead: acquired remote=192.0.2.1:1234 generation=111022026/11/11 23:37:26 INFO lead: released remote=192.0.2.1:1234 generation=11103--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.89s)1104=== CONT TestCacheStatsHandler11052026/11/11 23:37:26 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43489/oidc11062026/11/11 23:37:26 INFO lead: acquired remote=192.0.2.1:1234 generation=111072026/11/11 23:37:26 INFO lead: acquired remote=192.0.2.1:1234 generation=111082026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures1109--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.84s)1110=== CONT TestCacheConfigHandler1111=== RUN TestCacheConfigHandler/full_config,_no_issuer1112=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1113=== RUN TestCacheConfigHandler/no_cache_url_configured1114=== PAUSE TestCacheConfigHandler/no_cache_url_configured1115=== RUN TestCacheConfigHandler/no_signing_keys1116=== PAUSE TestCacheConfigHandler/no_signing_keys1117=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1118=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1119=== CONT TestService_ReadScope_PublicByDefault11202026/11/11 23:37:26 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11212026/11/11 23:37:26 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1122--- PASS: TestCompleteMultipartUnregistered (0.84s)1123=== CONT TestService_RequireScope_OIDC1124--- PASS: TestGCBugBareHashReferences (1.19s)1125=== CONT TestSharedNarBothUploadsComplete11262026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures11272026/11/11 23:37:26 INFO Received cleanup request method=DELETE path=/api/pending_closures11282026/11/11 23:37:26 INFO Aborted multipart uploads count=0 kept=011292026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures11302026/11/11 23:37:26 INFO Received cleanup request method=DELETE path=/api/pending_closures11312026/11/11 23:37:26 INFO Aborted multipart uploads count=1 kept=011322026/11/11 23:37:26 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11332026-11-11 23:37:26.267 UTC [739] ERROR: Closure does not exist: id=111342026-11-11 23:37:26.267 UTC [739] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE11352026-11-11 23:37:26.267 UTC [739] STATEMENT: -- name: CommitPendingClosure :one1136 SELECT commit_pending_closure($1::bigint)::bigint1137 1138--- PASS: TestService_cleanupPendingClosuresHandler (0.79s)11392026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures1140=== CONT TestParseSize11412026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures1142--- PASS: TestParseSize (0.00s)1143=== CONT TestService_Rustfstest11442026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures11452026/11/11 23:37:26 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45693/oidc1146--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (1.04s)1147=== CONT TestPresignedUploadRegisteredBeforeCommit11482026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures1149=== NAME TestClientCADerivations1150 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations3584875164/001/store/ba5mssg0ldlsfdh3mnrsxr4n2l6cv8fy-ca-test1151 client_ca_test.go:139: Found 1 dependencies (including self)11522026/11/11 23:37:26 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1153=== NAME TestPinProtectsFromGC1154 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC442583994/001/store/1hp7dvbj8f65jj9jcfxppxf77rgwcn7z-pinned-file.txt1155 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC442583994/001/store/fnb279965smk0hnnvgxygnq4fpj1w0df-unpinned-file.txt11562026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes1157=== NAME TestClientMultipleUploads1158 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads4110903559/001/store/djp3zaf3z5cxdicfyz85wrm5zvvz007c-test-file-0.txt11592026/11/11 23:37:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11602026/11/11 23:37:26 INFO Uploading ba5mssg0ldlsfdh3mnrsxr4n2l6cv8fy-ca-test (144B)11612026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11622026/11/11 23:37:26 WARN Failed to register uploaded object key=log/x525vwif8fj3radmi48gfa5kk9cqr07f-ca-test.drv error="server returned 404: 404 page not found\n"11632026/11/11 23:37:26 WARN Failed to register uploaded object key=ba5mssg0ldlsfdh3mnrsxr4n2l6cv8fy.ls error="server returned 404: 404 page not found\n"11642026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"11652026/11/11 23:37:26 INFO Signed narinfos id=1 count=111662026/11/11 23:37:26 INFO Uploading 1 narinfos1167--- PASS: TestCacheStatsHandler (0.52s)1168=== CONT TestCompletedNarNotReofferedAcrossClosures11692026/11/11 23:37:26 WARN Failed to register uploaded object key=ba5mssg0ldlsfdh3mnrsxr4n2l6cv8fy.narinfo error="server returned 404: 404 page not found\n"11702026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete11712026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures11722026/11/11 23:37:26 INFO Completed push id=1 objects=4 duration=4.467267ms11732026/11/11 23:37:26 INFO Upload complete. (93ms)11742026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures11752026/11/11 23:37:26 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)11762026/11/11 23:37:26 INFO Uploading 6f687gb58120d0pwmlvnaminnggji85l-shared-dep (136B)11772026/11/11 23:37:26 INFO Uploading wfyx6agsysg0hxmimn7ajlhvwkra9npk-b (216B)1178=== NAME TestClientWithDependencies1179 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies966845910/001/store/6m4ckjvp6cqp89hval508gb1gbjbmsid-test-script1180=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1181=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1182=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1183=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1184=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1185=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1186=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1187=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1188=== CONT TestCompleteMultipartUpload_ErrorButObjectExists1189=== NAME TestClientCADerivations1190 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations3584875164/001/store/ba5mssg0ldlsfdh3mnrsxr4n2l6cv8fy-ca-test1191 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1192 Compression: zstd1193 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1194 NarSize: 1441195 References: 1196 Deriver: /build/TestClientCADerivations3584875164/001/store/x525vwif8fj3radmi48gfa5kk9cqr07f-ca-test.drv1197 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1198 client_ca_test.go:185: Checking for realisation files in S3...1199 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1200 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1201=== NAME TestClientIntegration1202 client_integration_test.go:292: Created store path: /build/TestClientIntegration1086282682/002/store/9hx24jj55z7zs0dpl2rpqiapixvr8ci8-test-file.txt12032026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12042026/11/11 23:37:26 WARN Failed to register uploaded object key=6wzdi647bhji063avy0n7n9azkqckfh9.ls error="server returned 404: 404 page not found\n"12052026/11/11 23:37:26 WARN Failed to register uploaded object key=wfyx6agsysg0hxmimn7ajlhvwkra9npk.ls error="server returned 404: 404 page not found\n"12062026/11/11 23:37:26 WARN Failed to register uploaded object key=6f687gb58120d0pwmlvnaminnggji85l.ls error="server returned 404: 404 page not found\n"12072026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign12082026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/15qzy45n4f6gbxllhbpj794bribi8p0irsg9grd4li00d6fdq0g3.nar.zst error="server returned 404: 404 page not found\n"12092026/11/11 23:37:26 INFO Signed narinfos id=1 count=212102026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign12112026/11/11 23:37:26 INFO Signed narinfos id=2 count=212122026/11/11 23:37:26 INFO Uploading 4 narinfos1213=== NAME TestClientMultipleUploads1214 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads4110903559/001/store/avjgiymzi9gm0vz2wdnrzx35lbg7vcr3-test-file-1.txt12152026/11/11 23:37:26 WARN Failed to register uploaded object key=wfyx6agsysg0hxmimn7ajlhvwkra9npk.narinfo error="server returned 404: 404 page not found\n"12162026/11/11 23:37:26 WARN Failed to register uploaded object key=6f687gb58120d0pwmlvnaminnggji85l.narinfo error="server returned 404: 404 page not found\n"12172026/11/11 23:37:26 WARN Failed to register uploaded object key=6wzdi647bhji063avy0n7n9azkqckfh9.narinfo error="server returned 404: 404 page not found\n"12182026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes12192026/11/11 23:37:26 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12202026/11/11 23:37:26 WARN Failed to register uploaded object key=6f687gb58120d0pwmlvnaminnggji85l.narinfo error="server returned 404: 404 page not found\n"12212026/11/11 23:37:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12222026/11/11 23:37:26 INFO Uploading 1hp7dvbj8f65jj9jcfxppxf77rgwcn7z-pinned-file.txt (128B)12232026/11/11 23:37:26 INFO Completed upload id=1 objects=6 duration=7.849716ms12242026/11/11 23:37:26 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12252026/11/11 23:37:26 INFO Completed upload id=2 objects=6 duration=1.3351ms12262026/11/11 23:37:26 INFO Upload complete. (75ms)1227--- PASS: TestService_ReadScope_PublicByDefault (0.43s)1228=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit12292026/11/11 23:37:26 WARN Failed to register uploaded object key=1hp7dvbj8f65jj9jcfxppxf77rgwcn7z.ls error="server returned 404: 404 page not found\n"12302026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes1231=== NAME TestClientFallsBackToClosures12322026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1233 client_pushes_test.go:112: Retrieved narinfo from S3:12342026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"1235 StorePath: /build/TestClientFallsBackToClosures3713942835/001/store/6f687gb58120d0pwmlvnaminnggji85l-shared-dep1236 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1237 Compression: zstd1238 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821239 NarSize: 13612402026/11/11 23:37:26 INFO Signed narinfos id=1 count=11241 References: 1242 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n12432026/11/11 23:37:26 INFO Uploading 1 narinfos1244 client_pushes_test.go:112: Retrieved narinfo from S3:1245 StorePath: /build/TestClientFallsBackToClosures3713942835/001/store/6wzdi647bhji063avy0n7n9azkqckfh9-a1246 URL: nar/15qzy45n4f6gbxllhbpj794bribi8p0irsg9grd4li00d6fdq0g3.nar.zst1247 Compression: zstd1248 NarHash: sha256:15qzy45n4f6gbxllhbpj794bribi8p0irsg9grd4li00d6fdq0g31249 NarSize: 2161250 References: /build/TestClientFallsBackToClosures3713942835/001/store/6f687gb58120d0pwmlvnaminnggji85l-shared-dep1251 CA: text:sha256:1zgy7kv06si7dv0qwl8r63x0284ivz3qsqniksxmzycglh7i98mv1252=== NAME TestClientWithDependencies1253 client_integration_test.go:663: Found 1 dependencies (including self)12542026/11/11 23:37:26 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)12552026/11/11 23:37:26 INFO Uploading hczhaw5i18wsj5xd04jdxj0rn7wsr2rr-shared-dep (136B)12562026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete12572026/11/11 23:37:26 INFO Uploading 6gblw8jbrzqgpdwqkg7563w3xz6crdns-b (208B)12582026/11/11 23:37:26 WARN Failed to register uploaded object key=1hp7dvbj8f65jj9jcfxppxf77rgwcn7z.narinfo error="server returned 404: 404 page not found\n"1259=== NAME TestClientFallsBackToClosures1260 client_pushes_test.go:112: Retrieved narinfo from S3:1261 StorePath: /build/TestClientFallsBackToClosures3713942835/001/store/wfyx6agsysg0hxmimn7ajlhvwkra9npk-b1262 URL: nar/15qzy45n4f6gbxllhbpj794bribi8p0irsg9grd4li00d6fdq0g3.nar.zst1263 Compression: zstd1264 NarHash: sha256:15qzy45n4f6gbxllhbpj794bribi8p0irsg9grd4li00d6fdq0g31265 NarSize: 2161266 References: /build/TestClientFallsBackToClosures3713942835/001/store/6f687gb58120d0pwmlvnaminnggji85l-shared-dep1267 CA: text:sha256:1zgy7kv06si7dv0qwl8r63x0284ivz3qsqniksxmzycglh7i98mv12682026/11/11 23:37:26 WARN Failed to register uploaded object key=hczhaw5i18wsj5xd04jdxj0rn7wsr2rr.ls error="server returned 404: 404 page not found\n"12692026/11/11 23:37:26 WARN Failed to register uploaded object key=6gblw8jbrzqgpdwqkg7563w3xz6crdns.ls error="server returned 404: 404 page not found\n"12702026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12712026/11/11 23:37:26 WARN Failed to register uploaded object key=5gyf3wwgjl3vrg9y6hz2fd2wf9plcdf0.ls error="server returned 404: 404 page not found\n"12722026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12732026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/1l3r72yrccgsk3mzl901djjc34w7fivm0abvaal6r4g2j6qxikma.nar.zst error="server returned 404: 404 page not found\n"12742026/11/11 23:37:26 INFO Completed push id=1 objects=3 duration=6.672167ms12752026/11/11 23:37:26 INFO Upload complete. (67ms)12762026/11/11 23:37:26 INFO Signed narinfos id=1 count=312772026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes12782026/11/11 23:37:26 INFO Uploading 3 narinfos1279--- PASS: TestClientFallsBackToClosures (0.86s)1280=== CONT TestPush_SignsNarinfosOfItsPendingObjects1281=== NAME TestClientMultipleUploads1282 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads4110903559/001/store/sn0fpz1552z19988hnjm279p2f81vgiq-test-file-2.txt12832026/11/11 23:37:26 WARN Failed to register uploaded object key=6gblw8jbrzqgpdwqkg7563w3xz6crdns.narinfo error="server returned 404: 404 page not found\n"12842026/11/11 23:37:26 WARN Failed to register uploaded object key=hczhaw5i18wsj5xd04jdxj0rn7wsr2rr.narinfo error="server returned 404: 404 page not found\n"12852026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete12862026/11/11 23:37:26 WARN Failed to register uploaded object key=5gyf3wwgjl3vrg9y6hz2fd2wf9plcdf0.narinfo error="server returned 404: 404 page not found\n"12872026/11/11 23:37:26 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)12882026/11/11 23:37:26 INFO Uploading gy221r9l9sn3cnh8flqli8djxz3i85j9-shared-dep (136B)12892026/11/11 23:37:26 INFO Uploading giq0w0w8rz9h8qni8fv21zzj9fyzilcj-top (224B)12902026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures12912026/11/11 23:37:26 WARN Failed to register uploaded object key=giq0w0w8rz9h8qni8fv21zzj9fyzilcj.ls error="server returned 404: 404 page not found\n"12922026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/167wqq99n5yw3i3khkq0012i18lg6ga3cjdi3ry9yar088ppphh2.nar.zst error="server returned 404: 404 page not found\n"12932026/11/11 23:37:26 WARN Failed to register uploaded object key=gy221r9l9sn3cnh8flqli8djxz3i85j9.ls error="server returned 404: 404 page not found\n"12942026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12952026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12962026/11/11 23:37:26 INFO Signed narinfos id=1 count=212972026/11/11 23:37:26 INFO Uploading 2 narinfos12982026/11/11 23:37:26 INFO Completed push id=1 objects=8 duration=9.839675ms12992026/11/11 23:37:26 INFO Upload complete. (70ms)1300=== NAME TestClientPushesUseOnePush1301 client_pushes_test.go:97: Retrieved narinfo from S3:1302 StorePath: /build/TestClientPushesUseOnePush702232688/001/store/hczhaw5i18wsj5xd04jdxj0rn7wsr2rr-shared-dep1303 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1304 Compression: zstd1305 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821306 NarSize: 1361307 References: 1308 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n13092026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete1310 client_pushes_test.go:97: Retrieved narinfo from S3:13112026/11/11 23:37:26 WARN Failed to register uploaded object key=gy221r9l9sn3cnh8flqli8djxz3i85j9.narinfo error="server returned 404: 404 page not found\n"1312 StorePath: /build/TestClientPushesUseOnePush702232688/001/store/5gyf3wwgjl3vrg9y6hz2fd2wf9plcdf0-a1313 URL: nar/1l3r72yrccgsk3mzl901djjc34w7fivm0abvaal6r4g2j6qxikma.nar.zst13142026/11/11 23:37:26 WARN Failed to register uploaded object key=giq0w0w8rz9h8qni8fv21zzj9fyzilcj.narinfo error="server returned 404: 404 page not found\n"1315 Compression: zstd1316 NarHash: sha256:1l3r72yrccgsk3mzl901djjc34w7fivm0abvaal6r4g2j6qxikma1317 NarSize: 2081318 References: /build/TestClientPushesUseOnePush702232688/001/store/hczhaw5i18wsj5xd04jdxj0rn7wsr2rr-shared-dep1319 CA: text:sha256:0bs3lsrwg0d3jp1vnczwgg04d0dfp8d1s53b2yy6dsi90j66mx6813202026/11/11 23:37:26 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1321 client_pushes_test.go:97: Retrieved narinfo from S3:1322 StorePath: /build/TestClientPushesUseOnePush702232688/001/store/6gblw8jbrzqgpdwqkg7563w3xz6crdns-b1323 URL: nar/1l3r72yrccgsk3mzl901djjc34w7fivm0abvaal6r4g2j6qxikma.nar.zst1324 Compression: zstd1325 NarHash: sha256:1l3r72yrccgsk3mzl901djjc34w7fivm0abvaal6r4g2j6qxikma1326 NarSize: 2081327 References: /build/TestClientPushesUseOnePush702232688/001/store/hczhaw5i18wsj5xd04jdxj0rn7wsr2rr-shared-dep1328 CA: text:sha256:0bs3lsrwg0d3jp1vnczwgg04d0dfp8d1s53b2yy6dsi90j66mx6813292026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures13302026/11/11 23:37:26 INFO Completed push id=1 objects=6 duration=9.256175ms13312026/11/11 23:37:26 INFO Upload complete. (68ms)1332--- PASS: TestClientPushesUseOnePush (0.84s)13332026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes1334=== CONT TestPush_RejectsBadRequests1335=== NAME TestClientSharedPathCommittedMidPush1336 client_integration_test.go:728: Retrieved narinfo from S3:1337 StorePath: /build/TestClientSharedPathCommittedMidPush734096266/001/store/gy221r9l9sn3cnh8flqli8djxz3i85j9-shared-dep1338 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1339 Compression: zstd1340 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821341 NarSize: 1361342 References: 1343 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1344 client_integration_test.go:728: Retrieved narinfo from S3:1345 StorePath: /build/TestClientSharedPathCommittedMidPush734096266/001/store/giq0w0w8rz9h8qni8fv21zzj9fyzilcj-top1346 URL: nar/167wqq99n5yw3i3khkq0012i18lg6ga3cjdi3ry9yar088ppphh2.nar.zst1347 Compression: zstd1348 NarHash: sha256:167wqq99n5yw3i3khkq0012i18lg6ga3cjdi3ry9yar088ppphh21349 NarSize: 2241350 References: /build/TestClientSharedPathCommittedMidPush734096266/001/store/gy221r9l9sn3cnh8flqli8djxz3i85j9-shared-dep1351 CA: text:sha256:0nvpihpv889354yh1a36yzz55rykrr0w9rg11bimcafragk2gxrf13522026/11/11 23:37:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13532026/11/11 23:37:26 INFO Uploading 9hx24jj55z7zs0dpl2rpqiapixvr8ci8-test-file.txt (152B)1354--- PASS: TestClientSharedPathCommittedMidPush (0.81s)1355=== CONT TestPush_CompleteWaitsForCleanupOfThePush13562026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"1357--- PASS: TestService_Rustfstest (0.38s)13582026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1359=== CONT TestPush_RootsAreLockedInByteOrder13602026/11/11 23:37:26 WARN Failed to register uploaded object key=9hx24jj55z7zs0dpl2rpqiapixvr8ci8.ls error="server returned 404: 404 page not found\n"13612026/11/11 23:37:26 INFO Signed narinfos id=1 count=113622026/11/11 23:37:26 INFO Uploading 1 narinfos13632026/11/11 23:37:26 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NmY5ODliZjctNzJiZC00OTRiLWJjNWUtYTQ5MTA4Yzk5YjYxLjA3MjZkN2QwLWQ4OWQtNDI5ZS04M2I3LTJlMmVjZDkxZTQ0OXgxNzk0NDQwMjQ2MjI1MTcyMTkw parts=1013642026/11/11 23:37:26 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13652026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete13662026/11/11 23:37:26 WARN Failed to register uploaded object key=9hx24jj55z7zs0dpl2rpqiapixvr8ci8.narinfo error="server returned 404: 404 page not found\n"13672026/11/11 23:37:26 INFO Completed upload id=1 objects=2 duration=5.278677ms13682026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures13692026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes13702026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes13712026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures13722026/11/11 23:37:26 INFO Completed push id=1 objects=3 duration=25.209906ms13732026/11/11 23:37:26 INFO Upload complete. (81ms)13742026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes1375=== RUN TestService_RequireScope_OIDC/builder_may_write1376=== PAUSE TestService_RequireScope_OIDC/builder_may_write1377=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1378=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1379=== RUN TestService_RequireScope_OIDC/ops_may_admin1380=== PAUSE TestService_RequireScope_OIDC/ops_may_admin13812026/11/11 23:37:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1382=== RUN TestService_RequireScope_OIDC/ops_may_not_write13832026/11/11 23:37:26 INFO Uploading fnb279965smk0hnnvgxygnq4fpj1w0df-unpinned-file.txt (128B)1384=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1385=== RUN TestService_RequireScope_OIDC/reader_may_not_write1386=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1387=== RUN TestService_RequireScope_OIDC/static_token_may_admin1388=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1389=== RUN TestService_RequireScope_OIDC/static_token_may_write1390=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1391=== RUN TestService_RequireScope_OIDC/reader_may_read1392=== PAUSE TestService_RequireScope_OIDC/reader_may_read1393=== RUN TestService_RequireScope_OIDC/writer_implies_read1394=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1395=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1396=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1397=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1398=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1399=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1400=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1401=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1402=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1403=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1404=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1405=== CONT TestService_AuthMiddleware_MTLSBoundSubjects14062026/11/11 23:37:26 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo14072026/11/11 23:37:26 WARN Found objects in DB but missing from S3, will re-upload count=114082026/11/11 23:37:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1409--- PASS: TestService_verifyS3Integrity (1.26s)14102026/11/11 23:37:26 INFO Uploading 6m4ckjvp6cqp89hval508gb1gbjbmsid-test-script (136B)1411=== CONT TestService_ReadAuthMiddleware14122026/11/11 23:37:26 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)14132026/11/11 23:37:26 INFO Uploading djp3zaf3z5cxdicfyz85wrm5zvvz007c-test-file-0.txt (160B)1414=== NAME TestClientCADerivations14152026/11/11 23:37:26 INFO Uploading sn0fpz1552z19988hnjm279p2f81vgiq-test-file-2.txt (160B)1416 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features14172026/11/11 23:37:26 INFO Uploading avjgiymzi9gm0vz2wdnrzx35lbg7vcr3-test-file-1.txt (160B)1418 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1419 error: binary cache 's3://bucket44?endpoint=http://localhost:40115®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations3584875164/001/store'1420 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 114212026/11/11 23:37:26 WARN Failed to register uploaded object key=fnb279965smk0hnnvgxygnq4fpj1w0df.ls error="server returned 404: 404 page not found\n"14222026/11/11 23:37:26 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14232026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"14242026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign14252026/11/11 23:37:26 INFO Signed narinfos id=2 count=114262026/11/11 23:37:26 INFO Uploading 1 narinfos14272026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"14282026/11/11 23:37:26 WARN Failed to register uploaded object key=log/6ha0a8k413vz38c3y7wrff2y3qz4mbcd-test-script.drv error="server returned 404: 404 page not found\n"14292026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14302026/11/11 23:37:26 WARN Failed to register uploaded object key=6m4ckjvp6cqp89hval508gb1gbjbmsid.ls error="server returned 404: 404 page not found\n"14312026/11/11 23:37:26 INFO Signed narinfos id=1 count=114322026/11/11 23:37:26 INFO Uploading 1 narinfos14332026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"14342026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"14352026/11/11 23:37:26 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"14362026/11/11 23:37:26 WARN Failed to register uploaded object key=djp3zaf3z5cxdicfyz85wrm5zvvz007c.ls error="server returned 404: 404 page not found\n"1437--- PASS: TestClientCADerivations (0.95s)1438=== CONT TestService_AuthMiddleware_MTLSProxyHeader14392026/11/11 23:37:26 WARN Failed to register uploaded object key=avjgiymzi9gm0vz2wdnrzx35lbg7vcr3.ls error="server returned 404: 404 page not found\n"14402026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14412026/11/11 23:37:26 WARN Failed to register uploaded object key=sn0fpz1552z19988hnjm279p2f81vgiq.ls error="server returned 404: 404 page not found\n"14422026/11/11 23:37:26 INFO Signed narinfos id=1 count=314432026/11/11 23:37:26 INFO Uploading 3 narinfos14442026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/2/complete14452026/11/11 23:37:26 WARN Failed to register uploaded object key=fnb279965smk0hnnvgxygnq4fpj1w0df.narinfo error="server returned 404: 404 page not found\n"14462026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete14472026/11/11 23:37:26 WARN Failed to register uploaded object key=6m4ckjvp6cqp89hval508gb1gbjbmsid.narinfo error="server returned 404: 404 page not found\n"14482026/11/11 23:37:26 INFO Completed push id=2 objects=3 duration=2.311109ms14492026/11/11 23:37:26 INFO Upload complete. (65ms)14502026/11/11 23:37:26 WARN Failed to register uploaded object key=avjgiymzi9gm0vz2wdnrzx35lbg7vcr3.narinfo error="server returned 404: 404 page not found\n"14512026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete14522026/11/11 23:37:26 WARN Failed to register uploaded object key=sn0fpz1552z19988hnjm279p2f81vgiq.narinfo error="server returned 404: 404 page not found\n"14532026/11/11 23:37:26 WARN Failed to register uploaded object key=djp3zaf3z5cxdicfyz85wrm5zvvz007c.narinfo error="server returned 404: 404 page not found\n"14542026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures14552026/11/11 23:37:26 INFO Completed push id=1 objects=4 duration=10.809574ms14562026/11/11 23:37:26 INFO Upload complete. (79ms)1457=== NAME TestClientWithDependencies1458 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies966845910/001/store) requires matching store prefix14592026/11/11 23:37:26 INFO Completed push id=1 objects=9 duration=11.161174ms14602026/11/11 23:37:26 INFO Upload complete. (76ms)1461=== NAME TestClientMultipleUploads1462 client_integration_test.go:417: Uploaded 3 paths in 107.318092ms14632026/11/11 23:37:26 INFO All 1 paths already cached14642026/11/11 23:37:26 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NmY5ODliZjctNzJiZC00OTRiLWJjNWUtYTQ5MTA4Yzk5YjYxLjA5NzljZWI3LTAxZWYtNGQ0MS05OWNlLTYzM2E1ODlkMjkxNngxNzk0NDQwMjQ2MjgxMjM0Nzk5 parts=1014652026/11/11 23:37:26 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1466=== NAME TestClientIntegration1467 client_integration_test.go:318: Retrieved narinfo from S3:1468 StorePath: /build/TestClientIntegration1086282682/002/store/9hx24jj55z7zs0dpl2rpqiapixvr8ci8-test-file.txt1469 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1470 Compression: zstd1471 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11472 NarSize: 1521473 References: 1474 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11475 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1476 client_integration_test.go:319: Decompressed .ls content (64 bytes):1477 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1478--- PASS: TestClientWithDependencies (0.84s)1479=== CONT TestService_AuthMiddleware14802026/11/11 23:37:26 INFO Completed upload id=1 objects=2 duration=6.283307ms14812026/11/11 23:37:26 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001482--- PASS: TestClientMultipleUploads (0.80s)1483=== CONT TestService_readinessHandler14842026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures14852026/11/11 23:37:26 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst14862026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures14872026/11/11 23:37:26 INFO Starting cleanup of old closures method=DELETE path=/api/closures14882026/11/11 23:37:26 INFO Garbage collection started14892026/11/11 23:37:26 INFO Starting cleanup of old closures method=DELETE path=/api/closures1490--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.41s)1491=== CONT TestCreatePin_ReservedPins14922026/11/11 23:37:26 INFO Aborted multipart uploads count=0 kept=014932026/11/11 23:37:26 INFO Aborted multipart uploads count=0 kept=014942026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures14952026/11/11 23:37:26 WARN Force mode enabled - objects will be deleted immediately without grace period14962026/11/11 23:37:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014972026/11/11 23:37:26 INFO Vacuumed table table=pending_closures14982026/11/11 23:37:26 INFO Vacuumed table table=pending_objects14992026/11/11 23:37:26 INFO Vacuumed table table=multipart_uploads15002026/11/11 23:37:26 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38701/oidc15012026/11/11 23:37:26 INFO Vacuumed table table=closures15022026/11/11 23:37:26 INFO Received uploads request method=POST path=/api/pending_closures15032026/11/11 23:37:26 INFO Vacuumed table table=objects15042026/11/11 23:37:26 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001505--- PASS: TestService_createPendingClosureHandler (1.33s)1506=== CONT TestOrphanedObjectsGCStressTest15072026/11/11 23:37:26 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15082026/11/11 23:37:26 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NmY5ODliZjctNzJiZC00OTRiLWJjNWUtYTQ5MTA4Yzk5YjYxLmQwODJkN2ExLWQ5ZjMtNGQyZi04MWI1LTZkMTZhNjA0NDFkYngxNzk0NDQwMjQ2NzY2NzMyMjM115092026/11/11 23:37:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=015102026/11/11 23:37:26 INFO Vacuumed table table=pending_closures15112026/11/11 23:37:26 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NmY5ODliZjctNzJiZC00OTRiLWJjNWUtYTQ5MTA4Yzk5YjYxLmQwODJkN2ExLWQ5ZjMtNGQyZi04MWI1LTZkMTZhNjA0NDFkYngxNzk0NDQwMjQ2NzY2NzMyMjM1 parts=11512--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.25s)1513=== CONT TestOrphanedObjectsGC15142026/11/11 23:37:26 INFO Vacuumed table table=pending_objects15152026/11/11 23:37:26 INFO Vacuumed table table=multipart_uploads15162026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes15172026/11/11 23:37:26 INFO Vacuumed table table=closures15182026/11/11 23:37:26 INFO Vacuumed table table=objects15192026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes15202026/11/11 23:37:26 INFO Object in database but missing from S3 key=9hx24jj55z7zs0dpl2rpqiapixvr8ci8.ls15212026/11/11 23:37:26 WARN Found objects in DB but missing from S3, will re-upload count=115222026/11/11 23:37:26 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15232026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/2/complete15242026/11/11 23:37:26 WARN Failed to register uploaded object key=9hx24jj55z7zs0dpl2rpqiapixvr8ci8.ls error="server returned 404: 404 page not found\n"15252026/11/11 23:37:26 INFO Completed push id=2 objects=3 duration=3.549388ms15262026/11/11 23:37:26 INFO Upload complete. (57ms)1527=== NAME TestClientIntegration1528 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1529 client_integration_test.go:336: Decompressed .ls content (49 bytes):1530 {"version":1,"root":{"type":"regular","size":39}}15312026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/1/complete15322026/11/11 23:37:26 INFO Completed push id=1 objects=4 duration=7.682206ms15332026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes15342026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes15352026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15362026/11/11 23:37:26 INFO Signed narinfos id=1 count=115372026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15382026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15392026/11/11 23:37:26 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"15402026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15412026/11/11 23:37:26 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"15422026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15432026/11/11 23:37:26 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=115442026/11/11 23:37:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15452026/11/11 23:37:26 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1546--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.28s)1547=== CONT TestObjectStatsUpdatedOncePerCommit1548=== RUN TestPush_RejectsBadRequests/no_roots1549=== PAUSE TestPush_RejectsBadRequests/no_roots1550=== RUN TestPush_RejectsBadRequests/no_objects1551=== PAUSE TestPush_RejectsBadRequests/no_objects1552=== RUN TestPush_RejectsBadRequests/bad_root1553=== PAUSE TestPush_RejectsBadRequests/bad_root1554=== RUN TestPush_RejectsBadRequests/root_not_in_objects1555=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1556=== CONT TestObjectStatsTrigger15572026/11/11 23:37:26 INFO Aborted multipart uploads count=0 kept=015582026/11/11 23:37:26 WARN Force mode enabled - objects will be deleted immediately without grace period15592026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes15602026/11/11 23:37:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=015612026/11/11 23:37:26 INFO Vacuumed table table=pending_closures15622026/11/11 23:37:26 INFO Vacuumed table table=pending_objects15632026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes15642026/11/11 23:37:26 INFO Vacuumed table table=multipart_uploads15652026/11/11 23:37:26 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15662026/11/11 23:37:26 INFO Vacuumed table table=closures15672026/11/11 23:37:26 INFO Vacuumed table table=objects15682026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/2/complete15692026/11/11 23:37:26 INFO Received push request method=POST path=/api/pushes15702026/11/11 23:37:26 INFO Received complete push request method=POST path=/api/pushes/3/complete15712026/11/11 23:37:26 WARN Failed to register uploaded object key=9hx24jj55z7zs0dpl2rpqiapixvr8ci8.ls error="server returned 404: 404 page not found\n"15722026/11/11 23:37:26 INFO Completed push id=3 objects=3 duration=2.981459ms15732026/11/11 23:37:26 INFO Upload complete. (51ms)15742026/11/11 23:37:26 INFO Completed push id=2 objects=4 duration=5.597037ms1575--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.38s)1576=== CONT TestPendingCleanupUsesOneCutoff1577=== NAME TestClientIntegration1578 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1579 client_integration_test.go:353: Decompressed .ls content (49 bytes):1580 {"version":1,"root":{"type":"regular","size":39}}1581 client_integration_test.go:364: Testing garbage collection...15822026-11-11 23:37:26.974 UTC [1882] ERROR: could not obtain lock on row in relation "closures"15832026-11-11 23:37:26.974 UTC [1882] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15842026-11-11 23:37:26.974 UTC [1882] ERROR: could not obtain lock on row in relation "closures"15852026-11-11 23:37:26.974 UTC [1882] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15862026-11-11 23:37:26.975 UTC [1882] ERROR: could not obtain lock on row in relation "closures"15872026-11-11 23:37:26.975 UTC [1882] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15882026-11-11 23:37:26.975 UTC [1882] ERROR: could not obtain lock on row in relation "closures"15892026-11-11 23:37:26.975 UTC [1882] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15902026-11-11 23:37:26.975 UTC [1882] ERROR: could not obtain lock on row in relation "closures"15912026-11-11 23:37:26.975 UTC [1882] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1592--- PASS: TestPush_RootsAreLockedInByteOrder (0.33s)1593=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded15942026/11/11 23:37:26 INFO lead: released remote=192.0.2.1:1234 generation=11595--- PASS: TestLeadEndsOnShutdown (1.94s)1596=== CONT TestMultipartCleanup15972026/11/11 23:37:26 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"15982026/11/11 23:37:26 WARN mTLS auth: bound subjects configured but subject DN unavailable15992026/11/11 23:37:26 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1600--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.31s)1601=== CONT TestServerTLSConfig1602=== RUN TestServerTLSConfig/no_client_CA1603=== PAUSE TestServerTLSConfig/no_client_CA1604=== RUN TestServerTLSConfig/missing_CA_file1605=== PAUSE TestServerTLSConfig/missing_CA_file1606=== RUN TestServerTLSConfig/not_a_PEM_file1607=== PAUSE TestServerTLSConfig/not_a_PEM_file1608=== CONT TestService_NativeMTLS16092026/11/11 23:37:27 INFO Starting cleanup of old closures method=DELETE path=/api/closures16102026/11/11 23:37:27 INFO Garbage collection started16112026/11/11 23:37:27 INFO Aborted multipart uploads count=0 kept=01612--- PASS: TestService_ReadAuthMiddleware (0.34s)1613=== CONT TestMetricsCommit16142026/11/11 23:37:27 WARN Force mode enabled - objects will be deleted immediately without grace period1615--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.36s)1616=== CONT TestMetricsInventory16172026/11/11 23:37:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016182026/11/11 23:37:27 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1619--- PASS: TestService_AuthMiddleware (0.37s)1620=== CONT TestNARDeduplicationMetadataUploadBug16212026/11/11 23:37:27 INFO Vacuumed table table=pending_closures16222026/11/11 23:37:27 INFO Vacuumed table table=pending_objects16232026/11/11 23:37:27 INFO Vacuumed table table=multipart_uploads16242026/11/11 23:37:27 INFO Vacuumed table table=closures16252026/11/11 23:37:27 INFO Vacuumed table table=objects16262026/11/11 23:37:27 INFO lead: released remote=192.0.2.1:1234 generation=116272026/11/11 23:37:27 INFO lead: acquired remote=192.0.2.1:1234 generation=216282026/11/11 23:37:27 WARN readiness check failed error="closed pool"1629--- PASS: TestService_readinessHandler (0.40s)1630=== CONT TestCreatePendingClosureRejectsOversizedNAR16312026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures1632--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1633=== CONT TestCacheConfigHandlerMaxNarSize1634--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1635=== CONT TestEnsureAbortUploadsRule16362026/11/11 23:37:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16372026/11/11 23:37:27 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux16382026/11/11 23:37:27 WARN Refused reserved pin name=worker-x86_64-linux16392026/11/11 23:37:27 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux16402026/11/11 23:37:27 INFO Received create pin request method=POST path=/api/pins/my-app16412026/11/11 23:37:27 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux16422026/11/11 23:37:27 INFO lead: released remote=192.0.2.1:1234 generation=11643--- PASS: TestCreatePin_ReservedPins (0.43s)1644=== CONT TestGenerateLandingPage1645--- PASS: TestGenerateLandingPage (0.00s)1646=== CONT TestPush_RepeatedRootCommitsOnce16472026/11/11 23:37:27 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NmY5ODliZjctNzJiZC00OTRiLWJjNWUtYTQ5MTA4Yzk5YjYxLjNlZjE3ZTc4LWQwY2QtNGNjZC04NTIwLWU2ZTc2MWUzNTViMngxNzk0NDQwMjQ2NjMwNzIxMzM5 parts=1216482026/11/11 23:37:27 INFO lead: acquired remote=192.0.2.1:1234 generation=216492026/11/11 23:37:27 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=216502026/11/11 23:37:27 INFO lead: acquired remote=192.0.2.1:1234 generation=316512026-11-11 23:37:27.286 UTC [1889] ERROR: Push does not exist: id=116522026-11-11 23:37:27.286 UTC [1889] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE16532026-11-11 23:37:27.286 UTC [1889] STATEMENT: SELECT commit_push($1)1654--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.64s)1655=== CONT TestGCTaskStore_DeduplicateSameParams1656--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1657=== CONT TestService_healthCheckHandler1658--- PASS: TestObjectStatsUpdatedOncePerCommit (0.43s)1659=== CONT TestGracefulShutdownDrainsInflight16602026/11/11 23:37:27 INFO Starting HTTP server address=127.0.0.1:4066316612026/11/11 23:37:27 INFO Shutdown signal received, draining in-flight requests timeout=10s1662--- PASS: TestObjectStatsTrigger (0.42s)1663=== CONT TestGCTaskStore_Fail1664--- PASS: TestGCTaskStore_Fail (0.00s)1665=== CONT TestGCTaskStore_PhaseUpdates1666--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1667=== CONT TestGCTaskStore_CompletedAllowsNewTask1668--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1669=== CONT TestGCTaskStore_GetReturnsLatest1670--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1671=== CONT TestGCTaskStore_GetEmpty1672--- PASS: TestGCTaskStore_GetEmpty (0.00s)1673=== CONT TestGCTaskStore_ConflictDifferentParams1674--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1675=== CONT TestGCEndsOnShutdown1676=== RUN TestGCEndsOnShutdown/before_the_run1677=== PAUSE TestGCEndsOnShutdown/before_the_run1678=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1679=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1680=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1681=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1682=== CONT TestGCTaskStore_StartNew1683--- PASS: TestGCTaskStore_StartNew (0.00s)1684=== CONT TestRegisterCompletedObjectNeedsPendingRow16852026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures16862026/11/11 23:37:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16872026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures16882026/11/11 23:37:27 INFO lead: released remote=192.0.2.1:1234 generation=116892026/11/11 23:37:27 INFO Received cleanup request method=DELETE path=/api/pending_closures16902026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures16912026/11/11 23:37:27 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NmY5ODliZjctNzJiZC00OTRiLWJjNWUtYTQ5MTA4Yzk5YjYxLmQ5Nzk4Zjk4LTFhYzEtNDYyYS05Y2RjLTNjZDI1YjQ4ZmExOHgxNzk0NDQwMjQ2ODA0NDcwODQ1 parts=1216922026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures1693--- PASS: TestCompletedNarNotReofferedAcrossClosures (0.81s)1694=== CONT TestGetExistingObjectsLocksRows1695--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1696=== CONT TestCommitRacingPendingCleanupKeepsObjects16972026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures16982026/11/11 23:37:27 INFO lead: acquired remote=192.0.2.1:1234 generation=21699--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.43s)1700=== CONT TestPushWaitsForSweepHoldingRow17012026/11/11 23:37:27 WARN mTLS auth: subject not in bound subjects subject="CN=reader"17022026/11/11 23:37:27 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1703--- PASS: TestService_NativeMTLS (0.42s)1704=== CONT TestSweepSkipsRowsLockedByPush17052026/11/11 23:37:27 INFO Received push request method=POST path=/api/pushes17062026/11/11 23:37:27 INFO Received cleanup request method=DELETE path=/api/pending_closures17072026/11/11 23:37:27 INFO Received complete push request method=POST path=/api/pushes/1/complete17082026/11/11 23:37:27 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket78/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""17092026/11/11 23:37:27 INFO Aborted multipart uploads count=0 kept=117102026/11/11 23:37:27 INFO Received cleanup request method=DELETE path=/api/pending_closures17112026/11/11 23:37:27 INFO Completed push id=1 objects=4 duration=7.633056ms17122026/11/11 23:37:27 INFO Aborted multipart uploads count=1 kept=017132026/11/11 23:37:27 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17142026/11/11 23:37:27 INFO Completed upload id=2 objects=2 duration=1.992429ms1715--- PASS: TestMultipartCleanup (0.51s)1716=== CONT TestFailedS3DeleteKeepsTombstone1717--- PASS: TestMetricsCommit (0.46s)1718=== CONT TestSweepRechecksPendingAfterLocking1719--- PASS: TestMetricsInventory (0.46s)1720=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale17212026/11/11 23:37:27 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket84 days=21722--- PASS: TestEnsureAbortUploadsRule (0.44s)1723=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1724=== NAME TestNARDeduplicationMetadataUploadBug1725 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug1257410539/001/store/nji02cg2wqcv95w98qamf5r5f5l39ilh-file1.txt17262026/11/11 23:37:27 INFO Received push request method=POST path=/api/pushes17272026/11/11 23:37:27 INFO Received complete push request method=POST path=/api/pushes/1/complete1728--- PASS: TestService_healthCheckHandler (0.34s)1729=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows17302026/11/11 23:37:27 INFO Completed push id=1 objects=4 duration=8.011886ms1731--- PASS: TestPush_RepeatedRootCommitsOnce (0.45s)1732=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check17332026/11/11 23:37:27 INFO Received push request method=POST path=/api/pushes17342026/11/11 23:37:27 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17352026/11/11 23:37:27 INFO Uploading nji02cg2wqcv95w98qamf5r5f5l39ilh-file1.txt (160B)17362026/11/11 23:37:27 WARN Failed to register uploaded object key=nji02cg2wqcv95w98qamf5r5f5l39ilh.ls error="server returned 404: 404 page not found\n"17372026/11/11 23:37:27 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17382026/11/11 23:37:27 INFO Signed narinfos id=1 count=117392026/11/11 23:37:27 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"17402026/11/11 23:37:27 INFO Uploading 1 narinfos1741--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.35s)1742=== CONT TestIsValidCachePath/narinfo1743=== CONT TestIsValidCachePath/short_hash1744=== CONT TestIsValidCachePath/wrong_extension1745=== CONT TestIsValidCachePath/leading_slash1746=== CONT TestIsValidCachePath/empty1747=== CONT TestIsValidCachePath/random_path1748=== CONT TestIsValidCachePath/invalid_char_u1749=== CONT TestIsValidCachePath/invalid_char_e1750=== CONT TestIsValidCachePath/traversal_in_middle1751=== CONT TestIsValidCachePath/traversal_parent1752=== CONT TestIsValidCachePath/index.html1753=== CONT TestIsValidCachePath/nix-cache-info1754=== CONT TestIsValidCachePath/realisation17552026/11/11 23:37:27 INFO Received complete push request method=POST path=/api/pushes/1/complete1756=== CONT TestIsValidCachePath/log1757=== CONT TestIsValidCachePath/ls1758=== CONT TestIsValidCachePath/nar_uncompressed17592026/11/11 23:37:27 WARN Failed to register uploaded object key=nji02cg2wqcv95w98qamf5r5f5l39ilh.narinfo error="server returned 404: 404 page not found\n"1760=== CONT TestIsValidCachePath/nar_bz21761=== CONT TestIsValidCachePath/nar_xz1762=== CONT TestIsValidCachePath/nar_zst1763=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1764--- PASS: TestIsValidCachePath (0.00s)1765 --- PASS: TestIsValidCachePath/narinfo (0.00s)1766 --- PASS: TestIsValidCachePath/short_hash (0.00s)1767 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1768 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1769 --- PASS: TestIsValidCachePath/empty (0.00s)1770 --- PASS: TestIsValidCachePath/random_path (0.00s)1771 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1772 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1773 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1774 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1775 --- PASS: TestIsValidCachePath/index.html (0.00s)1776 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1777 --- PASS: TestIsValidCachePath/realisation (0.00s)1778 --- PASS: TestIsValidCachePath/log (0.00s)1779 --- PASS: TestIsValidCachePath/ls (0.00s)1780 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1781 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1782 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1783 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1784 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1785=== CONT TestParseSingleRange/none1786=== CONT TestParseSingleRange/open-ended1787=== CONT TestParseSingleRange/start_far_past_EOF1788=== CONT TestParseSingleRange/start_past_EOF1789=== CONT TestParseSingleRange/single_byte1790=== CONT TestParseSingleRange/suffix_exceeds_size1791=== CONT TestParseSingleRange/suffix1792=== CONT TestParseSingleRange/end_clamped_to_size1793=== CONT TestParseSingleRange/malformed_both_empty1794=== CONT TestParseSingleRange/closed1795=== CONT TestParseSingleRange/malformed_end_before_start1796=== CONT TestParseSingleRange/multi-range_ignored1797=== CONT TestParseSingleRange/malformed_no_dash1798=== CONT TestParseSingleRange/unknown_unit1799--- PASS: TestParseSingleRange (0.00s)1800 --- PASS: TestParseSingleRange/none (0.00s)1801 --- PASS: TestParseSingleRange/open-ended (0.00s)1802 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1803 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1804 --- PASS: TestParseSingleRange/single_byte (0.00s)1805 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1806 --- PASS: TestParseSingleRange/suffix (0.00s)1807 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1808 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1809 --- PASS: TestParseSingleRange/closed (0.00s)1810 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1811 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1812 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1813 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1814=== CONT TestResolveDBConnectionString/flag_wins1815=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1816=== CONT TestResolveDBConnectionString/nothing_configured1817=== CONT TestResolveDBConnectionString/missing_file_is_an_error1818=== CONT TestResolveDBConnectionString/file_when_flag_empty1819=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1820--- PASS: TestResolveDBConnectionString (0.00s)1821 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1822 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1823 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1824 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1825 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1826=== NAME TestOrphanedObjectsGC1827 orphaned_objects_gc_test.go:287: GC Test Summary:1828 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1829 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1830 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1831 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1832 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1833--- PASS: TestOrphanedObjectsGC (0.86s)1834=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed18352026/11/11 23:37:27 INFO Completed push id=1 objects=3 duration=9.124015ms18362026/11/11 23:37:27 INFO Upload complete. (72ms)1837=== NAME TestNARDeduplicationMetadataUploadBug1838 metadata_upload_test.go:54: Retrieved narinfo from S3:1839 StorePath: /build/TestNARDeduplicationMetadataUploadBug1257410539/001/store/nji02cg2wqcv95w98qamf5r5f5l39ilh-file1.txt1840 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1841 Compression: zstd1842 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1843 NarSize: 1601844 References: 1845 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1846 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1847 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1848 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1849--- PASS: TestGetExistingObjectsLocksRows (0.34s)1850=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1851=== NAME TestNARDeduplicationMetadataUploadBug1852 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug1257410539/001/store/rwiffg157cbj1bq0sshi3snahjfbvqpg-file2.txt18532026/11/11 23:37:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18542026/11/11 23:37:27 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NmY5ODliZjctNzJiZC00OTRiLWJjNWUtYTQ5MTA4Yzk5YjYxLmU2ZjQxMzEyLTUxOTQtNDFkOC1hYWQyLTQ5MjQ3ZmZmZWFlMHgxNzk0NDQwMjQ2NjM5MzY4NjU0 parts=121855--- PASS: TestSharedNarBothUploadsComplete (1.57s)1856=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost18572026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures18582026/11/11 23:37:27 INFO Received push request method=POST path=/api/pushes18592026/11/11 23:37:27 INFO Aborted multipart uploads count=0 kept=018602026/11/11 23:37:27 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18612026/11/11 23:37:27 WARN Force mode enabled - objects will be deleted immediately without grace period18622026/11/11 23:37:27 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign18632026/11/11 23:37:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=018642026/11/11 23:37:27 INFO Signed narinfos id=2 count=118652026/11/11 23:37:27 WARN Failed to register uploaded object key=rwiffg157cbj1bq0sshi3snahjfbvqpg.ls error="server returned 404: 404 page not found\n"18662026/11/11 23:37:27 INFO Uploading 1 narinfos18672026/11/11 23:37:27 INFO Vacuumed table table=pending_closures18682026/11/11 23:37:27 INFO Received complete push request method=POST path=/api/pushes/2/complete18692026/11/11 23:37:27 INFO Vacuumed table table=pending_objects18702026/11/11 23:37:27 WARN Failed to register uploaded object key=rwiffg157cbj1bq0sshi3snahjfbvqpg.narinfo error="server returned 404: 404 page not found\n"18712026/11/11 23:37:27 INFO Vacuumed table table=multipart_uploads18722026/11/11 23:37:27 INFO Vacuumed table table=closures18732026/11/11 23:37:27 INFO Vacuumed table table=objects18742026/11/11 23:37:27 INFO Completed push id=2 objects=3 duration=3.213188ms18752026/11/11 23:37:27 INFO Upload complete. (58ms)1876=== NAME TestNARDeduplicationMetadataUploadBug1877 metadata_upload_test.go:76: Retrieved narinfo from S3:1878 StorePath: /build/TestNARDeduplicationMetadataUploadBug1257410539/001/store/rwiffg157cbj1bq0sshi3snahjfbvqpg-file2.txt1879 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1880 Compression: zstd1881 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1882 NarSize: 1601883 References: 1884 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1885 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1886 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1887 {"version":1,"root":{"type":"regular","size":44}}18882026/11/11 23:37:27 INFO Aborted multipart uploads count=0 kept=01889--- PASS: TestNARDeduplicationMetadataUploadBug (0.72s)1890=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1891--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.43s)1892=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info18932026/11/11 23:37:27 INFO Received uploads request method=POST path=/1894=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key18952026/11/11 23:37:27 INFO Received complete multipart upload request method=POST path=/1896=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key18972026/11/11 23:37:27 INFO Received request for more parts method=POST path=/1898=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18992026/11/11 23:37:27 INFO Received uploads request method=POST path=/1900--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1901 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1902 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1903 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1904 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1905=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19062026/11/11 23:37:27 INFO Received uploads request method=POST path=/19072026/11/11 23:37:27 WARN Force mode enabled - objects will be deleted immediately without grace period19082026/11/11 23:37:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019092026/11/11 23:37:27 INFO Vacuumed table table=pending_closures19102026/11/11 23:37:27 INFO Vacuumed table table=pending_objects19112026/11/11 23:37:27 INFO Vacuumed table table=multipart_uploads19122026/11/11 23:37:27 INFO Vacuumed table table=closures19132026/11/11 23:37:27 INFO Vacuumed table table=objects19142026/11/11 23:37:27 INFO Aborted multipart uploads count=0 kept=019152026/11/11 23:37:27 WARN Force mode enabled - objects will be deleted immediately without grace period19162026/11/11 23:37:27 INFO Aborted multipart uploads count=0 kept=019172026/11/11 23:37:27 WARN Force mode enabled - objects will be deleted immediately without grace period19182026/11/11 23:37:27 INFO Received uploads request method=POST path=/api/pending_closures19192026/11/11 23:37:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019202026/11/11 23:37:27 INFO Vacuumed table table=pending_closures19212026/11/11 23:37:27 INFO Vacuumed table table=pending_objects19222026/11/11 23:37:27 INFO Vacuumed table table=multipart_uploads19232026/11/11 23:37:27 INFO Vacuumed table table=closures19242026/11/11 23:37:27 INFO Vacuumed table table=objects19252026/11/11 23:37:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=019262026/11/11 23:37:27 INFO Vacuumed table table=pending_closures1927--- PASS: TestSweepRechecksPendingAfterLocking (0.38s)19282026/11/11 23:37:27 INFO Vacuumed table table=pending_objects1929=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19302026/11/11 23:37:27 INFO Received request for more parts method=POST path=/19312026/11/11 23:37:27 INFO Vacuumed table table=multipart_uploads19322026/11/11 23:37:27 INFO Vacuumed table table=closures19332026/11/11 23:37:27 INFO Vacuumed table table=objects1934--- PASS: TestSweepSkipsRowsLockedByPush (0.46s)1935=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19362026/11/11 23:37:27 INFO Received complete multipart upload request method=POST path=/1937=== CONT TestIsValidUploadKey/narinfo1938=== CONT TestIsValidUploadKey/nix-cache-info1939=== CONT TestIsValidUploadKey/realisation_plus_in_output1940=== CONT TestIsValidUploadKey/realisation1941=== CONT TestIsValidUploadKey/build_log_equals1942=== CONT TestIsValidUploadKey/unknown_type1943=== CONT TestIsValidUploadKey/empty_key1944=== CONT TestIsValidUploadKey/build_log_question_mark1945=== CONT TestIsValidUploadKey/build_log_plus_in_name1946=== CONT TestIsValidUploadKey/absolute1947=== CONT TestIsValidUploadKey/build_log_home-manager_file1948=== CONT TestIsValidUploadKey/traversal_nar1949=== CONT TestIsValidUploadKey/build_log1950=== CONT TestIsValidUploadKey/traversal1951=== CONT TestIsValidUploadKey/nar_xz1952=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1953=== CONT TestIsValidUploadKey/nar_zst1954=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1955=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1956=== CONT TestIsValidUploadKey/index.html1957=== CONT TestIsValidUploadKey/nar_plain1958=== CONT TestIsValidUploadKey/listing1959--- PASS: TestIsValidUploadKey (0.17s)1960 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1961 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1962 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1963 --- PASS: TestIsValidUploadKey/realisation (0.00s)1964 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1965 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1966 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1967 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1968 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1969 --- PASS: TestIsValidUploadKey/absolute (0.00s)1970 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1971 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1972 --- PASS: TestIsValidUploadKey/build_log (0.00s)1973 --- PASS: TestIsValidUploadKey/traversal (0.00s)1974 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1975 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1976 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1977 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1978 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1979 --- PASS: TestIsValidUploadKey/index.html (0.00s)1980 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1981 --- PASS: TestIsValidUploadKey/listing (0.00s)1982=== CONT TestPendingClosureWriteTimeout/empty1983=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline19842026/11/11 23:37:27 INFO Aborted multipart uploads count=0 kept=01985=== CONT TestPendingClosureWriteTimeout/670k_objects1986=== CONT TestPendingClosureWriteTimeout/400_objects1987=== CONT TestProxyWriteTimeout/narinfo1988=== CONT TestProxyWriteTimeout/10_GiB_nar1989=== CONT TestProxyWriteTimeout/unknown_size1990=== CONT TestProxyWriteTimeout/1_GiB_nar1991--- PASS: TestProxyWriteTimeout (0.00s)1992 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1993 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1994 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1995 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1996=== CONT TestClientErrorHandling/InvalidStorePath19972026/11/11 23:37:27 WARN Force mode enabled - objects will be deleted immediately without grace period19982026/11/11 23:37:27 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."19992026/11/11 23:37:27 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."2000--- PASS: TestFailedS3DeleteKeepsTombstone (0.47s)
loading 619 hidden lines…
26202026/11/11 23:37:41 INFO Vacuumed table table=closures26212026/11/11 23:37:41 INFO Received uploads request method=POST path=/26222026/11/11 23:37:41 INFO Vacuumed table table=objects26232026/11/11 23:37:41 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."26242026/11/11 23:37:41 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."26252026/11/11 23:37:41 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."26262026/11/11 23:37:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."26272026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."26282026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."26292026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."26302026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."26312026/11/11 23:37:41 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."26322026/11/11 23:37:41 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."26332026/11/11 23:37:41 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."26342026/11/11 23:37:41 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."26352026/11/11 23:37:41 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."26362026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."26372026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."26382026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."26392026/11/11 23:37:41 INFO Completed push id=1 objects=5 duration=7.370836ms26402026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=026412026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period26422026/11/11 23:37:41 INFO Received uploads request method=POST path=/26432026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026442026/11/11 23:37:41 INFO Vacuumed table table=pending_closures26452026/11/11 23:37:41 INFO Received push request method=POST path=/26462026/11/11 23:37:41 INFO Received complete upload request method=POST path=/26472026/11/11 23:37:41 INFO Vacuumed table table=pending_objects26482026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads26492026/11/11 23:37:41 INFO Vacuumed table table=closures26502026/11/11 23:37:41 INFO Vacuumed table table=objects26512026/11/11 23:37:41 INFO Completed upload id=11 objects=14 duration=1.994309ms26522026/11/11 23:37:41 INFO Received push request method=POST path=/26532026/11/11 23:37:41 INFO Received uploads request method=POST path=/26542026/11/11 23:37:41 INFO Received uploads request method=POST path=/26552026/11/11 23:37:41 INFO Received push request method=POST path=/26562026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=026572026/11/11 23:37:41 INFO Received push request method=POST path=/26582026/11/11 23:37:41 INFO Received uploads request method=POST path=/26592026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=026602026/11/11 23:37:41 INFO Received complete push request method=POST path=/26612026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=026622026/11/11 23:37:41 INFO Completed push id=7 objects=20 duration=3.067239ms26632026/11/11 23:37:41 INFO Received push request method=POST path=/26642026/11/11 23:37:41 INFO Received push request method=POST path=/26652026/11/11 23:37:41 INFO Received push request method=POST path=/26662026/11/11 23:37:41 INFO Received push request method=POST path=/26672026/11/11 23:37:41 INFO Received push request method=POST path=/26682026/11/11 23:37:41 INFO Received push request method=POST path=/26692026/11/11 23:37:41 INFO Vacuumed table table=pending_closures26702026/11/11 23:37:41 INFO Received uploads request method=POST path=/26712026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=026722026/11/11 23:37:41 INFO Vacuumed table table=pending_objects26732026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads26742026/11/11 23:37:41 INFO Vacuumed table table=closures26752026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period26762026/11/11 23:37:41 INFO Vacuumed table table=objects26772026/11/11 23:37:41 INFO Received complete upload request method=POST path=/26782026/11/11 23:37:41 INFO Completed upload id=9 objects=7 duration=2.579648ms26792026/11/11 23:37:41 INFO Received uploads request method=POST path=/26802026/11/11 23:37:41 INFO Received uploads request method=POST path=/26812026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=026822026/11/11 23:37:41 INFO Vacuumed table table=pending_closures26832026/11/11 23:37:41 INFO Received push request method=POST path=/26842026/11/11 23:37:41 INFO Vacuumed table table=pending_objects26852026/11/11 23:37:41 INFO Received push request method=POST path=/26862026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads26872026/11/11 23:37:41 INFO Received complete upload request method=POST path=/26882026/11/11 23:37:41 INFO Vacuumed table table=closures26892026/11/11 23:37:41 INFO Completed upload id=2 objects=2 duration=2.486709ms26902026/11/11 23:37:41 INFO Vacuumed table table=objects26912026/11/11 23:37:41 INFO Received push request method=POST path=/26922026/11/11 23:37:41 INFO Received complete upload request method=POST path=/26932026/11/11 23:37:41 INFO Completed upload id=2 objects=7 duration=5.338347ms26942026/11/11 23:37:41 INFO Received push request method=POST path=/26952026/11/11 23:37:41 INFO Received push request method=POST path=/26962026/11/11 23:37:41 INFO Received push request method=POST path=/26972026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=026982026/11/11 23:37:41 INFO Received push request method=POST path=/26992026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027002026/11/11 23:37:41 INFO Received push request method=POST path=/27012026/11/11 23:37:41 INFO Vacuumed table table=pending_closures27022026/11/11 23:37:41 INFO Vacuumed table table=pending_objects27032026/11/11 23:37:41 INFO Received uploads request method=POST path=/27042026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads27052026/11/11 23:37:41 INFO Vacuumed table table=closures27062026/11/11 23:37:41 INFO Received push request method=POST path=/27072026/11/11 23:37:41 INFO Received complete push request method=POST path=/27082026/11/11 23:37:41 INFO Vacuumed table table=objects27092026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=027102026/11/11 23:37:41 INFO Received push request method=POST path=/27112026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period27122026/11/11 23:37:41 INFO Completed push id=6 objects=5 duration=2.198909ms27132026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027142026/11/11 23:37:41 INFO Vacuumed table table=pending_closures27152026/11/11 23:37:41 INFO Vacuumed table table=pending_objects27162026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads27172026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=027182026/11/11 23:37:41 INFO Vacuumed table table=closures27192026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=027202026/11/11 23:37:41 INFO Received complete push request method=POST path=/27212026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=027222026/11/11 23:37:41 INFO Vacuumed table table=objects27232026/11/11 23:37:41 INFO Received complete upload request method=POST path=/27242026/11/11 23:37:41 INFO Completed push id=1 objects=2 duration=2.948569ms27252026/11/11 23:37:41 INFO Completed upload id=5 objects=5 duration=2.080329ms27262026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027272026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period27282026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=027292026/11/11 23:37:41 INFO Vacuumed table table=pending_closures27302026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period27312026/11/11 23:37:41 INFO Vacuumed table table=pending_objects27322026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads27332026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27342026/11/11 23:37:41 INFO Vacuumed table table=closures27352026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027362026/11/11 23:37:41 INFO Vacuumed table table=objects27372026/11/11 23:37:41 INFO Vacuumed table table=pending_closures27382026/11/11 23:37:41 INFO Vacuumed table table=pending_objects27392026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads27402026/11/11 23:37:41 INFO Received push request method=POST path=/27412026/11/11 23:37:41 INFO Vacuumed table table=closures27422026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=027432026/11/11 23:37:41 INFO Vacuumed table table=objects27442026/11/11 23:37:41 INFO Received uploads request method=POST path=/27452026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period27462026/11/11 23:37:41 INFO Received uploads request method=POST path=/27472026/11/11 23:37:41 INFO Received complete push request method=POST path=/27482026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027492026/11/11 23:37:41 INFO Received complete upload request method=POST path=/27502026/11/11 23:37:41 INFO Completed push id=5 objects=3 duration=4.447737ms27512026/11/11 23:37:41 INFO Vacuumed table table=pending_closures27522026/11/11 23:37:41 INFO Completed upload id=5 objects=7 duration=3.243379ms27532026/11/11 23:37:41 INFO Received push request method=POST path=/27542026/11/11 23:37:41 INFO Received uploads request method=POST path=/27552026/11/11 23:37:41 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."27562026/11/11 23:37:41 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27572026/11/11 23:37:41 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27582026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27592026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27602026/11/11 23:37:41 INFO Vacuumed table table=pending_objects27612026/11/11 23:37:41 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27622026/11/11 23:37:41 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27632026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27642026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27652026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads27662026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27672026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."27682026/11/11 23:37:41 INFO Received complete push request method=POST path=/27692026/11/11 23:37:41 INFO Vacuumed table table=closures27702026/11/11 23:37:41 INFO Vacuumed table table=objects27712026/11/11 23:37:41 INFO Received complete push request method=POST path=/27722026/11/11 23:37:41 INFO Completed push id=2 objects=7 duration=3.044139ms27732026/11/11 23:37:41 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."27742026/11/11 23:37:41 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."27752026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27762026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27772026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27782026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27792026/11/11 23:37:41 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27802026/11/11 23:37:41 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27812026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27822026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27832026/11/11 23:37:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."27842026/11/11 23:37:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."27852026/11/11 23:37:41 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."27862026/11/11 23:37:41 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27872026/11/11 23:37:41 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27882026/11/11 23:37:41 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27892026/11/11 23:37:41 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27902026/11/11 23:37:41 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."27912026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27922026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27932026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."27942026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."27952026/11/11 23:37:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27962026/11/11 23:37:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27972026/11/11 23:37:41 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27982026/11/11 23:37:41 INFO Completed push id=4 objects=4 duration=3.458658ms27992026/11/11 23:37:41 INFO Received push request method=POST path=/28002026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028012026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028022026/11/11 23:37:41 INFO Vacuumed table table=pending_closures28032026/11/11 23:37:41 INFO Received push request method=POST path=/28042026/11/11 23:37:41 INFO Vacuumed table table=pending_objects28052026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads28062026/11/11 23:37:41 INFO Vacuumed table table=closures28072026/11/11 23:37:41 INFO Vacuumed table table=objects28082026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028092026/11/11 23:37:41 INFO Received uploads request method=POST path=/28102026/11/11 23:37:41 INFO Received push request method=POST path=/28112026/11/11 23:37:41 INFO Received uploads request method=POST path=/28122026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028132026/11/11 23:37:41 INFO Received uploads request method=POST path=/28142026/11/11 23:37:41 INFO Received complete push request method=POST path=/28152026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028162026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028172026/11/11 23:37:41 INFO Received complete upload request method=POST path=/28182026/11/11 23:37:41 INFO Vacuumed table table=pending_closures28192026/11/11 23:37:41 INFO Received uploads request method=POST path=/28202026/11/11 23:37:41 INFO Completed upload id=6 objects=9 duration=1.700849ms28212026/11/11 23:37:41 INFO Completed push id=6 objects=5 duration=4.122948ms28222026/11/11 23:37:41 INFO Vacuumed table table=pending_objects28232026/11/11 23:37:41 INFO Received push request method=POST path=/28242026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads28252026/11/11 23:37:41 INFO Vacuumed table table=closures28262026/11/11 23:37:41 INFO Vacuumed table table=objects28272026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028282026/11/11 23:37:41 INFO Received uploads request method=POST path=/28292026/11/11 23:37:41 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28302026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28312026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28322026/11/11 23:37:41 INFO Received uploads request method=POST path=/28332026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028342026/11/11 23:37:41 INFO Vacuumed table table=pending_closures28352026/11/11 23:37:41 INFO Vacuumed table table=pending_objects28362026/11/11 23:37:41 INFO Received complete push request method=POST path=/28372026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads28382026/11/11 23:37:41 INFO Received push request method=POST path=/28392026/11/11 23:37:41 INFO Vacuumed table table=closures28402026/11/11 23:37:41 INFO Received push request method=POST path=/28412026/11/11 23:37:41 INFO Completed push id=6 objects=6 duration=2.250188ms28422026/11/11 23:37:41 INFO Vacuumed table table=objects28432026/11/11 23:37:41 INFO Received complete push request method=POST path=/28442026/11/11 23:37:41 INFO Completed push id=8 objects=2 duration=2.164839ms28452026/11/11 23:37:41 INFO Received push request method=POST path=/28462026/11/11 23:37:41 INFO Received uploads request method=POST path=/28472026/11/11 23:37:41 INFO Received complete push request method=POST path=/28482026/11/11 23:37:41 INFO Completed push id=14 objects=3 duration=3.450598ms28492026/11/11 23:37:41 INFO Received uploads request method=POST path=/28502026/11/11 23:37:41 INFO Received uploads request method=POST path=/28512026/11/11 23:37:41 INFO Received uploads request method=POST path=/28522026/11/11 23:37:41 INFO Received uploads request method=POST path=/28532026/11/11 23:37:41 INFO Received push request method=POST path=/28542026/11/11 23:37:41 INFO Received push request method=POST path=/28552026/11/11 23:37:41 INFO Received push request method=POST path=/28562026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028572026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028582026/11/11 23:37:41 INFO Vacuumed table table=pending_closures28592026/11/11 23:37:41 INFO Received uploads request method=POST path=/28602026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period28612026/11/11 23:37:41 INFO Vacuumed table table=pending_objects28622026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads28632026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028642026/11/11 23:37:41 INFO Received uploads request method=POST path=/28652026/11/11 23:37:41 INFO Vacuumed table table=pending_closures28662026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028672026/11/11 23:37:41 INFO Vacuumed table table=closures28682026/11/11 23:37:41 INFO Received uploads request method=POST path=/28692026/11/11 23:37:41 INFO Vacuumed table table=pending_objects28702026/11/11 23:37:41 INFO Vacuumed table table=objects28712026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads28722026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period28732026/11/11 23:37:41 INFO Vacuumed table table=closures28742026/11/11 23:37:41 INFO Vacuumed table table=objects28752026/11/11 23:37:41 INFO Received uploads request method=POST path=/28762026/11/11 23:37:41 INFO Received uploads request method=POST path=/28772026/11/11 23:37:41 INFO Received complete push request method=POST path=/28782026/11/11 23:37:41 INFO Completed push id=8 objects=6 duration=1.984499ms28792026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=028802026/11/11 23:37:41 INFO Vacuumed table table=pending_closures28812026/11/11 23:37:41 INFO Vacuumed table table=pending_objects28822026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads28832026/11/11 23:37:41 INFO Vacuumed table table=closures28842026/11/11 23:37:41 INFO Vacuumed table table=objects28852026/11/11 23:37:41 INFO Received uploads request method=POST path=/28862026/11/11 23:37:41 INFO Received complete push request method=POST path=/28872026/11/11 23:37:41 INFO Received complete upload request method=POST path=/28882026/11/11 23:37:41 INFO Received uploads request method=POST path=/28892026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028902026/11/11 23:37:41 INFO Received push request method=POST path=/28912026/11/11 23:37:41 INFO Completed push id=5 objects=6 duration=2.147219ms28922026/11/11 23:37:41 INFO Completed upload id=3 objects=5 duration=2.039779ms28932026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period28942026/11/11 23:37:41 INFO Received complete upload request method=POST path=/28952026/11/11 23:37:41 INFO Completed upload id=10 objects=6 duration=3.023318ms28962026/11/11 23:37:41 INFO Received complete push request method=POST path=/28972026/11/11 23:37:41 INFO Completed push id=10 objects=2 duration=2.466619ms28982026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=028992026/11/11 23:37:41 INFO Received push request method=POST path=/29002026/11/11 23:37:41 INFO Received complete push request method=POST path=/29012026/11/11 23:37:41 INFO Completed push id=7 objects=27 duration=2.477549ms29022026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029032026/11/11 23:37:41 INFO Received push request method=POST path=/29042026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period29052026/11/11 23:37:41 INFO Received push request method=POST path=/29062026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029072026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029082026/11/11 23:37:41 INFO Vacuumed table table=pending_closures29092026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029102026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029112026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period29122026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029132026/11/11 23:37:41 INFO Vacuumed table table=pending_objects29142026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads29152026/11/11 23:37:41 INFO Vacuumed table table=pending_closures29162026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029172026/11/11 23:37:41 INFO Vacuumed table table=closures29182026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period29192026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029202026/11/11 23:37:41 INFO Vacuumed table table=pending_objects29212026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads29222026/11/11 23:37:41 INFO Vacuumed table table=objects29232026/11/11 23:37:41 INFO Received push request method=POST path=/29242026/11/11 23:37:41 INFO Vacuumed table table=pending_closures29252026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period29262026/11/11 23:37:41 INFO Vacuumed table table=closures29272026/11/11 23:37:41 INFO Vacuumed table table=pending_objects29282026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads29292026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029302026/11/11 23:37:41 INFO Vacuumed table table=objects29312026/11/11 23:37:41 INFO Vacuumed table table=closures29322026/11/11 23:37:41 INFO Vacuumed table table=pending_closures29332026/11/11 23:37:41 INFO Received uploads request method=POST path=/29342026/11/11 23:37:41 INFO Vacuumed table table=objects29352026/11/11 23:37:41 INFO Vacuumed table table=pending_objects29362026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads29372026/11/11 23:37:41 INFO Received complete upload request method=POST path=/29382026/11/11 23:37:41 INFO Vacuumed table table=closures29392026/11/11 23:37:41 INFO Completed upload id=12 objects=21 duration=1.3872ms29402026/11/11 23:37:41 INFO Vacuumed table table=objects29412026/11/11 23:37:41 INFO Received push request method=POST path=/29422026/11/11 23:37:41 INFO Received push request method=POST path=/29432026/11/11 23:37:41 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29442026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29452026/11/11 23:37:41 INFO Received uploads request method=POST path=/29462026/11/11 23:37:41 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29472026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29482026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29492026/11/11 23:37:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."29502026/11/11 23:37:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29512026/11/11 23:37:41 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29522026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29532026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29542026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29552026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029562026/11/11 23:37:41 INFO Received uploads request method=POST path=/29572026/11/11 23:37:41 INFO Received complete upload request method=POST path=/29582026/11/11 23:37:41 INFO Received complete push request method=POST path=/29592026/11/11 23:37:41 INFO Received uploads request method=POST path=/29602026/11/11 23:37:41 INFO Completed upload id=3 objects=9 duration=1.534259ms29612026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029622026/11/11 23:37:41 INFO Vacuumed table table=pending_closures29632026/11/11 23:37:41 INFO Vacuumed table table=pending_objects29642026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads29652026/11/11 23:37:41 INFO Received push request method=POST path=/29662026/11/11 23:37:41 INFO Vacuumed table table=closures29672026/11/11 23:37:41 INFO Completed push id=5 objects=9 duration=5.613567ms29682026/11/11 23:37:41 INFO Vacuumed table table=objects29692026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029702026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period29712026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=029722026/11/11 23:37:41 INFO Received uploads request method=POST path=/29732026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029742026/11/11 23:37:41 INFO Vacuumed table table=pending_closures29752026/11/11 23:37:41 INFO Received uploads request method=POST path=/29762026/11/11 23:37:41 INFO Received complete upload request method=POST path=/29772026/11/11 23:37:41 INFO Vacuumed table table=pending_objects29782026/11/11 23:37:41 INFO Received push request method=POST path=/29792026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads29802026/11/11 23:37:41 INFO Received push request method=POST path=/29812026/11/11 23:37:41 INFO Vacuumed table table=closures29822026/11/11 23:37:41 INFO Completed upload id=3 objects=8 duration=3.870298ms29832026/11/11 23:37:41 INFO Vacuumed table table=objects29842026/11/11 23:37:41 INFO Received complete upload request method=POST path=/29852026/11/11 23:37:41 INFO Received push request method=POST path=/29862026/11/11 23:37:41 INFO Completed upload id=4 objects=14 duration=3.456428ms29872026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=029882026/11/11 23:37:41 INFO Vacuumed table table=pending_closures29892026/11/11 23:37:41 INFO Vacuumed table table=pending_objects29902026/11/11 23:37:41 INFO Received complete push request method=POST path=/29912026/11/11 23:37:41 INFO Received complete push request method=POST path=/29922026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads29932026/11/11 23:37:41 INFO Received push request method=POST path=/29942026/11/11 23:37:41 INFO Vacuumed table table=closures29952026/11/11 23:37:41 INFO Completed push id=13 objects=23 duration=2.473119ms29962026/11/11 23:37:41 INFO Vacuumed table table=objects29972026/11/11 23:37:41 INFO Completed push id=2 objects=10 duration=3.678708ms29982026/11/11 23:37:41 INFO Received push request method=POST path=/29992026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30002026/11/11 23:37:41 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."30012026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30022026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30032026/11/11 23:37:41 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."30042026/11/11 23:37:41 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30052026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30062026/11/11 23:37:41 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30072026/11/11 23:37:41 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30082026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."30092026/11/11 23:37:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30102026/11/11 23:37:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30112026/11/11 23:37:41 INFO Received push request method=POST path=/30122026/11/11 23:37:41 INFO Received push request method=POST path=/30132026/11/11 23:37:41 INFO Received push request method=POST path=/30142026/11/11 23:37:41 INFO Received uploads request method=POST path=/30152026/11/11 23:37:41 INFO Received complete push request method=POST path=/30162026/11/11 23:37:41 INFO Completed push id=5 objects=6 duration=1.836279ms30172026/11/11 23:37:41 INFO Received uploads request method=POST path=/30182026/11/11 23:37:41 INFO Received uploads request method=POST path=/30192026/11/11 23:37:41 INFO Received uploads request method=POST path=/30202026/11/11 23:37:41 INFO Received uploads request method=POST path=/30212026/11/11 23:37:41 INFO Received uploads request method=POST path=/30222026/11/11 23:37:41 INFO Received uploads request method=POST path=/30232026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030242026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period30252026/11/11 23:37:41 INFO Received uploads request method=POST path=/30262026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030272026/11/11 23:37:41 INFO Vacuumed table table=pending_closures30282026/11/11 23:37:41 INFO Received push request method=POST path=/30292026/11/11 23:37:41 INFO Vacuumed table table=pending_objects30302026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads30312026/11/11 23:37:41 INFO Vacuumed table table=closures30322026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030332026/11/11 23:37:41 INFO Received complete upload request method=POST path=/30342026/11/11 23:37:41 INFO Vacuumed table table=objects30352026/11/11 23:37:41 INFO Completed upload id=10 objects=17 duration=1.43906ms30362026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period30372026/11/11 23:37:41 INFO Received push request method=POST path=/30382026/11/11 23:37:41 INFO Received complete upload request method=POST path=/30392026/11/11 23:37:41 INFO Completed upload id=10 objects=6 duration=2.931959ms30402026/11/11 23:37:41 INFO Received push request method=POST path=/30412026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030422026/11/11 23:37:41 INFO Received uploads request method=POST path=/30432026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period30442026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030452026/11/11 23:37:41 INFO Received push request method=POST path=/30462026/11/11 23:37:41 INFO Vacuumed table table=pending_closures30472026/11/11 23:37:41 INFO Received uploads request method=POST path=/30482026/11/11 23:37:41 INFO Vacuumed table table=pending_objects30492026/11/11 23:37:41 INFO Received complete upload request method=POST path=/30502026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads30512026/11/11 23:37:41 INFO Vacuumed table table=closures30522026/11/11 23:37:41 INFO Completed upload id=10 objects=23 duration=2.168528ms30532026/11/11 23:37:41 INFO Vacuumed table table=objects30542026/11/11 23:37:41 INFO Received complete upload request method=POST path=/30552026/11/11 23:37:41 INFO Received complete upload request method=POST path=/30562026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030572026/11/11 23:37:41 INFO Received complete push request method=POST path=/30582026/11/11 23:37:41 INFO Received push request method=POST path=/30592026/11/11 23:37:41 INFO Completed upload id=11 objects=6 duration=2.550589ms30602026/11/11 23:37:41 INFO Completed upload id=6 objects=24 duration=3.625128ms30612026/11/11 23:37:41 INFO Completed push id=8 objects=7 duration=2.277389ms30622026/11/11 23:37:41 INFO Received uploads request method=POST path=/30632026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030642026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030652026/11/11 23:37:41 INFO Received complete push request method=POST path=/30662026/11/11 23:37:41 INFO Received push request method=POST path=/30672026/11/11 23:37:41 INFO Vacuumed table table=pending_closures30682026/11/11 23:37:41 INFO Vacuumed table table=pending_objects30692026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads30702026/11/11 23:37:41 INFO Received push request method=POST path=/30712026/11/11 23:37:41 INFO Completed push id=6 objects=26 duration=4.659018ms30722026/11/11 23:37:41 INFO Received push request method=POST path=/30732026/11/11 23:37:41 INFO Vacuumed table table=closures30742026/11/11 23:37:41 INFO Received push request method=POST path=/30752026/11/11 23:37:41 INFO Received uploads request method=POST path=/30762026/11/11 23:37:41 INFO Vacuumed table table=objects30772026/11/11 23:37:41 INFO Received push request method=POST path=/30782026/11/11 23:37:41 INFO Received push request method=POST path=/30792026/11/11 23:37:41 INFO Received push request method=POST path=/30802026/11/11 23:37:41 INFO Received push request method=POST path=/30812026/11/11 23:37:41 INFO Received uploads request method=POST path=/30822026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030832026/11/11 23:37:41 INFO Received uploads request method=POST path=/30842026/11/11 23:37:41 INFO Received uploads request method=POST path=/30852026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030862026/11/11 23:37:41 INFO Received push request method=POST path=/30872026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030882026/11/11 23:37:41 INFO Received push request method=POST path=/30892026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=030902026/11/11 23:37:41 INFO Received push request method=POST path=/30912026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030922026/11/11 23:37:41 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30932026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30942026/11/11 23:37:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30952026/11/11 23:37:41 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30962026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30972026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period30982026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030992026/11/11 23:37:41 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31002026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31012026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031022026/11/11 23:37:41 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31032026/11/11 23:37:41 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31042026/11/11 23:37:41 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31052026/11/11 23:37:41 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31062026/11/11 23:37:41 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31072026/11/11 23:37:41 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31082026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."31092026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31102026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31112026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31122026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31132026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."31142026/11/11 23:37:41 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31152026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31162026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31172026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31182026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031192026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31202026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31212026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31222026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31232026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=031242026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31252026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31262026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31272026/11/11 23:37:41 INFO Vacuumed table table=closures31282026/11/11 23:37:41 INFO Vacuumed table table=closures31292026/11/11 23:37:41 INFO Vacuumed table table=objects31302026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31312026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31322026/11/11 23:37:41 INFO Vacuumed table table=objects31332026/11/11 23:37:41 INFO Vacuumed table table=closures31342026/11/11 23:37:41 INFO Vacuumed table table=closures31352026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period31362026/11/11 23:37:41 INFO Vacuumed table table=objects31372026/11/11 23:37:41 INFO Vacuumed table table=objects31382026/11/11 23:37:41 INFO Received complete push request method=POST path=/31392026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031402026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31412026/11/11 23:37:41 INFO Completed push id=8 objects=6 duration=2.291768ms31422026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31432026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=031442026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31452026/11/11 23:37:41 INFO Vacuumed table table=closures31462026/11/11 23:37:41 INFO Received complete upload request method=POST path=/31472026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period31482026/11/11 23:37:41 INFO Received push request method=POST path=/31492026/11/11 23:37:41 INFO Vacuumed table table=objects31502026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=031512026/11/11 23:37:41 INFO Received complete upload request method=POST path=/31522026/11/11 23:37:41 INFO Received complete push request method=POST path=/31532026/11/11 23:37:41 INFO Completed upload id=11 objects=6 duration=2.039898ms31542026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031552026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31562026/11/11 23:37:41 INFO Completed upload id=9 objects=16 duration=1.772809ms31572026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31582026/11/11 23:37:41 INFO Completed push id=8 objects=6 duration=2.327788ms31592026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31602026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31612026/11/11 23:37:41 INFO Received push request method=POST path=/31622026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31632026/11/11 23:37:41 INFO Received complete push request method=POST path=/31642026/11/11 23:37:41 INFO Vacuumed table table=closures31652026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31662026/11/11 23:37:41 INFO Vacuumed table table=closures31672026/11/11 23:37:41 INFO Vacuumed table table=objects31682026/11/11 23:37:41 INFO Completed push id=14 objects=6 duration=2.055109ms31692026/11/11 23:37:41 INFO Vacuumed table table=objects31702026/11/11 23:37:41 INFO Received push request method=POST path=/31712026/11/11 23:37:41 INFO Received push request method=POST path=/31722026/11/11 23:37:41 INFO Received complete push request method=POST path=/31732026/11/11 23:37:41 INFO Received uploads request method=POST path=/31742026/11/11 23:37:41 INFO Completed push id=5 objects=8 duration=3.775898ms31752026/11/11 23:37:41 INFO Aborted multipart uploads count=0 kept=031762026/11/11 23:37:41 WARN Force mode enabled - objects will be deleted immediately without grace period31772026/11/11 23:37:41 INFO Received push request method=POST path=/31782026/11/11 23:37:41 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031792026/11/11 23:37:41 INFO Received complete upload request method=POST path=/31802026/11/11 23:37:41 INFO Vacuumed table table=pending_closures31812026/11/11 23:37:41 INFO Vacuumed table table=pending_objects31822026/11/11 23:37:41 INFO Received push request method=POST path=/31832026/11/11 23:37:41 INFO Vacuumed table table=multipart_uploads31842026/11/11 23:37:41 INFO Completed upload id=9 objects=3 duration=1.684619ms31852026/11/11 23:37:41 INFO Vacuumed table table=closures31862026/11/11 23:37:41 INFO Received complete push request method=POST path=/31872026/11/11 23:37:41 INFO Vacuumed table table=objects31882026/11/11 23:37:41 INFO Received push request method=POST path=/31892026/11/11 23:37:41 INFO Completed push id=9 objects=5 duration=2.385218ms31902026/11/11 23:37:41 INFO Received uploads request method=POST path=/31912026/11/11 23:37:41 INFO Received complete push request method=POST path=/31922026/11/11 23:37:41 INFO Completed push id=12 objects=7 duration=1.949319ms31932026/11/11 23:37:41 INFO Received uploads request method=POST path=/31942026/11/11 23:37:41 INFO Received complete upload request method=POST path=/31952026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=031962026/11/11 23:37:42 INFO Completed upload id=10 objects=2 duration=3.100449ms31972026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=031982026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=031992026/11/11 23:37:42 INFO Received complete push request method=POST path=/32002026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period32012026/11/11 23:37:42 INFO Received push request method=POST path=/32022026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032032026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period32042026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32052026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032062026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32072026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32082026/11/11 23:37:42 INFO Completed push id=13 objects=12 duration=4.494367ms32092026/11/11 23:37:42 INFO Vacuumed table table=closures32102026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32112026/11/11 23:37:42 INFO Vacuumed table table=objects32122026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32132026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32142026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032152026/11/11 23:37:42 INFO Vacuumed table table=closures32162026/11/11 23:37:42 INFO Vacuumed table table=objects32172026/11/11 23:37:42 INFO Received push request method=POST path=/32182026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period32192026/11/11 23:37:42 INFO Received push request method=POST path=/32202026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032212026/11/11 23:37:42 INFO Received push request method=POST path=/32222026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32232026/11/11 23:37:42 INFO Received push request method=POST path=/32242026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32252026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32262026/11/11 23:37:42 INFO Received uploads request method=POST path=/32272026/11/11 23:37:42 INFO Vacuumed table table=closures32282026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032292026/11/11 23:37:42 INFO Vacuumed table table=objects32302026/11/11 23:37:42 INFO Received complete push request method=POST path=/32312026/11/11 23:37:42 INFO Completed push id=12 objects=15 duration=2.330819ms32322026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032332026/11/11 23:37:42 INFO Received uploads request method=POST path=/32342026/11/11 23:37:42 INFO Received push request method=POST path=/32352026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032362026/11/11 23:37:42 INFO Received push request method=POST path=/32372026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=032382026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032392026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32402026/11/11 23:37:42 INFO Received complete push request method=POST path=/32412026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032422026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32432026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32442026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032452026/11/11 23:37:42 INFO Completed push id=10 objects=3 duration=2.016529ms32462026/11/11 23:37:42 INFO Vacuumed table table=closures32472026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32482026/11/11 23:37:42 INFO Vacuumed table table=objects32492026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32502026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32512026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032522026/11/11 23:37:42 INFO Vacuumed table table=closures32532026/11/11 23:37:42 INFO Received complete upload request method=POST path=/32542026/11/11 23:37:42 INFO Received push request method=POST path=/32552026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32562026/11/11 23:37:42 INFO Vacuumed table table=objects32572026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=032582026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32592026/11/11 23:37:42 INFO Completed upload id=15 objects=4 duration=3.032618ms32602026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32612026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032622026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32632026/11/11 23:37:42 INFO Vacuumed table table=closures32642026/11/11 23:37:42 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32652026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32662026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32672026/11/11 23:37:42 INFO Vacuumed table table=objects32682026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32692026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032702026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032712026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period32722026/11/11 23:37:42 INFO Vacuumed table table=closures32732026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=032742026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32752026/11/11 23:37:42 INFO Vacuumed table table=objects32762026/11/11 23:37:42 INFO Received uploads request method=POST path=/32772026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32782026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032792026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32802026/11/11 23:37:42 INFO Received push request method=POST path=/32812026/11/11 23:37:42 INFO Vacuumed table table=closures32822026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32832026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032842026/11/11 23:37:42 INFO Received complete push request method=POST path=/32852026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32862026/11/11 23:37:42 INFO Vacuumed table table=pending_closures32872026/11/11 23:37:42 INFO Vacuumed table table=objects32882026/11/11 23:37:42 INFO Received push request method=POST path=/32892026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32902026/11/11 23:37:42 INFO Vacuumed table table=pending_objects32912026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads32922026/11/11 23:37:42 INFO Vacuumed table table=closures32932026/11/11 23:37:42 INFO Received uploads request method=POST path=/32942026/11/11 23:37:42 INFO Vacuumed table table=closures32952026/11/11 23:37:42 INFO Vacuumed table table=objects32962026/11/11 23:37:42 INFO Vacuumed table table=objects32972026/11/11 23:37:42 INFO Completed push id=11 objects=11 duration=4.514517ms32982026/11/11 23:37:42 INFO Received push request method=POST path=/32992026/11/11 23:37:42 INFO Received uploads request method=POST path=/33002026/11/11 23:37:42 INFO Received push request method=POST path=/33012026/11/11 23:37:42 INFO Received push request method=POST path=/33022026/11/11 23:37:42 INFO Received push request method=POST path=/33032026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33042026/11/11 23:37:42 INFO Received push request method=POST path=/33052026/11/11 23:37:42 INFO Received uploads request method=POST path=/33062026/11/11 23:37:42 INFO Received uploads request method=POST path=/33072026/11/11 23:37:42 INFO Received push request method=POST path=/33082026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=033092026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033102026/11/11 23:37:42 INFO Vacuumed table table=pending_closures33112026/11/11 23:37:42 INFO Vacuumed table table=pending_objects33122026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads33132026/11/11 23:37:42 INFO Vacuumed table table=closures33142026/11/11 23:37:42 INFO Vacuumed table table=objects33152026/11/11 23:37:42 INFO Received uploads request method=POST path=/33162026/11/11 23:37:42 INFO Received complete push request method=POST path=/33172026/11/11 23:37:42 INFO Received uploads request method=POST path=/33182026/11/11 23:37:42 INFO Completed push id=8 objects=6 duration=3.867288ms33192026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=033202026/11/11 23:37:42 INFO Received push request method=POST path=/33212026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period33222026/11/11 23:37:42 INFO Received complete upload request method=POST path=/33232026/11/11 23:37:42 INFO Received complete push request method=POST path=/33242026/11/11 23:37:42 INFO Completed upload id=16 objects=3 duration=1.29784ms33252026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033262026/11/11 23:37:42 INFO Completed push id=12 objects=3 duration=2.176259ms33272026/11/11 23:37:42 INFO Received complete push request method=POST path=/33282026/11/11 23:37:42 INFO Vacuumed table table=pending_closures33292026/11/11 23:37:42 INFO Received uploads request method=POST path=/33302026/11/11 23:37:42 INFO Received complete push request method=POST path=/33312026/11/11 23:37:42 INFO Vacuumed table table=pending_objects33322026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads33332026/11/11 23:37:42 INFO Completed push id=4 objects=6 duration=2.819578ms33342026/11/11 23:37:42 INFO Vacuumed table table=closures33352026/11/11 23:37:42 INFO Vacuumed table table=objects33362026/11/11 23:37:42 INFO Completed push id=3 objects=21 duration=4.977677ms33372026/11/11 23:37:42 INFO Received uploads request method=POST path=/33382026/11/11 23:37:42 INFO Received complete push request method=POST path=/33392026/11/11 23:37:42 INFO Received uploads request method=POST path=/33402026/11/11 23:37:42 INFO Completed push id=11 objects=8 duration=3.446528ms33412026/11/11 23:37:42 INFO Received complete push request method=POST path=/33422026/11/11 23:37:42 INFO Received uploads request method=POST path=/33432026/11/11 23:37:42 INFO Received complete push request method=POST path=/33442026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=033452026/11/11 23:37:42 INFO Completed push id=9 objects=7 duration=4.117748ms33462026/11/11 23:37:42 INFO Received push request method=POST path=/33472026/11/11 23:37:42 INFO Completed push id=19 objects=6 duration=3.715778ms33482026/11/11 23:37:42 INFO Received uploads request method=POST path=/33492026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033502026/11/11 23:37:42 INFO Vacuumed table table=pending_closures33512026/11/11 23:37:42 INFO Received uploads request method=POST path=/33522026/11/11 23:37:42 INFO Vacuumed table table=pending_objects33532026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads33542026/11/11 23:37:42 INFO Received push request method=POST path=/33552026/11/11 23:37:42 INFO Vacuumed table table=closures33562026/11/11 23:37:42 INFO Received uploads request method=POST path=/33572026/11/11 23:37:42 INFO Vacuumed table table=objects33582026/11/11 23:37:42 INFO Received complete upload request method=POST path=/33592026/11/11 23:37:42 INFO Received push request method=POST path=/33602026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=033612026/11/11 23:37:42 INFO Received uploads request method=POST path=/33622026/11/11 23:37:42 INFO Completed upload id=19 objects=25 duration=2.856918ms33632026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033642026/11/11 23:37:42 INFO Vacuumed table table=pending_closures33652026/11/11 23:37:42 INFO Vacuumed table table=pending_objects33662026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads33672026/11/11 23:37:42 INFO Vacuumed table table=closures33682026/11/11 23:37:42 INFO Vacuumed table table=objects33692026/11/11 23:37:42 INFO Received uploads request method=POST path=/33702026/11/11 23:37:42 INFO Received uploads request method=POST path=/33712026/11/11 23:37:42 INFO Received complete push request method=POST path=/33722026/11/11 23:37:42 INFO Completed push id=12 objects=2 duration=4.403708ms33732026/11/11 23:37:42 INFO Received uploads request method=POST path=/33742026/11/11 23:37:42 INFO Received complete upload request method=POST path=/33752026/11/11 23:37:42 INFO Received uploads request method=POST path=/33762026/11/11 23:37:42 INFO Received complete upload request method=POST path=/33772026/11/11 23:37:42 INFO Completed upload id=10 objects=7 duration=1.6957ms33782026/11/11 23:37:42 INFO Received push request method=POST path=/33792026/11/11 23:37:42 INFO Received complete upload request method=POST path=/33802026/11/11 23:37:42 INFO Completed upload id=9 objects=2 duration=1.908399ms33812026/11/11 23:37:42 INFO Received uploads request method=POST path=/33822026/11/11 23:37:42 INFO Completed upload id=8 objects=3 duration=2.631818ms33832026/11/11 23:37:42 INFO Received push request method=POST path=/33842026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=033852026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period33862026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033872026/11/11 23:37:42 INFO Vacuumed table table=pending_closures33882026/11/11 23:37:42 INFO Received uploads request method=POST path=/33892026/11/11 23:37:42 INFO Received complete push request method=POST path=/33902026/11/11 23:37:42 INFO Received uploads request method=POST path=/33912026/11/11 23:37:42 INFO Vacuumed table table=pending_objects33922026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads33932026/11/11 23:37:42 INFO Completed push id=8 objects=3 duration=2.187099ms33942026/11/11 23:37:42 INFO Vacuumed table table=closures33952026/11/11 23:37:42 INFO Vacuumed table table=objects33962026/11/11 23:37:42 INFO Received complete push request method=POST path=/33972026/11/11 23:37:42 INFO Completed push id=13 objects=6 duration=3.392878ms33982026/11/11 23:37:42 INFO Received uploads request method=POST path=/33992026/11/11 23:37:42 INFO Received uploads request method=POST path=/34002026/11/11 23:37:42 INFO Received push request method=POST path=/34012026/11/11 23:37:42 INFO Received uploads request method=POST path=/34022026/11/11 23:37:42 INFO Received uploads request method=POST path=/34032026/11/11 23:37:42 INFO Received push request method=POST path=/34042026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=034052026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period34062026/11/11 23:37:42 INFO Received push request method=POST path=/34072026/11/11 23:37:42 INFO Received push request method=POST path=/34082026/11/11 23:37:42 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34092026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34102026/11/11 23:37:42 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34112026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34122026/11/11 23:37:42 INFO Received complete push request method=POST path=/34132026/11/11 23:37:42 INFO Received uploads request method=POST path=/34142026/11/11 23:37:42 INFO Completed upload id=18 objects=4 duration=2.220308ms34152026/11/11 23:37:42 INFO Received complete push request method=POST path=/34162026/11/11 23:37:42 INFO Completed push id=9 objects=24 duration=2.126659ms34172026/11/11 23:37:42 INFO Completed push id=15 objects=24 duration=3.975748ms34182026/11/11 23:37:42 INFO Received uploads request method=POST path=/34192026/11/11 23:37:42 INFO Received uploads request method=POST path=/34202026/11/11 23:37:42 INFO Received push request method=POST path=/34212026/11/11 23:37:42 INFO Received uploads request method=POST path=/34222026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34232026/11/11 23:37:42 INFO Completed upload id=16 objects=23 duration=1.577739ms34242026/11/11 23:37:42 INFO Received push request method=POST path=/34252026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=034262026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period34272026/11/11 23:37:42 INFO Received uploads request method=POST path=/34282026/11/11 23:37:42 INFO Received push request method=POST path=/34292026/11/11 23:37:42 INFO Received uploads request method=POST path=/34302026/11/11 23:37:42 INFO Received push request method=POST path=/34312026/11/11 23:37:42 INFO Received uploads request method=POST path=/34322026/11/11 23:37:42 INFO Received uploads request method=POST path=/34332026/11/11 23:37:42 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34342026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34352026/11/11 23:37:42 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34362026/11/11 23:37:42 INFO Received push request method=POST path=/34372026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34382026/11/11 23:37:42 INFO Completed upload id=13 objects=10 duration=1.366469ms34392026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34402026/11/11 23:37:42 INFO Completed upload id=15 objects=6 duration=1.26237ms34412026/11/11 23:37:42 INFO Received complete push request method=POST path=/34422026/11/11 23:37:42 INFO Received uploads request method=POST path=/34432026/11/11 23:37:42 INFO Received complete push request method=POST path=/34442026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=034452026/11/11 23:37:42 INFO Completed push id=17 objects=6 duration=2.211499ms34462026/11/11 23:37:42 INFO Received complete push request method=POST path=/34472026/11/11 23:37:42 INFO Completed push id=6 objects=6 duration=2.614129ms34482026/11/11 23:37:42 INFO Received uploads request method=POST path=/34492026/11/11 23:37:42 INFO Completed push id=15 objects=5 duration=2.286329ms34502026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=034512026/11/11 23:37:42 INFO Received uploads request method=POST path=/34522026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34532026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034542026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period34552026/11/11 23:37:42 INFO Vacuumed table table=pending_closures34562026/11/11 23:37:42 INFO Completed upload id=18 objects=4 duration=2.136589ms34572026/11/11 23:37:42 INFO Vacuumed table table=pending_objects34582026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads34592026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034602026/11/11 23:37:42 INFO Vacuumed table table=closures34612026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=034622026/11/11 23:37:42 INFO Vacuumed table table=pending_closures34632026/11/11 23:37:42 INFO Vacuumed table table=objects34642026/11/11 23:37:42 INFO Vacuumed table table=pending_objects34652026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period34662026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads34672026/11/11 23:37:42 INFO Received complete push request method=POST path=/34682026/11/11 23:37:42 INFO Vacuumed table table=closures34692026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=034702026/11/11 23:37:42 INFO Received uploads request method=POST path=/34712026/11/11 23:37:42 INFO Vacuumed table table=objects34722026/11/11 23:37:42 INFO Completed push id=17 objects=6 duration=2.586708ms34732026/11/11 23:37:42 INFO Received uploads request method=POST path=/34742026/11/11 23:37:42 INFO Received push request method=POST path=/34752026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34762026/11/11 23:37:42 INFO Completed upload id=15 objects=6 duration=1.18638ms34772026/11/11 23:37:42 INFO Received uploads request method=POST path=/34782026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34792026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034802026/11/11 23:37:42 INFO Completed upload id=10 objects=10 duration=1.082619ms34812026/11/11 23:37:42 INFO Vacuumed table table=pending_closures34822026/11/11 23:37:42 INFO Vacuumed table table=pending_objects34832026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads34842026/11/11 23:37:42 INFO Vacuumed table table=closures34852026/11/11 23:37:42 INFO Vacuumed table table=objects34862026/11/11 23:37:42 INFO Received push request method=POST path=/34872026/11/11 23:37:42 INFO Received push request method=POST path=/34882026/11/11 23:37:42 INFO Received uploads request method=POST path=/34892026/11/11 23:37:42 INFO Received complete upload request method=POST path=/34902026/11/11 23:37:42 INFO Completed upload id=17 objects=7 duration=1.734789ms34912026/11/11 23:37:42 INFO Received push request method=POST path=/34922026/11/11 23:37:42 INFO Received uploads request method=POST path=/34932026/11/11 23:37:42 INFO Received complete push request method=POST path=/34942026/11/11 23:37:42 INFO Received complete push request method=POST path=/34952026/11/11 23:37:42 INFO Completed push id=15 objects=7 duration=2.497889ms34962026/11/11 23:37:42 INFO Received complete push request method=POST path=/34972026/11/11 23:37:42 INFO Completed push id=19 objects=8 duration=4.013878ms34982026/11/11 23:37:42 INFO Completed push id=4 objects=7 duration=2.020299ms34992026/11/11 23:37:42 INFO Received uploads request method=POST path=/35002026/11/11 23:37:42 INFO Received push request method=POST path=/35012026/11/11 23:37:42 INFO Received push request method=POST path=/35022026/11/11 23:37:42 INFO Received complete upload request method=POST path=/35032026/11/11 23:37:42 INFO Received push request method=POST path=/35042026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035052026/11/11 23:37:42 INFO Received push request method=POST path=/35062026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035072026/11/11 23:37:42 INFO Completed upload id=17 objects=6 duration=3.087798ms35082026/11/11 23:37:42 INFO Received uploads request method=POST path=/35092026/11/11 23:37:42 INFO Received complete push request method=POST path=/35102026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035112026/11/11 23:37:42 INFO Completed push id=10 objects=15 duration=2.147539ms35122026/11/11 23:37:42 INFO Vacuumed table table=pending_closures35132026/11/11 23:37:42 INFO Received uploads request method=POST path=/35142026/11/11 23:37:42 INFO Vacuumed table table=pending_objects35152026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads35162026/11/11 23:37:42 INFO Received uploads request method=POST path=/35172026/11/11 23:37:42 INFO Vacuumed table table=closures35182026/11/11 23:37:42 INFO Received uploads request method=POST path=/35192026/11/11 23:37:42 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35202026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35212026/11/11 23:37:42 INFO Vacuumed table table=objects35222026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=035232026/11/11 23:37:42 INFO Received complete upload request method=POST path=/35242026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035252026/11/11 23:37:42 INFO Vacuumed table table=pending_closures35262026/11/11 23:37:42 INFO Completed upload id=6 objects=6 duration=1.535219ms35272026/11/11 23:37:42 INFO Received uploads request method=POST path=/35282026/11/11 23:37:42 INFO Vacuumed table table=pending_objects35292026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads35302026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035312026/11/11 23:37:42 INFO Vacuumed table table=closures35322026/11/11 23:37:42 INFO Vacuumed table table=pending_closures35332026/11/11 23:37:42 INFO Vacuumed table table=pending_objects35342026/11/11 23:37:42 INFO Vacuumed table table=objects35352026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads35362026/11/11 23:37:42 INFO Vacuumed table table=closures35372026/11/11 23:37:42 INFO Vacuumed table table=objects35382026/11/11 23:37:42 INFO Received push request method=POST path=/35392026/11/11 23:37:42 INFO Received uploads request method=POST path=/35402026/11/11 23:37:42 INFO Received uploads request method=POST path=/35412026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035422026/11/11 23:37:42 INFO Received uploads request method=POST path=/35432026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035442026/11/11 23:37:42 INFO Received push request method=POST path=/35452026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035462026/11/11 23:37:42 INFO Vacuumed table table=pending_closures35472026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period35482026/11/11 23:37:42 INFO Received push request method=POST path=/35492026/11/11 23:37:42 INFO Vacuumed table table=pending_objects35502026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads35512026/11/11 23:37:42 INFO Received complete push request method=POST path=/35522026/11/11 23:37:42 INFO Vacuumed table table=closures35532026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035542026/11/11 23:37:42 INFO Vacuumed table table=objects35552026/11/11 23:37:42 INFO Completed push id=16 objects=27 duration=2.245999ms35562026/11/11 23:37:42 INFO Vacuumed table table=pending_closures35572026/11/11 23:37:42 INFO Received complete push request method=POST path=/35582026/11/11 23:37:42 INFO Vacuumed table table=pending_objects35592026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads35602026/11/11 23:37:42 INFO Vacuumed table table=closures35612026/11/11 23:37:42 INFO Completed push id=22 objects=3 duration=3.235709ms35622026/11/11 23:37:42 INFO Vacuumed table table=objects35632026/11/11 23:37:42 INFO Received push request method=POST path=/35642026/11/11 23:37:42 INFO Received complete upload request method=POST path=/35652026/11/11 23:37:42 INFO Received push request method=POST path=/35662026/11/11 23:37:42 INFO Received push request method=POST path=/35672026/11/11 23:37:42 INFO Completed upload id=14 objects=6 duration=1.21654ms35682026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035692026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period35702026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035712026/11/11 23:37:42 INFO Received uploads request method=POST path=/35722026/11/11 23:37:42 INFO Vacuumed table table=pending_closures35732026/11/11 23:37:42 INFO Received uploads request method=POST path=/35742026/11/11 23:37:42 INFO Vacuumed table table=pending_objects35752026/11/11 23:37:42 INFO Received uploads request method=POST path=/35762026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads35772026/11/11 23:37:42 INFO Received push request method=POST path=/35782026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035792026/11/11 23:37:42 INFO Vacuumed table table=closures35802026/11/11 23:37:42 INFO Vacuumed table table=objects35812026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period35822026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=035832026/11/11 23:37:42 INFO Received complete push request method=POST path=/35842026/11/11 23:37:42 INFO Completed push id=1 objects=11 duration=1.675609ms35852026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period35862026/11/11 23:37:42 INFO Received push request method=POST path=/35872026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035882026/11/11 23:37:42 INFO Received push request method=POST path=/35892026/11/11 23:37:42 INFO Vacuumed table table=pending_closures35902026/11/11 23:37:42 INFO Received complete push request method=POST path=/35912026/11/11 23:37:42 INFO Vacuumed table table=pending_objects35922026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads35932026/11/11 23:37:42 INFO Vacuumed table table=closures35942026/11/11 23:37:42 INFO Vacuumed table table=objects35952026/11/11 23:37:42 INFO Completed push id=14 objects=4 duration=3.551868ms35962026/11/11 23:37:42 INFO Received push request method=POST path=/35972026/11/11 23:37:42 INFO Received push request method=POST path=/35982026/11/11 23:37:42 INFO Received complete push request method=POST path=/35992026/11/11 23:37:42 INFO Received uploads request method=POST path=/36002026/11/11 23:37:42 INFO Completed push id=16 objects=4 duration=3.180058ms36012026/11/11 23:37:42 INFO Received complete push request method=POST path=/36022026/11/11 23:37:42 INFO Completed push id=7 objects=7 duration=1.707429ms36032026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36042026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=036052026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036062026/11/11 23:37:42 INFO Received uploads request method=POST path=/36072026/11/11 23:37:42 INFO Received complete upload request method=POST path=/36082026/11/11 23:37:42 INFO Vacuumed table table=pending_closures36092026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=036102026/11/11 23:37:42 INFO Completed upload id=22 objects=2 duration=1.452279ms36112026/11/11 23:37:42 INFO Vacuumed table table=pending_objects36122026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads36132026/11/11 23:37:42 INFO Received uploads request method=POST path=/36142026/11/11 23:37:42 INFO Vacuumed table table=closures36152026/11/11 23:37:42 INFO Received complete upload request method=POST path=/36162026/11/11 23:37:42 INFO Vacuumed table table=objects36172026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period36182026/11/11 23:37:42 INFO Completed upload id=16 objects=3 duration=1.224559ms36192026/11/11 23:37:42 INFO Received push request method=POST path=/36202026/11/11 23:37:42 INFO Received complete upload request method=POST path=/36212026/11/11 23:37:42 INFO Completed upload id=7 objects=3 duration=1.23857ms36222026/11/11 23:37:42 INFO Received uploads request method=POST path=/36232026/11/11 23:37:42 INFO Received uploads request method=POST path=/36242026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036252026/11/11 23:37:42 INFO Received push request method=POST path=/36262026/11/11 23:37:42 INFO Received complete push request method=POST path=/36272026/11/11 23:37:42 INFO Vacuumed table table=pending_closures36282026/11/11 23:37:42 INFO Vacuumed table table=pending_objects36292026/11/11 23:37:42 INFO Completed push id=18 objects=10 duration=2.096859ms36302026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads36312026/11/11 23:37:42 INFO Vacuumed table table=closures36322026/11/11 23:37:42 INFO Received push request method=POST path=/36332026/11/11 23:37:42 INFO Vacuumed table table=objects36342026/11/11 23:37:42 INFO Received uploads request method=POST path=/36352026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=036362026/11/11 23:37:42 INFO Received uploads request method=POST path=/36372026/11/11 23:37:42 INFO Received complete push request method=POST path=/36382026/11/11 23:37:42 INFO Completed push id=14 objects=3 duration=988.73µs36392026/11/11 23:37:42 INFO Received push request method=POST path=/36402026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=036412026/11/11 23:37:42 INFO Received uploads request method=POST path=/36422026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036432026/11/11 23:37:42 INFO Received push request method=POST path=/36442026/11/11 23:37:42 INFO Vacuumed table table=pending_closures36452026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period36462026/11/11 23:37:42 INFO Received complete upload request method=POST path=/36472026/11/11 23:37:42 INFO Vacuumed table table=pending_objects36482026/11/11 23:37:42 INFO Received uploads request method=POST path=/36492026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads36502026/11/11 23:37:42 INFO Completed upload id=20 objects=5 duration=1.457679ms36512026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036522026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=036532026/11/11 23:37:42 INFO Vacuumed table table=closures36542026/11/11 23:37:42 INFO Vacuumed table table=pending_closures36552026/11/11 23:37:42 INFO Vacuumed table table=objects36562026/11/11 23:37:42 INFO Received push request method=POST path=/36572026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period36582026/11/11 23:37:42 INFO Vacuumed table table=pending_objects36592026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads36602026/11/11 23:37:42 INFO Vacuumed table table=closures36612026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036622026/11/11 23:37:42 INFO Received complete push request method=POST path=/36632026/11/11 23:37:42 INFO Vacuumed table table=objects36642026/11/11 23:37:42 INFO Vacuumed table table=pending_closures36652026/11/11 23:37:42 INFO Completed push id=18 objects=23 duration=1.948349ms36662026/11/11 23:37:42 INFO Vacuumed table table=pending_objects36672026/11/11 23:37:42 INFO Received uploads request method=POST path=/36682026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads36692026/11/11 23:37:42 INFO Vacuumed table table=closures36702026/11/11 23:37:42 INFO Vacuumed table table=objects36712026/11/11 23:37:42 INFO Received push request method=POST path=/36722026/11/11 23:37:42 INFO Received push request method=POST path=/36732026/11/11 23:37:42 INFO Received complete upload request method=POST path=/36742026/11/11 23:37:42 INFO Received push request method=POST path=/36752026/11/11 23:37:42 INFO Received uploads request method=POST path=/36762026/11/11 23:37:42 INFO Completed upload id=18 objects=11 duration=1.5247ms36772026/11/11 23:37:42 INFO Received uploads request method=POST path=/36782026/11/11 23:37:42 INFO Received complete push request method=POST path=/36792026/11/11 23:37:42 INFO Completed push id=9 objects=4 duration=3.393518ms36802026/11/11 23:37:42 INFO Received uploads request method=POST path=/36812026/11/11 23:37:42 INFO Received uploads request method=POST path=/36822026/11/11 23:37:42 INFO Received push request method=POST path=/36832026/11/11 23:37:42 INFO Received push request method=POST path=/36842026/11/11 23:37:42 INFO Received uploads request method=POST path=/36852026/11/11 23:37:42 INFO Received uploads request method=POST path=/36862026/11/11 23:37:42 INFO Received push request method=POST path=/36872026/11/11 23:37:42 INFO Received uploads request method=POST path=/36882026/11/11 23:37:42 INFO Received push request method=POST path=/36892026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=036902026/11/11 23:37:42 INFO Received uploads request method=POST path=/36912026/11/11 23:37:42 INFO Received uploads request method=POST path=/36922026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period36932026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036942026/11/11 23:37:42 INFO Received uploads request method=POST path=/36952026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=036962026/11/11 23:37:42 INFO Vacuumed table table=pending_closures36972026/11/11 23:37:42 INFO Received push request method=POST path=/36982026/11/11 23:37:42 INFO Vacuumed table table=pending_objects36992026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period37002026/11/11 23:37:42 INFO Received push request method=POST path=/37012026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37022026/11/11 23:37:42 INFO Received push request method=POST path=/37032026/11/11 23:37:42 INFO Received complete push request method=POST path=/37042026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037052026/11/11 23:37:42 INFO Vacuumed table table=closures37062026/11/11 23:37:42 INFO Vacuumed table table=pending_closures37072026/11/11 23:37:42 INFO Completed push id=11 objects=9 duration=2.304809ms37082026/11/11 23:37:42 INFO Received complete upload request method=POST path=/37092026/11/11 23:37:42 INFO Vacuumed table table=objects37102026/11/11 23:37:42 INFO Completed upload id=15 objects=9 duration=1.293829ms37112026/11/11 23:37:42 INFO Vacuumed table table=pending_objects37122026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37132026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037142026/11/11 23:37:42 INFO Vacuumed table table=closures37152026/11/11 23:37:42 INFO Vacuumed table table=objects37162026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period37172026/11/11 23:37:42 INFO Received push request method=POST path=/37182026/11/11 23:37:42 INFO Received push request method=POST path=/37192026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037202026/11/11 23:37:42 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37212026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037222026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period37232026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037242026/11/11 23:37:42 INFO Received complete push request method=POST path=/37252026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037262026/11/11 23:37:42 INFO Received push request method=POST path=/37272026/11/11 23:37:42 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37282026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37292026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period37302026/11/11 23:37:42 INFO Vacuumed table table=pending_closures37312026/11/11 23:37:42 INFO Completed push id=14 objects=3 duration=3.308488ms37322026/11/11 23:37:42 INFO Received push request method=POST path=/37332026/11/11 23:37:42 INFO Vacuumed table table=pending_objects37342026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037352026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37362026/11/11 23:37:42 INFO Vacuumed table table=closures37372026/11/11 23:37:42 INFO Vacuumed table table=pending_closures37382026/11/11 23:37:42 INFO Vacuumed table table=objects37392026/11/11 23:37:42 INFO Vacuumed table table=pending_objects37402026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37412026/11/11 23:37:42 INFO Vacuumed table table=closures37422026/11/11 23:37:42 INFO Received push request method=POST path=/37432026/11/11 23:37:42 INFO Vacuumed table table=objects37442026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037452026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037462026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037472026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period37482026/11/11 23:37:42 INFO Vacuumed table table=pending_closures37492026/11/11 23:37:42 INFO Received push request method=POST path=/37502026/11/11 23:37:42 INFO Vacuumed table table=pending_objects37512026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37522026/11/11 23:37:42 INFO Vacuumed table table=closures37532026/11/11 23:37:42 INFO Vacuumed table table=objects37542026/11/11 23:37:42 INFO Received complete push request method=POST path=/37552026/11/11 23:37:42 INFO Received push request method=POST path=/37562026/11/11 23:37:42 INFO Completed push id=21 objects=9 duration=2.254659ms37572026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037582026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period37592026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037602026/11/11 23:37:42 INFO Received push request method=POST path=/37612026/11/11 23:37:42 INFO Vacuumed table table=pending_closures37622026/11/11 23:37:42 INFO Received complete push request method=POST path=/37632026/11/11 23:37:42 INFO Received complete push request method=POST path=/37642026/11/11 23:37:42 INFO Vacuumed table table=pending_objects37652026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37662026/11/11 23:37:42 INFO Completed push id=21 objects=17 duration=2.256899ms37672026/11/11 23:37:42 INFO Vacuumed table table=closures37682026/11/11 23:37:42 INFO Completed push id=4 objects=6 duration=2.220029ms37692026/11/11 23:37:42 INFO Vacuumed table table=objects37702026/11/11 23:37:42 INFO Received complete push request method=POST path=/37712026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037722026/11/11 23:37:42 INFO Received complete upload request method=POST path=/37732026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period37742026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=037752026/11/11 23:37:42 INFO Completed push id=16 objects=16 duration=3.930958ms37762026/11/11 23:37:42 INFO Completed upload id=14 objects=8 duration=2.457039ms37772026/11/11 23:37:42 INFO Received push request method=POST path=/37782026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037792026/11/11 23:37:42 INFO Received push request method=POST path=/37802026/11/11 23:37:42 INFO Vacuumed table table=pending_closures37812026/11/11 23:37:42 INFO Vacuumed table table=pending_objects37822026/11/11 23:37:42 INFO Received uploads request method=POST path=/37832026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37842026/11/11 23:37:42 INFO Vacuumed table table=closures37852026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=037862026/11/11 23:37:42 INFO Vacuumed table table=objects37872026/11/11 23:37:42 INFO Vacuumed table table=pending_closures37882026/11/11 23:37:42 INFO Received complete upload request method=POST path=/37892026/11/11 23:37:42 INFO Vacuumed table table=pending_objects37902026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads37912026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37922026/11/11 23:37:42 INFO Vacuumed table table=closures37932026/11/11 23:37:42 INFO Received uploads request method=POST path=/37942026/11/11 23:37:42 INFO Completed upload id=14 objects=16 duration=2.606409ms37952026/11/11 23:37:42 INFO Vacuumed table table=objects37962026/11/11 23:37:42 INFO Received complete push request method=POST path=/37972026/11/11 23:37:42 INFO Received uploads request method=POST path=/37982026/11/11 23:37:42 INFO Received push request method=POST path=/37992026/11/11 23:37:42 INFO Completed push id=19 objects=7 duration=2.002829ms38002026/11/11 23:37:42 INFO Received uploads request method=POST path=/38012026/11/11 23:37:42 INFO Received uploads request method=POST path=/38022026/11/11 23:37:42 INFO Received push request method=POST path=/38032026/11/11 23:37:42 INFO Received push request method=POST path=/38042026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038052026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038062026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period38072026/11/11 23:37:42 INFO Received uploads request method=POST path=/38082026/11/11 23:37:42 INFO Received complete upload request method=POST path=/38092026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038102026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period38112026/11/11 23:37:42 INFO Received push request method=POST path=/38122026/11/11 23:37:42 INFO Vacuumed table table=pending_closures38132026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038142026/11/11 23:37:42 INFO Completed upload id=21 objects=7 duration=3.342998ms38152026/11/11 23:37:42 INFO Vacuumed table table=pending_objects38162026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads38172026/11/11 23:37:42 INFO Vacuumed table table=pending_closures38182026/11/11 23:37:42 INFO Vacuumed table table=closures38192026/11/11 23:37:42 INFO Received push request method=POST path=/38202026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038212026/11/11 23:37:42 INFO Vacuumed table table=pending_objects38222026/11/11 23:37:42 INFO Vacuumed table table=objects38232026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads38242026/11/11 23:37:42 INFO Vacuumed table table=closures38252026/11/11 23:37:42 INFO Received uploads request method=POST path=/38262026/11/11 23:37:42 INFO Vacuumed table table=objects38272026/11/11 23:37:42 INFO Received push request method=POST path=/38282026/11/11 23:37:42 INFO Received push request method=POST path=/38292026/11/11 23:37:42 INFO Received complete push request method=POST path=/38302026/11/11 23:37:42 INFO Completed push id=17 objects=22 duration=2.996708ms38312026/11/11 23:37:42 INFO Received push request method=POST path=/38322026/11/11 23:37:42 INFO Received uploads request method=POST path=/38332026/11/11 23:37:42 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."38342026/11/11 23:37:42 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."38352026/11/11 23:37:42 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."38362026/11/11 23:37:42 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."38372026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."38382026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."38392026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38402026/11/11 23:37:42 INFO Received complete push request method=POST path=/38412026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038422026/11/11 23:37:42 INFO Completed push id=15 objects=4 duration=1.4282ms38432026/11/11 23:37:42 INFO Received uploads request method=POST path=/38442026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period38452026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038462026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038472026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038482026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038492026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038502026/11/11 23:37:42 INFO Vacuumed table table=pending_closures38512026/11/11 23:37:42 INFO Vacuumed table table=pending_closures38522026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period38532026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period38542026/11/11 23:37:42 INFO Received complete push request method=POST path=/38552026/11/11 23:37:42 INFO Vacuumed table table=pending_objects38562026/11/11 23:37:42 INFO Vacuumed table table=pending_objects38572026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads38582026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads38592026/11/11 23:37:42 INFO Vacuumed table table=closures38602026/11/11 23:37:42 INFO Vacuumed table table=closures38612026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038622026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038632026/11/11 23:37:42 INFO Completed push id=26 objects=14 duration=3.040188ms38642026/11/11 23:37:42 INFO Vacuumed table table=objects38652026/11/11 23:37:42 INFO Vacuumed table table=objects38662026/11/11 23:37:42 INFO Vacuumed table table=pending_closures38672026/11/11 23:37:42 INFO Received complete push request method=POST path=/38682026/11/11 23:37:42 INFO Vacuumed table table=pending_closures38692026/11/11 23:37:42 INFO Received push request method=POST path=/38702026/11/11 23:37:42 INFO Vacuumed table table=pending_objects38712026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads38722026/11/11 23:37:42 INFO Vacuumed table table=pending_objects38732026/11/11 23:37:42 INFO Completed push id=17 objects=11 duration=2.410809ms38742026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads38752026/11/11 23:37:42 INFO Vacuumed table table=closures38762026/11/11 23:37:42 INFO Vacuumed table table=closures38772026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038782026/11/11 23:37:42 INFO Vacuumed table table=objects38792026/11/11 23:37:42 INFO Received complete upload request method=POST path=/38802026/11/11 23:37:42 INFO Vacuumed table table=objects38812026/11/11 23:37:42 INFO Received uploads request method=POST path=/38822026/11/11 23:37:42 INFO Received push request method=POST path=/38832026/11/11 23:37:42 INFO Completed upload id=23 objects=16 duration=2.650478ms38842026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038852026/11/11 23:37:42 INFO Vacuumed table table=pending_closures38862026/11/11 23:37:42 INFO Vacuumed table table=pending_objects38872026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads38882026/11/11 23:37:42 INFO Vacuumed table table=closures38892026/11/11 23:37:42 INFO Vacuumed table table=objects38902026/11/11 23:37:42 INFO Received push request method=POST path=/38912026/11/11 23:37:42 INFO Received uploads request method=POST path=/38922026/11/11 23:37:42 INFO Received push request method=POST path=/38932026/11/11 23:37:42 INFO Received push request method=POST path=/38942026/11/11 23:37:42 INFO Received push request method=POST path=/38952026/11/11 23:37:42 INFO Received uploads request method=POST path=/38962026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=038972026/11/11 23:37:42 INFO Received uploads request method=POST path=/38982026/11/11 23:37:42 INFO Received uploads request method=POST path=/38992026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period39002026/11/11 23:37:42 INFO Received uploads request method=POST path=/39012026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=039022026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period39032026/11/11 23:37:42 INFO Received push request method=POST path=/39042026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039052026/11/11 23:37:42 INFO Vacuumed table table=pending_closures39062026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039072026/11/11 23:37:42 INFO Vacuumed table table=pending_objects39082026/11/11 23:37:42 INFO Received uploads request method=POST path=/39092026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads39102026/11/11 23:37:42 INFO Vacuumed table table=pending_closures39112026/11/11 23:37:42 INFO Vacuumed table table=closures39122026/11/11 23:37:42 INFO Received push request method=POST path=/39132026/11/11 23:37:42 INFO Vacuumed table table=pending_objects39142026/11/11 23:37:42 INFO Vacuumed table table=objects39152026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads39162026/11/11 23:37:42 INFO Vacuumed table table=closures39172026/11/11 23:37:42 INFO Received uploads request method=POST path=/39182026/11/11 23:37:42 INFO Vacuumed table table=objects39192026/11/11 23:37:42 INFO Received uploads request method=POST path=/39202026/11/11 23:37:42 INFO Received push request method=POST path=/39212026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=039222026/11/11 23:37:42 INFO Received uploads request method=POST path=/39232026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period39242026/11/11 23:37:42 INFO Received push request method=POST path=/39252026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039262026/11/11 23:37:42 INFO Received uploads request method=POST path=/39272026/11/11 23:37:42 INFO Vacuumed table table=pending_closures39282026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39292026/11/11 23:37:42 INFO Vacuumed table table=pending_objects39302026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads39312026/11/11 23:37:42 INFO Vacuumed table table=closures39322026/11/11 23:37:42 INFO Completed upload id=23 objects=15 duration=2.603369ms39332026/11/11 23:37:42 INFO Vacuumed table table=objects39342026/11/11 23:37:42 INFO Received uploads request method=POST path=/39352026/11/11 23:37:42 INFO Received uploads request method=POST path=/39362026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39372026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39382026/11/11 23:37:42 INFO Completed upload id=7 objects=10 duration=2.331729ms39392026/11/11 23:37:42 INFO Received complete push request method=POST path=/39402026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=039412026/11/11 23:37:42 INFO Completed push id=14 objects=3 duration=1.397039ms39422026/11/11 23:37:42 INFO Completed upload id=12 objects=26 duration=3.075698ms39432026/11/11 23:37:42 INFO Received push request method=POST path=/39442026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039452026/11/11 23:37:42 INFO Received uploads request method=POST path=/39462026/11/11 23:37:42 INFO Received push request method=POST path=/39472026/11/11 23:37:42 INFO Vacuumed table table=pending_closures39482026/11/11 23:37:42 INFO Vacuumed table table=pending_objects39492026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads39502026/11/11 23:37:42 INFO Vacuumed table table=closures39512026/11/11 23:37:42 INFO Vacuumed table table=objects39522026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=039532026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period39542026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039552026/11/11 23:37:42 INFO Vacuumed table table=pending_closures39562026/11/11 23:37:42 INFO Vacuumed table table=pending_objects39572026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads39582026/11/11 23:37:42 INFO Vacuumed table table=closures39592026/11/11 23:37:42 INFO Received uploads request method=POST path=/39602026/11/11 23:37:42 INFO Vacuumed table table=objects39612026/11/11 23:37:42 INFO Received uploads request method=POST path=/39622026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39632026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39642026/11/11 23:37:42 INFO Completed upload id=26 objects=6 duration=1.336669ms39652026/11/11 23:37:42 INFO Completed upload id=27 objects=4 duration=1.465299ms39662026/11/11 23:37:42 INFO Received uploads request method=POST path=/39672026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=039682026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period39692026/11/11 23:37:42 INFO Received uploads request method=POST path=/39702026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039712026/11/11 23:37:42 INFO Vacuumed table table=pending_closures39722026/11/11 23:37:42 INFO Vacuumed table table=pending_objects39732026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads39742026/11/11 23:37:42 INFO Vacuumed table table=closures39752026/11/11 23:37:42 INFO Vacuumed table table=objects39762026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39772026/11/11 23:37:42 INFO Completed upload id=19 objects=6 duration=1.346549ms39782026/11/11 23:37:42 INFO Received uploads request method=POST path=/39792026/11/11 23:37:42 INFO Received complete push request method=POST path=/39802026/11/11 23:37:42 INFO Received uploads request method=POST path=/39812026/11/11 23:37:42 INFO Completed push id=13 objects=11 duration=2.001459ms39822026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=039832026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period39842026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039852026/11/11 23:37:42 INFO Received push request method=POST path=/39862026/11/11 23:37:42 INFO Vacuumed table table=pending_closures39872026/11/11 23:37:42 INFO Vacuumed table table=pending_objects39882026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39892026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads39902026/11/11 23:37:42 INFO Vacuumed table table=closures39912026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39922026/11/11 23:37:42 INFO Vacuumed table table=objects39932026/11/11 23:37:42 INFO Completed upload id=16 objects=17 duration=2.810768ms39942026/11/11 23:37:42 INFO Completed upload id=17 objects=25 duration=2.987909ms39952026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39962026/11/11 23:37:42 INFO Completed upload id=20 objects=4 duration=968.96µs39972026/11/11 23:37:42 INFO Received uploads request method=POST path=/39982026/11/11 23:37:42 INFO Received complete upload request method=POST path=/39992026/11/11 23:37:42 INFO Received uploads request method=POST path=/40002026/11/11 23:37:42 INFO Completed upload id=29 objects=5 duration=1.460279ms40012026/11/11 23:37:42 INFO Received complete push request method=POST path=/40022026/11/11 23:37:42 INFO Completed push id=16 objects=8 duration=2.114219ms40032026/11/11 23:37:42 INFO Received complete upload request method=POST path=/40042026/11/11 23:37:42 INFO Received uploads request method=POST path=/40052026/11/11 23:37:42 INFO Completed upload id=22 objects=7 duration=1.574579ms40062026/11/11 23:37:42 INFO Received push request method=POST path=/40072026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040082026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period40092026/11/11 23:37:42 INFO Received push request method=POST path=/40102026/11/11 23:37:42 INFO Received uploads request method=POST path=/40112026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040122026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period40132026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040142026/11/11 23:37:42 INFO Received complete upload request method=POST path=/40152026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period40162026/11/11 23:37:42 INFO Completed upload id=23 objects=18 duration=2.766199ms40172026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040182026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040192026/11/11 23:37:42 INFO Vacuumed table table=pending_closures40202026/11/11 23:37:42 INFO Vacuumed table table=pending_objects40212026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads40222026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period40232026/11/11 23:37:42 INFO Vacuumed table table=closures40242026/11/11 23:37:42 INFO Vacuumed table table=objects40252026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040262026/11/11 23:37:42 INFO Received push request method=POST path=/40272026/11/11 23:37:42 INFO Vacuumed table table=pending_closures40282026/11/11 23:37:42 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40292026/11/11 23:37:42 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40302026/11/11 23:37:42 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40312026/11/11 23:37:42 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40322026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40332026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40342026/11/11 23:37:42 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40352026/11/11 23:37:42 INFO Received complete push request method=POST path=/40362026/11/11 23:37:42 INFO Vacuumed table table=pending_objects40372026/11/11 23:37:42 INFO Received push request method=POST path=/40382026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads40392026/11/11 23:37:42 INFO Completed push id=16 objects=5 duration=1.36174ms40402026/11/11 23:37:42 INFO Vacuumed table table=closures40412026/11/11 23:37:42 INFO Received complete upload request method=POST path=/40422026/11/11 23:37:42 INFO Vacuumed table table=objects40432026/11/11 23:37:42 INFO Completed upload id=13 objects=8 duration=1.804299ms40442026/11/11 23:37:42 INFO Received complete upload request method=POST path=/40452026/11/11 23:37:42 INFO Received complete upload request method=POST path=/40462026/11/11 23:37:42 INFO Completed upload id=21 objects=2 duration=1.176879ms40472026/11/11 23:37:42 INFO Completed upload id=11 objects=7 duration=1.61034ms40482026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040492026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period40502026/11/11 23:37:42 INFO Received uploads request method=POST path=/40512026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040522026/11/11 23:37:42 INFO Received complete push request method=POST path=/40532026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040542026/11/11 23:37:42 INFO Vacuumed table table=pending_closures40552026/11/11 23:37:42 INFO Vacuumed table table=pending_objects40562026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads40572026/11/11 23:37:42 INFO Completed push id=7 objects=15 duration=3.420868ms40582026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040592026/11/11 23:37:42 INFO Vacuumed table table=closures40602026/11/11 23:37:42 INFO Vacuumed table table=pending_closures40612026/11/11 23:37:42 INFO Vacuumed table table=objects40622026/11/11 23:37:42 INFO Received uploads request method=POST path=/40632026/11/11 23:37:42 INFO Vacuumed table table=pending_objects40642026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads40652026/11/11 23:37:42 INFO Vacuumed table table=closures40662026/11/11 23:37:42 INFO Vacuumed table table=objects40672026/11/11 23:37:42 INFO Received push request method=POST path=/40682026/11/11 23:37:42 INFO Received uploads request method=POST path=/40692026/11/11 23:37:42 INFO Received uploads request method=POST path=/40702026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040712026/11/11 23:37:42 WARN Force mode enabled - objects will be deleted immediately without grace period40722026/11/11 23:37:42 INFO Received complete push request method=POST path=/40732026/11/11 23:37:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040742026/11/11 23:37:42 INFO Completed push id=18 objects=6 duration=2.082458ms40752026/11/11 23:37:42 INFO Vacuumed table table=pending_closures40762026/11/11 23:37:42 INFO Received complete push request method=POST path=/40772026/11/11 23:37:42 INFO Vacuumed table table=pending_objects40782026/11/11 23:37:42 INFO Vacuumed table table=multipart_uploads40792026/11/11 23:37:42 INFO Received push request method=POST path=/40802026/11/11 23:37:42 INFO Vacuumed table table=closures40812026/11/11 23:37:42 INFO Received uploads request method=POST path=/40822026/11/11 23:37:42 INFO Completed push id=11 objects=6 duration=3.347609ms40832026/11/11 23:37:42 INFO Vacuumed table table=objects40842026/11/11 23:37:42 INFO Received complete push request method=POST path=/40852026/11/11 23:37:42 INFO Completed push id=24 objects=9 duration=2.100909ms40862026/11/11 23:37:42 INFO Received complete upload request method=POST path=/40872026/11/11 23:37:42 INFO Received push request method=POST path=/40882026/11/11 23:37:42 INFO Completed upload id=21 objects=3 duration=1.094409ms40892026/11/11 23:37:42 INFO Aborted multipart uploads count=0 kept=040902026/11/11 23:37:42 INFO Received complete push request method=POST path=/40912026/11/11 23:37:42 INFO Received uploads request method=POST path=/40922026/11/11 23:37:43 INFO Completed push id=31 objects=14 duration=3.473738ms40932026/11/11 23:37:43 INFO Received uploads request method=POST path=/40942026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040952026/11/11 23:37:43 INFO Received complete upload request method=POST path=/40962026/11/11 23:37:43 INFO Vacuumed table table=pending_closures40972026/11/11 23:37:43 INFO Completed upload id=23 objects=9 duration=823.909µs40982026/11/11 23:37:43 INFO Vacuumed table table=pending_objects40992026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41002026/11/11 23:37:43 INFO Vacuumed table table=closures41012026/11/11 23:37:43 INFO Vacuumed table table=objects41022026/11/11 23:37:43 INFO Received complete upload request method=POST path=/41032026/11/11 23:37:43 INFO Completed upload id=22 objects=22 duration=1.274409ms41042026/11/11 23:37:43 INFO Received push request method=POST path=/41052026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=041062026/11/11 23:37:43 INFO Received push request method=POST path=/41072026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period41082026/11/11 23:37:43 INFO Received complete push request method=POST path=/41092026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=041102026/11/11 23:37:43 INFO Received complete push request method=POST path=/41112026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41122026/11/11 23:37:43 INFO Received complete upload request method=POST path=/41132026/11/11 23:37:43 INFO Completed push id=18 objects=6 duration=1.275179ms41142026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41152026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41162026/11/11 23:37:43 INFO Completed push id=16 objects=14 duration=3.057659ms41172026/11/11 23:37:43 INFO Vacuumed table table=closures41182026/11/11 23:37:43 INFO Completed upload id=33 objects=5 duration=2.718578ms41192026/11/11 23:37:43 INFO Vacuumed table table=objects41202026/11/11 23:37:43 INFO Received push request method=POST path=/41212026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=041222026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041232026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=041242026/11/11 23:37:43 INFO Received push request method=POST path=/41252026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41262026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41272026/11/11 23:37:43 INFO Received uploads request method=POST path=/41282026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41292026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41302026/11/11 23:37:43 INFO Vacuumed table table=closures41312026/11/11 23:37:43 INFO Received push request method=POST path=/41322026/11/11 23:37:43 INFO Vacuumed table table=objects41332026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41342026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41352026/11/11 23:37:43 INFO Vacuumed table table=closures41362026/11/11 23:37:43 INFO Vacuumed table table=objects41372026/11/11 23:37:43 INFO Received push request method=POST path=/41382026/11/11 23:37:43 INFO Received push request method=POST path=/41392026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=041402026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=041412026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period41422026/11/11 23:37:43 INFO Received uploads request method=POST path=/41432026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041442026/11/11 23:37:43 INFO Received push request method=POST path=/41452026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041462026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41472026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41482026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41492026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41502026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41512026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41522026/11/11 23:37:43 INFO Vacuumed table table=closures41532026/11/11 23:37:43 INFO Vacuumed table table=closures41542026/11/11 23:37:43 INFO Received uploads request method=POST path=/41552026/11/11 23:37:43 INFO Vacuumed table table=objects41562026/11/11 23:37:43 INFO Vacuumed table table=objects41572026/11/11 23:37:43 INFO Received complete push request method=POST path=/41582026/11/11 23:37:43 INFO Completed push id=26 objects=5 duration=2.279729ms41592026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=041602026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period41612026/11/11 23:37:43 INFO Received complete push request method=POST path=/41622026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041632026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41642026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41652026/11/11 23:37:43 INFO Received complete push request method=POST path=/41662026/11/11 23:37:43 INFO Completed push id=7 objects=24 duration=4.317918ms41672026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41682026/11/11 23:37:43 INFO Vacuumed table table=closures41692026/11/11 23:37:43 INFO Vacuumed table table=objects41702026/11/11 23:37:43 INFO Completed push id=27 objects=25 duration=2.957849ms41712026/11/11 23:37:43 INFO Received uploads request method=POST path=/41722026/11/11 23:37:43 INFO Received uploads request method=POST path=/41732026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=041742026/11/11 23:37:43 INFO Received complete upload request method=POST path=/41752026/11/11 23:37:43 INFO Received complete upload request method=POST path=/41762026/11/11 23:37:43 INFO Received complete push request method=POST path=/41772026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041782026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=041792026/11/11 23:37:43 INFO Completed upload id=20 objects=3 duration=2.472639ms41802026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41812026/11/11 23:37:43 INFO Received complete upload request method=POST path=/41822026/11/11 23:37:43 INFO Completed upload id=31 objects=3 duration=2.836418ms41832026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period41842026/11/11 23:37:43 INFO Completed push id=5 objects=28 duration=2.454739ms41852026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41862026/11/11 23:37:43 INFO Completed upload id=24 objects=14 duration=1.47821ms41872026/11/11 23:37:43 INFO Received uploads request method=POST path=/41882026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41892026/11/11 23:37:43 INFO Vacuumed table table=closures41902026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041912026/11/11 23:37:43 INFO Vacuumed table table=pending_closures41922026/11/11 23:37:43 INFO Received complete upload request method=POST path=/41932026/11/11 23:37:43 INFO Vacuumed table table=pending_objects41942026/11/11 23:37:43 INFO Vacuumed table table=objects41952026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads41962026/11/11 23:37:43 INFO Vacuumed table table=closures41972026/11/11 23:37:43 INFO Completed upload id=21 objects=25 duration=1.850638ms41982026/11/11 23:37:43 INFO Vacuumed table table=objects41992026/11/11 23:37:43 INFO Received uploads request method=POST path=/42002026/11/11 23:37:43 INFO Received complete upload request method=POST path=/42012026/11/11 23:37:43 INFO Completed upload id=15 objects=3 duration=2.106209ms42022026/11/11 23:37:43 INFO Received push request method=POST path=/42032026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=042042026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042052026/11/11 23:37:43 INFO Received complete push request method=POST path=/42062026/11/11 23:37:43 INFO Vacuumed table table=pending_closures42072026/11/11 23:37:43 INFO Completed push id=7 objects=7 duration=1.560899ms42082026/11/11 23:37:43 INFO Vacuumed table table=pending_objects42092026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads42102026/11/11 23:37:43 INFO Vacuumed table table=closures42112026/11/11 23:37:43 INFO Vacuumed table table=objects42122026/11/11 23:37:43 INFO Received push request method=POST path=/42132026/11/11 23:37:43 INFO Received complete upload request method=POST path=/42142026/11/11 23:37:43 INFO Received complete push request method=POST path=/42152026/11/11 23:37:43 INFO Completed upload id=22 objects=2 duration=1.332129ms42162026/11/11 23:37:43 INFO Completed push id=10 objects=2 duration=1.640509ms42172026/11/11 23:37:43 INFO Received complete push request method=POST path=/42182026/11/11 23:37:43 INFO Received push request method=POST path=/42192026/11/11 23:37:43 INFO Received push request method=POST path=/42202026/11/11 23:37:43 INFO Received uploads request method=POST path=/42212026/11/11 23:37:43 INFO Completed push id=19 objects=3 duration=1.64493ms42222026/11/11 23:37:43 INFO Received uploads request method=POST path=/42232026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=042242026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042252026/11/11 23:37:43 INFO Received push request method=POST path=/42262026/11/11 23:37:43 INFO Vacuumed table table=pending_closures42272026/11/11 23:37:43 INFO Vacuumed table table=pending_objects42282026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads42292026/11/11 23:37:43 INFO Vacuumed table table=closures42302026/11/11 23:37:43 INFO Vacuumed table table=objects42312026/11/11 23:37:43 INFO Received push request method=POST path=/42322026/11/11 23:37:43 INFO Received complete push request method=POST path=/42332026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=042342026/11/11 23:37:43 INFO Received complete push request method=POST path=/42352026/11/11 23:37:43 INFO Completed push id=13 objects=15 duration=2.411269ms42362026/11/11 23:37:43 INFO Received complete push request method=POST path=/42372026/11/11 23:37:43 INFO Received complete push request method=POST path=/42382026/11/11 23:37:43 INFO Completed push id=12 objects=6 duration=1.691989ms42392026/11/11 23:37:43 INFO Received uploads request method=POST path=/42402026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period42412026/11/11 23:37:43 INFO Completed push id=24 objects=5 duration=3.126378ms42422026/11/11 23:37:43 INFO Completed push id=22 objects=3 duration=3.188148ms42432026/11/11 23:37:43 INFO Received uploads request method=POST path=/42442026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042452026/11/11 23:37:43 INFO Vacuumed table table=pending_closures42462026/11/11 23:37:43 INFO Received complete upload request method=POST path=/42472026/11/11 23:37:43 INFO Vacuumed table table=pending_objects42482026/11/11 23:37:43 INFO Completed upload id=17 objects=5 duration=1.391249ms42492026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads42502026/11/11 23:37:43 INFO Vacuumed table table=closures42512026/11/11 23:37:43 INFO Vacuumed table table=objects42522026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=042532026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042542026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=042552026/11/11 23:37:43 INFO Vacuumed table table=pending_closures42562026/11/11 23:37:43 INFO Vacuumed table table=pending_objects42572026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads42582026/11/11 23:37:43 INFO Vacuumed table table=closures42592026/11/11 23:37:43 INFO Vacuumed table table=objects42602026/11/11 23:37:43 INFO Received uploads request method=POST path=/42612026/11/11 23:37:43 INFO Received complete push request method=POST path=/42622026/11/11 23:37:43 INFO Received uploads request method=POST path=/42632026/11/11 23:37:43 INFO Received complete upload request method=POST path=/42642026/11/11 23:37:43 INFO Received complete upload request method=POST path=/42652026/11/11 23:37:43 INFO Completed upload id=24 objects=7 duration=1.28381ms42662026/11/11 23:37:43 INFO Completed push id=19 objects=14 duration=2.326809ms42672026/11/11 23:37:43 INFO Completed upload id=25 objects=6 duration=1.842759ms42682026/11/11 23:37:43 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42692026/11/11 23:37:43 INFO Received push request method=POST path=/42702026/11/11 23:37:43 INFO Received complete upload request method=POST path=/42712026/11/11 23:37:43 INFO Completed upload id=12 objects=6 duration=1.440809ms42722026/11/11 23:37:43 INFO Received complete upload request method=POST path=/42732026/11/11 23:37:43 INFO Completed upload id=18 objects=2 duration=2.267409ms42742026/11/11 23:37:43 INFO Received complete push request method=POST path=/42752026/11/11 23:37:43 INFO Completed push id=19 objects=3 duration=1.13173ms42762026/11/11 23:37:43 INFO Received complete push request method=POST path=/42772026/11/11 23:37:43 INFO Completed push id=16 objects=26 duration=1.29484ms42782026/11/11 23:37:43 INFO Received uploads request method=POST path=/42792026/11/11 23:37:43 INFO Received complete push request method=POST path=/42802026/11/11 23:37:43 INFO Completed push id=21 objects=3 duration=1.758809ms42812026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=042822026/11/11 23:37:43 INFO Received uploads request method=POST path=/42832026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=042842026/11/11 23:37:43 INFO Received complete push request method=POST path=/42852026/11/11 23:37:43 INFO Received push request method=POST path=/42862026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period42872026/11/11 23:37:43 INFO Completed push id=17 objects=3 duration=2.094559ms42882026/11/11 23:37:43 INFO Received complete push request method=POST path=/42892026/11/11 23:37:43 INFO Completed push id=13 objects=22 duration=1.756009ms42902026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042912026/11/11 23:37:43 INFO Vacuumed table table=pending_closures42922026/11/11 23:37:43 INFO Received complete push request method=POST path=/42932026/11/11 23:37:43 INFO Received push request method=POST path=/42942026/11/11 23:37:43 INFO Vacuumed table table=pending_objects42952026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads42962026/11/11 23:37:43 INFO Vacuumed table table=closures42972026/11/11 23:37:43 INFO Completed push id=17 objects=22 duration=2.656509ms42982026/11/11 23:37:43 INFO Vacuumed table table=objects42992026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=043002026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period43012026/11/11 23:37:43 INFO Received uploads request method=POST path=/43022026/11/11 23:37:43 INFO Received push request method=POST path=/43032026/11/11 23:37:43 INFO Received complete push request method=POST path=/43042026/11/11 23:37:43 INFO Received push request method=POST path=/43052026/11/11 23:37:43 INFO Received uploads request method=POST path=/43062026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043072026/11/11 23:37:43 INFO Completed push id=20 objects=3 duration=2.392458ms43082026/11/11 23:37:43 INFO Vacuumed table table=pending_closures43092026/11/11 23:37:43 INFO Received complete push request method=POST path=/43102026/11/11 23:37:43 INFO Vacuumed table table=pending_objects43112026/11/11 23:37:43 INFO Received complete push request method=POST path=/43122026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads43132026/11/11 23:37:43 INFO Completed push id=20 objects=4 duration=1.470039ms43142026/11/11 23:37:43 INFO Vacuumed table table=closures43152026/11/11 23:37:43 INFO Completed push id=13 objects=2 duration=1.538469ms43162026/11/11 23:37:43 INFO Vacuumed table table=objects43172026/11/11 23:37:43 INFO Received complete push request method=POST path=/43182026/11/11 23:37:43 INFO Received complete push request method=POST path=/43192026/11/11 23:37:43 INFO Completed push id=21 objects=6 duration=1.563629ms43202026/11/11 23:37:43 INFO Completed push id=23 objects=5 duration=1.547459ms43212026/11/11 23:37:43 INFO Received uploads request method=POST path=/43222026/11/11 23:37:43 INFO Received push request method=POST path=/43232026/11/11 23:37:43 INFO Received complete push request method=POST path=/43242026/11/11 23:37:43 INFO Received uploads request method=POST path=/43252026/11/11 23:37:43 INFO Completed push id=21 objects=26 duration=1.674759ms43262026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43272026/11/11 23:37:43 INFO Received complete push request method=POST path=/43282026/11/11 23:37:43 INFO Completed upload id=13 objects=7 duration=1.05074ms43292026/11/11 23:37:43 INFO Completed push id=24 objects=3 duration=1.506149ms43302026/11/11 23:37:43 INFO Received uploads request method=POST path=/43312026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43322026/11/11 23:37:43 INFO Completed upload id=25 objects=4 duration=1.414499ms43332026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43342026/11/11 23:37:43 INFO Completed upload id=27 objects=13 duration=1.779609ms43352026/11/11 23:37:43 INFO Received complete push request method=POST path=/43362026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43372026/11/11 23:37:43 INFO Completed push id=27 objects=6 duration=2.144439ms43382026/11/11 23:37:43 INFO Completed upload id=22 objects=7 duration=1.051659ms43392026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43402026/11/11 23:37:43 INFO Completed upload id=24 objects=3 duration=1.165799ms43412026/11/11 23:37:43 INFO Received uploads request method=POST path=/43422026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=043432026/11/11 23:37:43 INFO Vacuumed table table=pending_closures43442026/11/11 23:37:43 INFO Vacuumed table table=pending_objects43452026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads43462026/11/11 23:37:43 INFO Vacuumed table table=closures43472026/11/11 23:37:43 INFO Vacuumed table table=objects43482026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43492026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43502026/11/11 23:37:43 INFO Received push request method=POST path=/43512026/11/11 23:37:43 INFO Completed upload id=20 objects=4 duration=975.509µs43522026/11/11 23:37:43 INFO Completed upload id=22 objects=8 duration=1.392659ms43532026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=043542026/11/11 23:37:43 INFO Received complete push request method=POST path=/43552026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period43562026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43572026/11/11 23:37:43 INFO Completed push id=11 objects=9 duration=1.683629ms43582026/11/11 23:37:43 INFO Completed upload id=28 objects=17 duration=1.271449ms43592026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=043602026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043612026/11/11 23:37:43 INFO Received push request method=POST path=/43622026/11/11 23:37:43 INFO Vacuumed table table=pending_closures43632026/11/11 23:37:43 INFO Vacuumed table table=pending_objects43642026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads43652026/11/11 23:37:43 INFO Vacuumed table table=closures43662026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=043672026/11/11 23:37:43 INFO Vacuumed table table=objects43682026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043692026/11/11 23:37:43 INFO Vacuumed table table=pending_closures43702026/11/11 23:37:43 INFO Received complete push request method=POST path=/43712026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043722026/11/11 23:37:43 INFO Received push request method=POST path=/43732026/11/11 23:37:43 INFO Completed push id=24 objects=4 duration=1.198239ms43742026/11/11 23:37:43 INFO Received complete push request method=POST path=/43752026/11/11 23:37:43 INFO Vacuumed table table=pending_objects43762026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads43772026/11/11 23:37:43 INFO Vacuumed table table=pending_closures43782026/11/11 23:37:43 INFO Completed push id=25 objects=4 duration=1.56785ms43792026/11/11 23:37:43 INFO Vacuumed table table=closures43802026/11/11 23:37:43 INFO Vacuumed table table=pending_objects43812026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads43822026/11/11 23:37:43 INFO Vacuumed table table=objects43832026/11/11 23:37:43 INFO Vacuumed table table=closures43842026/11/11 23:37:43 INFO Vacuumed table table=objects43852026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43862026/11/11 23:37:43 INFO Received push request method=POST path=/43872026/11/11 23:37:43 INFO Completed upload id=23 objects=7 duration=1.101579ms43882026/11/11 23:37:43 INFO Received complete push request method=POST path=/43892026/11/11 23:37:43 INFO Completed push id=23 objects=5 duration=1.113519ms43902026/11/11 23:37:43 INFO Received complete push request method=POST path=/43912026/11/11 23:37:43 INFO Completed push id=29 objects=2 duration=1.309259ms43922026/11/11 23:37:43 INFO Received complete push request method=POST path=/43932026/11/11 23:37:43 INFO Completed push id=15 objects=24 duration=1.612069ms43942026/11/11 23:37:43 INFO Received complete upload request method=POST path=/43952026/11/11 23:37:43 INFO Completed upload id=15 objects=2 duration=884.969µs43962026/11/11 23:37:43 INFO Received complete push request method=POST path=/43972026/11/11 23:37:43 INFO Completed push id=25 objects=9 duration=1.252859ms43982026/11/11 23:37:43 INFO Received complete push request method=POST path=/43992026/11/11 23:37:43 INFO Received uploads request method=POST path=/44002026/11/11 23:37:43 INFO Completed push id=23 objects=28 duration=3.575999ms44012026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=044022026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period44032026/11/11 23:37:43 INFO Received push request method=POST path=/44042026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044052026/11/11 23:37:43 INFO Received push request method=POST path=/44062026/11/11 23:37:43 INFO Received uploads request method=POST path=/44072026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44082026/11/11 23:37:43 INFO Vacuumed table table=pending_closures44092026/11/11 23:37:43 INFO Completed upload id=24 objects=8 duration=1.282329ms44102026/11/11 23:37:43 INFO Vacuumed table table=pending_objects44112026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads44122026/11/11 23:37:43 INFO Received complete push request method=POST path=/44132026/11/11 23:37:43 INFO Vacuumed table table=closures44142026/11/11 23:37:43 INFO Received complete push request method=POST path=/44152026/11/11 23:37:43 INFO Vacuumed table table=objects44162026/11/11 23:37:43 INFO Completed push id=31 objects=6 duration=1.42411ms44172026/11/11 23:37:43 INFO Completed push id=20 objects=8 duration=1.20245ms44182026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44192026/11/11 23:37:43 INFO Completed upload id=19 objects=2 duration=998.729µs44202026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=044212026/11/11 23:37:43 INFO Received push request method=POST path=/44222026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044232026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44242026/11/11 23:37:43 INFO Vacuumed table table=pending_closures44252026/11/11 23:37:43 INFO Completed upload id=26 objects=3 duration=1.432309ms44262026/11/11 23:37:43 INFO Vacuumed table table=pending_objects44272026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads44282026/11/11 23:37:43 INFO Received push request method=POST path=/44292026/11/11 23:37:43 INFO Vacuumed table table=closures44302026/11/11 23:37:43 INFO Vacuumed table table=objects44312026/11/11 23:37:43 INFO Received push request method=POST path=/44322026/11/11 23:37:43 INFO Received push request method=POST path=/44332026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=044342026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44352026/11/11 23:37:43 INFO Completed upload id=28 objects=5 duration=1.048219ms44362026/11/11 23:37:43 INFO Received uploads request method=POST path=/44372026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044382026/11/11 23:37:43 INFO Received complete push request method=POST path=/44392026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44402026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44412026/11/11 23:37:43 INFO Received complete push request method=POST path=/44422026/11/11 23:37:43 INFO Vacuumed table table=pending_closures44432026/11/11 23:37:43 INFO Completed upload id=24 objects=7 duration=1.059169ms44442026/11/11 23:37:43 INFO Completed push id=32 objects=9 duration=1.674409ms44452026/11/11 23:37:43 INFO Completed upload id=24 objects=18 duration=1.481569ms44462026/11/11 23:37:43 INFO Completed push id=26 objects=23 duration=1.397969ms44472026/11/11 23:37:43 INFO Vacuumed table table=pending_objects44482026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads44492026/11/11 23:37:43 INFO Vacuumed table table=closures44502026/11/11 23:37:43 INFO Vacuumed table table=objects44512026/11/11 23:37:43 INFO Received complete push request method=POST path=/44522026/11/11 23:37:43 INFO Completed push id=19 objects=14 duration=1.205859ms44532026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=044542026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period44552026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44562026/11/11 23:37:43 INFO Received push request method=POST path=/44572026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44582026/11/11 23:37:43 INFO Completed upload id=27 objects=8 duration=1.318969ms44592026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044602026/11/11 23:37:43 INFO Completed upload id=27 objects=7 duration=1.13744ms44612026/11/11 23:37:43 INFO Received push request method=POST path=/44622026/11/11 23:37:43 INFO Vacuumed table table=pending_closures44632026/11/11 23:37:43 INFO Vacuumed table table=pending_objects44642026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads44652026/11/11 23:37:43 INFO Received complete push request method=POST path=/44662026/11/11 23:37:43 INFO Vacuumed table table=closures44672026/11/11 23:37:43 INFO Vacuumed table table=objects44682026/11/11 23:37:43 INFO Completed push id=30 objects=4 duration=3.373679ms44692026/11/11 23:37:43 INFO Received complete push request method=POST path=/44702026/11/11 23:37:43 INFO Received push request method=POST path=/44712026/11/11 23:37:43 INFO Completed push id=26 objects=3 duration=1.31237ms44722026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44732026/11/11 23:37:43 INFO Completed upload id=30 objects=5 duration=1.082089ms44742026/11/11 23:37:43 INFO Received complete push request method=POST path=/44752026/11/11 23:37:43 INFO Received uploads request method=POST path=/44762026/11/11 23:37:43 INFO Completed push id=25 objects=2 duration=1.864629ms44772026/11/11 23:37:43 INFO Received push request method=POST path=/44782026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44792026/11/11 23:37:43 INFO Completed upload id=33 objects=9 duration=1.388649ms44802026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44812026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=044822026/11/11 23:37:43 INFO Completed upload id=22 objects=6 duration=1.194499ms44832026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44842026/11/11 23:37:43 INFO Received uploads request method=POST path=/44852026/11/11 23:37:43 INFO Completed upload id=28 objects=3 duration=1.05999ms44862026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44872026/11/11 23:37:43 INFO Received uploads request method=POST path=/44882026/11/11 23:37:43 INFO Completed upload id=19 objects=2 duration=2.555828ms44892026/11/11 23:37:43 INFO Received complete upload request method=POST path=/44902026/11/11 23:37:43 INFO Received complete push request method=POST path=/44912026/11/11 23:37:43 INFO Completed upload id=28 objects=8 duration=1.110549ms44922026/11/11 23:37:43 INFO Completed push id=27 objects=15 duration=1.741109ms44932026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=044942026/11/11 23:37:43 INFO Received push request method=POST path=/44952026/11/11 23:37:43 INFO Received complete push request method=POST path=/44962026/11/11 23:37:43 INFO Vacuumed table table=pending_closures44972026/11/11 23:37:43 INFO Completed push id=27 objects=4 duration=1.512399ms44982026/11/11 23:37:43 INFO Received complete push request method=POST path=/44992026/11/11 23:37:43 INFO Vacuumed table table=pending_objects45002026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads45012026/11/11 23:37:43 INFO Completed push id=31 objects=3 duration=1.247449ms45022026/11/11 23:37:43 INFO Vacuumed table table=closures45032026/11/11 23:37:43 INFO Vacuumed table table=objects45042026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=045052026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=045062026/11/11 23:37:43 INFO Received uploads request method=POST path=/45072026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period45082026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045092026/11/11 23:37:43 INFO Received uploads request method=POST path=/45102026/11/11 23:37:43 INFO Received uploads request method=POST path=/45112026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045122026/11/11 23:37:43 INFO Vacuumed table table=pending_closures45132026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45142026/11/11 23:37:43 INFO Received push request method=POST path=/45152026/11/11 23:37:43 INFO Vacuumed table table=pending_closures45162026/11/11 23:37:43 INFO Completed upload id=20 objects=6 duration=1.23385ms45172026/11/11 23:37:43 INFO Vacuumed table table=pending_objects45182026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45192026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads45202026/11/11 23:37:43 INFO Completed upload id=29 objects=10 duration=1.00024ms45212026/11/11 23:37:43 INFO Vacuumed table table=closures45222026/11/11 23:37:43 INFO Received complete push request method=POST path=/45232026/11/11 23:37:43 INFO Vacuumed table table=pending_objects45242026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads45252026/11/11 23:37:43 INFO Received complete push request method=POST path=/45262026/11/11 23:37:43 INFO Completed push id=27 objects=11 duration=929.03µs45272026/11/11 23:37:43 INFO Vacuumed table table=objects45282026/11/11 23:37:43 INFO Vacuumed table table=closures45292026/11/11 23:37:43 INFO Completed push id=29 objects=23 duration=1.449559ms45302026/11/11 23:37:43 INFO Vacuumed table table=objects45312026/11/11 23:37:43 INFO Received uploads request method=POST path=/45322026/11/11 23:37:43 INFO Received complete push request method=POST path=/45332026/11/11 23:37:43 INFO Completed push id=33 objects=7 duration=2.061259ms45342026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45352026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45362026/11/11 23:37:43 INFO Received complete push request method=POST path=/45372026/11/11 23:37:43 INFO Completed upload id=29 objects=4 duration=1.980279ms45382026/11/11 23:37:43 INFO Received complete push request method=POST path=/45392026/11/11 23:37:43 INFO Completed upload id=13 objects=14 duration=3.120708ms45402026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45412026/11/11 23:37:43 INFO Completed push id=28 objects=26 duration=1.868569ms45422026/11/11 23:37:43 INFO Completed push id=32 objects=6 duration=1.333339ms45432026/11/11 23:37:43 INFO Completed upload id=30 objects=20 duration=1.211419ms45442026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45452026/11/11 23:37:43 INFO Received push request method=POST path=/45462026/11/11 23:37:43 INFO Completed upload id=29 objects=26 duration=806.589µs45472026/11/11 23:37:43 INFO Received complete push request method=POST path=/45482026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=045492026/11/11 23:37:43 INFO Received uploads request method=POST path=/45502026/11/11 23:37:43 INFO Completed push id=30 objects=6 duration=1.175339ms45512026/11/11 23:37:43 INFO Received uploads request method=POST path=/45522026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045532026/11/11 23:37:43 INFO Vacuumed table table=pending_closures45542026/11/11 23:37:43 INFO Received complete push request method=POST path=/45552026/11/11 23:37:43 INFO Vacuumed table table=pending_objects45562026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads45572026/11/11 23:37:43 INFO Vacuumed table table=closures45582026/11/11 23:37:43 INFO Received uploads request method=POST path=/45592026/11/11 23:37:43 INFO Completed push id=28 objects=19 duration=2.174329ms45602026/11/11 23:37:43 INFO Vacuumed table table=objects45612026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45622026/11/11 23:37:43 INFO Completed upload id=26 objects=3 duration=1.33064ms45632026/11/11 23:37:43 INFO Received uploads request method=POST path=/45642026/11/11 23:37:43 INFO Received uploads request method=POST path=/45652026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=045662026/11/11 23:37:43 INFO Received complete push request method=POST path=/45672026/11/11 23:37:43 INFO Completed push id=31 objects=11 duration=1.245649ms45682026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45692026/11/11 23:37:43 INFO Received complete push request method=POST path=/45702026/11/11 23:37:43 INFO Completed upload id=33 objects=7 duration=1.007189ms45712026/11/11 23:37:43 INFO Completed push id=30 objects=3 duration=1.320949ms45722026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045732026/11/11 23:37:43 INFO Received complete push request method=POST path=/45742026/11/11 23:37:43 INFO Completed push id=30 objects=8 duration=822.399µs45752026/11/11 23:37:43 INFO Vacuumed table table=pending_closures45762026/11/11 23:37:43 INFO Received complete push request method=POST path=/45772026/11/11 23:37:43 INFO Vacuumed table table=pending_objects45782026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads45792026/11/11 23:37:43 INFO Completed push id=32 objects=7 duration=1.727379ms45802026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45812026/11/11 23:37:43 INFO Vacuumed table table=closures45822026/11/11 23:37:43 INFO Vacuumed table table=objects45832026/11/11 23:37:43 INFO Completed upload id=26 objects=2 duration=2.008469ms45842026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45852026/11/11 23:37:43 INFO Completed upload id=26 objects=8 duration=1.05281ms45862026/11/11 23:37:43 INFO Received complete push request method=POST path=/45872026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45882026/11/11 23:37:43 INFO Completed push id=33 objects=3 duration=1.17353ms45892026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=045902026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period45912026/11/11 23:37:43 INFO Received push request method=POST path=/45922026/11/11 23:37:43 INFO Completed upload id=25 objects=16 duration=2.736379ms45932026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045942026/11/11 23:37:43 INFO Vacuumed table table=pending_closures45952026/11/11 23:37:43 INFO Vacuumed table table=pending_objects45962026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads45972026/11/11 23:37:43 INFO Received complete upload request method=POST path=/45982026/11/11 23:37:43 INFO Vacuumed table table=closures45992026/11/11 23:37:43 INFO Received uploads request method=POST path=/46002026/11/11 23:37:43 INFO Completed upload id=32 objects=2 duration=968.859µs46012026/11/11 23:37:43 INFO Vacuumed table table=objects46022026/11/11 23:37:43 INFO Received complete push request method=POST path=/46032026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46042026/11/11 23:37:43 INFO Completed push id=32 objects=19 duration=1.09939ms46052026/11/11 23:37:43 INFO Completed upload id=34 objects=3 duration=996.359µs46062026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46072026/11/11 23:37:43 INFO Completed upload id=31 objects=5 duration=750.779µs46082026/11/11 23:37:43 INFO Received complete push request method=POST path=/46092026/11/11 23:37:43 INFO Received complete push request method=POST path=/46102026/11/11 23:37:43 INFO Completed push id=26 objects=4 duration=1.298779ms46112026/11/11 23:37:43 INFO Completed push id=29 objects=10 duration=1.412089ms46122026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=046132026/11/11 23:37:43 INFO Received complete push request method=POST path=/46142026/11/11 23:37:43 INFO Completed push id=28 objects=14 duration=970.109µs46152026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046162026/11/11 23:37:43 INFO Received complete push request method=POST path=/46172026/11/11 23:37:43 INFO Received complete push request method=POST path=/46182026/11/11 23:37:43 INFO Vacuumed table table=pending_closures46192026/11/11 23:37:43 INFO Completed push id=34 objects=6 duration=1.15907ms46202026/11/11 23:37:43 INFO Vacuumed table table=pending_objects46212026/11/11 23:37:43 INFO Received push request method=POST path=/46222026/11/11 23:37:43 INFO Completed push id=31 objects=6 duration=3.129238ms46232026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads46242026/11/11 23:37:43 INFO Vacuumed table table=closures46252026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=046262026/11/11 23:37:43 INFO Vacuumed table table=objects46272026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period46282026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046292026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46302026/11/11 23:37:43 INFO Completed upload id=21 objects=2 duration=948.069µs46312026/11/11 23:37:43 INFO Vacuumed table table=pending_closures46322026/11/11 23:37:43 INFO Vacuumed table table=pending_objects46332026/11/11 23:37:43 INFO Received complete push request method=POST path=/46342026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads46352026/11/11 23:37:43 INFO Vacuumed table table=closures46362026/11/11 23:37:43 INFO Completed push id=33 objects=3 duration=1.482769ms46372026/11/11 23:37:43 INFO Vacuumed table table=objects46382026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=046392026/11/11 23:37:43 INFO Received complete push request method=POST path=/46402026/11/11 23:37:43 INFO Received complete push request method=POST path=/46412026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period46422026/11/11 23:37:43 INFO Completed push id=35 objects=11 duration=1.18611ms46432026/11/11 23:37:43 INFO Completed push id=32 objects=3 duration=730.8µs46442026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46452026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046462026/11/11 23:37:43 INFO Completed upload id=28 objects=13 duration=2.055159ms46472026/11/11 23:37:43 INFO Received complete push request method=POST path=/46482026/11/11 23:37:43 INFO Vacuumed table table=pending_closures46492026/11/11 23:37:43 INFO Received push request method=POST path=/46502026/11/11 23:37:43 INFO Completed push id=11 objects=6 duration=1.973239ms46512026/11/11 23:37:43 INFO Vacuumed table table=pending_objects46522026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads46532026/11/11 23:37:43 INFO Vacuumed table table=closures46542026/11/11 23:37:43 INFO Vacuumed table table=objects46552026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46562026/11/11 23:37:43 INFO Completed upload id=30 objects=6 duration=1.184639ms46572026/11/11 23:37:43 INFO Received uploads request method=POST path=/46582026/11/11 23:37:43 INFO Received complete push request method=POST path=/46592026/11/11 23:37:43 INFO Completed push id=35 objects=8 duration=1.280229ms46602026/11/11 23:37:43 INFO Received uploads request method=POST path=/46612026/11/11 23:37:43 INFO Received uploads request method=POST path=/46622026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46632026/11/11 23:37:43 INFO Received push request method=POST path=/46642026/11/11 23:37:43 INFO Completed upload id=34 objects=3 duration=1.013189ms46652026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=046662026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=046672026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period46682026/11/11 23:37:43 INFO Received complete push request method=POST path=/46692026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period46702026/11/11 23:37:43 INFO Completed push id=29 objects=3 duration=1.26393ms46712026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046722026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046732026/11/11 23:37:43 INFO Vacuumed table table=pending_closures46742026/11/11 23:37:43 INFO Received uploads request method=POST path=/46752026/11/11 23:37:43 INFO Received push request method=POST path=/46762026/11/11 23:37:43 INFO Vacuumed table table=pending_closures46772026/11/11 23:37:43 INFO Vacuumed table table=pending_objects46782026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads46792026/11/11 23:37:43 INFO Vacuumed table table=closures46802026/11/11 23:37:43 INFO Vacuumed table table=pending_objects46812026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads46822026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46832026/11/11 23:37:43 INFO Vacuumed table table=objects46842026/11/11 23:37:43 INFO Vacuumed table table=closures46852026/11/11 23:37:43 INFO Received complete push request method=POST path=/46862026/11/11 23:37:43 INFO Completed upload id=31 objects=5 duration=1.428599ms46872026/11/11 23:37:43 INFO Received uploads request method=POST path=/46882026/11/11 23:37:43 INFO Vacuumed table table=objects46892026/11/11 23:37:43 INFO Completed push id=20 objects=13 duration=2.083629ms46902026/11/11 23:37:43 INFO Received complete upload request method=POST path=/46912026/11/11 23:37:43 INFO Completed upload id=36 objects=9 duration=1.03948ms46922026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=046932026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=046942026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period46952026/11/11 23:37:43 INFO Received push request method=POST path=/46962026/11/11 23:37:43 INFO Received complete push request method=POST path=/46972026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period46982026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046992026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047002026/11/11 23:37:43 INFO Vacuumed table table=pending_closures47012026/11/11 23:37:43 INFO Vacuumed table table=pending_closures47022026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47032026/11/11 23:37:43 INFO Completed push id=9 objects=9 duration=3.840258ms47042026/11/11 23:37:43 INFO Vacuumed table table=pending_objects47052026/11/11 23:37:43 INFO Completed upload id=31 objects=2 duration=1.317219ms47062026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads47072026/11/11 23:37:43 INFO Vacuumed table table=pending_objects47082026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads47092026/11/11 23:37:43 INFO Vacuumed table table=closures47102026/11/11 23:37:43 INFO Vacuumed table table=closures47112026/11/11 23:37:43 INFO Vacuumed table table=objects47122026/11/11 23:37:43 INFO Vacuumed table table=objects47132026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=047142026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period47152026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47162026/11/11 23:37:43 INFO Completed upload id=34 objects=3 duration=1.016019ms47172026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047182026/11/11 23:37:43 INFO Received uploads request method=POST path=/47192026/11/11 23:37:43 INFO Vacuumed table table=pending_closures47202026/11/11 23:37:43 INFO Received complete push request method=POST path=/47212026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=047222026/11/11 23:37:43 INFO Completed push id=22 objects=4 duration=1.659679ms47232026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period47242026/11/11 23:37:43 INFO Vacuumed table table=pending_objects47252026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads47262026/11/11 23:37:43 INFO Vacuumed table table=closures47272026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047282026/11/11 23:37:43 INFO Vacuumed table table=objects47292026/11/11 23:37:43 INFO Vacuumed table table=pending_closures47302026/11/11 23:37:43 INFO Vacuumed table table=pending_objects47312026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47322026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads47332026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=047342026/11/11 23:37:43 INFO Received complete push request method=POST path=/47352026/11/11 23:37:43 INFO Vacuumed table table=closures47362026/11/11 23:37:43 INFO Completed upload id=21 objects=8 duration=1.559639ms47372026/11/11 23:37:43 INFO Completed push id=33 objects=6 duration=1.230209ms47382026/11/11 23:37:43 INFO Vacuumed table table=objects47392026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=047402026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period47412026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047422026/11/11 23:37:43 INFO Received uploads request method=POST path=/47432026/11/11 23:37:43 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."47442026/11/11 23:37:43 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."47452026/11/11 23:37:43 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."47462026/11/11 23:37:43 INFO Vacuumed table table=pending_closures47472026/11/11 23:37:43 INFO Vacuumed table table=pending_objects47482026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads47492026/11/11 23:37:43 INFO Received uploads request method=POST path=/47502026/11/11 23:37:43 INFO Received complete push request method=POST path=/47512026/11/11 23:37:43 INFO Vacuumed table table=closures47522026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47532026/11/11 23:37:43 INFO Completed push id=22 objects=6 duration=1.646869ms47542026/11/11 23:37:43 INFO Vacuumed table table=objects47552026/11/11 23:37:43 INFO Completed upload id=23 objects=7 duration=1.641659ms47562026/11/11 23:37:43 INFO Received uploads request method=POST path=/47572026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=047582026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period47592026/11/11 23:37:43 INFO Received complete push request method=POST path=/47602026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047612026/11/11 23:37:43 INFO Received push request method=POST path=/47622026/11/11 23:37:43 INFO Completed push id=1 objects=9 duration=2.109179ms47632026/11/11 23:37:43 INFO Vacuumed table table=pending_closures47642026/11/11 23:37:43 INFO Received uploads request method=POST path=/47652026/11/11 23:37:43 INFO Vacuumed table table=pending_objects47662026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads47672026/11/11 23:37:43 INFO Received complete push request method=POST path=/47682026/11/11 23:37:43 INFO Vacuumed table table=closures47692026/11/11 23:37:43 INFO Completed push id=24 objects=8 duration=1.339359ms47702026/11/11 23:37:43 INFO Received uploads request method=POST path=/47712026/11/11 23:37:43 INFO Vacuumed table table=objects47722026/11/11 23:37:43 INFO Received complete push request method=POST path=/47732026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47742026/11/11 23:37:43 INFO Completed push id=24 objects=5 duration=1.24898ms47752026/11/11 23:37:43 INFO Completed upload id=21 objects=9 duration=1.314329ms47762026/11/11 23:37:43 INFO Received push request method=POST path=/47772026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47782026/11/11 23:37:43 INFO Received push request method=POST path=/47792026/11/11 23:37:43 INFO Completed upload id=25 objects=9 duration=1.111479ms47802026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47812026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47822026/11/11 23:37:43 INFO Completed upload id=22 objects=5 duration=1.903219ms47832026/11/11 23:37:43 INFO Completed upload id=25 objects=2 duration=1.01703ms47842026/11/11 23:37:43 INFO Received complete push request method=POST path=/47852026/11/11 23:37:43 INFO Received complete push request method=POST path=/47862026/11/11 23:37:43 INFO Completed push id=25 objects=10 duration=1.185239ms47872026/11/11 23:37:43 INFO Completed push id=24 objects=24 duration=2.047269ms47882026/11/11 23:37:43 INFO Received push request method=POST path=/47892026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47902026/11/11 23:37:43 INFO Completed upload id=27 objects=3 duration=1.046619ms47912026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47922026/11/11 23:37:43 INFO Received complete push request method=POST path=/47932026/11/11 23:37:43 INFO Completed upload id=26 objects=20 duration=1.148169ms47942026/11/11 23:37:43 INFO Completed push id=28 objects=3 duration=1.898499ms47952026/11/11 23:37:43 INFO Received complete upload request method=POST path=/47962026/11/11 23:37:43 INFO Completed upload id=29 objects=5 duration=732.71µs47972026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=047982026/11/11 23:37:43 INFO Received complete push request method=POST path=/47992026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48002026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period48012026/11/11 23:37:43 INFO Completed upload id=28 objects=6 duration=966.95µs48022026/11/11 23:37:43 INFO Completed push id=28 objects=11 duration=2.577258ms48032026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048042026/11/11 23:37:43 INFO Vacuumed table table=pending_closures48052026/11/11 23:37:43 INFO Vacuumed table table=pending_objects48062026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads48072026/11/11 23:37:43 INFO Vacuumed table table=closures48082026/11/11 23:37:43 INFO Vacuumed table table=objects48092026/11/11 23:37:43 INFO Received complete push request method=POST path=/48102026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48112026/11/11 23:37:43 INFO Completed push id=29 objects=15 duration=1.417049ms48122026/11/11 23:37:43 INFO Completed upload id=2 objects=7 duration=1.572839ms48132026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48142026/11/11 23:37:43 INFO Completed upload id=30 objects=2 duration=786.469µs48152026/11/11 23:37:43 INFO Received complete push request method=POST path=/48162026/11/11 23:37:43 INFO Completed push id=30 objects=4 duration=991.07µs48172026/11/11 23:37:43 INFO Received complete push request method=POST path=/48182026/11/11 23:37:43 INFO Completed push id=28 objects=7 duration=1.22813ms48192026/11/11 23:37:43 INFO Received uploads request method=POST path=/48202026/11/11 23:37:43 INFO Received complete push request method=POST path=/48212026/11/11 23:37:43 INFO Received complete push request method=POST path=/48222026/11/11 23:37:43 INFO Received complete push request method=POST path=/48232026/11/11 23:37:43 INFO Completed push id=16 objects=7 duration=2.081699ms48242026/11/11 23:37:43 INFO Received complete push request method=POST path=/48252026/11/11 23:37:43 INFO Completed push id=30 objects=6 duration=1.26082ms48262026/11/11 23:37:43 INFO Completed push id=31 objects=7 duration=1.08445ms48272026/11/11 23:37:43 INFO Completed push id=31 objects=20 duration=784.929µs48282026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48292026/11/11 23:37:43 INFO Received complete push request method=POST path=/48302026/11/11 23:37:43 INFO Completed upload id=30 objects=6 duration=1.148379ms48312026/11/11 23:37:43 INFO Received uploads request method=POST path=/48322026/11/11 23:37:43 INFO Completed push id=29 objects=24 duration=2.189079ms48332026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48342026/11/11 23:37:43 INFO Completed upload id=32 objects=5 duration=890.88µs48352026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48362026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48372026/11/11 23:37:43 INFO Completed upload id=31 objects=3 duration=1.16016ms48382026/11/11 23:37:43 INFO Completed upload id=32 objects=5 duration=1.13334ms48392026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48402026/11/11 23:37:43 INFO Completed upload id=8 objects=6 duration=1.22184ms48412026/11/11 23:37:43 INFO Received complete push request method=POST path=/48422026/11/11 23:37:43 INFO Completed push id=31 objects=5 duration=1.150319ms48432026/11/11 23:37:43 INFO Received complete push request method=POST path=/48442026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=048452026/11/11 23:37:43 INFO Completed push id=23 objects=6 duration=1.387979ms48462026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period48472026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=048482026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48492026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=048502026/11/11 23:37:43 INFO Completed upload id=33 objects=3 duration=907.859µs48512026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period48522026/11/11 23:37:43 INFO Vacuumed table table=pending_closures48532026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048542026/11/11 23:37:43 INFO Vacuumed table table=pending_objects48552026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads48562026/11/11 23:37:43 INFO Vacuumed table table=pending_closures48572026/11/11 23:37:43 INFO Received complete push request method=POST path=/48582026/11/11 23:37:43 INFO Vacuumed table table=closures48592026/11/11 23:37:43 INFO Vacuumed table table=pending_objects48602026/11/11 23:37:43 INFO Completed push id=24 objects=6 duration=1.392159ms48612026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads48622026/11/11 23:37:43 INFO Vacuumed table table=objects48632026/11/11 23:37:43 INFO Received push request method=POST path=/48642026/11/11 23:37:43 INFO Vacuumed table table=closures48652026/11/11 23:37:43 INFO Vacuumed table table=objects48662026/11/11 23:37:43 INFO Received complete push request method=POST path=/48672026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=048682026/11/11 23:37:43 INFO Completed push id=32 objects=26 duration=1.541259ms48692026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period48702026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048712026/11/11 23:37:43 INFO Vacuumed table table=pending_closures48722026/11/11 23:37:43 INFO Vacuumed table table=pending_objects48732026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads48742026/11/11 23:37:43 INFO Received uploads request method=POST path=/48752026/11/11 23:37:43 INFO Vacuumed table table=closures48762026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48772026/11/11 23:37:43 INFO Vacuumed table table=objects48782026/11/11 23:37:43 INFO Completed upload id=34 objects=24 duration=1.374719ms48792026/11/11 23:37:43 INFO Received complete push request method=POST path=/48802026/11/11 23:37:43 INFO Completed push id=25 objects=7 duration=1.461779ms48812026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48822026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48832026/11/11 23:37:43 INFO Completed upload id=33 objects=2 duration=907.229µs48842026/11/11 23:37:43 INFO Completed upload id=12 objects=9 duration=1.570599ms48852026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=048862026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period48872026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=048882026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048892026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period48902026/11/11 23:37:43 INFO Vacuumed table table=pending_closures48912026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048922026/11/11 23:37:43 INFO Vacuumed table table=pending_objects48932026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads48942026/11/11 23:37:43 INFO Received complete upload request method=POST path=/48952026/11/11 23:37:43 INFO Vacuumed table table=closures48962026/11/11 23:37:43 INFO Completed upload id=34 objects=5 duration=1.719889ms48972026/11/11 23:37:43 INFO Vacuumed table table=pending_closures48982026/11/11 23:37:43 INFO Vacuumed table table=objects48992026/11/11 23:37:43 INFO Vacuumed table table=pending_objects49002026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads49012026/11/11 23:37:43 INFO Vacuumed table table=closures49022026/11/11 23:37:43 INFO Vacuumed table table=objects49032026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49042026/11/11 23:37:43 INFO Completed upload id=35 objects=11 duration=984.96µs49052026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49062026/11/11 23:37:43 INFO Completed upload id=26 objects=24 duration=1.809449ms49072026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49082026/11/11 23:37:43 INFO Completed upload id=36 objects=3 duration=710.15µs49092026/11/11 23:37:43 INFO Received push request method=POST path=/49102026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=049112026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period49122026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049132026/11/11 23:37:43 INFO Vacuumed table table=pending_closures49142026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49152026/11/11 23:37:43 INFO Vacuumed table table=pending_objects49162026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads49172026/11/11 23:37:43 INFO Vacuumed table table=closures49182026/11/11 23:37:43 INFO Completed upload id=23 objects=7 duration=2.988679ms49192026/11/11 23:37:43 INFO Vacuumed table table=objects49202026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49212026/11/11 23:37:43 INFO Completed upload id=27 objects=26 duration=1.497989ms49222026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49232026/11/11 23:37:43 INFO Completed upload id=26 objects=5 duration=1.01488ms49242026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49252026/11/11 23:37:43 INFO Completed upload id=18 objects=22 duration=1.721479ms49262026/11/11 23:37:43 INFO Received complete push request method=POST path=/49272026/11/11 23:37:43 INFO Completed push id=28 objects=6 duration=1.237779ms49282026/11/11 23:37:43 INFO Received complete push request method=POST path=/49292026/11/11 23:37:43 INFO Completed push id=27 objects=3 duration=1.953459ms49302026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49312026/11/11 23:37:43 INFO Completed upload id=21 objects=8 duration=1.122219ms49322026/11/11 23:37:43 INFO Received complete push request method=POST path=/49332026/11/11 23:37:43 INFO Completed push id=29 objects=6 duration=1.134839ms49342026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49352026/11/11 23:37:43 INFO Completed upload id=28 objects=5 duration=1.33891ms49362026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49372026/11/11 23:37:43 INFO Completed upload id=22 objects=7 duration=1.22396ms49382026/11/11 23:37:43 INFO Received complete push request method=POST path=/49392026/11/11 23:37:43 INFO Completed push id=29 objects=17 duration=1.46252ms49402026/11/11 23:37:43 INFO Received complete push request method=POST path=/49412026/11/11 23:37:43 INFO Completed push id=24 objects=5 duration=1.322399ms49422026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49432026/11/11 23:37:43 INFO Completed upload id=30 objects=8 duration=1.433559ms49442026/11/11 23:37:43 INFO Aborted multipart uploads count=0 kept=049452026/11/11 23:37:43 WARN Force mode enabled - objects will be deleted immediately without grace period49462026/11/11 23:37:43 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049472026/11/11 23:37:43 INFO Vacuumed table table=pending_closures49482026/11/11 23:37:43 INFO Vacuumed table table=pending_objects49492026/11/11 23:37:43 INFO Vacuumed table table=multipart_uploads49502026/11/11 23:37:43 INFO Vacuumed table table=closures49512026/11/11 23:37:43 INFO Vacuumed table table=objects49522026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49532026/11/11 23:37:43 INFO Completed upload id=25 objects=8 duration=1.084159ms49542026/11/11 23:37:43 INFO Received complete upload request method=POST path=/49552026/11/11 23:37:43 INFO Completed upload id=31 objects=6 duration=1.12712ms49562026/11/11 23:37:44 INFO Received complete push request method=POST path=/49572026/11/11 23:37:44 INFO Completed push id=26 objects=4 duration=1.326129ms49582026/11/11 23:37:44 INFO Received complete upload request method=POST path=/49592026/11/11 23:37:44 INFO Completed upload id=32 objects=4 duration=1.350619ms49602026/11/11 23:37:44 INFO Received complete push request method=POST path=/49612026/11/11 23:37:44 INFO Completed push id=30 objects=2 duration=1.360989ms49622026/11/11 23:37:44 INFO Received complete upload request method=POST path=/49632026/11/11 23:37:44 INFO Completed upload id=33 objects=10 duration=1.140779ms49642026/11/11 23:37:44 INFO Received complete push request method=POST path=/49652026/11/11 23:37:44 INFO Completed push id=32 objects=7 duration=1.560219ms49662026/11/11 23:37:44 INFO Received complete upload request method=POST path=/49672026/11/11 23:37:44 INFO Completed upload id=34 objects=23 duration=1.38616ms49682026/11/11 23:37:44 INFO Received complete upload request method=POST path=/49692026/11/11 23:37:44 INFO Completed upload id=33 objects=6 duration=1.331799ms49702026/11/11 23:37:44 INFO Received complete push request method=POST path=/49712026/11/11 23:37:44 INFO Completed push id=35 objects=6 duration=1.34742ms49722026/11/11 23:37:44 INFO Received complete upload request method=POST path=/49732026/11/11 23:37:44 INFO Completed upload id=34 objects=3 duration=1.08989ms49742026/11/11 23:37:44 INFO Aborted multipart uploads count=0 kept=049752026/11/11 23:37:44 WARN Force mode enabled - objects will be deleted immediately without grace period49762026/11/11 23:37:44 INFO Aborted multipart uploads count=0 kept=049772026/11/11 23:37:44 WARN Force mode enabled - objects will be deleted immediately without grace period49782026/11/11 23:37:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049792026/11/11 23:37:44 INFO Vacuumed table table=pending_closures49802026/11/11 23:37:44 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049812026/11/11 23:37:44 INFO Vacuumed table table=pending_objects49822026/11/11 23:37:44 INFO Vacuumed table table=multipart_uploads49832026/11/11 23:37:44 INFO Vacuumed table table=closures49842026/11/11 23:37:44 INFO Vacuumed table table=pending_closures49852026/11/11 23:37:44 INFO Vacuumed table table=objects49862026/11/11 23:37:44 INFO Vacuumed table table=pending_objects49872026/11/11 23:37:44 INFO Vacuumed table table=multipart_uploads49882026/11/11 23:37:44 INFO Vacuumed table table=closures49892026/11/11 23:37:44 INFO Vacuumed table table=objects4990--- PASS: FuzzPushGC (0.00s)4991 --- PASS: FuzzPushGC/seed#0 (2.89s)4992 --- PASS: FuzzPushGC/seed#7 (3.00s)4993 --- PASS: FuzzPushGC/seed#3 (3.02s)4994 --- PASS: FuzzPushGC/seed#8 (3.03s)4995 --- PASS: FuzzPushGC/seed#5 (3.04s)4996 --- PASS: FuzzPushGC/seed#4 (3.06s)4997 --- PASS: FuzzPushGC/seed#13 (3.07s)4998 --- PASS: FuzzPushGC/seed#14 (3.08s)4999 --- PASS: FuzzPushGC/seed#2 (3.20s)5000 --- PASS: FuzzPushGC/seed#15 (3.20s)5001 --- PASS: FuzzPushGC/seed#6 (3.22s)5002 --- PASS: FuzzPushGC/seed#9 (3.26s)5003 --- PASS: FuzzPushGC/seed#11 (3.32s)5004 --- PASS: FuzzPushGC/seed#1 (3.44s)5005 --- PASS: FuzzPushGC/seed#10 (3.61s)5006 --- PASS: FuzzPushGC/seed#12 (3.61s)5007PASS5008{"timestamp":"2026-11-11T23:37:44.181760044Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:34128","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(395)"}5009{"timestamp":"2026-11-11T23:37:44.181876574Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:34136","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(392)"}5010{"timestamp":"2026-11-11T23:37:44.181973434Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:34154","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(397)"}50112026-11-11 23:37:44.634 UTC [129] LOG: received smart shutdown request50122026-11-11 23:37:44.638 UTC [129] LOG: background worker "logical replication launcher" (PID 139) exited with exit code 150132026-11-11 23:37:45.463 UTC [134] LOG: checkpoint complete: wrote 3992 buffers (24.4%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=4.137 s, sync=0.680 s, total=4.876 s; sync files=42363, longest=0.001 s, average=0.001 s; distance=534215 kB, estimate=534215 kB; lsn=0/26D4A258, redo lsn=0/221179E050142026-11-11 23:37:45.466 UTC [134] LOG: shutting down50152026-11-11 23:37:45.467 UTC [134] LOG: checkpoint starting: shutdown immediate50162026-11-11 23:37:46.472 UTC [134] LOG: checkpoint complete: wrote 9421 buffers (57.5%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.158 s, sync=0.832 s, total=1.006 s; sync files=2823, longest=0.002 s, average=0.001 s; distance=78026 kB, estimate=488596 kB; lsn=0/26D4A2D0, redo lsn=0/26D4A2D050172026-11-11 23:37:46.567 UTC [129] LOG: database system is shut down5018Running OIDC tests...5019=== RUN TestAudienceForIssuer5020=== PAUSE TestAudienceForIssuer5021=== RUN TestGlobMatch5022=== PAUSE TestGlobMatch5023=== RUN TestValidateToken_ValidToken5024=== PAUSE TestValidateToken_ValidToken5025=== RUN TestValidateToken_WrongAudience5026=== PAUSE TestValidateToken_WrongAudience5027=== RUN TestValidateToken_Expired5028=== PAUSE TestValidateToken_Expired5029=== RUN TestValidateToken_BoundClaimsMismatch5030=== PAUSE TestValidateToken_BoundClaimsMismatch5031=== RUN TestValidateToken_BoundSubjectMismatch5032=== PAUSE TestValidateToken_BoundSubjectMismatch5033=== RUN TestValidateToken_MultipleProviders5034=== PAUSE TestValidateToken_MultipleProviders5035=== RUN TestValidateToken_NoMatchingProvider5036=== PAUSE TestValidateToken_NoMatchingProvider5037=== RUN TestValidateToken_KubernetesServiceAccount5038=== PAUSE TestValidateToken_KubernetesServiceAccount5039=== RUN TestNewValidator_KubernetesRequiresCA5040=== PAUSE TestNewValidator_KubernetesRequiresCA5041=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5042=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5043=== RUN TestPins_ReservedForMatchingRule5044=== PAUSE TestPins_ReservedForMatchingRule5045=== RUN TestPins_TopLevelShorthand5046=== PAUSE TestPins_TopLevelShorthand5047=== RUN TestPins_ConfigValidation5048=== PAUSE TestPins_ConfigValidation5049=== RUN TestScopes_LegacyProviderDefaultsToWrite5050=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5051=== RUN TestScopes_Rules5052=== PAUSE TestScopes_Rules5053=== RUN TestScopes_ConfigValidation5054=== PAUSE TestScopes_ConfigValidation5055=== CONT TestAudienceForIssuer5056=== CONT TestValidateToken_KubernetesServiceAccount5057=== CONT TestValidateToken_MultipleProviders5058--- PASS: TestAudienceForIssuer (0.00s)5059=== CONT TestValidateToken_Expired5060=== CONT TestValidateToken_WrongAudience5061=== CONT TestValidateToken_ValidToken5062=== CONT TestGlobMatch5063=== CONT TestValidateToken_BoundSubjectMismatch5064=== CONT TestPins_ConfigValidation5065=== CONT TestScopes_ConfigValidation5066=== CONT TestScopes_Rules5067=== CONT TestScopes_LegacyProviderDefaultsToWrite5068=== CONT TestValidateToken_NoMatchingProvider5069=== CONT TestPins_ReservedForMatchingRule5070--- PASS: TestPins_ConfigValidation (0.00s)5071--- PASS: TestScopes_ConfigValidation (0.00s)5072=== CONT TestPins_TopLevelShorthand5073=== CONT TestValidateToken_BoundClaimsMismatch5074=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5075=== CONT TestNewValidator_KubernetesRequiresCA5076=== RUN TestGlobMatch/foo_foo5077=== PAUSE TestGlobMatch/foo_foo5078=== RUN TestGlobMatch/foo_bar5079=== PAUSE TestGlobMatch/foo_bar5080=== RUN TestGlobMatch/*_5081=== PAUSE TestGlobMatch/*_5082=== RUN TestGlobMatch/*_anything5083=== PAUSE TestGlobMatch/*_anything5084=== RUN TestGlobMatch/foo*_foo5085=== PAUSE TestGlobMatch/foo*_foo5086=== RUN TestGlobMatch/foo*_foobar5087=== PAUSE TestGlobMatch/foo*_foobar5088=== RUN TestGlobMatch/foo*_bar5089=== PAUSE TestGlobMatch/foo*_bar5090=== RUN TestGlobMatch/*bar_bar5091=== PAUSE TestGlobMatch/*bar_bar5092=== RUN TestGlobMatch/*bar_foobar5093=== PAUSE TestGlobMatch/*bar_foobar5094=== RUN TestGlobMatch/*bar_foo5095=== PAUSE TestGlobMatch/*bar_foo5096=== RUN TestGlobMatch/foo*bar_foobar5097=== PAUSE TestGlobMatch/foo*bar_foobar5098=== RUN TestGlobMatch/foo*bar_foo123bar5099=== PAUSE TestGlobMatch/foo*bar_foo123bar5100=== RUN TestGlobMatch/foo*bar_foobarbaz5101=== PAUSE TestGlobMatch/foo*bar_foobarbaz5102=== RUN TestGlobMatch/*/*_foo/bar5103=== PAUSE TestGlobMatch/*/*_foo/bar5104=== RUN TestGlobMatch/*/*_foo5105=== PAUSE TestGlobMatch/*/*_foo5106=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5107=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5108=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05109=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05110=== RUN TestGlobMatch/refs/*/main_refs/heads/main5111=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5112=== RUN TestGlobMatch/fo?_foo5113=== PAUSE TestGlobMatch/fo?_foo5114=== RUN TestGlobMatch/fo?_fo5115=== PAUSE TestGlobMatch/fo?_fo5116=== RUN TestGlobMatch/fo?_fooo5117=== PAUSE TestGlobMatch/fo?_fooo5118=== RUN TestGlobMatch/?oo_foo5119=== PAUSE TestGlobMatch/?oo_foo5120=== RUN TestGlobMatch/?oo_boo5121=== PAUSE TestGlobMatch/?oo_boo5122=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5123=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5124=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5125=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5126=== CONT TestGlobMatch/foo_foo5127=== CONT TestGlobMatch/foo*bar_foobarbaz5128=== CONT TestGlobMatch/foo*bar_foo123bar5129=== CONT TestGlobMatch/foo*bar_foobar5130=== CONT TestGlobMatch/foo*_foo5131=== CONT TestGlobMatch/fo?_foo5132=== CONT TestGlobMatch/foo_bar5133=== CONT TestGlobMatch/*/*_foo/bar5134=== CONT TestGlobMatch/refs/*/main_refs/heads/main5135=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05136=== CONT TestGlobMatch/*bar_foo5137=== CONT TestGlobMatch/*bar_foobar5138=== CONT TestGlobMatch/*/*_foo5139=== CONT TestGlobMatch/*bar_bar5140=== CONT TestGlobMatch/foo*_bar5141=== CONT TestGlobMatch/foo*_foobar5142=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5143=== CONT TestGlobMatch/?oo_boo5144=== CONT TestGlobMatch/?oo_foo5145=== CONT TestGlobMatch/fo?_fooo5146=== CONT TestGlobMatch/fo?_fo5147=== CONT TestGlobMatch/*_anything5148=== CONT TestGlobMatch/*_5149=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5150=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5151--- PASS: TestGlobMatch (0.01s)5152 --- PASS: TestGlobMatch/foo_foo (0.00s)5153 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5154 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5155 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5156 --- PASS: TestGlobMatch/foo*_foo (0.00s)5157 --- PASS: TestGlobMatch/fo?_foo (0.00s)5158 --- PASS: TestGlobMatch/foo_bar (0.00s)5159 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5160 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5161 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5162 --- PASS: TestGlobMatch/*bar_foo (0.00s)5163 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5164 --- PASS: TestGlobMatch/*/*_foo (0.00s)5165 --- PASS: TestGlobMatch/*bar_bar (0.00s)5166 --- PASS: TestGlobMatch/foo*_bar (0.00s)5167 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5168 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5169 --- PASS: TestGlobMatch/?oo_boo (0.00s)5170 --- PASS: TestGlobMatch/?oo_foo (0.00s)5171 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5172 --- PASS: TestGlobMatch/fo?_fo (0.00s)5173 --- PASS: TestGlobMatch/*_anything (0.00s)5174 --- PASS: TestGlobMatch/*_ (0.00s)5175 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5176 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)51772026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38329/oidc51782026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37511/oidc5179--- PASS: TestValidateToken_Expired (0.03s)5180--- PASS: TestScopes_Rules (0.02s)51812026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45469/oidc5182--- PASS: TestValidateToken_BoundSubjectMismatch (0.04s)51832026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43329/oidc5184--- PASS: TestValidateToken_BoundClaimsMismatch (0.04s)51852026/11/11 23:37:49 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235186--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.06s)51872026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37647/oidc51882026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44821/oidc5189--- PASS: TestValidateToken_WrongAudience (0.07s)5190--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.07s)51912026/11/11 23:37:49 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:43147/oidc51922026/11/11 23:37:49 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:45025/oidc5193--- PASS: TestValidateToken_MultipleProviders (0.08s)51942026/11/11 23:37:49 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:330095195--- PASS: TestValidateToken_KubernetesServiceAccount (0.10s)51962026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43695/oidc5197--- PASS: TestPins_TopLevelShorthand (0.10s)51982026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40891/oidc5199--- PASS: TestValidateToken_ValidToken (0.11s)52002026/11/11 23:37:49 http: TLS handshake error from 127.0.0.1:53770: remote error: tls: bad certificate5201--- PASS: TestNewValidator_KubernetesRequiresCA (0.11s)52022026/11/11 23:37:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45043/oidc5203--- PASS: TestPins_ReservedForMatchingRule (0.16s)52042026/11/11 23:37:49 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:46389/oidc5205--- PASS: TestValidateToken_NoMatchingProvider (0.29s)5206PASS5207Running hook tests...5208=== RUN TestSendPathsEmpty5209=== PAUSE TestSendPathsEmpty5210=== RUN TestQueueEnqueueAndFetch5211=== PAUSE TestQueueEnqueueAndFetch5212=== RUN TestQueueDeduplication5213=== PAUSE TestQueueDeduplication5214=== RUN TestQueueRemove5215=== PAUSE TestQueueRemove5216=== RUN TestQueueFetchBatchLimit5217=== PAUSE TestQueueFetchBatchLimit5218=== RUN TestQueueRetryMovesToBack5219=== PAUSE TestQueueRetryMovesToBack5220=== RUN TestQueueFetchRemoveLifecycle5221=== PAUSE TestQueueFetchRemoveLifecycle5222=== RUN TestQueueConcurrentWriters5223=== PAUSE TestQueueConcurrentWriters5224=== RUN TestQueueRemoveLargeClosure5225=== PAUSE TestQueueRemoveLargeClosure5226=== RUN TestServerClientIntegration5227--- PASS: TestServerClientIntegration (0.00s)5228=== RUN TestServerQueueError52292026/11/11 23:37:49 ERROR Failed to queue paths error="permission denied" count=15230--- PASS: TestServerQueueError (0.00s)5231=== RUN TestGetListenerSocketActivation5232 server_test.go:220: === RUN TestGetListenerSocketActivation5233 --- PASS: TestGetListenerSocketActivation (0.00s)5234 PASS5235 5236--- PASS: TestGetListenerSocketActivation (0.01s)5237=== RUN TestServerStalledClientDoesNotBlockShutdown5238=== PAUSE TestServerStalledClientDoesNotBlockShutdown5239=== RUN TestServerBacksOffOnAcceptErrors5240=== PAUSE TestServerBacksOffOnAcceptErrors5241=== RUN TestDrainIsolatesPoisonPath5242=== PAUSE TestDrainIsolatesPoisonPath5243=== RUN TestRunNotBlockedByPoisonHead5244=== PAUSE TestRunNotBlockedByPoisonHead5245=== RUN TestDrainGivesUpWhenServerDown5246=== PAUSE TestDrainGivesUpWhenServerDown5247=== RUN TestFailedPathPrunedByLaterClosure5248=== PAUSE TestFailedPathPrunedByLaterClosure5249=== RUN TestWorkerUploadsAndRemoves5250=== PAUSE TestWorkerUploadsAndRemoves5251=== RUN TestWorkerSkipsGCdPaths5252=== PAUSE TestWorkerSkipsGCdPaths5253=== RUN TestWorkerPrunesClosureDeps5254=== PAUSE TestWorkerPrunesClosureDeps5255=== RUN TestDrainTimeout5256=== PAUSE TestDrainTimeout5257=== RUN TestWorkerRemoveFailureIsNotProgress5258=== PAUSE TestWorkerRemoveFailureIsNotProgress5259=== RUN TestWorkerKeepsPathItCannotStat5260=== PAUSE TestWorkerKeepsPathItCannotStat5261=== CONT TestSendPathsEmpty5262=== CONT TestWorkerRemoveFailureIsNotProgress5263--- PASS: TestSendPathsEmpty (0.00s)5264=== CONT TestWorkerKeepsPathItCannotStat5265=== CONT TestDrainIsolatesPoisonPath5266=== CONT TestQueueRetryMovesToBack5267=== CONT TestQueueFetchBatchLimit5268=== CONT TestQueueRemove5269=== CONT TestQueueDeduplication5270=== CONT TestQueueEnqueueAndFetch5271=== CONT TestWorkerUploadsAndRemoves5272=== CONT TestDrainTimeout5273=== CONT TestWorkerPrunesClosureDeps5274=== CONT TestWorkerSkipsGCdPaths5275=== CONT TestQueueFetchRemoveLifecycle5276=== CONT TestServerStalledClientDoesNotBlockShutdown5277=== CONT TestQueueRemoveLargeClosure5278=== CONT TestServerBacksOffOnAcceptErrors52792000/01/01 00:00:00 ERROR Accept failed error="too many open files"52802000/01/01 00:00:00 ERROR Accept failed error="too many open files"52812000/01/01 00:00:00 ERROR Accept failed error="too many open files"52822000/01/01 00:00:00 ERROR Accept failed error="too many open files"52832000/01/01 00:00:00 ERROR Accept failed error="too many open files"52842000/01/01 00:00:00 ERROR Accept failed error="too many open files"52852000/01/01 00:00:00 ERROR Accept failed error="too many open files"52862000/01/01 00:00:00 ERROR Accept failed error="too many open files"5287=== CONT TestQueueConcurrentWriters5288--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5289=== CONT TestRunNotBlockedByPoisonHead5290=== CONT TestFailedPathPrunedByLaterClosure5291=== CONT TestDrainGivesUpWhenServerDown52922026/11/11 23:37:49 INFO Upload queue status pending=252932026/11/11 23:37:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths923227052/002/nonexistent52942026/11/11 23:37:49 INFO Uploading batch count=452952026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=452962026/11/11 23:37:49 INFO Uploading batch count=152972026/11/11 23:37:49 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat427038353/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat427038353/002/locked/aaa: permission denied"52982026/11/11 23:37:49 INFO Upload queue status pending=252992026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=153002026/11/11 23:37:49 INFO Upload queue status pending=353012026/11/11 23:37:49 INFO Uploading batch count=153022026/11/11 23:37:49 INFO Upload queue status pending=253032000/01/01 00:00:00 INFO Uploading batch count=253042026/11/11 23:37:49 INFO Uploading batch count=253052026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=253062026/11/11 23:37:49 INFO Uploading batch count=25307--- PASS: TestQueueEnqueueAndFetch (0.02s)53082026/11/11 23:37:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath2851699683/002/bbb53092000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253102026/11/11 23:37:49 INFO Uploading batch count=153112026/11/11 23:37:49 INFO Uploading batch count=15312--- PASS: TestQueueDeduplication (0.02s)53132026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=15314--- PASS: TestQueueFetchBatchLimit (0.02s)53152026/11/11 23:37:49 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat427038353/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat427038353/002/locked/aaa: permission denied"5316--- PASS: TestQueueRetryMovesToBack (0.02s)53172026/11/11 23:37:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown444319323/002/a53182000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453192026/11/11 23:37:49 INFO Uploading batch count=15320--- PASS: TestQueueFetchRemoveLifecycle (0.02s)53212026/11/11 23:37:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown444319323/002/b5322--- PASS: TestQueueRemove (0.02s)53232026/11/11 23:37:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1614008826/002/nonexistent53242026/11/11 23:37:49 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat427038353/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat427038353/002/locked/aaa: permission denied"53252026/11/11 23:37:49 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153262026/11/11 23:37:49 INFO Uploading batch count=253272026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=253282026/11/11 23:37:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown444319323/002/c53292026/11/11 23:37:49 INFO Uploading batch count=153302026/11/11 23:37:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1614008826/002/nonexistent53312026/11/11 23:37:49 ERROR Drain finished with paths left in queue remaining=153322026/11/11 23:37:49 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153332026/11/11 23:37:49 INFO Uploading batch count=153342026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=153352026/11/11 23:37:49 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1614008826/002/nonexistent53362026/11/11 23:37:49 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153372026/11/11 23:37:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown444319323/002/d53382026/11/11 23:37:49 ERROR Drain finished with paths left in queue remaining=15339--- PASS: TestDrainTimeout (0.02s)53402026/11/11 23:37:49 INFO Uploading batch count=253412026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=253422026/11/11 23:37:49 INFO Uploading batch count=153432026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=15344--- PASS: TestWorkerRemoveFailureIsNotProgress (0.03s)53452026/11/11 23:37:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown444319323/002/e5346--- PASS: TestWorkerKeepsPathItCannotStat (0.03s)53472026/11/11 23:37:49 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown444319323/002/f5348--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)53492026/11/11 23:37:49 INFO Uploading batch count=153502026/11/11 23:37:49 ERROR Upload failed error="upload failed" count=153512026/11/11 23:37:49 ERROR Drain finished with paths left in queue remaining=1053522026/11/11 23:37:49 ERROR Drain finished with paths left in queue remaining=15353--- PASS: TestDrainIsolatesPoisonPath (0.03s)5354--- PASS: TestDrainGivesUpWhenServerDown (0.03s)5355--- PASS: TestWorkerSkipsGCdPaths (0.04s)5356--- PASS: TestWorkerUploadsAndRemoves (0.04s)5357--- PASS: TestWorkerPrunesClosureDeps (0.04s)5358--- PASS: TestQueueRemoveLargeClosure (0.07s)53592026/11/11 23:37:49 ERROR Failed to decode request error="read unix /build/hook2219336594/test.sock->@: i/o timeout"53602026/11/11 23:37:49 ERROR Failed to write response error="write unix /build/hook2219336594/test.sock->@: i/o timeout"5361--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)5362--- PASS: TestQueueConcurrentWriters (0.22s)53632026/11/11 23:37:50 INFO Uploading batch count=153642026/11/11 23:37:50 INFO Uploading batch count=153652026/11/11 23:37:50 INFO Uploading batch count=153662026/11/11 23:37:50 ERROR Upload failed error="upload failed" count=153672026/11/11 23:37:50 INFO Uploading batch count=153682026/11/11 23:37:50 ERROR Upload failed error="upload failed" count=153692026/11/11 23:37:50 INFO Uploading batch count=153702026/11/11 23:37:50 ERROR Upload failed error="upload failed" count=153712026/11/11 23:37:50 INFO Uploading batch count=153722026/11/11 23:37:50 ERROR Upload failed error="upload failed" count=153732026/11/11 23:37:50 ERROR Drain finished with paths left in queue remaining=15374--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5375PASS5376Running signing tests...5377=== RUN TestExternalSigner5378=== PAUSE TestExternalSigner5379=== RUN TestExternalSignerWrongPublicKey5380=== PAUSE TestExternalSignerWrongPublicKey5381=== RUN TestExternalSignerErrors5382=== PAUSE TestExternalSignerErrors5383=== RUN TestExternalSignerRestartsAfterCrash5384=== PAUSE TestExternalSignerRestartsAfterCrash5385=== RUN TestExternalSignerProgramErrorKeepsProcess5386=== PAUSE TestExternalSignerProgramErrorKeepsProcess5387=== RUN TestExternalSignerClose5388=== PAUSE TestExternalSignerClose5389=== RUN TestExternalSignerRejectsInvalidPublicKey5390=== PAUSE TestExternalSignerRejectsInvalidPublicKey5391=== RUN TestExternalSignerCancellation5392=== PAUSE TestExternalSignerCancellation5393=== RUN TestGenerateFingerprint5394=== PAUSE TestGenerateFingerprint5395=== RUN TestParseSigningKey5396=== PAUSE TestParseSigningKey5397=== RUN TestSign5398=== PAUSE TestSign5399=== RUN TestSignBatch5400=== PAUSE TestSignBatch5401=== RUN TestNilKeyPublicKey5402=== PAUSE TestNilKeyPublicKey5403=== RUN TestSignErrors5404=== PAUSE TestSignErrors5405=== CONT TestExternalSigner5406=== CONT TestExternalSignerProgramErrorKeepsProcess5407=== CONT TestExternalSignerCancellation5408=== CONT TestSignErrors5409=== CONT TestNilKeyPublicKey5410=== CONT TestSignBatch5411=== CONT TestSign5412=== CONT TestParseSigningKey5413=== RUN TestParseSigningKey/valid_32-byte_key5414=== CONT TestGenerateFingerprint5415=== CONT TestExternalSignerErrors5416=== RUN TestExternalSignerErrors/wrong_shape5417=== CONT TestExternalSignerRestartsAfterCrash5418=== CONT TestExternalSignerWrongPublicKey5419=== CONT TestExternalSignerClose5420=== CONT TestExternalSignerRejectsInvalidPublicKey5421--- PASS: TestNilKeyPublicKey (0.00s)5422=== PAUSE TestParseSigningKey/valid_32-byte_key5423=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5424=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5425=== RUN TestParseSigningKey/no_colon5426=== PAUSE TestParseSigningKey/no_colon5427=== RUN TestParseSigningKey/empty_name5428=== PAUSE TestParseSigningKey/empty_name5429=== RUN TestParseSigningKey/invalid_base645430=== PAUSE TestParseSigningKey/invalid_base645431=== RUN TestParseSigningKey/wrong_length5432=== PAUSE TestParseSigningKey/wrong_length5433=== CONT TestParseSigningKey/valid_32-byte_key5434=== CONT TestParseSigningKey/empty_name5435=== CONT TestParseSigningKey/no_colon5436=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5437=== RUN TestExternalSignerRejectsInvalidPublicKey/#005438=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005439=== RUN TestSignErrors/nil_key5440=== CONT TestParseSigningKey/wrong_length5441=== PAUSE TestSignErrors/nil_key5442=== PAUSE TestExternalSignerErrors/wrong_shape5443=== RUN TestGenerateFingerprint/basic_with_references5444=== PAUSE TestGenerateFingerprint/basic_with_references5445=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5446=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5447=== CONT TestParseSigningKey/invalid_base645448--- PASS: TestSignBatch (0.00s)5449--- PASS: TestSign (0.00s)5450=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5451=== RUN TestExternalSignerErrors/null5452=== PAUSE TestExternalSignerErrors/null5453=== RUN TestExternalSignerErrors/missing_signatures5454=== PAUSE TestExternalSignerErrors/missing_signatures5455=== RUN TestExternalSignerErrors/null_signatures5456=== PAUSE TestExternalSignerErrors/null_signatures5457=== RUN TestExternalSignerErrors/wrong_signature_length5458=== RUN TestGenerateFingerprint/no_references5459=== PAUSE TestGenerateFingerprint/no_references5460=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5461=== RUN TestSignErrors/nil_narinfo5462=== PAUSE TestSignErrors/nil_narinfo5463--- PASS: TestParseSigningKey (0.00s)5464 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5465 --- PASS: TestParseSigningKey/empty_name (0.00s)5466 --- PASS: TestParseSigningKey/no_colon (0.00s)5467 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5468 --- PASS: TestParseSigningKey/wrong_length (0.00s)5469 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5470=== PAUSE TestExternalSignerErrors/wrong_signature_length5471--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5472=== RUN TestExternalSignerErrors/exit_on_request5473=== PAUSE TestExternalSignerErrors/exit_on_request5474=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5475=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5476=== RUN TestSignErrors/invalid_narinfo5477=== PAUSE TestSignErrors/invalid_narinfo5478=== RUN TestSignErrors/canceled5479=== PAUSE TestSignErrors/canceled5480=== CONT TestSignErrors/nil_key5481=== CONT TestSignErrors/invalid_narinfo5482=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5483=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5484=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5485=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5486=== CONT TestExternalSignerRejectsInvalidPublicKey/#005487=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5488=== CONT TestSignErrors/canceled5489=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5490=== RUN TestExternalSignerErrors/invalid_JSON5491=== PAUSE TestExternalSignerErrors/invalid_JSON5492=== RUN TestExternalSignerErrors/missing_signature_name5493=== PAUSE TestExternalSignerErrors/missing_signature_name5494=== RUN TestExternalSignerErrors/wrong_signature_name5495=== PAUSE TestExternalSignerErrors/wrong_signature_name5496=== RUN TestExternalSignerErrors/invalid_signature_base645497=== PAUSE TestExternalSignerErrors/invalid_signature_base645498=== RUN TestExternalSignerErrors/program_error5499--- PASS: TestExternalSignerWrongPublicKey (0.00s)5500=== PAUSE TestExternalSignerErrors/program_error5501=== CONT TestSignErrors/nil_narinfo5502--- PASS: TestSignErrors (0.00s)5503 --- PASS: TestSignErrors/nil_key (0.00s)5504 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5505 --- PASS: TestSignErrors/canceled (0.00s)5506 --- PASS: TestSignErrors/nil_narinfo (0.00s)5507=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5508=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5509--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5510 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5511 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5512 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5513 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5514 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5515=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5516=== RUN TestExternalSignerErrors/exit_at_startup5517=== PAUSE TestExternalSignerErrors/exit_at_startup5518=== RUN TestExternalSignerErrors/wrong_id5519=== PAUSE TestExternalSignerErrors/wrong_id5520=== RUN TestExternalSignerErrors/wrong_signatures_shape5521--- PASS: TestExternalSignerClose (0.00s)5522=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5523=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5524=== RUN TestExternalSignerErrors/too_few_signatures5525=== PAUSE TestExternalSignerErrors/too_few_signatures5526=== RUN TestExternalSignerErrors/unrequested_key5527=== PAUSE TestExternalSignerErrors/unrequested_key5528=== CONT TestExternalSignerErrors/wrong_shape5529=== CONT TestExternalSignerErrors/too_few_signatures5530=== RUN TestGenerateFingerprint/invalid_nar_hash_length5531=== CONT TestExternalSignerErrors/missing_signature_name5532=== CONT TestExternalSignerErrors/wrong_signature_name5533=== CONT TestExternalSignerErrors/unrequested_key5534=== CONT TestExternalSignerErrors/wrong_signature_length5535=== CONT TestExternalSignerErrors/invalid_JSON5536=== CONT TestExternalSignerErrors/exit_on_request5537=== CONT TestExternalSignerErrors/exit_at_startup5538=== CONT TestExternalSignerErrors/missing_signatures5539=== CONT TestExternalSignerErrors/wrong_signatures_shape5540=== CONT TestExternalSignerErrors/null_signatures5541=== CONT TestExternalSignerErrors/wrong_id5542=== CONT TestExternalSignerErrors/null5543=== CONT TestExternalSignerErrors/invalid_signature_base645544=== CONT TestExternalSignerErrors/program_error5545=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5546--- PASS: TestExternalSigner (0.01s)5547=== RUN TestGenerateFingerprint/invalid_store_path_prefix5548=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5549=== RUN TestGenerateFingerprint/invalid_reference_prefix5550=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5551=== CONT TestGenerateFingerprint/basic_with_references5552=== CONT TestGenerateFingerprint/invalid_nar_hash_length5553=== CONT TestGenerateFingerprint/invalid_reference_prefix5554=== CONT TestGenerateFingerprint/invalid_store_path_prefix5555=== CONT TestGenerateFingerprint/no_references5556=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5557=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5558--- PASS: TestGenerateFingerprint (0.01s)5559 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5560 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5561 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5562 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5563 --- PASS: TestGenerateFingerprint/no_references (0.00s)5564 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5565 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5566--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5567--- PASS: TestExternalSignerErrors (0.00s)5568 --- PASS: TestExternalSignerErrors/too_few_signatures (0.00s)5569 --- PASS: TestExternalSignerErrors/exit_at_startup (0.00s)5570 --- PASS: TestExternalSignerErrors/unrequested_key (0.00s)5571 --- PASS: TestExternalSignerErrors/wrong_shape (0.00s)5572 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.00s)5573 --- PASS: TestExternalSignerErrors/missing_signature_name (0.00s)5574 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.00s)5575 --- PASS: TestExternalSignerErrors/invalid_JSON (0.00s)5576 --- PASS: TestExternalSignerErrors/exit_on_request (0.00s)5577 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5578 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5579 --- PASS: TestExternalSignerErrors/null (0.00s)5580 --- PASS: TestExternalSignerErrors/program_error (0.00s)5581 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5582 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5583 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5584--- PASS: TestExternalSignerCancellation (0.02s)5585PASS5586Running OpenBao signer tests...5587=== RUN TestSign5588=== PAUSE TestSign5589=== RUN TestSignErrors5590=== PAUSE TestSignErrors5591=== RUN TestServe5592=== PAUSE TestServe5593=== RUN TestServeRejectsGarbage5594=== PAUSE TestServeRejectsGarbage5595=== CONT TestSign5596=== CONT TestServeRejectsGarbage5597=== CONT TestServe5598=== CONT TestSignErrors5599--- PASS: TestServeRejectsGarbage (0.00s)5600=== RUN TestSignErrors/no_prefix5601=== PAUSE TestSignErrors/no_prefix5602=== RUN TestSignErrors/invalid_base645603=== PAUSE TestSignErrors/invalid_base645604=== RUN TestSignErrors/http_error5605=== PAUSE TestSignErrors/http_error5606=== RUN TestSignErrors/invalid_json5607=== PAUSE TestSignErrors/invalid_json5608=== CONT TestSignErrors/no_prefix5609--- PASS: TestServe (0.00s)5610=== CONT TestSignErrors/http_error5611=== CONT TestSignErrors/invalid_json5612=== CONT TestSignErrors/invalid_base645613--- PASS: TestSign (0.00s)5614--- PASS: TestSignErrors (0.00s)5615 --- PASS: TestSignErrors/invalid_json (0.00s)5616 --- PASS: TestSignErrors/http_error (0.00s)5617 --- PASS: TestSignErrors/invalid_base64 (0.00s)5618 --- PASS: TestSignErrors/no_prefix (0.00s)5619PASS