niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #493
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestDoServerRequestAttachesToken116=== CONT TestDoWithRetry_BodyReplayedViaGetBody117=== CONT TestEncodeNixBase32118=== RUN TestEncodeNixBase32/test_string_hash119=== PAUSE TestEncodeNixBase32/test_string_hash120=== RUN TestEncodeNixBase32/empty_input121=== PAUSE TestEncodeNixBase32/empty_input122=== CONT TestParsePathInfoJSONMultiplePaths123=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths124=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths125=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths126=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths127=== CONT TestParsePathInfoJSON128=== RUN TestParsePathInfoJSON/Nix_format129=== PAUSE TestParsePathInfoJSON/Nix_format130=== RUN TestParsePathInfoJSON/Lix_format131=== PAUSE TestParsePathInfoJSON/Lix_format132=== RUN TestParsePathInfoJSON/empty_input133=== PAUSE TestParsePathInfoJSON/empty_input134=== RUN TestParsePathInfoJSON/whitespace_only135=== PAUSE TestParsePathInfoJSON/whitespace_only136=== RUN TestParsePathInfoJSON/invalid_JSON137=== PAUSE TestParsePathInfoJSON/invalid_JSON138=== CONT TestClientSignaturesByStorePath139--- PASS: TestClientSignaturesByStorePath (0.00s)140=== CONT TestStreamPushReportsSkippedPaths141=== CONT TestPathInfoHashCompatibility142=== CONT TestGetStorePathHash143=== CONT TestConvertHashToNix32144=== RUN TestGetStorePathHash/valid_store_path145=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)146=== PAUSE TestGetStorePathHash/valid_store_path147=== RUN TestConvertHashToNix32/SRI_format_to_Nix32148=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32149=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer150--- PASS: TestStreamPushReportsSkippedPaths (0.00s)151=== CONT TestStreamPushReportsEveryPath152=== CONT TestResolveStorePath153=== CONT TestRegisterUploadedObject_OutlivesPush154=== CONT TestUploadMultipart_PartsInParallel155=== CONT TestDumpPathWriterError156=== CONT TestRateLimiterFeedback157=== CONT TestDumpPathSingleFile158=== RUN TestRateLimiterFeedback/429_enables_limiter159=== CONT TestPathInfoCACompatibility160=== CONT TestUploadMultipart_FailedProducerIsNotCompleted161=== CONT TestUploadMultipart_ReturnsAfterDroppedBody162=== CONT TestDumpPathMatchesNix163=== CONT TestStreamPushIsolatesFailures164=== CONT TestStreamPushStopsWhenCanceled165=== CONT TestSetClientTLS166=== CONT TestShellSplitErrors167=== CONT TestStreamPushBatchesUnderLoad168=== CONT TestEncodeNixBase32WithRealHash169=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)170=== RUN TestGetStorePathHash/basename_without_hyphen_should_error171=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess172=== RUN TestConvertHashToNix32/already_Nix32_format173--- PASS: TestStreamPushReportsEveryPath (0.00s)174=== CONT TestStreamPushReportsSignatures175=== CONT TestStreamPushRequestLine176=== RUN TestPathInfoCACompatibility/null_ca_field177=== PAUSE TestRateLimiterFeedback/429_enables_limiter178=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part179=== CONT TestStreamPushGivesUpOnDeadServer180--- PASS: TestResolveStorePath (0.00s)181--- PASS: TestStreamPushStopsWhenCanceled (0.00s)182=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon1832026/11/11 23:58:09 ERROR Upload failed error="bad path" count=3184--- PASS: TestEncodeNixBase32WithRealHash (0.00s)1852026/11/11 23:58:09 WARN Rate limiter enabled after throttle name=server-test rate=5186=== CONT TestDoWithRetry_FinalResponseBodyReadable187--- PASS: TestRegisterUploadedObject_OutlivesPush (0.03s)1882026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:34897189--- PASS: TestStreamPushIsolatesFailures (0.03s)190=== CONT TestShellSplit191--- PASS: TestShellSplit (0.00s)192=== CONT TestScriptTokenScriptFails193=== RUN TestRateLimiterFeedback/503_enables_limiter1942026/11/11 23:58:09 ERROR Upload failed error="connection refused" count=201952026/11/11 23:58:09 ERROR Server seems unavailable, giving up on batch untried=17196=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error1972026/11/11 23:58:09 WARN Rate limiter backed off name=server-test rate=5198=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error1992026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:34897200=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error201=== CONT TestScriptTokenBadJSON2022026/11/11 23:58:09 ERROR Upload failed error=boom count=12032026/11/11 23:58:09 ERROR Upload failed error=boom count=1204=== PAUSE TestConvertHashToNix32/already_Nix32_format205=== RUN TestConvertHashToNix32/invalid_format206=== CONT TestFileTokenMissing2072026/11/11 23:58:09 WARN Rate limiter enabled after throttle name=server-test rate=5208=== PAUSE TestConvertHashToNix32/invalid_format209=== PAUSE TestRateLimiterFeedback/503_enables_limiter210=== CONT TestFileTokenReadsAndCaches211=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error212=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error213=== CONT TestCaseHackSuffix214=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter2152026/11/11 23:58:09 WARN Rate limiter enabled after throttle name=server-test rate=5216=== CONT TestRetryTimeoutOutlastsMaxRetries2172026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:42923218--- PASS: TestShellSplitErrors (0.00s)219--- PASS: TestDoServerRequestAttachesToken (0.01s)220=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun221=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter222=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon2232026/11/11 23:58:09 WARN Rate limiter backed off name=server-test rate=5224=== PAUSE TestPathInfoCACompatibility/null_ca_field225=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part2262026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:40259227=== CONT TestScriptTokenEmptyCommand228=== CONT TestScriptTokenNoExpiryRerunsEveryCall229=== CONT TestSetClientTLSDoesNotMutateDefaultTransport230=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false231=== CONT TestFileTokenEmpty232=== CONT TestFilterOversizedClosures233=== CONT TestScriptTokenEmptyToken234=== CONT TestScriptTokenCachesUntilRefresh235=== RUN TestFilterOversizedClosures/no_limit_keeps_everything236=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything237--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)238--- PASS: TestStreamPushReportsSignatures (0.00s)239=== CONT TestStaticToken240=== CONT TestSetClientTLSErrors241=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI242=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter243=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter244=== RUN TestPathInfoCACompatibility/old_string_format_-_text245=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text246=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry247=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false2482026/11/11 23:58:09 INFO Garbage collection started249=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true250=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true251=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped252=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped253=== RUN TestFilterOversizedClosures/all_closures_skipped254=== PAUSE TestFilterOversizedClosures/all_closures_skipped255=== CONT TestRegisterUploadedObjectReusesConnections256=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica257=== CONT TestPartSizeForNAR258=== RUN TestPartSizeForNAR/zero_stays_at_minimum259=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum260=== RUN TestPartSizeForNAR/small_stays_at_minimum261=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI262=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512263=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512264=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary265=== RUN TestSetClientTLSErrors/missing_cert_file266=== PAUSE TestSetClientTLSErrors/missing_cert_file267=== RUN TestSetClientTLSErrors/missing_key_file268=== CONT TestEncodeNixBase32/empty_input269=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths270=== RUN TestSetClientTLS/rejects_connection_without_client_cert271=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert272=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA273=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA274=== RUN TestSetClientTLS/preserves_debug_logging_transport275=== PAUSE TestSetClientTLS/preserves_debug_logging_transport276=== CONT TestParsePathInfoJSON/Nix_format277=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths278=== CONT TestParsePathInfoJSON/empty_input279=== CONT TestParsePathInfoJSON/whitespace_only280=== CONT TestParsePathInfoJSON/Lix_format281=== CONT TestConvertHashToNix32/SRI_format_to_Nix32282=== CONT TestConvertHashToNix32/invalid_format283=== CONT TestConvertHashToNix32/already_Nix32_format284=== CONT TestGetStorePathHash/valid_store_path285=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error286=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error287=== CONT TestGetStorePathHash/basename_without_hyphen_should_error288=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter289=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter290=== CONT TestRateLimiterFeedback/429_enables_limiter291=== CONT TestRateLimiterFeedback/503_enables_limiter292=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary293=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true294=== CONT TestEncodeNixBase32/test_string_hash295=== CONT TestFilterOversizedClosures/no_limit_keeps_everything296=== CONT TestFilterOversizedClosures/all_closures_skipped2972026/11/11 23:58:09 INFO Garbage collection started298=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false2992026/11/11 23:58:09 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50300=== PAUSE TestSetClientTLSErrors/missing_key_file301=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3022026/11/11 23:58:09 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000303=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)304=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512305=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive306=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive307=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI308=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon309=== CONT TestSetClientTLS/rejects_connection_without_client_cert310=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part311=== RUN TestSetClientTLSErrors/missing_ca_file312=== PAUSE TestSetClientTLSErrors/missing_ca_file313=== RUN TestSetClientTLSErrors/invalid_ca_file314=== PAUSE TestSetClientTLSErrors/invalid_ca_file315=== RUN TestPathInfoCACompatibility/new_structured_format_-_text316=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text317=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method318=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method319=== CONT TestParsePathInfoJSON/invalid_JSON3202026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:40259321=== CONT TestSetClientTLSErrors/invalid_ca_file3222026/11/11 23:58:09 WARN Rate limiter enabled after throttle name=server-test rate=53232026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:360313242026/11/11 23:58:09 WARN Rate limiter backed off name=server-test rate=5325=== CONT TestSetClientTLS/preserves_debug_logging_transport3262026/11/11 23:58:09 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0327=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method3282026/11/11 23:58:09 WARN Rate limiter enabled after throttle name=server-test rate=53292026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:43997330=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3312026/11/11 23:58:09 WARN Rate limiter enabled after throttle name=s3-test rate=5332=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary3332026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:336153342026/11/11 23:58:09 WARN Rate limiter backed off name=server-test rate=5335--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.04s)336--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.03s)337--- PASS: TestDumpPathSingleFile (0.03s)3382026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:40259339--- PASS: TestFileTokenMissing (0.00s)340--- PASS: TestScriptTokenEmptyCommand (0.00s)341=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA3422026/11/11 23:58:09 WARN Garbage collection finished on another replica. Its statistics are not available here343=== CONT TestPathInfoCACompatibility/old_string_format_-_text344=== PAUSE TestPartSizeForNAR/small_stays_at_minimum345=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum346=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum347=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts348=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts349=== RUN TestPartSizeForNAR/1_TiB350=== PAUSE TestPartSizeForNAR/1_TiB351=== RUN TestPartSizeForNAR/5_TiB_S3_max_object352=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object353=== RUN TestPartSizeForNAR/capped_at_5_GiB354=== PAUSE TestPartSizeForNAR/capped_at_5_GiB355=== CONT TestPartSizeForNAR/zero_stays_at_minimum356=== CONT TestSetClientTLSErrors/missing_cert_file357=== CONT TestPathInfoCACompatibility/null_ca_field358=== CONT TestPartSizeForNAR/5_TiB_S3_max_object359=== CONT TestSetClientTLSErrors/missing_ca_file3602026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:40259361=== CONT TestSetClientTLSErrors/missing_key_file362=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive363--- PASS: TestFileTokenReadsAndCaches (0.00s)364--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)365=== CONT TestPartSizeForNAR/1_TiB366=== CONT TestPartSizeForNAR/capped_at_5_GiB367=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts368=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum369=== CONT TestPartSizeForNAR/small_stays_at_minimum370--- PASS: TestStaticToken (0.00s)371--- PASS: TestFileTokenEmpty (0.00s)372--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)373--- PASS: TestScriptTokenScriptFails (0.00s)374--- PASS: TestScriptTokenBadJSON (0.00s)375--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)376--- PASS: TestParsePathInfoJSON (0.00s)377 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)378 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)379 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)380 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)381 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)382--- PASS: TestScriptTokenEmptyToken (0.00s)383--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.00s)384--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)385--- PASS: TestPathInfoHashCompatibility (0.04s)386 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)387 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)388 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)389 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)390--- PASS: TestRateLimiterFeedback (0.03s)391 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)392 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)393 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)394 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)395--- PASS: TestEncodeNixBase32 (0.01s)396 --- PASS: TestEncodeNixBase32/empty_input (0.00s)397 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)398--- PASS: TestFilterOversizedClosures (0.01s)399 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)400 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)401 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)402--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.03s)403 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)404 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)4052026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:40259406--- PASS: TestParsePathInfoJSONMultiplePaths (0.01s)407 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)408 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)409--- PASS: TestStreamPushBatchesUnderLoad (0.01s)410 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)411 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)412--- PASS: TestConvertHashToNix32 (0.04s)413 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)414 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)415 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)416--- PASS: TestPartSizeForNAR (0.00s)417 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)418 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)419 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)420 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)421 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)422 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)423 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)424--- PASS: TestGetStorePathHash (0.04s)425 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)426 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)427 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)428 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)429--- PASS: TestPathInfoCACompatibility (0.03s)430 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)431 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)432 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)433 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)434 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)435--- PASS: TestScriptTokenCachesUntilRefresh (0.02s)436--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.02s)437--- PASS: TestSetClientTLSErrors (0.00s)438 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)439 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)440 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)441 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)4422026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:402594432026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:402594442026/11/11 23:58:09 http: TLS handshake error from 127.0.0.1:52894: remote error: tls: bad certificate4452026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:40259446--- PASS: TestSetClientTLS (0.03s)447 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)448 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)449 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4502026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:402594512026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:40259452--- PASS: TestStreamPushRequestLine (0.03s)4532026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:402594542026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:40259455--- PASS: TestCaseHackSuffix (0.04s)4562026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:402594572026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:402594582026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:40259459--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)4602026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:402594612026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:402594622026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:40259463--- PASS: TestDumpPathWriterError (0.08s)4642026/11/11 23:58:09 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:40259465--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.05s)4662026/11/11 23:58:09 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:40647/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:40647/api/uploads/complete4672026/11/11 23:58:09 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4682026/11/11 23:58:09 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:40647/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:40647/api/uploads/complete4692026/11/11 23:58:09 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4702026/11/11 23:58:09 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:40647/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:40647/api/uploads/complete4712026/11/11 23:58:09 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"472--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)473--- PASS: TestDumpPathMatchesNix (0.11s)474--- PASS: TestUploadMultipart_PartsInParallel (0.66s)475--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)476PASS477Running server tests...478The files belonging to this database system will be owned by user "nixbld".479This user must also own the server process.480481The database cluster will be initialized with locale "C".482The default database encoding has accordingly been set to "SQL_ASCII".483The default text search configuration will be set to "english".484485Data page checksums are enabled.486487creating directory /build/postgres1301084863/data ... ok488creating subdirectories ... ok489selecting dynamic shared memory implementation ... posix490selecting default "max_connections" ... 100491selecting default "shared_buffers" ... 128MB492selecting default time zone ... UTC493creating configuration files ... ok494running bootstrap script ... ok495performing post-bootstrap initialization ... ok496syncing data to disk ... ok497498initdb: warning: enabling "trust" authentication for local connections499initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.500501Success. You can now start the database server using:502503 pg_ctl -D /build/postgres1301084863/data -l logfile start504505/build/postgres1301084863:5432 - no response5062026-11-11 23:58:11.352 UTC [129] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit5072026-11-11 23:58:11.353 UTC [129] LOG: listening on Unix socket "/build/postgres1301084863/.s.PGSQL.5432"5082026-11-11 23:58:11.358 UTC [136] LOG: database system was shut down at 2026-11-11 23:58:11 UTC5092026-11-11 23:58:11.361 UTC [129] LOG: database system is ready to accept connections510/build/postgres1301084863:5432 - accepting connections511=== RUN TestAbortSucceeded512=== PAUSE TestAbortSucceeded513=== RUN TestService_AuthMiddleware514=== PAUSE TestService_AuthMiddleware515=== RUN TestService_AuthMiddleware_MTLSProxyHeader516=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader517=== RUN TestService_AuthMiddleware_MTLSBoundSubjects518=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects519=== RUN TestService_ReadAuthMiddleware520=== PAUSE TestService_ReadAuthMiddleware521=== RUN TestService_AuthMiddleware_OIDC522=== PAUSE TestService_AuthMiddleware_OIDC523=== RUN TestService_RequireScope_OIDC524=== PAUSE TestService_RequireScope_OIDC525=== RUN TestService_ReadScope_PublicByDefault526=== PAUSE TestService_ReadScope_PublicByDefault527=== RUN TestCacheConfigHandler528=== PAUSE TestCacheConfigHandler529=== RUN TestCacheStatsHandler530=== PAUSE TestCacheStatsHandler531=== RUN TestClientCADerivations532=== PAUSE TestClientCADerivations533=== RUN TestClientErrorHandling534=== PAUSE TestClientErrorHandling535=== RUN TestClientIntegration536=== PAUSE TestClientIntegration537=== RUN TestClientMultipleUploads538=== PAUSE TestClientMultipleUploads539=== RUN TestClientWithDependencies540=== PAUSE TestClientWithDependencies541=== RUN TestClientSharedPathCommittedMidPush542=== PAUSE TestClientSharedPathCommittedMidPush543=== RUN TestPinProtectsFromGC544=== PAUSE TestPinProtectsFromGC545=== RUN TestClientPushesUseOnePush546=== PAUSE TestClientPushesUseOnePush547=== RUN TestClientFallsBackToClosures548=== PAUSE TestClientFallsBackToClosures549=== RUN TestDeleteClosuresRacingPins550=== PAUSE TestDeleteClosuresRacingPins551=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock552=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock553=== RUN TestResolveDBConnectionString554=== PAUSE TestResolveDBConnectionString555=== RUN TestConnectWaitsForAPeerMigration556=== PAUSE TestConnectWaitsForAPeerMigration557=== RUN TestLeadElectsOneAndHandsOver558=== PAUSE TestLeadElectsOneAndHandsOver559=== RUN TestLeadIncumbentWinsAfterRestart5602026/11/11 23:58:11 INFO lead: acquired remote=192.0.2.1:1234 generation=15612026/11/11 23:58:12 INFO lead: released remote=192.0.2.1:1234 generation=15622026/11/11 23:58:12 INFO lead: acquired remote=192.0.2.1:1234 generation=25632026/11/11 23:58:13 INFO lead: released remote=192.0.2.1:1234 generation=2564--- PASS: TestLeadIncumbentWinsAfterRestart (1.88s)565=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent566=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent567=== RUN TestLeadIncumbentAnnouncesAtOnce568=== PAUSE TestLeadIncumbentAnnouncesAtOnce569=== RUN TestLeadGenerationGrowsWithEveryGrant570=== PAUSE TestLeadGenerationGrowsWithEveryGrant571=== RUN TestLeadEndsOnShutdown572=== PAUSE TestLeadEndsOnShutdown573=== RUN TestLeadEndsWhenItsConnectionHangs5742026/11/11 23:58:13 INFO lead: acquired remote=192.0.2.1:1234 generation=15752026-11-11 23:58:15.163 UTC [573] FATAL: terminating connection due to administrator command5762026/11/11 23:58:15 INFO lead: acquired remote=192.0.2.1:1234 generation=25772026/11/11 23:58:16 WARN lead: lock connection lost error="timeout: context deadline exceeded"5782026/11/11 23:58:16 INFO lead: released remote=192.0.2.1:1234 generation=15792026/11/11 23:58:16 INFO lead: released remote=192.0.2.1:1234 generation=2580--- PASS: TestLeadEndsWhenItsConnectionHangs (2.92s)581=== RUN TestGCAdvisoryLockBlocksConcurrentRun582=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun583=== RUN TestGCBugBareHashReferences584=== PAUSE TestGCBugBareHashReferences585=== RUN TestGCMetrics586=== PAUSE TestGCMetrics587=== RUN TestPushDedupSurvivesConcurrentGC588=== PAUSE TestPushDedupSurvivesConcurrentGC589=== RUN TestDeduplicatedObjectsRecordedAsPending590=== PAUSE TestDeduplicatedObjectsRecordedAsPending591=== RUN TestGCSweepSkipsPendingObjects592=== PAUSE TestGCSweepSkipsPendingObjects593=== RUN TestTombstonedObjectOfferedWithoutWaiting594=== PAUSE TestTombstonedObjectOfferedWithoutWaiting595=== RUN TestGCSweepDeliversEachKeyOnce596=== PAUSE TestGCSweepDeliversEachKeyOnce597=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection598=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection599=== RUN TestForceGCDuringPushOffersSweptObject600=== PAUSE TestForceGCDuringPushOffersSweptObject601=== RUN TestFailedS3DeleteKeepsTombstone602=== PAUSE TestFailedS3DeleteKeepsTombstone603=== RUN TestSweepSkipsRowsLockedByPush604=== PAUSE TestSweepSkipsRowsLockedByPush605=== RUN TestRegisterCompletedObjectNeedsPendingRow606=== PAUSE TestRegisterCompletedObjectNeedsPendingRow607=== RUN TestGetExistingObjectsLocksRows608=== PAUSE TestGetExistingObjectsLocksRows609=== RUN TestSweepRechecksPendingAfterLocking610=== PAUSE TestSweepRechecksPendingAfterLocking611=== RUN TestPushWaitsForSweepHoldingRow612=== PAUSE TestPushWaitsForSweepHoldingRow613=== RUN TestCommitRacingPendingCleanupKeepsObjects614=== PAUSE TestCommitRacingPendingCleanupKeepsObjects615=== RUN TestGCEndsOnShutdown616=== PAUSE TestGCEndsOnShutdown617=== RUN TestGCTaskStore_StartNew618=== PAUSE TestGCTaskStore_StartNew619=== RUN TestGCTaskStore_DeduplicateSameParams620=== PAUSE TestGCTaskStore_DeduplicateSameParams621=== RUN TestGCTaskStore_ConflictDifferentParams622=== PAUSE TestGCTaskStore_ConflictDifferentParams623=== RUN TestGCTaskStore_GetEmpty624=== PAUSE TestGCTaskStore_GetEmpty625=== RUN TestGCTaskStore_GetReturnsLatest626=== PAUSE TestGCTaskStore_GetReturnsLatest627=== RUN TestGCTaskStore_CompletedAllowsNewTask628=== PAUSE TestGCTaskStore_CompletedAllowsNewTask629=== RUN TestGCTaskStore_PhaseUpdates630=== PAUSE TestGCTaskStore_PhaseUpdates631=== RUN TestGCTaskStore_Fail632=== PAUSE TestGCTaskStore_Fail633=== RUN TestGracefulShutdownDrainsInflight634=== PAUSE TestGracefulShutdownDrainsInflight635=== RUN TestService_healthCheckHandler636=== PAUSE TestService_healthCheckHandler637=== RUN TestService_readinessHandler638=== PAUSE TestService_readinessHandler639=== RUN TestGenerateLandingPage640=== PAUSE TestGenerateLandingPage641=== RUN TestEnsureAbortUploadsRule642=== PAUSE TestEnsureAbortUploadsRule643=== RUN TestCacheConfigHandlerMaxNarSize644=== PAUSE TestCacheConfigHandlerMaxNarSize645=== RUN TestCreatePendingClosureRejectsOversizedNAR646=== PAUSE TestCreatePendingClosureRejectsOversizedNAR647=== RUN TestNARDeduplicationMetadataUploadBug648=== PAUSE TestNARDeduplicationMetadataUploadBug649=== RUN TestMetricsInventory650=== PAUSE TestMetricsInventory651=== RUN TestMetricsCommit652=== PAUSE TestMetricsCommit653=== RUN TestService_NativeMTLS654=== PAUSE TestService_NativeMTLS655=== RUN TestServerTLSConfig656=== PAUSE TestServerTLSConfig657=== RUN TestMultipartCleanup658=== PAUSE TestMultipartCleanup659=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded660=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded661=== RUN TestPendingCleanupUsesOneCutoff662=== PAUSE TestPendingCleanupUsesOneCutoff663=== RUN TestObjectStatsTrigger664=== PAUSE TestObjectStatsTrigger665=== RUN TestObjectStatsUpdatedOncePerCommit666=== PAUSE TestObjectStatsUpdatedOncePerCommit667=== RUN TestOrphanedObjectsGC668=== PAUSE TestOrphanedObjectsGC669=== RUN TestOrphanedObjectsGCStressTest670=== PAUSE TestOrphanedObjectsGCStressTest671=== RUN TestCreatePin_ReservedPins672=== PAUSE TestCreatePin_ReservedPins673=== RUN TestPresentReportsOnlyClosureRoots674=== PAUSE TestPresentReportsOnlyClosureRoots675=== RUN TestPresentRefreshesOnlyStaleClosures676=== PAUSE TestPresentRefreshesOnlyStaleClosures677=== RUN TestPresentNotReportedWhileGCDeletesClosure678=== PAUSE TestPresentNotReportedWhileGCDeletesClosure679=== RUN TestParseSingleRange680=== PAUSE TestParseSingleRange681=== RUN TestProxyHeadersOnlyTrustedOnSocket6822026/11/11 23:58:16 INFO Starting HTTP server address=127.0.0.1:461996832026/11/11 23:58:16 INFO Starting HTTP server address=proxy.sock6842026/11/11 23:58:16 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6852026/11/11 23:58:16 INFO Shutdown signal received, draining in-flight requests timeout=10s686--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)687=== RUN TestIsValidCachePath688=== PAUSE TestIsValidCachePath689=== RUN TestReadProxyNarinfo690=== PAUSE TestReadProxyNarinfo691=== RUN TestReadProxyNarinfoAlreadyDecompressed692=== PAUSE TestReadProxyNarinfoAlreadyDecompressed693=== RUN TestReadProxyContentNegotiation694=== PAUSE TestReadProxyContentNegotiation695=== RUN TestReadProxyNarStreaming696=== PAUSE TestReadProxyNarStreaming697=== RUN TestReadProxy404698=== PAUSE TestReadProxy404699=== RUN TestReadProxyInvalidPath700=== PAUSE TestReadProxyInvalidPath701=== RUN TestReadProxyHead702=== PAUSE TestReadProxyHead703=== RUN TestReadProxyOutlastsServerWriteTimeout704=== PAUSE TestReadProxyOutlastsServerWriteTimeout705=== RUN TestReadProxyConditionalGet706=== PAUSE TestReadProxyConditionalGet707=== RUN TestReadProxyRootRedirectsToIndexHTML708=== PAUSE TestReadProxyRootRedirectsToIndexHTML709=== RUN TestReadProxyDisabled710=== PAUSE TestReadProxyDisabled711=== RUN TestReadRedirectNar712=== PAUSE TestReadRedirectNar713=== RUN TestReadRedirectKeepsNarinfoProxied714=== PAUSE TestReadRedirectKeepsNarinfoProxied715=== RUN TestReadProxyRangeRequest716=== PAUSE TestReadProxyRangeRequest717=== RUN TestReadRedirectUsesPublicS3URL718=== PAUSE TestReadRedirectUsesPublicS3URL719=== RUN TestPush_OverlappingRootsStoreOneRowPerKey720=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey721=== RUN TestPush_CompleteCommitsEveryRoot722=== PAUSE TestPush_CompleteCommitsEveryRoot723=== RUN TestPush_RepeatedRootCommitsOnce724=== PAUSE TestPush_RepeatedRootCommitsOnce725=== RUN TestPush_CompleteWaitsForCleanupOfThePush726=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush727=== RUN TestPush_RootsAreLockedInByteOrder728=== PAUSE TestPush_RootsAreLockedInByteOrder729=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit730=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit731=== RUN TestPush_RejectsBadRequests732=== PAUSE TestPush_RejectsBadRequests733=== RUN TestPush_SignsNarinfosOfItsPendingObjects734=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects735=== RUN TestSharedNarBothUploadsComplete736=== PAUSE TestSharedNarBothUploadsComplete737=== RUN TestCompleteMultipartUpload_ErrorButObjectExists738=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists739=== RUN TestCompletedNarNotReofferedAcrossClosures740=== PAUSE TestCompletedNarNotReofferedAcrossClosures741=== RUN TestPresignedUploadRegisteredBeforeCommit742=== PAUSE TestPresignedUploadRegisteredBeforeCommit743=== RUN TestService_Rustfstest744=== PAUSE TestService_Rustfstest745=== RUN TestParseSize746=== PAUSE TestParseSize747=== RUN TestSkippedUploadsHandler748=== PAUSE TestSkippedUploadsHandler749=== RUN TestSystemdListenerNotActivated750--- PASS: TestSystemdListenerNotActivated (0.00s)751=== RUN TestWatchdogBeatsWhenHealthy752--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)753=== RUN TestWatchdogSkipsWhenUnhealthy7542026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7552026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7562026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7582026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7592026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7602026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/11 23:58:16 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"764--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)765=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle766=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle767=== RUN TestProxyWriteTimeout768=== PAUSE TestProxyWriteTimeout769=== RUN TestPendingClosureWriteTimeout770=== PAUSE TestPendingClosureWriteTimeout771=== RUN TestIsValidUploadKey772=== PAUSE TestIsValidUploadKey773=== RUN TestUploadHandlersRejectInvalidKeys774=== PAUSE TestUploadHandlersRejectInvalidKeys775=== RUN TestUploadHandlersRejectOversizedBody776=== PAUSE TestUploadHandlersRejectOversizedBody777=== RUN TestService_cleanupPendingClosuresHandler778=== PAUSE TestService_cleanupPendingClosuresHandler779=== RUN TestService_createPendingClosureHandler780=== PAUSE TestService_createPendingClosureHandler781=== RUN TestService_verifyS3Integrity782=== PAUSE TestService_verifyS3Integrity783=== RUN TestService_verifyS3ReuploadsStaleNarinfo784=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo785=== RUN TestCompleteMultipartUnregistered786=== PAUSE TestCompleteMultipartUnregistered787=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT788=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT789=== CONT TestAbortSucceeded790=== CONT TestProxyWriteTimeout791--- PASS: TestAbortSucceeded (0.00s)792=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT793=== CONT TestCompleteMultipartUnregistered794=== CONT TestService_verifyS3ReuploadsStaleNarinfo795=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed796=== CONT TestService_verifyS3Integrity797=== CONT TestService_createPendingClosureHandler798=== CONT TestService_cleanupPendingClosuresHandler799=== CONT TestUploadHandlersRejectOversizedBody800=== CONT TestUploadHandlersRejectInvalidKeys801=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info802=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info803=== CONT TestIsValidUploadKey804=== CONT TestPendingClosureWriteTimeout805=== RUN TestPendingClosureWriteTimeout/empty806=== CONT TestReadProxyContentNegotiation807=== CONT TestReadProxyNarinfoAlreadyDecompressed808=== CONT TestReadProxyNarinfo809=== CONT TestIsValidCachePath810=== RUN TestIsValidCachePath/narinfo811=== PAUSE TestIsValidCachePath/narinfo812=== CONT TestParseSingleRange813=== CONT TestPresentNotReportedWhileGCDeletesClosure814=== CONT TestPresentRefreshesOnlyStaleClosures815=== CONT TestPresentReportsOnlyClosureRoots816=== CONT TestPush_RepeatedRootCommitsOnce817=== CONT TestCreatePin_ReservedPins818=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle819=== CONT TestSkippedUploadsHandler820=== CONT TestOrphanedObjectsGCStressTest821=== RUN TestProxyWriteTimeout/narinfo822=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed823=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal824=== RUN TestIsValidUploadKey/narinfo825=== PAUSE TestPendingClosureWriteTimeout/empty826=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars827=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars828=== RUN TestIsValidCachePath/nar_zst829=== PAUSE TestIsValidCachePath/nar_zst830=== RUN TestIsValidCachePath/nar_xz831=== PAUSE TestIsValidCachePath/nar_xz832=== RUN TestIsValidCachePath/nar_bz2833=== PAUSE TestIsValidCachePath/nar_bz2834=== RUN TestIsValidCachePath/nar_uncompressed835=== PAUSE TestProxyWriteTimeout/narinfo836=== RUN TestProxyWriteTimeout/1_GiB_nar837=== RUN TestPendingClosureWriteTimeout/400_objects838=== RUN TestParseSingleRange/none839=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale8402026/11/11 23:58:16 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000841=== PAUSE TestPendingClosureWriteTimeout/400_objects842=== PAUSE TestIsValidUploadKey/narinfo843=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed844=== PAUSE TestIsValidCachePath/nar_uncompressed845=== RUN TestIsValidCachePath/ls846=== PAUSE TestProxyWriteTimeout/1_GiB_nar847=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale848=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh849=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh850=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal851=== PAUSE TestParseSingleRange/none852=== RUN TestParseSingleRange/unknown_unit853=== PAUSE TestParseSingleRange/unknown_unit854=== RUN TestParseSingleRange/multi-range_ignored855=== PAUSE TestParseSingleRange/multi-range_ignored856=== RUN TestParseSingleRange/malformed_no_dash857=== PAUSE TestParseSingleRange/malformed_no_dash858=== RUN TestParseSingleRange/malformed_both_empty859=== RUN TestPendingClosureWriteTimeout/670k_objects860=== RUN TestIsValidUploadKey/nar_zst861=== PAUSE TestPendingClosureWriteTimeout/670k_objects862=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline863=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline864=== PAUSE TestIsValidCachePath/ls865=== RUN TestProxyWriteTimeout/10_GiB_nar866=== PAUSE TestProxyWriteTimeout/10_GiB_nar867=== CONT TestParseSize868--- PASS: TestParseSize (0.00s)869=== CONT TestOrphanedObjectsGC870=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key871=== PAUSE TestParseSingleRange/malformed_both_empty872=== RUN TestParseSingleRange/malformed_end_before_start873=== PAUSE TestParseSingleRange/malformed_end_before_start874=== PAUSE TestIsValidUploadKey/nar_zst875=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed876=== CONT TestPush_SignsNarinfosOfItsPendingObjects877=== RUN TestIsValidCachePath/log878=== PAUSE TestIsValidCachePath/log879=== RUN TestIsValidCachePath/realisation880=== PAUSE TestIsValidCachePath/realisation881=== RUN TestIsValidCachePath/nix-cache-info882=== PAUSE TestIsValidCachePath/nix-cache-info883=== RUN TestProxyWriteTimeout/unknown_size884=== PAUSE TestProxyWriteTimeout/unknown_size885=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key886=== RUN TestParseSingleRange/closed887=== PAUSE TestParseSingleRange/closed888=== RUN TestParseSingleRange/open-ended889=== PAUSE TestParseSingleRange/open-ended890=== RUN TestIsValidUploadKey/nar_xz891=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost892=== RUN TestIsValidCachePath/index.html893=== PAUSE TestIsValidCachePath/index.html894=== RUN TestIsValidCachePath/traversal_parent895=== PAUSE TestIsValidCachePath/traversal_parent896=== RUN TestIsValidCachePath/traversal_in_middle897=== PAUSE TestIsValidCachePath/traversal_in_middle898=== RUN TestIsValidCachePath/invalid_char_e899=== PAUSE TestIsValidCachePath/invalid_char_e900=== RUN TestIsValidCachePath/invalid_char_u901=== PAUSE TestIsValidCachePath/invalid_char_u902=== RUN TestIsValidCachePath/random_path903=== PAUSE TestIsValidCachePath/random_path904=== RUN TestIsValidCachePath/empty905=== PAUSE TestIsValidCachePath/empty906=== RUN TestIsValidCachePath/leading_slash907=== PAUSE TestIsValidCachePath/leading_slash908=== RUN TestIsValidCachePath/wrong_extension909=== PAUSE TestIsValidCachePath/wrong_extension910=== CONT TestObjectStatsUpdatedOncePerCommit911=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost912=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key913=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key914=== RUN TestParseSingleRange/end_clamped_to_size915=== PAUSE TestIsValidUploadKey/nar_xz916=== CONT TestPush_RejectsBadRequests917--- PASS: TestSkippedUploadsHandler (0.09s)918=== RUN TestIsValidUploadKey/nar_plain919=== RUN TestIsValidCachePath/short_hash920=== PAUSE TestIsValidCachePath/short_hash921=== CONT TestPush_RootsAreLockedInByteOrder922=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed923=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit924=== PAUSE TestParseSingleRange/end_clamped_to_size925=== RUN TestParseSingleRange/suffix926=== PAUSE TestIsValidUploadKey/nar_plain927=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed928=== RUN TestIsValidUploadKey/listing929=== PAUSE TestIsValidUploadKey/listing930=== RUN TestIsValidUploadKey/build_log931=== PAUSE TestIsValidUploadKey/build_log932=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed933=== PAUSE TestParseSingleRange/suffix934=== RUN TestIsValidUploadKey/build_log_home-manager_file935=== PAUSE TestIsValidUploadKey/build_log_home-manager_file936=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed937=== RUN TestParseSingleRange/suffix_exceeds_size938=== PAUSE TestParseSingleRange/suffix_exceeds_size939=== RUN TestParseSingleRange/single_byte940=== PAUSE TestParseSingleRange/single_byte941=== RUN TestParseSingleRange/start_past_EOF942=== RUN TestIsValidUploadKey/build_log_plus_in_name943=== CONT TestObjectStatsTrigger944=== PAUSE TestParseSingleRange/start_past_EOF945=== PAUSE TestIsValidUploadKey/build_log_plus_in_name946=== RUN TestParseSingleRange/start_far_past_EOF947=== PAUSE TestParseSingleRange/start_far_past_EOF948=== RUN TestIsValidUploadKey/build_log_question_mark949=== PAUSE TestIsValidUploadKey/build_log_question_mark950=== RUN TestIsValidUploadKey/build_log_equals951=== PAUSE TestIsValidUploadKey/build_log_equals952=== RUN TestIsValidUploadKey/realisation953=== PAUSE TestIsValidUploadKey/realisation954=== CONT TestPush_CompleteWaitsForCleanupOfThePush955=== RUN TestIsValidUploadKey/realisation_plus_in_output956=== PAUSE TestIsValidUploadKey/realisation_plus_in_output957=== RUN TestIsValidUploadKey/nix-cache-info958=== PAUSE TestIsValidUploadKey/nix-cache-info959=== RUN TestIsValidUploadKey/index.html960=== PAUSE TestIsValidUploadKey/index.html961=== RUN TestIsValidUploadKey/narinfo_key,_nar_type962=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type963=== RUN TestIsValidUploadKey/nar_key,_narinfo_type964=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type965=== RUN TestIsValidUploadKey/listing_key,_narinfo_type966=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type967=== RUN TestIsValidUploadKey/traversal968=== PAUSE TestIsValidUploadKey/traversal969=== RUN TestIsValidUploadKey/traversal_nar970=== PAUSE TestIsValidUploadKey/traversal_nar971=== RUN TestIsValidUploadKey/absolute972=== PAUSE TestIsValidUploadKey/absolute973=== RUN TestIsValidUploadKey/empty_key974=== PAUSE TestIsValidUploadKey/empty_key975=== RUN TestIsValidUploadKey/unknown_type976=== PAUSE TestIsValidUploadKey/unknown_type977=== CONT TestPendingCleanupUsesOneCutoff978=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure979=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure980=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart981=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart982=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts983=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts984=== CONT TestSharedNarBothUploadsComplete9852026/11/11 23:58:17 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45131/oidc9862026/11/11 23:58:17 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9872026/11/11 23:58:17 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst988--- PASS: TestCompleteMultipartUnregistered (0.41s)989=== CONT TestService_Rustfstest9902026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures9912026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures9922026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures9932026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures994--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.45s)995=== CONT TestTombstonedObjectOfferedWithoutWaiting9962026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures9972026/11/11 23:58:17 INFO Received cleanup request method=DELETE path=/api/pending_closures9982026/11/11 23:58:17 INFO Aborted multipart uploads count=0 kept=09992026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures10002026/11/11 23:58:17 INFO Received cleanup request method=DELETE path=/api/pending_closures10012026/11/11 23:58:17 INFO Aborted multipart uploads count=1 kept=010022026/11/11 23:58:17 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10032026-11-11 23:58:17.426 UTC [651] ERROR: Closure does not exist: id=110042026-11-11 23:58:17.426 UTC [651] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE10052026-11-11 23:58:17.426 UTC [651] STATEMENT: -- name: CommitPendingClosure :one1006 SELECT commit_pending_closure($1::bigint)::bigint1007 1008--- PASS: TestService_cleanupPendingClosuresHandler (0.52s)1009=== CONT TestGCSweepSkipsPendingObjects1010--- PASS: TestPresentReportsOnlyClosureRoots (0.53s)1011=== CONT TestPresignedUploadRegisteredBeforeCommit1012--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.56s)1013=== CONT TestDeduplicatedObjectsRecordedAsPending10142026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes10152026/11/11 23:58:17 INFO Received complete push request method=POST path=/api/pushes/1/complete10162026/11/11 23:58:17 INFO Completed push id=1 objects=4 duration=11.800403ms1017--- PASS: TestPush_RepeatedRootCommitsOnce (0.61s)1018=== CONT TestCompletedNarNotReofferedAcrossClosures1019--- PASS: TestPresentRefreshesOnlyStaleClosures (0.64s)1020=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded10212026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes10222026/11/11 23:58:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10232026/11/11 23:58:17 INFO Signed narinfos id=1 count=110242026/11/11 23:58:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10252026/11/11 23:58:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10262026/11/11 23:58:17 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"10272026/11/11 23:58:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10282026/11/11 23:58:17 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"10292026/11/11 23:58:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10302026/11/11 23:58:17 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=110312026/11/11 23:58:17 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10322026/11/11 23:58:17 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1033--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.59s)1034=== CONT TestPushDedupSurvivesConcurrentGC10352026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures1036--- PASS: TestReadProxyContentNegotiation (0.76s)1037=== CONT TestMultipartCleanup10382026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes10392026/11/11 23:58:17 INFO Received complete push request method=POST path=/api/pushes/1/complete10402026/11/11 23:58:17 INFO Completed push id=1 objects=4 duration=8.527815ms10412026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes10422026/11/11 23:58:17 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1043--- PASS: TestObjectStatsTrigger (0.73s)1044=== CONT TestCompleteMultipartUpload_ErrorButObjectExists1045=== RUN TestPush_RejectsBadRequests/no_objects1046=== PAUSE TestPush_RejectsBadRequests/no_objects1047=== RUN TestPush_RejectsBadRequests/bad_root1048=== PAUSE TestPush_RejectsBadRequests/bad_root1049=== RUN TestPush_RejectsBadRequests/root_not_in_objects1050=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1051=== RUN TestPush_RejectsBadRequests/no_roots1052=== PAUSE TestPush_RejectsBadRequests/no_roots1053=== CONT TestGCMetrics10542026/11/11 23:58:17 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1055--- PASS: TestReadProxyNarinfo (0.86s)1056=== CONT TestServerTLSConfig1057=== RUN TestServerTLSConfig/no_client_CA1058=== PAUSE TestServerTLSConfig/no_client_CA1059=== RUN TestServerTLSConfig/missing_CA_file1060=== PAUSE TestServerTLSConfig/missing_CA_file1061=== RUN TestServerTLSConfig/not_a_PEM_file1062=== PAUSE TestServerTLSConfig/not_a_PEM_file1063=== CONT TestGCBugBareHashReferences10642026/11/11 23:58:17 INFO Aborted multipart uploads count=0 kept=010652026/11/11 23:58:17 WARN Force mode enabled - objects will be deleted immediately without grace period10662026/11/11 23:58:17 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10672026/11/11 23:58:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=010682026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes10692026/11/11 23:58:17 INFO Vacuumed table table=pending_closures10702026/11/11 23:58:17 INFO Vacuumed table table=pending_objects10712026/11/11 23:58:17 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=NDAwODgwZjktMGFhNi00NTlmLWIyZWMtZjY3ZGY0NmViODZmLjgzNmNkZGQxLWQ2OWQtNGZhOS04YTQ3LTdjOGU1ZmUxN2I4NHgxNzk0NDQxNDk3MzY0MDU3NjIx parts=1010722026/11/11 23:58:17 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10732026/11/11 23:58:17 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10742026/11/11 23:58:17 INFO Vacuumed table table=multipart_uploads10752026/11/11 23:58:17 INFO Vacuumed table table=closures10762026/11/11 23:58:17 INFO Completed upload id=1 objects=2 duration=7.312286ms10772026/11/11 23:58:17 INFO Vacuumed table table=objects10782026/11/11 23:58:17 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010792026/11/11 23:58:17 INFO Received complete push request method=POST path=/api/pushes/2/complete10802026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures10812026/11/11 23:58:17 INFO Completed push id=2 objects=4 duration=3.440078ms1082--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.84s)1083=== CONT TestGCTaskStore_GetReturnsLatest1084--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1085=== CONT TestGCTaskStore_GetEmpty1086--- PASS: TestGCTaskStore_GetEmpty (0.00s)1087=== CONT TestService_NativeMTLS10882026/11/11 23:58:17 INFO Starting cleanup of old closures method=DELETE path=/api/closures10892026/11/11 23:58:17 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=NDAwODgwZjktMGFhNi00NTlmLWIyZWMtZjY3ZGY0NmViODZmLjliZjgwNDIxLTg0ZjItNDU1NC1iNTc3LWFhNjlmMWJjY2Y5ZHgxNzk0NDQxNDk3MzgwMDc1MTgx parts=1010902026/11/11 23:58:17 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10912026/11/11 23:58:17 INFO Completed upload id=1 objects=2 duration=3.395108ms10922026/11/11 23:58:17 INFO Aborted multipart uploads count=0 kept=010932026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures10942026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures10952026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures10962026/11/11 23:58:17 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10972026/11/11 23:58:17 WARN Found objects in DB but missing from S3, will re-upload count=11098--- PASS: TestService_verifyS3Integrity (0.96s)1099=== CONT TestMetricsCommit11002026/11/11 23:58:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011012026/11/11 23:58:17 INFO Vacuumed table table=pending_closures11022026/11/11 23:58:17 INFO Vacuumed table table=pending_objects11032026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures11042026/11/11 23:58:17 INFO Vacuumed table table=multipart_uploads11052026/11/11 23:58:17 INFO Vacuumed table table=closures11062026/11/11 23:58:17 INFO Vacuumed table table=objects11072026/11/11 23:58:17 INFO Received cleanup request method=DELETE path=/api/pending_closures11082026/11/11 23:58:17 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001109--- PASS: TestService_createPendingClosureHandler (0.98s)1110=== CONT TestGCTaskStore_ConflictDifferentParams1111--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1112=== CONT TestGCTaskStore_DeduplicateSameParams1113--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1114=== CONT TestGCAdvisoryLockBlocksConcurrentRun11152026/11/11 23:58:17 INFO Received push request method=POST path=/api/pushes1116--- PASS: TestObjectStatsUpdatedOncePerCommit (0.94s)1117=== CONT TestGCTaskStore_StartNew1118--- PASS: TestGCTaskStore_StartNew (0.00s)1119=== CONT TestMetricsInventory11202026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures11212026-11-11 23:58:17.970 UTC [740] ERROR: could not obtain lock on row in relation "closures"11222026-11-11 23:58:17.970 UTC [740] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11232026-11-11 23:58:17.971 UTC [740] ERROR: could not obtain lock on row in relation "closures"11242026-11-11 23:58:17.971 UTC [740] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11252026-11-11 23:58:17.971 UTC [740] ERROR: could not obtain lock on row in relation "closures"11262026-11-11 23:58:17.971 UTC [740] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11272026-11-11 23:58:17.971 UTC [740] ERROR: could not obtain lock on row in relation "closures"11282026-11-11 23:58:17.971 UTC [740] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11292026-11-11 23:58:17.972 UTC [740] ERROR: could not obtain lock on row in relation "closures"11302026-11-11 23:58:17.972 UTC [740] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11312026/11/11 23:58:17 INFO Received uploads request method=POST path=/api/pending_closures1132--- PASS: TestPush_RootsAreLockedInByteOrder (0.97s)1133=== CONT TestGCEndsOnShutdown1134=== RUN TestGCEndsOnShutdown/before_the_run1135=== PAUSE TestGCEndsOnShutdown/before_the_run1136=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1137=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1138=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1139=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1140=== CONT TestLeadEndsOnShutdown11412026/11/11 23:58:17 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11422026/11/11 23:58:17 WARN Refused reserved pin name=worker-x86_64-linux11432026/11/11 23:58:17 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11442026/11/11 23:58:17 INFO Received create pin request method=POST path=/api/pins/my-app11452026/11/11 23:58:17 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1146--- PASS: TestCreatePin_ReservedPins (1.09s)1147=== CONT TestCommitRacingPendingCleanupKeepsObjects1148--- PASS: TestService_Rustfstest (0.73s)1149=== CONT TestNARDeduplicationMetadataUploadBug11502026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1151--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.73s)1152=== CONT TestPushWaitsForSweepHoldingRow11532026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures11542026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=011552026-11-11 23:58:18.138 UTC [726] ERROR: Push does not exist: id=111562026-11-11 23:58:18.138 UTC [726] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE11572026-11-11 23:58:18.138 UTC [726] STATEMENT: SELECT commit_push($1)1158--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (1.13s)1159=== CONT TestLeadGenerationGrowsWithEveryGrant11602026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures11612026/11/11 23:58:18 WARN Force mode enabled - objects will be deleted immediately without grace period11622026/11/11 23:58:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011632026/11/11 23:58:18 INFO Vacuumed table table=pending_closures11642026/11/11 23:58:18 INFO Vacuumed table table=pending_objects11652026/11/11 23:58:18 INFO Vacuumed table table=multipart_uploads11662026/11/11 23:58:18 INFO Vacuumed table table=closures11672026/11/11 23:58:18 INFO Vacuumed table table=objects1168--- PASS: TestGCSweepSkipsPendingObjects (0.73s)1169=== CONT TestCreatePendingClosureRejectsOversizedNAR11702026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1171--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1172=== CONT TestLeadIncumbentAnnouncesAtOnce11732026/11/11 23:58:18 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst11742026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1175--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.72s)1176=== CONT TestSweepRechecksPendingAfterLocking11772026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1178--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.72s)1179=== CONT TestCacheConfigHandlerMaxNarSize1180--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1181=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent11822026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1183=== NAME TestOrphanedObjectsGC1184 orphaned_objects_gc_test.go:287: GC Test Summary:1185 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1186 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1187 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1188 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1189 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1190--- PASS: TestOrphanedObjectsGC (1.24s)1191=== CONT TestGetExistingObjectsLocksRows11922026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1193--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.72s)1194=== CONT TestEnsureAbortUploadsRule11952026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures11962026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures11972026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures11982026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=011992026/11/11 23:58:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12002026/11/11 23:58:18 WARN Force mode enabled - objects will be deleted immediately without grace period12012026/11/11 23:58:18 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NDAwODgwZjktMGFhNi00NTlmLWIyZWMtZjY3ZGY0NmViODZmLmRiNWM4ZGNkLTk2YmMtNGFmMC05MWFkLTY4NDJiZmRkODkxMXgxNzk0NDQxNDk4MzU0MTM1NDQ212022026/11/11 23:58:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012032026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=012042026/11/11 23:58:18 INFO Vacuumed table table=pending_closures12052026/11/11 23:58:18 INFO Vacuumed table table=pending_objects12062026/11/11 23:58:18 INFO Vacuumed table table=multipart_uploads12072026/11/11 23:58:18 INFO Vacuumed table table=closures12082026/11/11 23:58:18 INFO Vacuumed table table=objects12092026/11/11 23:58:18 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=NDAwODgwZjktMGFhNi00NTlmLWIyZWMtZjY3ZGY0NmViODZmLmRiNWM4ZGNkLTk2YmMtNGFmMC05MWFkLTY4NDJiZmRkODkxMXgxNzk0NDQxNDk4MzU0MTM1NDQ2 parts=112102026/11/11 23:58:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01211--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.66s)1212=== CONT TestRegisterCompletedObjectNeedsPendingRow1213--- PASS: TestGCMetrics (0.65s)1214=== CONT TestLeadElectsOneAndHandsOver12152026/11/11 23:58:18 INFO Vacuumed table table=pending_closures12162026/11/11 23:58:18 INFO Vacuumed table table=pending_objects12172026/11/11 23:58:18 INFO Vacuumed table table=multipart_uploads12182026/11/11 23:58:18 INFO Vacuumed table table=closures12192026/11/11 23:58:18 INFO Vacuumed table table=objects12202026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=012212026/11/11 23:58:18 WARN Force mode enabled - objects will be deleted immediately without grace period12222026/11/11 23:58:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012232026/11/11 23:58:18 INFO Vacuumed table table=pending_closures12242026/11/11 23:58:18 INFO Vacuumed table table=pending_objects12252026/11/11 23:58:18 INFO Vacuumed table table=multipart_uploads12262026/11/11 23:58:18 INFO Vacuumed table table=closures12272026/11/11 23:58:18 INFO Vacuumed table table=objects1228--- PASS: TestPushDedupSurvivesConcurrentGC (0.83s)1229=== CONT TestSweepSkipsRowsLockedByPush12302026/11/11 23:58:18 WARN mTLS auth: subject not in bound subjects subject="CN=reader"12312026/11/11 23:58:18 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1232--- PASS: TestService_NativeMTLS (0.59s)1233=== CONT TestGenerateLandingPage1234--- PASS: TestGenerateLandingPage (0.00s)1235=== CONT TestConnectWaitsForAPeerMigration12362026/11/11 23:58:18 INFO Received cleanup request method=DELETE path=/api/pending_closures12372026/11/11 23:58:18 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket36/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""12382026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=112392026/11/11 23:58:18 INFO Received cleanup request method=DELETE path=/api/pending_closures12402026/11/11 23:58:18 INFO Aborted multipart uploads count=1 kept=01241--- PASS: TestMultipartCleanup (0.79s)1242=== CONT TestFailedS3DeleteKeepsTombstone12432026/11/11 23:58:18 INFO Received push request method=POST path=/api/pushes12442026/11/11 23:58:18 INFO Received complete push request method=POST path=/api/pushes/1/complete12452026/11/11 23:58:18 INFO Completed push id=1 objects=4 duration=9.727574ms12462026/11/11 23:58:18 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1247--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.61s)1248=== CONT TestResolveDBConnectionString1249=== RUN TestResolveDBConnectionString/flag_wins1250=== PAUSE TestResolveDBConnectionString/flag_wins1251=== RUN TestResolveDBConnectionString/file_when_flag_empty1252=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1253=== RUN TestResolveDBConnectionString/missing_file_is_an_error1254=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1255=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1256=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1257=== RUN TestResolveDBConnectionString/nothing_configured1258=== PAUSE TestResolveDBConnectionString/nothing_configured1259=== CONT TestForceGCDuringPushOffersSweptObject1260=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1261=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1262=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1263=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1264=== CONT TestService_readinessHandler12652026/11/11 23:58:18 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12662026/11/11 23:58:18 INFO Completed upload id=2 objects=2 duration=1.81781ms1267--- PASS: TestMetricsCommit (0.65s)1268=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1269--- PASS: TestMetricsInventory (0.61s)1270=== CONT TestService_healthCheckHandler12712026/11/11 23:58:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12722026/11/11 23:58:18 INFO lead: acquired remote=192.0.2.1:1234 generation=112732026/11/11 23:58:18 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NDAwODgwZjktMGFhNi00NTlmLWIyZWMtZjY3ZGY0NmViODZmLjFiODI3MTZlLTllMmYtNGNiYi1hYjk3LTc2NjMxOGI3ZWQxOXgxNzk0NDQxNDk3OTY5NzgxNTE5 parts=121274--- PASS: TestGCBugBareHashReferences (0.86s)1275=== CONT TestGracefulShutdownDrainsInflight12762026/11/11 23:58:18 INFO Starting HTTP server address=127.0.0.1:3610712772026/11/11 23:58:18 INFO Shutdown signal received, draining in-flight requests timeout=10s12782026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=012792026/11/11 23:58:18 WARN Force mode enabled - objects will be deleted immediately without grace period12802026/11/11 23:58:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012812026/11/11 23:58:18 INFO Vacuumed table table=pending_closures12822026/11/11 23:58:18 INFO Vacuumed table table=pending_objects12832026/11/11 23:58:18 INFO Vacuumed table table=multipart_uploads12842026/11/11 23:58:18 INFO Vacuumed table table=closures12852026/11/11 23:58:18 INFO Vacuumed table table=objects1286--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.67s)1287=== CONT TestGCTaskStore_Fail1288--- PASS: TestGCTaskStore_Fail (0.00s)1289=== CONT TestGCTaskStore_PhaseUpdates1290--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1291=== CONT TestGCTaskStore_CompletedAllowsNewTask1292--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1293=== CONT TestDeleteClosuresRacingPins12942026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1295=== NAME TestNARDeduplicationMetadataUploadBug1296 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3986758916/001/store/jryp5vxqc6ib5mhixiwyawnspmkjzf98-file1.txt12972026/11/11 23:58:18 INFO lead: acquired remote=192.0.2.1:1234 generation=11298--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1299=== CONT TestClientCADerivations13002026/11/11 23:58:18 INFO lead: acquired remote=192.0.2.1:1234 generation=113012026/11/11 23:58:18 INFO lead: released remote=192.0.2.1:1234 generation=11302--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.57s)1303=== CONT TestClientFallsBackToClosures13042026/11/11 23:58:18 INFO Received push request method=POST path=/api/pushes13052026/11/11 23:58:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete13062026/11/11 23:58:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13072026/11/11 23:58:18 INFO Uploading jryp5vxqc6ib5mhixiwyawnspmkjzf98-file1.txt (160B)13082026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=013092026/11/11 23:58:18 WARN Failed to register uploaded object key=jryp5vxqc6ib5mhixiwyawnspmkjzf98.ls error="server returned 404: 404 page not found\n"13102026/11/11 23:58:18 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"13112026/11/11 23:58:18 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13122026/11/11 23:58:18 INFO Signed narinfos id=1 count=113132026/11/11 23:58:18 INFO Uploading 1 narinfos13142026/11/11 23:58:18 WARN Force mode enabled - objects will be deleted immediately without grace period13152026/11/11 23:58:18 INFO lead: acquired remote=192.0.2.1:1234 generation=113162026/11/11 23:58:18 INFO Received complete push request method=POST path=/api/pushes/1/complete13172026/11/11 23:58:18 WARN Failed to register uploaded object key=jryp5vxqc6ib5mhixiwyawnspmkjzf98.narinfo error="server returned 404: 404 page not found\n"13182026/11/11 23:58:18 INFO Completed push id=1 objects=3 duration=9.934885ms13192026/11/11 23:58:18 INFO Upload complete. (76ms)1320=== NAME TestNARDeduplicationMetadataUploadBug1321 metadata_upload_test.go:54: Retrieved narinfo from S3:1322 StorePath: /build/TestNARDeduplicationMetadataUploadBug3986758916/001/store/jryp5vxqc6ib5mhixiwyawnspmkjzf98-file1.txt1323 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1324 Compression: zstd1325 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1326 NarSize: 1601327 References: 1328 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf13292026/11/11 23:58:18 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=NDAwODgwZjktMGFhNi00NTlmLWIyZWMtZjY3ZGY0NmViODZmLmE2YmJlNjBkLWMzNjktNGE0OC1hNTE3LWE5ZTE4ZDE0MWQwYngxNzk0NDQxNDk4MjE4NTU3MTM0 parts=1213302026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures1331 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1332 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1333 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}13342026/11/11 23:58:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01335--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.28s)1336=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection13372026/11/11 23:58:18 INFO Vacuumed table table=pending_closures13382026/11/11 23:58:18 INFO Vacuumed table table=pending_objects13392026/11/11 23:58:18 INFO Vacuumed table table=multipart_uploads13402026/11/11 23:58:18 INFO Vacuumed table table=closures13412026/11/11 23:58:18 INFO Vacuumed table table=objects1342--- PASS: TestSweepRechecksPendingAfterLocking (0.65s)1343=== CONT TestClientPushesUseOnePush1344--- PASS: TestGetExistingObjectsLocksRows (0.59s)1345=== CONT TestPinProtectsFromGC1346=== NAME TestNARDeduplicationMetadataUploadBug1347 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug3986758916/001/store/jlnjxw9pmi5ydf96as6dwdwcx7r0ih6x-file2.txt13482026/11/11 23:58:18 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket53 days=21349--- PASS: TestEnsureAbortUploadsRule (0.59s)1350=== CONT TestService_AuthMiddleware_OIDC1351--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.49s)1352=== CONT TestClientSharedPathCommittedMidPush13532026/11/11 23:58:18 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38191/oidc13542026/11/11 23:58:18 INFO Received push request method=POST path=/api/pushes13552026/11/11 23:58:18 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)13562026/11/11 23:58:18 INFO lead: acquired remote=192.0.2.1:1234 generation=113572026/11/11 23:58:18 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13582026/11/11 23:58:18 INFO Signed narinfos id=2 count=113592026/11/11 23:58:18 WARN Failed to register uploaded object key=jlnjxw9pmi5ydf96as6dwdwcx7r0ih6x.ls error="server returned 404: 404 page not found\n"13602026/11/11 23:58:18 INFO Uploading 1 narinfos13612026/11/11 23:58:18 INFO Received complete push request method=POST path=/api/pushes/2/complete13622026/11/11 23:58:18 WARN Failed to register uploaded object key=jlnjxw9pmi5ydf96as6dwdwcx7r0ih6x.narinfo error="server returned 404: 404 page not found\n"13632026/11/11 23:58:18 INFO Completed push id=2 objects=3 duration=2.947309ms13642026/11/11 23:58:18 INFO Upload complete. (60ms)1365=== NAME TestNARDeduplicationMetadataUploadBug1366 metadata_upload_test.go:76: Retrieved narinfo from S3:1367 StorePath: /build/TestNARDeduplicationMetadataUploadBug3986758916/001/store/jlnjxw9pmi5ydf96as6dwdwcx7r0ih6x-file2.txt1368 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1369 Compression: zstd1370 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1371 NarSize: 1601372 References: 1373 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1374 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1375 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1376 {"version":1,"root":{"type":"regular","size":44}}1377--- PASS: TestNARDeduplicationMetadataUploadBug (0.90s)1378=== CONT TestClientWithDependencies13792026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=013802026/11/11 23:58:18 INFO Received uploads request method=POST path=/api/pending_closures13812026/11/11 23:58:18 WARN Force mode enabled - objects will be deleted immediately without grace period13822026/11/11 23:58:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013832026/11/11 23:58:18 INFO Vacuumed table table=pending_closures13842026/11/11 23:58:18 INFO Vacuumed table table=pending_objects13852026/11/11 23:58:18 INFO Vacuumed table table=multipart_uploads13862026/11/11 23:58:18 INFO Vacuumed table table=closures13872026/11/11 23:58:18 INFO Vacuumed table table=objects13882026/11/11 23:58:18 INFO Aborted multipart uploads count=0 kept=013892026/11/11 23:58:18 WARN Force mode enabled - objects will be deleted immediately without grace period13902026/11/11 23:58:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=013912026/11/11 23:58:19 INFO Vacuumed table table=pending_closures13922026/11/11 23:58:19 INFO Vacuumed table table=pending_objects13932026/11/11 23:58:19 INFO Vacuumed table table=multipart_uploads13942026/11/11 23:58:19 INFO Vacuumed table table=closures13952026/11/11 23:58:19 INFO Vacuumed table table=objects1396--- PASS: TestSweepSkipsRowsLockedByPush (0.61s)1397=== CONT TestCacheConfigHandler1398=== RUN TestCacheConfigHandler/full_config,_no_issuer1399=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1400=== RUN TestCacheConfigHandler/no_cache_url_configured1401=== PAUSE TestCacheConfigHandler/no_cache_url_configured1402=== RUN TestCacheConfigHandler/no_signing_keys1403=== PAUSE TestCacheConfigHandler/no_signing_keys1404=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1405=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1406=== CONT TestClientMultipleUploads14072026/11/11 23:58:19 WARN readiness check failed error="closed pool"1408--- PASS: TestService_readinessHandler (0.53s)1409=== CONT TestClientIntegration14102026/11/11 23:58:19 INFO Aborted multipart uploads count=0 kept=014112026/11/11 23:58:19 WARN Force mode enabled - objects will be deleted immediately without grace period1412--- PASS: TestService_healthCheckHandler (0.51s)1413=== CONT TestClientErrorHandling1414=== RUN TestClientErrorHandling/InvalidStorePath14152026/11/11 23:58:19 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."1416=== PAUSE TestClientErrorHandling/InvalidStorePath14172026/11/11 23:58:19 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1418=== RUN TestClientErrorHandling/InvalidAuthToken1419=== PAUSE TestClientErrorHandling/InvalidAuthToken1420=== RUN TestClientErrorHandling/ServerNotAvailable1421=== PAUSE TestClientErrorHandling/ServerNotAvailable1422=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1423--- PASS: TestFailedS3DeleteKeepsTombstone (0.62s)1424=== CONT TestService_ReadAuthMiddleware14252026/11/11 23:58:19 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14262026/11/11 23:58:19 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=NDAwODgwZjktMGFhNi00NTlmLWIyZWMtZjY3ZGY0NmViODZmLmU4YTVlOTc3LWY0MDYtNDBmMi05YTI5LWRmODljMmE5OTk2MngxNzk0NDQxNDk3OTc5MjY5Mzgz parts=121427--- PASS: TestSharedNarBothUploadsComplete (2.07s)1428=== CONT TestCacheStatsHandler1429--- PASS: TestPushWaitsForSweepHoldingRow (1.09s)1430=== CONT TestService_ReadScope_PublicByDefault1431=== NAME TestClientCADerivations1432 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1699515404/001/store/z9a2b6ipswr4cnz7vbj5fznqm7zzlbil-ca-test14332026/11/11 23:58:19 INFO Received uploads request method=POST path=/api/pending_closures1434--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.75s)1435=== CONT TestService_RequireScope_OIDC1436=== NAME TestClientCADerivations1437 client_ca_test.go:139: Found 1 dependencies (including self)14382026/11/11 23:58:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39169/oidc1439=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1440=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1441=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1442=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1443=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1444=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1445=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1446=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1447=== CONT TestReadProxyDisabled14482026/11/11 23:58:19 INFO Received uploads request method=POST path=/api/pending_closures14492026/11/11 23:58:19 INFO Received uploads request method=POST path=/api/pending_closures14502026/11/11 23:58:19 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)14512026/11/11 23:58:19 INFO Uploading mgz9dyacscwm1x44ys0hcr8n9k4n3cpm-a (216B)14522026/11/11 23:58:19 INFO Uploading pil5gvkkr021kkxxfy3q98si6iw9nly6-shared-dep (136B)14532026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/0vw3gxld2rdivfak6gdl72188lfvgw884pmapcrqki25ihzald1y.nar.zst error="server returned 404: 404 page not found\n"14542026/11/11 23:58:19 WARN Failed to register uploaded object key=3l6z6nh75ca0jpb0aq164sipi3i14ps9.ls error="server returned 404: 404 page not found\n"14552026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14562026/11/11 23:58:19 WARN Failed to register uploaded object key=mgz9dyacscwm1x44ys0hcr8n9k4n3cpm.ls error="server returned 404: 404 page not found\n"14572026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign14582026/11/11 23:58:19 WARN Failed to register uploaded object key=pil5gvkkr021kkxxfy3q98si6iw9nly6.ls error="server returned 404: 404 page not found\n"14592026/11/11 23:58:19 INFO Signed narinfos id=1 count=214602026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign14612026/11/11 23:58:19 INFO Signed narinfos id=2 count=214622026/11/11 23:58:19 INFO Uploading 4 narinfos1463=== NAME TestPinProtectsFromGC1464 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC3849869425/001/store/k7h182f4fl39nz4iifbf9w2bsrr9c5lj-pinned-file.txt1465 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC3849869425/001/store/5lli9jbv8fnw05ybn9df3n7bv427nn42-unpinned-file.txt14662026/11/11 23:58:19 WARN Failed to register uploaded object key=3l6z6nh75ca0jpb0aq164sipi3i14ps9.narinfo error="server returned 404: 404 page not found\n"14672026/11/11 23:58:19 WARN Failed to register uploaded object key=mgz9dyacscwm1x44ys0hcr8n9k4n3cpm.narinfo error="server returned 404: 404 page not found\n"14682026/11/11 23:58:19 WARN Failed to register uploaded object key=pil5gvkkr021kkxxfy3q98si6iw9nly6.narinfo error="server returned 404: 404 page not found\n"14692026/11/11 23:58:19 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete14702026/11/11 23:58:19 WARN Failed to register uploaded object key=pil5gvkkr021kkxxfy3q98si6iw9nly6.narinfo error="server returned 404: 404 page not found\n"14712026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes14722026/11/11 23:58:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14732026/11/11 23:58:19 INFO Uploading z9a2b6ipswr4cnz7vbj5fznqm7zzlbil-ca-test (144B)14742026/11/11 23:58:19 INFO Completed upload id=1 objects=6 duration=7.695486ms14752026/11/11 23:58:19 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete14762026/11/11 23:58:19 INFO Completed upload id=2 objects=6 duration=2.278429ms14772026/11/11 23:58:19 INFO Upload complete. (70ms)1478=== NAME TestClientFallsBackToClosures1479 client_pushes_test.go:112: Retrieved narinfo from S3:1480 StorePath: /build/TestClientFallsBackToClosures444634957/001/store/pil5gvkkr021kkxxfy3q98si6iw9nly6-shared-dep1481 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1482 Compression: zstd1483 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y8214842026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"1485 NarSize: 1361486 References: 1487 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n14882026/11/11 23:58:19 WARN Failed to register uploaded object key=z9a2b6ipswr4cnz7vbj5fznqm7zzlbil.ls error="server returned 404: 404 page not found\n"14892026/11/11 23:58:19 WARN Failed to register uploaded object key=log/960yllg0dad12hcg9dg9ms7r51a6nhkv-ca-test.drv error="server returned 404: 404 page not found\n"14902026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14912026/11/11 23:58:19 INFO Signed narinfos id=1 count=114922026/11/11 23:58:19 INFO Uploading 1 narinfos1493 client_pushes_test.go:112: Retrieved narinfo from S3:14942026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes1495 StorePath: /build/TestClientFallsBackToClosures444634957/001/store/mgz9dyacscwm1x44ys0hcr8n9k4n3cpm-a1496 URL: nar/0vw3gxld2rdivfak6gdl72188lfvgw884pmapcrqki25ihzald1y.nar.zst1497 Compression: zstd1498 NarHash: sha256:0vw3gxld2rdivfak6gdl72188lfvgw884pmapcrqki25ihzald1y1499 NarSize: 2161500 References: /build/TestClientFallsBackToClosures444634957/001/store/pil5gvkkr021kkxxfy3q98si6iw9nly6-shared-dep1501 CA: text:sha256:0i9r4wv7c1ylpfak7bldiw8vlb3979384jmkkl4fkm9z76aqk5h81502 client_pushes_test.go:112: Retrieved narinfo from S3:1503 StorePath: /build/TestClientFallsBackToClosures444634957/001/store/3l6z6nh75ca0jpb0aq164sipi3i14ps9-b1504 URL: nar/0vw3gxld2rdivfak6gdl72188lfvgw884pmapcrqki25ihzald1y.nar.zst1505 Compression: zstd1506 NarHash: sha256:0vw3gxld2rdivfak6gdl72188lfvgw884pmapcrqki25ihzald1y1507 NarSize: 2161508 References: /build/TestClientFallsBackToClosures444634957/001/store/pil5gvkkr021kkxxfy3q98si6iw9nly6-shared-dep1509 CA: text:sha256:0i9r4wv7c1ylpfak7bldiw8vlb3979384jmkkl4fkm9z76aqk5h815102026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete15112026/11/11 23:58:19 WARN Failed to register uploaded object key=z9a2b6ipswr4cnz7vbj5fznqm7zzlbil.narinfo error="server returned 404: 404 page not found\n"15122026/11/11 23:58:19 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)15132026/11/11 23:58:19 INFO Uploading 4f5qjs3y9l40rjmya2mscx97f6nahbqb-a (216B)15142026/11/11 23:58:19 INFO Uploading 5xyszxmian6j0cv4hjidq7b1995slq40-shared-dep (136B)15152026/11/11 23:58:19 INFO Completed push id=1 objects=4 duration=4.632177ms15162026/11/11 23:58:19 INFO Upload complete. (95ms)1517--- PASS: TestClientFallsBackToClosures (0.69s)1518=== CONT TestService_AuthMiddleware_MTLSProxyHeader15192026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15202026/11/11 23:58:19 WARN Failed to register uploaded object key=4f5qjs3y9l40rjmya2mscx97f6nahbqb.ls error="server returned 404: 404 page not found\n"15212026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/0g1rkbr38blqm9a7fmikbbvxgyzy858fi5j14aw4c1qlv1qjiznl.nar.zst error="server returned 404: 404 page not found\n"15222026/11/11 23:58:19 WARN Failed to register uploaded object key=8yjziz779invksm8am7w92777frrkx5m.ls error="server returned 404: 404 page not found\n"1523=== NAME TestClientCADerivations15242026/11/11 23:58:19 WARN Failed to register uploaded object key=5xyszxmian6j0cv4hjidq7b1995slq40.ls error="server returned 404: 404 page not found\n"1525 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1699515404/001/store/z9a2b6ipswr4cnz7vbj5fznqm7zzlbil-ca-test15262026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1527 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1528 Compression: zstd1529 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1530 NarSize: 1441531 References: 1532 Deriver: /build/TestClientCADerivations1699515404/001/store/960yllg0dad12hcg9dg9ms7r51a6nhkv-ca-test.drv1533 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1534 client_ca_test.go:185: Checking for realisation files in S3...15352026/11/11 23:58:19 INFO Signed narinfos id=1 count=315362026/11/11 23:58:19 INFO Uploading 3 narinfos1537 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1538 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache15392026/11/11 23:58:19 WARN Failed to register uploaded object key=8yjziz779invksm8am7w92777frrkx5m.narinfo error="server returned 404: 404 page not found\n"15402026/11/11 23:58:19 WARN Failed to register uploaded object key=5xyszxmian6j0cv4hjidq7b1995slq40.narinfo error="server returned 404: 404 page not found\n"15412026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete15422026/11/11 23:58:19 WARN Failed to register uploaded object key=4f5qjs3y9l40rjmya2mscx97f6nahbqb.narinfo error="server returned 404: 404 page not found\n"15432026/11/11 23:58:19 INFO Completed push id=1 objects=8 duration=9.651314ms15442026/11/11 23:58:19 INFO Upload complete. (66ms)1545=== NAME TestClientPushesUseOnePush1546 client_pushes_test.go:97: Retrieved narinfo from S3:1547 StorePath: /build/TestClientPushesUseOnePush2837005656/001/store/5xyszxmian6j0cv4hjidq7b1995slq40-shared-dep1548 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1549 Compression: zstd1550 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821551 NarSize: 1361552 References: 1553 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1554=== NAME TestClientIntegration1555 client_integration_test.go:292: Created store path: /build/TestClientIntegration4220651982/002/store/fr60hlhcqwpcdf2d6ska624m8i5csq1i-test-file.txt1556=== NAME TestClientPushesUseOnePush1557 client_pushes_test.go:97: Retrieved narinfo from S3:1558 StorePath: /build/TestClientPushesUseOnePush2837005656/001/store/4f5qjs3y9l40rjmya2mscx97f6nahbqb-a1559 URL: nar/0g1rkbr38blqm9a7fmikbbvxgyzy858fi5j14aw4c1qlv1qjiznl.nar.zst1560 Compression: zstd1561 NarHash: sha256:0g1rkbr38blqm9a7fmikbbvxgyzy858fi5j14aw4c1qlv1qjiznl1562 NarSize: 2161563 References: /build/TestClientPushesUseOnePush2837005656/001/store/5xyszxmian6j0cv4hjidq7b1995slq40-shared-dep1564 CA: text:sha256:1v3m443v7gypmss5wmjl6ajypl7w0a9lpw7x9i71zzs76ismicj81565 client_pushes_test.go:97: Retrieved narinfo from S3:1566 StorePath: /build/TestClientPushesUseOnePush2837005656/001/store/8yjziz779invksm8am7w92777frrkx5m-b1567 URL: nar/0g1rkbr38blqm9a7fmikbbvxgyzy858fi5j14aw4c1qlv1qjiznl.nar.zst1568 Compression: zstd1569 NarHash: sha256:0g1rkbr38blqm9a7fmikbbvxgyzy858fi5j14aw4c1qlv1qjiznl1570 NarSize: 2161571 References: /build/TestClientPushesUseOnePush2837005656/001/store/5xyszxmian6j0cv4hjidq7b1995slq40-shared-dep1572 CA: text:sha256:1v3m443v7gypmss5wmjl6ajypl7w0a9lpw7x9i71zzs76ismicj81573--- PASS: TestService_ReadAuthMiddleware (0.37s)1574=== CONT TestPush_CompleteCommitsEveryRoot1575=== CONT TestService_AuthMiddleware1576--- PASS: TestClientPushesUseOnePush (0.64s)1577--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.65s)1578=== CONT TestReadProxyRangeRequest15792026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes15802026/11/11 23:58:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15812026/11/11 23:58:19 INFO Uploading k7h182f4fl39nz4iifbf9w2bsrr9c5lj-pinned-file.txt (128B)1582=== NAME TestClientWithDependencies1583 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies185474448/001/store/rm53lr4476jzf6imrf4pzlckzsv1xlr3-test-script15842026/11/11 23:58:19 WARN Failed to register uploaded object key=k7h182f4fl39nz4iifbf9w2bsrr9c5lj.ls error="server returned 404: 404 page not found\n"15852026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15862026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"15872026/11/11 23:58:19 INFO Signed narinfos id=1 count=115882026/11/11 23:58:19 INFO Uploading 1 narinfos1589=== NAME TestClientMultipleUploads1590 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1552391571/001/store/0j8v9a1sw4dv2gfp3zmm7712k041davr-test-file-0.txt15912026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes15922026/11/11 23:58:19 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"15932026/11/11 23:58:19 WARN mTLS auth: bound subjects configured but subject DN unavailable15942026/11/11 23:58:19 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1595--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.40s)1596=== CONT TestGCSweepDeliversEachKeyOnce15972026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete15982026/11/11 23:58:19 WARN Failed to register uploaded object key=k7h182f4fl39nz4iifbf9w2bsrr9c5lj.narinfo error="server returned 404: 404 page not found\n"15992026/11/11 23:58:19 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)16002026/11/11 23:58:19 INFO Uploading h9v27r0hhsihx3cfyhja15b29d68jpc9-shared-dep (136B)16012026/11/11 23:58:19 INFO Uploading ympyych82rz73gzrz2c3ak9gnfzrdi78-top (224B)16022026/11/11 23:58:19 INFO Completed push id=1 objects=3 duration=6.374706ms16032026/11/11 23:58:19 INFO Upload complete. (64ms)16042026/11/11 23:58:19 WARN Failed to register uploaded object key=h9v27r0hhsihx3cfyhja15b29d68jpc9.ls error="server returned 404: 404 page not found\n"16052026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"16062026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/1y7fda0f783ry9i9had7f5dm2ssr118g7yfsv9x0s63jjw63pjx4.nar.zst error="server returned 404: 404 page not found\n"16072026/11/11 23:58:19 WARN Failed to register uploaded object key=ympyych82rz73gzrz2c3ak9gnfzrdi78.ls error="server returned 404: 404 page not found\n"16082026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16092026/11/11 23:58:19 INFO Signed narinfos id=1 count=216102026/11/11 23:58:19 INFO Uploading 2 narinfos16112026/11/11 23:58:19 WARN Failed to register uploaded object key=h9v27r0hhsihx3cfyhja15b29d68jpc9.narinfo error="server returned 404: 404 page not found\n"16122026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete16132026/11/11 23:58:19 WARN Failed to register uploaded object key=ympyych82rz73gzrz2c3ak9gnfzrdi78.narinfo error="server returned 404: 404 page not found\n"1614=== NAME TestClientWithDependencies1615 client_integration_test.go:663: Found 1 dependencies (including self)16162026/11/11 23:58:19 INFO Completed push id=1 objects=6 duration=5.823286ms16172026/11/11 23:58:19 INFO Upload complete. (63ms)1618=== NAME TestClientSharedPathCommittedMidPush1619 client_integration_test.go:728: Retrieved narinfo from S3:1620 StorePath: /build/TestClientSharedPathCommittedMidPush1056297480/001/store/h9v27r0hhsihx3cfyhja15b29d68jpc9-shared-dep1621 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1622 Compression: zstd1623 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821624 NarSize: 1361625 References: 1626 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1627 client_integration_test.go:728: Retrieved narinfo from S3:1628 StorePath: /build/TestClientSharedPathCommittedMidPush1056297480/001/store/ympyych82rz73gzrz2c3ak9gnfzrdi78-top1629 URL: nar/1y7fda0f783ry9i9had7f5dm2ssr118g7yfsv9x0s63jjw63pjx4.nar.zst1630 Compression: zstd1631 NarHash: sha256:1y7fda0f783ry9i9had7f5dm2ssr118g7yfsv9x0s63jjw63pjx41632 NarSize: 2241633 References: /build/TestClientSharedPathCommittedMidPush1056297480/001/store/h9v27r0hhsihx3cfyhja15b29d68jpc9-shared-dep1634 CA: text:sha256:0svy8v6kwim1833sj47xm43kzhx2rns9hlj5vswhn9iadgvzxn421635=== NAME TestClientMultipleUploads1636 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1552391571/001/store/8vbqcpgqf2y957xci99y9p4m47f447jk-test-file-1.txt16372026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes1638--- PASS: TestClientSharedPathCommittedMidPush (0.62s)1639=== CONT TestPush_OverlappingRootsStoreOneRowPerKey16402026/11/11 23:58:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16412026/11/11 23:58:19 INFO Uploading fr60hlhcqwpcdf2d6ska624m8i5csq1i-test-file.txt (152B)1642--- PASS: TestService_ReadScope_PublicByDefault (0.35s)1643=== CONT TestReadRedirectKeepsNarinfoProxied16442026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"16452026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16462026/11/11 23:58:19 WARN Failed to register uploaded object key=fr60hlhcqwpcdf2d6ska624m8i5csq1i.ls error="server returned 404: 404 page not found\n"16472026/11/11 23:58:19 INFO Signed narinfos id=1 count=116482026/11/11 23:58:19 INFO Uploading 1 narinfos1649--- PASS: TestCacheStatsHandler (0.38s)1650=== CONT TestReadRedirectUsesPublicS3URL16512026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete16522026/11/11 23:58:19 WARN Failed to register uploaded object key=fr60hlhcqwpcdf2d6ska624m8i5csq1i.narinfo error="server returned 404: 404 page not found\n"1653=== NAME TestClientMultipleUploads1654 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1552391571/001/store/cx2zwj9haj3jn456756j5d4sjyz0gi74-test-file-2.txt16552026/11/11 23:58:19 INFO Completed push id=1 objects=3 duration=10.211695ms16562026/11/11 23:58:19 INFO Upload complete. (71ms)16572026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes16582026/11/11 23:58:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16592026/11/11 23:58:19 INFO Uploading 5lli9jbv8fnw05ybn9df3n7bv427nn42-unpinned-file.txt (128B)1660=== NAME TestClientCADerivations1661 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1662 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1663 error: binary cache 's3://bucket62?endpoint=http://localhost:33047®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations1699515404/001/store'1664 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11665--- PASS: TestClientCADerivations (0.85s)1666=== CONT TestReadProxyHead16672026/11/11 23:58:19 WARN Failed to register uploaded object key=5lli9jbv8fnw05ybn9df3n7bv427nn42.ls error="server returned 404: 404 page not found\n"16682026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign16692026/11/11 23:58:19 INFO Signed narinfos id=2 count=116702026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"16712026/11/11 23:58:19 INFO Uploading 1 narinfos16722026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes16732026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/2/complete16742026/11/11 23:58:19 WARN Failed to register uploaded object key=5lli9jbv8fnw05ybn9df3n7bv427nn42.narinfo error="server returned 404: 404 page not found\n"16752026/11/11 23:58:19 INFO Completed push id=2 objects=3 duration=7.145996ms16762026/11/11 23:58:19 INFO Upload complete. (57ms)16772026/11/11 23:58:19 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16782026/11/11 23:58:19 INFO Uploading rm53lr4476jzf6imrf4pzlckzsv1xlr3-test-script (136B)1679=== RUN TestService_RequireScope_OIDC/builder_may_write1680=== PAUSE TestService_RequireScope_OIDC/builder_may_write1681=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1682=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1683=== RUN TestService_RequireScope_OIDC/ops_may_admin1684=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1685=== RUN TestService_RequireScope_OIDC/ops_may_not_write1686=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1687=== RUN TestService_RequireScope_OIDC/reader_may_not_write1688=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1689=== RUN TestService_RequireScope_OIDC/static_token_may_admin1690=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1691=== RUN TestService_RequireScope_OIDC/static_token_may_write1692=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1693=== CONT TestReadRedirectNar1694--- PASS: TestReadProxyDisabled (0.23s)1695=== RUN TestService_RequireScope_OIDC/reader_may_read1696=== PAUSE TestService_RequireScope_OIDC/reader_may_read1697=== RUN TestService_RequireScope_OIDC/writer_implies_read1698=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1699=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1700=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1701=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1702=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1703=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1704=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1705=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1706=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1707=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1708=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1709=== CONT TestReadProxyRootRedirectsToIndexHTML17102026/11/11 23:58:19 WARN Failed to register uploaded object key=rm53lr4476jzf6imrf4pzlckzsv1xlr3.ls error="server returned 404: 404 page not found\n"17112026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"17122026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17132026/11/11 23:58:19 WARN Failed to register uploaded object key=log/lvqwrsib07xjjcyrx6zs87lgxggkcc79-test-script.drv error="server returned 404: 404 page not found\n"17142026/11/11 23:58:19 INFO Signed narinfos id=1 count=117152026/11/11 23:58:19 INFO Uploading 1 narinfos17162026/11/11 23:58:19 INFO All 1 paths already cached1717=== NAME TestClientIntegration1718 client_integration_test.go:318: Retrieved narinfo from S3:1719 StorePath: /build/TestClientIntegration4220651982/002/store/fr60hlhcqwpcdf2d6ska624m8i5csq1i-test-file.txt1720 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1721 Compression: zstd1722 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11723 NarSize: 1521724 References: 1725 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11726 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1727 client_integration_test.go:319: Decompressed .ls content (64 bytes):17282026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete1729 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}17302026/11/11 23:58:19 WARN Failed to register uploaded object key=rm53lr4476jzf6imrf4pzlckzsv1xlr3.narinfo error="server returned 404: 404 page not found\n"17312026/11/11 23:58:19 INFO Completed push id=1 objects=4 duration=10.274244ms17322026/11/11 23:58:19 INFO Upload complete. (66ms)1733=== NAME TestClientWithDependencies1734 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies185474448/001/store) requires matching store prefix1735--- PASS: TestClientWithDependencies (0.66s)1736=== CONT TestReadProxyConditionalGet1737--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.19s)1738=== CONT TestReadProxy40417392026/11/11 23:58:19 INFO Starting cleanup of old closures method=DELETE path=/api/closures17402026/11/11 23:58:19 INFO Garbage collection started17412026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes17422026/11/11 23:58:19 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)17432026/11/11 23:58:19 INFO Uploading 8vbqcpgqf2y957xci99y9p4m47f447jk-test-file-1.txt (160B)17442026/11/11 23:58:19 INFO Uploading cx2zwj9haj3jn456756j5d4sjyz0gi74-test-file-2.txt (160B)17452026/11/11 23:58:19 INFO Uploading 0j8v9a1sw4dv2gfp3zmm7712k041davr-test-file-0.txt (160B)17462026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"17472026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"17482026/11/11 23:58:19 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17492026/11/11 23:58:19 WARN Failed to register uploaded object key=0j8v9a1sw4dv2gfp3zmm7712k041davr.ls error="server returned 404: 404 page not found\n"17502026/11/11 23:58:19 WARN Failed to register uploaded object key=8vbqcpgqf2y957xci99y9p4m47f447jk.ls error="server returned 404: 404 page not found\n"17512026/11/11 23:58:19 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"17522026/11/11 23:58:19 WARN Failed to register uploaded object key=cx2zwj9haj3jn456756j5d4sjyz0gi74.ls error="server returned 404: 404 page not found\n"17532026/11/11 23:58:19 INFO Signed narinfos id=1 count=317542026/11/11 23:58:19 INFO Uploading 3 narinfos17552026/11/11 23:58:19 INFO Aborted multipart uploads count=0 kept=017562026/11/11 23:58:19 WARN Failed to register uploaded object key=cx2zwj9haj3jn456756j5d4sjyz0gi74.narinfo error="server returned 404: 404 page not found\n"17572026/11/11 23:58:19 WARN Failed to register uploaded object key=0j8v9a1sw4dv2gfp3zmm7712k041davr.narinfo error="server returned 404: 404 page not found\n"17582026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete17592026/11/11 23:58:19 WARN Failed to register uploaded object key=8vbqcpgqf2y957xci99y9p4m47f447jk.narinfo error="server returned 404: 404 page not found\n"17602026/11/11 23:58:19 WARN Force mode enabled - objects will be deleted immediately without grace period17612026/11/11 23:58:19 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1762--- PASS: TestService_AuthMiddleware (0.19s)1763=== CONT TestReadProxyOutlastsServerWriteTimeout17642026/11/11 23:58:19 INFO Completed push id=1 objects=9 duration=16.245031ms17652026/11/11 23:58:19 INFO Upload complete. (80ms)1766=== NAME TestClientMultipleUploads1767 client_integration_test.go:417: Uploaded 3 paths in 114.316034ms1768--- PASS: TestClientMultipleUploads (0.63s)1769=== CONT TestReadProxyInvalidPath17702026/11/11 23:58:19 INFO lead: released remote=192.0.2.1:1234 generation=11771--- PASS: TestLeadEndsOnShutdown (1.69s)1772=== CONT TestReadProxyNarStreaming17732026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes17742026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes17752026/11/11 23:58:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=017762026/11/11 23:58:19 INFO Object in database but missing from S3 key=fr60hlhcqwpcdf2d6ska624m8i5csq1i.ls17772026/11/11 23:58:19 WARN Found objects in DB but missing from S3, will re-upload count=117782026/11/11 23:58:19 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17792026/11/11 23:58:19 INFO Vacuumed table table=pending_closures17802026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/2/complete17812026/11/11 23:58:19 WARN Failed to register uploaded object key=fr60hlhcqwpcdf2d6ska624m8i5csq1i.ls error="server returned 404: 404 page not found\n"17822026/11/11 23:58:19 INFO Completed push id=2 objects=3 duration=2.938659ms17832026/11/11 23:58:19 INFO Upload complete. (65ms)17842026/11/11 23:58:19 INFO Vacuumed table table=pending_objects17852026/11/11 23:58:19 INFO Vacuumed table table=multipart_uploads17862026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/1/complete1787=== NAME TestClientIntegration1788 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1789 client_integration_test.go:336: Decompressed .ls content (49 bytes):1790 {"version":1,"root":{"type":"regular","size":39}}17912026/11/11 23:58:19 INFO Vacuumed table table=closures1792=== NAME TestOrphanedObjectsGCStressTest1793 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains17942026/11/11 23:58:19 INFO Vacuumed table table=objects17952026/11/11 23:58:19 INFO Completed push id=1 objects=6 duration=8.027945ms1796--- PASS: TestPush_CompleteCommitsEveryRoot (0.27s)1797=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1798--- PASS: TestReadProxyRangeRequest (0.27s)1799=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1800=== NAME TestOrphanedObjectsGCStressTest1801 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion18022026/11/11 23:58:19 INFO Aborted multipart uploads count=0 kept=018032026/11/11 23:58:19 WARN Force mode enabled - objects will be deleted immediately without grace period1804--- PASS: TestReadRedirectKeepsNarinfoProxied (0.27s)1805=== CONT TestPendingClosureWriteTimeout/empty1806=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline18072026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes18082026/11/11 23:58:19 INFO Received push request method=POST path=/api/pushes18092026/11/11 23:58:19 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18102026/11/11 23:58:19 INFO Received complete push request method=POST path=/api/pushes/3/complete18112026/11/11 23:58:19 WARN Failed to register uploaded object key=fr60hlhcqwpcdf2d6ska624m8i5csq1i.ls error="server returned 404: 404 page not found\n"18122026/11/11 23:58:19 INFO Completed push id=3 objects=3 duration=2.154148ms18132026/11/11 23:58:19 INFO Upload complete. (56ms)1814=== NAME TestClientIntegration1815 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1816 client_integration_test.go:353: Decompressed .ls content (49 bytes):1817 {"version":1,"root":{"type":"regular","size":39}}1818 client_integration_test.go:364: Testing garbage collection...1819--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.32s)1820=== CONT TestPendingClosureWriteTimeout/400_objects1821=== CONT TestPendingClosureWriteTimeout/670k_objects1822=== CONT TestProxyWriteTimeout/narinfo1823=== CONT TestProxyWriteTimeout/10_GiB_nar1824=== CONT TestProxyWriteTimeout/1_GiB_nar1825=== CONT TestProxyWriteTimeout/unknown_size1826--- PASS: TestProxyWriteTimeout (0.09s)1827 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1828 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1829 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1830 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1831=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info18322026/11/11 23:58:19 INFO Received uploads request method=POST path=/1833=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key18342026/11/11 23:58:19 INFO Received request for more parts method=POST path=/1835=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key18362026/11/11 23:58:19 INFO Received complete multipart upload request method=POST path=/1837=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18382026/11/11 23:58:19 INFO Received uploads request method=POST path=/1839--- PASS: TestUploadHandlersRejectInvalidKeys (0.09s)1840 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1841 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1842 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1843 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1844=== CONT TestIsValidCachePath/narinfo1845=== CONT TestIsValidCachePath/short_hash1846=== CONT TestIsValidCachePath/wrong_extension1847=== CONT TestIsValidCachePath/leading_slash1848=== CONT TestIsValidCachePath/empty1849=== CONT TestIsValidCachePath/random_path1850=== CONT TestIsValidCachePath/invalid_char_u1851=== CONT TestIsValidCachePath/invalid_char_e1852=== CONT TestIsValidCachePath/traversal_in_middle1853=== CONT TestIsValidCachePath/traversal_parent1854=== CONT TestIsValidCachePath/index.html1855=== CONT TestIsValidCachePath/nix-cache-info1856=== CONT TestIsValidCachePath/realisation1857=== CONT TestIsValidCachePath/log1858=== CONT TestIsValidCachePath/ls1859=== CONT TestIsValidCachePath/nar_uncompressed1860=== CONT TestIsValidCachePath/nar_bz21861=== CONT TestIsValidCachePath/nar_xz1862=== CONT TestIsValidCachePath/nar_zst1863=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1864--- PASS: TestIsValidCachePath (0.09s)1865 --- PASS: TestIsValidCachePath/narinfo (0.00s)1866 --- PASS: TestIsValidCachePath/short_hash (0.00s)1867 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1868 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1869 --- PASS: TestIsValidCachePath/empty (0.00s)1870 --- PASS: TestIsValidCachePath/random_path (0.00s)1871 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1872 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1873 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1874 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1875 --- PASS: TestIsValidCachePath/index.html (0.00s)1876 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1877 --- PASS: TestIsValidCachePath/realisation (0.00s)1878 --- PASS: TestIsValidCachePath/log (0.00s)1879 --- PASS: TestIsValidCachePath/ls (0.00s)1880 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1881 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1882 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1883 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1884 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1885=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed18862026/11/11 23:58:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=018872026/11/11 23:58:19 INFO Vacuumed table table=pending_closures18882026/11/11 23:58:19 INFO Vacuumed table table=pending_objects18892026/11/11 23:58:19 INFO Vacuumed table table=multipart_uploads18902026/11/11 23:58:19 INFO Vacuumed table table=closures18912026/11/11 23:58:19 INFO lead: released remote=192.0.2.1:1234 generation=118922026/11/11 23:58:19 INFO lead: acquired remote=192.0.2.1:1234 generation=218932026/11/11 23:58:19 INFO Vacuumed table table=objects1894--- PASS: TestGCSweepDeliversEachKeyOnce (0.37s)1895=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1896--- PASS: TestReadRedirectUsesPublicS3URL (0.33s)1897=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed18982026/11/11 23:58:19 INFO Starting cleanup of old closures method=DELETE path=/api/closures18992026/11/11 23:58:19 INFO Garbage collection started19002026/11/11 23:58:19 INFO Aborted multipart uploads count=0 kept=01901--- PASS: TestReadProxyHead (0.32s)1902=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost19032026/11/11 23:58:19 WARN Force mode enabled - objects will be deleted immediately without grace period19042026/11/11 23:58:19 INFO lead: released remote=192.0.2.1:1234 generation=11905--- PASS: TestReadRedirectNar (0.33s)1906=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed19072026/11/11 23:58:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01908--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.35s)1909=== CONT TestParseSingleRange/none1910=== CONT TestParseSingleRange/start_far_past_EOF1911=== CONT TestParseSingleRange/start_past_EOF1912=== CONT TestParseSingleRange/single_byte1913=== CONT TestParseSingleRange/suffix_exceeds_size1914=== CONT TestParseSingleRange/suffix1915=== CONT TestParseSingleRange/malformed_both_empty1916=== CONT TestParseSingleRange/malformed_no_dash1917=== CONT TestParseSingleRange/multi-range_ignored1918=== CONT TestParseSingleRange/unknown_unit1919=== CONT TestParseSingleRange/open-ended1920=== CONT TestParseSingleRange/end_clamped_to_size1921=== CONT TestParseSingleRange/closed1922=== CONT TestParseSingleRange/malformed_end_before_start1923--- PASS: TestParseSingleRange (0.09s)1924 --- PASS: TestParseSingleRange/none (0.00s)1925 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1926 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1927 --- PASS: TestParseSingleRange/single_byte (0.00s)1928 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1929 --- PASS: TestParseSingleRange/suffix (0.00s)1930 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1931 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1932 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1933 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1934 --- PASS: TestParseSingleRange/open-ended (0.00s)1935 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1936 --- PASS: TestParseSingleRange/closed (0.00s)1937 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1938=== CONT TestIsValidUploadKey/narinfo19392026/11/11 23:58:19 INFO Vacuumed table table=pending_closures1940=== CONT TestIsValidUploadKey/index.html1941=== CONT TestIsValidUploadKey/nix-cache-info1942=== CONT TestIsValidUploadKey/realisation_plus_in_output1943=== CONT TestIsValidUploadKey/realisation1944=== CONT TestIsValidUploadKey/build_log_equals1945=== CONT TestIsValidUploadKey/build_log_question_mark1946=== CONT TestIsValidUploadKey/build_log_plus_in_name1947=== CONT TestIsValidUploadKey/build_log_home-manager_file1948=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1949=== CONT TestIsValidUploadKey/build_log1950=== CONT TestIsValidUploadKey/listing1951=== CONT TestIsValidUploadKey/nar_plain1952=== CONT TestIsValidUploadKey/nar_xz1953=== CONT TestIsValidUploadKey/nar_zst1954=== CONT TestIsValidUploadKey/traversal_nar1955=== CONT TestIsValidUploadKey/unknown_type1956=== CONT TestIsValidUploadKey/empty_key1957=== CONT TestIsValidUploadKey/absolute1958=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1959=== CONT TestIsValidUploadKey/traversal1960=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1961--- PASS: TestIsValidUploadKey (0.10s)1962 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1963 --- PASS: TestIsValidUploadKey/index.html (0.00s)1964 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1965 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1966 --- PASS: TestIsValidUploadKey/realisation (0.00s)1967 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1968 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1969 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1970 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1971 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1972 --- PASS: TestIsValidUploadKey/build_log (0.00s)1973 --- PASS: TestIsValidUploadKey/listing (0.00s)1974 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1975 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1976 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1977 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1978 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1979 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1980 --- PASS: TestIsValidUploadKey/absolute (0.00s)1981 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1982 --- PASS: TestIsValidUploadKey/traversal (0.00s)1983 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1984=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19852026/11/11 23:58:19 INFO Received uploads request method=POST path=/19862026/11/11 23:58:19 INFO lead: acquired remote=192.0.2.1:1234 generation=219872026/11/11 23:58:19 INFO Vacuumed table table=pending_objects19882026/11/11 23:58:19 INFO Vacuumed table table=multipart_uploads19892026/11/11 23:58:19 INFO Vacuumed table table=closures19902026/11/11 23:58:19 INFO Vacuumed table table=objects19912026/11/11 23:58:19 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=21992--- PASS: TestReadProxyConditionalGet (0.39s)19932026/11/11 23:58:19 INFO lead: acquired remote=192.0.2.1:1234 generation=31994=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19952026/11/11 23:58:19 INFO Received request for more parts method=POST path=/1996--- PASS: TestReadProxy404 (0.41s)1997=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19982026/11/11 23:58:20 INFO Received complete multipart upload request method=POST path=/1999--- PASS: TestReadProxyInvalidPath (0.37s)2000=== CONT TestPush_RejectsBadRequests/no_objects
loading 688 hidden lines…
26892026/11/11 23:58:33 INFO Vacuumed table table=pending_closures26902026/11/11 23:58:33 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."26912026/11/11 23:58:33 INFO Received complete push request method=POST path=/26922026/11/11 23:58:33 INFO Completed upload id=5 objects=7 duration=3.156268ms26932026/11/11 23:58:33 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."26942026/11/11 23:58:33 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."26952026/11/11 23:58:33 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."26962026/11/11 23:58:33 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."26972026/11/11 23:58:33 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."26982026/11/11 23:58:33 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."26992026/11/11 23:58:33 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27002026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27012026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27022026/11/11 23:58:33 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."27032026/11/11 23:58:33 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."27042026/11/11 23:58:33 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."27052026/11/11 23:58:33 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27062026/11/11 23:58:33 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27072026/11/11 23:58:33 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27082026/11/11 23:58:33 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27092026/11/11 23:58:33 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."27102026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27112026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27122026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."27132026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."27142026/11/11 23:58:33 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27152026/11/11 23:58:33 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27162026/11/11 23:58:33 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27172026/11/11 23:58:33 INFO Vacuumed table table=pending_objects27182026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads27192026/11/11 23:58:33 INFO Completed push id=8 objects=13 duration=2.424399ms27202026/11/11 23:58:33 INFO Received uploads request method=POST path=/27212026/11/11 23:58:33 INFO Received push request method=POST path=/27222026/11/11 23:58:33 INFO Vacuumed table table=closures27232026/11/11 23:58:33 INFO Received push request method=POST path=/27242026/11/11 23:58:33 INFO Vacuumed table table=objects27252026/11/11 23:58:33 INFO Received push request method=POST path=/27262026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=027272026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period27282026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027292026/11/11 23:58:33 INFO Vacuumed table table=pending_closures27302026/11/11 23:58:33 INFO Vacuumed table table=pending_objects27312026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads27322026/11/11 23:58:33 INFO Vacuumed table table=closures27332026/11/11 23:58:33 INFO Vacuumed table table=objects27342026/11/11 23:58:33 INFO Received push request method=POST path=/27352026/11/11 23:58:33 INFO Received uploads request method=POST path=/27362026/11/11 23:58:33 INFO Received complete upload request method=POST path=/27372026/11/11 23:58:33 INFO Received push request method=POST path=/27382026/11/11 23:58:33 INFO Completed upload id=2 objects=10 duration=2.386539ms27392026/11/11 23:58:33 INFO Received complete upload request method=POST path=/27402026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=027412026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=027422026/11/11 23:58:33 INFO Received uploads request method=POST path=/27432026/11/11 23:58:33 INFO Completed upload id=6 objects=9 duration=1.557709ms27442026/11/11 23:58:33 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27452026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27462026/11/11 23:58:33 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27472026/11/11 23:58:33 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27482026/11/11 23:58:33 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27492026/11/11 23:58:33 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27502026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27512026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27522026/11/11 23:58:33 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."27532026/11/11 23:58:33 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27542026/11/11 23:58:33 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27552026/11/11 23:58:33 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27562026/11/11 23:58:33 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."27572026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."27582026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27592026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27602026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."27612026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27622026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."27632026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."27642026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period27652026/11/11 23:58:33 INFO Received complete upload request method=POST path=/27662026/11/11 23:58:33 INFO Received push request method=POST path=/27672026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027682026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027692026/11/11 23:58:33 INFO Completed upload id=6 objects=7 duration=2.909009ms27702026/11/11 23:58:33 INFO Vacuumed table table=pending_closures27712026/11/11 23:58:33 INFO Vacuumed table table=pending_closures27722026/11/11 23:58:33 INFO Received complete upload request method=POST path=/27732026/11/11 23:58:33 INFO Vacuumed table table=pending_objects27742026/11/11 23:58:33 INFO Received uploads request method=POST path=/27752026/11/11 23:58:33 INFO Vacuumed table table=pending_objects27762026/11/11 23:58:33 INFO Received push request method=POST path=/27772026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads27782026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads27792026/11/11 23:58:33 INFO Received push request method=POST path=/27802026/11/11 23:58:33 INFO Vacuumed table table=closures27812026/11/11 23:58:33 INFO Vacuumed table table=closures27822026/11/11 23:58:33 INFO Received push request method=POST path=/27832026/11/11 23:58:33 INFO Vacuumed table table=objects27842026/11/11 23:58:33 INFO Vacuumed table table=objects27852026/11/11 23:58:33 INFO Completed upload id=3 objects=6 duration=5.725276ms27862026/11/11 23:58:33 INFO Received push request method=POST path=/27872026/11/11 23:58:33 INFO Received complete upload request method=POST path=/27882026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=027892026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period27902026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027912026/11/11 23:58:33 INFO Vacuumed table table=pending_closures27922026/11/11 23:58:33 INFO Completed upload id=1 objects=15 duration=5.688686ms27932026/11/11 23:58:33 INFO Vacuumed table table=pending_objects27942026/11/11 23:58:33 INFO Received push request method=POST path=/27952026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads27962026/11/11 23:58:33 INFO Vacuumed table table=closures27972026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=027982026/11/11 23:58:33 INFO Vacuumed table table=objects27992026/11/11 23:58:33 INFO Received uploads request method=POST path=/28002026/11/11 23:58:33 INFO Received uploads request method=POST path=/28012026/11/11 23:58:33 INFO Received complete push request method=POST path=/28022026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=028032026/11/11 23:58:33 INFO Received uploads request method=POST path=/28042026/11/11 23:58:33 INFO Received complete upload request method=POST path=/28052026/11/11 23:58:33 INFO Received push request method=POST path=/28062026/11/11 23:58:33 INFO Completed upload id=9 objects=7 duration=2.483148ms28072026/11/11 23:58:33 INFO Received uploads request method=POST path=/28082026/11/11 23:58:33 INFO Received push request method=POST path=/28092026/11/11 23:58:33 INFO Completed push id=6 objects=5 duration=4.065527ms28102026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period28112026/11/11 23:58:33 INFO Received uploads request method=POST path=/28122026/11/11 23:58:33 INFO Received uploads request method=POST path=/28132026/11/11 23:58:33 INFO Received uploads request method=POST path=/28142026/11/11 23:58:33 INFO Received complete upload request method=POST path=/28152026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=028162026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period28172026/11/11 23:58:33 INFO Completed upload id=6 objects=24 duration=3.459308ms28182026/11/11 23:58:33 INFO Received push request method=POST path=/28192026/11/11 23:58:33 INFO Received push request method=POST path=/28202026/11/11 23:58:33 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."28212026/11/11 23:58:33 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28222026/11/11 23:58:33 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."28232026/11/11 23:58:33 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."28242026/11/11 23:58:33 INFO Received complete push request method=POST path=/28252026/11/11 23:58:33 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28262026/11/11 23:58:33 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."28272026/11/11 23:58:33 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."28282026/11/11 23:58:33 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."28292026/11/11 23:58:33 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."28302026/11/11 23:58:33 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."28312026/11/11 23:58:33 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."28322026/11/11 23:58:33 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."28332026/11/11 23:58:33 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."28342026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."28352026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28362026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."28372026/11/11 23:58:33 INFO Received uploads request method=POST path=/28382026/11/11 23:58:33 INFO Completed push id=3 objects=5 duration=2.495869ms28392026/11/11 23:58:33 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28402026/11/11 23:58:33 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28412026/11/11 23:58:33 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28422026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28432026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28442026/11/11 23:58:33 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28452026/11/11 23:58:33 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28462026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28472026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28482026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28492026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."28502026/11/11 23:58:33 INFO Received complete upload request method=POST path=/28512026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=028522026/11/11 23:58:33 INFO Completed upload id=3 objects=5 duration=1.287959ms28532026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028542026/11/11 23:58:33 INFO Vacuumed table table=pending_closures28552026/11/11 23:58:33 INFO Vacuumed table table=pending_objects28562026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads28572026/11/11 23:58:33 INFO Vacuumed table table=closures28582026/11/11 23:58:33 INFO Vacuumed table table=objects28592026/11/11 23:58:33 INFO Received complete upload request method=POST path=/28602026/11/11 23:58:33 INFO Received push request method=POST path=/28612026/11/11 23:58:33 INFO Received push request method=POST path=/28622026/11/11 23:58:33 INFO Completed upload id=10 objects=4 duration=3.076048ms28632026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=028642026/11/11 23:58:33 INFO Received uploads request method=POST path=/28652026/11/11 23:58:33 INFO Received push request method=POST path=/28662026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=028672026/11/11 23:58:33 INFO Vacuumed table table=pending_closures28682026/11/11 23:58:33 INFO Received push request method=POST path=/28692026/11/11 23:58:33 INFO Vacuumed table table=pending_objects28702026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads28712026/11/11 23:58:33 INFO Vacuumed table table=closures28722026/11/11 23:58:33 INFO Vacuumed table table=objects28732026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=028742026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=028752026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period28762026/11/11 23:58:33 INFO Received complete push request method=POST path=/28772026/11/11 23:58:33 INFO Completed push id=4 objects=4 duration=3.180528ms28782026/11/11 23:58:33 INFO Received push request method=POST path=/28792026/11/11 23:58:33 INFO Received uploads request method=POST path=/28802026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028812026/11/11 23:58:33 INFO Vacuumed table table=pending_closures28822026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=028832026/11/11 23:58:33 INFO Received push request method=POST path=/28842026/11/11 23:58:33 INFO Vacuumed table table=pending_objects28852026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads28862026/11/11 23:58:33 INFO Vacuumed table table=closures28872026/11/11 23:58:33 INFO Vacuumed table table=objects28882026/11/11 23:58:33 INFO Received complete push request method=POST path=/28892026/11/11 23:58:33 INFO Received push request method=POST path=/28902026/11/11 23:58:33 INFO Completed push id=8 objects=6 duration=2.421418ms28912026/11/11 23:58:33 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28922026/11/11 23:58:33 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28932026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28942026/11/11 23:58:33 INFO Received complete push request method=POST path=/28952026/11/11 23:58:33 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."28962026/11/11 23:58:33 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."28972026/11/11 23:58:33 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."28982026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28992026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29002026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29012026/11/11 23:58:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=029022026/11/11 23:58:33 INFO Completed push id=8 objects=2 duration=965.73µs29032026/11/11 23:58:33 INFO Received uploads request method=POST path=/29042026/11/11 23:58:33 INFO Vacuumed table table=pending_closures29052026/11/11 23:58:33 INFO Vacuumed table table=pending_objects29062026/11/11 23:58:33 INFO Vacuumed table table=multipart_uploads29072026/11/11 23:58:33 INFO Vacuumed table table=closures29082026/11/11 23:58:33 INFO Received push request method=POST path=/29092026/11/11 23:58:33 INFO Vacuumed table table=objects29102026/11/11 23:58:33 INFO Received push request method=POST path=/29112026/11/11 23:58:33 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29122026/11/11 23:58:33 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29132026/11/11 23:58:33 INFO Received complete push request method=POST path=/29142026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29152026/11/11 23:58:33 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29162026/11/11 23:58:33 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29172026/11/11 23:58:33 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29182026/11/11 23:58:33 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29192026/11/11 23:58:33 INFO Received uploads request method=POST path=/29202026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=029212026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=029222026/11/11 23:58:33 INFO Completed push id=3 objects=22 duration=6.770876ms29232026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period29242026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=029252026/11/11 23:58:33 INFO Received complete push request method=POST path=/29262026/11/11 23:58:33 INFO Aborted multipart uploads count=0 kept=029272026/11/11 23:58:33 INFO Completed push id=6 objects=6 duration=2.200519ms29282026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period29292026/11/11 23:58:33 WARN Force mode enabled - objects will be deleted immediately without grace period29302026/11/11 23:58:34 INFO Received complete upload request method=POST path=/29312026/11/11 23:58:34 INFO Received push request method=POST path=/29322026/11/11 23:58:34 INFO Completed upload id=9 objects=3 duration=1.461979ms29332026/11/11 23:58:34 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29342026/11/11 23:58:34 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29352026/11/11 23:58:34 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29362026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=029372026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=029382026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=029392026/11/11 23:58:34 INFO Received push request method=POST path=/29402026/11/11 23:58:34 INFO Vacuumed table table=pending_closures29412026/11/11 23:58:34 INFO Vacuumed table table=pending_closures29422026/11/11 23:58:34 INFO Vacuumed table table=pending_objects29432026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads29442026/11/11 23:58:34 INFO Received push request method=POST path=/29452026/11/11 23:58:34 INFO Vacuumed table table=pending_objects29462026/11/11 23:58:34 INFO Vacuumed table table=closures29472026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads29482026/11/11 23:58:34 INFO Received push request method=POST path=/29492026/11/11 23:58:34 INFO Vacuumed table table=closures29502026/11/11 23:58:34 INFO Vacuumed table table=objects29512026/11/11 23:58:34 INFO Received complete push request method=POST path=/29522026/11/11 23:58:34 INFO Received push request method=POST path=/29532026/11/11 23:58:34 INFO Vacuumed table table=objects29542026/11/11 23:58:34 INFO Completed push id=4 objects=4 duration=1.950219ms29552026/11/11 23:58:34 INFO Received complete push request method=POST path=/29562026/11/11 23:58:34 INFO Received uploads request method=POST path=/29572026/11/11 23:58:34 INFO Received complete upload request method=POST path=/29582026/11/11 23:58:34 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29592026/11/11 23:58:34 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29602026/11/11 23:58:34 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29612026/11/11 23:58:34 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29622026/11/11 23:58:34 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29632026/11/11 23:58:34 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29642026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29652026/11/11 23:58:34 INFO Completed push id=2 objects=10 duration=3.230338ms29662026/11/11 23:58:34 INFO Received push request method=POST path=/29672026/11/11 23:58:34 INFO Completed upload id=3 objects=24 duration=2.000328ms29682026/11/11 23:58:34 INFO Received complete upload request method=POST path=/29692026/11/11 23:58:34 INFO Completed upload id=10 objects=6 duration=1.610249ms29702026/11/11 23:58:34 INFO Received push request method=POST path=/29712026/11/11 23:58:34 INFO Received uploads request method=POST path=/29722026/11/11 23:58:34 INFO Received complete push request method=POST path=/29732026/11/11 23:58:34 INFO Completed push id=2 objects=7 duration=4.934507ms29742026/11/11 23:58:34 INFO Received push request method=POST path=/29752026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=029762026/11/11 23:58:34 INFO Received uploads request method=POST path=/29772026/11/11 23:58:34 INFO Received uploads request method=POST path=/29782026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period29792026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=029802026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029812026/11/11 23:58:34 INFO Vacuumed table table=pending_closures29822026/11/11 23:58:34 INFO Vacuumed table table=pending_objects29832026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads29842026/11/11 23:58:34 INFO Vacuumed table table=closures29852026/11/11 23:58:34 INFO Vacuumed table table=objects29862026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=029872026/11/11 23:58:34 INFO Received push request method=POST path=/29882026/11/11 23:58:34 INFO Received complete upload request method=POST path=/29892026/11/11 23:58:34 INFO Completed upload id=11 objects=14 duration=2.063469ms29902026/11/11 23:58:34 INFO Received uploads request method=POST path=/29912026/11/11 23:58:34 INFO Received uploads request method=POST path=/29922026/11/11 23:58:34 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29932026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29942026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029952026/11/11 23:58:34 INFO Received push request method=POST path=/29962026/11/11 23:58:34 INFO Vacuumed table table=pending_closures29972026/11/11 23:58:34 INFO Received push request method=POST path=/29982026/11/11 23:58:34 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29992026/11/11 23:58:34 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."30002026/11/11 23:58:34 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30012026/11/11 23:58:34 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30022026/11/11 23:58:34 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."30032026/11/11 23:58:34 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30042026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30052026/11/11 23:58:34 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30062026/11/11 23:58:34 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30072026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."30082026/11/11 23:58:34 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30092026/11/11 23:58:34 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30102026/11/11 23:58:34 INFO Vacuumed table table=pending_objects30112026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads30122026/11/11 23:58:34 INFO Vacuumed table table=closures30132026/11/11 23:58:34 INFO Vacuumed table table=objects30142026/11/11 23:58:34 INFO Received complete push request method=POST path=/30152026/11/11 23:58:34 INFO Received uploads request method=POST path=/30162026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=030172026/11/11 23:58:34 INFO Completed push id=7 objects=27 duration=2.424229ms30182026/11/11 23:58:34 INFO Received complete push request method=POST path=/30192026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030202026/11/11 23:58:34 INFO Completed push id=10 objects=2 duration=2.096499ms30212026/11/11 23:58:34 INFO Vacuumed table table=pending_closures30222026/11/11 23:58:34 INFO Vacuumed table table=pending_objects30232026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads30242026/11/11 23:58:34 INFO Vacuumed table table=closures30252026/11/11 23:58:34 INFO Vacuumed table table=objects30262026/11/11 23:58:34 INFO Received push request method=POST path=/30272026/11/11 23:58:34 INFO Received push request method=POST path=/30282026/11/11 23:58:34 INFO Received push request method=POST path=/30292026/11/11 23:58:34 INFO Received uploads request method=POST path=/30302026/11/11 23:58:34 INFO Received uploads request method=POST path=/30312026/11/11 23:58:34 INFO Received uploads request method=POST path=/30322026/11/11 23:58:34 INFO Received push request method=POST path=/30332026/11/11 23:58:34 INFO Received complete push request method=POST path=/30342026/11/11 23:58:34 INFO Completed push id=8 objects=7 duration=1.804069ms30352026/11/11 23:58:34 INFO Received uploads request method=POST path=/30362026/11/11 23:58:34 INFO Received push request method=POST path=/30372026/11/11 23:58:34 INFO Received push request method=POST path=/30382026/11/11 23:58:34 INFO Received push request method=POST path=/30392026/11/11 23:58:34 INFO Received push request method=POST path=/30402026/11/11 23:58:34 INFO Received uploads request method=POST path=/30412026/11/11 23:58:34 INFO Received push request method=POST path=/30422026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=030432026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period30442026/11/11 23:58:34 INFO Received complete upload request method=POST path=/30452026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030462026/11/11 23:58:34 INFO Received uploads request method=POST path=/30472026/11/11 23:58:34 INFO Vacuumed table table=pending_closures30482026/11/11 23:58:34 INFO Completed upload id=12 objects=21 duration=1.383839ms30492026/11/11 23:58:34 INFO Vacuumed table table=pending_objects30502026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads30512026/11/11 23:58:34 INFO Vacuumed table table=closures30522026/11/11 23:58:34 INFO Vacuumed table table=objects30532026/11/11 23:58:34 INFO Received push request method=POST path=/30542026/11/11 23:58:34 INFO Received complete upload request method=POST path=/30552026/11/11 23:58:34 INFO Received uploads request method=POST path=/30562026/11/11 23:58:34 INFO Received uploads request method=POST path=/30572026/11/11 23:58:34 INFO Received uploads request method=POST path=/30582026/11/11 23:58:34 INFO Received push request method=POST path=/30592026/11/11 23:58:34 INFO Received complete upload request method=POST path=/30602026/11/11 23:58:34 INFO Completed upload id=3 objects=9 duration=1.573139ms30612026/11/11 23:58:34 INFO Completed upload id=2 objects=2 duration=2.077059ms30622026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=030632026/11/11 23:58:34 INFO Received complete push request method=POST path=/30642026/11/11 23:58:34 INFO Received push request method=POST path=/30652026/11/11 23:58:34 INFO Completed push id=7 objects=20 duration=2.815509ms30662026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30672026/11/11 23:58:34 INFO Received push request method=POST path=/30682026/11/11 23:58:34 INFO Received push request method=POST path=/30692026/11/11 23:58:34 INFO Received push request method=POST path=/30702026/11/11 23:58:34 INFO Received complete push request method=POST path=/30712026/11/11 23:58:34 INFO Completed push id=12 objects=3 duration=2.159778ms30722026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=030732026/11/11 23:58:34 INFO Received complete push request method=POST path=/30742026/11/11 23:58:34 INFO Completed push id=5 objects=3 duration=2.085069ms30752026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period30762026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030772026/11/11 23:58:34 INFO Vacuumed table table=pending_closures30782026/11/11 23:58:34 INFO Received uploads request method=POST path=/30792026/11/11 23:58:34 INFO Received complete push request method=POST path=/30802026/11/11 23:58:34 INFO Vacuumed table table=pending_objects30812026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads30822026/11/11 23:58:34 INFO Vacuumed table table=closures30832026/11/11 23:58:34 INFO Completed push id=6 objects=5 duration=2.332018ms30842026/11/11 23:58:34 INFO Vacuumed table table=objects30852026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=030862026/11/11 23:58:34 INFO Received push request method=POST path=/30872026/11/11 23:58:34 INFO Received uploads request method=POST path=/30882026/11/11 23:58:34 INFO Received push request method=POST path=/30892026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period30902026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030912026/11/11 23:58:34 INFO Vacuumed table table=pending_closures30922026/11/11 23:58:34 INFO Received complete upload request method=POST path=/30932026/11/11 23:58:34 INFO Vacuumed table table=pending_objects30942026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads30952026/11/11 23:58:34 INFO Completed upload id=9 objects=16 duration=1.255009ms30962026/11/11 23:58:34 INFO Vacuumed table table=closures30972026/11/11 23:58:34 INFO Vacuumed table table=objects30982026/11/11 23:58:34 INFO Received uploads request method=POST path=/30992026/11/11 23:58:34 INFO Received complete push request method=POST path=/31002026/11/11 23:58:34 INFO Completed push id=13 objects=23 duration=1.933198ms31012026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031022026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031032026/11/11 23:58:34 INFO Received push request method=POST path=/31042026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031052026/11/11 23:58:34 INFO Received complete push request method=POST path=/31062026/11/11 23:58:34 INFO Completed push id=5 objects=6 duration=1.734429ms31072026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031082026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031092026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31102026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31112026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31122026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31132026/11/11 23:58:34 INFO Received push request method=POST path=/31142026/11/11 23:58:34 INFO Vacuumed table table=closures31152026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31162026/11/11 23:58:34 INFO Vacuumed table table=objects31172026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31182026/11/11 23:58:34 INFO Vacuumed table table=closures31192026/11/11 23:58:34 INFO Received uploads request method=POST path=/31202026/11/11 23:58:34 INFO Vacuumed table table=objects31212026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031222026/11/11 23:58:34 INFO Received complete upload request method=POST path=/31232026/11/11 23:58:34 INFO Completed upload id=10 objects=23 duration=2.132238ms31242026/11/11 23:58:34 INFO Received push request method=POST path=/31252026/11/11 23:58:34 INFO Received complete upload request method=POST path=/31262026/11/11 23:58:34 INFO Received push request method=POST path=/31272026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031282026/11/11 23:58:34 INFO Completed upload id=11 objects=6 duration=2.262788ms31292026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031302026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period31312026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031322026/11/11 23:58:34 INFO Received uploads request method=POST path=/31332026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031342026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31352026/11/11 23:58:34 INFO Received complete push request method=POST path=/31362026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031372026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=031382026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period31392026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31402026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31412026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31422026/11/11 23:58:34 INFO Received push request method=POST path=/31432026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31442026/11/11 23:58:34 INFO Vacuumed table table=closures31452026/11/11 23:58:34 INFO Completed push id=13 objects=12 duration=3.997177ms31462026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31472026/11/11 23:58:34 INFO Vacuumed table table=objects31482026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31492026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31502026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31512026/11/11 23:58:34 INFO Vacuumed table table=closures31522026/11/11 23:58:34 INFO Vacuumed table table=objects31532026/11/11 23:58:34 INFO Vacuumed table table=closures31542026/11/11 23:58:34 INFO Vacuumed table table=objects31552026/11/11 23:58:34 INFO Received complete push request method=POST path=/31562026/11/11 23:58:34 INFO Received uploads request method=POST path=/31572026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031582026/11/11 23:58:34 INFO Received push request method=POST path=/31592026/11/11 23:58:34 INFO Completed push id=5 objects=3 duration=3.317608ms31602026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31612026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31622026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31632026/11/11 23:58:34 INFO Vacuumed table table=closures31642026/11/11 23:58:34 INFO Received push request method=POST path=/31652026/11/11 23:58:34 INFO Received complete upload request method=POST path=/31662026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031672026/11/11 23:58:34 INFO Vacuumed table table=objects31682026/11/11 23:58:34 INFO Received complete upload request method=POST path=/31692026/11/11 23:58:34 INFO Received complete push request method=POST path=/31702026/11/11 23:58:34 INFO Completed upload id=4 objects=14 duration=4.102197ms31712026/11/11 23:58:34 INFO Completed upload id=10 objects=2 duration=2.831958ms31722026/11/11 23:58:34 INFO Completed push id=14 objects=3 duration=3.464978ms31732026/11/11 23:58:34 INFO Received push request method=POST path=/31742026/11/11 23:58:34 INFO Received complete push request method=POST path=/31752026/11/11 23:58:34 INFO Completed push id=10 objects=3 duration=1.610409ms31762026/11/11 23:58:34 INFO Received uploads request method=POST path=/31772026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031782026/11/11 23:58:34 INFO Received uploads request method=POST path=/31792026/11/11 23:58:34 INFO Received push request method=POST path=/31802026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=031812026/11/11 23:58:34 INFO Received push request method=POST path=/31822026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031832026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31842026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31852026/11/11 23:58:34 INFO Vacuumed table table=pending_closures31862026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31872026/11/11 23:58:34 INFO Received push request method=POST path=/31882026/11/11 23:58:34 INFO Vacuumed table table=closures31892026/11/11 23:58:34 INFO Vacuumed table table=pending_objects31902026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads31912026/11/11 23:58:34 INFO Vacuumed table table=objects31922026/11/11 23:58:34 INFO Vacuumed table table=closures31932026/11/11 23:58:34 INFO Vacuumed table table=objects31942026/11/11 23:58:34 INFO Received uploads request method=POST path=/31952026/11/11 23:58:34 INFO Received uploads request method=POST path=/31962026/11/11 23:58:34 INFO Received complete upload request method=POST path=/31972026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=031982026/11/11 23:58:34 INFO Completed upload id=16 objects=23 duration=1.524149ms31992026/11/11 23:58:34 INFO Received complete upload request method=POST path=/32002026/11/11 23:58:34 INFO Completed upload id=11 objects=6 duration=1.608399ms32012026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=032022026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period32032026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032042026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32052026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32062026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32072026/11/11 23:58:34 INFO Vacuumed table table=closures32082026/11/11 23:58:34 INFO Received push request method=POST path=/32092026/11/11 23:58:34 INFO Received complete push request method=POST path=/32102026/11/11 23:58:34 INFO Vacuumed table table=objects32112026/11/11 23:58:34 INFO Received uploads request method=POST path=/32122026/11/11 23:58:34 INFO Completed push id=5 objects=6 duration=2.031008ms32132026/11/11 23:58:34 INFO Received push request method=POST path=/32142026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032152026/11/11 23:58:34 INFO Received push request method=POST path=/32162026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32172026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32182026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32192026/11/11 23:58:34 INFO Vacuumed table table=closures32202026/11/11 23:58:34 INFO Vacuumed table table=objects32212026/11/11 23:58:34 INFO Received push request method=POST path=/32222026/11/11 23:58:34 INFO Received uploads request method=POST path=/32232026/11/11 23:58:34 INFO Received push request method=POST path=/32242026/11/11 23:58:34 INFO Received complete upload request method=POST path=/32252026/11/11 23:58:34 INFO Completed upload id=15 objects=6 duration=1.10375ms32262026/11/11 23:58:34 INFO Received push request method=POST path=/32272026/11/11 23:58:34 INFO Received complete push request method=POST path=/32282026/11/11 23:58:34 INFO Completed push id=9 objects=5 duration=2.288379ms32292026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=032302026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=032312026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period32322026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32332026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032342026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32352026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32362026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32372026/11/11 23:58:34 INFO Vacuumed table table=closures32382026/11/11 23:58:34 INFO Received complete push request method=POST path=/32392026/11/11 23:58:34 INFO Vacuumed table table=objects32402026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32412026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32422026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=032432026/11/11 23:58:34 INFO Vacuumed table table=closures32442026/11/11 23:58:34 INFO Vacuumed table table=objects32452026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period32462026/11/11 23:58:34 INFO Received complete push request method=POST path=/32472026/11/11 23:58:34 INFO Completed push id=5 objects=9 duration=5.016677ms32482026/11/11 23:58:34 INFO Completed push id=8 objects=6 duration=1.631839ms32492026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032502026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32512026/11/11 23:58:34 INFO Received uploads request method=POST path=/32522026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32532026/11/11 23:58:34 INFO Received push request method=POST path=/32542026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=032552026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32562026/11/11 23:58:34 INFO Received uploads request method=POST path=/32572026/11/11 23:58:34 INFO Vacuumed table table=closures32582026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period32592026/11/11 23:58:34 INFO Vacuumed table table=objects32602026/11/11 23:58:34 INFO Received push request method=POST path=/32612026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032622026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32632026/11/11 23:58:34 INFO Received complete push request method=POST path=/32642026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=032652026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32662026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32672026/11/11 23:58:34 INFO Vacuumed table table=closures32682026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period32692026/11/11 23:58:34 INFO Vacuumed table table=objects32702026/11/11 23:58:34 INFO Completed push id=3 objects=21 duration=4.750418ms32712026/11/11 23:58:34 INFO Received complete push request method=POST path=/32722026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032732026/11/11 23:58:34 INFO Completed push id=12 objects=15 duration=2.239509ms32742026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32752026/11/11 23:58:34 INFO Received push request method=POST path=/32762026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32772026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32782026/11/11 23:58:34 INFO Vacuumed table table=closures32792026/11/11 23:58:34 INFO Received uploads request method=POST path=/32802026/11/11 23:58:34 INFO Vacuumed table table=objects32812026/11/11 23:58:34 INFO Received push request method=POST path=/32822026/11/11 23:58:34 INFO Received uploads request method=POST path=/32832026/11/11 23:58:34 INFO Received complete upload request method=POST path=/32842026/11/11 23:58:34 INFO Completed upload id=17 objects=7 duration=1.163369ms32852026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=032862026/11/11 23:58:34 INFO Received complete push request method=POST path=/32872026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032882026/11/11 23:58:34 INFO Vacuumed table table=pending_closures32892026/11/11 23:58:34 INFO Completed push id=19 objects=6 duration=4.006857ms32902026/11/11 23:58:34 INFO Vacuumed table table=pending_objects32912026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads32922026/11/11 23:58:34 INFO Vacuumed table table=closures32932026/11/11 23:58:34 INFO Vacuumed table table=objects32942026/11/11 23:58:34 INFO Received push request method=POST path=/32952026/11/11 23:58:34 INFO Received uploads request method=POST path=/32962026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=032972026/11/11 23:58:34 INFO Received push request method=POST path=/32982026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032992026/11/11 23:58:34 INFO Vacuumed table table=pending_closures33002026/11/11 23:58:34 INFO Vacuumed table table=pending_objects33012026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads33022026/11/11 23:58:34 INFO Received push request method=POST path=/33032026/11/11 23:58:34 INFO Vacuumed table table=closures33042026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033052026/11/11 23:58:34 INFO Vacuumed table table=objects33062026/11/11 23:58:34 INFO Received uploads request method=POST path=/33072026/11/11 23:58:34 INFO Received push request method=POST path=/33082026/11/11 23:58:34 INFO Received complete push request method=POST path=/33092026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033102026/11/11 23:58:34 INFO Completed push id=11 objects=8 duration=3.361219ms33112026/11/11 23:58:34 INFO Vacuumed table table=pending_closures33122026/11/11 23:58:34 INFO Vacuumed table table=pending_objects33132026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads33142026/11/11 23:58:34 INFO Received uploads request method=POST path=/33152026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033162026/11/11 23:58:34 INFO Vacuumed table table=closures33172026/11/11 23:58:34 INFO Vacuumed table table=objects33182026/11/11 23:58:34 INFO Received push request method=POST path=/33192026/11/11 23:58:34 INFO Received complete push request method=POST path=/33202026/11/11 23:58:34 INFO Received push request method=POST path=/33212026/11/11 23:58:34 INFO Completed push id=5 objects=8 duration=4.156267ms33222026/11/11 23:58:34 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33232026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33242026/11/11 23:58:34 INFO Received uploads request method=POST path=/33252026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033262026/11/11 23:58:34 INFO Received complete upload request method=POST path=/33272026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033282026/11/11 23:58:34 INFO Vacuumed table table=pending_closures33292026/11/11 23:58:34 INFO Completed upload id=15 objects=4 duration=1.726629ms33302026/11/11 23:58:34 INFO Vacuumed table table=pending_objects33312026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads33322026/11/11 23:58:34 INFO Vacuumed table table=closures33332026/11/11 23:58:34 INFO Vacuumed table table=objects33342026/11/11 23:58:34 INFO Received uploads request method=POST path=/33352026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033362026/11/11 23:58:34 INFO Received push request method=POST path=/33372026/11/11 23:58:34 INFO Received uploads request method=POST path=/33382026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033392026/11/11 23:58:34 INFO Vacuumed table table=pending_closures33402026/11/11 23:58:34 INFO Vacuumed table table=pending_objects33412026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads33422026/11/11 23:58:34 INFO Vacuumed table table=closures33432026/11/11 23:58:34 INFO Vacuumed table table=objects33442026/11/11 23:58:34 INFO Received uploads request method=POST path=/33452026/11/11 23:58:34 INFO Received push request method=POST path=/33462026/11/11 23:58:34 INFO Received push request method=POST path=/33472026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033482026/11/11 23:58:34 INFO Received push request method=POST path=/33492026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period33502026/11/11 23:58:34 INFO Received uploads request method=POST path=/33512026/11/11 23:58:34 INFO Received uploads request method=POST path=/33522026/11/11 23:58:34 INFO Received complete push request method=POST path=/33532026/11/11 23:58:34 INFO Received complete push request method=POST path=/33542026/11/11 23:58:34 INFO Received uploads request method=POST path=/33552026/11/11 23:58:34 INFO Completed push id=13 objects=6 duration=3.143598ms33562026/11/11 23:58:34 INFO Completed push id=9 objects=4 duration=3.866698ms33572026/11/11 23:58:34 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33582026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33592026/11/11 23:58:34 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33602026/11/11 23:58:34 INFO Received push request method=POST path=/33612026/11/11 23:58:34 INFO Received push request method=POST path=/33622026/11/11 23:58:34 INFO Received push request method=POST path=/33632026/11/11 23:58:34 INFO Received uploads request method=POST path=/33642026/11/11 23:58:34 INFO Received uploads request method=POST path=/33652026/11/11 23:58:34 INFO Received complete upload request method=POST path=/33662026/11/11 23:58:34 INFO Completed upload id=14 objects=6 duration=1.08594ms33672026/11/11 23:58:34 INFO Received push request method=POST path=/33682026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033692026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period33702026/11/11 23:58:34 INFO Received push request method=POST path=/33712026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033722026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033732026/11/11 23:58:34 INFO Vacuumed table table=pending_closures33742026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033752026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period33762026/11/11 23:58:34 INFO Vacuumed table table=pending_objects33772026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads33782026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033792026/11/11 23:58:34 INFO Vacuumed table table=closures33802026/11/11 23:58:34 INFO Received complete push request method=POST path=/33812026/11/11 23:58:34 INFO Vacuumed table table=objects33822026/11/11 23:58:34 INFO Vacuumed table table=pending_closures33832026/11/11 23:58:34 INFO Vacuumed table table=pending_objects33842026/11/11 23:58:34 INFO Received push request method=POST path=/33852026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads33862026/11/11 23:58:34 INFO Completed push id=8 objects=6 duration=3.262848ms33872026/11/11 23:58:34 INFO Received uploads request method=POST path=/33882026/11/11 23:58:34 INFO Vacuumed table table=closures33892026/11/11 23:58:34 INFO Received complete upload request method=POST path=/33902026/11/11 23:58:34 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33912026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33922026/11/11 23:58:34 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33932026/11/11 23:58:34 INFO Completed upload id=16 objects=3 duration=995.009µs33942026/11/11 23:58:34 INFO Vacuumed table table=objects33952026/11/11 23:58:34 INFO Received push request method=POST path=/33962026/11/11 23:58:34 INFO Received complete push request method=POST path=/33972026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=033982026/11/11 23:58:34 INFO Completed push id=6 objects=26 duration=4.426257ms33992026/11/11 23:58:34 INFO Received complete upload request method=POST path=/34002026/11/11 23:58:34 INFO Completed upload id=10 objects=17 duration=1.265899ms34012026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period34022026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034032026/11/11 23:58:34 INFO Vacuumed table table=pending_closures34042026/11/11 23:58:34 INFO Vacuumed table table=pending_objects34052026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads34062026/11/11 23:58:34 INFO Vacuumed table table=closures34072026/11/11 23:58:34 INFO Received push request method=POST path=/34082026/11/11 23:58:34 INFO Vacuumed table table=objects34092026/11/11 23:58:34 INFO Received uploads request method=POST path=/34102026/11/11 23:58:34 INFO Received complete push request method=POST path=/34112026/11/11 23:58:34 INFO Completed push id=15 objects=5 duration=2.212999ms34122026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=034132026/11/11 23:58:34 INFO Received uploads request method=POST path=/34142026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034152026/11/11 23:58:34 INFO Vacuumed table table=pending_closures34162026/11/11 23:58:34 INFO Received uploads request method=POST path=/34172026/11/11 23:58:34 INFO Vacuumed table table=pending_objects34182026/11/11 23:58:34 INFO Received complete upload request method=POST path=/34192026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads34202026/11/11 23:58:34 INFO Completed upload id=7 objects=3 duration=1.357619ms34212026/11/11 23:58:34 INFO Vacuumed table table=closures34222026/11/11 23:58:34 INFO Vacuumed table table=objects34232026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=034242026/11/11 23:58:34 INFO Received push request method=POST path=/34252026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period34262026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=034272026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034282026/11/11 23:58:34 INFO Vacuumed table table=pending_closures34292026/11/11 23:58:34 INFO Vacuumed table table=pending_objects34302026/11/11 23:58:34 INFO Received uploads request method=POST path=/34312026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads34322026/11/11 23:58:34 INFO Vacuumed table table=closures34332026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034342026/11/11 23:58:34 INFO Received uploads request method=POST path=/34352026/11/11 23:58:34 INFO Vacuumed table table=objects34362026/11/11 23:58:34 INFO Vacuumed table table=pending_closures34372026/11/11 23:58:34 INFO Received uploads request method=POST path=/34382026/11/11 23:58:34 INFO Received complete push request method=POST path=/34392026/11/11 23:58:34 INFO Vacuumed table table=pending_objects34402026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads34412026/11/11 23:58:34 INFO Vacuumed table table=closures34422026/11/11 23:58:34 INFO Completed push id=17 objects=6 duration=2.186418ms34432026/11/11 23:58:34 INFO Vacuumed table table=objects34442026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=034452026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period34462026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034472026/11/11 23:58:34 INFO Vacuumed table table=pending_closures34482026/11/11 23:58:34 INFO Vacuumed table table=pending_objects34492026/11/11 23:58:34 INFO Received push request method=POST path=/34502026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads34512026/11/11 23:58:34 INFO Vacuumed table table=closures34522026/11/11 23:58:34 INFO Received uploads request method=POST path=/34532026/11/11 23:58:34 INFO Vacuumed table table=objects34542026/11/11 23:58:34 INFO Received complete push request method=POST path=/34552026/11/11 23:58:34 INFO Received uploads request method=POST path=/34562026/11/11 23:58:34 INFO Received complete push request method=POST path=/34572026/11/11 23:58:34 INFO Completed push id=15 objects=7 duration=2.349629ms34582026/11/11 23:58:34 INFO Completed push id=18 objects=23 duration=2.099599ms34592026/11/11 23:58:34 INFO Received complete push request method=POST path=/34602026/11/11 23:58:34 INFO Received push request method=POST path=/34612026/11/11 23:58:34 INFO Completed push id=15 objects=24 duration=3.967218ms34622026/11/11 23:58:34 INFO Received push request method=POST path=/34632026/11/11 23:58:34 INFO Received uploads request method=POST path=/34642026/11/11 23:58:34 INFO Received complete upload request method=POST path=/34652026/11/11 23:58:34 INFO Received uploads request method=POST path=/34662026/11/11 23:58:34 INFO Received complete upload request method=POST path=/34672026/11/11 23:58:34 INFO Received push request method=POST path=/34682026/11/11 23:58:34 INFO Completed upload id=10 objects=6 duration=2.662779ms34692026/11/11 23:58:34 INFO Received complete push request method=POST path=/34702026/11/11 23:58:34 INFO Completed upload id=17 objects=6 duration=2.520279ms34712026/11/11 23:58:34 INFO Completed push id=4 objects=7 duration=2.054749ms34722026/11/11 23:58:34 INFO Received push request method=POST path=/34732026/11/11 23:58:34 INFO Received uploads request method=POST path=/34742026/11/11 23:58:34 INFO Received uploads request method=POST path=/34752026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=034762026/11/11 23:58:34 INFO Received complete push request method=POST path=/34772026/11/11 23:58:34 INFO Completed push id=14 objects=6 duration=1.752738ms34782026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034792026/11/11 23:58:34 INFO Vacuumed table table=pending_closures34802026/11/11 23:58:34 INFO Received complete upload request method=POST path=/34812026/11/11 23:58:34 INFO Vacuumed table table=pending_objects34822026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads34832026/11/11 23:58:34 INFO Vacuumed table table=closures34842026/11/11 23:58:34 INFO Completed upload id=18 objects=4 duration=4.302308ms34852026/11/11 23:58:34 INFO Vacuumed table table=objects34862026/11/11 23:58:34 INFO Received uploads request method=POST path=/34872026/11/11 23:58:34 INFO Received complete push request method=POST path=/34882026/11/11 23:58:34 INFO Received complete push request method=POST path=/34892026/11/11 23:58:34 INFO Received uploads request method=POST path=/34902026/11/11 23:58:34 INFO Completed push id=9 objects=24 duration=1.208659ms34912026/11/11 23:58:34 INFO Completed push id=11 objects=11 duration=2.329958ms34922026/11/11 23:58:34 INFO Received complete upload request method=POST path=/34932026/11/11 23:58:34 INFO Completed upload id=8 objects=3 duration=2.316429ms34942026/11/11 23:58:34 INFO Received complete push request method=POST path=/34952026/11/11 23:58:34 INFO Received uploads request method=POST path=/34962026/11/11 23:58:34 INFO Completed push id=17 objects=22 duration=1.823819ms34972026/11/11 23:58:34 INFO Received push request method=POST path=/34982026/11/11 23:58:34 INFO Received push request method=POST path=/34992026/11/11 23:58:34 INFO Received complete push request method=POST path=/35002026/11/11 23:58:34 INFO Completed push id=11 objects=9 duration=2.194339ms35012026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035022026/11/11 23:58:34 INFO Received uploads request method=POST path=/35032026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period35042026/11/11 23:58:34 INFO Received uploads request method=POST path=/35052026/11/11 23:58:34 INFO Received complete upload request method=POST path=/35062026/11/11 23:58:34 INFO Completed upload id=15 objects=6 duration=938.889µs35072026/11/11 23:58:34 INFO Received push request method=POST path=/35082026/11/11 23:58:34 INFO Received uploads request method=POST path=/35092026/11/11 23:58:34 INFO Received uploads request method=POST path=/35102026/11/11 23:58:34 INFO Received complete push request method=POST path=/35112026/11/11 23:58:34 INFO Received push request method=POST path=/35122026/11/11 23:58:34 INFO Completed push id=17 objects=6 duration=1.884439ms35132026/11/11 23:58:34 INFO Received push request method=POST path=/35142026/11/11 23:58:34 INFO Received push request method=POST path=/35152026/11/11 23:58:34 INFO Received uploads request method=POST path=/35162026/11/11 23:58:34 INFO Received uploads request method=POST path=/35172026/11/11 23:58:34 INFO Received uploads request method=POST path=/35182026/11/11 23:58:34 INFO Received complete push request method=POST path=/35192026/11/11 23:58:34 INFO Completed push id=4 objects=6 duration=1.393169ms35202026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35212026/11/11 23:58:34 INFO Received push request method=POST path=/35222026/11/11 23:58:34 INFO Received complete push request method=POST path=/35232026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035242026/11/11 23:58:34 INFO Received push request method=POST path=/35252026/11/11 23:58:34 INFO Completed push id=8 objects=6 duration=1.960579ms35262026/11/11 23:58:34 INFO Received push request method=POST path=/35272026/11/11 23:58:34 INFO Received complete upload request method=POST path=/35282026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=035292026/11/11 23:58:34 INFO Completed upload id=10 objects=10 duration=1.333679ms35302026/11/11 23:58:34 INFO Vacuumed table table=pending_closures35312026/11/11 23:58:34 INFO Vacuumed table table=pending_objects35322026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads35332026/11/11 23:58:34 INFO Vacuumed table table=closures35342026/11/11 23:58:34 INFO Vacuumed table table=objects35352026/11/11 23:58:34 INFO Received complete upload request method=POST path=/35362026/11/11 23:58:34 INFO Received complete upload request method=POST path=/35372026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035382026/11/11 23:58:34 INFO Completed upload id=22 objects=2 duration=1.252799ms35392026/11/11 23:58:34 INFO Completed upload id=6 objects=6 duration=1.315779ms35402026/11/11 23:58:34 INFO Received complete push request method=POST path=/35412026/11/11 23:58:34 INFO Completed push id=7 objects=7 duration=965.98µs35422026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035432026/11/11 23:58:34 INFO Received push request method=POST path=/35442026/11/11 23:58:34 INFO Vacuumed table table=pending_closures35452026/11/11 23:58:34 INFO Vacuumed table table=pending_objects35462026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads35472026/11/11 23:58:34 INFO Vacuumed table table=closures35482026/11/11 23:58:34 INFO Received complete push request method=POST path=/35492026/11/11 23:58:34 INFO Vacuumed table table=objects35502026/11/11 23:58:34 INFO Received complete push request method=POST path=/35512026/11/11 23:58:34 INFO Completed push id=6 objects=6 duration=2.138849ms35522026/11/11 23:58:34 INFO Received uploads request method=POST path=/35532026/11/11 23:58:34 INFO Completed push id=12 objects=7 duration=1.722909ms35542026/11/11 23:58:34 INFO Received push request method=POST path=/35552026/11/11 23:58:34 INFO Received complete upload request method=POST path=/35562026/11/11 23:58:34 INFO Completed upload id=7 objects=10 duration=1.30797ms35572026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035582026/11/11 23:58:34 INFO Received uploads request method=POST path=/35592026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035602026/11/11 23:58:34 INFO Received complete upload request method=POST path=/35612026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period35622026/11/11 23:58:34 INFO Completed upload id=9 objects=2 duration=1.640069ms35632026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035642026/11/11 23:58:34 INFO Vacuumed table table=pending_closures35652026/11/11 23:58:34 INFO Received push request method=POST path=/35662026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035672026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035682026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035692026/11/11 23:58:34 INFO Vacuumed table table=pending_objects35702026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads35712026/11/11 23:58:34 INFO Vacuumed table table=pending_closures35722026/11/11 23:58:34 INFO Received complete push request method=POST path=/35732026/11/11 23:58:34 INFO Vacuumed table table=closures35742026/11/11 23:58:34 INFO Vacuumed table table=pending_objects35752026/11/11 23:58:34 INFO Completed push id=16 objects=27 duration=1.462649ms35762026/11/11 23:58:34 INFO Vacuumed table table=objects35772026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads35782026/11/11 23:58:34 INFO Received uploads request method=POST path=/35792026/11/11 23:58:34 INFO Vacuumed table table=closures35802026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035812026/11/11 23:58:34 INFO Vacuumed table table=objects35822026/11/11 23:58:34 INFO Vacuumed table table=pending_closures35832026/11/11 23:58:34 INFO Received uploads request method=POST path=/35842026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035852026/11/11 23:58:34 INFO Vacuumed table table=pending_objects35862026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads35872026/11/11 23:58:34 INFO Received push request method=POST path=/35882026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=035892026/11/11 23:58:34 INFO Vacuumed table table=closures35902026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period35912026/11/11 23:58:34 INFO Received complete upload request method=POST path=/35922026/11/11 23:58:34 INFO Vacuumed table table=objects35932026/11/11 23:58:34 INFO Received complete push request method=POST path=/35942026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035952026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period35962026/11/11 23:58:34 INFO Completed upload id=19 objects=25 duration=3.017168ms35972026/11/11 23:58:34 INFO Vacuumed table table=pending_closures35982026/11/11 23:58:34 INFO Received complete push request method=POST path=/35992026/11/11 23:58:34 INFO Vacuumed table table=pending_objects36002026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036012026/11/11 23:58:34 INFO Completed push id=9 objects=7 duration=4.412687ms36022026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads36032026/11/11 23:58:34 INFO Completed push id=8 objects=3 duration=1.914929ms36042026/11/11 23:58:34 INFO Vacuumed table table=closures36052026/11/11 23:58:34 INFO Vacuumed table table=pending_closures36062026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036072026/11/11 23:58:34 INFO Vacuumed table table=objects36082026/11/11 23:58:34 INFO Vacuumed table table=pending_objects36092026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads36102026/11/11 23:58:34 INFO Vacuumed table table=pending_closures36112026/11/11 23:58:34 INFO Vacuumed table table=closures36122026/11/11 23:58:34 INFO Vacuumed table table=pending_objects36132026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads36142026/11/11 23:58:34 INFO Vacuumed table table=objects36152026/11/11 23:58:34 INFO Vacuumed table table=closures36162026/11/11 23:58:34 INFO Vacuumed table table=objects36172026/11/11 23:58:34 INFO Received uploads request method=POST path=/36182026/11/11 23:58:34 INFO Received push request method=POST path=/36192026/11/11 23:58:34 INFO Received uploads request method=POST path=/36202026/11/11 23:58:34 INFO Received push request method=POST path=/36212026/11/11 23:58:34 INFO Received uploads request method=POST path=/36222026/11/11 23:58:34 INFO Received push request method=POST path=/36232026/11/11 23:58:34 INFO Received complete upload request method=POST path=/36242026/11/11 23:58:34 INFO Completed upload id=18 objects=11 duration=1.168449ms36252026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=036262026/11/11 23:58:34 INFO Received push request method=POST path=/36272026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period36282026/11/11 23:58:34 INFO Received uploads request method=POST path=/36292026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036302026/11/11 23:58:34 INFO Vacuumed table table=pending_closures36312026/11/11 23:58:34 INFO Vacuumed table table=pending_objects36322026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads36332026/11/11 23:58:34 INFO Received complete upload request method=POST path=/36342026/11/11 23:58:34 INFO Vacuumed table table=closures36352026/11/11 23:58:34 INFO Received uploads request method=POST path=/36362026/11/11 23:58:34 INFO Vacuumed table table=objects36372026/11/11 23:58:34 INFO Completed upload id=10 objects=7 duration=1.925488ms36382026/11/11 23:58:34 INFO Received push request method=POST path=/36392026/11/11 23:58:34 INFO Received complete push request method=POST path=/36402026/11/11 23:58:34 INFO Received uploads request method=POST path=/36412026/11/11 23:58:34 INFO Completed push id=10 objects=15 duration=2.130749ms36422026/11/11 23:58:34 INFO Received uploads request method=POST path=/36432026/11/11 23:58:34 INFO Received push request method=POST path=/36442026/11/11 23:58:34 INFO Received uploads request method=POST path=/36452026/11/11 23:58:34 INFO Received uploads request method=POST path=/36462026/11/11 23:58:34 INFO Received push request method=POST path=/36472026/11/11 23:58:34 INFO Received uploads request method=POST path=/36482026/11/11 23:58:34 INFO Received uploads request method=POST path=/36492026/11/11 23:58:34 INFO Received push request method=POST path=/36502026/11/11 23:58:34 INFO Received complete upload request method=POST path=/36512026/11/11 23:58:34 INFO Completed upload id=15 objects=9 duration=1.215859ms36522026/11/11 23:58:34 INFO Received complete push request method=POST path=/36532026/11/11 23:58:34 INFO Received complete upload request method=POST path=/36542026/11/11 23:58:34 INFO Completed upload id=13 objects=10 duration=1.061999ms36552026/11/11 23:58:34 INFO Received uploads request method=POST path=/36562026/11/11 23:58:34 INFO Completed push id=12 objects=2 duration=3.337648ms36572026/11/11 23:58:34 INFO Received push request method=POST path=/36582026/11/11 23:58:34 INFO Received complete push request method=POST path=/36592026/11/11 23:58:34 INFO Completed push id=18 objects=10 duration=1.30983ms36602026/11/11 23:58:34 INFO Received push request method=POST path=/36612026/11/11 23:58:34 INFO Received complete upload request method=POST path=/36622026/11/11 23:58:34 INFO Completed upload id=11 objects=7 duration=1.00446ms36632026/11/11 23:58:34 INFO Received push request method=POST path=/36642026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=036652026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period36662026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=036672026/11/11 23:58:34 INFO Received uploads request method=POST path=/36682026/11/11 23:58:34 INFO Received complete push request method=POST path=/36692026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period36702026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036712026/11/11 23:58:34 INFO Received uploads request method=POST path=/36722026/11/11 23:58:34 INFO Vacuumed table table=pending_closures36732026/11/11 23:58:34 INFO Completed push id=14 objects=3 duration=3.337378ms36742026/11/11 23:58:34 INFO Received uploads request method=POST path=/36752026/11/11 23:58:34 INFO Vacuumed table table=pending_objects36762026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads36772026/11/11 23:58:34 INFO Received push request method=POST path=/36782026/11/11 23:58:34 INFO Vacuumed table table=closures36792026/11/11 23:58:34 INFO Vacuumed table table=objects36802026/11/11 23:58:34 INFO Received uploads request method=POST path=/36812026/11/11 23:58:34 INFO Received complete upload request method=POST path=/36822026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=036832026/11/11 23:58:34 INFO Completed upload id=23 objects=15 duration=2.887008ms36842026/11/11 23:58:34 INFO Received uploads request method=POST path=/36852026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period36862026/11/11 23:58:34 INFO Received uploads request method=POST path=/36872026/11/11 23:58:34 INFO Received complete push request method=POST path=/36882026/11/11 23:58:34 INFO Received complete push request method=POST path=/36892026/11/11 23:58:34 INFO Completed push id=14 objects=3 duration=1.249429ms36902026/11/11 23:58:34 INFO Completed push id=21 objects=17 duration=2.199139ms36912026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=036922026/11/11 23:58:34 INFO Received uploads request method=POST path=/36932026/11/11 23:58:34 INFO Received push request method=POST path=/36942026/11/11 23:58:34 INFO Received complete push request method=POST path=/36952026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036962026/11/11 23:58:34 INFO Completed push id=21 objects=9 duration=2.050789ms36972026/11/11 23:58:34 INFO Received uploads request method=POST path=/36982026/11/11 23:58:34 INFO Vacuumed table table=pending_closures36992026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37002026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37012026/11/11 23:58:34 INFO Vacuumed table table=closures37022026/11/11 23:58:34 INFO Vacuumed table table=objects37032026/11/11 23:58:34 INFO Received complete upload request method=POST path=/37042026/11/11 23:58:34 INFO Completed upload id=18 objects=4 duration=1.638039ms37052026/11/11 23:58:34 INFO Received push request method=POST path=/37062026/11/11 23:58:34 INFO Received push request method=POST path=/37072026/11/11 23:58:34 INFO Received complete push request method=POST path=/37082026/11/11 23:58:34 INFO Completed push id=19 objects=7 duration=1.720479ms37092026/11/11 23:58:34 INFO Received uploads request method=POST path=/37102026/11/11 23:58:34 INFO Received uploads request method=POST path=/37112026/11/11 23:58:34 INFO Received uploads request method=POST path=/37122026/11/11 23:58:34 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37132026/11/11 23:58:34 INFO Received push request method=POST path=/37142026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=037152026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=037162026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period37172026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period37182026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037192026/11/11 23:58:34 INFO Received uploads request method=POST path=/37202026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=037212026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37222026/11/11 23:58:34 INFO Received complete upload request method=POST path=/37232026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37242026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037252026/11/11 23:58:34 INFO Completed upload id=19 objects=6 duration=1.512889ms37262026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37272026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37282026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37292026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37302026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37312026/11/11 23:58:34 INFO Received push request method=POST path=/37322026/11/11 23:58:34 INFO Vacuumed table table=closures37332026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37342026/11/11 23:58:34 INFO Vacuumed table table=closures37352026/11/11 23:58:34 INFO Received push request method=POST path=/37362026/11/11 23:58:34 INFO Vacuumed table table=objects37372026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37382026/11/11 23:58:34 INFO Vacuumed table table=objects37392026/11/11 23:58:34 INFO Vacuumed table table=closures37402026/11/11 23:58:34 INFO Received uploads request method=POST path=/37412026/11/11 23:58:34 INFO Vacuumed table table=objects37422026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=037432026/11/11 23:58:34 INFO Received push request method=POST path=/37442026/11/11 23:58:34 INFO Received uploads request method=POST path=/37452026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=037462026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period37472026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037482026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037492026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37502026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37512026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37522026/11/11 23:58:34 INFO Received push request method=POST path=/37532026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37542026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37552026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37562026/11/11 23:58:34 INFO Vacuumed table table=closures37572026/11/11 23:58:34 INFO Vacuumed table table=closures37582026/11/11 23:58:34 INFO Vacuumed table table=objects37592026/11/11 23:58:34 INFO Vacuumed table table=objects37602026/11/11 23:58:34 INFO Received push request method=POST path=/37612026/11/11 23:58:34 INFO Received push request method=POST path=/37622026/11/11 23:58:34 INFO Received push request method=POST path=/37632026/11/11 23:58:34 INFO Received uploads request method=POST path=/37642026/11/11 23:58:34 INFO Received push request method=POST path=/37652026/11/11 23:58:34 INFO Received complete upload request method=POST path=/37662026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=037672026/11/11 23:58:34 INFO Completed upload id=22 objects=7 duration=1.35161ms37682026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period37692026/11/11 23:58:34 INFO Received complete push request method=POST path=/37702026/11/11 23:58:34 INFO Completed push id=15 objects=4 duration=1.602959ms37712026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037722026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37732026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37742026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37752026/11/11 23:58:34 INFO Vacuumed table table=closures37762026/11/11 23:58:34 INFO Vacuumed table table=objects37772026/11/11 23:58:34 INFO Received push request method=POST path=/37782026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=037792026/11/11 23:58:34 INFO Received push request method=POST path=/37802026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period37812026/11/11 23:58:34 INFO Received push request method=POST path=/37822026/11/11 23:58:34 INFO Received push request method=POST path=/37832026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037842026/11/11 23:58:34 INFO Received push request method=POST path=/37852026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37862026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37872026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37882026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=037892026/11/11 23:58:34 INFO Vacuumed table table=closures37902026/11/11 23:58:34 INFO Vacuumed table table=objects37912026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period37922026/11/11 23:58:34 INFO Received uploads request method=POST path=/37932026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037942026/11/11 23:58:34 INFO Vacuumed table table=pending_closures37952026/11/11 23:58:34 INFO Received push request method=POST path=/37962026/11/11 23:58:34 INFO Vacuumed table table=pending_objects37972026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads37982026/11/11 23:58:34 INFO Vacuumed table table=closures37992026/11/11 23:58:34 INFO Vacuumed table table=objects38002026/11/11 23:58:34 INFO Received uploads request method=POST path=/38012026/11/11 23:58:34 INFO Received complete push request method=POST path=/38022026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=038032026/11/11 23:58:34 INFO Aborted multipart uploads count=0 kept=038042026/11/11 23:58:34 INFO Completed push id=14 objects=4 duration=3.222209ms38052026/11/11 23:58:34 INFO Received uploads request method=POST path=/38062026/11/11 23:58:34 WARN Force mode enabled - objects will be deleted immediately without grace period38072026/11/11 23:58:34 INFO Received push request method=POST path=/38082026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038092026/11/11 23:58:34 INFO Received uploads request method=POST path=/38102026/11/11 23:58:34 INFO Vacuumed table table=pending_closures38112026/11/11 23:58:34 INFO Received push request method=POST path=/38122026/11/11 23:58:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038132026/11/11 23:58:34 INFO Vacuumed table table=pending_objects38142026/11/11 23:58:34 INFO Vacuumed table table=pending_closures38152026/11/11 23:58:34 INFO Vacuumed table table=multipart_uploads38162026/11/11 23:58:34 INFO Vacuumed table table=closures38172026/11/11 23:58:34 INFO Vacuumed table table=pending_objects38182026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads38192026/11/11 23:58:35 INFO Vacuumed table table=objects38202026/11/11 23:58:35 INFO Received uploads request method=POST path=/38212026/11/11 23:58:35 INFO Vacuumed table table=closures38222026/11/11 23:58:35 INFO Received push request method=POST path=/38232026/11/11 23:58:35 INFO Vacuumed table table=objects38242026/11/11 23:58:35 INFO Received push request method=POST path=/38252026/11/11 23:58:35 INFO Received complete push request method=POST path=/38262026/11/11 23:58:35 INFO Completed push id=13 objects=15 duration=1.375649ms38272026/11/11 23:58:35 INFO Received uploads request method=POST path=/38282026/11/11 23:58:35 INFO Received uploads request method=POST path=/38292026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038302026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038312026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038322026/11/11 23:58:35 INFO Vacuumed table table=pending_closures38332026/11/11 23:58:35 INFO Vacuumed table table=pending_objects38342026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads38352026/11/11 23:58:35 INFO Received push request method=POST path=/38362026/11/11 23:58:35 INFO Vacuumed table table=closures38372026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period38382026/11/11 23:58:35 INFO Received push request method=POST path=/38392026/11/11 23:58:35 INFO Received complete push request method=POST path=/38402026/11/11 23:58:35 INFO Vacuumed table table=objects38412026/11/11 23:58:35 INFO Received push request method=POST path=/38422026/11/11 23:58:35 INFO Completed push id=24 objects=9 duration=2.166208ms38432026/11/11 23:58:35 INFO Received complete upload request method=POST path=/38442026/11/11 23:58:35 INFO Received push request method=POST path=/38452026/11/11 23:58:35 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."38462026/11/11 23:58:35 INFO Completed upload id=12 objects=26 duration=2.693869ms38472026/11/11 23:58:35 INFO Received uploads request method=POST path=/38482026/11/11 23:58:35 INFO Received complete push request method=POST path=/38492026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038502026/11/11 23:58:35 INFO Received push request method=POST path=/38512026/11/11 23:58:35 INFO Received complete push request method=POST path=/38522026/11/11 23:58:35 INFO Completed push id=1 objects=11 duration=2.736208ms38532026/11/11 23:58:35 INFO Received complete push request method=POST path=/38542026/11/11 23:58:35 INFO Completed push id=19 objects=8 duration=4.410568ms38552026/11/11 23:58:35 INFO Completed push id=16 objects=4 duration=2.038588ms38562026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."38572026/11/11 23:58:35 INFO Received complete push request method=POST path=/38582026/11/11 23:58:35 INFO Received complete push request method=POST path=/38592026/11/11 23:58:35 INFO Completed push id=14 objects=3 duration=1.315129ms38602026/11/11 23:58:35 INFO Received push request method=POST path=/38612026/11/11 23:58:35 INFO Received uploads request method=POST path=/38622026/11/11 23:58:35 INFO Completed push id=22 objects=3 duration=2.929028ms38632026/11/11 23:58:35 INFO Received uploads request method=POST path=/38642026/11/11 23:58:35 INFO Received push request method=POST path=/38652026/11/11 23:58:35 INFO Received uploads request method=POST path=/38662026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038672026/11/11 23:58:35 INFO Received complete upload request method=POST path=/38682026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period38692026/11/11 23:58:35 INFO Completed upload id=23 objects=16 duration=2.520309ms38702026/11/11 23:58:35 INFO Received uploads request method=POST path=/38712026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038722026/11/11 23:58:35 INFO Vacuumed table table=pending_closures38732026/11/11 23:58:35 INFO Received uploads request method=POST path=/38742026/11/11 23:58:35 INFO Received complete upload request method=POST path=/38752026/11/11 23:58:35 INFO Vacuumed table table=pending_objects38762026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads38772026/11/11 23:58:35 INFO Completed upload id=16 objects=3 duration=1.529329ms38782026/11/11 23:58:35 INFO Vacuumed table table=closures38792026/11/11 23:58:35 INFO Vacuumed table table=objects38802026/11/11 23:58:35 INFO Received push request method=POST path=/38812026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038822026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038832026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period38842026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038852026/11/11 23:58:35 INFO Received push request method=POST path=/38862026/11/11 23:58:35 INFO Received uploads request method=POST path=/38872026/11/11 23:58:35 INFO Vacuumed table table=pending_closures38882026/11/11 23:58:35 INFO Vacuumed table table=pending_objects38892026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038902026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads38912026/11/11 23:58:35 INFO Vacuumed table table=closures38922026/11/11 23:58:35 INFO Vacuumed table table=objects38932026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period38942026/11/11 23:58:35 INFO Received complete upload request method=POST path=/38952026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=038962026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038972026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period38982026/11/11 23:58:35 INFO Completed upload id=14 objects=8 duration=2.324519ms38992026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39002026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039012026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=039022026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39032026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39042026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39052026/11/11 23:58:35 INFO Received complete upload request method=POST path=/39062026/11/11 23:58:35 INFO Vacuumed table table=closures39072026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39082026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39092026/11/11 23:58:35 INFO Completed upload id=20 objects=5 duration=1.165849ms39102026/11/11 23:58:35 INFO Received complete push request method=POST path=/39112026/11/11 23:58:35 INFO Vacuumed table table=objects39122026/11/11 23:58:35 INFO Received push request method=POST path=/39132026/11/11 23:58:35 INFO Vacuumed table table=closures39142026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039152026/11/11 23:58:35 INFO Completed push id=20 objects=3 duration=1.088639ms39162026/11/11 23:58:35 INFO Vacuumed table table=objects39172026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39182026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39192026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39202026/11/11 23:58:35 INFO Received complete push request method=POST path=/39212026/11/11 23:58:35 INFO Vacuumed table table=closures39222026/11/11 23:58:35 INFO Vacuumed table table=objects39232026/11/11 23:58:35 INFO Completed push id=26 objects=14 duration=3.098789ms39242026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=039252026/11/11 23:58:35 INFO Received complete push request method=POST path=/39262026/11/11 23:58:35 INFO Received complete upload request method=POST path=/39272026/11/11 23:58:35 INFO Completed push id=16 objects=8 duration=2.077369ms39282026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period39292026/11/11 23:58:35 INFO Completed upload id=20 objects=4 duration=1.423309ms39302026/11/11 23:58:35 INFO Received uploads request method=POST path=/39312026/11/11 23:58:35 INFO Received uploads request method=POST path=/39322026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039332026/11/11 23:58:35 INFO Received uploads request method=POST path=/39342026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39352026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39362026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39372026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39382026/11/11 23:58:35 INFO Received uploads request method=POST path=/39392026/11/11 23:58:35 INFO Vacuumed table table=closures39402026/11/11 23:58:35 INFO Received push request method=POST path=/39412026/11/11 23:58:35 INFO Vacuumed table table=objects39422026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=039432026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period39442026/11/11 23:58:35 INFO Received push request method=POST path=/39452026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039462026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39472026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39482026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39492026/11/11 23:58:35 INFO Vacuumed table table=closures39502026/11/11 23:58:35 INFO Vacuumed table table=objects39512026/11/11 23:58:35 INFO Received uploads request method=POST path=/39522026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=039532026/11/11 23:58:35 INFO Received push request method=POST path=/39542026/11/11 23:58:35 INFO Received uploads request method=POST path=/39552026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period39562026/11/11 23:58:35 INFO Received uploads request method=POST path=/39572026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039582026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39592026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39602026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39612026/11/11 23:58:35 INFO Vacuumed table table=closures39622026/11/11 23:58:35 INFO Vacuumed table table=objects39632026/11/11 23:58:35 INFO Received push request method=POST path=/39642026/11/11 23:58:35 INFO Received uploads request method=POST path=/39652026/11/11 23:58:35 INFO Received complete push request method=POST path=/39662026/11/11 23:58:35 INFO Completed push id=16 objects=5 duration=1.494599ms39672026/11/11 23:58:35 INFO Received uploads request method=POST path=/39682026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=039692026/11/11 23:58:35 INFO Received push request method=POST path=/39702026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039712026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39722026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39732026/11/11 23:58:35 INFO Received uploads request method=POST path=/39742026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39752026/11/11 23:58:35 INFO Vacuumed table table=closures39762026/11/11 23:58:35 INFO Vacuumed table table=objects39772026/11/11 23:58:35 INFO Received uploads request method=POST path=/39782026/11/11 23:58:35 INFO Received complete upload request method=POST path=/39792026/11/11 23:58:35 INFO Received uploads request method=POST path=/39802026/11/11 23:58:35 INFO Completed upload id=13 objects=8 duration=1.975819ms39812026/11/11 23:58:35 INFO Received uploads request method=POST path=/39822026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=039832026/11/11 23:58:35 INFO Received complete push request method=POST path=/39842026/11/11 23:58:35 INFO Received complete push request method=POST path=/39852026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039862026/11/11 23:58:35 INFO Vacuumed table table=pending_closures39872026/11/11 23:58:35 INFO Vacuumed table table=pending_objects39882026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads39892026/11/11 23:58:35 INFO Vacuumed table table=closures39902026/11/11 23:58:35 INFO Received push request method=POST path=/39912026/11/11 23:58:35 INFO Completed push id=11 objects=6 duration=3.347448ms39922026/11/11 23:58:35 INFO Completed push id=16 objects=16 duration=4.037408ms39932026/11/11 23:58:35 INFO Vacuumed table table=objects39942026/11/11 23:58:35 INFO Received uploads request method=POST path=/39952026/11/11 23:58:35 INFO Received push request method=POST path=/39962026/11/11 23:58:35 INFO Received uploads request method=POST path=/39972026/11/11 23:58:35 INFO Received uploads request method=POST path=/39982026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=039992026/11/11 23:58:35 INFO Received uploads request method=POST path=/40002026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040012026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period40022026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040032026/11/11 23:58:35 INFO Vacuumed table table=pending_closures40042026/11/11 23:58:35 INFO Received push request method=POST path=/40052026/11/11 23:58:35 INFO Vacuumed table table=pending_objects40062026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads40072026/11/11 23:58:35 INFO Vacuumed table table=closures40082026/11/11 23:58:35 INFO Vacuumed table table=objects40092026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040102026/11/11 23:58:35 INFO Received complete upload request method=POST path=/40112026/11/11 23:58:35 INFO Received uploads request method=POST path=/40122026/11/11 23:58:35 INFO Vacuumed table table=pending_closures40132026/11/11 23:58:35 INFO Vacuumed table table=pending_objects40142026/11/11 23:58:35 INFO Completed upload id=33 objects=5 duration=2.857979ms40152026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads40162026/11/11 23:58:35 INFO Received complete push request method=POST path=/40172026/11/11 23:58:35 INFO Received uploads request method=POST path=/40182026/11/11 23:58:35 INFO Vacuumed table table=closures40192026/11/11 23:58:35 INFO Completed push id=18 objects=6 duration=1.31512ms40202026/11/11 23:58:35 INFO Vacuumed table table=objects40212026/11/11 23:58:35 INFO Received uploads request method=POST path=/40222026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040232026/11/11 23:58:35 INFO Received push request method=POST path=/40242026/11/11 23:58:35 INFO Received push request method=POST path=/40252026/11/11 23:58:35 INFO Received complete push request method=POST path=/40262026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040272026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040282026/11/11 23:58:35 INFO Received complete push request method=POST path=/40292026/11/11 23:58:35 INFO Completed push id=4 objects=6 duration=2.635428ms40302026/11/11 23:58:35 INFO Completed push id=21 objects=3 duration=1.477409ms40312026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040322026/11/11 23:58:35 INFO Vacuumed table table=pending_closures40332026/11/11 23:58:35 INFO Vacuumed table table=pending_objects40342026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads40352026/11/11 23:58:35 INFO Vacuumed table table=closures40362026/11/11 23:58:35 INFO Received complete upload request method=POST path=/40372026/11/11 23:58:35 INFO Vacuumed table table=objects40382026/11/11 23:58:35 INFO Completed upload id=23 objects=18 duration=1.589189ms40392026/11/11 23:58:35 INFO Received push request method=POST path=/40402026/11/11 23:58:35 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."40412026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."40422026/11/11 23:58:35 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40432026/11/11 23:58:35 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40442026/11/11 23:58:35 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40452026/11/11 23:58:35 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40462026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40472026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40482026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40492026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040502026/11/11 23:58:35 INFO Received uploads request method=POST path=/40512026/11/11 23:58:35 INFO Received push request method=POST path=/40522026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040532026/11/11 23:58:35 INFO Received complete push request method=POST path=/40542026/11/11 23:58:35 INFO Received complete upload request method=POST path=/40552026/11/11 23:58:35 INFO Vacuumed table table=pending_closures40562026/11/11 23:58:35 INFO Completed upload id=22 objects=6 duration=959.73µs40572026/11/11 23:58:35 INFO Vacuumed table table=pending_objects40582026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads40592026/11/11 23:58:35 INFO Completed push id=7 objects=15 duration=3.013788ms40602026/11/11 23:58:35 INFO Vacuumed table table=closures40612026/11/11 23:58:35 INFO Vacuumed table table=objects40622026/11/11 23:58:35 INFO Received push request method=POST path=/40632026/11/11 23:58:35 INFO Received uploads request method=POST path=/40642026/11/11 23:58:35 INFO Received uploads request method=POST path=/40652026/11/11 23:58:35 INFO Received uploads request method=POST path=/40662026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040672026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period40682026/11/11 23:58:35 INFO Received uploads request method=POST path=/40692026/11/11 23:58:35 INFO Received uploads request method=POST path=/40702026/11/11 23:58:35 INFO Received uploads request method=POST path=/40712026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040722026/11/11 23:58:35 INFO Received uploads request method=POST path=/40732026/11/11 23:58:35 INFO Vacuumed table table=pending_closures40742026/11/11 23:58:35 INFO Vacuumed table table=pending_objects40752026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads40762026/11/11 23:58:35 INFO Vacuumed table table=closures40772026/11/11 23:58:35 INFO Received uploads request method=POST path=/40782026/11/11 23:58:35 INFO Vacuumed table table=objects40792026/11/11 23:58:35 INFO Received complete upload request method=POST path=/40802026/11/11 23:58:35 INFO Received complete upload request method=POST path=/40812026/11/11 23:58:35 INFO Completed upload id=20 objects=3 duration=2.487069ms40822026/11/11 23:58:35 INFO Received push request method=POST path=/40832026/11/11 23:58:35 INFO Received push request method=POST path=/40842026/11/11 23:58:35 INFO Received uploads request method=POST path=/40852026/11/11 23:58:35 INFO Completed upload id=21 objects=25 duration=1.774309ms40862026/11/11 23:58:35 INFO Received push request method=POST path=/40872026/11/11 23:58:35 INFO Received uploads request method=POST path=/40882026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040892026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period40902026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=040912026/11/11 23:58:35 INFO Received complete push request method=POST path=/40922026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040932026/11/11 23:58:35 INFO Completed push id=18 objects=6 duration=3.519498ms40942026/11/11 23:58:35 INFO Received push request method=POST path=/40952026/11/11 23:58:35 INFO Vacuumed table table=pending_closures40962026/11/11 23:58:35 INFO Vacuumed table table=pending_objects40972026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads40982026/11/11 23:58:35 INFO Vacuumed table table=closures40992026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=041002026/11/11 23:58:35 INFO Vacuumed table table=objects41012026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41022026/11/11 23:58:35 INFO Received uploads request method=POST path=/41032026/11/11 23:58:35 INFO Vacuumed table table=pending_closures41042026/11/11 23:58:35 INFO Received push request method=POST path=/41052026/11/11 23:58:35 INFO Vacuumed table table=pending_objects41062026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads41072026/11/11 23:58:35 INFO Completed upload id=16 objects=17 duration=2.844988ms41082026/11/11 23:58:35 INFO Vacuumed table table=closures41092026/11/11 23:58:35 INFO Received push request method=POST path=/41102026/11/11 23:58:35 INFO Vacuumed table table=objects41112026/11/11 23:58:35 INFO Received complete push request method=POST path=/41122026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=041132026/11/11 23:58:35 INFO Completed push id=5 objects=28 duration=2.339579ms41142026/11/11 23:58:35 INFO Received push request method=POST path=/41152026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period41162026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=041172026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41182026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041192026/11/11 23:58:35 INFO Received uploads request method=POST path=/41202026/11/11 23:58:35 INFO Completed upload id=22 objects=22 duration=1.504639ms41212026/11/11 23:58:35 INFO Vacuumed table table=pending_closures41222026/11/11 23:58:35 INFO Vacuumed table table=pending_objects41232026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads41242026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041252026/11/11 23:58:35 INFO Vacuumed table table=closures41262026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41272026/11/11 23:58:35 INFO Vacuumed table table=pending_closures41282026/11/11 23:58:35 INFO Vacuumed table table=objects41292026/11/11 23:58:35 INFO Completed upload id=23 objects=9 duration=1.521179ms41302026/11/11 23:58:35 INFO Received push request method=POST path=/41312026/11/11 23:58:35 INFO Vacuumed table table=pending_objects41322026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads41332026/11/11 23:58:35 INFO Vacuumed table table=closures41342026/11/11 23:58:35 INFO Received push request method=POST path=/41352026/11/11 23:58:35 INFO Vacuumed table table=objects41362026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=041372026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041382026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41392026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=041402026/11/11 23:58:35 INFO Vacuumed table table=pending_closures41412026/11/11 23:58:35 INFO Vacuumed table table=pending_objects41422026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads41432026/11/11 23:58:35 INFO Completed upload id=21 objects=7 duration=2.508288ms41442026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period41452026/11/11 23:58:35 INFO Vacuumed table table=closures41462026/11/11 23:58:35 INFO Vacuumed table table=objects41472026/11/11 23:58:35 INFO Received complete push request method=POST path=/41482026/11/11 23:58:35 INFO Completed push id=10 objects=2 duration=1.660669ms41492026/11/11 23:58:35 INFO Received push request method=POST path=/41502026/11/11 23:58:35 INFO Received push request method=POST path=/41512026/11/11 23:58:35 INFO Received uploads request method=POST path=/41522026/11/11 23:58:35 INFO Received push request method=POST path=/41532026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=041542026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41552026/11/11 23:58:35 INFO Received uploads request method=POST path=/41562026/11/11 23:58:35 INFO Completed upload id=24 objects=7 duration=1.078749ms41572026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041582026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41592026/11/11 23:58:35 INFO Vacuumed table table=pending_closures41602026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41612026/11/11 23:58:35 INFO Vacuumed table table=pending_objects41622026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads41632026/11/11 23:58:35 INFO Completed upload id=12 objects=6 duration=1.309609ms41642026/11/11 23:58:35 INFO Received push request method=POST path=/41652026/11/11 23:58:35 INFO Vacuumed table table=closures41662026/11/11 23:58:35 INFO Completed upload id=14 objects=16 duration=2.935028ms41672026/11/11 23:58:35 INFO Vacuumed table table=objects41682026/11/11 23:58:35 INFO Received complete push request method=POST path=/41692026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41702026/11/11 23:58:35 INFO Completed push id=26 objects=5 duration=1.245619ms41712026/11/11 23:58:35 INFO Completed upload id=17 objects=5 duration=1.191729ms41722026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=041732026/11/11 23:58:35 INFO Received push request method=POST path=/41742026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period41752026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041762026/11/11 23:58:35 INFO Received push request method=POST path=/41772026/11/11 23:58:35 INFO Vacuumed table table=pending_closures41782026/11/11 23:58:35 INFO Vacuumed table table=pending_objects41792026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads41802026/11/11 23:58:35 INFO Vacuumed table table=closures41812026/11/11 23:58:35 INFO Received push request method=POST path=/41822026/11/11 23:58:35 INFO Vacuumed table table=objects41832026/11/11 23:58:35 INFO Received push request method=POST path=/41842026/11/11 23:58:35 INFO Received push request method=POST path=/41852026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=041862026/11/11 23:58:35 INFO Received complete push request method=POST path=/41872026/11/11 23:58:35 INFO Received uploads request method=POST path=/41882026/11/11 23:58:35 INFO Completed push id=17 objects=3 duration=2.504748ms41892026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041902026/11/11 23:58:35 INFO Vacuumed table table=pending_closures41912026/11/11 23:58:35 INFO Vacuumed table table=pending_objects41922026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads41932026/11/11 23:58:35 INFO Vacuumed table table=closures41942026/11/11 23:58:35 INFO Received complete upload request method=POST path=/41952026/11/11 23:58:35 INFO Vacuumed table table=objects41962026/11/11 23:58:35 INFO Completed upload id=24 objects=14 duration=784.25µs41972026/11/11 23:58:35 INFO Received uploads request method=POST path=/41982026/11/11 23:58:35 INFO Received complete push request method=POST path=/41992026/11/11 23:58:35 INFO Completed push id=19 objects=3 duration=1.254989ms42002026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=042012026/11/11 23:58:35 INFO Received push request method=POST path=/42022026/11/11 23:58:35 INFO Received complete push request method=POST path=/42032026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period42042026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042052026/11/11 23:58:35 INFO Vacuumed table table=pending_closures42062026/11/11 23:58:35 INFO Completed push id=30 objects=4 duration=3.440958ms42072026/11/11 23:58:35 INFO Vacuumed table table=pending_objects42082026/11/11 23:58:35 INFO Received uploads request method=POST path=/42092026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads42102026/11/11 23:58:35 INFO Vacuumed table table=closures42112026/11/11 23:58:35 INFO Received complete push request method=POST path=/42122026/11/11 23:58:35 INFO Vacuumed table table=objects42132026/11/11 23:58:35 INFO Received complete push request method=POST path=/42142026/11/11 23:58:35 INFO Completed push id=27 objects=25 duration=3.296168ms42152026/11/11 23:58:35 INFO Completed push id=23 objects=5 duration=1.59305ms42162026/11/11 23:58:35 INFO Received complete push request method=POST path=/42172026/11/11 23:58:35 INFO Completed push id=19 objects=3 duration=847.48µs42182026/11/11 23:58:35 INFO Received uploads request method=POST path=/42192026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=042202026/11/11 23:58:35 INFO Vacuumed table table=pending_closures42212026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=042222026/11/11 23:58:35 INFO Vacuumed table table=pending_objects42232026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads42242026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=042252026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period42262026/11/11 23:58:35 INFO Vacuumed table table=closures42272026/11/11 23:58:35 INFO Received push request method=POST path=/42282026/11/11 23:58:35 INFO Vacuumed table table=objects42292026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period42302026/11/11 23:58:35 INFO Received uploads request method=POST path=/42312026/11/11 23:58:35 INFO Received complete push request method=POST path=/42322026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042332026/11/11 23:58:35 INFO Vacuumed table table=pending_closures42342026/11/11 23:58:35 INFO Completed push id=17 objects=11 duration=2.678739ms42352026/11/11 23:58:35 INFO Vacuumed table table=pending_objects42362026/11/11 23:58:35 INFO Received uploads request method=POST path=/42372026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads42382026/11/11 23:58:35 INFO Received uploads request method=POST path=/42392026/11/11 23:58:35 INFO Vacuumed table table=closures42402026/11/11 23:58:35 INFO Vacuumed table table=objects42412026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=042422026/11/11 23:58:35 INFO Received complete upload request method=POST path=/42432026/11/11 23:58:35 INFO Received complete push request method=POST path=/42442026/11/11 23:58:35 INFO Completed upload id=24 objects=3 duration=1.262649ms42452026/11/11 23:58:35 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."42462026/11/11 23:58:35 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."42472026/11/11 23:58:35 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."42482026/11/11 23:58:35 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."42492026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42502026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."42512026/11/11 23:58:35 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42522026/11/11 23:58:35 INFO Completed push id=20 objects=4 duration=1.20364ms42532026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042542026/11/11 23:58:35 INFO Vacuumed table table=pending_closures42552026/11/11 23:58:35 INFO Vacuumed table table=pending_objects42562026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads42572026/11/11 23:58:35 INFO Vacuumed table table=closures42582026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=042592026/11/11 23:58:35 INFO Vacuumed table table=objects42602026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period42612026/11/11 23:58:35 INFO Received complete upload request method=POST path=/42622026/11/11 23:58:35 INFO Received uploads request method=POST path=/42632026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042642026/11/11 23:58:35 INFO Received complete push request method=POST path=/42652026/11/11 23:58:35 INFO Vacuumed table table=pending_closures42662026/11/11 23:58:35 INFO Received push request method=POST path=/42672026/11/11 23:58:35 INFO Completed upload id=17 objects=25 duration=3.475998ms42682026/11/11 23:58:35 INFO Vacuumed table table=pending_objects42692026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads42702026/11/11 23:58:35 INFO Received uploads request method=POST path=/42712026/11/11 23:58:35 INFO Completed push id=19 objects=14 duration=2.150759ms42722026/11/11 23:58:35 INFO Vacuumed table table=closures42732026/11/11 23:58:35 INFO Received push request method=POST path=/42742026/11/11 23:58:35 INFO Vacuumed table table=objects42752026/11/11 23:58:35 INFO Received uploads request method=POST path=/42762026/11/11 23:58:35 INFO Received push request method=POST path=/42772026/11/11 23:58:35 INFO Received complete push request method=POST path=/42782026/11/11 23:58:35 INFO Completed push id=25 objects=4 duration=1.334479ms42792026/11/11 23:58:35 INFO Received uploads request method=POST path=/42802026/11/11 23:58:35 INFO Received complete push request method=POST path=/42812026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=042822026/11/11 23:58:35 INFO Received push request method=POST path=/42832026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period42842026/11/11 23:58:35 INFO Completed push id=24 objects=5 duration=2.971248ms42852026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042862026/11/11 23:58:35 INFO Received push request method=POST path=/42872026/11/11 23:58:35 INFO Vacuumed table table=pending_closures42882026/11/11 23:58:35 INFO Vacuumed table table=pending_objects42892026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads42902026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=042912026/11/11 23:58:35 INFO Vacuumed table table=closures42922026/11/11 23:58:35 INFO Received complete upload request method=POST path=/42932026/11/11 23:58:35 INFO Vacuumed table table=objects42942026/11/11 23:58:35 INFO Completed upload id=27 objects=4 duration=1.501939ms42952026/11/11 23:58:35 INFO Received push request method=POST path=/42962026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period42972026/11/11 23:58:35 INFO Received uploads request method=POST path=/42982026/11/11 23:58:35 INFO Received push request method=POST path=/42992026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43002026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43012026/11/11 23:58:35 INFO Completed upload id=22 objects=8 duration=1.141219ms43022026/11/11 23:58:35 INFO Received push request method=POST path=/43032026/11/11 23:58:35 INFO Completed upload id=21 objects=3 duration=1.523249ms43042026/11/11 23:58:35 INFO Received complete push request method=POST path=/43052026/11/11 23:58:35 INFO Completed push id=23 objects=28 duration=3.458128ms43062026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43072026/11/11 23:58:35 INFO Completed upload id=15 objects=3 duration=1.873588ms43082026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43092026/11/11 23:58:35 INFO Completed upload id=21 objects=2 duration=1.32183ms43102026/11/11 23:58:35 INFO Received complete push request method=POST path=/43112026/11/11 23:58:35 INFO Received complete push request method=POST path=/43122026/11/11 23:58:35 INFO Completed push id=13 objects=11 duration=1.625979ms43132026/11/11 23:58:35 INFO Completed push id=16 objects=26 duration=1.52787ms43142026/11/11 23:58:35 INFO Received push request method=POST path=/43152026/11/11 23:58:35 INFO Received push request method=POST path=/43162026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=043172026/11/11 23:58:35 INFO Received complete push request method=POST path=/43182026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period43192026/11/11 23:58:35 INFO Completed push id=23 objects=5 duration=1.02707ms43202026/11/11 23:58:35 INFO Received complete push request method=POST path=/43212026/11/11 23:58:35 INFO Completed push id=32 objects=7 duration=1.3157ms43222026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=043232026/11/11 23:58:35 INFO Received uploads request method=POST path=/43242026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043252026/11/11 23:58:35 INFO Vacuumed table table=pending_closures43262026/11/11 23:58:35 INFO Vacuumed table table=pending_objects43272026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads43282026/11/11 23:58:35 INFO Received uploads request method=POST path=/43292026/11/11 23:58:35 INFO Vacuumed table table=closures43302026/11/11 23:58:35 INFO Vacuumed table table=objects43312026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43322026/11/11 23:58:35 INFO Received uploads request method=POST path=/43332026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=043342026/11/11 23:58:35 INFO Completed upload id=26 objects=6 duration=1.218999ms43352026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period43362026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=043372026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043382026/11/11 23:58:35 INFO Received complete push request method=POST path=/43392026/11/11 23:58:35 INFO Vacuumed table table=pending_closures43402026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=043412026/11/11 23:58:35 INFO Vacuumed table table=pending_closures43422026/11/11 23:58:35 INFO Completed push id=26 objects=23 duration=1.590779ms43432026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=043442026/11/11 23:58:35 INFO Received complete push request method=POST path=/43452026/11/11 23:58:35 INFO Vacuumed table table=pending_objects43462026/11/11 23:58:35 INFO Vacuumed table table=pending_objects43472026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads43482026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads43492026/11/11 23:58:35 INFO Vacuumed table table=pending_closures43502026/11/11 23:58:35 INFO Vacuumed table table=closures43512026/11/11 23:58:35 INFO Vacuumed table table=closures43522026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period43532026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43542026/11/11 23:58:35 INFO Completed push id=31 objects=14 duration=3.140288ms43552026/11/11 23:58:35 INFO Vacuumed table table=objects43562026/11/11 23:58:35 INFO Vacuumed table table=objects43572026/11/11 23:58:35 INFO Completed upload id=24 objects=8 duration=1.211099ms43582026/11/11 23:58:35 INFO Received complete push request method=POST path=/43592026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043602026/11/11 23:58:35 INFO Vacuumed table table=pending_objects43612026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads43622026/11/11 23:58:35 INFO Vacuumed table table=pending_closures43632026/11/11 23:58:35 INFO Received push request method=POST path=/43642026/11/11 23:58:35 INFO Completed push id=26 objects=4 duration=1.616299ms43652026/11/11 23:58:35 INFO Vacuumed table table=closures43662026/11/11 23:58:35 INFO Vacuumed table table=pending_objects43672026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads43682026/11/11 23:58:35 INFO Vacuumed table table=objects43692026/11/11 23:58:35 INFO Vacuumed table table=closures43702026/11/11 23:58:35 INFO Vacuumed table table=objects43712026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43722026/11/11 23:58:35 INFO Completed upload id=27 objects=8 duration=1.131509ms43732026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=043742026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period43752026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043762026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43772026/11/11 23:58:35 INFO Vacuumed table table=pending_closures43782026/11/11 23:58:35 INFO Completed upload id=25 objects=4 duration=867.88µs43792026/11/11 23:58:35 INFO Vacuumed table table=pending_objects43802026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads43812026/11/11 23:58:35 INFO Received uploads request method=POST path=/43822026/11/11 23:58:35 INFO Vacuumed table table=closures43832026/11/11 23:58:35 INFO Vacuumed table table=objects43842026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=043852026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period43862026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043872026/11/11 23:58:35 INFO Vacuumed table table=pending_closures43882026/11/11 23:58:35 INFO Vacuumed table table=pending_objects43892026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads43902026/11/11 23:58:35 INFO Vacuumed table table=closures43912026/11/11 23:58:35 INFO Vacuumed table table=objects43922026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43932026/11/11 23:58:35 INFO Completed upload id=28 objects=5 duration=1.06347ms43942026/11/11 23:58:35 INFO Received complete upload request method=POST path=/43952026/11/11 23:58:35 INFO Completed upload id=28 objects=13 duration=1.05936ms43962026/11/11 23:58:35 INFO Received push request method=POST path=/43972026/11/11 23:58:35 INFO Received complete push request method=POST path=/43982026/11/11 23:58:35 INFO Received complete push request method=POST path=/43992026/11/11 23:58:35 INFO Received push request method=POST path=/44002026/11/11 23:58:35 INFO Completed push id=21 objects=26 duration=1.759209ms44012026/11/11 23:58:35 INFO Received complete upload request method=POST path=/44022026/11/11 23:58:35 INFO Completed push id=27 objects=6 duration=3.632408ms44032026/11/11 23:58:35 INFO Received push request method=POST path=/44042026/11/11 23:58:35 INFO Completed upload id=19 objects=2 duration=2.325348ms44052026/11/11 23:58:35 INFO Received complete upload request method=POST path=/44062026/11/11 23:58:35 INFO Received complete push request method=POST path=/44072026/11/11 23:58:35 INFO Completed upload id=29 objects=5 duration=1.12113ms44082026/11/11 23:58:35 INFO Completed push id=27 objects=15 duration=1.272019ms44092026/11/11 23:58:35 INFO Received complete push request method=POST path=/44102026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=044112026/11/11 23:58:35 INFO Completed push id=16 objects=7 duration=1.377639ms44122026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period44132026/11/11 23:58:35 INFO Received uploads request method=POST path=/44142026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044152026/11/11 23:58:35 INFO Vacuumed table table=pending_closures44162026/11/11 23:58:35 INFO Vacuumed table table=pending_objects44172026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads44182026/11/11 23:58:35 INFO Vacuumed table table=closures44192026/11/11 23:58:35 INFO Vacuumed table table=objects44202026/11/11 23:58:35 INFO Received uploads request method=POST path=/44212026/11/11 23:58:35 INFO Received complete push request method=POST path=/44222026/11/11 23:58:35 INFO Received uploads request method=POST path=/44232026/11/11 23:58:35 INFO Completed push id=29 objects=3 duration=1.23706ms44242026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=044252026/11/11 23:58:35 INFO Received complete upload request method=POST path=/44262026/11/11 23:58:35 INFO Completed upload id=30 objects=5 duration=935.44µs44272026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044282026/11/11 23:58:35 INFO Received complete upload request method=POST path=/44292026/11/11 23:58:35 INFO Vacuumed table table=pending_closures44302026/11/11 23:58:35 INFO Completed upload id=28 objects=17 duration=696.079µs44312026/11/11 23:58:35 INFO Vacuumed table table=pending_objects44322026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads44332026/11/11 23:58:35 INFO Vacuumed table table=closures44342026/11/11 23:58:35 INFO Vacuumed table table=objects44352026/11/11 23:58:35 INFO Received uploads request method=POST path=/44362026/11/11 23:58:35 INFO Received complete push request method=POST path=/44372026/11/11 23:58:35 INFO Received push request method=POST path=/44382026/11/11 23:58:35 INFO Completed push id=17 objects=22 duration=2.821148ms44392026/11/11 23:58:35 INFO Received complete push request method=POST path=/44402026/11/11 23:58:35 INFO Received complete push request method=POST path=/44412026/11/11 23:58:35 INFO Received complete upload request method=POST path=/44422026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=044432026/11/11 23:58:35 INFO Completed push id=33 objects=7 duration=1.953268ms44442026/11/11 23:58:35 INFO Received complete push request method=POST path=/44452026/11/11 23:58:35 INFO Completed upload id=31 objects=2 duration=1.178379ms44462026/11/11 23:58:35 INFO Completed push id=23 objects=6 duration=1.282579ms44472026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044482026/11/11 23:58:35 INFO Received complete upload request method=POST path=/44492026/11/11 23:58:35 INFO Completed push id=7 objects=24 duration=3.957738ms44502026/11/11 23:58:35 INFO Vacuumed table table=pending_closures44512026/11/11 23:58:35 INFO Completed upload id=15 objects=2 duration=1.086189ms44522026/11/11 23:58:35 INFO Received complete push request method=POST path=/44532026/11/11 23:58:35 INFO Vacuumed table table=pending_objects44542026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads44552026/11/11 23:58:35 INFO Completed push id=20 objects=8 duration=775.82µs44562026/11/11 23:58:35 INFO Vacuumed table table=closures44572026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=044582026/11/11 23:58:35 INFO Vacuumed table table=objects44592026/11/11 23:58:35 INFO Received complete push request method=POST path=/44602026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period44612026/11/11 23:58:35 INFO Completed push id=29 objects=2 duration=1.164359ms44622026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044632026/11/11 23:58:35 INFO Received complete push request method=POST path=/44642026/11/11 23:58:35 INFO Vacuumed table table=pending_closures44652026/11/11 23:58:35 INFO Completed push id=31 objects=3 duration=1.12566ms44662026/11/11 23:58:35 INFO Vacuumed table table=pending_objects44672026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads44682026/11/11 23:58:35 INFO Vacuumed table table=closures44692026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=044702026/11/11 23:58:35 INFO Received complete push request method=POST path=/44712026/11/11 23:58:35 INFO Vacuumed table table=objects44722026/11/11 23:58:35 INFO Received uploads request method=POST path=/44732026/11/11 23:58:35 INFO Received complete push request method=POST path=/44742026/11/11 23:58:35 INFO Completed push id=28 objects=19 duration=1.198389ms44752026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044762026/11/11 23:58:35 INFO Received complete upload request method=POST path=/44772026/11/11 23:58:35 INFO Completed push id=22 objects=3 duration=3.769638ms44782026/11/11 23:58:35 INFO Vacuumed table table=pending_closures44792026/11/11 23:58:35 INFO Completed upload id=26 objects=3 duration=1.090889ms44802026/11/11 23:58:35 INFO Received complete push request method=POST path=/44812026/11/11 23:58:35 INFO Vacuumed table table=pending_objects44822026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads44832026/11/11 23:58:35 INFO Received complete push request method=POST path=/44842026/11/11 23:58:35 INFO Vacuumed table table=closures44852026/11/11 23:58:35 INFO Completed push id=24 objects=6 duration=1.644809ms44862026/11/11 23:58:35 INFO Completed push id=33 objects=6 duration=1.377599ms44872026/11/11 23:58:35 INFO Vacuumed table table=objects44882026/11/11 23:58:35 INFO Received complete push request method=POST path=/44892026/11/11 23:58:35 INFO Received complete push request method=POST path=/44902026/11/11 23:58:35 INFO Received uploads request method=POST path=/44912026/11/11 23:58:35 INFO Completed push id=16 objects=14 duration=2.599359ms44922026/11/11 23:58:35 INFO Completed push id=7 objects=7 duration=2.009299ms44932026/11/11 23:58:35 INFO Received uploads request method=POST path=/44942026/11/11 23:58:35 INFO Received complete push request method=POST path=/44952026/11/11 23:58:35 INFO Completed push id=31 objects=6 duration=1.25891ms44962026/11/11 23:58:35 INFO Received push request method=POST path=/44972026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=044982026/11/11 23:58:35 INFO Received uploads request method=POST path=/44992026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period45002026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045012026/11/11 23:58:35 INFO Vacuumed table table=pending_closures45022026/11/11 23:58:35 INFO Vacuumed table table=pending_objects45032026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads45042026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=045052026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45062026/11/11 23:58:35 INFO Vacuumed table table=closures45072026/11/11 23:58:35 INFO Vacuumed table table=objects45082026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period45092026/11/11 23:58:35 INFO Completed upload id=25 objects=16 duration=2.175019ms45102026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=045112026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045122026/11/11 23:58:35 INFO Received complete push request method=POST path=/45132026/11/11 23:58:35 INFO Completed push id=12 objects=6 duration=1.4754ms45142026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period45152026/11/11 23:58:35 INFO Vacuumed table table=pending_closures45162026/11/11 23:58:35 INFO Received complete push request method=POST path=/45172026/11/11 23:58:35 INFO Vacuumed table table=pending_objects45182026/11/11 23:58:35 INFO Received uploads request method=POST path=/45192026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads45202026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045212026/11/11 23:58:35 INFO Completed push id=32 objects=6 duration=1.326449ms45222026/11/11 23:58:35 INFO Received push request method=POST path=/45232026/11/11 23:58:35 INFO Vacuumed table table=closures45242026/11/11 23:58:35 INFO Received complete push request method=POST path=/45252026/11/11 23:58:35 INFO Vacuumed table table=pending_closures45262026/11/11 23:58:35 INFO Received complete push request method=POST path=/45272026/11/11 23:58:35 INFO Vacuumed table table=objects45282026/11/11 23:58:35 INFO Completed push id=29 objects=10 duration=1.489819ms45292026/11/11 23:58:35 INFO Received complete push request method=POST path=/45302026/11/11 23:58:35 INFO Vacuumed table table=pending_objects45312026/11/11 23:58:35 INFO Completed push id=25 objects=7 duration=1.825539ms45322026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads45332026/11/11 23:58:35 INFO Completed push id=32 objects=9 duration=1.33119ms45342026/11/11 23:58:35 INFO Vacuumed table table=closures45352026/11/11 23:58:35 INFO Vacuumed table table=objects45362026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=045372026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period45382026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045392026/11/11 23:58:35 INFO Vacuumed table table=pending_closures45402026/11/11 23:58:35 INFO Vacuumed table table=pending_objects45412026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads45422026/11/11 23:58:35 INFO Vacuumed table table=closures45432026/11/11 23:58:35 INFO Vacuumed table table=objects45442026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45452026/11/11 23:58:35 INFO Completed upload id=33 objects=7 duration=907.839µs45462026/11/11 23:58:35 INFO Received uploads request method=POST path=/45472026/11/11 23:58:35 INFO Received uploads request method=POST path=/45482026/11/11 23:58:35 INFO Received push request method=POST path=/45492026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=045502026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45512026/11/11 23:58:35 INFO Completed upload id=30 objects=6 duration=1.028ms45522026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45532026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045542026/11/11 23:58:35 INFO Completed upload id=33 objects=9 duration=1.056379ms45552026/11/11 23:58:35 INFO Vacuumed table table=pending_closures45562026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45572026/11/11 23:58:35 INFO Vacuumed table table=pending_objects45582026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads45592026/11/11 23:58:35 INFO Vacuumed table table=closures45602026/11/11 23:58:35 INFO Completed upload id=13 objects=14 duration=3.016728ms45612026/11/11 23:58:35 INFO Vacuumed table table=objects45622026/11/11 23:58:35 INFO Received push request method=POST path=/45632026/11/11 23:58:35 INFO Received push request method=POST path=/45642026/11/11 23:58:35 INFO Received push request method=POST path=/45652026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45662026/11/11 23:58:35 INFO Received complete push request method=POST path=/45672026/11/11 23:58:35 INFO Completed upload id=34 objects=3 duration=1.242889ms45682026/11/11 23:58:35 INFO Received push request method=POST path=/45692026/11/11 23:58:35 INFO Completed push id=27 objects=11 duration=936.69µs45702026/11/11 23:58:35 INFO Received complete push request method=POST path=/45712026/11/11 23:58:35 INFO Received complete push request method=POST path=/45722026/11/11 23:58:35 INFO Completed push id=24 objects=3 duration=1.377529ms45732026/11/11 23:58:35 INFO Completed push id=13 objects=22 duration=2.017268ms45742026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45752026/11/11 23:58:35 INFO Completed upload id=26 objects=24 duration=1.32058ms45762026/11/11 23:58:35 INFO Received complete upload request method=POST path=/45772026/11/11 23:58:35 INFO Received complete push request method=POST path=/45782026/11/11 23:58:35 INFO Completed upload id=31 objects=5 duration=903.779µs45792026/11/11 23:58:35 INFO Completed push id=9 objects=9 duration=3.313848ms45802026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=045812026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period45822026/11/11 23:58:35 INFO Received push request method=POST path=/45832026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=045842026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045852026/11/11 23:58:35 INFO Vacuumed table table=pending_closures45862026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045872026/11/11 23:58:35 INFO Vacuumed table table=pending_objects45882026/11/11 23:58:35 INFO Received uploads request method=POST path=/45892026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads45902026/11/11 23:58:35 INFO Vacuumed table table=pending_closures45912026/11/11 23:58:35 INFO Vacuumed table table=closures45922026/11/11 23:58:35 INFO Vacuumed table table=objects45932026/11/11 23:58:35 INFO Vacuumed table table=pending_objects45942026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads45952026/11/11 23:58:35 INFO Received uploads request method=POST path=/45962026/11/11 23:58:35 INFO Vacuumed table table=closures45972026/11/11 23:58:35 INFO Vacuumed table table=objects45982026/11/11 23:58:35 INFO Received complete push request method=POST path=/45992026/11/11 23:58:35 INFO Completed push id=35 objects=11 duration=1.15507ms46002026/11/11 23:58:35 INFO Received complete push request method=POST path=/46012026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46022026/11/11 23:58:35 INFO Received push request method=POST path=/46032026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46042026/11/11 23:58:35 INFO Completed upload id=29 objects=26 duration=739.619µs46052026/11/11 23:58:35 INFO Completed push id=19 objects=14 duration=1.950159ms46062026/11/11 23:58:35 INFO Received push request method=POST path=/46072026/11/11 23:58:35 INFO Received complete push request method=POST path=/46082026/11/11 23:58:35 INFO Completed upload id=31 objects=3 duration=2.422468ms46092026/11/11 23:58:35 INFO Completed push id=21 objects=6 duration=1.669639ms46102026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46112026/11/11 23:58:35 INFO Completed upload id=34 objects=3 duration=891.12µs46122026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46132026/11/11 23:58:35 INFO Completed upload id=21 objects=8 duration=1.230709ms46142026/11/11 23:58:35 INFO Received uploads request method=POST path=/46152026/11/11 23:58:35 INFO Received uploads request method=POST path=/46162026/11/11 23:58:35 INFO Received complete push request method=POST path=/46172026/11/11 23:58:35 INFO Completed push id=13 objects=2 duration=1.630049ms46182026/11/11 23:58:35 INFO Received complete push request method=POST path=/46192026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46202026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=046212026/11/11 23:58:35 INFO Received uploads request method=POST path=/46222026/11/11 23:58:35 INFO Completed push id=30 objects=8 duration=866.02µs46232026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46242026/11/11 23:58:35 INFO Completed upload id=22 objects=2 duration=1.308069ms46252026/11/11 23:58:35 INFO Received complete push request method=POST path=/46262026/11/11 23:58:35 INFO Completed upload id=22 objects=7 duration=1.221389ms46272026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period46282026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046292026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=046302026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46312026/11/11 23:58:35 INFO Completed push id=31 objects=6 duration=3.214318ms46322026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period46332026/11/11 23:58:35 INFO Vacuumed table table=pending_closures46342026/11/11 23:58:35 INFO Received complete push request method=POST path=/46352026/11/11 23:58:35 INFO Completed upload id=27 objects=26 duration=1.560559ms46362026/11/11 23:58:35 INFO Completed push id=22 objects=6 duration=1.119309ms46372026/11/11 23:58:35 INFO Vacuumed table table=pending_objects46382026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads46392026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046402026/11/11 23:58:35 INFO Vacuumed table table=closures46412026/11/11 23:58:35 INFO Vacuumed table table=objects46422026/11/11 23:58:35 INFO Vacuumed table table=pending_closures46432026/11/11 23:58:35 INFO Vacuumed table table=pending_objects46442026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46452026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads46462026/11/11 23:58:35 INFO Vacuumed table table=closures46472026/11/11 23:58:35 INFO Completed upload id=25 objects=6 duration=1.615729ms46482026/11/11 23:58:35 INFO Vacuumed table table=objects46492026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46502026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=046512026/11/11 23:58:35 INFO Completed upload id=26 objects=8 duration=1.264359ms46522026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period46532026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46542026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046552026/11/11 23:58:35 INFO Completed upload id=27 objects=13 duration=1.503929ms46562026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46572026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=046582026/11/11 23:58:35 INFO Vacuumed table table=pending_closures46592026/11/11 23:58:35 INFO Completed upload id=31 objects=5 duration=1.805739ms46602026/11/11 23:58:35 INFO Received complete push request method=POST path=/46612026/11/11 23:58:35 INFO Vacuumed table table=pending_objects46622026/11/11 23:58:35 INFO Completed push id=24 objects=4 duration=1.36859ms46632026/11/11 23:58:35 INFO Received uploads request method=POST path=/46642026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads46652026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period46662026/11/11 23:58:35 INFO Received complete push request method=POST path=/46672026/11/11 23:58:35 INFO Vacuumed table table=closures46682026/11/11 23:58:35 INFO Vacuumed table table=objects46692026/11/11 23:58:35 INFO Completed push id=24 objects=8 duration=1.405269ms46702026/11/11 23:58:35 INFO Received complete push request method=POST path=/46712026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046722026/11/11 23:58:35 INFO Completed push id=28 objects=6 duration=1.420459ms46732026/11/11 23:58:35 INFO Vacuumed table table=pending_closures46742026/11/11 23:58:35 INFO Vacuumed table table=pending_objects46752026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads46762026/11/11 23:58:35 INFO Vacuumed table table=closures46772026/11/11 23:58:35 INFO Vacuumed table table=objects46782026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46792026/11/11 23:58:35 INFO Completed upload id=20 objects=6 duration=1.209819ms46802026/11/11 23:58:35 INFO Received complete push request method=POST path=/46812026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=046822026/11/11 23:58:35 INFO Completed push id=28 objects=14 duration=1.333589ms46832026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046842026/11/11 23:58:35 INFO Received complete push request method=POST path=/46852026/11/11 23:58:35 INFO Vacuumed table table=pending_closures46862026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46872026/11/11 23:58:35 INFO Received complete push request method=POST path=/46882026/11/11 23:58:35 INFO Completed push id=32 objects=3 duration=960.72µs46892026/11/11 23:58:35 INFO Completed upload id=25 objects=9 duration=1.253899ms46902026/11/11 23:58:35 INFO Completed push id=25 objects=9 duration=1.630369ms46912026/11/11 23:58:35 INFO Vacuumed table table=pending_objects46922026/11/11 23:58:35 INFO Received uploads request method=POST path=/46932026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads46942026/11/11 23:58:35 INFO Vacuumed table table=closures46952026/11/11 23:58:35 INFO Received complete push request method=POST path=/46962026/11/11 23:58:35 INFO Vacuumed table table=objects46972026/11/11 23:58:35 INFO Completed push id=29 objects=6 duration=1.048759ms46982026/11/11 23:58:35 INFO Received complete upload request method=POST path=/46992026/11/11 23:58:35 INFO Completed upload id=18 objects=2 duration=2.116439ms47002026/11/11 23:58:35 INFO Received push request method=POST path=/47012026/11/11 23:58:35 INFO Received push request method=POST path=/47022026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47032026/11/11 23:58:35 INFO Completed upload id=27 objects=3 duration=1.139019ms47042026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=047052026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47062026/11/11 23:58:35 INFO Completed upload id=26 objects=3 duration=1.132949ms47072026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period47082026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47092026/11/11 23:58:35 INFO Received uploads request method=POST path=/47102026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047112026/11/11 23:58:35 INFO Received push request method=POST path=/47122026/11/11 23:58:35 INFO Completed upload id=22 objects=5 duration=2.018309ms47132026/11/11 23:58:35 INFO Vacuumed table table=pending_closures47142026/11/11 23:58:35 INFO Received uploads request method=POST path=/47152026/11/11 23:58:35 INFO Vacuumed table table=pending_objects47162026/11/11 23:58:35 INFO Received uploads request method=POST path=/47172026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads47182026/11/11 23:58:35 INFO Vacuumed table table=closures47192026/11/11 23:58:35 INFO Vacuumed table table=objects47202026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47212026/11/11 23:58:35 INFO Completed upload id=28 objects=6 duration=1.09158ms47222026/11/11 23:58:35 INFO Received uploads request method=POST path=/47232026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=047242026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47252026/11/11 23:58:35 INFO Received complete push request method=POST path=/47262026/11/11 23:58:35 INFO Completed upload id=30 objects=8 duration=1.340609ms47272026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47282026/11/11 23:58:35 INFO Completed push id=28 objects=3 duration=1.924499ms47292026/11/11 23:58:35 INFO Completed upload id=27 objects=7 duration=1.15279ms47302026/11/11 23:58:35 INFO Received push request method=POST path=/47312026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=047322026/11/11 23:58:35 INFO Received complete push request method=POST path=/47332026/11/11 23:58:35 INFO Vacuumed table table=pending_closures47342026/11/11 23:58:35 INFO Received complete push request method=POST path=/47352026/11/11 23:58:35 INFO Completed push id=11 objects=9 duration=1.806598ms47362026/11/11 23:58:35 INFO Completed push id=30 objects=4 duration=1.304399ms47372026/11/11 23:58:35 INFO Vacuumed table table=pending_objects47382026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads47392026/11/11 23:58:35 INFO Vacuumed table table=closures47402026/11/11 23:58:35 INFO Vacuumed table table=objects47412026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=047422026/11/11 23:58:35 INFO Received complete push request method=POST path=/47432026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period47442026/11/11 23:58:35 INFO Completed push id=1 objects=9 duration=1.646939ms47452026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047462026/11/11 23:58:35 INFO Vacuumed table table=pending_closures47472026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47482026/11/11 23:58:35 INFO Completed upload id=13 objects=7 duration=1.442549ms47492026/11/11 23:58:35 INFO Vacuumed table table=pending_objects47502026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads47512026/11/11 23:58:35 INFO Vacuumed table table=closures47522026/11/11 23:58:35 INFO Vacuumed table table=objects47532026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47542026/11/11 23:58:35 INFO Received complete push request method=POST path=/47552026/11/11 23:58:35 INFO Completed upload id=28 objects=3 duration=1.012939ms47562026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47572026/11/11 23:58:35 INFO Completed push id=29 objects=15 duration=1.35644ms47582026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47592026/11/11 23:58:35 INFO Completed upload id=21 objects=2 duration=1.241909ms47602026/11/11 23:58:35 INFO Received complete push request method=POST path=/47612026/11/11 23:58:35 INFO Received push request method=POST path=/47622026/11/11 23:58:35 INFO Completed upload id=31 objects=6 duration=1.187139ms47632026/11/11 23:58:35 INFO Completed push id=31 objects=7 duration=1.099619ms47642026/11/11 23:58:35 INFO Received complete push request method=POST path=/47652026/11/11 23:58:35 INFO Completed push id=15 objects=24 duration=1.487579ms47662026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47672026/11/11 23:58:35 INFO Completed upload id=20 objects=4 duration=1.157079ms47682026/11/11 23:58:35 INFO Received uploads request method=POST path=/47692026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47702026/11/11 23:58:35 INFO Received uploads request method=POST path=/47712026/11/11 23:58:35 INFO Completed upload id=32 objects=5 duration=927.009µs47722026/11/11 23:58:35 INFO Received complete push request method=POST path=/47732026/11/11 23:58:35 INFO Completed push id=30 objects=6 duration=1.067669ms47742026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47752026/11/11 23:58:35 INFO Received uploads request method=POST path=/47762026/11/11 23:58:35 INFO Completed upload id=32 objects=4 duration=1.105449ms47772026/11/11 23:58:35 INFO Received uploads request method=POST path=/47782026/11/11 23:58:35 INFO Received complete push request method=POST path=/47792026/11/11 23:58:35 INFO Completed push id=29 objects=23 duration=1.696249ms47802026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47812026/11/11 23:58:35 INFO Completed upload id=19 objects=2 duration=994.339µs47822026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47832026/11/11 23:58:35 INFO Completed upload id=23 objects=7 duration=1.274899ms47842026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47852026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47862026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=047872026/11/11 23:58:35 INFO Completed upload id=21 objects=9 duration=2.025739ms47882026/11/11 23:58:35 INFO Completed upload id=31 objects=3 duration=1.327189ms47892026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47902026/11/11 23:58:35 WARN Force mode enabled - objects will be deleted immediately without grace period47912026/11/11 23:58:35 INFO Completed upload id=26 objects=2 duration=1.957779ms47922026/11/11 23:58:35 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047932026/11/11 23:58:35 INFO Received complete push request method=POST path=/47942026/11/11 23:58:35 INFO Received uploads request method=POST path=/47952026/11/11 23:58:35 INFO Vacuumed table table=pending_closures47962026/11/11 23:58:35 INFO Completed push id=30 objects=6 duration=1.594259ms47972026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47982026/11/11 23:58:35 INFO Received complete upload request method=POST path=/47992026/11/11 23:58:35 INFO Vacuumed table table=pending_objects48002026/11/11 23:58:35 INFO Vacuumed table table=multipart_uploads48012026/11/11 23:58:35 INFO Completed upload id=33 objects=10 duration=1.164869ms48022026/11/11 23:58:35 INFO Completed upload id=24 objects=7 duration=1.23423ms48032026/11/11 23:58:35 INFO Vacuumed table table=closures48042026/11/11 23:58:35 INFO Vacuumed table table=objects48052026/11/11 23:58:35 INFO Received complete upload request method=POST path=/48062026/11/11 23:58:35 INFO Received complete push request method=POST path=/48072026/11/11 23:58:35 INFO Completed upload id=2 objects=7 duration=1.443459ms48082026/11/11 23:58:35 INFO Completed push id=25 objects=10 duration=1.893749ms48092026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=048102026/11/11 23:58:35 INFO Aborted multipart uploads count=0 kept=048112026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period48122026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48132026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048142026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048152026/11/11 23:58:36 INFO Completed upload id=29 objects=4 duration=3.376158ms48162026/11/11 23:58:36 INFO Received complete push request method=POST path=/48172026/11/11 23:58:36 INFO Vacuumed table table=pending_closures48182026/11/11 23:58:36 INFO Vacuumed table table=pending_closures48192026/11/11 23:58:36 INFO Completed push id=26 objects=3 duration=1.38071ms48202026/11/11 23:58:36 INFO Received complete push request method=POST path=/48212026/11/11 23:58:36 INFO Vacuumed table table=pending_objects48222026/11/11 23:58:36 INFO Vacuumed table table=pending_objects48232026/11/11 23:58:36 INFO Completed push id=33 objects=3 duration=1.378289ms48242026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads48252026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads48262026/11/11 23:58:36 INFO Vacuumed table table=closures48272026/11/11 23:58:36 INFO Vacuumed table table=closures48282026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48292026/11/11 23:58:36 INFO Vacuumed table table=objects48302026/11/11 23:58:36 INFO Vacuumed table table=objects48312026/11/11 23:58:36 INFO Completed upload id=34 objects=24 duration=1.502319ms48322026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48332026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48342026/11/11 23:58:36 INFO Completed upload id=29 objects=5 duration=1.168129ms48352026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48362026/11/11 23:58:36 INFO Completed upload id=34 objects=23 duration=1.571919ms48372026/11/11 23:58:36 INFO Completed upload id=24 objects=18 duration=1.759489ms48382026/11/11 23:58:36 INFO Received uploads request method=POST path=/48392026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48402026/11/11 23:58:36 INFO Received uploads request method=POST path=/48412026/11/11 23:58:36 INFO Completed upload id=28 objects=8 duration=1.201599ms48422026/11/11 23:58:36 INFO Received complete push request method=POST path=/48432026/11/11 23:58:36 INFO Completed push id=34 objects=6 duration=1.073779ms48442026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=048452026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period48462026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048472026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48482026/11/11 23:58:36 INFO Vacuumed table table=pending_closures48492026/11/11 23:58:36 INFO Completed upload id=30 objects=2 duration=1.059379ms48502026/11/11 23:58:36 INFO Received complete push request method=POST path=/48512026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48522026/11/11 23:58:36 INFO Vacuumed table table=pending_objects48532026/11/11 23:58:36 INFO Completed upload id=8 objects=6 duration=1.237209ms48542026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads48552026/11/11 23:58:36 INFO Completed push id=25 objects=2 duration=2.079239ms48562026/11/11 23:58:36 INFO Vacuumed table table=closures48572026/11/11 23:58:36 INFO Received uploads request method=POST path=/48582026/11/11 23:58:36 INFO Vacuumed table table=objects48592026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48602026/11/11 23:58:36 INFO Received push request method=POST path=/48612026/11/11 23:58:36 INFO Completed upload id=29 objects=10 duration=1.079439ms48622026/11/11 23:58:36 INFO Received complete push request method=POST path=/48632026/11/11 23:58:36 INFO Received push request method=POST path=/48642026/11/11 23:58:36 INFO Completed push id=35 objects=6 duration=1.622579ms48652026/11/11 23:58:36 INFO Received complete push request method=POST path=/48662026/11/11 23:58:36 INFO Completed push id=35 objects=8 duration=1.311959ms48672026/11/11 23:58:36 INFO Received complete push request method=POST path=/48682026/11/11 23:58:36 INFO Completed push id=31 objects=20 duration=1.284449ms48692026/11/11 23:58:36 INFO Received complete push request method=POST path=/48702026/11/11 23:58:36 INFO Completed push id=27 objects=4 duration=1.002359ms48712026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48722026/11/11 23:58:36 INFO Completed upload id=30 objects=20 duration=1.054949ms48732026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=048742026/11/11 23:58:36 INFO Received complete push request method=POST path=/48752026/11/11 23:58:36 INFO Received complete push request method=POST path=/48762026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period48772026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48782026/11/11 23:58:36 INFO Completed push id=11 objects=6 duration=2.082209ms48792026/11/11 23:58:36 INFO Completed upload id=36 objects=9 duration=1.289929ms48802026/11/11 23:58:36 INFO Completed push id=24 objects=24 duration=3.538428ms48812026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=048822026/11/11 23:58:36 INFO Received complete upload request method=POST path=/48832026/11/11 23:58:36 INFO Vacuumed table table=pending_closures48842026/11/11 23:58:36 INFO Completed upload id=32 objects=5 duration=1.173849ms48852026/11/11 23:58:36 INFO Vacuumed table table=pending_objects48862026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads48872026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=048882026/11/11 23:58:36 INFO Received complete push request method=POST path=/48892026/11/11 23:58:36 INFO Vacuumed table table=closures48902026/11/11 23:58:36 INFO Completed push id=31 objects=11 duration=1.021369ms48912026/11/11 23:58:36 INFO Vacuumed table table=objects48922026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=048932026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=048942026/11/11 23:58:36 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."48952026/11/11 23:58:36 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."48962026/11/11 23:58:36 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48972026/11/11 23:58:36 INFO Received complete push request method=POST path=/48982026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period48992026/11/11 23:58:36 INFO Received complete push request method=POST path=/49002026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49012026/11/11 23:58:36 INFO Completed push id=28 objects=26 duration=1.67493ms49022026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period49032026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049042026/11/11 23:58:36 INFO Completed push id=20 objects=13 duration=1.619819ms49052026/11/11 23:58:36 INFO Completed upload id=12 objects=9 duration=1.450289ms49062026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49072026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049082026/11/11 23:58:36 INFO Vacuumed table table=pending_closures49092026/11/11 23:58:36 INFO Completed upload id=33 objects=3 duration=811.29µs49102026/11/11 23:58:36 INFO Vacuumed table table=pending_closures49112026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49122026/11/11 23:58:36 INFO Vacuumed table table=pending_objects49132026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads49142026/11/11 23:58:36 INFO Completed upload id=32 objects=2 duration=903.919µs49152026/11/11 23:58:36 INFO Vacuumed table table=pending_objects49162026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads49172026/11/11 23:58:36 INFO Vacuumed table table=closures49182026/11/11 23:58:36 INFO Vacuumed table table=closures49192026/11/11 23:58:36 INFO Vacuumed table table=objects49202026/11/11 23:58:36 INFO Vacuumed table table=objects49212026/11/11 23:58:36 INFO Received push request method=POST path=/49222026/11/11 23:58:36 INFO Received complete push request method=POST path=/49232026/11/11 23:58:36 INFO Completed push id=30 objects=3 duration=1.004529ms49242026/11/11 23:58:36 INFO Received complete push request method=POST path=/49252026/11/11 23:58:36 INFO Completed push id=22 objects=4 duration=1.373639ms49262026/11/11 23:58:36 INFO Received complete push request method=POST path=/49272026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=049282026/11/11 23:58:36 INFO Completed push id=33 objects=3 duration=1.151089ms49292026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period49302026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049312026/11/11 23:58:36 INFO Received uploads request method=POST path=/49322026/11/11 23:58:36 INFO Vacuumed table table=pending_closures49332026/11/11 23:58:36 INFO Vacuumed table table=pending_objects49342026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads49352026/11/11 23:58:36 INFO Vacuumed table table=closures49362026/11/11 23:58:36 INFO Vacuumed table table=objects49372026/11/11 23:58:36 INFO Received complete push request method=POST path=/49382026/11/11 23:58:36 INFO Completed push id=32 objects=19 duration=1.268409ms49392026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49402026/11/11 23:58:36 INFO Completed upload id=34 objects=3 duration=838.06µs49412026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49422026/11/11 23:58:36 INFO Received push request method=POST path=/49432026/11/11 23:58:36 INFO Completed upload id=23 objects=7 duration=1.257019ms49442026/11/11 23:58:36 INFO Received uploads request method=POST path=/49452026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=049462026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period49472026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049482026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=049492026/11/11 23:58:36 INFO Vacuumed table table=pending_closures49502026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period49512026/11/11 23:58:36 INFO Received complete push request method=POST path=/49522026/11/11 23:58:36 INFO Vacuumed table table=pending_objects49532026/11/11 23:58:36 INFO Received complete push request method=POST path=/49542026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049552026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads49562026/11/11 23:58:36 INFO Completed push id=24 objects=5 duration=1.824388ms49572026/11/11 23:58:36 INFO Vacuumed table table=closures49582026/11/11 23:58:36 INFO Vacuumed table table=pending_closures49592026/11/11 23:58:36 INFO Completed push id=28 objects=11 duration=2.750919ms49602026/11/11 23:58:36 INFO Vacuumed table table=objects49612026/11/11 23:58:36 INFO Vacuumed table table=pending_objects49622026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads49632026/11/11 23:58:36 INFO Vacuumed table table=closures49642026/11/11 23:58:36 INFO Vacuumed table table=objects49652026/11/11 23:58:36 INFO Received push request method=POST path=/49662026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49672026/11/11 23:58:36 INFO Completed upload id=25 objects=2 duration=1.424629ms49682026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49692026/11/11 23:58:36 INFO Completed upload id=18 objects=22 duration=1.632059ms49702026/11/11 23:58:36 INFO Received uploads request method=POST path=/49712026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49722026/11/11 23:58:36 INFO Completed upload id=26 objects=20 duration=1.324779ms49732026/11/11 23:58:36 INFO Received complete push request method=POST path=/49742026/11/11 23:58:36 INFO Completed push id=29 objects=24 duration=1.833089ms49752026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49762026/11/11 23:58:36 INFO Completed upload id=21 objects=8 duration=1.119829ms49772026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=049782026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period49792026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049802026/11/11 23:58:36 INFO Vacuumed table table=pending_closures49812026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=049822026/11/11 23:58:36 INFO Vacuumed table table=pending_objects49832026/11/11 23:58:36 INFO Received complete push request method=POST path=/49842026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads49852026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period49862026/11/11 23:58:36 INFO Completed push id=28 objects=7 duration=1.329759ms49872026/11/11 23:58:36 INFO Vacuumed table table=closures49882026/11/11 23:58:36 INFO Vacuumed table table=objects49892026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049902026/11/11 23:58:36 INFO Vacuumed table table=pending_closures49912026/11/11 23:58:36 INFO Vacuumed table table=pending_objects49922026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads49932026/11/11 23:58:36 INFO Vacuumed table table=closures49942026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49952026/11/11 23:58:36 INFO Completed upload id=22 objects=7 duration=1.120419ms49962026/11/11 23:58:36 INFO Vacuumed table table=objects49972026/11/11 23:58:36 INFO Received complete upload request method=POST path=/49982026/11/11 23:58:36 INFO Completed upload id=30 objects=6 duration=1.271759ms49992026/11/11 23:58:36 INFO Received complete push request method=POST path=/50002026/11/11 23:58:36 INFO Completed push id=24 objects=5 duration=1.104499ms50012026/11/11 23:58:36 INFO Received push request method=POST path=/50022026/11/11 23:58:36 INFO Received complete push request method=POST path=/50032026/11/11 23:58:36 INFO Completed push id=31 objects=5 duration=1.293619ms50042026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50052026/11/11 23:58:36 INFO Completed upload id=25 objects=8 duration=1.09882ms50062026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50072026/11/11 23:58:36 INFO Completed upload id=23 objects=7 duration=3.269138ms50082026/11/11 23:58:36 INFO Received complete push request method=POST path=/50092026/11/11 23:58:36 INFO Completed push id=26 objects=4 duration=1.171589ms50102026/11/11 23:58:36 INFO Received complete push request method=POST path=/50112026/11/11 23:58:36 INFO Completed push id=32 objects=26 duration=1.738259ms50122026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50132026/11/11 23:58:36 INFO Completed upload id=26 objects=5 duration=1.130379ms50142026/11/11 23:58:36 INFO Received complete push request method=POST path=/50152026/11/11 23:58:36 INFO Completed push id=30 objects=2 duration=1.06163ms50162026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50172026/11/11 23:58:36 INFO Completed upload id=33 objects=2 duration=1.150279ms50182026/11/11 23:58:36 INFO Received complete push request method=POST path=/50192026/11/11 23:58:36 INFO Completed push id=27 objects=3 duration=2.097168ms50202026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50212026/11/11 23:58:36 INFO Completed upload id=34 objects=5 duration=1.14732ms50222026/11/11 23:58:36 INFO Received complete push request method=POST path=/50232026/11/11 23:58:36 INFO Completed push id=32 objects=7 duration=1.45372ms50242026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50252026/11/11 23:58:36 INFO Completed upload id=28 objects=5 duration=1.170139ms50262026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50272026/11/11 23:58:36 INFO Completed upload id=35 objects=11 duration=1.15188ms50282026/11/11 23:58:36 INFO Received complete push request method=POST path=/50292026/11/11 23:58:36 INFO Completed push id=29 objects=17 duration=1.589879ms50302026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50312026/11/11 23:58:36 INFO Completed upload id=33 objects=6 duration=1.262079ms50322026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50332026/11/11 23:58:36 INFO Completed upload id=36 objects=3 duration=1.354779ms50342026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=050352026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period50362026/11/11 23:58:36 INFO Received complete upload request method=POST path=/50372026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050382026/11/11 23:58:36 INFO Completed upload id=34 objects=3 duration=1.001799ms50392026/11/11 23:58:36 INFO Vacuumed table table=pending_closures50402026/11/11 23:58:36 INFO Vacuumed table table=pending_objects50412026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads50422026/11/11 23:58:36 INFO Vacuumed table table=closures50432026/11/11 23:58:36 INFO Vacuumed table table=objects50442026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=050452026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period50462026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050472026/11/11 23:58:36 INFO Vacuumed table table=pending_closures50482026/11/11 23:58:36 INFO Vacuumed table table=pending_objects50492026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads50502026/11/11 23:58:36 INFO Vacuumed table table=closures50512026/11/11 23:58:36 INFO Vacuumed table table=objects50522026/11/11 23:58:36 INFO Aborted multipart uploads count=0 kept=050532026/11/11 23:58:36 WARN Force mode enabled - objects will be deleted immediately without grace period50542026/11/11 23:58:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050552026/11/11 23:58:36 INFO Vacuumed table table=pending_closures50562026/11/11 23:58:36 INFO Vacuumed table table=pending_objects50572026/11/11 23:58:36 INFO Vacuumed table table=multipart_uploads50582026/11/11 23:58:36 INFO Vacuumed table table=closures50592026/11/11 23:58:36 INFO Vacuumed table table=objects5060--- PASS: FuzzPushGC (0.00s)5061 --- PASS: FuzzPushGC/seed#14 (2.83s)5062 --- PASS: FuzzPushGC/seed#0 (2.90s)5063 --- PASS: FuzzPushGC/seed#8 (2.94s)5064 --- PASS: FuzzPushGC/seed#13 (2.94s)5065 --- PASS: FuzzPushGC/seed#3 (3.01s)5066 --- PASS: FuzzPushGC/seed#15 (3.12s)5067 --- PASS: FuzzPushGC/seed#9 (3.14s)5068 --- PASS: FuzzPushGC/seed#12 (3.19s)5069 --- PASS: FuzzPushGC/seed#4 (3.20s)5070 --- PASS: FuzzPushGC/seed#6 (3.22s)5071 --- PASS: FuzzPushGC/seed#7 (3.25s)5072 --- PASS: FuzzPushGC/seed#5 (3.26s)5073 --- PASS: FuzzPushGC/seed#2 (3.33s)5074 --- PASS: FuzzPushGC/seed#1 (3.50s)5075 --- PASS: FuzzPushGC/seed#11 (3.51s)5076 --- PASS: FuzzPushGC/seed#10 (3.52s)5077PASS5078{"timestamp":"2026-11-11T23:58:36.439253542Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:38818","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(399)"}5079{"timestamp":"2026-11-11T23:58:36.439593772Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:39280","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(400)"}50802026-11-11 23:58:36.867 UTC [129] LOG: received smart shutdown request50812026-11-11 23:58:36.871 UTC [129] LOG: background worker "logical replication launcher" (PID 139) exited with exit code 150822026-11-11 23:58:37.908 UTC [134] LOG: checkpoint complete: wrote 4222 buffers (25.8%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=4.028 s, sync=0.877 s, total=4.971 s; sync files=42363, longest=0.008 s, average=0.001 s; distance=534977 kB, estimate=534977 kB; lsn=0/26D618F0, redo lsn=0/221D621850832026-11-11 23:58:37.911 UTC [134] LOG: shutting down50842026-11-11 23:58:37.911 UTC [134] LOG: checkpoint starting: shutdown immediate50852026-11-11 23:58:38.728 UTC [134] LOG: checkpoint complete: wrote 9496 buffers (58.0%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.173 s, sync=0.636 s, total=0.818 s; sync files=2795, longest=0.003 s, average=0.001 s; distance=77357 kB, estimate=489215 kB; lsn=0/26D61968, redo lsn=0/26D6196850862026-11-11 23:58:38.789 UTC [129] LOG: database system is shut down5087Running OIDC tests...5088=== RUN TestAudienceForIssuer5089=== PAUSE TestAudienceForIssuer5090=== RUN TestGlobMatch5091=== PAUSE TestGlobMatch5092=== RUN TestValidateToken_ValidToken5093=== PAUSE TestValidateToken_ValidToken5094=== RUN TestValidateToken_WrongAudience5095=== PAUSE TestValidateToken_WrongAudience5096=== RUN TestValidateToken_Expired5097=== PAUSE TestValidateToken_Expired5098=== RUN TestValidateToken_BoundClaimsMismatch5099=== PAUSE TestValidateToken_BoundClaimsMismatch5100=== RUN TestValidateToken_BoundSubjectMismatch5101=== PAUSE TestValidateToken_BoundSubjectMismatch5102=== RUN TestValidateToken_MultipleProviders5103=== PAUSE TestValidateToken_MultipleProviders5104=== RUN TestValidateToken_NoMatchingProvider5105=== PAUSE TestValidateToken_NoMatchingProvider5106=== RUN TestValidateToken_KubernetesServiceAccount5107=== PAUSE TestValidateToken_KubernetesServiceAccount5108=== RUN TestNewValidator_KubernetesRequiresCA5109=== PAUSE TestNewValidator_KubernetesRequiresCA5110=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5111=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5112=== RUN TestPins_ReservedForMatchingRule5113=== PAUSE TestPins_ReservedForMatchingRule5114=== RUN TestPins_TopLevelShorthand5115=== PAUSE TestPins_TopLevelShorthand5116=== RUN TestPins_ConfigValidation5117=== PAUSE TestPins_ConfigValidation5118=== RUN TestScopes_LegacyProviderDefaultsToWrite5119=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5120=== RUN TestScopes_Rules5121=== PAUSE TestScopes_Rules5122=== RUN TestScopes_ConfigValidation5123=== PAUSE TestScopes_ConfigValidation5124=== CONT TestAudienceForIssuer5125=== CONT TestPins_ConfigValidation5126=== CONT TestScopes_Rules5127=== CONT TestValidateToken_KubernetesServiceAccount5128=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5129=== CONT TestScopes_ConfigValidation5130--- PASS: TestAudienceForIssuer (0.00s)5131=== CONT TestNewValidator_KubernetesRequiresCA5132--- PASS: TestScopes_ConfigValidation (0.00s)5133--- PASS: TestPins_ConfigValidation (0.00s)5134=== CONT TestValidateToken_NoMatchingProvider5135=== CONT TestPins_TopLevelShorthand5136=== CONT TestValidateToken_WrongAudience5137=== CONT TestValidateToken_Expired5138=== CONT TestValidateToken_ValidToken5139=== CONT TestValidateToken_MultipleProviders5140=== CONT TestValidateToken_BoundSubjectMismatch5141=== CONT TestScopes_LegacyProviderDefaultsToWrite5142=== CONT TestGlobMatch5143=== CONT TestValidateToken_BoundClaimsMismatch5144=== CONT TestPins_ReservedForMatchingRule5145=== RUN TestGlobMatch/foo_foo5146=== PAUSE TestGlobMatch/foo_foo5147=== RUN TestGlobMatch/foo_bar5148=== PAUSE TestGlobMatch/foo_bar5149=== RUN TestGlobMatch/*_5150=== PAUSE TestGlobMatch/*_5151=== RUN TestGlobMatch/*_anything5152=== PAUSE TestGlobMatch/*_anything5153=== RUN TestGlobMatch/foo*_foo5154=== PAUSE TestGlobMatch/foo*_foo5155=== RUN TestGlobMatch/foo*_foobar5156=== PAUSE TestGlobMatch/foo*_foobar5157=== RUN TestGlobMatch/foo*_bar5158=== PAUSE TestGlobMatch/foo*_bar5159=== RUN TestGlobMatch/*bar_bar5160=== PAUSE TestGlobMatch/*bar_bar5161=== RUN TestGlobMatch/*bar_foobar5162=== PAUSE TestGlobMatch/*bar_foobar5163=== RUN TestGlobMatch/*bar_foo5164=== PAUSE TestGlobMatch/*bar_foo5165=== RUN TestGlobMatch/foo*bar_foobar5166=== PAUSE TestGlobMatch/foo*bar_foobar5167=== RUN TestGlobMatch/foo*bar_foo123bar5168=== PAUSE TestGlobMatch/foo*bar_foo123bar5169=== RUN TestGlobMatch/foo*bar_foobarbaz5170=== PAUSE TestGlobMatch/foo*bar_foobarbaz5171=== RUN TestGlobMatch/*/*_foo/bar5172=== PAUSE TestGlobMatch/*/*_foo/bar5173=== RUN TestGlobMatch/*/*_foo5174=== PAUSE TestGlobMatch/*/*_foo5175=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5176=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5177=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05178=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05179=== RUN TestGlobMatch/refs/*/main_refs/heads/main5180=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5181=== RUN TestGlobMatch/fo?_foo5182=== PAUSE TestGlobMatch/fo?_foo5183=== RUN TestGlobMatch/fo?_fo5184=== PAUSE TestGlobMatch/fo?_fo5185=== RUN TestGlobMatch/fo?_fooo5186=== PAUSE TestGlobMatch/fo?_fooo5187=== RUN TestGlobMatch/?oo_foo5188=== PAUSE TestGlobMatch/?oo_foo5189=== RUN TestGlobMatch/?oo_boo5190=== PAUSE TestGlobMatch/?oo_boo5191=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5192=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5193=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5194=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5195=== CONT TestGlobMatch/foo_foo5196=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5197=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5198=== CONT TestGlobMatch/?oo_boo5199=== CONT TestGlobMatch/foo*bar_foobar5200=== CONT TestGlobMatch/foo*_foo5201=== CONT TestGlobMatch/*_anything5202=== CONT TestGlobMatch/*_5203=== CONT TestGlobMatch/foo_bar5204=== CONT TestGlobMatch/*bar_foobar5205=== CONT TestGlobMatch/*bar_foo5206=== CONT TestGlobMatch/foo*bar_foo123bar5207=== CONT TestGlobMatch/*bar_bar5208=== CONT TestGlobMatch/*/*_foo/bar5209=== CONT TestGlobMatch/foo*bar_foobarbaz5210=== CONT TestGlobMatch/foo*_foobar5211=== CONT TestGlobMatch/foo*_bar5212=== CONT TestGlobMatch/refs/*/main_refs/heads/main5213=== CONT TestGlobMatch/?oo_foo5214=== CONT TestGlobMatch/fo?_fooo5215=== CONT TestGlobMatch/fo?_fo5216=== CONT TestGlobMatch/fo?_foo5217=== CONT TestGlobMatch/*/*_foo5218=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05219=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5220--- PASS: TestGlobMatch (0.00s)5221 --- PASS: TestGlobMatch/foo_foo (0.00s)5222 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5223 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5224 --- PASS: TestGlobMatch/?oo_boo (0.00s)5225 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5226 --- PASS: TestGlobMatch/foo*_foo (0.00s)5227 --- PASS: TestGlobMatch/*_anything (0.00s)5228 --- PASS: TestGlobMatch/*_ (0.00s)5229 --- PASS: TestGlobMatch/foo_bar (0.00s)5230 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5231 --- PASS: TestGlobMatch/*bar_foo (0.00s)5232 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5233 --- PASS: TestGlobMatch/*bar_bar (0.00s)5234 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5235 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5236 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5237 --- PASS: TestGlobMatch/foo*_bar (0.00s)5238 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5239 --- PASS: TestGlobMatch/?oo_foo (0.00s)5240 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5241 --- PASS: TestGlobMatch/fo?_fo (0.00s)5242 --- PASS: TestGlobMatch/fo?_foo (0.00s)5243 --- PASS: TestGlobMatch/*/*_foo (0.00s)5244 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5245 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)52462026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43741/oidc5247--- PASS: TestPins_TopLevelShorthand (0.04s)52482026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42171/oidc5249--- PASS: TestValidateToken_ValidToken (0.05s)52502026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35291/oidc52512026/11/11 23:58:41 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352522026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35249/oidc5253--- PASS: TestScopes_Rules (0.06s)52542026/11/11 23:58:41 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:439515255--- PASS: TestValidateToken_WrongAudience (0.06s)5256--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.06s)52572026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34545/oidc5258--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.07s)5259--- PASS: TestValidateToken_KubernetesServiceAccount (0.07s)52602026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41859/oidc52612026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42687/oidc52622026/11/11 23:58:41 http: TLS handshake error from 127.0.0.1:35342: remote error: tls: bad certificate5263--- PASS: TestNewValidator_KubernetesRequiresCA (0.08s)5264--- PASS: TestValidateToken_BoundSubjectMismatch (0.08s)5265--- PASS: TestValidateToken_BoundClaimsMismatch (0.08s)52662026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43247/oidc5267--- PASS: TestPins_ReservedForMatchingRule (0.11s)52682026/11/11 23:58:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41177/oidc5269--- PASS: TestValidateToken_Expired (0.17s)52702026/11/11 23:58:41 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:34655/oidc5271--- PASS: TestValidateToken_NoMatchingProvider (0.17s)52722026/11/11 23:58:41 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:33043/oidc52732026/11/11 23:58:41 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:37485/oidc5274--- PASS: TestValidateToken_MultipleProviders (0.21s)5275PASS5276Running hook tests...5277=== RUN TestSendPathsEmpty5278=== PAUSE TestSendPathsEmpty5279=== RUN TestQueueEnqueueAndFetch5280=== PAUSE TestQueueEnqueueAndFetch5281=== RUN TestQueueDeduplication5282=== PAUSE TestQueueDeduplication5283=== RUN TestQueueRemove5284=== PAUSE TestQueueRemove5285=== RUN TestQueueFetchBatchLimit5286=== PAUSE TestQueueFetchBatchLimit5287=== RUN TestQueueRetryMovesToBack5288=== PAUSE TestQueueRetryMovesToBack5289=== RUN TestQueueFetchRemoveLifecycle5290=== PAUSE TestQueueFetchRemoveLifecycle5291=== RUN TestQueueConcurrentWriters5292=== PAUSE TestQueueConcurrentWriters5293=== RUN TestQueueRemoveLargeClosure5294=== PAUSE TestQueueRemoveLargeClosure5295=== RUN TestServerClientIntegration5296--- PASS: TestServerClientIntegration (0.00s)5297=== RUN TestServerQueueError52982026/11/11 23:58:41 ERROR Failed to queue paths error="permission denied" count=15299--- PASS: TestServerQueueError (0.00s)5300=== RUN TestGetListenerSocketActivation5301 server_test.go:220: === RUN TestGetListenerSocketActivation5302 --- PASS: TestGetListenerSocketActivation (0.00s)5303 PASS5304 5305--- PASS: TestGetListenerSocketActivation (0.01s)5306=== RUN TestServerStalledClientDoesNotBlockShutdown5307=== PAUSE TestServerStalledClientDoesNotBlockShutdown5308=== RUN TestServerBacksOffOnAcceptErrors5309=== PAUSE TestServerBacksOffOnAcceptErrors5310=== RUN TestDrainIsolatesPoisonPath5311=== PAUSE TestDrainIsolatesPoisonPath5312=== RUN TestRunNotBlockedByPoisonHead5313=== PAUSE TestRunNotBlockedByPoisonHead5314=== RUN TestDrainGivesUpWhenServerDown5315=== PAUSE TestDrainGivesUpWhenServerDown5316=== RUN TestFailedPathPrunedByLaterClosure5317=== PAUSE TestFailedPathPrunedByLaterClosure5318=== RUN TestWorkerUploadsAndRemoves5319=== PAUSE TestWorkerUploadsAndRemoves5320=== RUN TestWorkerSkipsGCdPaths5321=== PAUSE TestWorkerSkipsGCdPaths5322=== RUN TestWorkerPrunesClosureDeps5323=== PAUSE TestWorkerPrunesClosureDeps5324=== RUN TestDrainTimeout5325=== PAUSE TestDrainTimeout5326=== RUN TestWorkerRemoveFailureIsNotProgress5327=== PAUSE TestWorkerRemoveFailureIsNotProgress5328=== RUN TestWorkerKeepsPathItCannotStat5329=== PAUSE TestWorkerKeepsPathItCannotStat5330=== CONT TestSendPathsEmpty5331--- PASS: TestSendPathsEmpty (0.00s)5332=== CONT TestWorkerPrunesClosureDeps5333=== CONT TestDrainIsolatesPoisonPath5334=== CONT TestWorkerUploadsAndRemoves5335=== CONT TestFailedPathPrunedByLaterClosure5336=== CONT TestDrainGivesUpWhenServerDown5337=== CONT TestWorkerSkipsGCdPaths5338=== CONT TestWorkerKeepsPathItCannotStat5339=== CONT TestRunNotBlockedByPoisonHead5340=== CONT TestWorkerRemoveFailureIsNotProgress5341=== CONT TestQueueRetryMovesToBack5342=== CONT TestDrainTimeout5343=== CONT TestQueueFetchBatchLimit5344=== CONT TestQueueRemove5345=== CONT TestServerStalledClientDoesNotBlockShutdown5346=== CONT TestQueueDeduplication5347=== CONT TestServerBacksOffOnAcceptErrors5348=== CONT TestQueueEnqueueAndFetch53492000/01/01 00:00:00 ERROR Accept failed error="too many open files"53502000/01/01 00:00:00 ERROR Accept failed error="too many open files"53512000/01/01 00:00:00 ERROR Accept failed error="too many open files"53522000/01/01 00:00:00 ERROR Accept failed error="too many open files"53532000/01/01 00:00:00 ERROR Accept failed error="too many open files"53542000/01/01 00:00:00 ERROR Accept failed error="too many open files"53552000/01/01 00:00:00 ERROR Accept failed error="too many open files"53562000/01/01 00:00:00 ERROR Accept failed error="too many open files"5357--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5358=== CONT TestQueueFetchRemoveLifecycle5359=== CONT TestQueueRemoveLargeClosure5360=== CONT TestQueueConcurrentWriters53612026/11/11 23:58:41 INFO Upload queue status pending=253622026/11/11 23:58:41 INFO Upload queue status pending=253632026/11/11 23:58:41 INFO Uploading batch count=253642026/11/11 23:58:41 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths979358925/002/nonexistent53652026/11/11 23:58:41 INFO Uploading batch count=253662026/11/11 23:58:41 INFO Uploading batch count=453672026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=453682026/11/11 23:58:41 INFO Uploading batch count=153692026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=253702026/11/11 23:58:41 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown134066468/002/a53712026/11/11 23:58:41 INFO Upload queue status pending=253722026/11/11 23:58:41 INFO Uploading batch count=153732026/11/11 23:58:41 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath1212304090/002/bbb53742026/11/11 23:58:41 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown134066468/002/b53752026/11/11 23:58:41 INFO Upload queue status pending=35376--- PASS: TestQueueFetchBatchLimit (0.01s)53772026/11/11 23:58:41 INFO Uploading batch count=153782026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=15379--- PASS: TestQueueEnqueueAndFetch (0.01s)53802026/11/11 23:58:41 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2506456824/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2506456824/002/locked/aaa: permission denied"53812026/11/11 23:58:41 INFO Uploading batch count=153822026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=15383--- PASS: TestQueueFetchRemoveLifecycle (0.01s)53842000/01/01 00:00:00 INFO Uploading batch count=25385--- PASS: TestQueueDeduplication (0.02s)53862000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253872026/11/11 23:58:41 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2506456824/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2506456824/002/locked/aaa: permission denied"53882026/11/11 23:58:41 INFO Uploading batch count=153892026/11/11 23:58:41 INFO Uploading batch count=253902000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45391--- PASS: TestQueueRemove (0.02s)53922026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=253932026/11/11 23:58:41 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2669809616/002/nonexistent5394--- PASS: TestQueueRetryMovesToBack (0.02s)53952026/11/11 23:58:41 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown134066468/002/c53962026/11/11 23:58:41 INFO Uploading batch count=153972026/11/11 23:58:41 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2506456824/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2506456824/002/locked/aaa: permission denied"53982026/11/11 23:58:41 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153992026/11/11 23:58:41 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2669809616/002/nonexistent54002026/11/11 23:58:41 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154012026/11/11 23:58:41 INFO Uploading batch count=154022026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=154032026/11/11 23:58:41 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2669809616/002/nonexistent54042026/11/11 23:58:41 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=154052026/11/11 23:58:41 ERROR Drain finished with paths left in queue remaining=154062026/11/11 23:58:41 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown134066468/002/d54072026/11/11 23:58:41 ERROR Drain finished with paths left in queue remaining=154082026/11/11 23:58:41 INFO Uploading batch count=154092026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=15410--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)54112026/11/11 23:58:41 INFO Uploading batch count=25412--- PASS: TestDrainTimeout (0.02s)54132026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=254142026/11/11 23:58:41 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown134066468/002/e54152026/11/11 23:58:41 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown134066468/002/f54162026/11/11 23:58:41 INFO Uploading batch count=154172026/11/11 23:58:41 ERROR Upload failed error="upload failed" count=15418--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)54192026/11/11 23:58:41 ERROR Drain finished with paths left in queue remaining=105420--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)54212026/11/11 23:58:41 ERROR Drain finished with paths left in queue remaining=15422--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5423--- PASS: TestDrainIsolatesPoisonPath (0.03s)5424--- PASS: TestWorkerSkipsGCdPaths (0.04s)5425--- PASS: TestWorkerUploadsAndRemoves (0.04s)5426--- PASS: TestWorkerPrunesClosureDeps (0.04s)5427--- PASS: TestQueueRemoveLargeClosure (0.07s)54282026/11/11 23:58:41 ERROR Failed to decode request error="read unix /build/hook1119170045/test.sock->@: i/o timeout"54292026/11/11 23:58:41 ERROR Failed to write response error="write unix /build/hook1119170045/test.sock->@: i/o timeout"5430--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)5431--- PASS: TestQueueConcurrentWriters (0.24s)54322026/11/11 23:58:42 INFO Uploading batch count=154332026/11/11 23:58:42 INFO Uploading batch count=154342026/11/11 23:58:42 INFO Uploading batch count=154352026/11/11 23:58:42 ERROR Upload failed error="upload failed" count=154362026/11/11 23:58:42 INFO Uploading batch count=154372026/11/11 23:58:42 ERROR Upload failed error="upload failed" count=154382026/11/11 23:58:42 INFO Uploading batch count=154392026/11/11 23:58:42 ERROR Upload failed error="upload failed" count=154402026/11/11 23:58:42 INFO Uploading batch count=154412026/11/11 23:58:42 ERROR Upload failed error="upload failed" count=154422026/11/11 23:58:42 ERROR Drain finished with paths left in queue remaining=15443--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5444PASS5445Running signing tests...5446=== RUN TestExternalSigner5447=== PAUSE TestExternalSigner5448=== RUN TestExternalSignerWrongPublicKey5449=== PAUSE TestExternalSignerWrongPublicKey5450=== RUN TestExternalSignerErrors5451=== PAUSE TestExternalSignerErrors5452=== RUN TestExternalSignerRestartsAfterCrash5453=== PAUSE TestExternalSignerRestartsAfterCrash5454=== RUN TestExternalSignerProgramErrorKeepsProcess5455=== PAUSE TestExternalSignerProgramErrorKeepsProcess5456=== RUN TestExternalSignerClose5457=== PAUSE TestExternalSignerClose5458=== RUN TestExternalSignerRejectsInvalidPublicKey5459=== PAUSE TestExternalSignerRejectsInvalidPublicKey5460=== RUN TestExternalSignerCancellation5461=== PAUSE TestExternalSignerCancellation5462=== RUN TestGenerateFingerprint5463=== PAUSE TestGenerateFingerprint5464=== RUN TestParseSigningKey5465=== PAUSE TestParseSigningKey5466=== RUN TestSign5467=== PAUSE TestSign5468=== RUN TestSignBatch5469=== PAUSE TestSignBatch5470=== RUN TestNilKeyPublicKey5471=== PAUSE TestNilKeyPublicKey5472=== RUN TestSignErrors5473=== PAUSE TestSignErrors5474=== CONT TestExternalSigner5475=== CONT TestExternalSignerCancellation5476=== CONT TestGenerateFingerprint5477=== CONT TestSignBatch5478=== CONT TestNilKeyPublicKey5479--- PASS: TestNilKeyPublicKey (0.00s)5480=== CONT TestExternalSignerProgramErrorKeepsProcess5481=== CONT TestExternalSignerRejectsInvalidPublicKey5482=== CONT TestExternalSignerClose5483=== CONT TestExternalSignerErrors5484=== CONT TestExternalSignerRestartsAfterCrash5485=== CONT TestParseSigningKey5486=== CONT TestSign5487=== CONT TestSignErrors5488=== CONT TestExternalSignerWrongPublicKey5489=== RUN TestGenerateFingerprint/basic_with_references5490=== PAUSE TestGenerateFingerprint/basic_with_references5491=== RUN TestExternalSignerRejectsInvalidPublicKey/#005492=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005493=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5494=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5495=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5496=== RUN TestGenerateFingerprint/no_references5497=== PAUSE TestGenerateFingerprint/no_references5498=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5499=== RUN TestParseSigningKey/valid_32-byte_key5500=== PAUSE TestParseSigningKey/valid_32-byte_key5501=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5502=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5503=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5504=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5505=== RUN TestSignErrors/nil_key5506=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5507=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5508=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5509=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5510=== PAUSE TestSignErrors/nil_key5511=== RUN TestSignErrors/nil_narinfo5512=== PAUSE TestSignErrors/nil_narinfo5513=== RUN TestSignErrors/invalid_narinfo5514=== PAUSE TestSignErrors/invalid_narinfo5515=== RUN TestSignErrors/canceled5516=== PAUSE TestSignErrors/canceled5517=== CONT TestSignErrors/nil_key5518=== CONT TestSignErrors/nil_narinfo5519=== RUN TestGenerateFingerprint/invalid_nar_hash_length5520=== CONT TestSignErrors/invalid_narinfo5521=== RUN TestExternalSignerErrors/missing_signatures5522=== PAUSE TestExternalSignerErrors/missing_signatures5523=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5524=== RUN TestExternalSignerErrors/wrong_signatures_shape5525=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5526=== RUN TestExternalSignerErrors/missing_signature_name5527=== PAUSE TestExternalSignerErrors/missing_signature_name5528=== RUN TestExternalSignerErrors/wrong_signature_name5529=== PAUSE TestExternalSignerErrors/wrong_signature_name5530=== RUN TestExternalSignerErrors/null5531=== PAUSE TestExternalSignerErrors/null5532=== RUN TestExternalSignerErrors/wrong_id5533=== PAUSE TestExternalSignerErrors/wrong_id5534=== RUN TestExternalSignerErrors/exit_on_request5535=== PAUSE TestExternalSignerErrors/exit_on_request5536=== RUN TestExternalSignerErrors/wrong_shape5537=== PAUSE TestExternalSignerErrors/wrong_shape5538=== RUN TestExternalSignerErrors/null_signatures5539--- PASS: TestSignBatch (0.00s)5540--- PASS: TestSign (0.00s)5541=== RUN TestParseSigningKey/no_colon5542=== CONT TestSignErrors/canceled5543--- PASS: TestSignErrors (0.00s)5544 --- PASS: TestSignErrors/nil_key (0.00s)5545 --- PASS: TestSignErrors/nil_narinfo (0.00s)5546 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5547 --- PASS: TestSignErrors/canceled (0.00s)5548=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5549=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5550=== CONT TestExternalSignerRejectsInvalidPublicKey/#005551=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5552=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5553=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5554=== PAUSE TestParseSigningKey/no_colon5555=== RUN TestParseSigningKey/empty_name5556=== PAUSE TestParseSigningKey/empty_name5557=== RUN TestGenerateFingerprint/invalid_store_path_prefix5558=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5559=== RUN TestGenerateFingerprint/invalid_reference_prefix5560=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5561=== CONT TestGenerateFingerprint/basic_with_references5562=== CONT TestGenerateFingerprint/invalid_reference_prefix5563=== PAUSE TestExternalSignerErrors/null_signatures5564=== CONT TestGenerateFingerprint/invalid_nar_hash_length5565=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5566=== RUN TestParseSigningKey/invalid_base645567=== PAUSE TestParseSigningKey/invalid_base645568--- PASS: TestExternalSignerWrongPublicKey (0.00s)5569=== CONT TestGenerateFingerprint/invalid_store_path_prefix5570=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5571--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5572 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5573 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5574 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5575 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5576 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5577=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5578=== CONT TestGenerateFingerprint/no_references5579--- PASS: TestGenerateFingerprint (0.00s)5580 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5581 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5582 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5583 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5584 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5585 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5586 --- PASS: TestGenerateFingerprint/no_references (0.00s)5587=== RUN TestExternalSignerErrors/too_few_signatures5588=== RUN TestParseSigningKey/wrong_length5589=== PAUSE TestParseSigningKey/wrong_length5590=== CONT TestParseSigningKey/valid_32-byte_key5591=== CONT TestParseSigningKey/wrong_length5592--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5593--- PASS: TestExternalSignerClose (0.00s)5594=== PAUSE TestExternalSignerErrors/too_few_signatures5595=== CONT TestParseSigningKey/no_colon5596=== CONT TestParseSigningKey/empty_name5597=== CONT TestParseSigningKey/invalid_base645598=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5599--- PASS: TestParseSigningKey (0.00s)5600 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5601 --- PASS: TestParseSigningKey/wrong_length (0.00s)5602 --- PASS: TestParseSigningKey/no_colon (0.00s)5603 --- PASS: TestParseSigningKey/empty_name (0.00s)5604 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5605 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5606=== RUN TestExternalSignerErrors/unrequested_key5607=== PAUSE TestExternalSignerErrors/unrequested_key5608=== RUN TestExternalSignerErrors/invalid_signature_base645609=== PAUSE TestExternalSignerErrors/invalid_signature_base645610=== RUN TestExternalSignerErrors/program_error5611=== PAUSE TestExternalSignerErrors/program_error5612=== RUN TestExternalSignerErrors/wrong_signature_length5613=== PAUSE TestExternalSignerErrors/wrong_signature_length5614=== RUN TestExternalSignerErrors/exit_at_startup5615=== PAUSE TestExternalSignerErrors/exit_at_startup5616=== RUN TestExternalSignerErrors/invalid_JSON5617=== PAUSE TestExternalSignerErrors/invalid_JSON5618=== CONT TestExternalSignerErrors/missing_signatures5619=== CONT TestExternalSignerErrors/unrequested_key5620=== CONT TestExternalSignerErrors/exit_on_request5621=== CONT TestExternalSignerErrors/invalid_JSON5622=== CONT TestExternalSignerErrors/wrong_shape5623=== CONT TestExternalSignerErrors/wrong_signature_length5624=== CONT TestExternalSignerErrors/program_error5625=== CONT TestExternalSignerErrors/null5626=== CONT TestExternalSignerErrors/invalid_signature_base645627=== CONT TestExternalSignerErrors/wrong_id5628=== CONT TestExternalSignerErrors/null_signatures5629=== CONT TestExternalSignerErrors/too_few_signatures5630=== CONT TestExternalSignerErrors/missing_signature_name5631=== CONT TestExternalSignerErrors/exit_at_startup5632--- PASS: TestExternalSigner (0.01s)5633=== CONT TestExternalSignerErrors/wrong_signatures_shape5634=== CONT TestExternalSignerErrors/wrong_signature_name5635--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5636--- PASS: TestExternalSignerErrors (0.00s)5637 --- PASS: TestExternalSignerErrors/missing_signatures (0.00s)5638 --- PASS: TestExternalSignerErrors/unrequested_key (0.00s)5639 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.00s)5640 --- PASS: TestExternalSignerErrors/wrong_shape (0.00s)5641 --- PASS: TestExternalSignerErrors/program_error (0.00s)5642 --- PASS: TestExternalSignerErrors/exit_on_request (0.00s)5643 --- PASS: TestExternalSignerErrors/null (0.00s)5644 --- PASS: TestExternalSignerErrors/invalid_JSON (0.00s)5645 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.00s)5646 --- PASS: TestExternalSignerErrors/missing_signature_name (0.00s)5647 --- PASS: TestExternalSignerErrors/null_signatures (0.00s)5648 --- PASS: TestExternalSignerErrors/exit_at_startup (0.00s)5649 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5650 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.00s)5651 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5652 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5653--- PASS: TestExternalSignerCancellation (0.02s)5654PASS5655Running OpenBao signer tests...5656=== RUN TestSign5657=== PAUSE TestSign5658=== RUN TestSignErrors5659=== PAUSE TestSignErrors5660=== RUN TestServe5661=== PAUSE TestServe5662=== RUN TestServeRejectsGarbage5663=== PAUSE TestServeRejectsGarbage5664=== CONT TestSign5665=== CONT TestServe5666=== CONT TestSignErrors5667=== CONT TestServeRejectsGarbage5668=== RUN TestSignErrors/invalid_json5669=== PAUSE TestSignErrors/invalid_json5670=== RUN TestSignErrors/no_prefix5671=== PAUSE TestSignErrors/no_prefix5672=== RUN TestSignErrors/invalid_base645673=== PAUSE TestSignErrors/invalid_base645674=== RUN TestSignErrors/http_error5675=== PAUSE TestSignErrors/http_error5676=== CONT TestSignErrors/invalid_json5677=== CONT TestSignErrors/http_error5678--- PASS: TestServeRejectsGarbage (0.00s)5679--- PASS: TestServe (0.00s)5680=== CONT TestSignErrors/invalid_base645681=== CONT TestSignErrors/no_prefix5682--- PASS: TestSign (0.00s)5683--- PASS: TestSignErrors (0.00s)5684 --- PASS: TestSignErrors/http_error (0.00s)5685 --- PASS: TestSignErrors/invalid_base64 (0.00s)5686 --- PASS: TestSignErrors/no_prefix (0.00s)5687 --- PASS: TestSignErrors/invalid_json (0.00s)5688PASS