niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #483
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestDoServerRequestAttachesToken114=== CONT TestRetryTimeoutOutlastsMaxRetries115=== CONT TestSetClientTLS116=== CONT TestScriptTokenEmptyCommand117--- PASS: TestScriptTokenEmptyCommand (0.00s)118=== CONT TestGetStorePathHash119=== RUN TestGetStorePathHash/valid_store_path120=== PAUSE TestGetStorePathHash/valid_store_path121=== RUN TestGetStorePathHash/basename_without_hyphen_should_error122=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error123=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error124=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error125=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error126=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error127=== CONT TestScriptTokenScriptFails128=== CONT TestConvertHashToNix32129=== CONT TestScriptTokenBadJSON130=== CONT TestScriptTokenEmptyToken131=== CONT TestScriptTokenCachesUntilRefresh132=== CONT TestScriptTokenNoExpiryRerunsEveryCall133=== CONT TestFileTokenEmpty134=== CONT TestFileTokenMissing135=== CONT TestFileTokenReadsAndCaches136=== CONT TestStaticToken137=== CONT TestSetClientTLSErrors138=== CONT TestEncodeNixBase32WithRealHash139=== CONT TestSetClientTLSDoesNotMutateDefaultTransport140=== CONT TestDoWithRetry_BodyReplayedViaGetBody141=== CONT TestPathInfoCACompatibility142=== CONT TestParsePathInfoJSONMultiplePaths143=== CONT TestResolveStorePath144=== CONT TestParsePathInfoJSON145=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess146=== CONT TestPathInfoHashCompatibility147=== CONT TestRateLimiterFeedback148=== RUN TestConvertHashToNix32/SRI_format_to_Nix32149=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths150--- PASS: TestScriptTokenScriptFails (0.00s)151=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)152=== RUN TestRateLimiterFeedback/429_enables_limiter153=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)154=== PAUSE TestRateLimiterFeedback/429_enables_limiter155=== RUN TestRateLimiterFeedback/503_enables_limiter156=== PAUSE TestRateLimiterFeedback/503_enables_limiter157=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter158=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter159=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32160=== RUN TestConvertHashToNix32/already_Nix32_format161=== PAUSE TestConvertHashToNix32/already_Nix32_format162=== RUN TestConvertHashToNix32/invalid_format163=== PAUSE TestConvertHashToNix32/invalid_format164=== CONT TestRegisterUploadedObject_OutlivesPush165=== CONT TestStreamPushGivesUpOnDeadServer1662026/11/11 22:34:52 WARN Rate limiter enabled after throttle name=server-test rate=5167=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter168=== CONT TestStreamPushRequestLine169=== CONT TestStreamPushStopsWhenCanceled170=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths171--- PASS: TestFileTokenEmpty (0.00s)172--- PASS: TestStaticToken (0.00s)173--- PASS: TestFileTokenMissing (0.00s)174--- PASS: TestResolveStorePath (0.00s)175--- PASS: TestScriptTokenBadJSON (0.01s)176=== CONT TestClientSignaturesByStorePath177=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon178=== CONT TestStreamPushIsolatesFailures1792026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:44899180=== RUN TestPathInfoCACompatibility/null_ca_field181=== PAUSE TestPathInfoCACompatibility/null_ca_field182=== RUN TestPathInfoCACompatibility/old_string_format_-_text183=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text184=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon1852026/11/11 22:34:52 ERROR Upload failed error="bad path" count=3186=== CONT TestStreamPushReportsSkippedPaths1872026/11/11 22:34:52 ERROR Upload failed error="connection refused" count=201882026/11/11 22:34:52 ERROR Server seems unavailable, giving up on batch untried=17189=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI190=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI191=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths192--- PASS: TestStreamPushStopsWhenCanceled (0.00s)193--- PASS: TestEncodeNixBase32WithRealHash (0.00s)194--- PASS: TestFileTokenReadsAndCaches (0.00s)195--- PASS: TestClientSignaturesByStorePath (0.00s)196=== CONT TestStreamPushReportsEveryPath1972026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:44899198=== CONT TestStreamPushReportsSignatures199=== CONT TestUploadMultipart_PartsInParallel200=== CONT TestDumpPathMatchesNix201=== RUN TestParsePathInfoJSON/Nix_format2022026/11/11 22:34:52 ERROR Upload failed error=boom count=1203=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive204=== CONT TestStreamPushBatchesUnderLoad2052026/11/11 22:34:52 ERROR Upload failed error=boom count=1206=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter207=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512208=== CONT TestEncodeNixBase32209--- PASS: TestStreamPushReportsSkippedPaths (0.00s)210=== CONT TestDumpPathWriterError211=== RUN TestSetClientTLSErrors/missing_cert_file212=== CONT TestDumpPathSingleFile2132026/11/11 22:34:52 WARN Rate limiter enabled after throttle name=server-test rate=5214=== CONT TestPartSizeForNAR2152026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36595216=== RUN TestPartSizeForNAR/zero_stays_at_minimum217=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun218=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths219=== CONT TestFilterOversizedClosures220=== CONT TestRegisterUploadedObjectReusesConnections221=== RUN TestFilterOversizedClosures/no_limit_keeps_everything2222026/11/11 22:34:52 WARN Rate limiter backed off name=server-test rate=5223=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything2242026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:36595225=== CONT TestCaseHackSuffix2262026/11/11 22:34:52 INFO Garbage collection started227=== CONT TestShellSplitErrors228=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica2292026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:44899230=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false231=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false232=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry233=== CONT TestDoWithRetry_FinalResponseBodyReadable234--- PASS: TestStreamPushIsolatesFailures (0.00s)235--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)236--- PASS: TestScriptTokenEmptyToken (0.01s)237--- PASS: TestDoServerRequestAttachesToken (0.01s)238=== CONT TestUploadMultipart_ReturnsAfterDroppedBody239=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512240--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)241--- PASS: TestStreamPushReportsSignatures (0.00s)242--- PASS: TestStreamPushReportsEveryPath (0.00s)243--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)244--- PASS: TestScriptTokenCachesUntilRefresh (0.09s)245--- PASS: TestShellSplitErrors (0.00s)246--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.09s)247--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.09s)248=== RUN TestEncodeNixBase32/test_string_hash249=== PAUSE TestEncodeNixBase32/test_string_hash250=== RUN TestEncodeNixBase32/empty_input251=== PAUSE TestEncodeNixBase32/empty_input252=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer253=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive254=== RUN TestPathInfoCACompatibility/new_structured_format_-_text255=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text256=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method257=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method258=== CONT TestGetStorePathHash/valid_store_path259=== CONT TestGetStorePathHash/basename_without_hyphen_should_error2602026/11/11 22:34:52 INFO Garbage collection started261=== CONT TestConvertHashToNix32/SRI_format_to_Nix32262=== CONT TestConvertHashToNix32/invalid_format263=== PAUSE TestSetClientTLSErrors/missing_cert_file264=== RUN TestSetClientTLSErrors/missing_key_file265=== PAUSE TestSetClientTLSErrors/missing_key_file266=== RUN TestSetClientTLSErrors/missing_ca_file267=== PAUSE TestSetClientTLSErrors/missing_ca_file268=== RUN TestSetClientTLSErrors/invalid_ca_file2692026/11/11 22:34:52 WARN Rate limiter enabled after throttle name=server-test rate=5270=== PAUSE TestSetClientTLSErrors/invalid_ca_file2712026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:347792722026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:44899273=== CONT TestRateLimiterFeedback/429_enables_limiter2742026/11/11 22:34:52 WARN Rate limiter backed off name=server-test rate=5275=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum276=== RUN TestPartSizeForNAR/small_stays_at_minimum277=== PAUSE TestPartSizeForNAR/small_stays_at_minimum278=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum279=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum280=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts281=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts282=== RUN TestPartSizeForNAR/1_TiB283=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter284=== CONT TestShellSplit2852026/11/11 22:34:52 WARN Rate limiter enabled after throttle name=server-test rate=5286=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true2872026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:36387288=== RUN TestSetClientTLS/rejects_connection_without_client_cert2892026/11/11 22:34:52 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0290=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped291=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped2922026/11/11 22:34:52 WARN Rate limiter backed off name=server-test rate=5293=== RUN TestFilterOversizedClosures/all_closures_skipped294=== PAUSE TestFilterOversizedClosures/all_closures_skipped295--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.09s)296=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths297=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error298=== CONT TestConvertHashToNix32/already_Nix32_format2992026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:44899300=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error301=== CONT TestRateLimiterFeedback/503_enables_limiter302=== CONT TestEncodeNixBase32/test_string_hash303=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon3042026/11/11 22:34:52 WARN Rate limiter enabled after throttle name=s3-test rate=5305=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method3062026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:43039307=== CONT TestPathInfoCACompatibility/null_ca_field308=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI309=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive310=== CONT TestPathInfoCACompatibility/old_string_format_-_text311=== CONT TestSetClientTLSErrors/missing_cert_file312=== PAUSE TestPartSizeForNAR/1_TiB313=== CONT TestSetClientTLSErrors/invalid_ca_file314=== RUN TestPartSizeForNAR/5_TiB_S3_max_object315=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object316=== RUN TestPartSizeForNAR/capped_at_5_GiB317=== PAUSE TestPartSizeForNAR/capped_at_5_GiB3182026/11/11 22:34:52 WARN Rate limiter enabled after throttle name=server-test rate=5319=== CONT TestSetClientTLSErrors/missing_ca_file3202026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:42091321=== CONT TestSetClientTLSErrors/missing_key_file3222026/11/11 22:34:52 WARN Garbage collection finished on another replica. Its statistics are not available here323=== CONT TestFilterOversizedClosures/all_closures_skipped3242026/11/11 22:34:52 WARN Rate limiter backed off name=server-test rate=5325=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true3262026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:44899327=== PAUSE TestParsePathInfoJSON/Nix_format3282026/11/11 22:34:52 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50329=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert330--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)331=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths332=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)333=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512334=== CONT TestPathInfoCACompatibility/new_structured_format_-_text335=== CONT TestEncodeNixBase32/empty_input336=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter337=== CONT TestFilterOversizedClosures/no_limit_keeps_everything338=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3392026/11/11 22:34:52 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000340=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false341=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true342=== CONT TestPartSizeForNAR/1_TiB343=== CONT TestPartSizeForNAR/capped_at_5_GiB344=== RUN TestParsePathInfoJSON/Lix_format345=== PAUSE TestParsePathInfoJSON/Lix_format346=== RUN TestParsePathInfoJSON/empty_input347=== PAUSE TestParsePathInfoJSON/empty_input348=== RUN TestParsePathInfoJSON/whitespace_only349=== PAUSE TestParsePathInfoJSON/whitespace_only350=== RUN TestParsePathInfoJSON/invalid_JSON351=== CONT TestPartSizeForNAR/5_TiB_S3_max_object352=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA353=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA354--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)3552026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:44899356--- PASS: TestShellSplit (0.00s)357=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum358--- PASS: TestConvertHashToNix32 (0.00s)359 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)360 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)361 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)362=== CONT TestPartSizeForNAR/small_stays_at_minimum363=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts364=== CONT TestPartSizeForNAR/zero_stays_at_minimum365=== PAUSE TestParsePathInfoJSON/invalid_JSON366--- PASS: TestGetStorePathHash (0.00s)367 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)368 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)369 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)370 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)371=== CONT TestParsePathInfoJSON/Nix_format372=== RUN TestSetClientTLS/preserves_debug_logging_transport373=== PAUSE TestSetClientTLS/preserves_debug_logging_transport374=== CONT TestSetClientTLS/rejects_connection_without_client_cert375=== CONT TestSetClientTLS/preserves_debug_logging_transport376=== CONT TestParsePathInfoJSON/invalid_JSON3772026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:44899378=== CONT TestParsePathInfoJSON/whitespace_only379=== CONT TestParsePathInfoJSON/empty_input380=== CONT TestParsePathInfoJSON/Lix_format381--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)382--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)383=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA384--- PASS: TestStreamPushBatchesUnderLoad (0.09s)385 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)386 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)387--- PASS: TestParsePathInfoJSONMultiplePaths (0.09s)388 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)389 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)390--- PASS: TestFilterOversizedClosures (0.00s)391 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)392 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)393 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)3942026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:44899395--- PASS: TestPartSizeForNAR (0.09s)396 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)397 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)398 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)399 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)400 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)401 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)402 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)403--- PASS: TestPathInfoHashCompatibility (0.09s)404 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)405 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)406 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)407 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)408--- PASS: TestRateLimiterFeedback (0.01s)409 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)410 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)411 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)412 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)413--- PASS: TestParsePathInfoJSON (0.10s)414 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)415 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)416 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)417 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)418 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)419--- PASS: TestEncodeNixBase32 (0.09s)420 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)421 --- PASS: TestEncodeNixBase32/empty_input (0.00s)422--- PASS: TestSetClientTLSErrors (0.09s)423 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)424 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)425 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)426 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)427--- PASS: TestPathInfoCACompatibility (0.09s)428 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)429 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)430 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)431 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)432 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)4332026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:44899434--- PASS: TestDumpPathSingleFile (0.12s)4352026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:448994362026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:448994372026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:44899438--- PASS: TestCaseHackSuffix (0.03s)4392026/11/11 22:34:52 http: TLS handshake error from 127.0.0.1:37176: remote error: tls: bad certificate4402026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:44899441--- PASS: TestStreamPushRequestLine (0.12s)442--- PASS: TestSetClientTLS (0.11s)443 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.02s)444 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.02s)445 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.03s)4462026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:448994472026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:448994482026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:448994492026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:448994502026/11/11 22:34:52 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:44899451--- PASS: TestRegisterUploadedObjectReusesConnections (0.05s)452--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.15s)453--- PASS: TestDumpPathWriterError (0.14s)454--- PASS: TestDumpPathMatchesNix (0.18s)4552026/11/11 22:34:52 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:41515/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:41515/api/uploads/complete4562026/11/11 22:34:52 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:41515/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:41515/api/uploads/complete4572026/11/11 22:34:52 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4582026/11/11 22:34:52 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4592026/11/11 22:34:52 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:41515/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:41515/api/uploads/complete4602026/11/11 22:34:52 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"461--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)462--- PASS: TestUploadMultipart_PartsInParallel (0.72s)463--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "nixbld".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /build/postgres3755943098/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /build/postgres3755943098/data -l logfile start492493/build/postgres3755943098:5432 - no response4942026-11-11 22:34:54.109 UTC [129] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit4952026-11-11 22:34:54.110 UTC [129] LOG: listening on Unix socket "/build/postgres3755943098/.s.PGSQL.5432"4962026-11-11 22:34:54.115 UTC [136] LOG: database system was shut down at 2026-11-11 22:34:53 UTC4972026-11-11 22:34:54.118 UTC [129] LOG: database system is ready to accept connections498/build/postgres3755943098:5432 - accepting connections499{"timestamp":"2026-11-11T22:34:54.417769052Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"00bed69f-bd5b-4710-8598-aa338befa0a6","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(386)"}500=== RUN TestAbortSucceeded501=== PAUSE TestAbortSucceeded502=== RUN TestService_AuthMiddleware503=== PAUSE TestService_AuthMiddleware504=== RUN TestService_AuthMiddleware_MTLSProxyHeader505=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader506=== RUN TestService_AuthMiddleware_MTLSBoundSubjects507=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects508=== RUN TestService_ReadAuthMiddleware509=== PAUSE TestService_ReadAuthMiddleware510=== RUN TestService_AuthMiddleware_OIDC511=== PAUSE TestService_AuthMiddleware_OIDC512=== RUN TestService_RequireScope_OIDC513=== PAUSE TestService_RequireScope_OIDC514=== RUN TestService_ReadScope_PublicByDefault515=== PAUSE TestService_ReadScope_PublicByDefault516=== RUN TestCacheConfigHandler517=== PAUSE TestCacheConfigHandler518=== RUN TestCacheStatsHandler519=== PAUSE TestCacheStatsHandler520=== RUN TestClientCADerivations521=== PAUSE TestClientCADerivations522=== RUN TestClientErrorHandling523=== PAUSE TestClientErrorHandling524=== RUN TestClientIntegration525=== PAUSE TestClientIntegration526=== RUN TestClientMultipleUploads527=== PAUSE TestClientMultipleUploads528=== RUN TestClientWithDependencies529=== PAUSE TestClientWithDependencies530=== RUN TestClientSharedPathCommittedMidPush531=== PAUSE TestClientSharedPathCommittedMidPush532=== RUN TestPinProtectsFromGC533=== PAUSE TestPinProtectsFromGC534=== RUN TestClientPushesUseOnePush535=== PAUSE TestClientPushesUseOnePush536=== RUN TestClientFallsBackToClosures537=== PAUSE TestClientFallsBackToClosures538=== RUN TestDeleteClosuresRacingPins539=== PAUSE TestDeleteClosuresRacingPins540=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock541=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock542=== RUN TestResolveDBConnectionString543=== PAUSE TestResolveDBConnectionString544=== RUN TestConnectWaitsForAPeerMigration545=== PAUSE TestConnectWaitsForAPeerMigration546=== RUN TestLeadElectsOneAndHandsOver547=== PAUSE TestLeadElectsOneAndHandsOver548=== RUN TestLeadIncumbentWinsAfterRestart5492026/11/11 22:34:54 INFO lead: acquired remote=192.0.2.1:1234 generation=15502026/11/11 22:34:55 INFO lead: released remote=192.0.2.1:1234 generation=15512026/11/11 22:34:55 INFO lead: acquired remote=192.0.2.1:1234 generation=25522026/11/11 22:34:56 INFO lead: released remote=192.0.2.1:1234 generation=2553--- PASS: TestLeadIncumbentWinsAfterRestart (1.88s)554=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent555=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent556=== RUN TestLeadIncumbentAnnouncesAtOnce557=== PAUSE TestLeadIncumbentAnnouncesAtOnce558=== RUN TestLeadGenerationGrowsWithEveryGrant559=== PAUSE TestLeadGenerationGrowsWithEveryGrant560=== RUN TestLeadEndsOnShutdown561=== PAUSE TestLeadEndsOnShutdown562=== RUN TestLeadEndsWhenItsConnectionHangs5632026/11/11 22:34:56 INFO lead: acquired remote=192.0.2.1:1234 generation=15642026-11-11 22:34:57.920 UTC [574] FATAL: terminating connection due to administrator command5652026/11/11 22:34:58 INFO lead: acquired remote=192.0.2.1:1234 generation=25662026/11/11 22:34:59 WARN lead: lock connection lost error="timeout: context deadline exceeded"5672026/11/11 22:34:59 INFO lead: released remote=192.0.2.1:1234 generation=15682026/11/11 22:34:59 INFO lead: released remote=192.0.2.1:1234 generation=2569--- PASS: TestLeadEndsWhenItsConnectionHangs (3.12s)570=== RUN TestGCAdvisoryLockBlocksConcurrentRun571=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun572=== RUN TestGCBugBareHashReferences573=== PAUSE TestGCBugBareHashReferences574=== RUN TestGCMetrics575=== PAUSE TestGCMetrics576=== RUN TestPushDedupSurvivesConcurrentGC577=== PAUSE TestPushDedupSurvivesConcurrentGC578=== RUN TestDeduplicatedObjectsRecordedAsPending579=== PAUSE TestDeduplicatedObjectsRecordedAsPending580=== RUN TestGCSweepSkipsPendingObjects581=== PAUSE TestGCSweepSkipsPendingObjects582=== RUN TestTombstonedObjectOfferedWithoutWaiting583=== PAUSE TestTombstonedObjectOfferedWithoutWaiting584=== RUN TestGCSweepDeliversEachKeyOnce585=== PAUSE TestGCSweepDeliversEachKeyOnce586=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection587=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection588=== RUN TestForceGCDuringPushOffersSweptObject589=== PAUSE TestForceGCDuringPushOffersSweptObject590=== RUN TestFailedS3DeleteKeepsTombstone591=== PAUSE TestFailedS3DeleteKeepsTombstone592=== RUN TestSweepSkipsRowsLockedByPush593=== PAUSE TestSweepSkipsRowsLockedByPush594=== RUN TestRegisterCompletedObjectNeedsPendingRow595=== PAUSE TestRegisterCompletedObjectNeedsPendingRow596=== RUN TestGetExistingObjectsLocksRows597=== PAUSE TestGetExistingObjectsLocksRows598=== RUN TestSweepRechecksPendingAfterLocking599=== PAUSE TestSweepRechecksPendingAfterLocking600=== RUN TestPushWaitsForSweepHoldingRow601=== PAUSE TestPushWaitsForSweepHoldingRow602=== RUN TestCommitRacingPendingCleanupKeepsObjects603=== PAUSE TestCommitRacingPendingCleanupKeepsObjects604=== RUN TestGCEndsOnShutdown605=== PAUSE TestGCEndsOnShutdown606=== RUN TestGCTaskStore_StartNew607=== PAUSE TestGCTaskStore_StartNew608=== RUN TestGCTaskStore_DeduplicateSameParams609=== PAUSE TestGCTaskStore_DeduplicateSameParams610=== RUN TestGCTaskStore_ConflictDifferentParams611=== PAUSE TestGCTaskStore_ConflictDifferentParams612=== RUN TestGCTaskStore_GetEmpty613=== PAUSE TestGCTaskStore_GetEmpty614=== RUN TestGCTaskStore_GetReturnsLatest615=== PAUSE TestGCTaskStore_GetReturnsLatest616=== RUN TestGCTaskStore_CompletedAllowsNewTask617=== PAUSE TestGCTaskStore_CompletedAllowsNewTask618=== RUN TestGCTaskStore_PhaseUpdates619=== PAUSE TestGCTaskStore_PhaseUpdates620=== RUN TestGCTaskStore_Fail621=== PAUSE TestGCTaskStore_Fail622=== RUN TestGracefulShutdownDrainsInflight623=== PAUSE TestGracefulShutdownDrainsInflight624=== RUN TestService_healthCheckHandler625=== PAUSE TestService_healthCheckHandler626=== RUN TestService_readinessHandler627=== PAUSE TestService_readinessHandler628=== RUN TestGenerateLandingPage629=== PAUSE TestGenerateLandingPage630=== RUN TestEnsureAbortUploadsRule631=== PAUSE TestEnsureAbortUploadsRule632=== RUN TestCacheConfigHandlerMaxNarSize633=== PAUSE TestCacheConfigHandlerMaxNarSize634=== RUN TestCreatePendingClosureRejectsOversizedNAR635=== PAUSE TestCreatePendingClosureRejectsOversizedNAR636=== RUN TestNARDeduplicationMetadataUploadBug637=== PAUSE TestNARDeduplicationMetadataUploadBug638=== RUN TestMetricsInventory639=== PAUSE TestMetricsInventory640=== RUN TestMetricsCommit641=== PAUSE TestMetricsCommit642=== RUN TestService_NativeMTLS643=== PAUSE TestService_NativeMTLS644=== RUN TestServerTLSConfig645=== PAUSE TestServerTLSConfig646=== RUN TestMultipartCleanup647=== PAUSE TestMultipartCleanup648=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded649=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded650=== RUN TestPendingCleanupUsesOneCutoff651=== PAUSE TestPendingCleanupUsesOneCutoff652=== RUN TestObjectStatsTrigger653=== PAUSE TestObjectStatsTrigger654=== RUN TestObjectStatsUpdatedOncePerCommit655=== PAUSE TestObjectStatsUpdatedOncePerCommit656=== RUN TestOrphanedObjectsGC657=== PAUSE TestOrphanedObjectsGC658=== RUN TestOrphanedObjectsGCStressTest659=== PAUSE TestOrphanedObjectsGCStressTest660=== RUN TestCreatePin_ReservedPins661=== PAUSE TestCreatePin_ReservedPins662=== RUN TestPresentReportsOnlyClosureRoots663=== PAUSE TestPresentReportsOnlyClosureRoots664=== RUN TestPresentRefreshesOnlyStaleClosures665=== PAUSE TestPresentRefreshesOnlyStaleClosures666=== RUN TestPresentNotReportedWhileGCDeletesClosure667=== PAUSE TestPresentNotReportedWhileGCDeletesClosure668=== RUN TestParseSingleRange669=== PAUSE TestParseSingleRange670=== RUN TestProxyHeadersOnlyTrustedOnSocket6712026/11/11 22:34:59 INFO Starting HTTP server address=127.0.0.1:337476722026/11/11 22:34:59 INFO Starting HTTP server address=proxy.sock6732026/11/11 22:34:59 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6742026/11/11 22:34:59 INFO Shutdown signal received, draining in-flight requests timeout=10s675--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)676=== RUN TestIsValidCachePath677=== PAUSE TestIsValidCachePath678=== RUN TestReadProxyNarinfo679=== PAUSE TestReadProxyNarinfo680=== RUN TestReadProxyNarinfoAlreadyDecompressed681=== PAUSE TestReadProxyNarinfoAlreadyDecompressed682=== RUN TestReadProxyContentNegotiation683=== PAUSE TestReadProxyContentNegotiation684=== RUN TestReadProxyNarStreaming685=== PAUSE TestReadProxyNarStreaming686=== RUN TestReadProxy404687=== PAUSE TestReadProxy404688=== RUN TestReadProxyInvalidPath689=== PAUSE TestReadProxyInvalidPath690=== RUN TestReadProxyHead691=== PAUSE TestReadProxyHead692=== RUN TestReadProxyOutlastsServerWriteTimeout693=== PAUSE TestReadProxyOutlastsServerWriteTimeout694=== RUN TestReadProxyConditionalGet695=== PAUSE TestReadProxyConditionalGet696=== RUN TestReadProxyRootRedirectsToIndexHTML697=== PAUSE TestReadProxyRootRedirectsToIndexHTML698=== RUN TestReadProxyDisabled699=== PAUSE TestReadProxyDisabled700=== RUN TestReadRedirectNar701=== PAUSE TestReadRedirectNar702=== RUN TestReadRedirectKeepsNarinfoProxied703=== PAUSE TestReadRedirectKeepsNarinfoProxied704=== RUN TestReadProxyRangeRequest705=== PAUSE TestReadProxyRangeRequest706=== RUN TestReadRedirectUsesPublicS3URL707=== PAUSE TestReadRedirectUsesPublicS3URL708=== RUN TestPush_OverlappingRootsStoreOneRowPerKey709=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey710=== RUN TestPush_CompleteCommitsEveryRoot711=== PAUSE TestPush_CompleteCommitsEveryRoot712=== RUN TestPush_RepeatedRootCommitsOnce713=== PAUSE TestPush_RepeatedRootCommitsOnce714=== RUN TestPush_CompleteWaitsForCleanupOfThePush715=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush716=== RUN TestPush_RootsAreLockedInByteOrder717=== PAUSE TestPush_RootsAreLockedInByteOrder718=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit719=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit720=== RUN TestPush_RejectsBadRequests721=== PAUSE TestPush_RejectsBadRequests722=== RUN TestPush_SignsNarinfosOfItsPendingObjects723=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects724=== RUN TestSharedNarBothUploadsComplete725=== PAUSE TestSharedNarBothUploadsComplete726=== RUN TestCompleteMultipartUpload_ErrorButObjectExists727=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists728=== RUN TestCompletedNarNotReofferedAcrossClosures729=== PAUSE TestCompletedNarNotReofferedAcrossClosures730=== RUN TestPresignedUploadRegisteredBeforeCommit731=== PAUSE TestPresignedUploadRegisteredBeforeCommit732=== RUN TestService_Rustfstest733=== PAUSE TestService_Rustfstest734=== RUN TestParseSize735=== PAUSE TestParseSize736=== RUN TestSkippedUploadsHandler737=== PAUSE TestSkippedUploadsHandler738=== RUN TestSystemdListenerNotActivated739--- PASS: TestSystemdListenerNotActivated (0.00s)740=== RUN TestWatchdogBeatsWhenHealthy741--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)742=== RUN TestWatchdogSkipsWhenUnhealthy7432026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7442026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7452026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7462026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7472026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7482026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7492026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/11 22:34:59 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"753--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)754=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle755=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle756=== RUN TestProxyWriteTimeout757=== PAUSE TestProxyWriteTimeout758=== RUN TestPendingClosureWriteTimeout759=== PAUSE TestPendingClosureWriteTimeout760=== RUN TestIsValidUploadKey761=== PAUSE TestIsValidUploadKey762=== RUN TestUploadHandlersRejectInvalidKeys763=== PAUSE TestUploadHandlersRejectInvalidKeys764=== RUN TestUploadHandlersRejectOversizedBody765=== PAUSE TestUploadHandlersRejectOversizedBody766=== RUN TestService_cleanupPendingClosuresHandler767=== PAUSE TestService_cleanupPendingClosuresHandler768=== RUN TestService_createPendingClosureHandler769=== PAUSE TestService_createPendingClosureHandler770=== RUN TestService_verifyS3Integrity771=== PAUSE TestService_verifyS3Integrity772=== RUN TestService_verifyS3ReuploadsStaleNarinfo773=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo774=== RUN TestCompleteMultipartUnregistered775=== PAUSE TestCompleteMultipartUnregistered776=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT777=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT778=== CONT TestAbortSucceeded779=== CONT TestPendingClosureWriteTimeout780=== RUN TestPendingClosureWriteTimeout/empty781=== PAUSE TestPendingClosureWriteTimeout/empty782=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT783=== CONT TestCompleteMultipartUnregistered784=== CONT TestService_verifyS3ReuploadsStaleNarinfo785=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed786=== CONT TestService_verifyS3Integrity787=== CONT TestService_createPendingClosureHandler788=== CONT TestService_cleanupPendingClosuresHandler789=== CONT TestUploadHandlersRejectOversizedBody790=== CONT TestUploadHandlersRejectInvalidKeys791=== CONT TestIsValidUploadKey792--- PASS: TestAbortSucceeded (0.00s)793=== CONT TestCreatePin_ReservedPins794=== CONT TestOrphanedObjectsGCStressTest795=== CONT TestOrphanedObjectsGC796=== CONT TestObjectStatsUpdatedOncePerCommit797=== CONT TestObjectStatsTrigger798=== CONT TestPendingCleanupUsesOneCutoff799=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded800=== CONT TestMultipartCleanup801=== CONT TestServerTLSConfig802=== CONT TestService_NativeMTLS803=== CONT TestMetricsCommit804=== CONT TestMetricsInventory805=== CONT TestNARDeduplicationMetadataUploadBug806=== CONT TestCreatePendingClosureRejectsOversizedNAR807=== RUN TestPendingClosureWriteTimeout/400_objects808=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed809=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed810=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed811=== RUN TestServerTLSConfig/no_client_CA812=== PAUSE TestServerTLSConfig/no_client_CA813=== RUN TestServerTLSConfig/missing_CA_file814=== PAUSE TestPendingClosureWriteTimeout/400_objects815=== RUN TestPendingClosureWriteTimeout/670k_objects816=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost8172026/11/11 22:34:59 INFO Received uploads request method=POST path=/api/pending_closures818=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info819=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info820=== RUN TestIsValidUploadKey/narinfo821--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)822=== CONT TestCacheConfigHandlerMaxNarSize823=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal824=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal825=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key826=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key827=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key828=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key829=== PAUSE TestServerTLSConfig/missing_CA_file830=== PAUSE TestPendingClosureWriteTimeout/670k_objects831=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost832=== PAUSE TestIsValidUploadKey/narinfo833=== RUN TestIsValidUploadKey/nar_zst834=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed835=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed836=== RUN TestServerTLSConfig/not_a_PEM_file837=== PAUSE TestIsValidUploadKey/nar_zst838--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)839=== CONT TestGenerateLandingPage840=== CONT TestEnsureAbortUploadsRule841=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline842=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed843=== PAUSE TestServerTLSConfig/not_a_PEM_file844=== CONT TestService_readinessHandler845=== RUN TestIsValidUploadKey/nar_xz846=== PAUSE TestIsValidUploadKey/nar_xz847=== RUN TestIsValidUploadKey/nar_plain848=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline849=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed850=== PAUSE TestIsValidUploadKey/nar_plain851=== RUN TestIsValidUploadKey/listing852=== CONT TestGracefulShutdownDrainsInflight853=== PAUSE TestIsValidUploadKey/listing8542026/11/11 22:34:59 INFO Starting HTTP server address=127.0.0.1:45345855=== CONT TestService_healthCheckHandler856=== RUN TestIsValidUploadKey/build_log857=== PAUSE TestIsValidUploadKey/build_log858--- PASS: TestGenerateLandingPage (0.00s)8592026/11/11 22:34:59 INFO Shutdown signal received, draining in-flight requests timeout=10s860=== RUN TestIsValidUploadKey/build_log_home-manager_file861=== PAUSE TestIsValidUploadKey/build_log_home-manager_file862=== CONT TestGCTaskStore_Fail863--- PASS: TestGCTaskStore_Fail (0.00s)864=== RUN TestIsValidUploadKey/build_log_plus_in_name865=== PAUSE TestIsValidUploadKey/build_log_plus_in_name866=== RUN TestIsValidUploadKey/build_log_question_mark867=== PAUSE TestIsValidUploadKey/build_log_question_mark868=== RUN TestIsValidUploadKey/build_log_equals869=== CONT TestGCTaskStore_PhaseUpdates870--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)871=== PAUSE TestIsValidUploadKey/build_log_equals872=== CONT TestGCTaskStore_CompletedAllowsNewTask873--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)874=== CONT TestGCTaskStore_GetReturnsLatest875--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)876=== CONT TestGCTaskStore_GetEmpty877--- PASS: TestGCTaskStore_GetEmpty (0.00s)878=== RUN TestIsValidUploadKey/realisation879=== PAUSE TestIsValidUploadKey/realisation880=== RUN TestIsValidUploadKey/realisation_plus_in_output881=== PAUSE TestIsValidUploadKey/realisation_plus_in_output882=== CONT TestGCTaskStore_ConflictDifferentParams883--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)884=== CONT TestGCTaskStore_DeduplicateSameParams885--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)886=== RUN TestIsValidUploadKey/nix-cache-info887=== CONT TestGCTaskStore_StartNew888--- PASS: TestGCTaskStore_StartNew (0.00s)889=== PAUSE TestIsValidUploadKey/nix-cache-info890=== RUN TestIsValidUploadKey/index.html891=== PAUSE TestIsValidUploadKey/index.html892=== RUN TestIsValidUploadKey/narinfo_key,_nar_type893=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type894=== RUN TestIsValidUploadKey/nar_key,_narinfo_type895=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type896=== RUN TestIsValidUploadKey/listing_key,_narinfo_type897=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type898=== RUN TestIsValidUploadKey/traversal899=== PAUSE TestIsValidUploadKey/traversal900=== RUN TestIsValidUploadKey/traversal_nar901=== PAUSE TestIsValidUploadKey/traversal_nar902=== RUN TestIsValidUploadKey/absolute903=== PAUSE TestIsValidUploadKey/absolute904=== RUN TestIsValidUploadKey/empty_key905=== PAUSE TestIsValidUploadKey/empty_key906=== RUN TestIsValidUploadKey/unknown_type907=== PAUSE TestIsValidUploadKey/unknown_type908=== CONT TestGCEndsOnShutdown909=== RUN TestGCEndsOnShutdown/before_the_run910=== CONT TestCommitRacingPendingCleanupKeepsObjects911=== PAUSE TestGCEndsOnShutdown/before_the_run912=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete913=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete914=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark915=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark916=== CONT TestPushWaitsForSweepHoldingRow9172026/11/11 22:34:59 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39581/oidc918--- PASS: TestGracefulShutdownDrainsInflight (0.07s)919=== CONT TestSweepRechecksPendingAfterLocking920=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure921=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure922=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart923=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart924=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts925=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts926=== CONT TestGetExistingObjectsLocksRows9272026/11/11 22:35:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9282026/11/11 22:35:00 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst929--- PASS: TestCompleteMultipartUnregistered (0.37s)930=== CONT TestRegisterCompletedObjectNeedsPendingRow9312026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures932--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.42s)933=== CONT TestSweepSkipsRowsLockedByPush9342026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures9352026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures9362026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures9372026/11/11 22:35:00 INFO Received cleanup request method=DELETE path=/api/pending_closures9382026/11/11 22:35:00 WARN mTLS auth: subject not in bound subjects subject="CN=reader"9392026/11/11 22:35:00 WARN mTLS auth: subject not in bound subjects subject="CN=reader"940--- PASS: TestService_NativeMTLS (0.51s)941=== CONT TestFailedS3DeleteKeepsTombstone942=== NAME TestNARDeduplicationMetadataUploadBug943 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug1302532486/001/store/vhc9nc91jv4cvxrrmn1fv7k5r946psqq-file1.txt9442026/11/11 22:35:00 INFO Received cleanup request method=DELETE path=/api/pending_closures9452026/11/11 22:35:00 INFO Aborted multipart uploads count=0 kept=09462026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures9472026/11/11 22:35:00 INFO Received cleanup request method=DELETE path=/api/pending_closures9482026/11/11 22:35:00 INFO Aborted multipart uploads count=1 kept=09492026/11/11 22:35:00 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9502026-11-11 22:35:00.422 UTC [654] ERROR: Closure does not exist: id=19512026-11-11 22:35:00.422 UTC [654] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE9522026-11-11 22:35:00.422 UTC [654] STATEMENT: -- name: CommitPendingClosure :one953 SELECT commit_pending_closure($1::bigint)::bigint954 955--- PASS: TestService_cleanupPendingClosuresHandler (0.56s)956=== CONT TestForceGCDuringPushOffersSweptObject957=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows958=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows959=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check960=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check961=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection9622026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures9632026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures9642026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures9652026/11/11 22:35:00 INFO Received push request method=POST path=/api/pushes9662026/11/11 22:35:00 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)9672026/11/11 22:35:00 INFO Uploading vhc9nc91jv4cvxrrmn1fv7k5r946psqq-file1.txt (160B)968--- PASS: TestObjectStatsTrigger (0.62s)969=== CONT TestGCSweepDeliversEachKeyOnce9702026/11/11 22:35:00 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"9712026/11/11 22:35:00 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9722026/11/11 22:35:00 WARN Failed to register uploaded object key=vhc9nc91jv4cvxrrmn1fv7k5r946psqq.ls error="server returned 404: 404 page not found\n"9732026/11/11 22:35:00 INFO Signed narinfos id=1 count=19742026/11/11 22:35:00 INFO Uploading 1 narinfos9752026/11/11 22:35:00 INFO Received complete push request method=POST path=/api/pushes/1/complete9762026/11/11 22:35:00 WARN Failed to register uploaded object key=vhc9nc91jv4cvxrrmn1fv7k5r946psqq.narinfo error="server returned 404: 404 page not found\n"9772026/11/11 22:35:00 INFO Completed push id=1 objects=3 duration=5.52084ms9782026/11/11 22:35:00 INFO Upload complete. (75ms)979=== NAME TestNARDeduplicationMetadataUploadBug980 metadata_upload_test.go:54: Retrieved narinfo from S3:981 StorePath: /build/TestNARDeduplicationMetadataUploadBug1302532486/001/store/vhc9nc91jv4cvxrrmn1fv7k5r946psqq-file1.txt982 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst983 Compression: zstd984 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf985 NarSize: 160986 References: 987 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf988 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)989 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):990 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}991--- PASS: TestMetricsInventory (0.67s)992=== CONT TestTombstonedObjectOfferedWithoutWaiting9932026/11/11 22:35:00 INFO Received push request method=POST path=/api/pushes994=== NAME TestNARDeduplicationMetadataUploadBug995 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug1302532486/001/store/2sxf9699143vml05bkg9m9c2rbsk444y-file2.txt9962026/11/11 22:35:00 INFO Received complete push request method=POST path=/api/pushes/1/complete9972026/11/11 22:35:00 INFO Completed push id=1 objects=4 duration=7.115281ms9982026/11/11 22:35:00 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete9992026/11/11 22:35:00 INFO Completed upload id=2 objects=2 duration=1.02186ms1000--- PASS: TestMetricsCommit (0.71s)1001=== CONT TestGCSweepSkipsPendingObjects10022026/11/11 22:35:00 INFO Received push request method=POST path=/api/pushes10032026/11/11 22:35:00 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)10042026/11/11 22:35:00 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign10052026/11/11 22:35:00 WARN Failed to register uploaded object key=2sxf9699143vml05bkg9m9c2rbsk444y.ls error="server returned 404: 404 page not found\n"10062026/11/11 22:35:00 INFO Signed narinfos id=2 count=110072026/11/11 22:35:00 INFO Uploading 1 narinfos10082026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures10092026/11/11 22:35:00 INFO Received complete push request method=POST path=/api/pushes/2/complete10102026/11/11 22:35:00 WARN Failed to register uploaded object key=2sxf9699143vml05bkg9m9c2rbsk444y.narinfo error="server returned 404: 404 page not found\n"10112026/11/11 22:35:00 INFO Completed push id=2 objects=3 duration=2.96496ms10122026/11/11 22:35:00 INFO Upload complete. (57ms)1013=== NAME TestNARDeduplicationMetadataUploadBug1014 metadata_upload_test.go:76: Retrieved narinfo from S3:1015 StorePath: /build/TestNARDeduplicationMetadataUploadBug1302532486/001/store/2sxf9699143vml05bkg9m9c2rbsk444y-file2.txt1016 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1017 Compression: zstd1018 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1019 NarSize: 1601020 References: 1021 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1022 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1023 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1024 {"version":1,"root":{"type":"regular","size":44}}1025--- PASS: TestNARDeduplicationMetadataUploadBug (0.78s)1026=== CONT TestDeduplicatedObjectsRecordedAsPending10272026/11/11 22:35:00 INFO Aborted multipart uploads count=0 kept=010282026/11/11 22:35:00 WARN Force mode enabled - objects will be deleted immediately without grace period10292026/11/11 22:35:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010302026/11/11 22:35:00 INFO Vacuumed table table=pending_closures10312026/11/11 22:35:00 INFO Vacuumed table table=pending_objects10322026/11/11 22:35:00 INFO Vacuumed table table=multipart_uploads10332026/11/11 22:35:00 INFO Vacuumed table table=closures10342026/11/11 22:35:00 INFO Vacuumed table table=objects10352026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures1036--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.76s)1037=== CONT TestPushDedupSurvivesConcurrentGC1038--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.84s)1039=== CONT TestGCMetrics1040--- PASS: TestObjectStatsUpdatedOncePerCommit (0.78s)1041=== CONT TestGCBugBareHashReferences10422026/11/11 22:35:00 WARN readiness check failed error="closed pool"1043--- PASS: TestService_readinessHandler (0.79s)1044=== CONT TestGCAdvisoryLockBlocksConcurrentRun1045--- PASS: TestService_healthCheckHandler (0.82s)1046=== CONT TestLeadEndsOnShutdown10472026/11/11 22:35:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10482026/11/11 22:35:00 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10492026/11/11 22:35:00 WARN Refused reserved pin name=worker-x86_64-linux10502026/11/11 22:35:00 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10512026/11/11 22:35:00 INFO Received create pin request method=POST path=/api/pins/my-app10522026/11/11 22:35:00 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1053--- PASS: TestCreatePin_ReservedPins (0.91s)1054=== CONT TestLeadGenerationGrowsWithEveryGrant10552026/11/11 22:35:00 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MzdhYjcwMGEtZjMwYi00NzhlLWIzNzgtZTkzNjRiNmUxZTQzLmFmMGVkYzM4LWExZDYtNGY3YS1hODMwLWZlYzdmMjZmMjRmYXgxNzk0NDM2NTAwMzA5Nzk2Mzky parts=1010562026/11/11 22:35:00 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10572026/11/11 22:35:00 INFO Completed upload id=1 objects=2 duration=4.39635ms10582026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures10592026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures10602026/11/11 22:35:00 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10612026/11/11 22:35:00 WARN Found objects in DB but missing from S3, will re-upload count=11062--- PASS: TestService_verifyS3Integrity (0.94s)1063=== CONT TestLeadIncumbentAnnouncesAtOnce1064--- PASS: TestGetExistingObjectsLocksRows (0.75s)1065=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent10662026/11/11 22:35:00 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket25 days=21067--- PASS: TestEnsureAbortUploadsRule (0.93s)1068=== CONT TestLeadElectsOneAndHandsOver10692026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures10702026/11/11 22:35:00 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10712026/11/11 22:35:00 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MzdhYjcwMGEtZjMwYi00NzhlLWIzNzgtZTkzNjRiNmUxZTQzLjQxYWE4Y2U5LTdmNTItNGU1My04ZDZiLWEwM2RmNjY5MWZiOHgxNzk0NDM2NTAwNDQwMjMwNTQx parts=1010722026/11/11 22:35:00 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10732026/11/11 22:35:00 INFO Aborted multipart uploads count=0 kept=010742026/11/11 22:35:00 INFO Completed upload id=1 objects=2 duration=5.19936ms10752026/11/11 22:35:00 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010762026/11/11 22:35:00 INFO Received uploads request method=POST path=/api/pending_closures10772026/11/11 22:35:00 WARN Force mode enabled - objects will be deleted immediately without grace period10782026/11/11 22:35:00 INFO Starting cleanup of old closures method=DELETE path=/api/closures10792026/11/11 22:35:00 INFO Aborted multipart uploads count=0 kept=01080--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.70s)1081=== CONT TestConnectWaitsForAPeerMigration10822026/11/11 22:35:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010832026/11/11 22:35:00 INFO Vacuumed table table=pending_closures10842026/11/11 22:35:00 INFO Vacuumed table table=pending_objects10852026/11/11 22:35:00 INFO Vacuumed table table=multipart_uploads10862026/11/11 22:35:00 INFO Vacuumed table table=closures10872026/11/11 22:35:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010882026/11/11 22:35:00 INFO Vacuumed table table=objects1089--- PASS: TestSweepRechecksPendingAfterLocking (0.95s)1090=== CONT TestResolveDBConnectionString1091=== RUN TestResolveDBConnectionString/flag_wins10922026/11/11 22:35:00 INFO Vacuumed table table=pending_closures1093=== PAUSE TestResolveDBConnectionString/flag_wins1094=== RUN TestResolveDBConnectionString/file_when_flag_empty1095=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1096=== RUN TestResolveDBConnectionString/missing_file_is_an_error1097=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1098=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1099=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1100=== RUN TestResolveDBConnectionString/nothing_configured1101=== PAUSE TestResolveDBConnectionString/nothing_configured1102=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock11032026/11/11 22:35:00 INFO Vacuumed table table=pending_objects11042026/11/11 22:35:00 INFO Vacuumed table table=multipart_uploads11052026/11/11 22:35:00 INFO Vacuumed table table=closures11062026/11/11 22:35:00 INFO Vacuumed table table=objects11072026/11/11 22:35:00 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001108--- PASS: TestService_createPendingClosureHandler (1.12s)1109=== CONT TestDeleteClosuresRacingPins11102026/11/11 22:35:00 INFO Received cleanup request method=DELETE path=/api/pending_closures11112026/11/11 22:35:00 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket26/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""11122026/11/11 22:35:00 INFO Aborted multipart uploads count=0 kept=111132026/11/11 22:35:00 INFO Received cleanup request method=DELETE path=/api/pending_closures11142026/11/11 22:35:00 INFO Aborted multipart uploads count=1 kept=011152026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures1116--- PASS: TestMultipartCleanup (1.08s)1117=== CONT TestClientFallsBackToClosures11182026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=011192026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period11202026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011212026/11/11 22:35:01 INFO Vacuumed table table=pending_closures11222026/11/11 22:35:01 INFO Vacuumed table table=pending_objects11232026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads11242026/11/11 22:35:01 INFO Vacuumed table table=closures11252026/11/11 22:35:01 INFO Vacuumed table table=objects11262026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=01127=== NAME TestOrphanedObjectsGC1128 orphaned_objects_gc_test.go:287: GC Test Summary:1129 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1130 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1131 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1132 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1133 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1134--- PASS: TestOrphanedObjectsGC (1.16s)1135=== CONT TestClientPushesUseOnePush11362026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period11372026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures11382026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=011392026/11/11 22:35:01 INFO Vacuumed table table=pending_closures11402026/11/11 22:35:01 INFO Vacuumed table table=pending_objects11412026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads11422026/11/11 22:35:01 INFO Vacuumed table table=closures11432026/11/11 22:35:01 INFO Vacuumed table table=objects1144--- PASS: TestSweepSkipsRowsLockedByPush (0.79s)1145=== CONT TestPinProtectsFromGC11462026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=011472026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period11482026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures11492026/11/11 22:35:01 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."11502026/11/11 22:35:01 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1151--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.58s)1152=== CONT TestReadProxyRangeRequest11532026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=01154--- PASS: TestFailedS3DeleteKeepsTombstone (0.74s)1155=== CONT TestClientSharedPathCommittedMidPush11562026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period1157--- PASS: TestPushWaitsForSweepHoldingRow (1.21s)1158=== CONT TestClientWithDependencies11592026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures11602026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=011612026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=011622026/11/11 22:35:01 INFO Vacuumed table table=pending_closures11632026/11/11 22:35:01 INFO Vacuumed table table=pending_objects11642026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads11652026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period11662026/11/11 22:35:01 INFO Vacuumed table table=closures11672026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures11682026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011692026/11/11 22:35:01 INFO Vacuumed table table=objects1170--- PASS: TestGCSweepDeliversEachKeyOnce (0.69s)1171=== CONT TestProxyWriteTimeout1172=== RUN TestProxyWriteTimeout/narinfo1173=== PAUSE TestProxyWriteTimeout/narinfo1174=== RUN TestProxyWriteTimeout/1_GiB_nar1175=== PAUSE TestProxyWriteTimeout/1_GiB_nar1176=== RUN TestProxyWriteTimeout/10_GiB_nar1177=== PAUSE TestProxyWriteTimeout/10_GiB_nar1178=== RUN TestProxyWriteTimeout/unknown_size1179=== PAUSE TestProxyWriteTimeout/unknown_size1180=== CONT TestClientMultipleUploads11812026/11/11 22:35:01 INFO Vacuumed table table=pending_closures1182--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.53s)1183=== CONT TestClientIntegration11842026/11/11 22:35:01 INFO Vacuumed table table=pending_objects11852026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads11862026/11/11 22:35:01 INFO Vacuumed table table=closures11872026/11/11 22:35:01 INFO Vacuumed table table=objects1188--- PASS: TestGCSweepSkipsPendingObjects (0.61s)1189=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle11902026/11/11 22:35:01 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1191--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.49s)1192=== CONT TestClientErrorHandling1193=== RUN TestClientErrorHandling/InvalidStorePath1194=== PAUSE TestClientErrorHandling/InvalidStorePath1195=== RUN TestClientErrorHandling/InvalidAuthToken1196=== PAUSE TestClientErrorHandling/InvalidAuthToken1197=== RUN TestClientErrorHandling/ServerNotAvailable1198=== PAUSE TestClientErrorHandling/ServerNotAvailable1199=== CONT TestSkippedUploadsHandler12002026/11/11 22:35:01 INFO Client skipped oversized paths paths=3 nar_bytes=50000000001201--- PASS: TestSkippedUploadsHandler (0.01s)1202=== CONT TestParseSize1203--- PASS: TestParseSize (0.00s)1204=== CONT TestService_Rustfstest12052026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=012062026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period12072026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012082026/11/11 22:35:01 INFO Vacuumed table table=pending_closures12092026/11/11 22:35:01 INFO Vacuumed table table=pending_objects12102026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads12112026/11/11 22:35:01 INFO Vacuumed table table=closures12122026/11/11 22:35:01 INFO Vacuumed table table=objects1213--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.83s)1214=== CONT TestClientCADerivations1215--- PASS: TestGCMetrics (0.56s)1216=== CONT TestPresignedUploadRegisteredBeforeCommit12172026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures12182026/11/11 22:35:01 INFO lead: acquired remote=192.0.2.1:1234 generation=112192026/11/11 22:35:01 INFO lead: acquired remote=192.0.2.1:1234 generation=112202026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=012212026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012222026/11/11 22:35:01 INFO Vacuumed table table=pending_closures12232026/11/11 22:35:01 INFO Vacuumed table table=pending_objects12242026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads12252026/11/11 22:35:01 INFO Vacuumed table table=closures12262026/11/11 22:35:01 INFO Vacuumed table table=objects12272026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=012282026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period12292026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012302026/11/11 22:35:01 INFO lead: acquired remote=192.0.2.1:1234 generation=112312026/11/11 22:35:01 INFO lead: released remote=192.0.2.1:1234 generation=11232--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.60s)1233=== CONT TestCacheStatsHandler12342026/11/11 22:35:01 INFO Vacuumed table table=pending_closures12352026/11/11 22:35:01 INFO Vacuumed table table=pending_objects12362026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads12372026/11/11 22:35:01 INFO Vacuumed table table=closures12382026/11/11 22:35:01 INFO Vacuumed table table=objects1239--- PASS: TestPushDedupSurvivesConcurrentGC (0.73s)1240=== CONT TestCompletedNarNotReofferedAcrossClosures12412026/11/11 22:35:01 INFO lead: acquired remote=192.0.2.1:1234 generation=112422026/11/11 22:35:01 INFO lead: acquired remote=192.0.2.1:1234 generation=11243--- PASS: TestGCBugBareHashReferences (0.84s)1244=== CONT TestCacheConfigHandler1245=== RUN TestCacheConfigHandler/full_config,_no_issuer1246=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1247=== RUN TestCacheConfigHandler/no_cache_url_configured1248=== PAUSE TestCacheConfigHandler/no_cache_url_configured1249=== RUN TestCacheConfigHandler/no_signing_keys1250=== PAUSE TestCacheConfigHandler/no_signing_keys1251=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1252=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1253=== CONT TestCompleteMultipartUpload_ErrorButObjectExists1254--- PASS: TestReadProxyRangeRequest (0.54s)1255=== CONT TestService_ReadScope_PublicByDefault1256--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.75s)1257=== CONT TestSharedNarBothUploadsComplete1258=== NAME TestPinProtectsFromGC1259 client_integration_test.go:778: Pinned store path: /build/TestPinProtectsFromGC3205746395/001/store/51vahj0rv9cwzpikyga4jx4rd0pmvswv-pinned-file.txt1260 client_integration_test.go:779: Unpinned store path: /build/TestPinProtectsFromGC3205746395/001/store/basdyl4zp0cwnkgys6yrkvbkcgfx9b34-unpinned-file.txt12612026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures12622026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures12632026/11/11 22:35:01 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)12642026/11/11 22:35:01 INFO Uploading sdcc8ri34ak7b6yzffq0gzqv9nyq1wkf-b (216B)12652026/11/11 22:35:01 INFO Uploading 657zbpxmz994w1cdfc5hc6r104cvdazw-shared-dep (136B)12662026/11/11 22:35:01 WARN Failed to register uploaded object key=8hjlsgnqc03wcql80hgpzf3shdhw0zsa.ls error="server returned 404: 404 page not found\n"12672026/11/11 22:35:01 WARN Failed to register uploaded object key=sdcc8ri34ak7b6yzffq0gzqv9nyq1wkf.ls error="server returned 404: 404 page not found\n"12682026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/1vhn2w4rw9sjf8hzj75ds7bz0y9yc02m40xpp1wa9jh1zdmll4cj.nar.zst error="server returned 404: 404 page not found\n"12692026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12702026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign12712026/11/11 22:35:01 WARN Failed to register uploaded object key=657zbpxmz994w1cdfc5hc6r104cvdazw.ls error="server returned 404: 404 page not found\n"12722026/11/11 22:35:01 INFO Signed narinfos id=1 count=212732026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign12742026/11/11 22:35:01 INFO Signed narinfos id=2 count=212752026/11/11 22:35:01 INFO Uploading 4 narinfos12762026/11/11 22:35:01 WARN Failed to register uploaded object key=sdcc8ri34ak7b6yzffq0gzqv9nyq1wkf.narinfo error="server returned 404: 404 page not found\n"12772026/11/11 22:35:01 WARN Failed to register uploaded object key=8hjlsgnqc03wcql80hgpzf3shdhw0zsa.narinfo error="server returned 404: 404 page not found\n"12782026/11/11 22:35:01 WARN Failed to register uploaded object key=657zbpxmz994w1cdfc5hc6r104cvdazw.narinfo error="server returned 404: 404 page not found\n"12792026/11/11 22:35:01 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete12802026/11/11 22:35:01 WARN Failed to register uploaded object key=657zbpxmz994w1cdfc5hc6r104cvdazw.narinfo error="server returned 404: 404 page not found\n"12812026/11/11 22:35:01 INFO Received push request method=POST path=/api/pushes1282=== NAME TestClientMultipleUploads1283 client_integration_test.go:405: Created store path 0: /build/TestClientMultipleUploads1278297814/001/store/m8kps0xsbajnar1jk0968f193dnm8nay-test-file-0.txt12842026/11/11 22:35:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12852026/11/11 22:35:01 INFO Uploading 51vahj0rv9cwzpikyga4jx4rd0pmvswv-pinned-file.txt (128B)12862026/11/11 22:35:01 INFO Completed upload id=1 objects=6 duration=5.75624ms12872026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures12882026/11/11 22:35:01 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12892026/11/11 22:35:01 INFO Completed upload id=2 objects=6 duration=1.3365ms12902026/11/11 22:35:01 INFO Upload complete. (74ms)12912026/11/11 22:35:01 WARN Failed to register uploaded object key=51vahj0rv9cwzpikyga4jx4rd0pmvswv.ls error="server returned 404: 404 page not found\n"12922026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12932026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"1294=== NAME TestClientFallsBackToClosures12952026/11/11 22:35:01 INFO Signed narinfos id=1 count=11296 client_pushes_test.go:112: Retrieved narinfo from S3:1297 StorePath: /build/TestClientFallsBackToClosures3632646175/001/store/657zbpxmz994w1cdfc5hc6r104cvdazw-shared-dep1298 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1299 Compression: zstd13002026/11/11 22:35:01 INFO Uploading 1 narinfos1301 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821302 NarSize: 1361303 References: 1304 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1305 client_pushes_test.go:112: Retrieved narinfo from S3:13062026/11/11 22:35:01 INFO Received push request method=POST path=/api/pushes1307 StorePath: /build/TestClientFallsBackToClosures3632646175/001/store/8hjlsgnqc03wcql80hgpzf3shdhw0zsa-a1308 URL: nar/1vhn2w4rw9sjf8hzj75ds7bz0y9yc02m40xpp1wa9jh1zdmll4cj.nar.zst1309 Compression: zstd1310 NarHash: sha256:1vhn2w4rw9sjf8hzj75ds7bz0y9yc02m40xpp1wa9jh1zdmll4cj1311 NarSize: 2161312 References: /build/TestClientFallsBackToClosures3632646175/001/store/657zbpxmz994w1cdfc5hc6r104cvdazw-shared-dep1313 CA: text:sha256:03gy7qxgn9d56hhgj3pnc7fh3jhxnyydsr6xwpf3qadv914j5p8c13142026/11/11 22:35:01 INFO Received complete push request method=POST path=/api/pushes/1/complete13152026/11/11 22:35:01 WARN Failed to register uploaded object key=51vahj0rv9cwzpikyga4jx4rd0pmvswv.narinfo error="server returned 404: 404 page not found\n"1316=== NAME TestClientWithDependencies1317 client_integration_test.go:660: Built derivation: /build/TestClientWithDependencies1884267549/001/store/9d47zb5yd9azhyya88d480rdazc4873z-test-script1318=== NAME TestClientFallsBackToClosures1319 client_pushes_test.go:112: Retrieved narinfo from S3:1320 StorePath: /build/TestClientFallsBackToClosures3632646175/001/store/sdcc8ri34ak7b6yzffq0gzqv9nyq1wkf-b1321 URL: nar/1vhn2w4rw9sjf8hzj75ds7bz0y9yc02m40xpp1wa9jh1zdmll4cj.nar.zst1322 Compression: zstd1323 NarHash: sha256:1vhn2w4rw9sjf8hzj75ds7bz0y9yc02m40xpp1wa9jh1zdmll4cj1324 NarSize: 2161325 References: /build/TestClientFallsBackToClosures3632646175/001/store/657zbpxmz994w1cdfc5hc6r104cvdazw-shared-dep1326 CA: text:sha256:03gy7qxgn9d56hhgj3pnc7fh3jhxnyydsr6xwpf3qadv914j5p8c13272026/11/11 22:35:01 INFO Completed push id=1 objects=3 duration=4.993641ms13282026/11/11 22:35:01 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)13292026/11/11 22:35:01 INFO Uploading zz4m913n1qqnqgjz83pw84jh4v7mvmww-b (216B)13302026/11/11 22:35:01 INFO Upload complete. (57ms)13312026/11/11 22:35:01 INFO Uploading b0qwgbvv5ydxa5ybdz01x7v2y06aljnw-shared-dep (136B)1332--- PASS: TestClientFallsBackToClosures (0.79s)1333=== CONT TestPush_SignsNarinfosOfItsPendingObjects13342026/11/11 22:35:01 WARN Failed to register uploaded object key=vjsh3b9mnpp110a5k6l42sipz2mzcbwd.ls error="server returned 404: 404 page not found\n"13352026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/0hqw79yk3vv6xx994wc594fbza2j8dabf17zal82qyh9nhsf90nf.nar.zst error="server returned 404: 404 page not found\n"13362026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13372026/11/11 22:35:01 WARN Failed to register uploaded object key=zz4m913n1qqnqgjz83pw84jh4v7mvmww.ls error="server returned 404: 404 page not found\n"13382026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13392026/11/11 22:35:01 WARN Failed to register uploaded object key=b0qwgbvv5ydxa5ybdz01x7v2y06aljnw.ls error="server returned 404: 404 page not found\n"13402026/11/11 22:35:01 INFO Signed narinfos id=1 count=313412026/11/11 22:35:01 INFO Uploading 3 narinfos1342=== NAME TestClientIntegration1343 client_integration_test.go:291: Created store path: /build/TestClientIntegration4144659877/002/store/jpmk50pkabqzg6axjsxijhj57irkmsi8-test-file.txt1344--- PASS: TestService_Rustfstest (0.59s)1345=== CONT TestService_RequireScope_OIDC13462026/11/11 22:35:01 WARN Failed to register uploaded object key=vjsh3b9mnpp110a5k6l42sipz2mzcbwd.narinfo error="server returned 404: 404 page not found\n"13472026/11/11 22:35:01 WARN Failed to register uploaded object key=b0qwgbvv5ydxa5ybdz01x7v2y06aljnw.narinfo error="server returned 404: 404 page not found\n"13482026/11/11 22:35:01 INFO Received complete push request method=POST path=/api/pushes/1/complete13492026/11/11 22:35:01 WARN Failed to register uploaded object key=zz4m913n1qqnqgjz83pw84jh4v7mvmww.narinfo error="server returned 404: 404 page not found\n"1350=== NAME TestClientMultipleUploads1351 client_integration_test.go:405: Created store path 1: /build/TestClientMultipleUploads1278297814/001/store/va3m2lkfbwll5lahzszi6fmbmk60yqnq-test-file-1.txt13522026/11/11 22:35:01 INFO Completed push id=1 objects=8 duration=9.31084ms13532026/11/11 22:35:01 INFO Upload complete. (69ms)1354=== NAME TestClientPushesUseOnePush1355 client_pushes_test.go:97: Retrieved narinfo from S3:1356 StorePath: /build/TestClientPushesUseOnePush3552073337/001/store/b0qwgbvv5ydxa5ybdz01x7v2y06aljnw-shared-dep1357 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1358 Compression: zstd1359 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821360 NarSize: 1361361 References: 1362 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1363 client_pushes_test.go:97: Retrieved narinfo from S3:1364 StorePath: /build/TestClientPushesUseOnePush3552073337/001/store/vjsh3b9mnpp110a5k6l42sipz2mzcbwd-a1365 URL: nar/0hqw79yk3vv6xx994wc594fbza2j8dabf17zal82qyh9nhsf90nf.nar.zst1366 Compression: zstd1367 NarHash: sha256:0hqw79yk3vv6xx994wc594fbza2j8dabf17zal82qyh9nhsf90nf1368 NarSize: 2161369 References: /build/TestClientPushesUseOnePush3552073337/001/store/b0qwgbvv5ydxa5ybdz01x7v2y06aljnw-shared-dep1370 CA: text:sha256:038wkdlsgpizq1kn68m2w6gqcn65r0fwmwzfhsc7j9irbl08iy3h1371 client_pushes_test.go:97: Retrieved narinfo from S3:1372 StorePath: /build/TestClientPushesUseOnePush3552073337/001/store/zz4m913n1qqnqgjz83pw84jh4v7mvmww-b1373 URL: nar/0hqw79yk3vv6xx994wc594fbza2j8dabf17zal82qyh9nhsf90nf.nar.zst1374 Compression: zstd1375 NarHash: sha256:0hqw79yk3vv6xx994wc594fbza2j8dabf17zal82qyh9nhsf90nf1376 NarSize: 2161377 References: /build/TestClientPushesUseOnePush3552073337/001/store/b0qwgbvv5ydxa5ybdz01x7v2y06aljnw-shared-dep1378 CA: text:sha256:038wkdlsgpizq1kn68m2w6gqcn65r0fwmwzfhsc7j9irbl08iy3h1379=== NAME TestClientWithDependencies1380 client_integration_test.go:662: Found 1 dependencies (including self)1381--- PASS: TestClientPushesUseOnePush (0.80s)1382=== CONT TestPush_RejectsBadRequests13832026/11/11 22:35:01 INFO Received push request method=POST path=/api/pushes13842026/11/11 22:35:01 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)13852026/11/11 22:35:01 INFO Uploading ykyw5nsgkxrsj4dy1v8mpw7nxzm2mvfh-shared-dep (136B)13862026/11/11 22:35:01 INFO Uploading rrbyp29s7lplzxsrsjk6yhli2nzk5imz-top (224B)13872026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/18bdrjmqbhvcav34jirily3x33bm53bmn8hddssy8jwmdx6xdzv7.nar.zst error="server returned 404: 404 page not found\n"13882026/11/11 22:35:01 WARN Failed to register uploaded object key=ykyw5nsgkxrsj4dy1v8mpw7nxzm2mvfh.ls error="server returned 404: 404 page not found\n"13892026/11/11 22:35:01 WARN Failed to register uploaded object key=rrbyp29s7lplzxsrsjk6yhli2nzk5imz.ls error="server returned 404: 404 page not found\n"13902026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13912026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13922026/11/11 22:35:01 INFO Signed narinfos id=1 count=213932026/11/11 22:35:01 INFO Uploading 2 narinfos1394=== NAME TestClientMultipleUploads1395 client_integration_test.go:405: Created store path 2: /build/TestClientMultipleUploads1278297814/001/store/q9jxlr7szj810mwsv22s8yrg038llgry-test-file-2.txt13962026/11/11 22:35:01 INFO Received complete push request method=POST path=/api/pushes/1/complete13972026/11/11 22:35:01 WARN Failed to register uploaded object key=ykyw5nsgkxrsj4dy1v8mpw7nxzm2mvfh.narinfo error="server returned 404: 404 page not found\n"13982026/11/11 22:35:01 WARN Failed to register uploaded object key=rrbyp29s7lplzxsrsjk6yhli2nzk5imz.narinfo error="server returned 404: 404 page not found\n"13992026/11/11 22:35:01 INFO Completed push id=1 objects=6 duration=5.73266ms14002026/11/11 22:35:01 INFO Upload complete. (64ms)1401=== NAME TestClientSharedPathCommittedMidPush1402 client_integration_test.go:727: Retrieved narinfo from S3:1403 StorePath: /build/TestClientSharedPathCommittedMidPush2375530141/001/store/ykyw5nsgkxrsj4dy1v8mpw7nxzm2mvfh-shared-dep1404 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1405 Compression: zstd1406 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821407 NarSize: 1361408 References: 1409 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n14102026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures1411 client_integration_test.go:727: Retrieved narinfo from S3:1412 StorePath: /build/TestClientSharedPathCommittedMidPush2375530141/001/store/rrbyp29s7lplzxsrsjk6yhli2nzk5imz-top1413 URL: nar/18bdrjmqbhvcav34jirily3x33bm53bmn8hddssy8jwmdx6xdzv7.nar.zst1414 Compression: zstd1415 NarHash: sha256:18bdrjmqbhvcav34jirily3x33bm53bmn8hddssy8jwmdx6xdzv71416 NarSize: 2241417 References: /build/TestClientSharedPathCommittedMidPush2375530141/001/store/ykyw5nsgkxrsj4dy1v8mpw7nxzm2mvfh-shared-dep1418 CA: text:sha256:08i7qmcxkjj0r0m7p8y0yz5sy9s15wqc9jh5p1xyqgjygyvvf7m014192026/11/11 22:35:01 INFO Received push request method=POST path=/api/pushes14202026/11/11 22:35:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14212026/11/11 22:35:01 INFO Uploading basdyl4zp0cwnkgys6yrkvbkcgfx9b34-unpinned-file.txt (128B)1422--- PASS: TestClientSharedPathCommittedMidPush (0.74s)1423=== CONT TestService_AuthMiddleware_OIDC14242026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"14252026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign14262026/11/11 22:35:01 WARN Failed to register uploaded object key=basdyl4zp0cwnkgys6yrkvbkcgfx9b34.ls error="server returned 404: 404 page not found\n"14272026/11/11 22:35:01 INFO Signed narinfos id=2 count=114282026/11/11 22:35:01 INFO Uploading 1 narinfos14292026/11/11 22:35:01 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14302026/11/11 22:35:01 INFO Received complete push request method=POST path=/api/pushes/2/complete14312026/11/11 22:35:01 WARN Failed to register uploaded object key=basdyl4zp0cwnkgys6yrkvbkcgfx9b34.narinfo error="server returned 404: 404 page not found\n"14322026/11/11 22:35:01 INFO Received push request method=POST path=/api/pushes14332026/11/11 22:35:01 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst14342026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures14352026/11/11 22:35:01 INFO Completed push id=2 objects=3 duration=2.36495ms14362026/11/11 22:35:01 INFO Upload complete. (51ms)1437--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.62s)1438=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit14392026/11/11 22:35:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14402026/11/11 22:35:01 INFO Uploading jpmk50pkabqzg6axjsxijhj57irkmsi8-test-file.txt (152B)14412026/11/11 22:35:01 WARN Failed to register uploaded object key=jpmk50pkabqzg6axjsxijhj57irkmsi8.ls error="server returned 404: 404 page not found\n"14422026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14432026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"14442026/11/11 22:35:01 INFO Signed narinfos id=1 count=114452026/11/11 22:35:01 INFO Uploading 1 narinfos1446--- PASS: TestCacheStatsHandler (0.48s)1447=== CONT TestPush_RootsAreLockedInByteOrder14482026/11/11 22:35:01 INFO Received complete push request method=POST path=/api/pushes/1/complete14492026/11/11 22:35:01 WARN Failed to register uploaded object key=jpmk50pkabqzg6axjsxijhj57irkmsi8.narinfo error="server returned 404: 404 page not found\n"14502026/11/11 22:35:01 INFO Received push request method=POST path=/api/pushes14512026/11/11 22:35:01 INFO Completed push id=1 objects=3 duration=9.640691ms14522026/11/11 22:35:01 INFO Upload complete. (65ms)14532026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures14542026/11/11 22:35:01 INFO Starting cleanup of old closures method=DELETE path=/api/closures14552026/11/11 22:35:01 INFO Garbage collection started14562026/11/11 22:35:01 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:44019/oidc14572026/11/11 22:35:01 INFO Received push request method=POST path=/api/pushes14582026/11/11 22:35:01 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14592026/11/11 22:35:01 INFO Uploading 9d47zb5yd9azhyya88d480rdazc4873z-test-script (136B)14602026/11/11 22:35:01 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)14612026/11/11 22:35:01 INFO Uploading q9jxlr7szj810mwsv22s8yrg038llgry-test-file-2.txt (160B)14622026/11/11 22:35:01 INFO Uploading va3m2lkfbwll5lahzszi6fmbmk60yqnq-test-file-1.txt (160B)14632026/11/11 22:35:01 INFO Uploading m8kps0xsbajnar1jk0968f193dnm8nay-test-file-0.txt (160B)14642026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"14652026/11/11 22:35:01 WARN Failed to register uploaded object key=log/5y5n3bh8xla68y56aazwpf5kyv1y083g-test-script.drv error="server returned 404: 404 page not found\n"14662026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14672026/11/11 22:35:01 WARN Failed to register uploaded object key=9d47zb5yd9azhyya88d480rdazc4873z.ls error="server returned 404: 404 page not found\n"14682026/11/11 22:35:01 INFO Aborted multipart uploads count=0 kept=014692026/11/11 22:35:01 INFO Signed narinfos id=1 count=114702026/11/11 22:35:01 INFO Uploading 1 narinfos1471=== NAME TestClientCADerivations1472 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1736911347/001/store/7zkwa557fzn436ysjz4vywg0isn1brr9-ca-test14732026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"14742026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"14752026/11/11 22:35:01 WARN Failed to register uploaded object key=q9jxlr7szj810mwsv22s8yrg038llgry.ls error="server returned 404: 404 page not found\n"14762026/11/11 22:35:01 WARN Force mode enabled - objects will be deleted immediately without grace period14772026/11/11 22:35:01 INFO Received complete push request method=POST path=/api/pushes/1/complete14782026/11/11 22:35:01 WARN Failed to register uploaded object key=9d47zb5yd9azhyya88d480rdazc4873z.narinfo error="server returned 404: 404 page not found\n"14792026/11/11 22:35:01 WARN Failed to register uploaded object key=va3m2lkfbwll5lahzszi6fmbmk60yqnq.ls error="server returned 404: 404 page not found\n"14802026/11/11 22:35:01 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"14812026/11/11 22:35:01 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14822026/11/11 22:35:01 WARN Failed to register uploaded object key=m8kps0xsbajnar1jk0968f193dnm8nay.ls error="server returned 404: 404 page not found\n"14832026/11/11 22:35:01 INFO Signed narinfos id=1 count=314842026/11/11 22:35:01 INFO Uploading 3 narinfos14852026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures14862026/11/11 22:35:01 INFO All 1 paths already cached14872026/11/11 22:35:01 WARN Failed to register uploaded object key=va3m2lkfbwll5lahzszi6fmbmk60yqnq.narinfo error="server returned 404: 404 page not found\n"14882026/11/11 22:35:01 WARN Failed to register uploaded object key=m8kps0xsbajnar1jk0968f193dnm8nay.narinfo error="server returned 404: 404 page not found\n"14892026/11/11 22:35:01 INFO Received complete push request method=POST path=/api/pushes/1/complete1490=== NAME TestClientIntegration14912026/11/11 22:35:01 WARN Failed to register uploaded object key=q9jxlr7szj810mwsv22s8yrg038llgry.narinfo error="server returned 404: 404 page not found\n"1492 client_integration_test.go:317: Retrieved narinfo from S3:1493 StorePath: /build/TestClientIntegration4144659877/002/store/jpmk50pkabqzg6axjsxijhj57irkmsi8-test-file.txt1494 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1495 Compression: zstd1496 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11497 NarSize: 1521498 References: 1499 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk115002026/11/11 22:35:01 INFO Completed push id=1 objects=4 duration=10.376111ms15012026/11/11 22:35:01 INFO Upload complete. (84ms)1502 client_integration_test.go:318: Retrieved .ls file from S3 (compressed size: 77 bytes)1503 client_integration_test.go:318: Decompressed .ls content (64 bytes):1504 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1505=== NAME TestClientWithDependencies1506 client_integration_test.go:664: Skipping nix copy test - isolated store (/build/TestClientWithDependencies1884267549/001/store) requires matching store prefix15072026/11/11 22:35:01 INFO Completed push id=1 objects=9 duration=9.70524ms15082026/11/11 22:35:01 INFO Upload complete. (73ms)1509=== NAME TestClientMultipleUploads1510 client_integration_test.go:416: Uploaded 3 paths in 104.522437ms1511--- PASS: TestClientWithDependencies (0.81s)1512=== CONT TestService_ReadAuthMiddleware1513=== NAME TestClientCADerivations1514 client_ca_test.go:139: Found 1 dependencies (including self)1515--- PASS: TestClientMultipleUploads (0.79s)1516=== CONT TestPush_CompleteWaitsForCleanupOfThePush1517--- PASS: TestService_ReadScope_PublicByDefault (0.32s)1518=== CONT TestService_AuthMiddleware_MTLSBoundSubjects15192026/11/11 22:35:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=015202026/11/11 22:35:01 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15212026/11/11 22:35:01 INFO Vacuumed table table=pending_closures15222026/11/11 22:35:01 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MzdhYjcwMGEtZjMwYi00NzhlLWIzNzgtZTkzNjRiNmUxZTQzLjM5OTk4OTY5LTVjNDAtNDQxNS04MDlhLTdmYzg4NzAyNTZhOXgxNzk0NDM2NTAxOTQzNjM1Njcx15232026/11/11 22:35:01 INFO Vacuumed table table=pending_objects15242026/11/11 22:35:01 INFO Vacuumed table table=multipart_uploads15252026/11/11 22:35:01 INFO Received uploads request method=POST path=/api/pending_closures15262026/11/11 22:35:02 INFO Vacuumed table table=closures15272026/11/11 22:35:02 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MzdhYjcwMGEtZjMwYi00NzhlLWIzNzgtZTkzNjRiNmUxZTQzLjM5OTk4OTY5LTVjNDAtNDQxNS04MDlhLTdmYzg4NzAyNTZhOXgxNzk0NDM2NTAxOTQzNjM1Njcx parts=11528--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.46s)1529=== CONT TestPush_RepeatedRootCommitsOnce15302026/11/11 22:35:02 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33355/oidc15312026/11/11 22:35:02 INFO Vacuumed table table=objects15322026/11/11 22:35:02 INFO Received uploads request method=POST path=/api/pending_closures15332026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes15342026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes15352026/11/11 22:35:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15362026/11/11 22:35:02 INFO Signed narinfos id=1 count=115372026/11/11 22:35:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15382026/11/11 22:35:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15392026/11/11 22:35:02 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"15402026/11/11 22:35:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15412026/11/11 22:35:02 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"15422026/11/11 22:35:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15432026/11/11 22:35:02 INFO Object in database but missing from S3 key=jpmk50pkabqzg6axjsxijhj57irkmsi8.ls15442026/11/11 22:35:02 WARN Found objects in DB but missing from S3, will re-upload count=115452026/11/11 22:35:02 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=115462026/11/11 22:35:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15472026/11/11 22:35:02 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15482026/11/11 22:35:02 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1549--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.25s)1550=== CONT TestService_AuthMiddleware_MTLSProxyHeader15512026/11/11 22:35:02 INFO Received complete push request method=POST path=/api/pushes/2/complete15522026/11/11 22:35:02 WARN Failed to register uploaded object key=jpmk50pkabqzg6axjsxijhj57irkmsi8.ls error="server returned 404: 404 page not found\n"1553=== RUN TestPush_RejectsBadRequests/no_roots1554=== PAUSE TestPush_RejectsBadRequests/no_roots1555=== RUN TestPush_RejectsBadRequests/no_objects1556=== PAUSE TestPush_RejectsBadRequests/no_objects1557=== RUN TestPush_RejectsBadRequests/bad_root1558=== PAUSE TestPush_RejectsBadRequests/bad_root1559=== RUN TestPush_RejectsBadRequests/root_not_in_objects1560=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1561=== CONT TestPush_CompleteCommitsEveryRoot15622026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes15632026/11/11 22:35:02 INFO Completed push id=2 objects=3 duration=11.584401ms15642026/11/11 22:35:02 INFO Upload complete. (66ms)1565=== NAME TestClientIntegration1566 client_integration_test.go:335: Retrieved .ls file from S3 (compressed size: 62 bytes)1567 client_integration_test.go:335: Decompressed .ls content (49 bytes):1568 {"version":1,"root":{"type":"regular","size":39}}15692026/11/11 22:35:02 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15702026/11/11 22:35:02 INFO Uploading 7zkwa557fzn436ysjz4vywg0isn1brr9-ca-test (144B)15712026/11/11 22:35:02 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"15722026/11/11 22:35:02 WARN Failed to register uploaded object key=7zkwa557fzn436ysjz4vywg0isn1brr9.ls error="server returned 404: 404 page not found\n"15732026/11/11 22:35:02 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15742026/11/11 22:35:02 WARN Failed to register uploaded object key=log/s1pkihzcbhylzqnfm4d3kkjsf7npmh0c-ca-test.drv error="server returned 404: 404 page not found\n"15752026/11/11 22:35:02 INFO Signed narinfos id=1 count=115762026/11/11 22:35:02 INFO Uploading 1 narinfos15772026/11/11 22:35:02 INFO Received complete push request method=POST path=/api/pushes/1/complete15782026/11/11 22:35:02 WARN Failed to register uploaded object key=7zkwa557fzn436ysjz4vywg0isn1brr9.narinfo error="server returned 404: 404 page not found\n"15792026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes15802026/11/11 22:35:02 INFO Completed push id=1 objects=4 duration=10.015091ms15812026/11/11 22:35:02 INFO Upload complete. (105ms)1582=== NAME TestClientCADerivations1583 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1736911347/001/store/7zkwa557fzn436ysjz4vywg0isn1brr9-ca-test1584 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1585 Compression: zstd1586 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1587 NarSize: 1441588 References: 1589 Deriver: /build/TestClientCADerivations1736911347/001/store/s1pkihzcbhylzqnfm4d3kkjsf7npmh0c-ca-test.drv1590 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1591 client_ca_test.go:185: Checking for realisation files in S3...1592 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1593 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache15942026/11/11 22:35:02 INFO Received complete push request method=POST path=/api/pushes/1/complete15952026/11/11 22:35:02 INFO Completed push id=1 objects=4 duration=7.065381ms15962026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes15972026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes15982026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes15992026/11/11 22:35:02 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)16002026/11/11 22:35:02 INFO Received complete push request method=POST path=/api/pushes/3/complete16012026/11/11 22:35:02 WARN Failed to register uploaded object key=jpmk50pkabqzg6axjsxijhj57irkmsi8.ls error="server returned 404: 404 page not found\n"16022026-11-11 22:35:02.180 UTC [1824] ERROR: could not obtain lock on row in relation "closures"16032026-11-11 22:35:02.180 UTC [1824] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16042026/11/11 22:35:02 INFO Aborted multipart uploads count=0 kept=016052026-11-11 22:35:02.180 UTC [1824] ERROR: could not obtain lock on row in relation "closures"16062026-11-11 22:35:02.180 UTC [1824] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16072026-11-11 22:35:02.181 UTC [1824] ERROR: could not obtain lock on row in relation "closures"16082026-11-11 22:35:02.181 UTC [1824] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16092026-11-11 22:35:02.181 UTC [1824] ERROR: could not obtain lock on row in relation "closures"16102026-11-11 22:35:02.181 UTC [1824] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16112026-11-11 22:35:02.181 UTC [1824] ERROR: could not obtain lock on row in relation "closures"16122026-11-11 22:35:02.181 UTC [1824] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT16132026/11/11 22:35:02 INFO Completed push id=3 objects=3 duration=2.743481ms16142026/11/11 22:35:02 INFO Upload complete. (48ms)1615=== RUN TestService_RequireScope_OIDC/builder_may_write1616=== PAUSE TestService_RequireScope_OIDC/builder_may_write1617=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1618=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1619=== RUN TestService_RequireScope_OIDC/ops_may_admin1620=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1621=== RUN TestService_RequireScope_OIDC/ops_may_not_write1622=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1623=== RUN TestService_RequireScope_OIDC/reader_may_not_write1624=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1625=== RUN TestService_RequireScope_OIDC/static_token_may_admin1626=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1627=== NAME TestClientIntegration1628 client_integration_test.go:352: Retrieved .ls file from S3 (compressed size: 62 bytes)1629=== RUN TestService_RequireScope_OIDC/static_token_may_write1630=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1631=== NAME TestClientIntegration1632 client_integration_test.go:352: Decompressed .ls content (49 bytes):1633 {"version":1,"root":{"type":"regular","size":39}}1634=== RUN TestService_RequireScope_OIDC/reader_may_read1635=== PAUSE TestService_RequireScope_OIDC/reader_may_read1636=== RUN TestService_RequireScope_OIDC/writer_implies_read1637=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1638=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1639=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1640=== NAME TestClientIntegration1641 client_integration_test.go:363: Testing garbage collection...1642=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1643=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1644=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1645=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1646=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1647=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1648=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1649=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1650=== CONT TestService_AuthMiddleware1651--- PASS: TestPush_RootsAreLockedInByteOrder (0.30s)1652=== CONT TestPush_OverlappingRootsStoreOneRowPerKey16532026/11/11 22:35:02 WARN Force mode enabled - objects will be deleted immediately without grace period16542026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes16552026/11/11 22:35:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=016562026/11/11 22:35:02 INFO Vacuumed table table=pending_closures16572026/11/11 22:35:02 INFO Vacuumed table table=pending_objects16582026/11/11 22:35:02 INFO Vacuumed table table=multipart_uploads16592026/11/11 22:35:02 INFO Vacuumed table table=closures16602026/11/11 22:35:02 INFO Vacuumed table table=objects16612026/11/11 22:35:02 INFO Received complete push request method=POST path=/api/pushes/2/complete16622026/11/11 22:35:02 INFO Completed push id=2 objects=4 duration=3.14334ms1663--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.34s)1664=== CONT TestReadProxy4041665--- PASS: TestService_ReadAuthMiddleware (0.27s)1666=== CONT TestReadRedirectUsesPublicS3URL16672026/11/11 22:35:02 INFO Starting cleanup of old closures method=DELETE path=/api/closures16682026/11/11 22:35:02 INFO Garbage collection started1669=== NAME TestClientCADerivations1670 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1671 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1672 error: binary cache 's3://bucket57?endpoint=http://localhost:36699®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations1736911347/001/store'1673 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 116742026/11/11 22:35:02 INFO Aborted multipart uploads count=0 kept=01675--- PASS: TestClientCADerivations (0.98s)1676=== CONT TestReadRedirectKeepsNarinfoProxied16772026/11/11 22:35:02 WARN Force mode enabled - objects will be deleted immediately without grace period16782026/11/11 22:35:02 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"16792026/11/11 22:35:02 WARN mTLS auth: bound subjects configured but subject DN unavailable16802026/11/11 22:35:02 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1681--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.28s)1682=== CONT TestReadRedirectNar16832026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes16842026/11/11 22:35:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016852026/11/11 22:35:02 INFO Vacuumed table table=pending_closures16862026/11/11 22:35:02 INFO Vacuumed table table=pending_objects16872026/11/11 22:35:02 INFO Vacuumed table table=multipart_uploads16882026/11/11 22:35:02 INFO Vacuumed table table=closures16892026/11/11 22:35:02 INFO Received complete push request method=POST path=/api/pushes/1/complete16902026/11/11 22:35:02 INFO Vacuumed table table=objects1691=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1692=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1693=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1694=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1695=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1696=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1697=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1698=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1699=== CONT TestReadProxyDisabled17002026/11/11 22:35:02 INFO Completed push id=1 objects=4 duration=8.031021ms1701--- PASS: TestPush_RepeatedRootCommitsOnce (0.30s)1702=== CONT TestIsValidCachePath1703=== RUN TestIsValidCachePath/narinfo1704=== PAUSE TestIsValidCachePath/narinfo1705=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1706=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1707=== RUN TestIsValidCachePath/nar_zst1708=== PAUSE TestIsValidCachePath/nar_zst1709=== RUN TestIsValidCachePath/nar_xz1710=== PAUSE TestIsValidCachePath/nar_xz1711=== RUN TestIsValidCachePath/nar_bz21712=== PAUSE TestIsValidCachePath/nar_bz21713=== RUN TestIsValidCachePath/nar_uncompressed1714=== PAUSE TestIsValidCachePath/nar_uncompressed1715=== RUN TestIsValidCachePath/ls1716=== PAUSE TestIsValidCachePath/ls1717=== RUN TestIsValidCachePath/log1718=== PAUSE TestIsValidCachePath/log1719=== RUN TestIsValidCachePath/realisation1720=== PAUSE TestIsValidCachePath/realisation1721=== RUN TestIsValidCachePath/nix-cache-info1722=== PAUSE TestIsValidCachePath/nix-cache-info1723=== RUN TestIsValidCachePath/index.html1724=== PAUSE TestIsValidCachePath/index.html1725=== RUN TestIsValidCachePath/traversal_parent1726=== PAUSE TestIsValidCachePath/traversal_parent1727=== RUN TestIsValidCachePath/traversal_in_middle1728=== PAUSE TestIsValidCachePath/traversal_in_middle1729=== RUN TestIsValidCachePath/invalid_char_e1730=== PAUSE TestIsValidCachePath/invalid_char_e1731=== RUN TestIsValidCachePath/invalid_char_u1732=== PAUSE TestIsValidCachePath/invalid_char_u1733=== RUN TestIsValidCachePath/random_path1734=== PAUSE TestIsValidCachePath/random_path1735=== RUN TestIsValidCachePath/empty1736=== PAUSE TestIsValidCachePath/empty1737=== RUN TestIsValidCachePath/leading_slash1738=== PAUSE TestIsValidCachePath/leading_slash1739=== RUN TestIsValidCachePath/wrong_extension1740=== PAUSE TestIsValidCachePath/wrong_extension1741=== RUN TestIsValidCachePath/short_hash1742=== PAUSE TestIsValidCachePath/short_hash1743=== CONT TestReadProxyRootRedirectsToIndexHTML1744--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.28s)1745=== CONT TestReadProxyNarStreaming17462026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes17472026/11/11 22:35:02 INFO Received complete push request method=POST path=/api/pushes/1/complete17482026/11/11 22:35:02 INFO Completed push id=1 objects=6 duration=10.480531ms1749--- PASS: TestPush_CompleteCommitsEveryRoot (0.34s)1750=== CONT TestReadProxyConditionalGet17512026/11/11 22:35:02 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1752--- PASS: TestService_AuthMiddleware (0.22s)1753=== CONT TestReadProxyOutlastsServerWriteTimeout17542026/11/11 22:35:02 INFO lead: released remote=192.0.2.1:1234 generation=11755--- PASS: TestLeadEndsOnShutdown (1.70s)1756=== CONT TestReadProxyContentNegotiation17572026/11/11 22:35:02 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17582026/11/11 22:35:02 INFO Received push request method=POST path=/api/pushes17592026/11/11 22:35:02 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=MzdhYjcwMGEtZjMwYi00NzhlLWIzNzgtZTkzNjRiNmUxZTQzLjJhMmRjZWRiLThlMDYtNDYzMC05MTYxLThkZmU3NmUwMDlhM3gxNzk0NDM2NTAxOTE2NjY1NzQ5 parts=121760--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.31s)1761=== CONT TestReadProxyHead17622026/11/11 22:35:02 INFO Received uploads request method=POST path=/api/pending_closures1763--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.08s)1764=== CONT TestReadProxyNarinfoAlreadyDecompressed17652026-11-11 22:35:02.501 UTC [2002] ERROR: Push does not exist: id=117662026-11-11 22:35:02.501 UTC [2002] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE17672026-11-11 22:35:02.501 UTC [2002] STATEMENT: SELECT commit_push($1)1768--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.54s)1769=== CONT TestReadProxyInvalidPath1770--- PASS: TestReadRedirectUsesPublicS3URL (0.30s)1771=== CONT TestReadProxyNarinfo17722026/11/11 22:35:02 INFO lead: released remote=192.0.2.1:1234 generation=117732026/11/11 22:35:02 INFO lead: acquired remote=192.0.2.1:1234 generation=21774--- PASS: TestReadRedirectKeepsNarinfoProxied (0.31s)1775=== CONT TestPresentNotReportedWhileGCDeletesClosure1776=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1777=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1778=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1779=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1780=== CONT TestPresentRefreshesOnlyStaleClosures17812026/11/11 22:35:02 INFO lead: released remote=192.0.2.1:1234 generation=11782--- PASS: TestDeleteClosuresRacingPins (1.58s)1783=== CONT TestParseSingleRange1784=== RUN TestParseSingleRange/none1785=== PAUSE TestParseSingleRange/none1786=== RUN TestParseSingleRange/unknown_unit1787=== PAUSE TestParseSingleRange/unknown_unit1788=== RUN TestParseSingleRange/multi-range_ignored1789=== PAUSE TestParseSingleRange/multi-range_ignored1790=== RUN TestParseSingleRange/malformed_no_dash1791=== PAUSE TestParseSingleRange/malformed_no_dash1792=== RUN TestParseSingleRange/malformed_both_empty1793=== PAUSE TestParseSingleRange/malformed_both_empty1794=== RUN TestParseSingleRange/malformed_end_before_start1795=== PAUSE TestParseSingleRange/malformed_end_before_start1796=== RUN TestParseSingleRange/closed1797=== PAUSE TestParseSingleRange/closed1798=== RUN TestParseSingleRange/open-ended1799=== PAUSE TestParseSingleRange/open-ended1800=== RUN TestParseSingleRange/end_clamped_to_size1801=== PAUSE TestParseSingleRange/end_clamped_to_size1802=== RUN TestParseSingleRange/suffix1803=== PAUSE TestParseSingleRange/suffix1804=== RUN TestParseSingleRange/suffix_exceeds_size1805=== PAUSE TestParseSingleRange/suffix_exceeds_size1806=== RUN TestParseSingleRange/single_byte1807=== PAUSE TestParseSingleRange/single_byte1808=== RUN TestParseSingleRange/start_past_EOF1809=== PAUSE TestParseSingleRange/start_past_EOF1810=== RUN TestParseSingleRange/start_far_past_EOF1811=== PAUSE TestParseSingleRange/start_far_past_EOF1812=== CONT TestPresentReportsOnlyClosureRoots18132026/11/11 22:35:02 INFO Received complete multipart upload request method=POST path=/api/multipart/complete18142026/11/11 22:35:02 INFO lead: acquired remote=192.0.2.1:1234 generation=21815--- PASS: TestReadRedirectNar (0.36s)1816=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info18172026/11/11 22:35:02 INFO Received uploads request method=POST path=/1818=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key18192026/11/11 22:35:02 INFO Received request for more parts method=POST path=/1820=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key18212026/11/11 22:35:02 INFO Received complete multipart upload request method=POST path=/1822=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18232026/11/11 22:35:02 INFO Received uploads request method=POST path=/1824--- PASS: TestUploadHandlersRejectInvalidKeys (0.06s)1825 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1826 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1827 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1828 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1829=== CONT TestServerTLSConfig/no_client_CA1830=== CONT TestServerTLSConfig/not_a_PEM_file18312026/11/11 22:35:02 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MzdhYjcwMGEtZjMwYi00NzhlLWIzNzgtZTkzNjRiNmUxZTQzLjMwYmNmNjAzLWVjMGItNDk3Yy1hMmMyLWRhMWJiZGNhOTE1OHgxNzk0NDM2NTAyMDEwOTYwMDI2 parts=121832=== CONT TestServerTLSConfig/missing_CA_file1833--- PASS: TestServerTLSConfig (0.06s)1834 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1835 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1836 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1837=== CONT TestPendingClosureWriteTimeout/empty1838=== CONT TestPendingClosureWriteTimeout/400_objects1839=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline18402026/11/11 22:35:02 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=21841--- PASS: TestReadProxyDisabled (0.33s)1842=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed18432026/11/11 22:35:02 INFO lead: acquired remote=192.0.2.1:1234 generation=31844--- PASS: TestReadProxy404 (0.43s)1845=== CONT TestPendingClosureWriteTimeout/670k_objects1846=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1847--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.37s)1848=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost18492026/11/11 22:35:02 INFO lead: released remote=192.0.2.1:1234 generation=11850--- PASS: TestReadProxyNarStreaming (0.39s)1851=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1852=== NAME TestOrphanedObjectsGCStressTest1853 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains1854--- PASS: TestReadProxyConditionalGet (0.35s)1855=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed18562026/11/11 22:35:02 INFO lead: acquired remote=192.0.2.1:1234 generation=21857=== NAME TestOrphanedObjectsGCStressTest1858 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion1859--- PASS: TestReadProxyContentNegotiation (0.40s)1860=== CONT TestIsValidUploadKey/narinfo1861=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1862=== CONT TestIsValidUploadKey/index.html1863=== CONT TestIsValidUploadKey/nix-cache-info1864=== CONT TestIsValidUploadKey/realisation_plus_in_output1865=== CONT TestIsValidUploadKey/realisation1866=== CONT TestIsValidUploadKey/build_log_equals1867=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1868=== CONT TestIsValidUploadKey/build_log_question_mark1869=== CONT TestIsValidUploadKey/build_log_plus_in_name1870=== CONT TestIsValidUploadKey/build_log_home-manager_file1871=== CONT TestIsValidUploadKey/build_log1872=== CONT TestIsValidUploadKey/listing1873=== CONT TestIsValidUploadKey/nar_plain1874=== CONT TestIsValidUploadKey/nar_xz1875=== CONT TestIsValidUploadKey/nar_zst1876=== CONT TestIsValidUploadKey/absolute1877=== CONT TestIsValidUploadKey/unknown_type1878=== CONT TestIsValidUploadKey/empty_key1879=== CONT TestIsValidUploadKey/traversal1880=== CONT TestIsValidUploadKey/traversal_nar1881=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1882--- PASS: TestIsValidUploadKey (0.06s)1883 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1884 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1885 --- PASS: TestIsValidUploadKey/index.html (0.00s)1886 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1887 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1888 --- PASS: TestIsValidUploadKey/realisation (0.00s)1889 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1890 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1891 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1892 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1893 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1894 --- PASS: TestIsValidUploadKey/build_log (0.00s)1895 --- PASS: TestIsValidUploadKey/listing (0.00s)1896 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1897 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1898 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1899 --- PASS: TestIsValidUploadKey/absolute (0.00s)1900 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1901 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1902 --- PASS: TestIsValidUploadKey/traversal (0.00s)1903 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1904 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1905=== CONT TestGCEndsOnShutdown/before_the_run1906--- PASS: TestReadProxyHead (0.36s)1907=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1908--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.38s)1909=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1910--- PASS: TestReadProxyInvalidPath (0.40s)1911=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19122026/11/11 22:35:02 INFO Received uploads request method=POST path=/1913--- PASS: TestPresentRefreshesOnlyStaleClosures (0.43s)1914=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19152026/11/11 22:35:02 INFO Received request for more parts method=POST path=/1916--- PASS: TestPresentReportsOnlyClosureRoots (0.45s)1917=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19182026/11/11 22:35:03 INFO Received complete multipart upload request method=POST path=/19192026/11/11 22:35:03 INFO Received uploads request method=POST path=/api/pending_closures1920=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows19212026/11/11 22:35:03 INFO Received uploads request method=POST path=/api/pending_closures19222026/11/11 22:35:03 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1923--- PASS: TestReadProxyNarinfo (0.54s)1924=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1925=== CONT TestResolveDBConnectionString/flag_wins1926=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1927=== CONT TestResolveDBConnectionString/nothing_configured1928=== CONT TestResolveDBConnectionString/missing_file_is_an_error1929=== CONT TestResolveDBConnectionString/file_when_flag_empty1930=== CONT TestProxyWriteTimeout/narinfo1931=== CONT TestProxyWriteTimeout/10_GiB_nar1932=== CONT TestProxyWriteTimeout/unknown_size1933=== CONT TestProxyWriteTimeout/1_GiB_nar1934--- PASS: TestProxyWriteTimeout (0.00s)1935 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1936 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1937 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1938 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1939--- PASS: TestResolveDBConnectionString (0.00s)1940 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1941 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1942 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1943 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1944 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1945=== CONT TestClientErrorHandling/InvalidStorePath19462026/11/11 22:35:03 INFO Received uploads request method=POST path=/api/pending_closures19472026/11/11 22:35:03 INFO Received uploads request method=POST path=/api/pending_closures19482026/11/11 22:35:03 INFO Received uploads request method=POST path=/api/pending_closures19492026/11/11 22:35:03 INFO Received uploads request method=POST path=/api/pending_closures19502026/11/11 22:35:03 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19512026/11/11 22:35:03 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MzdhYjcwMGEtZjMwYi00NzhlLWIzNzgtZTkzNjRiNmUxZTQzLjUyZTViMDY3LTQzNmMtNDFmNy1iNDg1LTExNzI3Y2MwNThmOHgxNzk0NDM2NTAyMDIwNDk4ODQ3 parts=121952--- PASS: TestSharedNarBothUploadsComplete (1.50s)1953=== CONT TestClientErrorHandling/ServerNotAvailable1954=== CONT TestClientErrorHandling/InvalidAuthToken19552026/11/11 22:35:03 INFO Aborted multipart uploads count=0 kept=019562026/11/11 22:35:03 WARN Force mode enabled - objects will be deleted immediately without grace period19572026/11/11 22:35:03 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1958=== CONT TestCacheConfigHandler/full_config,_no_issuer1959=== CONT TestCacheConfigHandler/no_signing_keys1960=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1961=== CONT TestCacheConfigHandler/no_cache_url_configured1962--- PASS: TestCacheConfigHandler (0.00s)1963 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1964 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1965 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1966 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1967=== CONT TestPush_RejectsBadRequests/no_roots19682026/11/11 22:35:03 INFO Received push request method=POST path=/api/pushes1969=== CONT TestPush_RejectsBadRequests/bad_root19702026/11/11 22:35:03 INFO Received push request method=POST path=/api/pushes1971=== CONT TestPush_RejectsBadRequests/root_not_in_objects19722026/11/11 22:35:03 INFO Received push request method=POST path=/api/pushes1973=== CONT TestPush_RejectsBadRequests/no_objects19742026/11/11 22:35:03 INFO Received push request method=POST path=/api/pushes1975=== CONT TestService_RequireScope_OIDC/writer_implies_read1976--- PASS: TestPush_RejectsBadRequests (0.23s)1977 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1978 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1979 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1980 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)19812026/11/11 22:35:03 INFO Aborted multipart uploads count=0 kept=01982=== CONT TestService_RequireScope_OIDC/builder_may_write1983=== CONT TestService_RequireScope_OIDC/reader_may_read1984=== CONT TestService_RequireScope_OIDC/static_token_may_write1985=== CONT TestService_RequireScope_OIDC/static_token_may_admin1986=== CONT TestService_RequireScope_OIDC/reader_may_not_write1987=== CONT TestService_RequireScope_OIDC/ops_may_not_write1988=== CONT TestService_RequireScope_OIDC/ops_may_admin1989=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1990=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1991=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1992=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1993=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1994=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19952026/11/11 22:35:03 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1996=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1997--- PASS: TestService_RequireScope_OIDC (0.38s)1998 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1999 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)2000 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)
loading 621 hidden lines…
26222026/11/11 22:35:16 INFO Vacuumed table table=pending_objects26232026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=026242026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads26252026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period26262026/11/11 22:35:16 INFO Vacuumed table table=closures26272026/11/11 22:35:16 INFO Received push request method=POST path=/26282026/11/11 22:35:16 INFO Received push request method=POST path=/26292026/11/11 22:35:16 INFO Completed upload id=6 objects=9 duration=1.7331ms26302026/11/11 22:35:16 INFO Vacuumed table table=objects26312026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026322026/11/11 22:35:16 INFO Vacuumed table table=pending_closures26332026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=026342026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=026352026/11/11 22:35:16 INFO Vacuumed table table=pending_objects26362026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads26372026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period26382026/11/11 22:35:16 INFO Vacuumed table table=closures26392026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026402026/11/11 22:35:16 INFO Vacuumed table table=pending_closures26412026/11/11 22:35:16 INFO Vacuumed table table=pending_closures26422026/11/11 22:35:16 INFO Vacuumed table table=objects26432026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=026442026/11/11 22:35:16 INFO Vacuumed table table=pending_objects26452026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads26462026/11/11 22:35:16 INFO Vacuumed table table=closures26472026/11/11 22:35:16 INFO Vacuumed table table=objects26482026/11/11 22:35:16 INFO Vacuumed table table=pending_objects26492026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads26502026/11/11 22:35:16 INFO Vacuumed table table=closures26512026/11/11 22:35:16 INFO Received push request method=POST path=/26522026/11/11 22:35:16 INFO Vacuumed table table=objects26532026/11/11 22:35:16 INFO Received complete push request method=POST path=/26542026/11/11 22:35:16 INFO Received push request method=POST path=/26552026/11/11 22:35:16 INFO Completed push id=6 objects=5 duration=2.91253ms26562026/11/11 22:35:16 INFO Received complete push request method=POST path=/26572026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=026582026/11/11 22:35:16 INFO Received uploads request method=POST path=/26592026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period26602026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026612026/11/11 22:35:16 INFO Received uploads request method=POST path=/26622026/11/11 22:35:16 INFO Vacuumed table table=pending_closures26632026/11/11 22:35:16 INFO Completed push id=4 objects=4 duration=3.66026ms26642026/11/11 22:35:16 INFO Vacuumed table table=pending_objects26652026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads26662026/11/11 22:35:16 INFO Vacuumed table table=closures26672026/11/11 22:35:16 INFO Vacuumed table table=objects26682026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=026692026/11/11 22:35:16 INFO Vacuumed table table=pending_closures26702026/11/11 22:35:16 INFO Received push request method=POST path=/26712026/11/11 22:35:16 INFO Received uploads request method=POST path=/26722026/11/11 22:35:16 INFO Vacuumed table table=pending_objects26732026/11/11 22:35:16 INFO Received push request method=POST path=/26742026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=026752026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads26762026/11/11 22:35:16 INFO Received push request method=POST path=/26772026/11/11 22:35:16 INFO Vacuumed table table=closures26782026/11/11 22:35:16 INFO Received uploads request method=POST path=/26792026/11/11 22:35:16 INFO Vacuumed table table=objects26802026/11/11 22:35:16 INFO Received push request method=POST path=/26812026/11/11 22:35:16 INFO Received complete upload request method=POST path=/26822026/11/11 22:35:16 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."26832026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."26842026/11/11 22:35:16 INFO Received push request method=POST path=/26852026/11/11 22:35:16 INFO Received complete upload request method=POST path=/26862026/11/11 22:35:16 INFO Completed upload id=9 objects=7 duration=2.42221ms26872026/11/11 22:35:16 INFO Received complete push request method=POST path=/26882026/11/11 22:35:16 INFO Completed upload id=3 objects=24 duration=2.477781ms26892026/11/11 22:35:16 INFO Completed push id=4 objects=7 duration=2.260441ms26902026/11/11 22:35:16 INFO Received complete push request method=POST path=/26912026/11/11 22:35:16 INFO Completed push id=6 objects=6 duration=1.97631ms26922026/11/11 22:35:16 INFO Received uploads request method=POST path=/26932026/11/11 22:35:16 INFO Received push request method=POST path=/26942026/11/11 22:35:16 INFO Received uploads request method=POST path=/26952026/11/11 22:35:16 INFO Received complete upload request method=POST path=/26962026/11/11 22:35:16 INFO Completed upload id=3 objects=5 duration=1.83171ms26972026/11/11 22:35:16 INFO Received uploads request method=POST path=/26982026/11/11 22:35:16 INFO Received push request method=POST path=/26992026/11/11 22:35:16 INFO Received complete upload request method=POST path=/27002026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027012026/11/11 22:35:16 INFO Received push request method=POST path=/27022026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period27032026/11/11 22:35:16 INFO Received push request method=POST path=/27042026/11/11 22:35:16 INFO Received uploads request method=POST path=/27052026/11/11 22:35:16 INFO Completed upload id=2 objects=7 duration=5.061941ms27062026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027072026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027082026/11/11 22:35:16 INFO Received complete push request method=POST path=/27092026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period27102026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027112026/11/11 22:35:16 INFO Completed push id=8 objects=13 duration=2.63895ms27122026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027132026/11/11 22:35:16 INFO Received uploads request method=POST path=/27142026/11/11 22:35:16 INFO Received complete upload request method=POST path=/27152026/11/11 22:35:16 INFO Vacuumed table table=pending_closures27162026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027172026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period27182026/11/11 22:35:16 INFO Vacuumed table table=pending_objects27192026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads27202026/11/11 22:35:16 INFO Vacuumed table table=pending_closures27212026/11/11 22:35:16 INFO Received complete push request method=POST path=/27222026/11/11 22:35:16 INFO Vacuumed table table=closures27232026/11/11 22:35:16 INFO Vacuumed table table=pending_objects27242026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads27252026/11/11 22:35:16 INFO Completed upload id=1 objects=15 duration=5.71218ms27262026/11/11 22:35:16 INFO Vacuumed table table=objects27272026/11/11 22:35:16 INFO Received uploads request method=POST path=/27282026/11/11 22:35:16 INFO Vacuumed table table=closures27292026/11/11 22:35:16 INFO Vacuumed table table=objects27302026/11/11 22:35:16 INFO Completed push id=2 objects=7 duration=5.15143ms27312026/11/11 22:35:16 INFO Received complete push request method=POST path=/27322026/11/11 22:35:16 INFO Completed push id=1 objects=2 duration=2.371041ms27332026/11/11 22:35:16 INFO Received uploads request method=POST path=/27342026/11/11 22:35:16 INFO Received complete upload request method=POST path=/27352026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027362026/11/11 22:35:16 INFO Received uploads request method=POST path=/27372026/11/11 22:35:16 INFO Received complete push request method=POST path=/27382026/11/11 22:35:16 INFO Received uploads request method=POST path=/27392026/11/11 22:35:16 INFO Completed upload id=2 objects=26 duration=5.78613ms27402026/11/11 22:35:16 INFO Completed push id=12 objects=3 duration=2.41382ms27412026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027422026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027432026/11/11 22:35:16 INFO Received complete push request method=POST path=/27442026/11/11 22:35:16 INFO Vacuumed table table=pending_closures27452026/11/11 22:35:16 INFO Vacuumed table table=pending_objects27462026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads27472026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027482026/11/11 22:35:16 INFO Completed push id=7 objects=27 duration=2.739531ms27492026/11/11 22:35:16 INFO Vacuumed table table=closures27502026/11/11 22:35:16 INFO Vacuumed table table=objects27512026/11/11 22:35:16 INFO Vacuumed table table=pending_closures27522026/11/11 22:35:16 INFO Vacuumed table table=pending_objects27532026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads27542026/11/11 22:35:16 INFO Vacuumed table table=closures27552026/11/11 22:35:16 INFO Vacuumed table table=objects27562026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027572026/11/11 22:35:16 INFO Received complete push request method=POST path=/27582026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027592026/11/11 22:35:16 INFO Received uploads request method=POST path=/27602026/11/11 22:35:16 INFO Completed push id=7 objects=20 duration=2.91957ms27612026/11/11 22:35:16 INFO Received uploads request method=POST path=/27622026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027632026/11/11 22:35:16 INFO Received uploads request method=POST path=/27642026/11/11 22:35:16 INFO Vacuumed table table=pending_closures27652026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27662026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27672026/11/11 22:35:16 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27682026/11/11 22:35:16 INFO Received uploads request method=POST path=/27692026/11/11 22:35:16 INFO Vacuumed table table=pending_objects27702026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads27712026/11/11 22:35:16 INFO Received uploads request method=POST path=/27722026/11/11 22:35:16 INFO Vacuumed table table=closures27732026/11/11 22:35:16 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27742026/11/11 22:35:16 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27752026/11/11 22:35:16 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27762026/11/11 22:35:16 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27772026/11/11 22:35:16 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27782026/11/11 22:35:16 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27792026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27802026/11/11 22:35:16 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27812026/11/11 22:35:16 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27822026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27832026/11/11 22:35:16 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27842026/11/11 22:35:16 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27852026/11/11 22:35:16 INFO Vacuumed table table=objects27862026/11/11 22:35:16 INFO Received push request method=POST path=/27872026/11/11 22:35:16 INFO Received complete push request method=POST path=/27882026/11/11 22:35:16 INFO Completed push id=2 objects=10 duration=3.73828ms27892026/11/11 22:35:16 INFO Received complete upload request method=POST path=/27902026/11/11 22:35:16 INFO Completed upload id=3 objects=9 duration=1.62957ms27912026/11/11 22:35:16 INFO Received uploads request method=POST path=/27922026/11/11 22:35:16 INFO Received uploads request method=POST path=/27932026/11/11 22:35:16 INFO Received uploads request method=POST path=/27942026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027952026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=027962026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period27972026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027982026/11/11 22:35:16 INFO Vacuumed table table=pending_closures27992026/11/11 22:35:16 INFO Vacuumed table table=pending_objects28002026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads28012026/11/11 22:35:16 INFO Vacuumed table table=closures28022026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028032026/11/11 22:35:16 INFO Vacuumed table table=objects28042026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=028052026/11/11 22:35:16 INFO Vacuumed table table=pending_closures28062026/11/11 22:35:16 INFO Vacuumed table table=pending_objects28072026/11/11 22:35:16 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28082026/11/11 22:35:16 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."28092026/11/11 22:35:16 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28102026/11/11 22:35:16 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."28112026/11/11 22:35:16 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."28122026/11/11 22:35:16 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."28132026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28142026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28152026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28162026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28172026/11/11 22:35:16 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."28182026/11/11 22:35:16 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."28192026/11/11 22:35:16 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."28202026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads28212026/11/11 22:35:16 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."28222026/11/11 22:35:16 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."28232026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=028242026/11/11 22:35:16 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28252026/11/11 22:35:16 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28262026/11/11 22:35:16 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."28272026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28282026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."28292026/11/11 22:35:16 INFO Received complete upload request method=POST path=/28302026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."28312026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."28322026/11/11 22:35:16 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."28332026/11/11 22:35:16 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."28342026/11/11 22:35:16 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."28352026/11/11 22:35:16 INFO Vacuumed table table=closures28362026/11/11 22:35:16 INFO Vacuumed table table=objects28372026/11/11 22:35:16 INFO Completed upload id=10 objects=4 duration=2.883901ms28382026/11/11 22:35:16 INFO Received push request method=POST path=/28392026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period28402026/11/11 22:35:16 INFO Received uploads request method=POST path=/28412026/11/11 22:35:16 INFO Received push request method=POST path=/28422026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=028432026/11/11 22:35:16 INFO Received uploads request method=POST path=/28442026/11/11 22:35:16 INFO Received uploads request method=POST path=/28452026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028462026/11/11 22:35:16 INFO Vacuumed table table=pending_closures28472026/11/11 22:35:16 INFO Vacuumed table table=pending_objects28482026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads28492026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28502026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28512026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28522026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28532026/11/11 22:35:16 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28542026/11/11 22:35:16 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28552026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28562026/11/11 22:35:16 INFO Received uploads request method=POST path=/28572026/11/11 22:35:16 INFO Vacuumed table table=closures28582026/11/11 22:35:16 INFO Vacuumed table table=objects28592026/11/11 22:35:16 INFO Received push request method=POST path=/28602026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=028612026/11/11 22:35:16 INFO Received push request method=POST path=/28622026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period28632026/11/11 22:35:16 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28642026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28652026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28662026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28672026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28682026/11/11 22:35:16 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28692026/11/11 22:35:16 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28702026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28712026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28722026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28732026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."28742026/11/11 22:35:16 INFO Received push request method=POST path=/28752026/11/11 22:35:16 INFO Received push request method=POST path=/28762026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=028772026/11/11 22:35:16 INFO Received push request method=POST path=/28782026/11/11 22:35:16 INFO Received uploads request method=POST path=/28792026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028802026/11/11 22:35:16 INFO Received complete push request method=POST path=/28812026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=028822026/11/11 22:35:16 INFO Vacuumed table table=pending_closures28832026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=028842026/11/11 22:35:16 INFO Completed push id=10 objects=2 duration=2.34294ms28852026/11/11 22:35:16 INFO Vacuumed table table=pending_closures28862026/11/11 22:35:16 INFO Vacuumed table table=pending_objects28872026/11/11 22:35:16 INFO Received push request method=POST path=/28882026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads28892026/11/11 22:35:16 INFO Received complete push request method=POST path=/28902026/11/11 22:35:16 INFO Vacuumed table table=pending_objects28912026/11/11 22:35:16 INFO Received uploads request method=POST path=/28922026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads28932026/11/11 22:35:16 INFO Vacuumed table table=closures28942026/11/11 22:35:16 INFO Vacuumed table table=closures28952026/11/11 22:35:16 INFO Completed push id=5 objects=6 duration=2.50003ms28962026/11/11 22:35:16 INFO Vacuumed table table=objects28972026/11/11 22:35:16 INFO Vacuumed table table=objects28982026/11/11 22:35:16 INFO Received push request method=POST path=/28992026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029002026/11/11 22:35:16 INFO Received push request method=POST path=/29012026/11/11 22:35:16 INFO Received complete push request method=POST path=/29022026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period29032026/11/11 22:35:16 INFO Completed push id=5 objects=3 duration=3.58086ms29042026/11/11 22:35:16 INFO Received complete push request method=POST path=/29052026/11/11 22:35:16 INFO Completed push id=8 objects=7 duration=2.180751ms29062026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29072026/11/11 22:35:16 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29082026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29092026/11/11 22:35:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29102026/11/11 22:35:16 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29112026/11/11 22:35:16 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29122026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29132026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29142026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29152026/11/11 22:35:16 INFO Completed upload id=16 objects=23 duration=1.50025ms29162026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029172026/11/11 22:35:16 INFO Completed upload id=10 objects=6 duration=2.79077ms29182026/11/11 22:35:16 INFO Received push request method=POST path=/29192026/11/11 22:35:16 INFO Received uploads request method=POST path=/29202026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29212026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029222026/11/11 22:35:16 INFO Completed upload id=12 objects=21 duration=1.55522ms29232026/11/11 22:35:16 INFO Received push request method=POST path=/29242026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period29252026/11/11 22:35:16 INFO Received complete push request method=POST path=/29262026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=029272026/11/11 22:35:16 INFO Completed push id=8 objects=2 duration=1.94223ms29282026/11/11 22:35:16 INFO Vacuumed table table=pending_closures29292026/11/11 22:35:16 INFO Received uploads request method=POST path=/29302026/11/11 22:35:16 INFO Received push request method=POST path=/29312026/11/11 22:35:16 INFO Vacuumed table table=pending_objects29322026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads29332026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029342026/11/11 22:35:16 INFO Received uploads request method=POST path=/29352026/11/11 22:35:16 INFO Vacuumed table table=closures29362026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period29372026/11/11 22:35:16 INFO Vacuumed table table=objects29382026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029392026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29402026/11/11 22:35:16 INFO Vacuumed table table=pending_closures29412026/11/11 22:35:16 INFO Completed upload id=15 objects=6 duration=1.46455ms29422026/11/11 22:35:16 INFO Received uploads request method=POST path=/29432026/11/11 22:35:16 INFO Vacuumed table table=pending_objects29442026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads29452026/11/11 22:35:16 INFO Vacuumed table table=closures29462026/11/11 22:35:16 INFO Vacuumed table table=objects29472026/11/11 22:35:16 INFO Received uploads request method=POST path=/29482026/11/11 22:35:16 INFO Received uploads request method=POST path=/29492026/11/11 22:35:16 INFO Received push request method=POST path=/29502026/11/11 22:35:16 INFO Received push request method=POST path=/29512026/11/11 22:35:16 INFO Received uploads request method=POST path=/29522026/11/11 22:35:16 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29532026/11/11 22:35:16 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29542026/11/11 22:35:16 INFO Received push request method=POST path=/29552026/11/11 22:35:16 INFO Received push request method=POST path=/29562026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029572026/11/11 22:35:16 INFO Received push request method=POST path=/29582026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29592026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029602026/11/11 22:35:16 INFO Completed upload id=4 objects=14 duration=3.170631ms29612026/11/11 22:35:16 INFO Vacuumed table table=pending_closures29622026/11/11 22:35:16 INFO Received uploads request method=POST path=/29632026/11/11 22:35:16 INFO Vacuumed table table=pending_objects29642026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads29652026/11/11 22:35:16 INFO Vacuumed table table=closures29662026/11/11 22:35:16 INFO Vacuumed table table=objects29672026/11/11 22:35:16 INFO Received push request method=POST path=/29682026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029692026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29702026/11/11 22:35:16 INFO Completed upload id=9 objects=16 duration=1.62034ms29712026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period29722026/11/11 22:35:16 INFO Received push request method=POST path=/29732026/11/11 22:35:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029742026/11/11 22:35:16 INFO Vacuumed table table=pending_closures29752026/11/11 22:35:16 INFO Vacuumed table table=pending_objects29762026/11/11 22:35:16 INFO Vacuumed table table=multipart_uploads29772026/11/11 22:35:16 INFO Vacuumed table table=closures29782026/11/11 22:35:16 INFO Received complete push request method=POST path=/29792026/11/11 22:35:16 INFO Vacuumed table table=objects29802026/11/11 22:35:16 INFO Completed push id=13 objects=23 duration=2.16413ms29812026/11/11 22:35:16 INFO Received uploads request method=POST path=/29822026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029832026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29842026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29852026/11/11 22:35:16 WARN Force mode enabled - objects will be deleted immediately without grace period29862026/11/11 22:35:16 INFO Received complete upload request method=POST path=/29872026/11/11 22:35:16 INFO Received push request method=POST path=/29882026/11/11 22:35:16 INFO Aborted multipart uploads count=0 kept=029892026/11/11 22:35:16 INFO Completed upload id=10 objects=2 duration=2.866481ms29902026/11/11 22:35:16 INFO Completed upload id=3 objects=8 duration=3.110981ms29912026/11/11 22:35:17 INFO Completed upload id=11 objects=14 duration=2.39669ms29922026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=029932026/11/11 22:35:17 INFO Vacuumed table table=pending_closures29942026/11/11 22:35:17 INFO Vacuumed table table=pending_objects29952026/11/11 22:35:17 INFO Received complete upload request method=POST path=/29962026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads29972026/11/11 22:35:17 INFO Completed upload id=17 objects=7 duration=1.32703ms29982026/11/11 22:35:17 INFO Received push request method=POST path=/29992026/11/11 22:35:17 INFO Vacuumed table table=closures30002026/11/11 22:35:17 INFO Vacuumed table table=objects30012026/11/11 22:35:17 INFO Received push request method=POST path=/30022026/11/11 22:35:17 INFO Received complete push request method=POST path=/30032026/11/11 22:35:17 INFO Received push request method=POST path=/30042026/11/11 22:35:17 INFO Completed push id=8 objects=6 duration=1.86958ms30052026/11/11 22:35:17 INFO Received push request method=POST path=/30062026/11/11 22:35:17 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."30072026/11/11 22:35:17 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30082026/11/11 22:35:17 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30092026/11/11 22:35:17 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."30102026/11/11 22:35:17 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30112026/11/11 22:35:17 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30122026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30132026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30142026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30152026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030162026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030172026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period30182026/11/11 22:35:17 INFO Received push request method=POST path=/30192026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030202026/11/11 22:35:17 INFO Received complete upload request method=POST path=/30212026/11/11 22:35:17 INFO Vacuumed table table=pending_closures30222026/11/11 22:35:17 INFO Vacuumed table table=pending_objects30232026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads30242026/11/11 22:35:17 INFO Completed upload id=10 objects=6 duration=3.25415ms30252026/11/11 22:35:17 INFO Received push request method=POST path=/30262026/11/11 22:35:17 INFO Vacuumed table table=closures30272026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30282026/11/11 22:35:17 INFO Vacuumed table table=objects30292026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030302026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=030312026/11/11 22:35:17 INFO Vacuumed table table=pending_closures30322026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030332026/11/11 22:35:17 INFO Vacuumed table table=pending_objects30342026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads30352026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030362026/11/11 22:35:17 INFO Received push request method=POST path=/30372026/11/11 22:35:17 INFO Received uploads request method=POST path=/30382026/11/11 22:35:17 INFO Vacuumed table table=closures30392026/11/11 22:35:17 INFO Received push request method=POST path=/30402026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period30412026/11/11 22:35:17 INFO Vacuumed table table=objects30422026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030432026/11/11 22:35:17 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30442026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30452026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030462026/11/11 22:35:17 INFO Vacuumed table table=pending_closures30472026/11/11 22:35:17 INFO Received push request method=POST path=/30482026/11/11 22:35:17 INFO Vacuumed table table=pending_objects30492026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads30502026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=030512026/11/11 22:35:17 INFO Received uploads request method=POST path=/30522026/11/11 22:35:17 INFO Vacuumed table table=closures30532026/11/11 22:35:17 INFO Received uploads request method=POST path=/30542026/11/11 22:35:17 INFO Vacuumed table table=pending_closures30552026/11/11 22:35:17 INFO Vacuumed table table=objects30562026/11/11 22:35:17 INFO Received uploads request method=POST path=/30572026/11/11 22:35:17 INFO Vacuumed table table=pending_objects30582026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads30592026/11/11 22:35:17 INFO Vacuumed table table=closures30602026/11/11 22:35:17 INFO Vacuumed table table=objects30612026/11/11 22:35:17 INFO Received push request method=POST path=/30622026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=030632026/11/11 22:35:17 INFO Vacuumed table table=pending_closures30642026/11/11 22:35:17 INFO Vacuumed table table=pending_objects30652026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads30662026/11/11 22:35:17 INFO Received push request method=POST path=/30672026/11/11 22:35:17 INFO Vacuumed table table=closures30682026/11/11 22:35:17 INFO Received complete push request method=POST path=/30692026/11/11 22:35:17 INFO Vacuumed table table=objects30702026/11/11 22:35:17 INFO Completed push id=10 objects=3 duration=1.98533ms30712026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030722026/11/11 22:35:17 INFO Received push request method=POST path=/30732026/11/11 22:35:17 INFO Received push request method=POST path=/30742026/11/11 22:35:17 INFO Received push request method=POST path=/30752026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030762026/11/11 22:35:17 INFO Received push request method=POST path=/30772026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030782026/11/11 22:35:17 INFO Vacuumed table table=pending_closures30792026/11/11 22:35:17 INFO Received push request method=POST path=/30802026/11/11 22:35:17 INFO Vacuumed table table=pending_objects30812026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads30822026/11/11 22:35:17 INFO Vacuumed table table=closures30832026/11/11 22:35:17 INFO Received uploads request method=POST path=/30842026/11/11 22:35:17 INFO Vacuumed table table=objects30852026/11/11 22:35:17 INFO Received push request method=POST path=/30862026/11/11 22:35:17 INFO Received uploads request method=POST path=/30872026/11/11 22:35:17 INFO Received push request method=POST path=/30882026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030892026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period30902026/11/11 22:35:17 INFO Received uploads request method=POST path=/30912026/11/11 22:35:17 INFO Received complete upload request method=POST path=/30922026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030932026/11/11 22:35:17 INFO Vacuumed table table=pending_closures30942026/11/11 22:35:17 INFO Completed upload id=11 objects=6 duration=2.255561ms30952026/11/11 22:35:17 INFO Received complete upload request method=POST path=/30962026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030972026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=030982026/11/11 22:35:17 INFO Vacuumed table table=pending_objects30992026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads31002026/11/11 22:35:17 INFO Completed upload id=14 objects=6 duration=1.6294ms31012026/11/11 22:35:17 INFO Vacuumed table table=closures31022026/11/11 22:35:17 INFO Vacuumed table table=objects31032026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031042026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031052026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=031062026/11/11 22:35:17 INFO Vacuumed table table=pending_closures31072026/11/11 22:35:17 INFO Vacuumed table table=pending_closures31082026/11/11 22:35:17 INFO Vacuumed table table=pending_objects31092026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads31102026/11/11 22:35:17 INFO Vacuumed table table=pending_objects31112026/11/11 22:35:17 INFO Vacuumed table table=closures31122026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads31132026/11/11 22:35:17 INFO Vacuumed table table=closures31142026/11/11 22:35:17 INFO Vacuumed table table=objects31152026/11/11 22:35:17 INFO Vacuumed table table=objects31162026/11/11 22:35:17 INFO Received push request method=POST path=/31172026/11/11 22:35:17 INFO Received push request method=POST path=/31182026/11/11 22:35:17 INFO Received push request method=POST path=/31192026/11/11 22:35:17 INFO Received complete push request method=POST path=/31202026/11/11 22:35:17 INFO Received uploads request method=POST path=/31212026/11/11 22:35:17 INFO Received push request method=POST path=/31222026/11/11 22:35:17 INFO Completed push id=8 objects=6 duration=2.58292ms31232026/11/11 22:35:17 INFO Received push request method=POST path=/31242026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031252026/11/11 22:35:17 INFO Vacuumed table table=pending_closures31262026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=031272026/11/11 22:35:17 INFO Vacuumed table table=pending_objects31282026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads31292026/11/11 22:35:17 INFO Received complete push request method=POST path=/31302026/11/11 22:35:17 INFO Vacuumed table table=closures31312026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031322026/11/11 22:35:17 INFO Received push request method=POST path=/31332026/11/11 22:35:17 INFO Vacuumed table table=pending_closures31342026/11/11 22:35:17 INFO Vacuumed table table=pending_objects31352026/11/11 22:35:17 INFO Vacuumed table table=objects31362026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads31372026/11/11 22:35:17 INFO Vacuumed table table=closures31382026/11/11 22:35:17 INFO Completed push id=11 objects=8 duration=3.82881ms31392026/11/11 22:35:17 INFO Vacuumed table table=objects31402026/11/11 22:35:17 INFO Received push request method=POST path=/31412026/11/11 22:35:17 INFO Received uploads request method=POST path=/31422026/11/11 22:35:17 INFO Received complete push request method=POST path=/31432026/11/11 22:35:17 INFO Completed push id=5 objects=8 duration=3.574151ms31442026/11/11 22:35:17 INFO Received uploads request method=POST path=/31452026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=031462026/11/11 22:35:17 INFO Received complete push request method=POST path=/31472026/11/11 22:35:17 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31482026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."31492026/11/11 22:35:17 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31502026/11/11 22:35:17 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31512026/11/11 22:35:17 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31522026/11/11 22:35:17 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31532026/11/11 22:35:17 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31542026/11/11 22:35:17 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."31552026/11/11 22:35:17 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."31562026/11/11 22:35:17 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31572026/11/11 22:35:17 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31582026/11/11 22:35:17 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."31592026/11/11 22:35:17 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31602026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."31612026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31622026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31632026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31642026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31652026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."31662026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31672026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period31682026/11/11 22:35:17 INFO Received uploads request method=POST path=/31692026/11/11 22:35:17 INFO Completed push id=12 objects=7 duration=2.25407ms31702026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031712026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=031722026/11/11 22:35:17 INFO Vacuumed table table=pending_closures31732026/11/11 22:35:17 INFO Vacuumed table table=pending_objects31742026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads31752026/11/11 22:35:17 INFO Vacuumed table table=closures31762026/11/11 22:35:17 INFO Received complete upload request method=POST path=/31772026/11/11 22:35:17 INFO Vacuumed table table=objects31782026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031792026/11/11 22:35:17 INFO Completed upload id=7 objects=3 duration=1.12059ms31802026/11/11 22:35:17 INFO Received push request method=POST path=/31812026/11/11 22:35:17 INFO Vacuumed table table=pending_closures31822026/11/11 22:35:17 INFO Received uploads request method=POST path=/31832026/11/11 22:35:17 INFO Vacuumed table table=pending_objects31842026/11/11 22:35:17 INFO Received push request method=POST path=/31852026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads31862026/11/11 22:35:17 INFO Vacuumed table table=closures31872026/11/11 22:35:17 INFO Vacuumed table table=objects31882026/11/11 22:35:17 INFO Received push request method=POST path=/31892026/11/11 22:35:17 INFO Received uploads request method=POST path=/31902026/11/11 22:35:17 INFO Received push request method=POST path=/31912026/11/11 22:35:17 INFO Received complete upload request method=POST path=/31922026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=031932026/11/11 22:35:17 INFO Received complete upload request method=POST path=/31942026/11/11 22:35:17 INFO Received push request method=POST path=/31952026/11/11 22:35:17 INFO Completed upload id=10 objects=23 duration=2.25718ms31962026/11/11 22:35:17 INFO Completed upload id=10 objects=17 duration=1.78087ms31972026/11/11 22:35:17 INFO Received complete push request method=POST path=/31982026/11/11 22:35:17 INFO Completed push id=14 objects=3 duration=3.38488ms31992026/11/11 22:35:17 INFO Received uploads request method=POST path=/32002026/11/11 22:35:17 INFO Received push request method=POST path=/32012026/11/11 22:35:17 INFO Received complete push request method=POST path=/32022026/11/11 22:35:17 INFO Received uploads request method=POST path=/32032026/11/11 22:35:17 INFO Received uploads request method=POST path=/32042026/11/11 22:35:17 INFO Completed push id=8 objects=6 duration=3.66092ms32052026/11/11 22:35:17 INFO Received complete push request method=POST path=/32062026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032072026/11/11 22:35:17 INFO Received uploads request method=POST path=/32082026/11/11 22:35:17 INFO Completed push id=9 objects=5 duration=2.81ms32092026/11/11 22:35:17 INFO Vacuumed table table=pending_closures32102026/11/11 22:35:17 INFO Received complete push request method=POST path=/32112026/11/11 22:35:17 INFO Vacuumed table table=pending_objects32122026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads32132026/11/11 22:35:17 INFO Vacuumed table table=closures32142026/11/11 22:35:17 INFO Completed push id=18 objects=23 duration=2.354231ms32152026/11/11 22:35:17 INFO Received complete push request method=POST path=/32162026/11/11 22:35:17 INFO Vacuumed table table=objects32172026/11/11 22:35:17 INFO Completed push id=13 objects=6 duration=3.1429ms32182026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=032192026/11/11 22:35:17 INFO Received push request method=POST path=/32202026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=032212026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period32222026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032232026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=032242026/11/11 22:35:17 INFO Vacuumed table table=pending_closures32252026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=032262026/11/11 22:35:17 INFO Vacuumed table table=pending_objects32272026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads32282026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032292026/11/11 22:35:17 INFO Vacuumed table table=closures32302026/11/11 22:35:17 INFO Received uploads request method=POST path=/32312026/11/11 22:35:17 INFO Vacuumed table table=objects32322026/11/11 22:35:17 INFO Vacuumed table table=pending_closures32332026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period32342026/11/11 22:35:17 INFO Vacuumed table table=pending_objects32352026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads32362026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032372026/11/11 22:35:17 INFO Received complete upload request method=POST path=/32382026/11/11 22:35:17 INFO Vacuumed table table=closures32392026/11/11 22:35:17 INFO Received complete push request method=POST path=/32402026/11/11 22:35:17 INFO Received push request method=POST path=/32412026/11/11 22:35:17 INFO Vacuumed table table=pending_closures32422026/11/11 22:35:17 INFO Vacuumed table table=objects32432026/11/11 22:35:17 INFO Completed upload id=15 objects=4 duration=2.68606ms32442026/11/11 22:35:17 INFO Vacuumed table table=pending_objects32452026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads32462026/11/11 22:35:17 INFO Vacuumed table table=closures32472026/11/11 22:35:17 INFO Received push request method=POST path=/32482026/11/11 22:35:17 INFO Received complete push request method=POST path=/32492026/11/11 22:35:17 INFO Vacuumed table table=objects32502026/11/11 22:35:17 INFO Received uploads request method=POST path=/32512026/11/11 22:35:17 INFO Completed push id=5 objects=9 duration=6.01019ms32522026/11/11 22:35:17 INFO Completed push id=5 objects=6 duration=2.04216ms32532026/11/11 22:35:17 INFO Received push request method=POST path=/32542026/11/11 22:35:17 INFO Received push request method=POST path=/32552026/11/11 22:35:17 INFO Received uploads request method=POST path=/32562026/11/11 22:35:17 INFO Received push request method=POST path=/32572026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=032582026/11/11 22:35:17 INFO Vacuumed table table=pending_closures32592026/11/11 22:35:17 INFO Received complete push request method=POST path=/32602026/11/11 22:35:17 INFO Received push request method=POST path=/32612026/11/11 22:35:17 INFO Vacuumed table table=pending_objects32622026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads32632026/11/11 22:35:17 INFO Vacuumed table table=closures32642026/11/11 22:35:17 INFO Received push request method=POST path=/32652026/11/11 22:35:17 INFO Vacuumed table table=objects32662026/11/11 22:35:17 INFO Received complete upload request method=POST path=/32672026/11/11 22:35:17 INFO Completed push id=13 objects=12 duration=4.27931ms32682026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=032692026/11/11 22:35:17 INFO Completed upload id=11 objects=6 duration=1.57065ms32702026/11/11 22:35:17 INFO Received push request method=POST path=/32712026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period32722026/11/11 22:35:17 INFO Received complete push request method=POST path=/32732026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032742026/11/11 22:35:17 INFO Vacuumed table table=pending_closures32752026/11/11 22:35:17 INFO Vacuumed table table=pending_objects32762026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads32772026/11/11 22:35:17 INFO Vacuumed table table=closures32782026/11/11 22:35:17 INFO Vacuumed table table=objects32792026/11/11 22:35:17 INFO Received uploads request method=POST path=/32802026/11/11 22:35:17 INFO Completed push id=9 objects=7 duration=4.71364ms32812026/11/11 22:35:17 INFO Received complete push request method=POST path=/32822026/11/11 22:35:17 INFO Completed push id=11 objects=9 duration=1.85004ms32832026/11/11 22:35:17 INFO Received push request method=POST path=/32842026/11/11 22:35:17 INFO Received push request method=POST path=/32852026/11/11 22:35:17 INFO Received uploads request method=POST path=/32862026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=032872026/11/11 22:35:17 INFO Received push request method=POST path=/32882026/11/11 22:35:17 INFO Received push request method=POST path=/32892026/11/11 22:35:17 INFO Received uploads request method=POST path=/32902026/11/11 22:35:17 INFO Received uploads request method=POST path=/32912026/11/11 22:35:17 INFO Received complete push request method=POST path=/32922026/11/11 22:35:17 INFO Completed push id=14 objects=6 duration=1.91578ms32932026/11/11 22:35:17 INFO Received uploads request method=POST path=/32942026/11/11 22:35:17 INFO Received push request method=POST path=/32952026/11/11 22:35:17 INFO Received push request method=POST path=/32962026/11/11 22:35:17 INFO Received uploads request method=POST path=/32972026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032982026/11/11 22:35:17 INFO Received uploads request method=POST path=/32992026/11/11 22:35:17 INFO Vacuumed table table=pending_closures33002026/11/11 22:35:17 INFO Vacuumed table table=pending_objects33012026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads33022026/11/11 22:35:17 INFO Vacuumed table table=closures33032026/11/11 22:35:17 INFO Received complete upload request method=POST path=/33042026/11/11 22:35:17 INFO Vacuumed table table=objects33052026/11/11 22:35:17 INFO Received complete push request method=POST path=/33062026/11/11 22:35:17 INFO Completed upload id=10 objects=7 duration=1.71992ms33072026/11/11 22:35:17 INFO Received complete upload request method=POST path=/33082026/11/11 22:35:17 INFO Completed push id=17 objects=6 duration=2.38952ms33092026/11/11 22:35:17 INFO Received uploads request method=POST path=/33102026/11/11 22:35:17 INFO Completed upload id=16 objects=3 duration=1.37584ms33112026/11/11 22:35:17 INFO Received push request method=POST path=/33122026/11/11 22:35:17 INFO Received push request method=POST path=/33132026/11/11 22:35:17 INFO Received complete push request method=POST path=/33142026/11/11 22:35:17 INFO Completed push id=12 objects=15 duration=2.220531ms33152026/11/11 22:35:17 INFO Received push request method=POST path=/33162026/11/11 22:35:17 INFO Received complete push request method=POST path=/33172026/11/11 22:35:17 INFO Completed push id=9 objects=24 duration=1.8807ms33182026/11/11 22:35:17 INFO Received push request method=POST path=/33192026/11/11 22:35:17 INFO Received push request method=POST path=/33202026/11/11 22:35:17 INFO Received uploads request method=POST path=/33212026/11/11 22:35:17 INFO Received uploads request method=POST path=/33222026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=033232026/11/11 22:35:17 INFO Received uploads request method=POST path=/33242026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033252026/11/11 22:35:17 INFO Received push request method=POST path=/33262026/11/11 22:35:17 INFO Vacuumed table table=pending_closures33272026/11/11 22:35:17 INFO Received push request method=POST path=/33282026/11/11 22:35:17 INFO Vacuumed table table=pending_objects33292026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads33302026/11/11 22:35:17 INFO Received push request method=POST path=/33312026/11/11 22:35:17 INFO Vacuumed table table=closures33322026/11/11 22:35:17 INFO Received push request method=POST path=/33332026/11/11 22:35:17 INFO Vacuumed table table=objects33342026/11/11 22:35:17 INFO Received uploads request method=POST path=/33352026/11/11 22:35:17 INFO Received uploads request method=POST path=/33362026/11/11 22:35:17 INFO Received uploads request method=POST path=/33372026/11/11 22:35:17 INFO Received complete upload request method=POST path=/33382026/11/11 22:35:17 INFO Received complete push request method=POST path=/33392026/11/11 22:35:17 INFO Completed push id=4 objects=7 duration=2.14168ms33402026/11/11 22:35:17 INFO Completed upload id=8 objects=3 duration=2.80821ms33412026/11/11 22:35:17 INFO Received uploads request method=POST path=/33422026/11/11 22:35:17 INFO Received push request method=POST path=/33432026/11/11 22:35:17 INFO Received push request method=POST path=/33442026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=033452026/11/11 22:35:17 INFO Received uploads request method=POST path=/33462026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=033472026/11/11 22:35:17 INFO Received uploads request method=POST path=/33482026/11/11 22:35:17 INFO Vacuumed table table=pending_closures33492026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=033502026/11/11 22:35:17 INFO Received uploads request method=POST path=/33512026/11/11 22:35:17 INFO Vacuumed table table=pending_objects33522026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads33532026/11/11 22:35:17 INFO Vacuumed table table=closures33542026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period33552026/11/11 22:35:17 INFO Vacuumed table table=objects33562026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033572026/11/11 22:35:17 INFO Received uploads request method=POST path=/33582026/11/11 22:35:17 INFO Vacuumed table table=pending_closures33592026/11/11 22:35:17 INFO Received uploads request method=POST path=/33602026/11/11 22:35:17 INFO Received uploads request method=POST path=/33612026/11/11 22:35:17 INFO Received push request method=POST path=/33622026/11/11 22:35:17 INFO Vacuumed table table=pending_objects33632026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads33642026/11/11 22:35:17 INFO Vacuumed table table=closures33652026/11/11 22:35:17 INFO Received push request method=POST path=/33662026/11/11 22:35:17 INFO Vacuumed table table=objects33672026/11/11 22:35:17 INFO Received uploads request method=POST path=/33682026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=033692026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period33702026/11/11 22:35:17 INFO Received push request method=POST path=/33712026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033722026/11/11 22:35:17 INFO Vacuumed table table=pending_closures33732026/11/11 22:35:17 INFO Received complete upload request method=POST path=/33742026/11/11 22:35:17 INFO Vacuumed table table=pending_objects33752026/11/11 22:35:17 INFO Received complete upload request method=POST path=/33762026/11/11 22:35:17 INFO Received complete push request method=POST path=/33772026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads33782026/11/11 22:35:17 INFO Completed upload id=10 objects=10 duration=1.1262ms33792026/11/11 22:35:17 INFO Received uploads request method=POST path=/33802026/11/11 22:35:17 INFO Completed upload id=18 objects=4 duration=2.45324ms33812026/11/11 22:35:17 INFO Received complete upload request method=POST path=/33822026/11/11 22:35:17 INFO Vacuumed table table=closures33832026/11/11 22:35:17 INFO Completed push id=19 objects=6 duration=3.48848ms33842026/11/11 22:35:17 INFO Vacuumed table table=objects33852026/11/11 22:35:17 INFO Received uploads request method=POST path=/33862026/11/11 22:35:17 INFO Completed upload id=17 objects=6 duration=2.88539ms33872026/11/11 22:35:17 INFO Received uploads request method=POST path=/33882026/11/11 22:35:17 INFO Received push request method=POST path=/33892026/11/11 22:35:17 INFO Received complete upload request method=POST path=/33902026/11/11 22:35:17 INFO Completed upload id=18 objects=4 duration=2.2138ms33912026/11/11 22:35:17 INFO Received complete push request method=POST path=/33922026/11/11 22:35:17 INFO Received push request method=POST path=/33932026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=033942026/11/11 22:35:17 INFO Completed push id=6 objects=26 duration=4.805641ms33952026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period33962026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033972026/11/11 22:35:17 INFO Vacuumed table table=pending_closures33982026/11/11 22:35:17 INFO Received push request method=POST path=/33992026/11/11 22:35:17 INFO Vacuumed table table=pending_objects34002026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads34012026/11/11 22:35:17 INFO Vacuumed table table=closures34022026/11/11 22:35:17 INFO Received uploads request method=POST path=/34032026/11/11 22:35:17 INFO Vacuumed table table=objects34042026/11/11 22:35:17 INFO Received uploads request method=POST path=/34052026/11/11 22:35:17 INFO Received complete push request method=POST path=/34062026/11/11 22:35:17 INFO Completed push id=6 objects=6 duration=2.31558ms34072026/11/11 22:35:17 INFO Received uploads request method=POST path=/34082026/11/11 22:35:17 INFO Received complete push request method=POST path=/34092026/11/11 22:35:17 INFO Completed push id=15 objects=24 duration=4.278981ms34102026/11/11 22:35:17 INFO Received uploads request method=POST path=/34112026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=034122026/11/11 22:35:17 INFO Received push request method=POST path=/34132026/11/11 22:35:17 INFO Received push request method=POST path=/34142026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period34152026/11/11 22:35:17 INFO Received push request method=POST path=/34162026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034172026/11/11 22:35:17 INFO Received push request method=POST path=/34182026/11/11 22:35:17 INFO Received complete push request method=POST path=/34192026/11/11 22:35:17 INFO Vacuumed table table=pending_closures34202026/11/11 22:35:17 INFO Completed push id=7 objects=7 duration=1.86201ms34212026/11/11 22:35:17 INFO Vacuumed table table=pending_objects34222026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads34232026/11/11 22:35:17 INFO Vacuumed table table=closures34242026/11/11 22:35:17 INFO Received complete push request method=POST path=/34252026/11/11 22:35:17 INFO Vacuumed table table=objects34262026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=034272026/11/11 22:35:17 INFO Completed push id=17 objects=6 duration=2.20102ms34282026/11/11 22:35:17 INFO Received uploads request method=POST path=/34292026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period34302026/11/11 22:35:17 INFO Received push request method=POST path=/34312026/11/11 22:35:17 INFO Received uploads request method=POST path=/34322026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=034332026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period34342026/11/11 22:35:17 INFO Received uploads request method=POST path=/34352026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034362026/11/11 22:35:17 INFO Vacuumed table table=pending_closures34372026/11/11 22:35:17 INFO Vacuumed table table=pending_objects34382026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads34392026/11/11 22:35:17 INFO Vacuumed table table=closures34402026/11/11 22:35:17 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34412026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34422026/11/11 22:35:17 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34432026/11/11 22:35:17 INFO Received push request method=POST path=/34442026/11/11 22:35:17 INFO Vacuumed table table=objects34452026/11/11 22:35:17 INFO Received push request method=POST path=/34462026/11/11 22:35:17 INFO Received uploads request method=POST path=/34472026/11/11 22:35:17 INFO Received complete upload request method=POST path=/34482026/11/11 22:35:17 INFO Completed upload id=23 objects=15 duration=3.0306ms34492026/11/11 22:35:17 INFO Received complete push request method=POST path=/34502026/11/11 22:35:17 INFO Received uploads request method=POST path=/34512026/11/11 22:35:17 INFO Completed push id=12 objects=2 duration=3.400831ms34522026/11/11 22:35:17 INFO Received uploads request method=POST path=/34532026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=034542026/11/11 22:35:17 INFO Received complete push request method=POST path=/34552026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period34562026/11/11 22:35:17 INFO Received complete push request method=POST path=/34572026/11/11 22:35:17 INFO Completed push id=19 objects=8 duration=4.73889ms34582026/11/11 22:35:17 INFO Completed push id=11 objects=11 duration=4.1538ms34592026/11/11 22:35:17 INFO Received uploads request method=POST path=/34602026/11/11 22:35:17 INFO Received push request method=POST path=/34612026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=034622026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=034632026/11/11 22:35:17 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34642026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34652026/11/11 22:35:17 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34662026/11/11 22:35:17 INFO Received push request method=POST path=/34672026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period34682026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034692026/11/11 22:35:17 INFO Vacuumed table table=pending_closures34702026/11/11 22:35:17 INFO Received uploads request method=POST path=/34712026/11/11 22:35:17 INFO Vacuumed table table=pending_objects34722026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads34732026/11/11 22:35:17 INFO Vacuumed table table=closures34742026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034752026/11/11 22:35:17 INFO Vacuumed table table=objects34762026/11/11 22:35:17 INFO Received complete push request method=POST path=/34772026/11/11 22:35:17 INFO Vacuumed table table=pending_closures34782026/11/11 22:35:17 INFO Completed push id=10 objects=15 duration=2.05602ms34792026/11/11 22:35:17 INFO Vacuumed table table=pending_objects34802026/11/11 22:35:17 INFO Received uploads request method=POST path=/34812026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads34822026/11/11 22:35:17 INFO Vacuumed table table=closures34832026/11/11 22:35:17 INFO Received push request method=POST path=/34842026/11/11 22:35:17 INFO Vacuumed table table=objects34852026/11/11 22:35:17 INFO Received complete upload request method=POST path=/34862026/11/11 22:35:17 INFO Completed upload id=18 objects=11 duration=1.59679ms34872026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=034882026/11/11 22:35:17 INFO Received uploads request method=POST path=/34892026/11/11 22:35:17 INFO Received complete upload request method=POST path=/34902026/11/11 22:35:17 INFO Completed upload id=19 objects=6 duration=1.74283ms34912026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034922026/11/11 22:35:17 INFO Vacuumed table table=pending_closures34932026/11/11 22:35:17 INFO Received push request method=POST path=/34942026/11/11 22:35:17 INFO Vacuumed table table=pending_objects34952026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads34962026/11/11 22:35:17 INFO Vacuumed table table=closures34972026/11/11 22:35:17 INFO Vacuumed table table=objects34982026/11/11 22:35:17 INFO Received complete push request method=POST path=/34992026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035002026/11/11 22:35:17 INFO Received uploads request method=POST path=/35012026/11/11 22:35:17 INFO Completed push id=1 objects=11 duration=2.16558ms35022026/11/11 22:35:17 INFO Received uploads request method=POST path=/35032026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035042026/11/11 22:35:17 INFO Vacuumed table table=pending_closures35052026/11/11 22:35:17 INFO Received uploads request method=POST path=/35062026/11/11 22:35:17 INFO Vacuumed table table=pending_objects35072026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads35082026/11/11 22:35:17 INFO Received complete push request method=POST path=/35092026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035102026/11/11 22:35:17 INFO Vacuumed table table=closures35112026/11/11 22:35:17 INFO Vacuumed table table=objects35122026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period35132026/11/11 22:35:17 INFO Completed push id=4 objects=6 duration=2.63396ms35142026/11/11 22:35:17 INFO Received push request method=POST path=/35152026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035162026/11/11 22:35:17 INFO Received uploads request method=POST path=/35172026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period35182026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035192026/11/11 22:35:17 INFO Received complete upload request method=POST path=/35202026/11/11 22:35:17 INFO Received complete upload request method=POST path=/35212026/11/11 22:35:17 INFO Vacuumed table table=pending_closures35222026/11/11 22:35:17 INFO Completed upload id=6 objects=6 duration=1.48244ms35232026/11/11 22:35:17 INFO Completed upload id=22 objects=7 duration=1.45969ms35242026/11/11 22:35:17 INFO Vacuumed table table=pending_objects35252026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads35262026/11/11 22:35:17 INFO Vacuumed table table=closures35272026/11/11 22:35:17 INFO Received push request method=POST path=/35282026/11/11 22:35:17 INFO Received push request method=POST path=/35292026/11/11 22:35:17 INFO Vacuumed table table=objects35302026/11/11 22:35:17 INFO Received complete push request method=POST path=/35312026/11/11 22:35:17 INFO Received uploads request method=POST path=/35322026/11/11 22:35:17 INFO Completed push id=15 objects=7 duration=2.6249ms35332026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035342026/11/11 22:35:17 INFO Received complete upload request method=POST path=/35352026/11/11 22:35:17 INFO Received complete push request method=POST path=/35362026/11/11 22:35:17 INFO Completed upload id=16 objects=3 duration=1.09213ms35372026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period35382026/11/11 22:35:17 INFO Completed push id=14 objects=3 duration=1.35778ms35392026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035402026/11/11 22:35:17 INFO Received complete upload request method=POST path=/35412026/11/11 22:35:17 INFO Vacuumed table table=pending_closures35422026/11/11 22:35:17 INFO Completed upload id=15 objects=6 duration=950.741µs35432026/11/11 22:35:17 INFO Vacuumed table table=pending_objects35442026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads35452026/11/11 22:35:17 INFO Vacuumed table table=closures35462026/11/11 22:35:17 INFO Vacuumed table table=objects35472026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035482026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035492026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035502026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period35512026/11/11 22:35:17 INFO Received push request method=POST path=/35522026/11/11 22:35:17 INFO Vacuumed table table=pending_closures35532026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35542026/11/11 22:35:17 INFO Received uploads request method=POST path=/35552026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035562026/11/11 22:35:17 INFO Vacuumed table table=pending_objects35572026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads35582026/11/11 22:35:17 INFO Vacuumed table table=pending_closures35592026/11/11 22:35:17 INFO Received uploads request method=POST path=/35602026/11/11 22:35:17 INFO Received uploads request method=POST path=/35612026/11/11 22:35:17 INFO Vacuumed table table=closures35622026/11/11 22:35:17 INFO Vacuumed table table=objects35632026/11/11 22:35:17 INFO Vacuumed table table=pending_objects35642026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads35652026/11/11 22:35:17 INFO Vacuumed table table=closures35662026/11/11 22:35:17 INFO Vacuumed table table=objects35672026/11/11 22:35:17 INFO Received push request method=POST path=/35682026/11/11 22:35:17 INFO Received push request method=POST path=/35692026/11/11 22:35:17 INFO Received push request method=POST path=/35702026/11/11 22:35:17 INFO Received complete upload request method=POST path=/35712026/11/11 22:35:17 INFO Completed upload id=9 objects=2 duration=1.78416ms35722026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035732026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period35742026/11/11 22:35:17 INFO Received uploads request method=POST path=/35752026/11/11 22:35:17 INFO Received complete upload request method=POST path=/35762026/11/11 22:35:17 INFO Completed upload id=15 objects=9 duration=1.97137ms35772026/11/11 22:35:17 INFO Received push request method=POST path=/35782026/11/11 22:35:17 INFO Received push request method=POST path=/35792026/11/11 22:35:17 INFO Received push request method=POST path=/35802026/11/11 22:35:17 INFO Received push request method=POST path=/35812026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035822026/11/11 22:35:17 INFO Received uploads request method=POST path=/35832026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period35842026/11/11 22:35:17 INFO Received complete push request method=POST path=/35852026/11/11 22:35:17 INFO Completed push id=8 objects=3 duration=1.86316ms35862026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035872026/11/11 22:35:17 INFO Received complete push request method=POST path=/35882026/11/11 22:35:17 INFO Vacuumed table table=pending_closures35892026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=035902026/11/11 22:35:17 INFO Vacuumed table table=pending_objects35912026/11/11 22:35:17 INFO Completed push id=24 objects=9 duration=2.477821ms35922026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads35932026/11/11 22:35:17 INFO Vacuumed table table=closures35942026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period35952026/11/11 22:35:17 INFO Vacuumed table table=objects35962026/11/11 22:35:17 INFO Received uploads request method=POST path=/35972026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035982026/11/11 22:35:17 INFO Vacuumed table table=pending_closures35992026/11/11 22:35:17 INFO Vacuumed table table=pending_objects36002026/11/11 22:35:17 INFO Received uploads request method=POST path=/36012026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads36022026/11/11 22:35:17 INFO Vacuumed table table=closures36032026/11/11 22:35:17 INFO Received push request method=POST path=/36042026/11/11 22:35:17 INFO Vacuumed table table=objects36052026/11/11 22:35:17 INFO Received uploads request method=POST path=/36062026/11/11 22:35:17 INFO Received push request method=POST path=/36072026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36082026/11/11 22:35:17 INFO Received complete upload request method=POST path=/36092026/11/11 22:35:17 INFO Received push request method=POST path=/36102026/11/11 22:35:17 INFO Completed upload id=19 objects=25 duration=3.2184ms36112026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036122026/11/11 22:35:17 INFO Received push request method=POST path=/36132026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036142026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036152026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036162026/11/11 22:35:17 INFO Vacuumed table table=pending_closures36172026/11/11 22:35:17 INFO Received uploads request method=POST path=/36182026/11/11 22:35:17 INFO Vacuumed table table=pending_closures36192026/11/11 22:35:17 INFO Received complete push request method=POST path=/36202026/11/11 22:35:17 INFO Vacuumed table table=pending_objects36212026/11/11 22:35:17 INFO Received complete push request method=POST path=/36222026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads36232026/11/11 22:35:17 INFO Vacuumed table table=pending_objects36242026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads36252026/11/11 22:35:17 INFO Vacuumed table table=closures36262026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036272026/11/11 22:35:17 INFO Vacuumed table table=closures36282026/11/11 22:35:17 INFO Completed push id=16 objects=27 duration=2.60369ms36292026/11/11 22:35:17 INFO Vacuumed table table=objects36302026/11/11 22:35:17 INFO Received uploads request method=POST path=/36312026/11/11 22:35:17 INFO Received push request method=POST path=/36322026/11/11 22:35:17 INFO Completed push id=21 objects=17 duration=2.83329ms36332026/11/11 22:35:17 INFO Received uploads request method=POST path=/36342026/11/11 22:35:17 INFO Vacuumed table table=objects36352026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period36362026/11/11 22:35:17 INFO Received complete push request method=POST path=/36372026/11/11 22:35:17 INFO Received push request method=POST path=/36382026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036392026/11/11 22:35:17 INFO Vacuumed table table=pending_closures36402026/11/11 22:35:17 INFO Completed push id=3 objects=21 duration=4.891101ms36412026/11/11 22:35:17 INFO Vacuumed table table=pending_objects36422026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads36432026/11/11 22:35:17 INFO Vacuumed table table=closures36442026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036452026/11/11 22:35:17 INFO Vacuumed table table=objects36462026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period36472026/11/11 22:35:17 INFO Received push request method=POST path=/36482026/11/11 22:35:17 INFO Received push request method=POST path=/36492026/11/11 22:35:17 INFO Received push request method=POST path=/36502026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036512026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036522026/11/11 22:35:17 INFO Received uploads request method=POST path=/36532026/11/11 22:35:17 INFO Vacuumed table table=pending_closures36542026/11/11 22:35:17 INFO Received complete push request method=POST path=/36552026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period36562026/11/11 22:35:17 INFO Vacuumed table table=pending_objects36572026/11/11 22:35:17 INFO Received complete upload request method=POST path=/36582026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads36592026/11/11 22:35:17 INFO Vacuumed table table=closures36602026/11/11 22:35:17 INFO Completed upload id=13 objects=10 duration=1.29568ms36612026/11/11 22:35:17 INFO Completed push id=22 objects=3 duration=2.91674ms36622026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036632026/11/11 22:35:17 INFO Vacuumed table table=objects36642026/11/11 22:35:17 INFO Received complete push request method=POST path=/36652026/11/11 22:35:17 INFO Vacuumed table table=pending_closures36662026/11/11 22:35:17 INFO Completed push id=19 objects=7 duration=1.976ms36672026/11/11 22:35:17 INFO Vacuumed table table=pending_objects36682026/11/11 22:35:17 INFO Received complete upload request method=POST path=/36692026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads36702026/11/11 22:35:17 INFO Vacuumed table table=closures36712026/11/11 22:35:17 INFO Completed upload id=22 objects=2 duration=1.37745ms36722026/11/11 22:35:17 INFO Vacuumed table table=objects36732026/11/11 22:35:17 INFO Received uploads request method=POST path=/36742026/11/11 22:35:17 INFO Received complete push request method=POST path=/36752026/11/11 22:35:17 INFO Completed push id=15 objects=4 duration=1.5148ms36762026/11/11 22:35:17 INFO Received uploads request method=POST path=/36772026/11/11 22:35:17 INFO Received push request method=POST path=/36782026/11/11 22:35:17 INFO Received uploads request method=POST path=/36792026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036802026/11/11 22:35:17 INFO Received push request method=POST path=/36812026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period36822026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036832026/11/11 22:35:17 INFO Received push request method=POST path=/36842026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036852026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period36862026/11/11 22:35:17 INFO Received push request method=POST path=/36872026/11/11 22:35:17 INFO Received push request method=POST path=/36882026/11/11 22:35:17 INFO Received push request method=POST path=/36892026/11/11 22:35:17 INFO Received push request method=POST path=/36902026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036912026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=036922026/11/11 22:35:17 INFO Received complete push request method=POST path=/36932026/11/11 22:35:17 INFO Vacuumed table table=pending_closures36942026/11/11 22:35:17 INFO Vacuumed table table=pending_objects36952026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads36962026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period36972026/11/11 22:35:17 INFO Vacuumed table table=closures36982026/11/11 22:35:17 INFO Received complete push request method=POST path=/36992026/11/11 22:35:17 INFO Completed push id=9 objects=4 duration=3.32548ms37002026/11/11 22:35:17 INFO Received uploads request method=POST path=/37012026/11/11 22:35:17 INFO Vacuumed table table=objects37022026/11/11 22:35:17 INFO Received uploads request method=POST path=/37032026/11/11 22:35:17 INFO Completed push id=4 objects=6 duration=2.05313ms37042026/11/11 22:35:17 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37052026/11/11 22:35:17 INFO Received push request method=POST path=/37062026/11/11 22:35:17 INFO Received push request method=POST path=/37072026/11/11 22:35:17 INFO Received uploads request method=POST path=/37082026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=037092026/11/11 22:35:17 INFO Received uploads request method=POST path=/37102026/11/11 22:35:17 INFO Received uploads request method=POST path=/37112026/11/11 22:35:17 INFO Received complete push request method=POST path=/37122026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period37132026/11/11 22:35:17 INFO Completed push id=18 objects=10 duration=1.82155ms37142026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037152026/11/11 22:35:17 INFO Received uploads request method=POST path=/37162026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37172026/11/11 22:35:17 INFO Vacuumed table table=pending_objects37182026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads37192026/11/11 22:35:17 INFO Received uploads request method=POST path=/37202026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37212026/11/11 22:35:17 INFO Vacuumed table table=closures37222026/11/11 22:35:17 INFO Vacuumed table table=objects37232026/11/11 22:35:17 INFO Received uploads request method=POST path=/37242026/11/11 22:35:17 INFO Received uploads request method=POST path=/37252026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=037262026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=037272026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period37282026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037292026/11/11 22:35:17 INFO Received uploads request method=POST path=/37302026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037312026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37322026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37332026/11/11 22:35:17 INFO Vacuumed table table=pending_objects37342026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads37352026/11/11 22:35:17 INFO Vacuumed table table=pending_objects37362026/11/11 22:35:17 INFO Vacuumed table table=closures37372026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads37382026/11/11 22:35:17 INFO Vacuumed table table=closures37392026/11/11 22:35:17 INFO Vacuumed table table=objects37402026/11/11 22:35:17 INFO Vacuumed table table=objects37412026/11/11 22:35:17 INFO Received uploads request method=POST path=/37422026/11/11 22:35:17 INFO Received push request method=POST path=/37432026/11/11 22:35:17 INFO Received uploads request method=POST path=/37442026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=037452026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37462026/11/11 22:35:17 INFO Vacuumed table table=pending_objects37472026/11/11 22:35:17 INFO Received complete push request method=POST path=/37482026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads37492026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=037502026/11/11 22:35:17 INFO Completed push id=14 objects=3 duration=1.5903ms37512026/11/11 22:35:17 INFO Vacuumed table table=closures37522026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period37532026/11/11 22:35:17 INFO Received complete upload request method=POST path=/37542026/11/11 22:35:17 INFO Vacuumed table table=objects37552026/11/11 22:35:17 INFO Received push request method=POST path=/37562026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037572026/11/11 22:35:17 INFO Completed upload id=14 objects=8 duration=2.49225ms37582026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37592026/11/11 22:35:17 INFO Vacuumed table table=pending_objects37602026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads37612026/11/11 22:35:17 INFO Vacuumed table table=closures37622026/11/11 22:35:17 INFO Vacuumed table table=objects37632026/11/11 22:35:17 INFO Received complete upload request method=POST path=/37642026/11/11 22:35:17 INFO Received push request method=POST path=/37652026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=037662026/11/11 22:35:17 INFO Completed upload id=14 objects=16 duration=3.31768ms37672026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period37682026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037692026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=037702026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37712026/11/11 22:35:17 INFO Vacuumed table table=pending_objects37722026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads37732026/11/11 22:35:17 INFO Vacuumed table table=closures37742026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period37752026/11/11 22:35:17 INFO Vacuumed table table=objects37762026/11/11 22:35:17 INFO Received uploads request method=POST path=/37772026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037782026/11/11 22:35:17 INFO Received push request method=POST path=/37792026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37802026/11/11 22:35:17 INFO Vacuumed table table=pending_objects37812026/11/11 22:35:17 INFO Received push request method=POST path=/37822026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads37832026/11/11 22:35:17 INFO Vacuumed table table=closures37842026/11/11 22:35:17 INFO Vacuumed table table=objects37852026/11/11 22:35:17 INFO Received complete push request method=POST path=/37862026/11/11 22:35:17 INFO Received complete upload request method=POST path=/37872026/11/11 22:35:17 INFO Completed push id=15 objects=5 duration=2.24411ms37882026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=037892026/11/11 22:35:17 INFO Completed upload id=12 objects=26 duration=3.08743ms37902026/11/11 22:35:17 INFO Received uploads request method=POST path=/37912026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period37922026/11/11 22:35:17 INFO Received uploads request method=POST path=/37932026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037942026/11/11 22:35:17 INFO Received push request method=POST path=/37952026/11/11 22:35:17 INFO Received complete push request method=POST path=/37962026/11/11 22:35:17 INFO Received push request method=POST path=/37972026/11/11 22:35:17 INFO Completed push id=21 objects=3 duration=1.5855ms37982026/11/11 22:35:17 INFO Vacuumed table table=pending_closures37992026/11/11 22:35:17 INFO Received push request method=POST path=/38002026/11/11 22:35:17 INFO Received push request method=POST path=/38012026/11/11 22:35:17 INFO Vacuumed table table=pending_objects38022026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads38032026/11/11 22:35:17 INFO Received uploads request method=POST path=/38042026/11/11 22:35:17 INFO Vacuumed table table=closures38052026/11/11 22:35:17 INFO Received uploads request method=POST path=/38062026/11/11 22:35:17 INFO Vacuumed table table=objects38072026/11/11 22:35:17 INFO Received uploads request method=POST path=/38082026/11/11 22:35:17 INFO Received complete upload request method=POST path=/38092026/11/11 22:35:17 INFO Completed upload id=20 objects=4 duration=1.17006ms38102026/11/11 22:35:17 INFO Received push request method=POST path=/38112026/11/11 22:35:17 INFO Received uploads request method=POST path=/38122026/11/11 22:35:17 INFO Received uploads request method=POST path=/38132026/11/11 22:35:17 INFO Received push request method=POST path=/38142026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038152026/11/11 22:35:17 INFO Received complete push request method=POST path=/38162026/11/11 22:35:17 INFO Completed push id=14 objects=4 duration=3.388561ms38172026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038182026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038192026/11/11 22:35:17 INFO Vacuumed table table=pending_closures38202026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period38212026/11/11 22:35:17 INFO Vacuumed table table=pending_objects38222026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads38232026/11/11 22:35:17 INFO Received complete push request method=POST path=/38242026/11/11 22:35:17 INFO Vacuumed table table=closures38252026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038262026/11/11 22:35:17 INFO Vacuumed table table=objects38272026/11/11 22:35:17 INFO Vacuumed table table=pending_closures38282026/11/11 22:35:17 INFO Completed push id=17 objects=11 duration=2.63622ms38292026/11/11 22:35:17 INFO Vacuumed table table=pending_objects38302026/11/11 22:35:17 INFO Received uploads request method=POST path=/38312026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads38322026/11/11 22:35:17 INFO Vacuumed table table=closures38332026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038342026/11/11 22:35:17 INFO Vacuumed table table=objects38352026/11/11 22:35:17 INFO Received uploads request method=POST path=/38362026/11/11 22:35:17 INFO Received complete push request method=POST path=/38372026/11/11 22:35:17 INFO Completed push id=26 objects=14 duration=3.08421ms38382026/11/11 22:35:17 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."38392026/11/11 22:35:17 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."38402026/11/11 22:35:17 INFO Received push request method=POST path=/38412026/11/11 22:35:17 INFO Received push request method=POST path=/38422026/11/11 22:35:17 INFO Received push request method=POST path=/38432026/11/11 22:35:17 INFO Received complete push request method=POST path=/38442026/11/11 22:35:17 INFO Received complete push request method=POST path=/38452026/11/11 22:35:17 INFO Completed push id=16 objects=5 duration=1.63199ms38462026/11/11 22:35:17 INFO Completed push id=17 objects=22 duration=2.84287ms38472026/11/11 22:35:17 INFO Received uploads request method=POST path=/38482026/11/11 22:35:17 INFO Received complete upload request method=POST path=/38492026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038502026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period38512026/11/11 22:35:17 INFO Received complete push request method=POST path=/38522026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038532026/11/11 22:35:17 INFO Completed upload id=23 objects=16 duration=2.857791ms38542026/11/11 22:35:17 INFO Received push request method=POST path=/38552026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038562026/11/11 22:35:17 INFO Completed push id=16 objects=8 duration=2.331761ms38572026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period38582026/11/11 22:35:17 INFO Received push request method=POST path=/38592026/11/11 22:35:17 INFO Vacuumed table table=pending_closures38602026/11/11 22:35:17 INFO Vacuumed table table=pending_objects38612026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads38622026/11/11 22:35:17 INFO Vacuumed table table=closures38632026/11/11 22:35:17 INFO Vacuumed table table=objects38642026/11/11 22:35:17 INFO Received uploads request method=POST path=/38652026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038662026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038672026/11/11 22:35:17 INFO Vacuumed table table=pending_closures38682026/11/11 22:35:17 INFO Vacuumed table table=pending_objects38692026/11/11 22:35:17 INFO Received push request method=POST path=/38702026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads38712026/11/11 22:35:17 INFO Received uploads request method=POST path=/38722026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038732026/11/11 22:35:17 INFO Vacuumed table table=closures38742026/11/11 22:35:17 INFO Received push request method=POST path=/38752026/11/11 22:35:17 INFO Vacuumed table table=objects38762026/11/11 22:35:17 INFO Vacuumed table table=pending_closures38772026/11/11 22:35:17 INFO Vacuumed table table=pending_objects38782026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads38792026/11/11 22:35:17 INFO Received push request method=POST path=/38802026/11/11 22:35:17 INFO Vacuumed table table=closures38812026/11/11 22:35:17 INFO Vacuumed table table=objects38822026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038832026/11/11 22:35:17 INFO Received uploads request method=POST path=/38842026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period38852026/11/11 22:35:17 INFO Received uploads request method=POST path=/38862026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038872026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038882026/11/11 22:35:17 INFO Vacuumed table table=pending_closures38892026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period38902026/11/11 22:35:17 INFO Vacuumed table table=pending_objects38912026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads38922026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=038932026/11/11 22:35:17 INFO Vacuumed table table=closures38942026/11/11 22:35:17 WARN Force mode enabled - objects will be deleted immediately without grace period38952026/11/11 22:35:17 INFO Received complete push request method=POST path=/38962026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038972026/11/11 22:35:17 INFO Vacuumed table table=objects38982026/11/11 22:35:17 INFO Vacuumed table table=pending_closures38992026/11/11 22:35:17 INFO Vacuumed table table=pending_objects39002026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads39012026/11/11 22:35:17 INFO Completed push id=16 objects=4 duration=3.29953ms39022026/11/11 22:35:17 INFO Vacuumed table table=closures39032026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=039042026/11/11 22:35:17 INFO Vacuumed table table=objects39052026/11/11 22:35:17 INFO Received push request method=POST path=/39062026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=039072026/11/11 22:35:17 INFO Received complete upload request method=POST path=/39082026/11/11 22:35:17 INFO Vacuumed table table=pending_closures39092026/11/11 22:35:17 INFO Received push request method=POST path=/39102026/11/11 22:35:17 INFO Vacuumed table table=pending_objects39112026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads39122026/11/11 22:35:17 INFO Received uploads request method=POST path=/39132026/11/11 22:35:17 INFO Vacuumed table table=closures39142026/11/11 22:35:17 INFO Completed upload id=17 objects=25 duration=3.56068ms39152026/11/11 22:35:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039162026/11/11 22:35:17 INFO Vacuumed table table=objects39172026/11/11 22:35:17 INFO Vacuumed table table=pending_closures39182026/11/11 22:35:17 INFO Aborted multipart uploads count=0 kept=039192026/11/11 22:35:17 INFO Vacuumed table table=pending_objects39202026/11/11 22:35:17 INFO Vacuumed table table=multipart_uploads39212026/11/11 22:35:17 INFO Vacuumed table table=closures39222026/11/11 22:35:18 INFO Vacuumed table table=objects39232026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039242026/11/11 22:35:18 INFO Received uploads request method=POST path=/39252026/11/11 22:35:18 INFO Vacuumed table table=pending_closures39262026/11/11 22:35:18 INFO Received uploads request method=POST path=/39272026/11/11 22:35:18 INFO Received uploads request method=POST path=/39282026/11/11 22:35:18 INFO Vacuumed table table=pending_objects39292026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads39302026/11/11 22:35:18 INFO Vacuumed table table=closures39312026/11/11 22:35:18 INFO Received uploads request method=POST path=/39322026/11/11 22:35:18 INFO Vacuumed table table=objects39332026/11/11 22:35:18 INFO Received uploads request method=POST path=/39342026/11/11 22:35:18 INFO Received push request method=POST path=/39352026/11/11 22:35:18 INFO Received complete upload request method=POST path=/39362026/11/11 22:35:18 INFO Received push request method=POST path=/39372026/11/11 22:35:18 INFO Completed upload id=26 objects=6 duration=1.38679ms39382026/11/11 22:35:18 INFO Received complete push request method=POST path=/39392026/11/11 22:35:18 INFO Received complete upload request method=POST path=/39402026/11/11 22:35:18 INFO Received push request method=POST path=/39412026/11/11 22:35:18 INFO Completed push id=11 objects=6 duration=3.62869ms39422026/11/11 22:35:18 INFO Completed upload id=21 objects=7 duration=3.01607ms39432026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=039442026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039452026/11/11 22:35:18 INFO Vacuumed table table=pending_closures39462026/11/11 22:35:18 INFO Vacuumed table table=pending_objects39472026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads39482026/11/11 22:35:18 INFO Vacuumed table table=closures39492026/11/11 22:35:18 INFO Vacuumed table table=objects39502026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=039512026/11/11 22:35:18 INFO Received complete upload request method=POST path=/39522026/11/11 22:35:18 INFO Completed upload id=20 objects=5 duration=1.29865ms39532026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period39542026/11/11 22:35:18 INFO Received uploads request method=POST path=/39552026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039562026/11/11 22:35:18 INFO Received complete upload request method=POST path=/39572026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=039582026/11/11 22:35:18 INFO Received complete upload request method=POST path=/39592026/11/11 22:35:18 INFO Vacuumed table table=pending_closures39602026/11/11 22:35:18 INFO Completed upload id=7 objects=10 duration=2.121251ms39612026/11/11 22:35:18 INFO Completed upload id=21 objects=2 duration=1.79809ms39622026/11/11 22:35:18 INFO Vacuumed table table=pending_objects39632026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039642026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads39652026/11/11 22:35:18 INFO Received complete upload request method=POST path=/39662026/11/11 22:35:18 INFO Vacuumed table table=closures39672026/11/11 22:35:18 INFO Received uploads request method=POST path=/39682026/11/11 22:35:18 INFO Vacuumed table table=pending_closures39692026/11/11 22:35:18 INFO Vacuumed table table=objects39702026/11/11 22:35:18 INFO Received complete push request method=POST path=/39712026/11/11 22:35:18 INFO Received complete push request method=POST path=/39722026/11/11 22:35:18 INFO Completed upload id=16 objects=17 duration=2.84449ms39732026/11/11 22:35:18 INFO Vacuumed table table=pending_objects39742026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads39752026/11/11 22:35:18 INFO Vacuumed table table=closures39762026/11/11 22:35:18 INFO Vacuumed table table=objects39772026/11/11 22:35:18 INFO Received uploads request method=POST path=/39782026/11/11 22:35:18 INFO Completed push id=30 objects=4 duration=4.09325ms39792026/11/11 22:35:18 INFO Completed push id=18 objects=6 duration=4.24183ms39802026/11/11 22:35:18 INFO Received complete push request method=POST path=/39812026/11/11 22:35:18 INFO Received uploads request method=POST path=/39822026/11/11 22:35:18 INFO Received push request method=POST path=/39832026/11/11 22:35:18 INFO Completed push id=18 objects=6 duration=1.99966ms39842026/11/11 22:35:18 INFO Received complete upload request method=POST path=/39852026/11/11 22:35:18 INFO Completed upload id=29 objects=5 duration=1.34073ms39862026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=039872026/11/11 22:35:18 INFO Received push request method=POST path=/39882026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period39892026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039902026/11/11 22:35:18 INFO Received push request method=POST path=/39912026/11/11 22:35:18 INFO Vacuumed table table=pending_closures39922026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=039932026/11/11 22:35:18 INFO Vacuumed table table=pending_objects39942026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads39952026/11/11 22:35:18 INFO Vacuumed table table=closures39962026/11/11 22:35:18 INFO Vacuumed table table=objects39972026/11/11 22:35:18 INFO Received uploads request method=POST path=/39982026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period39992026/11/11 22:35:18 INFO Received uploads request method=POST path=/40002026/11/11 22:35:18 INFO Received uploads request method=POST path=/40012026/11/11 22:35:18 INFO Received complete upload request method=POST path=/40022026/11/11 22:35:18 INFO Completed upload id=23 objects=9 duration=1.556921ms40032026/11/11 22:35:18 INFO Received push request method=POST path=/40042026/11/11 22:35:18 INFO Received push request method=POST path=/40052026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=040062026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=040072026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period40082026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040092026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040102026/11/11 22:35:18 INFO Vacuumed table table=pending_closures40112026/11/11 22:35:18 INFO Vacuumed table table=pending_objects40122026/11/11 22:35:18 INFO Vacuumed table table=pending_closures40132026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads40142026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=040152026/11/11 22:35:18 INFO Vacuumed table table=closures40162026/11/11 22:35:18 INFO Vacuumed table table=pending_objects40172026/11/11 22:35:18 INFO Vacuumed table table=objects40182026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads40192026/11/11 22:35:18 INFO Received push request method=POST path=/40202026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period40212026/11/11 22:35:18 INFO Vacuumed table table=closures40222026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040232026/11/11 22:35:18 INFO Vacuumed table table=objects40242026/11/11 22:35:18 INFO Vacuumed table table=pending_closures40252026/11/11 22:35:18 INFO Received push request method=POST path=/40262026/11/11 22:35:18 INFO Vacuumed table table=pending_objects40272026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads40282026/11/11 22:35:18 INFO Vacuumed table table=closures40292026/11/11 22:35:18 INFO Vacuumed table table=objects40302026/11/11 22:35:18 INFO Received push request method=POST path=/40312026/11/11 22:35:18 INFO Received complete push request method=POST path=/40322026/11/11 22:35:18 INFO Received push request method=POST path=/40332026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=040342026/11/11 22:35:18 INFO Completed push id=16 objects=16 duration=3.89557ms40352026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period40362026/11/11 22:35:18 INFO Received uploads request method=POST path=/40372026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040382026/11/11 22:35:18 INFO Vacuumed table table=pending_closures40392026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=040402026/11/11 22:35:18 INFO Vacuumed table table=pending_objects40412026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads40422026/11/11 22:35:18 INFO Received complete upload request method=POST path=/40432026/11/11 22:35:18 INFO Vacuumed table table=closures40442026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040452026/11/11 22:35:18 INFO Vacuumed table table=objects40462026/11/11 22:35:18 INFO Vacuumed table table=pending_closures40472026/11/11 22:35:18 INFO Completed upload id=20 objects=3 duration=3.063341ms40482026/11/11 22:35:18 INFO Vacuumed table table=pending_objects40492026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads40502026/11/11 22:35:18 INFO Vacuumed table table=closures40512026/11/11 22:35:18 INFO Received push request method=POST path=/40522026/11/11 22:35:18 INFO Vacuumed table table=objects40532026/11/11 22:35:18 INFO Received push request method=POST path=/40542026/11/11 22:35:18 INFO Received uploads request method=POST path=/40552026/11/11 22:35:18 INFO Received push request method=POST path=/40562026/11/11 22:35:18 INFO Received complete push request method=POST path=/40572026/11/11 22:35:18 INFO Received uploads request method=POST path=/40582026/11/11 22:35:18 INFO Completed push id=7 objects=15 duration=3.01195ms40592026/11/11 22:35:18 INFO Received uploads request method=POST path=/40602026/11/11 22:35:18 INFO Received push request method=POST path=/40612026/11/11 22:35:18 INFO Received uploads request method=POST path=/40622026/11/11 22:35:18 INFO Received push request method=POST path=/40632026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=040642026/11/11 22:35:18 INFO Received complete push request method=POST path=/40652026/11/11 22:35:18 INFO Received complete push request method=POST path=/40662026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040672026/11/11 22:35:18 INFO Completed push id=32 objects=7 duration=1.90017ms40682026/11/11 22:35:18 INFO Vacuumed table table=pending_closures40692026/11/11 22:35:18 INFO Received uploads request method=POST path=/40702026/11/11 22:35:18 INFO Vacuumed table table=pending_objects40712026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads40722026/11/11 22:35:18 INFO Completed push id=7 objects=24 duration=3.6209ms40732026/11/11 22:35:18 INFO Vacuumed table table=closures40742026/11/11 22:35:18 INFO Vacuumed table table=objects40752026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=040762026/11/11 22:35:18 INFO Received complete upload request method=POST path=/40772026/11/11 22:35:18 INFO Completed upload id=11 objects=7 duration=1.52563ms40782026/11/11 22:35:18 INFO Received complete upload request method=POST path=/40792026/11/11 22:35:18 INFO Completed upload id=27 objects=4 duration=1.20467ms40802026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=040812026/11/11 22:35:18 INFO Vacuumed table table=pending_closures40822026/11/11 22:35:18 INFO Vacuumed table table=pending_objects40832026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads40842026/11/11 22:35:18 INFO Vacuumed table table=closures40852026/11/11 22:35:18 INFO Vacuumed table table=objects40862026/11/11 22:35:18 INFO Received complete push request method=POST path=/40872026/11/11 22:35:18 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40882026/11/11 22:35:18 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40892026/11/11 22:35:18 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40902026/11/11 22:35:18 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40912026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40922026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40932026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40942026/11/11 22:35:18 INFO Received uploads request method=POST path=/40952026/11/11 22:35:18 INFO Completed push id=26 objects=4 duration=1.35608ms40962026/11/11 22:35:18 INFO Received uploads request method=POST path=/40972026/11/11 22:35:18 INFO Received uploads request method=POST path=/40982026/11/11 22:35:18 INFO Received complete push request method=POST path=/40992026/11/11 22:35:18 INFO Received complete push request method=POST path=/41002026/11/11 22:35:18 INFO Completed push id=13 objects=11 duration=1.82222ms41012026/11/11 22:35:18 INFO Received push request method=POST path=/41022026/11/11 22:35:18 INFO Completed push id=14 objects=3 duration=3.87835ms41032026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=041042026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041052026/11/11 22:35:18 INFO Vacuumed table table=pending_closures41062026/11/11 22:35:18 INFO Received uploads request method=POST path=/41072026/11/11 22:35:18 INFO Received push request method=POST path=/41082026/11/11 22:35:18 INFO Vacuumed table table=pending_objects41092026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads41102026/11/11 22:35:18 INFO Received uploads request method=POST path=/41112026/11/11 22:35:18 INFO Vacuumed table table=closures41122026/11/11 22:35:18 INFO Vacuumed table table=objects41132026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41142026/11/11 22:35:18 INFO Completed upload id=28 objects=13 duration=1.55315ms41152026/11/11 22:35:18 INFO Received push request method=POST path=/41162026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41172026/11/11 22:35:18 INFO Received uploads request method=POST path=/41182026/11/11 22:35:18 INFO Completed upload id=24 objects=7 duration=1.51587ms41192026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=041202026/11/11 22:35:18 INFO Received push request method=POST path=/41212026/11/11 22:35:18 INFO Received uploads request method=POST path=/41222026/11/11 22:35:18 INFO Received uploads request method=POST path=/41232026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41242026/11/11 22:35:18 INFO Received push request method=POST path=/41252026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period41262026/11/11 22:35:18 INFO Completed upload id=23 objects=18 duration=2.59917ms41272026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041282026/11/11 22:35:18 INFO Vacuumed table table=pending_closures41292026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=041302026/11/11 22:35:18 INFO Vacuumed table table=pending_objects41312026/11/11 22:35:18 INFO Received complete push request method=POST path=/41322026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads41332026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period41342026/11/11 22:35:18 INFO Vacuumed table table=closures41352026/11/11 22:35:18 INFO Vacuumed table table=objects41362026/11/11 22:35:18 INFO Completed push id=21 objects=9 duration=2.158101ms41372026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041382026/11/11 22:35:18 INFO Received push request method=POST path=/41392026/11/11 22:35:18 INFO Vacuumed table table=pending_closures41402026/11/11 22:35:18 INFO Vacuumed table table=pending_objects41412026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads41422026/11/11 22:35:18 INFO Vacuumed table table=closures41432026/11/11 22:35:18 INFO Vacuumed table table=objects41442026/11/11 22:35:18 INFO Received uploads request method=POST path=/41452026/11/11 22:35:18 INFO Received complete push request method=POST path=/41462026/11/11 22:35:18 INFO Completed push id=29 objects=3 duration=1.31772ms41472026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=041482026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=041492026/11/11 22:35:18 INFO Received uploads request method=POST path=/41502026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041512026/11/11 22:35:18 INFO Received uploads request method=POST path=/41522026/11/11 22:35:18 INFO Received complete push request method=POST path=/41532026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period41542026/11/11 22:35:18 INFO Vacuumed table table=pending_closures41552026/11/11 22:35:18 INFO Vacuumed table table=pending_objects41562026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads41572026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041582026/11/11 22:35:18 INFO Vacuumed table table=closures41592026/11/11 22:35:18 INFO Completed push id=27 objects=25 duration=3.49095ms41602026/11/11 22:35:18 INFO Vacuumed table table=pending_closures41612026/11/11 22:35:18 INFO Vacuumed table table=objects41622026/11/11 22:35:18 INFO Vacuumed table table=pending_objects41632026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=041642026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads41652026/11/11 22:35:18 INFO Vacuumed table table=closures41662026/11/11 22:35:18 INFO Vacuumed table table=objects41672026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041682026/11/11 22:35:18 INFO Received uploads request method=POST path=/41692026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41702026/11/11 22:35:18 INFO Vacuumed table table=pending_closures41712026/11/11 22:35:18 INFO Vacuumed table table=pending_objects41722026/11/11 22:35:18 INFO Completed upload id=15 objects=3 duration=1.91337ms41732026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads41742026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41752026/11/11 22:35:18 INFO Vacuumed table table=closures41762026/11/11 22:35:18 INFO Vacuumed table table=objects41772026/11/11 22:35:18 INFO Received push request method=POST path=/41782026/11/11 22:35:18 INFO Completed upload id=31 objects=3 duration=3.12563ms41792026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41802026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41812026/11/11 22:35:18 INFO Completed upload id=31 objects=2 duration=1.42231ms41822026/11/11 22:35:18 INFO Completed upload id=33 objects=5 duration=2.93489ms41832026/11/11 22:35:18 INFO Received complete push request method=POST path=/41842026/11/11 22:35:18 INFO Received push request method=POST path=/41852026/11/11 22:35:18 INFO Completed push id=16 objects=26 duration=1.66542ms41862026/11/11 22:35:18 INFO Received push request method=POST path=/41872026/11/11 22:35:18 INFO Received push request method=POST path=/41882026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=041892026/11/11 22:35:18 INFO Received uploads request method=POST path=/41902026/11/11 22:35:18 INFO Received uploads request method=POST path=/41912026/11/11 22:35:18 INFO Received push request method=POST path=/41922026/11/11 22:35:18 INFO Received complete upload request method=POST path=/41932026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041942026/11/11 22:35:18 INFO Completed upload id=22 objects=2 duration=1.22657ms41952026/11/11 22:35:18 INFO Vacuumed table table=pending_closures41962026/11/11 22:35:18 INFO Received complete push request method=POST path=/41972026/11/11 22:35:18 INFO Vacuumed table table=pending_objects41982026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads41992026/11/11 22:35:18 INFO Completed push id=33 objects=6 duration=1.52851ms42002026/11/11 22:35:18 INFO Vacuumed table table=closures42012026/11/11 22:35:18 INFO Vacuumed table table=objects42022026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042032026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period42042026/11/11 22:35:18 INFO Received uploads request method=POST path=/42052026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042062026/11/11 22:35:18 INFO Received push request method=POST path=/42072026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42082026/11/11 22:35:18 INFO Received uploads request method=POST path=/42092026/11/11 22:35:18 INFO Vacuumed table table=pending_objects42102026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads42112026/11/11 22:35:18 INFO Vacuumed table table=closures42122026/11/11 22:35:18 INFO Vacuumed table table=objects42132026/11/11 22:35:18 INFO Received complete upload request method=POST path=/42142026/11/11 22:35:18 INFO Completed upload id=22 objects=22 duration=1.3142ms42152026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042162026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042172026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period42182026/11/11 22:35:18 INFO Received uploads request method=POST path=/42192026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042202026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period42212026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42222026/11/11 22:35:18 INFO Received complete push request method=POST path=/42232026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042242026/11/11 22:35:18 INFO Vacuumed table table=pending_objects42252026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads42262026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042272026/11/11 22:35:18 INFO Vacuumed table table=closures42282026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42292026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period42302026/11/11 22:35:18 INFO Vacuumed table table=objects42312026/11/11 22:35:18 INFO Completed push id=24 objects=5 duration=3.24678ms42322026/11/11 22:35:18 INFO Vacuumed table table=pending_objects42332026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads42342026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042352026/11/11 22:35:18 INFO Vacuumed table table=closures42362026/11/11 22:35:18 INFO Received complete push request method=POST path=/42372026/11/11 22:35:18 INFO Vacuumed table table=objects42382026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042392026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42402026/11/11 22:35:18 INFO Completed push id=13 objects=2 duration=1.56277ms42412026/11/11 22:35:18 INFO Vacuumed table table=pending_objects42422026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads42432026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042442026/11/11 22:35:18 INFO Vacuumed table table=closures42452026/11/11 22:35:18 INFO Vacuumed table table=objects42462026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42472026/11/11 22:35:18 INFO Received push request method=POST path=/42482026/11/11 22:35:18 INFO Vacuumed table table=pending_objects42492026/11/11 22:35:18 INFO Received complete push request method=POST path=/42502026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads42512026/11/11 22:35:18 INFO Vacuumed table table=closures42522026/11/11 22:35:18 INFO Received complete push request method=POST path=/42532026/11/11 22:35:18 INFO Received complete upload request method=POST path=/42542026/11/11 22:35:18 INFO Received push request method=POST path=/42552026/11/11 22:35:18 INFO Vacuumed table table=objects42562026/11/11 22:35:18 INFO Completed push id=27 objects=6 duration=3.34159ms42572026/11/11 22:35:18 INFO Completed upload id=21 objects=25 duration=2.165261ms42582026/11/11 22:35:18 INFO Completed push id=16 objects=14 duration=2.815551ms42592026/11/11 22:35:18 INFO Received complete upload request method=POST path=/42602026/11/11 22:35:18 INFO Received complete upload request method=POST path=/42612026/11/11 22:35:18 INFO Completed upload id=18 objects=2 duration=2.42533ms42622026/11/11 22:35:18 INFO Received complete push request method=POST path=/42632026/11/11 22:35:18 INFO Completed upload id=27 objects=13 duration=1.2326ms42642026/11/11 22:35:18 INFO Received complete push request method=POST path=/42652026/11/11 22:35:18 INFO Received push request method=POST path=/42662026/11/11 22:35:18 INFO Received complete push request method=POST path=/42672026/11/11 22:35:18 INFO Completed push id=19 objects=14 duration=2.196941ms42682026/11/11 22:35:18 INFO Completed push id=13 objects=15 duration=1.98612ms42692026/11/11 22:35:18 INFO Completed push id=23 objects=28 duration=3.948581ms42702026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042712026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period42722026/11/11 22:35:18 INFO Received complete push request method=POST path=/42732026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042742026/11/11 22:35:18 INFO Completed push id=26 objects=5 duration=1.79563ms42752026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42762026/11/11 22:35:18 INFO Vacuumed table table=pending_objects42772026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads42782026/11/11 22:35:18 INFO Vacuumed table table=closures42792026/11/11 22:35:18 INFO Received uploads request method=POST path=/42802026/11/11 22:35:18 INFO Vacuumed table table=objects42812026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042822026/11/11 22:35:18 INFO Received uploads request method=POST path=/42832026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period42842026/11/11 22:35:18 INFO Received push request method=POST path=/42852026/11/11 22:35:18 INFO Received complete push request method=POST path=/42862026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042872026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42882026/11/11 22:35:18 INFO Completed push id=17 objects=22 duration=2.52309ms42892026/11/11 22:35:18 INFO Vacuumed table table=pending_objects42902026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads42912026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042922026/11/11 22:35:18 INFO Vacuumed table table=closures42932026/11/11 22:35:18 INFO Vacuumed table table=objects42942026/11/11 22:35:18 INFO Received push request method=POST path=/42952026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042962026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=042972026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042982026/11/11 22:35:18 INFO Vacuumed table table=pending_closures42992026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period43002026/11/11 22:35:18 INFO Vacuumed table table=pending_objects43012026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43022026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=043032026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43042026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads43052026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43062026/11/11 22:35:18 INFO Completed upload id=24 objects=14 duration=1.1128ms43072026/11/11 22:35:18 INFO Vacuumed table table=closures43082026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period43092026/11/11 22:35:18 INFO Completed upload id=15 objects=2 duration=1.52446ms43102026/11/11 22:35:18 INFO Completed upload id=19 objects=2 duration=2.86538ms43112026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=043122026/11/11 22:35:18 INFO Vacuumed table table=objects43132026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43142026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period43152026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043162026/11/11 22:35:18 INFO Vacuumed table table=pending_closures43172026/11/11 22:35:18 INFO Vacuumed table table=pending_objects43182026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads43192026/11/11 22:35:18 INFO Vacuumed table table=closures43202026/11/11 22:35:18 INFO Received uploads request method=POST path=/43212026/11/11 22:35:18 INFO Vacuumed table table=objects43222026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=043232026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043242026/11/11 22:35:18 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43252026/11/11 22:35:18 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."43262026/11/11 22:35:18 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."43272026/11/11 22:35:18 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."43282026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43292026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."43302026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43312026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43322026/11/11 22:35:18 INFO Vacuumed table table=pending_closures43332026/11/11 22:35:18 INFO Completed upload id=27 objects=8 duration=1.28338ms43342026/11/11 22:35:18 INFO Vacuumed table table=pending_objects43352026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads43362026/11/11 22:35:18 INFO Received uploads request method=POST path=/43372026/11/11 22:35:18 INFO Vacuumed table table=closures43382026/11/11 22:35:18 INFO Vacuumed table table=objects43392026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43402026/11/11 22:35:18 INFO Received complete push request method=POST path=/43412026/11/11 22:35:18 INFO Completed upload id=13 objects=7 duration=1.33583ms43422026/11/11 22:35:18 INFO Completed push id=19 objects=3 duration=1.18353ms43432026/11/11 22:35:18 INFO Received complete push request method=POST path=/43442026/11/11 22:35:18 INFO Received uploads request method=POST path=/43452026/11/11 22:35:18 INFO Completed push id=21 objects=26 duration=2.15089ms43462026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43472026/11/11 22:35:18 INFO Completed upload id=12 objects=6 duration=1.28339ms43482026/11/11 22:35:18 INFO Received push request method=POST path=/43492026/11/11 22:35:18 INFO Received complete push request method=POST path=/43502026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43512026/11/11 22:35:18 INFO Received complete push request method=POST path=/43522026/11/11 22:35:18 INFO Received push request method=POST path=/43532026/11/11 22:35:18 INFO Completed upload id=20 objects=4 duration=1.396061ms43542026/11/11 22:35:18 INFO Completed push id=33 objects=7 duration=2.259681ms43552026/11/11 22:35:18 INFO Completed push id=20 objects=3 duration=2.298071ms43562026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=043572026/11/11 22:35:18 INFO Received push request method=POST path=/43582026/11/11 22:35:18 INFO Vacuumed table table=pending_closures43592026/11/11 22:35:18 INFO Received push request method=POST path=/43602026/11/11 22:35:18 INFO Vacuumed table table=pending_objects43612026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads43622026/11/11 22:35:18 INFO Received complete push request method=POST path=/43632026/11/11 22:35:18 INFO Vacuumed table table=closures43642026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43652026/11/11 22:35:18 INFO Completed push id=17 objects=3 duration=2.13718ms43662026/11/11 22:35:18 INFO Vacuumed table table=objects43672026/11/11 22:35:18 INFO Completed upload id=28 objects=17 duration=1.78255ms43682026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43692026/11/11 22:35:18 INFO Completed upload id=21 objects=3 duration=1.408311ms43702026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43712026/11/11 22:35:18 INFO Completed upload id=26 objects=3 duration=1.17152ms43722026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=043732026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43742026/11/11 22:35:18 INFO Completed upload id=23 objects=7 duration=1.35337ms43752026/11/11 22:35:18 INFO Received uploads request method=POST path=/43762026/11/11 22:35:18 INFO Received uploads request method=POST path=/43772026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043782026/11/11 22:35:18 INFO Vacuumed table table=pending_closures43792026/11/11 22:35:18 INFO Received push request method=POST path=/43802026/11/11 22:35:18 INFO Vacuumed table table=pending_objects43812026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads43822026/11/11 22:35:18 INFO Vacuumed table table=closures43832026/11/11 22:35:18 INFO Vacuumed table table=objects43842026/11/11 22:35:18 INFO Received push request method=POST path=/43852026/11/11 22:35:18 INFO Received push request method=POST path=/43862026/11/11 22:35:18 INFO Received complete push request method=POST path=/43872026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=043882026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period43892026/11/11 22:35:18 INFO Completed push id=29 objects=2 duration=1.462791ms43902026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=043912026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period43922026/11/11 22:35:18 INFO Received complete push request method=POST path=/43932026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043942026/11/11 22:35:18 INFO Completed push id=23 objects=5 duration=1.45461ms43952026/11/11 22:35:18 INFO Vacuumed table table=pending_closures43962026/11/11 22:35:18 INFO Received complete upload request method=POST path=/43972026/11/11 22:35:18 INFO Vacuumed table table=pending_objects43982026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads43992026/11/11 22:35:18 INFO Vacuumed table table=closures44002026/11/11 22:35:18 INFO Completed upload id=25 objects=6 duration=1.85968ms44012026/11/11 22:35:18 INFO Received push request method=POST path=/44022026/11/11 22:35:18 INFO Vacuumed table table=objects44032026/11/11 22:35:18 INFO Received push request method=POST path=/44042026/11/11 22:35:18 INFO Received uploads request method=POST path=/44052026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=044062026/11/11 22:35:18 INFO Received complete push request method=POST path=/44072026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=044082026/11/11 22:35:18 INFO Vacuumed table table=pending_closures44092026/11/11 22:35:18 INFO Completed push id=31 objects=14 duration=3.51255ms44102026/11/11 22:35:18 INFO Received uploads request method=POST path=/44112026/11/11 22:35:18 INFO Vacuumed table table=pending_objects44122026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads44132026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044142026/11/11 22:35:18 INFO Vacuumed table table=pending_closures44152026/11/11 22:35:18 INFO Vacuumed table table=closures44162026/11/11 22:35:18 INFO Received complete push request method=POST path=/44172026/11/11 22:35:18 INFO Vacuumed table table=pending_objects44182026/11/11 22:35:18 INFO Vacuumed table table=objects44192026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads44202026/11/11 22:35:18 INFO Completed push id=31 objects=6 duration=1.560931ms44212026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44222026/11/11 22:35:18 INFO Vacuumed table table=closures44232026/11/11 22:35:18 INFO Completed upload id=24 objects=3 duration=1.08426ms44242026/11/11 22:35:18 INFO Vacuumed table table=objects44252026/11/11 22:35:18 INFO Received complete push request method=POST path=/44262026/11/11 22:35:18 INFO Completed push id=19 objects=14 duration=1.93271ms44272026/11/11 22:35:18 INFO Received uploads request method=POST path=/44282026/11/11 22:35:18 INFO Received uploads request method=POST path=/44292026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44302026/11/11 22:35:18 INFO Received uploads request method=POST path=/44312026/11/11 22:35:18 INFO Completed upload id=24 objects=18 duration=1.812471ms44322026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44332026/11/11 22:35:18 INFO Completed upload id=13 objects=8 duration=1.75976ms44342026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=044352026/11/11 22:35:18 INFO Received push request method=POST path=/44362026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044372026/11/11 22:35:18 INFO Vacuumed table table=pending_closures44382026/11/11 22:35:18 INFO Received complete push request method=POST path=/44392026/11/11 22:35:18 INFO Received complete push request method=POST path=/44402026/11/11 22:35:18 INFO Vacuumed table table=pending_objects44412026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads44422026/11/11 22:35:18 INFO Completed push id=25 objects=4 duration=1.2602ms44432026/11/11 22:35:18 INFO Completed push id=32 objects=9 duration=1.92927ms44442026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44452026/11/11 22:35:18 INFO Received push request method=POST path=/44462026/11/11 22:35:18 INFO Vacuumed table table=closures44472026/11/11 22:35:18 INFO Vacuumed table table=objects44482026/11/11 22:35:18 INFO Received uploads request method=POST path=/44492026/11/11 22:35:18 INFO Completed upload id=25 objects=16 duration=2.8357ms44502026/11/11 22:35:18 INFO Received complete push request method=POST path=/44512026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44522026/11/11 22:35:18 INFO Completed push id=25 objects=2 duration=2.16395ms44532026/11/11 22:35:18 INFO Completed upload id=25 objects=4 duration=1.15958ms44542026/11/11 22:35:18 INFO Received uploads request method=POST path=/44552026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44562026/11/11 22:35:18 INFO Received uploads request method=POST path=/44572026/11/11 22:35:18 INFO Completed upload id=29 objects=4 duration=2.10269ms44582026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=044592026/11/11 22:35:18 INFO Received uploads request method=POST path=/44602026/11/11 22:35:18 INFO Received push request method=POST path=/44612026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period44622026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044632026/11/11 22:35:18 INFO Vacuumed table table=pending_closures44642026/11/11 22:35:18 INFO Vacuumed table table=pending_objects44652026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads44662026/11/11 22:35:18 INFO Vacuumed table table=closures44672026/11/11 22:35:18 INFO Vacuumed table table=objects44682026/11/11 22:35:18 INFO Received complete push request method=POST path=/44692026/11/11 22:35:18 INFO Received uploads request method=POST path=/44702026/11/11 22:35:18 INFO Completed push id=9 objects=9 duration=3.18044ms44712026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=044722026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44732026/11/11 22:35:18 INFO Received complete push request method=POST path=/44742026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period44752026/11/11 22:35:18 INFO Completed upload id=33 objects=9 duration=1.32337ms44762026/11/11 22:35:18 INFO Completed push id=27 objects=4 duration=1.32462ms44772026/11/11 22:35:18 INFO Received push request method=POST path=/44782026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044792026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44802026/11/11 22:35:18 INFO Vacuumed table table=pending_closures44812026/11/11 22:35:18 INFO Completed upload id=26 objects=8 duration=1.32918ms44822026/11/11 22:35:18 INFO Vacuumed table table=pending_objects44832026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads44842026/11/11 22:35:18 INFO Received uploads request method=POST path=/44852026/11/11 22:35:18 INFO Vacuumed table table=closures44862026/11/11 22:35:18 INFO Vacuumed table table=objects44872026/11/11 22:35:18 INFO Received push request method=POST path=/44882026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44892026/11/11 22:35:18 INFO Received uploads request method=POST path=/44902026/11/11 22:35:18 INFO Completed upload id=20 objects=6 duration=1.82894ms44912026/11/11 22:35:18 INFO Received push request method=POST path=/44922026/11/11 22:35:18 INFO Received complete push request method=POST path=/44932026/11/11 22:35:18 INFO Completed push id=11 objects=9 duration=1.582ms44942026/11/11 22:35:18 INFO Received complete upload request method=POST path=/44952026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=044962026/11/11 22:35:18 INFO Completed upload id=21 objects=8 duration=1.22208ms44972026/11/11 22:35:18 INFO Received complete push request method=POST path=/44982026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period44992026/11/11 22:35:18 INFO Completed push id=28 objects=14 duration=1.37986ms45002026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045012026/11/11 22:35:18 INFO Received uploads request method=POST path=/45022026/11/11 22:35:18 INFO Received complete push request method=POST path=/45032026/11/11 22:35:18 INFO Vacuumed table table=pending_closures45042026/11/11 22:35:18 INFO Received push request method=POST path=/45052026/11/11 22:35:18 INFO Completed push id=26 objects=23 duration=1.59369ms45062026/11/11 22:35:18 INFO Vacuumed table table=pending_objects45072026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45082026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads45092026/11/11 22:35:18 INFO Completed upload id=22 objects=6 duration=1.65468ms45102026/11/11 22:35:18 INFO Vacuumed table table=closures45112026/11/11 22:35:18 INFO Vacuumed table table=objects45122026/11/11 22:35:18 INFO Received push request method=POST path=/45132026/11/11 22:35:18 INFO Received complete push request method=POST path=/45142026/11/11 22:35:18 INFO Received complete push request method=POST path=/45152026/11/11 22:35:18 INFO Completed push id=28 objects=26 duration=1.87914ms45162026/11/11 22:35:18 INFO Completed push id=22 objects=3 duration=3.75042ms45172026/11/11 22:35:18 INFO Received complete push request method=POST path=/45182026/11/11 22:35:18 INFO Received complete push request method=POST path=/45192026/11/11 22:35:18 INFO Completed push id=22 objects=6 duration=1.64348ms45202026/11/11 22:35:18 INFO Completed push id=7 objects=7 duration=1.95557ms45212026/11/11 22:35:18 INFO Received complete push request method=POST path=/45222026/11/11 22:35:18 INFO Completed push id=15 objects=24 duration=1.56093ms45232026/11/11 22:35:18 INFO Received complete push request method=POST path=/45242026/11/11 22:35:18 INFO Completed push id=20 objects=8 duration=1.189851ms45252026/11/11 22:35:18 INFO Received push request method=POST path=/45262026/11/11 22:35:18 INFO Received uploads request method=POST path=/45272026/11/11 22:35:18 INFO Received push request method=POST path=/45282026/11/11 22:35:18 INFO Received uploads request method=POST path=/45292026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45302026/11/11 22:35:18 INFO Received complete push request method=POST path=/45312026/11/11 22:35:18 INFO Completed upload id=13 objects=14 duration=3.50152ms45322026/11/11 22:35:18 INFO Received complete push request method=POST path=/45332026/11/11 22:35:18 INFO Completed push id=11 objects=6 duration=2.13688ms45342026/11/11 22:35:18 INFO Received uploads request method=POST path=/45352026/11/11 22:35:18 INFO Completed push id=30 objects=3 duration=1.187201ms45362026/11/11 22:35:18 INFO Received complete push request method=POST path=/45372026/11/11 22:35:18 INFO Completed push id=24 objects=8 duration=1.20625ms45382026/11/11 22:35:18 INFO Received complete push request method=POST path=/45392026/11/11 22:35:18 INFO Completed push id=12 objects=6 duration=1.40777ms45402026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45412026/11/11 22:35:18 INFO Completed upload id=19 objects=2 duration=1.00572ms45422026/11/11 22:35:18 INFO Received uploads request method=POST path=/45432026/11/11 22:35:18 INFO Received uploads request method=POST path=/45442026/11/11 22:35:18 INFO Received complete push request method=POST path=/45452026/11/11 22:35:18 INFO Completed push id=27 objects=15 duration=1.578621ms45462026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=045472026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45482026/11/11 22:35:18 INFO Received complete push request method=POST path=/45492026/11/11 22:35:18 INFO Completed upload id=25 objects=9 duration=1.26381ms45502026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045512026/11/11 22:35:18 INFO Completed push id=32 objects=19 duration=1.42387ms45522026/11/11 22:35:18 INFO Vacuumed table table=pending_closures45532026/11/11 22:35:18 INFO Vacuumed table table=pending_objects45542026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45552026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads45562026/11/11 22:35:18 INFO Received complete push request method=POST path=/45572026/11/11 22:35:18 INFO Vacuumed table table=closures45582026/11/11 22:35:18 INFO Vacuumed table table=objects45592026/11/11 22:35:18 INFO Completed upload id=24 objects=7 duration=2.34554ms45602026/11/11 22:35:18 INFO Completed push id=20 objects=13 duration=2.58588ms45612026/11/11 22:35:18 INFO Received push request method=POST path=/45622026/11/11 22:35:18 INFO Received complete push request method=POST path=/45632026/11/11 22:35:18 INFO Received uploads request method=POST path=/45642026/11/11 22:35:18 INFO Completed push id=5 objects=28 duration=2.47695ms45652026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45662026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=045672026/11/11 22:35:18 INFO Completed upload id=21 objects=2 duration=1.13224ms45682026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period45692026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045702026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45712026/11/11 22:35:18 INFO Vacuumed table table=pending_closures45722026/11/11 22:35:18 INFO Received push request method=POST path=/45732026/11/11 22:35:18 INFO Completed upload id=27 objects=3 duration=1.39497ms45742026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=045752026/11/11 22:35:18 INFO Vacuumed table table=pending_objects45762026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period45772026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads45782026/11/11 22:35:18 INFO Vacuumed table table=closures45792026/11/11 22:35:18 INFO Received complete push request method=POST path=/45802026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045812026/11/11 22:35:18 INFO Vacuumed table table=objects45822026/11/11 22:35:18 INFO Completed push id=26 objects=3 duration=1.38197ms45832026/11/11 22:35:18 INFO Vacuumed table table=pending_closures45842026/11/11 22:35:18 INFO Received complete push request method=POST path=/45852026/11/11 22:35:18 INFO Vacuumed table table=pending_objects45862026/11/11 22:35:18 INFO Completed push id=22 objects=4 duration=1.640271ms45872026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads45882026/11/11 22:35:18 INFO Received complete push request method=POST path=/45892026/11/11 22:35:18 INFO Received push request method=POST path=/45902026/11/11 22:35:18 INFO Vacuumed table table=closures45912026/11/11 22:35:18 INFO Received complete push request method=POST path=/45922026/11/11 22:35:18 INFO Vacuumed table table=objects45932026/11/11 22:35:18 INFO Completed push id=10 objects=2 duration=1.87049ms45942026/11/11 22:35:18 INFO Received uploads request method=POST path=/45952026/11/11 22:35:18 INFO Completed push id=31 objects=6 duration=2.80008ms45962026/11/11 22:35:18 INFO Received uploads request method=POST path=/45972026/11/11 22:35:18 INFO Received complete push request method=POST path=/45982026/11/11 22:35:18 INFO Received complete upload request method=POST path=/45992026/11/11 22:35:18 INFO Completed upload id=28 objects=6 duration=1.66239ms46002026/11/11 22:35:18 INFO Received complete push request method=POST path=/46012026/11/11 22:35:18 INFO Completed push id=13 objects=22 duration=2.35195ms46022026/11/11 22:35:18 INFO Completed push id=24 objects=3 duration=1.60945ms46032026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46042026/11/11 22:35:18 INFO Received uploads request method=POST path=/46052026/11/11 22:35:18 INFO Completed upload id=28 objects=8 duration=1.09093ms46062026/11/11 22:35:18 INFO Received complete push request method=POST path=/46072026/11/11 22:35:18 INFO Completed push id=28 objects=19 duration=1.38241ms46082026/11/11 22:35:18 INFO Received push request method=POST path=/46092026/11/11 22:35:18 INFO Received uploads request method=POST path=/46102026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46112026/11/11 22:35:18 INFO Received complete push request method=POST path=/46122026/11/11 22:35:18 INFO Completed upload id=23 objects=7 duration=1.15148ms46132026/11/11 22:35:18 INFO Completed push id=27 objects=11 duration=1.38523ms46142026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=046152026/11/11 22:35:18 INFO Received complete push request method=POST path=/46162026/11/11 22:35:18 INFO Completed push id=30 objects=4 duration=1.24718ms46172026/11/11 22:35:18 INFO Received push request method=POST path=/46182026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period46192026/11/11 22:35:18 INFO Received complete push request method=POST path=/46202026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46212026/11/11 22:35:18 INFO Completed push id=21 objects=6 duration=1.42414ms46222026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046232026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46242026/11/11 22:35:18 INFO Completed upload id=22 objects=5 duration=2.18412ms46252026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46262026/11/11 22:35:18 INFO Completed upload id=17 objects=5 duration=1.37314ms46272026/11/11 22:35:18 INFO Vacuumed table table=pending_closures46282026/11/11 22:35:18 INFO Completed upload id=29 objects=10 duration=1.12991ms46292026/11/11 22:35:18 INFO Vacuumed table table=pending_objects46302026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads46312026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46322026/11/11 22:35:18 INFO Vacuumed table table=closures46332026/11/11 22:35:18 INFO Vacuumed table table=objects46342026/11/11 22:35:18 INFO Completed upload id=21 objects=9 duration=2.145931ms46352026/11/11 22:35:18 INFO Received uploads request method=POST path=/46362026/11/11 22:35:18 INFO Received complete push request method=POST path=/46372026/11/11 22:35:18 INFO Received complete push request method=POST path=/46382026/11/11 22:35:18 INFO Completed push id=31 objects=7 duration=1.15215ms46392026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46402026/11/11 22:35:18 INFO Completed push id=24 objects=5 duration=1.31465ms46412026/11/11 22:35:18 INFO Completed upload id=29 objects=26 duration=1.11167ms46422026/11/11 22:35:18 INFO Received complete push request method=POST path=/46432026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46442026/11/11 22:35:18 INFO Completed push id=29 objects=10 duration=1.21181ms46452026/11/11 22:35:18 INFO Completed upload id=22 objects=7 duration=1.14837ms46462026/11/11 22:35:18 INFO Received complete push request method=POST path=/46472026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46482026/11/11 22:35:18 INFO Completed upload id=30 objects=20 duration=1.08229ms46492026/11/11 22:35:18 INFO Completed push id=28 objects=3 duration=1.96974ms46502026/11/11 22:35:18 INFO Received complete push request method=POST path=/46512026/11/11 22:35:18 INFO Completed push id=25 objects=10 duration=1.95186ms46522026/11/11 22:35:18 INFO Received complete push request method=POST path=/46532026/11/11 22:35:18 INFO Received uploads request method=POST path=/46542026/11/11 22:35:18 INFO Completed push id=19 objects=3 duration=1.253331ms46552026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46562026/11/11 22:35:18 INFO Completed upload id=32 objects=5 duration=1.28019ms46572026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46582026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=046592026/11/11 22:35:18 INFO Completed upload id=25 objects=2 duration=1.13599ms46602026/11/11 22:35:18 INFO Received complete push request method=POST path=/46612026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period46622026/11/11 22:35:18 INFO Completed push id=30 objects=8 duration=1.94301ms46632026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046642026/11/11 22:35:18 INFO Received complete push request method=POST path=/46652026/11/11 22:35:18 INFO Received complete push request method=POST path=/46662026/11/11 22:35:18 INFO Completed push id=31 objects=11 duration=1.87188ms46672026/11/11 22:35:18 INFO Completed push id=24 objects=4 duration=1.293651ms46682026/11/11 22:35:18 INFO Vacuumed table table=pending_closures46692026/11/11 22:35:18 INFO Received complete push request method=POST path=/46702026/11/11 22:35:18 INFO Received uploads request method=POST path=/46712026/11/11 22:35:18 INFO Vacuumed table table=pending_objects46722026/11/11 22:35:18 INFO Completed push id=29 objects=15 duration=1.61595ms46732026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46742026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads46752026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46762026/11/11 22:35:18 INFO Completed upload id=30 objects=6 duration=1.29861ms46772026/11/11 22:35:18 INFO Completed upload id=29 objects=5 duration=1.13278ms46782026/11/11 22:35:18 INFO Vacuumed table table=closures46792026/11/11 22:35:18 INFO Vacuumed table table=objects46802026/11/11 22:35:18 INFO Received uploads request method=POST path=/46812026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46822026/11/11 22:35:18 INFO Completed upload id=26 objects=20 duration=1.05149ms46832026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46842026/11/11 22:35:18 INFO Completed upload id=31 objects=5 duration=1.04626ms46852026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46862026/11/11 22:35:18 INFO Completed upload id=32 objects=2 duration=1.09025ms46872026/11/11 22:35:18 INFO Received complete push request method=POST path=/46882026/11/11 22:35:18 INFO Completed push id=25 objects=9 duration=1.2414ms46892026/11/11 22:35:18 INFO Received complete push request method=POST path=/46902026/11/11 22:35:18 INFO Received complete push request method=POST path=/46912026/11/11 22:35:18 INFO Completed push id=20 objects=4 duration=1.28673ms46922026/11/11 22:35:18 INFO Completed push id=30 objects=6 duration=1.63169ms46932026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46942026/11/11 22:35:18 INFO Completed upload id=31 objects=5 duration=1.05533ms46952026/11/11 22:35:18 INFO Received complete upload request method=POST path=/46962026/11/11 22:35:18 INFO Received push request method=POST path=/46972026/11/11 22:35:18 INFO Completed upload id=30 objects=2 duration=1.13833ms46982026/11/11 22:35:18 INFO Received complete push request method=POST path=/46992026/11/11 22:35:18 INFO Received uploads request method=POST path=/47002026/11/11 22:35:18 INFO Completed push id=1 objects=9 duration=1.70967ms47012026/11/11 22:35:18 INFO Received complete push request method=POST path=/47022026/11/11 22:35:18 INFO Completed push id=32 objects=3 duration=1.08257ms47032026/11/11 22:35:18 INFO Received complete push request method=POST path=/47042026/11/11 22:35:18 INFO Received complete push request method=POST path=/47052026/11/11 22:35:18 INFO Completed push id=33 objects=3 duration=1.15071ms47062026/11/11 22:35:18 INFO Completed push id=28 objects=7 duration=1.29792ms47072026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47082026/11/11 22:35:18 INFO Completed upload id=26 objects=3 duration=990.921µs47092026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47102026/11/11 22:35:18 INFO Completed upload id=31 objects=3 duration=1.27098ms47112026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47122026/11/11 22:35:18 INFO Completed upload id=34 objects=3 duration=927.86µs47132026/11/11 22:35:18 INFO Received complete push request method=POST path=/47142026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47152026/11/11 22:35:18 INFO Received complete push request method=POST path=/47162026/11/11 22:35:18 INFO Completed push id=31 objects=20 duration=1.37241ms47172026/11/11 22:35:18 INFO Received uploads request method=POST path=/47182026/11/11 22:35:18 INFO Completed upload id=34 objects=24 duration=1.58047ms47192026/11/11 22:35:18 INFO Completed push id=24 objects=24 duration=3.369741ms47202026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=047212026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47222026/11/11 22:35:18 INFO Completed upload id=22 objects=8 duration=1.90991ms47232026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period47242026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47252026/11/11 22:35:18 INFO Completed upload id=34 objects=3 duration=1.08719ms47262026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047272026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=047282026/11/11 22:35:18 INFO Vacuumed table table=pending_closures47292026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=047302026/11/11 22:35:18 INFO Vacuumed table table=pending_objects47312026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads47322026/11/11 22:35:18 INFO Vacuumed table table=closures47332026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47342026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=047352026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period47362026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47372026/11/11 22:35:18 INFO Completed upload id=30 objects=6 duration=1.087931ms47382026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47392026/11/11 22:35:18 INFO Vacuumed table table=objects47402026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period47412026/11/11 22:35:18 INFO Completed upload id=27 objects=7 duration=1.09039ms47422026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047432026/11/11 22:35:18 INFO Completed upload id=32 objects=5 duration=1.12434ms47442026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047452026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=047462026/11/11 22:35:18 INFO Vacuumed table table=pending_closures47472026/11/11 22:35:18 INFO Received push request method=POST path=/47482026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period47492026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=047502026/11/11 22:35:18 INFO Vacuumed table table=pending_closures47512026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=047522026/11/11 22:35:18 INFO Vacuumed table table=pending_objects47532026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads47542026/11/11 22:35:18 INFO Received complete push request method=POST path=/47552026/11/11 22:35:18 INFO Vacuumed table table=pending_objects47562026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047572026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads47582026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period47592026/11/11 22:35:18 INFO Vacuumed table table=closures47602026/11/11 22:35:18 INFO Vacuumed table table=pending_closures47612026/11/11 22:35:18 INFO Completed push id=23 objects=5 duration=1.6322ms47622026/11/11 22:35:18 INFO Vacuumed table table=closures47632026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=047642026/11/11 22:35:18 INFO Vacuumed table table=objects47652026/11/11 22:35:18 INFO Vacuumed table table=pending_closures47662026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period47672026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047682026/11/11 22:35:18 INFO Vacuumed table table=pending_objects47692026/11/11 22:35:18 INFO Vacuumed table table=objects47702026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads47712026/11/11 22:35:18 INFO Vacuumed table table=pending_objects47722026/11/11 22:35:18 INFO Vacuumed table table=closures47732026/11/11 22:35:18 INFO Vacuumed table table=pending_closures47742026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047752026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads47762026/11/11 22:35:18 INFO Vacuumed table table=closures47772026/11/11 22:35:18 INFO Vacuumed table table=objects47782026/11/11 22:35:18 INFO Vacuumed table table=pending_closures47792026/11/11 22:35:18 INFO Vacuumed table table=pending_objects47802026/11/11 22:35:18 INFO Vacuumed table table=objects47812026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads47822026/11/11 22:35:18 INFO Vacuumed table table=closures47832026/11/11 22:35:18 INFO Vacuumed table table=pending_objects47842026/11/11 22:35:18 INFO Received complete push request method=POST path=/47852026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads47862026/11/11 22:35:18 INFO Vacuumed table table=objects47872026/11/11 22:35:18 INFO Vacuumed table table=closures47882026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47892026/11/11 22:35:18 INFO Completed push id=31 objects=5 duration=1.63831ms47902026/11/11 22:35:18 INFO Vacuumed table table=objects47912026/11/11 22:35:18 INFO Completed upload id=28 objects=3 duration=1.25536ms47922026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47932026/11/11 22:35:18 INFO Completed upload id=33 objects=3 duration=815.23µs47942026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47952026/11/11 22:35:18 INFO Completed upload id=2 objects=7 duration=1.29321ms47962026/11/11 22:35:18 INFO Received complete upload request method=POST path=/47972026/11/11 22:35:18 INFO Completed upload id=24 objects=8 duration=1.25581ms47982026/11/11 22:35:18 INFO Received uploads request method=POST path=/47992026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=048002026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period48012026/11/11 22:35:18 INFO Received complete push request method=POST path=/48022026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48032026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048042026/11/11 22:35:18 INFO Completed push id=29 objects=23 duration=1.58845ms48052026/11/11 22:35:18 INFO Completed upload id=28 objects=5 duration=1.52632ms48062026/11/11 22:35:18 INFO Received uploads request method=POST path=/48072026/11/11 22:35:18 INFO Vacuumed table table=pending_closures48082026/11/11 22:35:18 INFO Received complete push request method=POST path=/48092026/11/11 22:35:18 INFO Vacuumed table table=pending_objects48102026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads48112026/11/11 22:35:18 INFO Completed push id=32 objects=26 duration=1.57474ms48122026/11/11 22:35:18 INFO Vacuumed table table=closures48132026/11/11 22:35:18 INFO Vacuumed table table=objects48142026/11/11 22:35:18 INFO Received uploads request method=POST path=/48152026/11/11 22:35:18 INFO Received complete push request method=POST path=/48162026/11/11 22:35:18 INFO Completed push id=30 objects=6 duration=1.13148ms48172026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48182026/11/11 22:35:18 INFO Received complete push request method=POST path=/48192026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48202026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48212026/11/11 22:35:18 INFO Completed upload id=8 objects=6 duration=1.19507ms48222026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48232026/11/11 22:35:18 INFO Completed upload id=26 objects=2 duration=2.349361ms48242026/11/11 22:35:18 INFO Completed upload id=33 objects=2 duration=1.66778ms48252026/11/11 22:35:18 INFO Completed push id=16 objects=7 duration=2.56278ms48262026/11/11 22:35:18 INFO Completed upload id=30 objects=5 duration=1.23171ms48272026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=048282026/11/11 22:35:18 INFO Received complete push request method=POST path=/48292026/11/11 22:35:18 INFO Completed push id=33 objects=3 duration=1.06761ms48302026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048312026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48322026/11/11 22:35:18 INFO Completed upload id=34 objects=5 duration=984.56µs48332026/11/11 22:35:18 INFO Received push request method=POST path=/48342026/11/11 22:35:18 INFO Vacuumed table table=pending_closures48352026/11/11 22:35:18 INFO Received complete push request method=POST path=/48362026/11/11 22:35:18 INFO Vacuumed table table=pending_objects48372026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads48382026/11/11 22:35:18 INFO Completed push id=31 objects=3 duration=1.28919ms48392026/11/11 22:35:18 INFO Vacuumed table table=closures48402026/11/11 22:35:18 INFO Vacuumed table table=objects48412026/11/11 22:35:18 INFO Received complete push request method=POST path=/48422026/11/11 22:35:18 INFO Completed push id=23 objects=6 duration=1.40069ms48432026/11/11 22:35:18 INFO Received complete push request method=POST path=/48442026/11/11 22:35:18 INFO Completed push id=34 objects=6 duration=1.0866ms48452026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48462026/11/11 22:35:18 INFO Completed upload id=35 objects=11 duration=805.94µs48472026/11/11 22:35:18 INFO Received uploads request method=POST path=/48482026/11/11 22:35:18 INFO Received complete push request method=POST path=/48492026/11/11 22:35:18 INFO Completed push id=24 objects=6 duration=1.08975ms48502026/11/11 22:35:18 INFO Received complete push request method=POST path=/48512026/11/11 22:35:18 INFO Completed push id=35 objects=8 duration=953.41µs48522026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48532026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48542026/11/11 22:35:18 INFO Completed upload id=36 objects=3 duration=806.43µs48552026/11/11 22:35:18 INFO Received complete push request method=POST path=/48562026/11/11 22:35:18 INFO Completed upload id=12 objects=9 duration=1.23419ms48572026/11/11 22:35:18 INFO Completed push id=32 objects=6 duration=1.39056ms48582026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=048592026/11/11 22:35:18 INFO Received push request method=POST path=/48602026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period48612026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048622026/11/11 22:35:18 INFO Vacuumed table table=pending_closures48632026/11/11 22:35:18 INFO Vacuumed table table=pending_objects48642026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads48652026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48662026/11/11 22:35:18 INFO Vacuumed table table=closures48672026/11/11 22:35:18 INFO Completed upload id=36 objects=9 duration=922.03µs48682026/11/11 22:35:18 INFO Vacuumed table table=objects48692026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=048702026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48712026/11/11 22:35:18 INFO Received complete push request method=POST path=/48722026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period48732026/11/11 22:35:18 INFO Completed upload id=33 objects=7 duration=1.21328ms48742026/11/11 22:35:18 INFO Completed push id=25 objects=7 duration=1.74575ms48752026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048762026/11/11 22:35:18 INFO Vacuumed table table=pending_closures48772026/11/11 22:35:18 INFO Vacuumed table table=pending_objects48782026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads48792026/11/11 22:35:18 INFO Vacuumed table table=closures48802026/11/11 22:35:18 INFO Vacuumed table table=objects48812026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=048822026/11/11 22:35:18 WARN Force mode enabled - objects will be deleted immediately without grace period48832026/11/11 22:35:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048842026/11/11 22:35:18 INFO Received complete upload request method=POST path=/48852026/11/11 22:35:18 INFO Vacuumed table table=pending_closures48862026/11/11 22:35:18 INFO Completed upload id=34 objects=3 duration=1.19852ms48872026/11/11 22:35:18 INFO Vacuumed table table=pending_objects48882026/11/11 22:35:18 INFO Aborted multipart uploads count=0 kept=048892026/11/11 22:35:18 INFO Vacuumed table table=multipart_uploads48902026/11/11 22:35:18 INFO Vacuumed table table=closures48912026/11/11 22:35:18 INFO Vacuumed table table=objects48922026/11/11 22:35:18 INFO Received push request method=POST path=/48932026/11/11 22:35:18 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."48942026/11/11 22:35:18 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."48952026/11/11 22:35:18 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48962026/11/11 22:35:18 INFO Received complete push request method=POST path=/48972026/11/11 22:35:19 INFO Completed push id=35 objects=11 duration=1.33929ms48982026/11/11 22:35:19 INFO Received complete upload request method=POST path=/48992026/11/11 22:35:19 INFO Completed upload id=26 objects=24 duration=1.48847ms49002026/11/11 22:35:19 INFO Received push request method=POST path=/49012026/11/11 22:35:19 INFO Aborted multipart uploads count=0 kept=049022026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49032026/11/11 22:35:19 WARN Force mode enabled - objects will be deleted immediately without grace period49042026/11/11 22:35:19 INFO Completed upload id=23 objects=7 duration=3.09086ms49052026/11/11 22:35:19 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049062026/11/11 22:35:19 INFO Vacuumed table table=pending_closures49072026/11/11 22:35:19 INFO Vacuumed table table=pending_objects49082026/11/11 22:35:19 INFO Vacuumed table table=multipart_uploads49092026/11/11 22:35:19 INFO Vacuumed table table=closures49102026/11/11 22:35:19 INFO Vacuumed table table=objects49112026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49122026/11/11 22:35:19 INFO Completed upload id=26 objects=5 duration=5.54077ms49132026/11/11 22:35:19 INFO Received push request method=POST path=/49142026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49152026/11/11 22:35:19 INFO Completed upload id=18 objects=22 duration=1.46042ms49162026/11/11 22:35:19 INFO Received complete push request method=POST path=/49172026/11/11 22:35:19 INFO Completed push id=27 objects=3 duration=993.4µs49182026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49192026/11/11 22:35:19 INFO Received complete push request method=POST path=/49202026/11/11 22:35:19 INFO Completed upload id=27 objects=26 duration=1.29012ms49212026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49222026/11/11 22:35:19 INFO Completed push id=28 objects=11 duration=2.28312ms49232026/11/11 22:35:19 INFO Completed upload id=21 objects=8 duration=942.46µs49242026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49252026/11/11 22:35:19 INFO Completed upload id=28 objects=5 duration=746.22µs49262026/11/11 22:35:19 INFO Received complete push request method=POST path=/49272026/11/11 22:35:19 INFO Completed push id=28 objects=6 duration=1.255671ms49282026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49292026/11/11 22:35:19 INFO Completed upload id=22 objects=7 duration=1.06634ms49302026/11/11 22:35:19 INFO Received complete push request method=POST path=/49312026/11/11 22:35:19 INFO Completed push id=29 objects=17 duration=841.83µs49322026/11/11 22:35:19 INFO Received complete push request method=POST path=/49332026/11/11 22:35:19 INFO Completed push id=29 objects=6 duration=1.16283ms49342026/11/11 22:35:19 INFO Received complete push request method=POST path=/49352026/11/11 22:35:19 INFO Received complete push request method=POST path=/49362026/11/11 22:35:19 INFO Completed push id=24 objects=5 duration=1.20156ms49372026/11/11 22:35:19 INFO Completed push id=29 objects=24 duration=1.86347ms49382026/11/11 22:35:19 INFO Aborted multipart uploads count=0 kept=049392026/11/11 22:35:19 WARN Force mode enabled - objects will be deleted immediately without grace period49402026/11/11 22:35:19 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049412026/11/11 22:35:19 INFO Vacuumed table table=pending_closures49422026/11/11 22:35:19 INFO Vacuumed table table=pending_objects49432026/11/11 22:35:19 INFO Vacuumed table table=multipart_uploads49442026/11/11 22:35:19 INFO Vacuumed table table=closures49452026/11/11 22:35:19 INFO Vacuumed table table=objects49462026/11/11 22:35:19 INFO Aborted multipart uploads count=0 kept=049472026/11/11 22:35:19 WARN Force mode enabled - objects will be deleted immediately without grace period49482026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49492026/11/11 22:35:19 INFO Completed upload id=25 objects=8 duration=1.16645ms49502026/11/11 22:35:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049512026/11/11 22:35:19 INFO Vacuumed table table=pending_closures49522026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49532026/11/11 22:35:19 INFO Vacuumed table table=pending_objects49542026/11/11 22:35:19 INFO Completed upload id=30 objects=8 duration=1.42445ms49552026/11/11 22:35:19 INFO Vacuumed table table=multipart_uploads49562026/11/11 22:35:19 INFO Vacuumed table table=closures49572026/11/11 22:35:19 INFO Vacuumed table table=objects49582026/11/11 22:35:19 INFO Received complete push request method=POST path=/49592026/11/11 22:35:19 INFO Completed push id=26 objects=4 duration=1.13491ms49602026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49612026/11/11 22:35:19 INFO Completed upload id=31 objects=6 duration=1.3138ms49622026/11/11 22:35:19 INFO Received complete push request method=POST path=/49632026/11/11 22:35:19 INFO Completed push id=30 objects=2 duration=1.155781ms49642026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49652026/11/11 22:35:19 INFO Completed upload id=32 objects=4 duration=1.0107ms49662026/11/11 22:35:19 INFO Received complete push request method=POST path=/49672026/11/11 22:35:19 INFO Completed push id=32 objects=7 duration=1.37781ms49682026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49692026/11/11 22:35:19 INFO Completed upload id=33 objects=10 duration=1.13677ms49702026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49712026/11/11 22:35:19 INFO Completed upload id=33 objects=6 duration=1.03189ms49722026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49732026/11/11 22:35:19 INFO Completed upload id=34 objects=23 duration=1.270921ms49742026/11/11 22:35:19 INFO Received complete upload request method=POST path=/49752026/11/11 22:35:19 INFO Completed upload id=34 objects=3 duration=788µs49762026/11/11 22:35:19 INFO Aborted multipart uploads count=0 kept=049772026/11/11 22:35:19 WARN Force mode enabled - objects will be deleted immediately without grace period49782026/11/11 22:35:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049792026/11/11 22:35:19 INFO Received complete push request method=POST path=/49802026/11/11 22:35:19 INFO Completed push id=35 objects=6 duration=1.157311ms49812026/11/11 22:35:19 INFO Vacuumed table table=pending_closures49822026/11/11 22:35:19 INFO Vacuumed table table=pending_objects49832026/11/11 22:35:19 INFO Vacuumed table table=multipart_uploads49842026/11/11 22:35:19 INFO Vacuumed table table=closures49852026/11/11 22:35:19 INFO Vacuumed table table=objects49862026/11/11 22:35:19 INFO Aborted multipart uploads count=0 kept=049872026/11/11 22:35:19 WARN Force mode enabled - objects will be deleted immediately without grace period49882026/11/11 22:35:19 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049892026/11/11 22:35:19 INFO Vacuumed table table=pending_closures49902026/11/11 22:35:19 INFO Vacuumed table table=pending_objects49912026/11/11 22:35:19 INFO Vacuumed table table=multipart_uploads49922026/11/11 22:35:19 INFO Vacuumed table table=closures49932026/11/11 22:35:19 INFO Vacuumed table table=objects4994--- PASS: FuzzPushGC (0.00s)4995 --- PASS: FuzzPushGC/seed#14 (2.60s)4996 --- PASS: FuzzPushGC/seed#0 (2.87s)4997 --- PASS: FuzzPushGC/seed#7 (2.95s)4998 --- PASS: FuzzPushGC/seed#3 (3.09s)4999 --- PASS: FuzzPushGC/seed#15 (3.10s)5000 --- PASS: FuzzPushGC/seed#13 (3.11s)5001 --- PASS: FuzzPushGC/seed#9 (3.11s)5002 --- PASS: FuzzPushGC/seed#5 (3.11s)5003 --- PASS: FuzzPushGC/seed#6 (3.14s)5004 --- PASS: FuzzPushGC/seed#11 (3.24s)5005 --- PASS: FuzzPushGC/seed#4 (3.25s)5006 --- PASS: FuzzPushGC/seed#8 (3.30s)5007 --- PASS: FuzzPushGC/seed#1 (3.41s)5008 --- PASS: FuzzPushGC/seed#2 (3.43s)5009 --- PASS: FuzzPushGC/seed#10 (3.57s)5010 --- PASS: FuzzPushGC/seed#12 (3.60s)5011PASS5012{"timestamp":"2026-11-11T22:35:19.353445888Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:54304","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(389)"}50132026-11-11 22:35:19.759 UTC [129] LOG: received smart shutdown request50142026-11-11 22:35:19.763 UTC [129] LOG: background worker "logical replication launcher" (PID 139) exited with exit code 150152026-11-11 22:35:20.548 UTC [134] LOG: checkpoint complete: wrote 4119 buffers (25.1%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=4.081 s, sync=0.630 s, total=4.771 s; sync files=42363, longest=0.002 s, average=0.001 s; distance=535527 kB, estimate=535527 kB; lsn=0/26D59440, redo lsn=0/2225F98850162026-11-11 22:35:20.550 UTC [134] LOG: shutting down50172026-11-11 22:35:20.551 UTC [134] LOG: checkpoint starting: shutdown immediate50182026-11-11 22:35:21.392 UTC [134] LOG: checkpoint complete: wrote 9600 buffers (58.6%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.169 s, sync=0.664 s, total=0.842 s; sync files=2842, longest=0.002 s, average=0.001 s; distance=76774 kB, estimate=489651 kB; lsn=0/26D594B8, redo lsn=0/26D594B850192026-11-11 22:35:21.453 UTC [129] LOG: database system is shut down5020Running OIDC tests...5021=== RUN TestAudienceForIssuer5022=== PAUSE TestAudienceForIssuer5023=== RUN TestGlobMatch5024=== PAUSE TestGlobMatch5025=== RUN TestValidateToken_ValidToken5026=== PAUSE TestValidateToken_ValidToken5027=== RUN TestValidateToken_WrongAudience5028=== PAUSE TestValidateToken_WrongAudience5029=== RUN TestValidateToken_Expired5030=== PAUSE TestValidateToken_Expired5031=== RUN TestValidateToken_BoundClaimsMismatch5032=== PAUSE TestValidateToken_BoundClaimsMismatch5033=== RUN TestValidateToken_BoundSubjectMismatch5034=== PAUSE TestValidateToken_BoundSubjectMismatch5035=== RUN TestValidateToken_MultipleProviders5036=== PAUSE TestValidateToken_MultipleProviders5037=== RUN TestValidateToken_NoMatchingProvider5038=== PAUSE TestValidateToken_NoMatchingProvider5039=== RUN TestValidateToken_KubernetesServiceAccount5040=== PAUSE TestValidateToken_KubernetesServiceAccount5041=== RUN TestNewValidator_KubernetesRequiresCA5042=== PAUSE TestNewValidator_KubernetesRequiresCA5043=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5044=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5045=== RUN TestPins_ReservedForMatchingRule5046=== PAUSE TestPins_ReservedForMatchingRule5047=== RUN TestPins_TopLevelShorthand5048=== PAUSE TestPins_TopLevelShorthand5049=== RUN TestPins_ConfigValidation5050=== PAUSE TestPins_ConfigValidation5051=== RUN TestScopes_LegacyProviderDefaultsToWrite5052=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5053=== RUN TestScopes_Rules5054=== PAUSE TestScopes_Rules5055=== RUN TestScopes_ConfigValidation5056=== PAUSE TestScopes_ConfigValidation5057=== CONT TestAudienceForIssuer5058=== CONT TestValidateToken_KubernetesServiceAccount5059--- PASS: TestAudienceForIssuer (0.00s)5060=== CONT TestPins_ConfigValidation5061=== CONT TestValidateToken_NoMatchingProvider5062=== CONT TestScopes_ConfigValidation5063=== CONT TestValidateToken_MultipleProviders5064--- PASS: TestPins_ConfigValidation (0.00s)5065=== CONT TestScopes_Rules5066--- PASS: TestScopes_ConfigValidation (0.00s)5067=== CONT TestValidateToken_BoundSubjectMismatch5068=== CONT TestValidateToken_BoundClaimsMismatch5069=== CONT TestValidateToken_Expired5070=== CONT TestValidateToken_WrongAudience5071=== CONT TestValidateToken_ValidToken5072=== CONT TestScopes_LegacyProviderDefaultsToWrite5073=== CONT TestGlobMatch5074=== CONT TestPins_TopLevelShorthand5075=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5076=== CONT TestNewValidator_KubernetesRequiresCA5077=== CONT TestPins_ReservedForMatchingRule5078=== RUN TestGlobMatch/foo_foo5079=== PAUSE TestGlobMatch/foo_foo5080=== RUN TestGlobMatch/foo_bar5081=== PAUSE TestGlobMatch/foo_bar5082=== RUN TestGlobMatch/*_5083=== PAUSE TestGlobMatch/*_5084=== RUN TestGlobMatch/*_anything5085=== PAUSE TestGlobMatch/*_anything5086=== RUN TestGlobMatch/foo*_foo5087=== PAUSE TestGlobMatch/foo*_foo5088=== RUN TestGlobMatch/foo*_foobar5089=== PAUSE TestGlobMatch/foo*_foobar5090=== RUN TestGlobMatch/foo*_bar5091=== PAUSE TestGlobMatch/foo*_bar5092=== RUN TestGlobMatch/*bar_bar5093=== PAUSE TestGlobMatch/*bar_bar5094=== RUN TestGlobMatch/*bar_foobar5095=== PAUSE TestGlobMatch/*bar_foobar5096=== RUN TestGlobMatch/*bar_foo5097=== PAUSE TestGlobMatch/*bar_foo5098=== RUN TestGlobMatch/foo*bar_foobar5099=== PAUSE TestGlobMatch/foo*bar_foobar5100=== RUN TestGlobMatch/foo*bar_foo123bar5101=== PAUSE TestGlobMatch/foo*bar_foo123bar5102=== RUN TestGlobMatch/foo*bar_foobarbaz5103=== PAUSE TestGlobMatch/foo*bar_foobarbaz5104=== RUN TestGlobMatch/*/*_foo/bar5105=== PAUSE TestGlobMatch/*/*_foo/bar5106=== RUN TestGlobMatch/*/*_foo5107=== PAUSE TestGlobMatch/*/*_foo5108=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5109=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5110=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05111=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05112=== RUN TestGlobMatch/refs/*/main_refs/heads/main5113=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5114=== RUN TestGlobMatch/fo?_foo5115=== PAUSE TestGlobMatch/fo?_foo5116=== RUN TestGlobMatch/fo?_fo5117=== PAUSE TestGlobMatch/fo?_fo5118=== RUN TestGlobMatch/fo?_fooo5119=== PAUSE TestGlobMatch/fo?_fooo5120=== RUN TestGlobMatch/?oo_foo5121=== PAUSE TestGlobMatch/?oo_foo5122=== RUN TestGlobMatch/?oo_boo5123=== PAUSE TestGlobMatch/?oo_boo5124=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5125=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5126=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5127=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5128=== CONT TestGlobMatch/foo_foo5129=== CONT TestGlobMatch/foo*bar_foobarbaz5130=== CONT TestGlobMatch/foo*bar_foo123bar5131=== CONT TestGlobMatch/foo*bar_foobar5132=== CONT TestGlobMatch/*bar_foo5133=== CONT TestGlobMatch/*bar_foobar5134=== CONT TestGlobMatch/*bar_bar5135=== CONT TestGlobMatch/foo*_bar5136=== CONT TestGlobMatch/foo*_foobar5137=== CONT TestGlobMatch/foo*_foo5138=== CONT TestGlobMatch/*/*_foo/bar5139=== CONT TestGlobMatch/*_anything5140=== CONT TestGlobMatch/*_5141=== CONT TestGlobMatch/foo_bar5142=== CONT TestGlobMatch/fo?_fo5143=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05144=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5145=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5146=== CONT TestGlobMatch/?oo_boo5147=== CONT TestGlobMatch/?oo_foo5148=== CONT TestGlobMatch/fo?_fooo5149=== CONT TestGlobMatch/fo?_foo5150=== CONT TestGlobMatch/refs/*/main_refs/heads/main5151=== CONT TestGlobMatch/*/*_foo5152=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5153--- PASS: TestGlobMatch (0.00s)5154 --- PASS: TestGlobMatch/foo_foo (0.00s)5155 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5156 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5157 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5158 --- PASS: TestGlobMatch/*bar_foo (0.00s)5159 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5160 --- PASS: TestGlobMatch/*bar_bar (0.00s)5161 --- PASS: TestGlobMatch/foo*_bar (0.00s)5162 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5163 --- PASS: TestGlobMatch/foo*_foo (0.00s)5164 --- PASS: TestGlobMatch/foo_bar (0.00s)5165 --- PASS: TestGlobMatch/fo?_fo (0.00s)5166 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5167 --- PASS: TestGlobMatch/*_ (0.00s)5168 --- PASS: TestGlobMatch/*_anything (0.00s)5169 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5170 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5171 --- PASS: TestGlobMatch/?oo_boo (0.00s)5172 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5173 --- PASS: TestGlobMatch/fo?_foo (0.00s)5174 --- PASS: TestGlobMatch/?oo_foo (0.00s)5175 --- PASS: TestGlobMatch/*/*_foo (0.00s)5176 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5177 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5178 --- PASS: TestGlobMatch/fo?_fooo (0.00s)51792026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33291/oidc5180--- PASS: TestValidateToken_WrongAudience (0.03s)51812026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45991/oidc51822026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36191/oidc5183--- PASS: TestValidateToken_Expired (0.04s)5184--- PASS: TestScopes_Rules (0.04s)51852026/11/11 22:35:24 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235186--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.04s)51872026/11/11 22:35:24 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:438995188--- PASS: TestValidateToken_KubernetesServiceAccount (0.06s)51892026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42223/oidc5190--- PASS: TestValidateToken_BoundClaimsMismatch (0.07s)51912026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41489/oidc5192--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.07s)51932026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35411/oidc5194--- PASS: TestPins_ReservedForMatchingRule (0.08s)51952026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42347/oidc5196--- PASS: TestValidateToken_ValidToken (0.09s)51972026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36753/oidc5198--- PASS: TestPins_TopLevelShorthand (0.09s)51992026/11/11 22:35:24 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:35991/oidc52002026/11/11 22:35:24 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:38243/oidc52012026/11/11 22:35:24 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43861/oidc5202--- PASS: TestValidateToken_MultipleProviders (0.11s)5203--- PASS: TestValidateToken_BoundSubjectMismatch (0.11s)52042026/11/11 22:35:24 http: TLS handshake error from 127.0.0.1:33776: remote error: tls: bad certificate5205--- PASS: TestNewValidator_KubernetesRequiresCA (0.14s)52062026/11/11 22:35:24 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:37097/oidc5207--- PASS: TestValidateToken_NoMatchingProvider (0.27s)5208PASS5209Running hook tests...5210=== RUN TestSendPathsEmpty5211=== PAUSE TestSendPathsEmpty5212=== RUN TestQueueEnqueueAndFetch5213=== PAUSE TestQueueEnqueueAndFetch5214=== RUN TestQueueDeduplication5215=== PAUSE TestQueueDeduplication5216=== RUN TestQueueRemove5217=== PAUSE TestQueueRemove5218=== RUN TestQueueFetchBatchLimit5219=== PAUSE TestQueueFetchBatchLimit5220=== RUN TestQueueRetryMovesToBack5221=== PAUSE TestQueueRetryMovesToBack5222=== RUN TestQueueFetchRemoveLifecycle5223=== PAUSE TestQueueFetchRemoveLifecycle5224=== RUN TestQueueConcurrentWriters5225=== PAUSE TestQueueConcurrentWriters5226=== RUN TestQueueRemoveLargeClosure5227=== PAUSE TestQueueRemoveLargeClosure5228=== RUN TestServerClientIntegration5229--- PASS: TestServerClientIntegration (0.00s)5230=== RUN TestServerQueueError52312026/11/11 22:35:24 ERROR Failed to queue paths error="permission denied" count=15232--- PASS: TestServerQueueError (0.00s)5233=== RUN TestGetListenerSocketActivation5234 server_test.go:218: === RUN TestGetListenerSocketActivation5235 --- PASS: TestGetListenerSocketActivation (0.00s)5236 PASS5237 5238--- PASS: TestGetListenerSocketActivation (0.01s)5239=== RUN TestServerStalledClientDoesNotBlockShutdown5240=== PAUSE TestServerStalledClientDoesNotBlockShutdown5241=== RUN TestServerBacksOffOnAcceptErrors5242=== PAUSE TestServerBacksOffOnAcceptErrors5243=== RUN TestDrainIsolatesPoisonPath5244=== PAUSE TestDrainIsolatesPoisonPath5245=== RUN TestRunNotBlockedByPoisonHead5246=== PAUSE TestRunNotBlockedByPoisonHead5247=== RUN TestDrainGivesUpWhenServerDown5248=== PAUSE TestDrainGivesUpWhenServerDown5249=== RUN TestFailedPathPrunedByLaterClosure5250=== PAUSE TestFailedPathPrunedByLaterClosure5251=== RUN TestWorkerUploadsAndRemoves5252=== PAUSE TestWorkerUploadsAndRemoves5253=== RUN TestWorkerSkipsGCdPaths5254=== PAUSE TestWorkerSkipsGCdPaths5255=== RUN TestWorkerPrunesClosureDeps5256=== PAUSE TestWorkerPrunesClosureDeps5257=== RUN TestDrainTimeout5258=== PAUSE TestDrainTimeout5259=== RUN TestWorkerRemoveFailureIsNotProgress5260=== PAUSE TestWorkerRemoveFailureIsNotProgress5261=== RUN TestWorkerKeepsPathItCannotStat5262=== PAUSE TestWorkerKeepsPathItCannotStat5263=== CONT TestSendPathsEmpty5264=== CONT TestWorkerSkipsGCdPaths5265--- PASS: TestSendPathsEmpty (0.00s)5266=== CONT TestWorkerKeepsPathItCannotStat5267=== CONT TestWorkerRemoveFailureIsNotProgress5268=== CONT TestDrainTimeout5269=== CONT TestWorkerPrunesClosureDeps5270=== CONT TestDrainIsolatesPoisonPath5271=== CONT TestQueueRetryMovesToBack5272=== CONT TestQueueFetchBatchLimit5273=== CONT TestQueueRemove5274=== CONT TestQueueFetchRemoveLifecycle5275=== CONT TestQueueDeduplication5276=== CONT TestServerBacksOffOnAcceptErrors52772026/11/11 22:35:24 ERROR Accept failed error="too many open files"5278=== CONT TestServerStalledClientDoesNotBlockShutdown5279=== CONT TestQueueRemoveLargeClosure5280=== CONT TestQueueEnqueueAndFetch5281=== CONT TestFailedPathPrunedByLaterClosure5282=== CONT TestQueueConcurrentWriters5283=== CONT TestWorkerUploadsAndRemoves5284=== CONT TestRunNotBlockedByPoisonHead5285=== CONT TestDrainGivesUpWhenServerDown52862026/11/11 22:35:24 ERROR Accept failed error="too many open files"52872026/11/11 22:35:24 INFO Upload queue status pending=352882026/11/11 22:35:24 INFO Upload queue status pending=252892026/11/11 22:35:24 INFO Uploading batch count=152902026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=152912026/11/11 22:35:24 INFO Uploading batch count=152922026/11/11 22:35:24 INFO Uploading batch count=152932026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=152942026/11/11 22:35:24 INFO Upload queue status pending=252952026/11/11 22:35:24 INFO Upload queue status pending=252962026/11/11 22:35:24 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths659142146/002/nonexistent52972026/11/11 22:35:24 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2493387143/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2493387143/002/locked/aaa: permission denied"52982026/11/11 22:35:24 INFO Uploading batch count=252992026/11/11 22:35:24 INFO Uploading batch count=453002026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=453012026/11/11 22:35:24 INFO Uploading batch count=153022026/11/11 22:35:24 INFO Uploading batch count=253032026/11/11 22:35:24 ERROR Accept failed error="too many open files"53042026/11/11 22:35:24 INFO Uploading batch count=153052026/11/11 22:35:24 INFO Uploading batch count=153062026/11/11 22:35:24 INFO Uploading batch count=253072026/11/11 22:35:24 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2493387143/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2493387143/002/locked/aaa: permission denied"53082026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=253092026/11/11 22:35:24 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2730979208/002/a53102026/11/11 22:35:24 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath549898429/002/bbb5311--- PASS: TestQueueDeduplication (0.02s)5312--- PASS: TestQueueEnqueueAndFetch (0.02s)5313--- PASS: TestQueueFetchBatchLimit (0.02s)53142026/11/11 22:35:24 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2493387143/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2493387143/002/locked/aaa: permission denied"5315--- PASS: TestQueueRetryMovesToBack (0.02s)53162026/11/11 22:35:24 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2730979208/002/b5317--- PASS: TestQueueRemove (0.02s)53182026/11/11 22:35:24 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1379871189/002/nonexistent5319--- PASS: TestQueueFetchRemoveLifecycle (0.02s)53202026/11/11 22:35:24 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153212026/11/11 22:35:24 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1379871189/002/nonexistent53222026/11/11 22:35:24 ERROR Drain finished with paths left in queue remaining=153232026/11/11 22:35:24 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153242026/11/11 22:35:24 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1379871189/002/nonexistent53252026/11/11 22:35:24 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153262026/11/11 22:35:24 INFO Uploading batch count=253272026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=253282026/11/11 22:35:24 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2730979208/002/c53292026/11/11 22:35:24 ERROR Drain finished with paths left in queue remaining=153302026/11/11 22:35:24 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2730979208/002/d53312026/11/11 22:35:24 INFO Uploading batch count=153322026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=15333--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5334--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)53352026/11/11 22:35:24 INFO Uploading batch count=253362026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=253372026/11/11 22:35:24 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2730979208/002/e5338--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)53392026/11/11 22:35:24 INFO Uploading batch count=153402026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=153412026/11/11 22:35:24 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2730979208/002/f53422026/11/11 22:35:24 INFO Uploading batch count=153432026/11/11 22:35:24 ERROR Upload failed error="upload failed" count=153442026/11/11 22:35:24 ERROR Drain finished with paths left in queue remaining=1053452026/11/11 22:35:24 ERROR Drain finished with paths left in queue remaining=15346--- PASS: TestDrainIsolatesPoisonPath (0.03s)5347--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5348--- PASS: TestWorkerPrunesClosureDeps (0.04s)53492026/11/11 22:35:24 ERROR Accept failed error="too many open files"5350--- PASS: TestWorkerSkipsGCdPaths (0.04s)5351--- PASS: TestWorkerUploadsAndRemoves (0.04s)5352--- PASS: TestQueueRemoveLargeClosure (0.05s)53532026/11/11 22:35:24 ERROR Accept failed error="too many open files"53542026/11/11 22:35:24 ERROR Accept failed error="too many open files"5355--- PASS: TestQueueConcurrentWriters (0.19s)53562026/11/11 22:35:24 ERROR Failed to decode request error="read unix /build/hook3087343359/test.sock->@: i/o timeout"53572026/11/11 22:35:24 ERROR Failed to write response error="write unix /build/hook3087343359/test.sock->@: i/o timeout"5358--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)53592026/11/11 22:35:24 ERROR Upload failed error="context deadline exceeded" count=253602026/11/11 22:35:24 ERROR Drain finished with paths left in queue remaining=45361--- PASS: TestDrainTimeout (0.22s)53622026/11/11 22:35:24 ERROR Accept failed error="too many open files"53632026/11/11 22:35:24 ERROR Accept failed error="too many open files"5364--- PASS: TestServerBacksOffOnAcceptErrors (0.64s)53652026/11/11 22:35:25 INFO Uploading batch count=153662026/11/11 22:35:25 INFO Uploading batch count=153672026/11/11 22:35:25 INFO Uploading batch count=153682026/11/11 22:35:25 ERROR Upload failed error="upload failed" count=153692026/11/11 22:35:25 INFO Uploading batch count=153702026/11/11 22:35:25 ERROR Upload failed error="upload failed" count=153712026/11/11 22:35:25 INFO Uploading batch count=153722026/11/11 22:35:25 ERROR Upload failed error="upload failed" count=153732026/11/11 22:35:25 INFO Uploading batch count=153742026/11/11 22:35:25 ERROR Upload failed error="upload failed" count=153752026/11/11 22:35:25 ERROR Drain finished with paths left in queue remaining=15376--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5377PASS5378Running signing tests...5379=== RUN TestExternalSigner5380=== PAUSE TestExternalSigner5381=== RUN TestExternalSignerWrongPublicKey5382=== PAUSE TestExternalSignerWrongPublicKey5383=== RUN TestExternalSignerErrors5384=== PAUSE TestExternalSignerErrors5385=== RUN TestExternalSignerRestartsAfterCrash5386=== PAUSE TestExternalSignerRestartsAfterCrash5387=== RUN TestExternalSignerProgramErrorKeepsProcess5388=== PAUSE TestExternalSignerProgramErrorKeepsProcess5389=== RUN TestExternalSignerClose5390=== PAUSE TestExternalSignerClose5391=== RUN TestExternalSignerRejectsInvalidPublicKey5392=== PAUSE TestExternalSignerRejectsInvalidPublicKey5393=== RUN TestExternalSignerCancellation5394=== PAUSE TestExternalSignerCancellation5395=== RUN TestGenerateFingerprint5396=== PAUSE TestGenerateFingerprint5397=== RUN TestParseSigningKey5398=== PAUSE TestParseSigningKey5399=== RUN TestSign5400=== PAUSE TestSign5401=== RUN TestSignBatch5402=== PAUSE TestSignBatch5403=== RUN TestNilKeyPublicKey5404=== PAUSE TestNilKeyPublicKey5405=== RUN TestSignErrors5406=== PAUSE TestSignErrors5407=== CONT TestExternalSigner5408=== CONT TestExternalSignerCancellation5409=== CONT TestSignBatch5410=== CONT TestSignErrors5411=== CONT TestNilKeyPublicKey5412--- PASS: TestNilKeyPublicKey (0.00s)5413=== CONT TestExternalSignerRejectsInvalidPublicKey5414=== CONT TestParseSigningKey5415=== RUN TestParseSigningKey/valid_32-byte_key5416=== CONT TestSign5417=== CONT TestGenerateFingerprint5418=== CONT TestExternalSignerErrors5419=== CONT TestExternalSignerRestartsAfterCrash5420=== CONT TestExternalSignerWrongPublicKey5421=== CONT TestExternalSignerClose5422=== CONT TestExternalSignerProgramErrorKeepsProcess5423=== RUN TestExternalSignerRejectsInvalidPublicKey/#005424=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005425=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5426=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5427=== RUN TestGenerateFingerprint/basic_with_references5428=== PAUSE TestGenerateFingerprint/basic_with_references5429=== RUN TestGenerateFingerprint/no_references5430=== PAUSE TestGenerateFingerprint/no_references5431=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5432=== RUN TestExternalSignerErrors/wrong_signature_name5433=== PAUSE TestExternalSignerErrors/wrong_signature_name5434=== PAUSE TestParseSigningKey/valid_32-byte_key5435=== RUN TestExternalSignerErrors/program_error5436=== PAUSE TestExternalSignerErrors/program_error5437=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5438=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5439=== RUN TestSignErrors/nil_key5440=== PAUSE TestSignErrors/nil_key5441=== RUN TestSignErrors/nil_narinfo5442=== PAUSE TestSignErrors/nil_narinfo5443--- PASS: TestSign (0.00s)5444--- PASS: TestSignBatch (0.00s)5445=== RUN TestExternalSignerErrors/invalid_JSON5446=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5447=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5448=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5449=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5450=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5451=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5452=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5453=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5454=== RUN TestGenerateFingerprint/invalid_nar_hash_length5455=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5456=== RUN TestGenerateFingerprint/invalid_store_path_prefix5457=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5458=== RUN TestGenerateFingerprint/invalid_reference_prefix5459=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5460=== CONT TestGenerateFingerprint/basic_with_references5461=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5462=== RUN TestSignErrors/invalid_narinfo5463=== PAUSE TestSignErrors/invalid_narinfo5464=== CONT TestGenerateFingerprint/no_references5465=== CONT TestGenerateFingerprint/invalid_nar_hash_length5466=== PAUSE TestExternalSignerErrors/invalid_JSON5467=== RUN TestExternalSignerErrors/wrong_shape5468=== RUN TestParseSigningKey/no_colon5469=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5470=== CONT TestExternalSignerRejectsInvalidPublicKey/#005471=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5472=== CONT TestGenerateFingerprint/invalid_reference_prefix5473=== CONT TestGenerateFingerprint/invalid_store_path_prefix5474=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5475--- PASS: TestGenerateFingerprint (0.00s)5476 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5477 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5478 --- PASS: TestGenerateFingerprint/no_references (0.00s)5479 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5480 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5481 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5482 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5483=== RUN TestSignErrors/canceled5484=== PAUSE TestSignErrors/canceled5485=== CONT TestSignErrors/nil_key5486=== CONT TestSignErrors/invalid_narinfo5487=== PAUSE TestExternalSignerErrors/wrong_shape5488=== CONT TestSignErrors/canceled5489=== PAUSE TestParseSigningKey/no_colon5490=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5491=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5492=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5493=== CONT TestSignErrors/nil_narinfo5494--- PASS: TestExternalSignerClose (0.00s)5495--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5496 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5497 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5498 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5499 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5500 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5501=== RUN TestExternalSignerErrors/wrong_id5502=== PAUSE TestExternalSignerErrors/wrong_id5503=== RUN TestExternalSignerErrors/unrequested_key5504=== PAUSE TestExternalSignerErrors/unrequested_key5505=== RUN TestExternalSignerErrors/missing_signature_name5506--- PASS: TestSignErrors (0.00s)5507 --- PASS: TestSignErrors/nil_key (0.00s)5508 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5509 --- PASS: TestSignErrors/canceled (0.00s)5510 --- PASS: TestSignErrors/nil_narinfo (0.00s)5511=== RUN TestParseSigningKey/empty_name5512=== PAUSE TestParseSigningKey/empty_name5513=== RUN TestParseSigningKey/invalid_base645514=== PAUSE TestParseSigningKey/invalid_base645515=== RUN TestParseSigningKey/wrong_length5516=== PAUSE TestParseSigningKey/wrong_length5517=== CONT TestParseSigningKey/valid_32-byte_key5518=== PAUSE TestExternalSignerErrors/missing_signature_name5519=== RUN TestExternalSignerErrors/invalid_signature_base645520=== CONT TestParseSigningKey/invalid_base645521=== CONT TestParseSigningKey/empty_name5522=== CONT TestParseSigningKey/no_colon5523=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5524--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5525--- PASS: TestExternalSignerWrongPublicKey (0.00s)5526=== PAUSE TestExternalSignerErrors/invalid_signature_base645527=== RUN TestExternalSignerErrors/exit_on_request5528=== CONT TestParseSigningKey/wrong_length5529--- PASS: TestParseSigningKey (0.00s)5530 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5531 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5532 --- PASS: TestParseSigningKey/empty_name (0.00s)5533 --- PASS: TestParseSigningKey/no_colon (0.00s)5534 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5535 --- PASS: TestParseSigningKey/wrong_length (0.00s)5536=== PAUSE TestExternalSignerErrors/exit_on_request5537=== RUN TestExternalSignerErrors/null5538=== PAUSE TestExternalSignerErrors/null5539=== RUN TestExternalSignerErrors/missing_signatures5540=== PAUSE TestExternalSignerErrors/missing_signatures5541=== RUN TestExternalSignerErrors/null_signatures5542=== PAUSE TestExternalSignerErrors/null_signatures5543=== RUN TestExternalSignerErrors/wrong_signature_length5544=== PAUSE TestExternalSignerErrors/wrong_signature_length5545=== RUN TestExternalSignerErrors/exit_at_startup5546=== PAUSE TestExternalSignerErrors/exit_at_startup5547=== RUN TestExternalSignerErrors/wrong_signatures_shape5548=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5549=== RUN TestExternalSignerErrors/too_few_signatures5550=== PAUSE TestExternalSignerErrors/too_few_signatures5551=== CONT TestExternalSignerErrors/wrong_signature_name5552=== CONT TestExternalSignerErrors/wrong_signature_length5553=== CONT TestExternalSignerErrors/missing_signature_name5554=== CONT TestExternalSignerErrors/too_few_signatures5555=== CONT TestExternalSignerErrors/wrong_signatures_shape5556=== CONT TestExternalSignerErrors/exit_at_startup5557=== CONT TestExternalSignerErrors/missing_signatures5558=== CONT TestExternalSignerErrors/null5559=== CONT TestExternalSignerErrors/exit_on_request5560=== CONT TestExternalSignerErrors/wrong_shape5561=== CONT TestExternalSignerErrors/unrequested_key5562=== CONT TestExternalSignerErrors/wrong_id5563=== CONT TestExternalSignerErrors/invalid_signature_base645564=== CONT TestExternalSignerErrors/invalid_JSON5565--- PASS: TestExternalSigner (0.01s)5566=== CONT TestExternalSignerErrors/null_signatures5567=== CONT TestExternalSignerErrors/program_error5568--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5569--- PASS: TestExternalSignerErrors (0.00s)5570 --- PASS: TestExternalSignerErrors/missing_signature_name (0.00s)5571 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.00s)5572 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.00s)5573 --- PASS: TestExternalSignerErrors/too_few_signatures (0.00s)5574 --- PASS: TestExternalSignerErrors/null (0.01s)5575 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5576 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5577 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5578 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5579 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5580 --- PASS: TestExternalSignerErrors/program_error (0.01s)5581 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5582 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5583 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5584 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5585 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5586--- PASS: TestExternalSignerCancellation (0.02s)5587PASS5588Running OpenBao signer tests...5589=== RUN TestSign5590=== PAUSE TestSign5591=== RUN TestSignErrors5592=== PAUSE TestSignErrors5593=== RUN TestServe5594=== PAUSE TestServe5595=== RUN TestServeRejectsGarbage5596=== PAUSE TestServeRejectsGarbage5597=== CONT TestSign5598=== CONT TestServe5599=== CONT TestServeRejectsGarbage5600=== CONT TestSignErrors5601=== RUN TestSignErrors/invalid_json5602--- PASS: TestServeRejectsGarbage (0.00s)5603=== PAUSE TestSignErrors/invalid_json5604=== RUN TestSignErrors/no_prefix5605=== PAUSE TestSignErrors/no_prefix5606=== RUN TestSignErrors/invalid_base645607=== PAUSE TestSignErrors/invalid_base645608--- PASS: TestServe (0.00s)5609=== RUN TestSignErrors/http_error5610=== PAUSE TestSignErrors/http_error5611=== CONT TestSignErrors/invalid_json5612=== CONT TestSignErrors/invalid_base645613=== CONT TestSignErrors/http_error5614=== CONT TestSignErrors/no_prefix5615--- PASS: TestSign (0.00s)5616--- PASS: TestSignErrors (0.00s)5617 --- PASS: TestSignErrors/invalid_json (0.00s)5618 --- PASS: TestSignErrors/no_prefix (0.00s)5619 --- PASS: TestSignErrors/http_error (0.00s)5620 --- PASS: TestSignErrors/invalid_base64 (0.00s)5621PASS