niks3-go-unit-tests log

succeeded `niks3-go-unit-tests` `checks.x86_64-linux.go-unit-tests` · build #498 · raw

1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushAnswersEveryLineWhenCanceled86=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestDoServerRequestAttachesToken116=== CONT TestPathInfoCACompatibility117=== CONT TestStreamPushAnswersEveryLineWhenCanceled118=== CONT TestFileTokenMissing119=== CONT TestFileTokenReadsAndCaches120=== CONT TestStaticToken121--- PASS: TestStaticToken (0.00s)122=== CONT TestParsePathInfoJSON123=== RUN TestParsePathInfoJSON/Nix_format124=== PAUSE TestParsePathInfoJSON/Nix_format125=== RUN TestParsePathInfoJSON/Lix_format126=== PAUSE TestParsePathInfoJSON/Lix_format127=== RUN TestParsePathInfoJSON/empty_input128=== PAUSE TestParsePathInfoJSON/empty_input129=== RUN TestParsePathInfoJSON/whitespace_only130=== PAUSE TestParsePathInfoJSON/whitespace_only131=== RUN TestParsePathInfoJSON/invalid_JSON132=== PAUSE TestParsePathInfoJSON/invalid_JSON133--- PASS: TestFileTokenMissing (0.00s)134=== CONT TestCaseHackSuffix135=== CONT TestSetClientTLSErrors136=== CONT TestEncodeNixBase32137=== RUN TestEncodeNixBase32/test_string_hash138--- PASS: TestFileTokenReadsAndCaches (0.00s)139=== CONT TestSetClientTLS140=== CONT TestSetClientTLSDoesNotMutateDefaultTransport141=== CONT TestFileTokenEmpty142=== CONT TestStreamPushRequestLine1432000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2144=== CONT TestScriptTokenEmptyCommand145--- PASS: TestScriptTokenEmptyCommand (0.00s)146=== CONT TestClientSignaturesByStorePath147=== CONT TestScriptTokenScriptFails148=== CONT TestStreamPushReportsSkippedPaths149=== CONT TestScriptTokenBadJSON150=== CONT TestScriptTokenEmptyToken151=== CONT TestStreamPushReportsSignatures152=== CONT TestScriptTokenCachesUntilRefresh153=== CONT TestDumpPathMatchesNix154=== CONT TestParsePathInfoJSONMultiplePaths155=== CONT TestScriptTokenNoExpiryRerunsEveryCall156=== CONT TestShellSplitErrors157=== RUN TestPathInfoCACompatibility/null_ca_field158=== CONT TestPartSizeForNAR159=== PAUSE TestEncodeNixBase32/test_string_hash160=== CONT TestStreamPushStopsWhenCanceled161=== CONT TestDumpPathWriterError1622026/11/12 00:22:57 ERROR Upload failed error=boom count=1163=== CONT TestUploadMultipart_PartsInParallel164=== RUN TestPartSizeForNAR/zero_stays_at_minimum165--- PASS: TestClientSignaturesByStorePath (0.00s)166--- PASS: TestStreamPushReportsSkippedPaths (0.00s)167=== RUN TestEncodeNixBase32/empty_input168=== PAUSE TestPathInfoCACompatibility/null_ca_field169=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum170=== RUN TestPathInfoCACompatibility/old_string_format_-_text171=== RUN TestPartSizeForNAR/small_stays_at_minimum172=== CONT TestDumpPathSingleFile173=== PAUSE TestEncodeNixBase32/empty_input174--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.00s)175=== CONT TestResolveStorePath176=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths177=== CONT TestFilterOversizedClosures178=== RUN TestFilterOversizedClosures/no_limit_keeps_everything179=== CONT TestUploadMultipart_ReturnsAfterDroppedBody180=== CONT TestStreamPushGivesUpOnDeadServer181=== PAUSE TestPartSizeForNAR/small_stays_at_minimum182=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text183=== CONT TestStreamPushIsolatesFailures184=== CONT TestStreamPushBatchesUnderLoad185--- PASS: TestShellSplitErrors (0.00s)186=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths187--- PASS: TestFileTokenEmpty (0.00s)1882026/11/12 00:22:57 ERROR Upload failed error=boom count=1189--- PASS: TestStreamPushStopsWhenCanceled (0.00s)1902026/11/12 00:22:57 ERROR Upload failed error="connection refused" count=20191--- PASS: TestScriptTokenScriptFails (0.00s)1922026/11/12 00:22:57 ERROR Server seems unavailable, giving up on batch untried=17193=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths194=== RUN TestSetClientTLSErrors/missing_cert_file195=== CONT TestShellSplit1962026/11/12 00:22:57 ERROR Upload failed error="bad path" count=3197=== CONT TestDoWithRetry_FinalResponseBodyReadable198=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths199=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica200=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun201=== CONT TestRegisterUploadedObject_OutlivesPush202=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything203=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped204=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped205=== CONT TestRetryTimeoutOutlastsMaxRetries206=== CONT TestRegisterUploadedObjectReusesConnections207=== CONT TestDoWithRetry_BodyReplayedViaGetBody2082026/11/12 00:22:57 WARN Rate limiter enabled after throttle name=server-test rate=52092026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:374072102026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:44749211=== RUN TestSetClientTLS/rejects_connection_without_client_cert2122026/11/12 00:22:57 WARN Rate limiter backed off name=server-test rate=5213=== CONT TestConvertHashToNix32214=== RUN TestConvertHashToNix32/SRI_format_to_Nix32215=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32216=== RUN TestConvertHashToNix32/already_Nix32_format217=== PAUSE TestConvertHashToNix32/already_Nix32_format218=== RUN TestConvertHashToNix32/invalid_format219=== PAUSE TestConvertHashToNix32/invalid_format220=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert221=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum222=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum2232026/11/12 00:22:57 INFO Garbage collection started224=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts225=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts226=== RUN TestPartSizeForNAR/1_TiB227=== PAUSE TestPartSizeForNAR/1_TiB228=== RUN TestPartSizeForNAR/5_TiB_S3_max_object229=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object230=== RUN TestPartSizeForNAR/capped_at_5_GiB231=== PAUSE TestPartSizeForNAR/capped_at_5_GiB232=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess2332026/11/12 00:22:57 INFO Garbage collection started2342026/11/12 00:22:57 WARN Rate limiter enabled after throttle name=server-test rate=52352026/11/12 00:22:57 WARN Rate limiter enabled after throttle name=server-test rate=5236=== CONT TestEncodeNixBase32WithRealHash237=== CONT TestParsePathInfoJSON/Nix_format238=== CONT TestRateLimiterFeedback239=== RUN TestRateLimiterFeedback/429_enables_limiter240=== PAUSE TestRateLimiterFeedback/429_enables_limiter241=== RUN TestRateLimiterFeedback/503_enables_limiter242=== PAUSE TestRateLimiterFeedback/503_enables_limiter243=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter244=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter245=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter246=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter247=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive248=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive249=== RUN TestPathInfoCACompatibility/new_structured_format_-_text250=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text251=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method252=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method2532026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:35487254=== CONT TestParsePathInfoJSON/whitespace_only255=== CONT TestParsePathInfoJSON/invalid_JSON256=== CONT TestEncodeNixBase32/test_string_hash257=== CONT TestEncodeNixBase32/empty_input258=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths259=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths260=== CONT TestConvertHashToNix32/SRI_format_to_Nix32261=== CONT TestConvertHashToNix32/invalid_format262=== CONT TestConvertHashToNix32/already_Nix32_format263=== CONT TestPartSizeForNAR/zero_stays_at_minimum264=== CONT TestRateLimiterFeedback/429_enables_limiter265--- PASS: TestScriptTokenBadJSON (0.00s)266--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)267--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)268--- PASS: TestStreamPushReportsSignatures (0.07s)269=== CONT TestParsePathInfoJSON/empty_input270=== CONT TestPartSizeForNAR/capped_at_5_GiB271=== CONT TestPartSizeForNAR/5_TiB_S3_max_object2722026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:44749273=== CONT TestPartSizeForNAR/1_TiB2742026/11/12 00:22:57 WARN Rate limiter backed off name=server-test rate=5275=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts2762026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:35487277=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum278=== CONT TestParsePathInfoJSON/Lix_format279=== CONT TestPathInfoCACompatibility/null_ca_field280=== CONT TestGetStorePathHash281=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer282=== CONT TestPathInfoHashCompatibility283=== CONT TestStreamPushReportsEveryPath284=== RUN TestFilterOversizedClosures/all_closures_skipped285=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry286=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA287--- PASS: TestDoServerRequestAttachesToken (0.07s)288=== RUN TestGetStorePathHash/valid_store_path289=== PAUSE TestGetStorePathHash/valid_store_path2902026/11/12 00:22:57 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0291=== RUN TestGetStorePathHash/basename_without_hyphen_should_error292=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error293=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error294=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error295=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error296=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error297=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA298=== RUN TestSetClientTLS/preserves_debug_logging_transport299=== PAUSE TestSetClientTLS/preserves_debug_logging_transport300=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false301=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false3022026/11/12 00:22:57 WARN Rate limiter enabled after throttle name=server-test rate=5303--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.00s)3042026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:38155305--- PASS: TestShellSplit (0.00s)306--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)307--- PASS: TestScriptTokenEmptyToken (0.07s)308--- PASS: TestStreamPushIsolatesFailures (0.00s)309--- PASS: TestCaseHackSuffix (0.07s)310--- PASS: TestResolveStorePath (0.07s)311--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)312--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)313=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive314=== CONT TestPathInfoCACompatibility/old_string_format_-_text3152026/11/12 00:22:57 WARN Rate limiter backed off name=server-test rate=5316=== CONT TestGetStorePathHash/valid_store_path317=== CONT TestSetClientTLS/rejects_connection_without_client_cert318=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error319=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error320=== PAUSE TestFilterOversizedClosures/all_closures_skipped321=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method322=== CONT TestSetClientTLS/preserves_debug_logging_transport323=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA324=== CONT TestGetStorePathHash/basename_without_hyphen_should_error325=== CONT TestFilterOversizedClosures/no_limit_keeps_everything326=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter3272026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:44749328--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)3292026/11/12 00:22:57 WARN Garbage collection finished on another replica. Its statistics are not available here330--- PASS: TestEncodeNixBase32WithRealHash (0.00s)331=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true332=== PAUSE TestSetClientTLSErrors/missing_cert_file333=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter334=== CONT TestPathInfoCACompatibility/new_structured_format_-_text335=== CONT TestRateLimiterFeedback/503_enables_limiter336=== CONT TestPartSizeForNAR/small_stays_at_minimum337=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)338=== CONT TestFilterOversizedClosures/all_closures_skipped3392026/11/12 00:22:57 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50340--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)341--- PASS: TestStreamPushReportsEveryPath (0.00s)342=== RUN TestSetClientTLSErrors/missing_key_file343=== PAUSE TestSetClientTLSErrors/missing_key_file344=== RUN TestSetClientTLSErrors/missing_ca_file345=== PAUSE TestSetClientTLSErrors/missing_ca_file346=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)3472026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:44749348=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon349=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon3502026/11/12 00:22:57 WARN Rate limiter enabled after throttle name=s3-test rate=5351=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI3522026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36841353=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI354=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512355=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha5123562026/11/12 00:22:57 WARN Rate limiter enabled after throttle name=server-test rate=5357=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3582026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:45749359--- PASS: TestGetStorePathHash (0.00s)360 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)361 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)362 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)363 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)3642026/11/12 00:22:57 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000365=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true366=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false3672026/11/12 00:22:57 WARN Rate limiter backed off name=server-test rate=5368=== RUN TestSetClientTLSErrors/invalid_ca_file3692026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:44749370=== PAUSE TestSetClientTLSErrors/invalid_ca_file371=== CONT TestSetClientTLSErrors/missing_cert_file372=== CONT TestSetClientTLSErrors/invalid_ca_file3732026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:44749374=== CONT TestSetClientTLSErrors/missing_ca_file375=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512376=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI377=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon378--- PASS: TestEncodeNixBase32 (0.01s)379 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)380 --- PASS: TestEncodeNixBase32/empty_input (0.00s)381--- PASS: TestParsePathInfoJSONMultiplePaths (0.07s)382 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)383 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)384--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.00s)385--- PASS: TestConvertHashToNix32 (0.00s)386 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)387 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)388 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)389--- PASS: TestParsePathInfoJSON (0.00s)390 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)391 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)392 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)393 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)394 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)395--- PASS: TestPartSizeForNAR (0.07s)396 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)397 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)398 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)399 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)400 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)401 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)402 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)403=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true404=== CONT TestSetClientTLSErrors/missing_key_file405--- PASS: TestPathInfoCACompatibility (0.08s)406 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)407 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)408 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)409 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)410 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)411=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)412--- PASS: TestFilterOversizedClosures (0.07s)413 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)414 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)415 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)416--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)417--- PASS: TestScriptTokenCachesUntilRefresh (0.08s)418--- PASS: TestRateLimiterFeedback (0.00s)419 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)420 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)421 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)422 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)4232026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:44749424--- PASS: TestPathInfoHashCompatibility (0.00s)425 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)426 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)427 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)428 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)429--- PASS: TestStreamPushBatchesUnderLoad (0.01s)430 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)431 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.01s)432--- PASS: TestSetClientTLSErrors (0.08s)433 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)434 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)435 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.01s)436 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)4372026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:447494382026/11/12 00:22:57 http: TLS handshake error from 127.0.0.1:36702: remote error: tls: bad certificate4392026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:44749440--- PASS: TestSetClientTLS (0.08s)441 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)442 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)443 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4442026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:44749445--- PASS: TestStreamPushRequestLine (0.09s)4462026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:447494472026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:447494482026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:447494492026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:447494502026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:447494512026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:447494522026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:447494532026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:44749454--- PASS: TestRegisterUploadedObjectReusesConnections (0.03s)455--- PASS: TestDumpPathSingleFile (0.10s)4562026/11/12 00:22:57 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:44749457--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.03s)458--- PASS: TestDumpPathWriterError (0.11s)459--- PASS: TestDumpPathMatchesNix (0.15s)4602026/11/12 00:22:57 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:39033/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:39033/api/uploads/complete4612026/11/12 00:22:57 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4622026/11/12 00:22:57 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:39033/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:39033/api/uploads/complete4632026/11/12 00:22:57 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4642026/11/12 00:22:57 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:39033/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:39033/api/uploads/complete4652026/11/12 00:22:57 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"466--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)467--- PASS: TestUploadMultipart_PartsInParallel (0.70s)468--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)469PASS470Running server tests...471The files belonging to this database system will be owned by user "nixbld".472This user must also own the server process.473474The database cluster will be initialized with locale "C".475The default database encoding has accordingly been set to "SQL_ASCII".476The default text search configuration will be set to "english".477478Data page checksums are enabled.479480creating directory /build/postgres3869257567/data ... ok481creating subdirectories ... ok482selecting dynamic shared memory implementation ... posix483selecting default "max_connections" ... 100484selecting default "shared_buffers" ... 128MB485selecting default time zone ... UTC486creating configuration files ... ok487running bootstrap script ... ok488performing post-bootstrap initialization ... ok489syncing data to disk ... ok490491initdb: warning: enabling "trust" authentication for local connections492initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.493494Success. You can now start the database server using:495496 pg_ctl -D /build/postgres3869257567/data -l logfile start497498/build/postgres3869257567:5432 - no response4992026-11-12 00:22:58.966 UTC [128] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit5002026-11-12 00:22:58.966 UTC [128] LOG: listening on Unix socket "/build/postgres3869257567/.s.PGSQL.5432"5012026-11-12 00:22:58.971 UTC [135] LOG: database system was shut down at 2026-11-12 00:22:58 UTC5022026-11-12 00:22:58.974 UTC [128] LOG: database system is ready to accept connections503/build/postgres3869257567:5432 - accepting connections504{"timestamp":"2026-11-12T00:22:59.270159605Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"6bece088-2641-4465-a09a-da1dfcf67805","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(386)"}505=== RUN TestAbortSucceeded506=== PAUSE TestAbortSucceeded507=== RUN TestService_AuthMiddleware508=== PAUSE TestService_AuthMiddleware509=== RUN TestService_AuthMiddleware_MTLSProxyHeader510=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader511=== RUN TestService_AuthMiddleware_MTLSBoundSubjects512=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects513=== RUN TestService_ReadAuthMiddleware514=== PAUSE TestService_ReadAuthMiddleware515=== RUN TestService_AuthMiddleware_OIDC516=== PAUSE TestService_AuthMiddleware_OIDC517=== RUN TestService_RequireScope_OIDC518=== PAUSE TestService_RequireScope_OIDC519=== RUN TestService_ReadScope_PublicByDefault520=== PAUSE TestService_ReadScope_PublicByDefault521=== RUN TestCacheConfigHandler522=== PAUSE TestCacheConfigHandler523=== RUN TestCacheStatsHandler524=== PAUSE TestCacheStatsHandler525=== RUN TestClientCADerivations526=== PAUSE TestClientCADerivations527=== RUN TestClientErrorHandling528=== PAUSE TestClientErrorHandling529=== RUN TestClientIntegration530=== PAUSE TestClientIntegration531=== RUN TestClientMultipleUploads532=== PAUSE TestClientMultipleUploads533=== RUN TestClientWithDependencies534=== PAUSE TestClientWithDependencies535=== RUN TestClientSharedPathCommittedMidPush536=== PAUSE TestClientSharedPathCommittedMidPush537=== RUN TestPinProtectsFromGC538=== PAUSE TestPinProtectsFromGC539=== RUN TestClientPushesUseOnePush540=== PAUSE TestClientPushesUseOnePush541=== RUN TestClientFallsBackToClosures542=== PAUSE TestClientFallsBackToClosures543=== RUN TestDeleteClosuresRacingPins544=== PAUSE TestDeleteClosuresRacingPins545=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock546=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock547=== RUN TestResolveDBConnectionString548=== PAUSE TestResolveDBConnectionString549=== RUN TestConnectWaitsForAPeerMigration550=== PAUSE TestConnectWaitsForAPeerMigration551=== RUN TestLeadElectsOneAndHandsOver552=== PAUSE TestLeadElectsOneAndHandsOver553=== RUN TestLeadIncumbentWinsAfterRestart5542026/11/12 00:22:59 INFO lead: acquired remote=192.0.2.1:1234 generation=15552026/11/12 00:23:00 INFO lead: released remote=192.0.2.1:1234 generation=15562026/11/12 00:23:00 INFO lead: acquired remote=192.0.2.1:1234 generation=25572026/11/12 00:23:01 INFO lead: released remote=192.0.2.1:1234 generation=2558--- PASS: TestLeadIncumbentWinsAfterRestart (1.88s)559=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent560=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent561=== RUN TestLeadIncumbentAnnouncesAtOnce562=== PAUSE TestLeadIncumbentAnnouncesAtOnce563=== RUN TestLeadGenerationGrowsWithEveryGrant564=== PAUSE TestLeadGenerationGrowsWithEveryGrant565=== RUN TestLeadEndsOnShutdown566=== PAUSE TestLeadEndsOnShutdown567=== RUN TestLeadEndsWhenItsConnectionHangs5682026/11/12 00:23:01 INFO lead: acquired remote=192.0.2.1:1234 generation=15692026-11-12 00:23:02.770 UTC [577] FATAL: terminating connection due to administrator command5702026/11/12 00:23:02 INFO lead: acquired remote=192.0.2.1:1234 generation=25712026/11/12 00:23:03 WARN lead: lock connection lost error="timeout: context deadline exceeded"5722026/11/12 00:23:03 INFO lead: released remote=192.0.2.1:1234 generation=15732026/11/12 00:23:04 INFO lead: released remote=192.0.2.1:1234 generation=2574--- PASS: TestLeadEndsWhenItsConnectionHangs (3.11s)575=== RUN TestGCAdvisoryLockBlocksConcurrentRun576=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun577=== RUN TestGCBugBareHashReferences578=== PAUSE TestGCBugBareHashReferences579=== RUN TestGCMetrics580=== PAUSE TestGCMetrics581=== RUN TestPushDedupSurvivesConcurrentGC582=== PAUSE TestPushDedupSurvivesConcurrentGC583=== RUN TestDeduplicatedObjectsRecordedAsPending584=== PAUSE TestDeduplicatedObjectsRecordedAsPending585=== RUN TestGCSweepSkipsPendingObjects586=== PAUSE TestGCSweepSkipsPendingObjects587=== RUN TestTombstonedObjectOfferedWithoutWaiting588=== PAUSE TestTombstonedObjectOfferedWithoutWaiting589=== RUN TestGCSweepDeliversEachKeyOnce590=== PAUSE TestGCSweepDeliversEachKeyOnce591=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection592=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection593=== RUN TestForceGCDuringPushOffersSweptObject594=== PAUSE TestForceGCDuringPushOffersSweptObject595=== RUN TestFailedS3DeleteKeepsTombstone596=== PAUSE TestFailedS3DeleteKeepsTombstone597=== RUN TestSweepSkipsRowsLockedByPush598=== PAUSE TestSweepSkipsRowsLockedByPush599=== RUN TestRegisterCompletedObjectNeedsPendingRow600=== PAUSE TestRegisterCompletedObjectNeedsPendingRow601=== RUN TestGetExistingObjectsLocksRows602=== PAUSE TestGetExistingObjectsLocksRows603=== RUN TestSweepRechecksPendingAfterLocking604=== PAUSE TestSweepRechecksPendingAfterLocking605=== RUN TestPushWaitsForSweepHoldingRow606=== PAUSE TestPushWaitsForSweepHoldingRow607=== RUN TestCommitRacingPendingCleanupKeepsObjects608=== PAUSE TestCommitRacingPendingCleanupKeepsObjects609=== RUN TestGCEndsOnShutdown610=== PAUSE TestGCEndsOnShutdown611=== RUN TestGCTaskStore_StartNew612=== PAUSE TestGCTaskStore_StartNew613=== RUN TestGCTaskStore_DeduplicateSameParams614=== PAUSE TestGCTaskStore_DeduplicateSameParams615=== RUN TestGCTaskStore_ConflictDifferentParams616=== PAUSE TestGCTaskStore_ConflictDifferentParams617=== RUN TestGCTaskStore_GetEmpty618=== PAUSE TestGCTaskStore_GetEmpty619=== RUN TestGCTaskStore_GetReturnsLatest620=== PAUSE TestGCTaskStore_GetReturnsLatest621=== RUN TestGCTaskStore_CompletedAllowsNewTask622=== PAUSE TestGCTaskStore_CompletedAllowsNewTask623=== RUN TestGCTaskStore_PhaseUpdates624=== PAUSE TestGCTaskStore_PhaseUpdates625=== RUN TestGCTaskStore_Fail626=== PAUSE TestGCTaskStore_Fail627=== RUN TestGracefulShutdownDrainsInflight628=== PAUSE TestGracefulShutdownDrainsInflight629=== RUN TestService_healthCheckHandler630=== PAUSE TestService_healthCheckHandler631=== RUN TestService_readinessHandler632=== PAUSE TestService_readinessHandler633=== RUN TestGenerateLandingPage634=== PAUSE TestGenerateLandingPage635=== RUN TestEnsureAbortUploadsRule636=== PAUSE TestEnsureAbortUploadsRule637=== RUN TestCacheConfigHandlerMaxNarSize638=== PAUSE TestCacheConfigHandlerMaxNarSize639=== RUN TestCreatePendingClosureRejectsOversizedNAR640=== PAUSE TestCreatePendingClosureRejectsOversizedNAR641=== RUN TestNARDeduplicationMetadataUploadBug642=== PAUSE TestNARDeduplicationMetadataUploadBug643=== RUN TestMetricsInventory644=== PAUSE TestMetricsInventory645=== RUN TestMetricsCommit646=== PAUSE TestMetricsCommit647=== RUN TestService_NativeMTLS648=== PAUSE TestService_NativeMTLS649=== RUN TestServerTLSConfig650=== PAUSE TestServerTLSConfig651=== RUN TestMultipartCleanup652=== PAUSE TestMultipartCleanup653=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded654=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded655=== RUN TestPendingCleanupUsesOneCutoff656=== PAUSE TestPendingCleanupUsesOneCutoff657=== RUN TestObjectStatsTrigger658=== PAUSE TestObjectStatsTrigger659=== RUN TestObjectStatsUpdatedOncePerCommit660=== PAUSE TestObjectStatsUpdatedOncePerCommit661=== RUN TestOrphanedObjectsGC662=== PAUSE TestOrphanedObjectsGC663=== RUN TestOrphanedObjectsGCStressTest664=== PAUSE TestOrphanedObjectsGCStressTest665=== RUN TestCreatePin_ReservedPins666=== PAUSE TestCreatePin_ReservedPins667=== RUN TestPresentReportsOnlyClosureRoots668=== PAUSE TestPresentReportsOnlyClosureRoots669=== RUN TestPresentRefreshesOnlyStaleClosures670=== PAUSE TestPresentRefreshesOnlyStaleClosures671=== RUN TestPresentNotReportedWhileGCDeletesClosure672=== PAUSE TestPresentNotReportedWhileGCDeletesClosure673=== RUN TestParseSingleRange674=== PAUSE TestParseSingleRange675=== RUN TestProxyHeadersOnlyTrustedOnSocket6762026/11/12 00:23:04 INFO Starting HTTP server address=127.0.0.1:348456772026/11/12 00:23:04 INFO Starting HTTP server address=proxy.sock6782026/11/12 00:23:04 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6792026/11/12 00:23:04 INFO Shutdown signal received, draining in-flight requests timeout=10s680--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)681=== RUN TestIsValidCachePath682=== PAUSE TestIsValidCachePath683=== RUN TestReadProxyNarinfo684=== PAUSE TestReadProxyNarinfo685=== RUN TestReadProxyNarinfoAlreadyDecompressed686=== PAUSE TestReadProxyNarinfoAlreadyDecompressed687=== RUN TestReadProxyContentNegotiation688=== PAUSE TestReadProxyContentNegotiation689=== RUN TestReadProxyNarStreaming690=== PAUSE TestReadProxyNarStreaming691=== RUN TestReadProxy404692=== PAUSE TestReadProxy404693=== RUN TestReadProxyInvalidPath694=== PAUSE TestReadProxyInvalidPath695=== RUN TestReadProxyHead696=== PAUSE TestReadProxyHead697=== RUN TestReadProxyOutlastsServerWriteTimeout698=== PAUSE TestReadProxyOutlastsServerWriteTimeout699=== RUN TestReadProxyConditionalGet700=== PAUSE TestReadProxyConditionalGet701=== RUN TestReadProxyRootRedirectsToIndexHTML702=== PAUSE TestReadProxyRootRedirectsToIndexHTML703=== RUN TestReadProxyDisabled704=== PAUSE TestReadProxyDisabled705=== RUN TestReadRedirectNar706=== PAUSE TestReadRedirectNar707=== RUN TestReadRedirectKeepsNarinfoProxied708=== PAUSE TestReadRedirectKeepsNarinfoProxied709=== RUN TestReadProxyRangeRequest710=== PAUSE TestReadProxyRangeRequest711=== RUN TestReadRedirectUsesPublicS3URL712=== PAUSE TestReadRedirectUsesPublicS3URL713=== RUN TestPush_OverlappingRootsStoreOneRowPerKey714=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey715=== RUN TestPush_CompleteCommitsEveryRoot716=== PAUSE TestPush_CompleteCommitsEveryRoot717=== RUN TestPush_RepeatedRootCommitsOnce718=== PAUSE TestPush_RepeatedRootCommitsOnce719=== RUN TestPush_CompleteWaitsForCleanupOfThePush720=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush721=== RUN TestPush_RootsAreLockedInByteOrder722=== PAUSE TestPush_RootsAreLockedInByteOrder723=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit724=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit725=== RUN TestPush_RejectsBadRequests726=== PAUSE TestPush_RejectsBadRequests727=== RUN TestPush_SignsNarinfosOfItsPendingObjects728=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects729=== RUN TestSharedNarBothUploadsComplete730=== PAUSE TestSharedNarBothUploadsComplete731=== RUN TestCompleteMultipartUpload_ErrorButObjectExists732=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists733=== RUN TestCompletedNarNotReofferedAcrossClosures734=== PAUSE TestCompletedNarNotReofferedAcrossClosures735=== RUN TestPresignedUploadRegisteredBeforeCommit736=== PAUSE TestPresignedUploadRegisteredBeforeCommit737=== RUN TestService_Rustfstest738=== PAUSE TestService_Rustfstest739=== RUN TestParseSize740=== PAUSE TestParseSize741=== RUN TestSkippedUploadsHandler742=== PAUSE TestSkippedUploadsHandler743=== RUN TestSystemdListenerNotActivated744--- PASS: TestSystemdListenerNotActivated (0.00s)745=== RUN TestWatchdogBeatsWhenHealthy746--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)747=== RUN TestWatchdogSkipsWhenUnhealthy7482026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7492026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7532026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7542026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7552026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7562026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/12 00:23:04 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"758--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)759=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle760=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle761=== RUN TestProxyWriteTimeout762=== PAUSE TestProxyWriteTimeout763=== RUN TestPendingClosureWriteTimeout764=== PAUSE TestPendingClosureWriteTimeout765=== RUN TestIsValidUploadKey766=== PAUSE TestIsValidUploadKey767=== RUN TestUploadHandlersRejectInvalidKeys768=== PAUSE TestUploadHandlersRejectInvalidKeys769=== RUN TestUploadHandlersRejectOversizedBody770=== PAUSE TestUploadHandlersRejectOversizedBody771=== RUN TestService_cleanupPendingClosuresHandler772=== PAUSE TestService_cleanupPendingClosuresHandler773=== RUN TestService_createPendingClosureHandler774=== PAUSE TestService_createPendingClosureHandler775=== RUN TestService_verifyS3Integrity776=== PAUSE TestService_verifyS3Integrity777=== RUN TestService_verifyS3ReuploadsStaleNarinfo778=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo779=== RUN TestCompleteMultipartUnregistered780=== PAUSE TestCompleteMultipartUnregistered781=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT782=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT783=== CONT TestAbortSucceeded784=== CONT TestParseSize785--- PASS: TestAbortSucceeded (0.00s)786--- PASS: TestParseSize (0.00s)787=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT788=== CONT TestCompleteMultipartUnregistered789=== CONT TestService_verifyS3ReuploadsStaleNarinfo790=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed791=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed792=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed793=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed794=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost795=== CONT TestService_verifyS3Integrity796=== CONT TestService_createPendingClosureHandler797=== CONT TestService_cleanupPendingClosuresHandler798=== CONT TestUploadHandlersRejectOversizedBody799=== CONT TestUploadHandlersRejectInvalidKeys800=== CONT TestIsValidUploadKey801=== CONT TestPendingClosureWriteTimeout802=== CONT TestProxyWriteTimeout803=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle804=== CONT TestSkippedUploadsHandler805=== CONT TestNARDeduplicationMetadataUploadBug806=== CONT TestCreatePendingClosureRejectsOversizedNAR807=== CONT TestCacheConfigHandlerMaxNarSize808=== CONT TestEnsureAbortUploadsRule809=== CONT TestGenerateLandingPage810=== CONT TestService_readinessHandler811=== CONT TestService_healthCheckHandler812=== CONT TestGracefulShutdownDrainsInflight813=== CONT TestGCTaskStore_Fail814=== CONT TestGCTaskStore_PhaseUpdates815=== CONT TestGCTaskStore_CompletedAllowsNewTask816=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost817=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info818=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info819=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal820=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal821=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key822=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key823=== RUN TestProxyWriteTimeout/narinfo824=== PAUSE TestProxyWriteTimeout/narinfo8252026/11/12 00:23:04 INFO Received uploads request method=POST path=/api/pending_closures826=== RUN TestIsValidUploadKey/narinfo8272026/11/12 00:23:04 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000828=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key829=== RUN TestProxyWriteTimeout/1_GiB_nar830--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)831=== CONT TestGCTaskStore_GetReturnsLatest832=== RUN TestPendingClosureWriteTimeout/empty833=== CONT TestGCTaskStore_GetEmpty834--- PASS: TestGCTaskStore_Fail (0.00s)835--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)836--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)837--- PASS: TestGCTaskStore_GetEmpty (0.00s)838--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)839--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)840=== CONT TestCommitRacingPendingCleanupKeepsObjects841=== PAUSE TestIsValidUploadKey/narinfo8422026/11/12 00:23:04 INFO Starting HTTP server address=127.0.0.1:38985843=== RUN TestIsValidUploadKey/nar_zst844=== CONT TestGCEndsOnShutdown845=== RUN TestGCEndsOnShutdown/before_the_run846=== CONT TestGCTaskStore_ConflictDifferentParams8472026/11/12 00:23:04 INFO Shutdown signal received, draining in-flight requests timeout=10s848--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)849=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed850=== PAUSE TestGCEndsOnShutdown/before_the_run851=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key852=== PAUSE TestIsValidUploadKey/nar_zst853=== RUN TestIsValidUploadKey/nar_xz854=== PAUSE TestIsValidUploadKey/nar_xz855=== RUN TestIsValidUploadKey/nar_plain856=== PAUSE TestIsValidUploadKey/nar_plain857=== PAUSE TestProxyWriteTimeout/1_GiB_nar858=== CONT TestGCTaskStore_DeduplicateSameParams859=== PAUSE TestPendingClosureWriteTimeout/empty860=== CONT TestSweepRechecksPendingAfterLocking861=== CONT TestReadProxyInvalidPath862=== CONT TestService_Rustfstest863=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed864=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete865=== CONT TestPushWaitsForSweepHoldingRow866=== RUN TestIsValidUploadKey/listing867=== PAUSE TestIsValidUploadKey/listing868=== RUN TestIsValidUploadKey/build_log869=== PAUSE TestIsValidUploadKey/build_log870=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete871--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)872=== RUN TestPendingClosureWriteTimeout/400_objects873=== PAUSE TestPendingClosureWriteTimeout/400_objects874=== RUN TestPendingClosureWriteTimeout/670k_objects875=== PAUSE TestPendingClosureWriteTimeout/670k_objects876=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline877=== CONT TestGCTaskStore_StartNew878=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed879=== RUN TestProxyWriteTimeout/10_GiB_nar880=== RUN TestIsValidUploadKey/build_log_home-manager_file881=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark882=== CONT TestGetExistingObjectsLocksRows883=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline884=== PAUSE TestProxyWriteTimeout/10_GiB_nar885=== RUN TestProxyWriteTimeout/unknown_size886=== PAUSE TestProxyWriteTimeout/unknown_size887=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark888=== CONT TestPresignedUploadRegisteredBeforeCommit889--- PASS: TestGenerateLandingPage (0.00s)890--- PASS: TestGCTaskStore_StartNew (0.00s)891=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed892=== CONT TestRegisterCompletedObjectNeedsPendingRow893=== PAUSE TestIsValidUploadKey/build_log_home-manager_file894=== RUN TestIsValidUploadKey/build_log_plus_in_name895=== CONT TestSweepSkipsRowsLockedByPush896=== CONT TestCompletedNarNotReofferedAcrossClosures897=== PAUSE TestIsValidUploadKey/build_log_plus_in_name898=== RUN TestIsValidUploadKey/build_log_question_mark899=== PAUSE TestIsValidUploadKey/build_log_question_mark900=== RUN TestIsValidUploadKey/build_log_equals901=== PAUSE TestIsValidUploadKey/build_log_equals902=== RUN TestIsValidUploadKey/realisation903=== PAUSE TestIsValidUploadKey/realisation904=== RUN TestIsValidUploadKey/realisation_plus_in_output905=== PAUSE TestIsValidUploadKey/realisation_plus_in_output906=== RUN TestIsValidUploadKey/nix-cache-info907=== PAUSE TestIsValidUploadKey/nix-cache-info908=== RUN TestIsValidUploadKey/index.html909=== PAUSE TestIsValidUploadKey/index.html910=== RUN TestIsValidUploadKey/narinfo_key,_nar_type911=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type912=== RUN TestIsValidUploadKey/nar_key,_narinfo_type913=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type914=== RUN TestIsValidUploadKey/listing_key,_narinfo_type915=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type916=== RUN TestIsValidUploadKey/traversal917=== PAUSE TestIsValidUploadKey/traversal918=== RUN TestIsValidUploadKey/traversal_nar919=== PAUSE TestIsValidUploadKey/traversal_nar920=== RUN TestIsValidUploadKey/absolute921=== PAUSE TestIsValidUploadKey/absolute922=== RUN TestIsValidUploadKey/empty_key923=== PAUSE TestIsValidUploadKey/empty_key924=== RUN TestIsValidUploadKey/unknown_type925=== PAUSE TestIsValidUploadKey/unknown_type926=== CONT TestCompleteMultipartUpload_ErrorButObjectExists927--- PASS: TestSkippedUploadsHandler (0.10s)928=== CONT TestFailedS3DeleteKeepsTombstone929--- PASS: TestGracefulShutdownDrainsInflight (0.07s)930=== CONT TestForceGCDuringPushOffersSweptObject931=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows932=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows933=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check934=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check935=== CONT TestSharedNarBothUploadsComplete936=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure937=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure938=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart939=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart940=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts941=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts942=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection9432026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures9442026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures9452026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures946=== NAME TestNARDeduplicationMetadataUploadBug947 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug36001168/001/store/11zprs4kfzv7aaq846j97l71m585s4yx-file1.txt9482026/11/12 00:23:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9492026/11/12 00:23:05 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst950--- PASS: TestCompleteMultipartUnregistered (0.41s)951=== CONT TestGCSweepDeliversEachKeyOnce9522026/11/12 00:23:05 INFO Received cleanup request method=DELETE path=/api/pending_closures9532026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=09542026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures9552026/11/12 00:23:05 INFO Received cleanup request method=DELETE path=/api/pending_closures9562026/11/12 00:23:05 INFO Aborted multipart uploads count=1 kept=09572026/11/12 00:23:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9582026-11-12 00:23:05.177 UTC [651] ERROR: Closure does not exist: id=19592026-11-12 00:23:05.177 UTC [651] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE9602026-11-12 00:23:05.177 UTC [651] STATEMENT: -- name: CommitPendingClosure :one961 SELECT commit_pending_closure($1::bigint)::bigint962 963--- PASS: TestService_cleanupPendingClosuresHandler (0.46s)964=== CONT TestPush_SignsNarinfosOfItsPendingObjects9652026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures966--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.48s)967=== CONT TestTombstonedObjectOfferedWithoutWaiting9682026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures9692026/11/12 00:23:05 INFO Received push request method=POST path=/api/pushes9702026/11/12 00:23:05 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)9712026/11/12 00:23:05 INFO Uploading 11zprs4kfzv7aaq846j97l71m585s4yx-file1.txt (160B)9722026/11/12 00:23:05 WARN Failed to register uploaded object key=11zprs4kfzv7aaq846j97l71m585s4yx.ls error="server returned 404: 404 page not found\n"9732026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9742026/11/12 00:23:05 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"9752026/11/12 00:23:05 INFO Signed narinfos id=1 count=19762026/11/12 00:23:05 INFO Uploading 1 narinfos9772026/11/12 00:23:05 INFO Received complete push request method=POST path=/api/pushes/1/complete9782026/11/12 00:23:05 WARN Failed to register uploaded object key=11zprs4kfzv7aaq846j97l71m585s4yx.narinfo error="server returned 404: 404 page not found\n"9792026/11/12 00:23:05 WARN readiness check failed error="closed pool"980--- PASS: TestService_readinessHandler (0.51s)981=== CONT TestPush_RejectsBadRequests9822026/11/12 00:23:05 INFO Completed push id=1 objects=3 duration=8.551336ms9832026/11/12 00:23:05 INFO Upload complete. (71ms)984=== NAME TestNARDeduplicationMetadataUploadBug985 metadata_upload_test.go:54: Retrieved narinfo from S3:986 StorePath: /build/TestNARDeduplicationMetadataUploadBug36001168/001/store/11zprs4kfzv7aaq846j97l71m585s4yx-file1.txt987 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst988 Compression: zstd989 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf990 NarSize: 160991 References: 992 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf993 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)994 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):995 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}9962026/11/12 00:23:05 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket11 days=2997--- PASS: TestEnsureAbortUploadsRule (0.56s)998=== CONT TestGCSweepSkipsPendingObjects999=== NAME TestNARDeduplicationMetadataUploadBug1000 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug36001168/001/store/p6awb1pqprrqf8fnfrwq7gh89nmvlxj3-file2.txt1001--- PASS: TestService_healthCheckHandler (0.51s)1002=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit10032026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=010042026/11/12 00:23:05 INFO Received push request method=POST path=/api/pushes10052026/11/12 00:23:05 WARN Force mode enabled - objects will be deleted immediately without grace period10062026/11/12 00:23:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010072026/11/12 00:23:05 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)10082026/11/12 00:23:05 INFO Vacuumed table table=pending_closures10092026/11/12 00:23:05 INFO Vacuumed table table=pending_objects10102026/11/12 00:23:05 INFO Vacuumed table table=multipart_uploads10112026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign10122026/11/12 00:23:05 INFO Vacuumed table table=closures10132026/11/12 00:23:05 WARN Failed to register uploaded object key=p6awb1pqprrqf8fnfrwq7gh89nmvlxj3.ls error="server returned 404: 404 page not found\n"10142026/11/12 00:23:05 INFO Signed narinfos id=2 count=110152026/11/12 00:23:05 INFO Vacuumed table table=objects10162026/11/12 00:23:05 INFO Uploading 1 narinfos10172026/11/12 00:23:05 INFO Received complete push request method=POST path=/api/pushes/2/complete10182026/11/12 00:23:05 WARN Failed to register uploaded object key=p6awb1pqprrqf8fnfrwq7gh89nmvlxj3.narinfo error="server returned 404: 404 page not found\n"1019--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.56s)1020=== CONT TestPinProtectsFromGC10212026/11/12 00:23:05 INFO Completed push id=2 objects=3 duration=2.479799ms10222026/11/12 00:23:05 INFO Upload complete. (59ms)10232026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=01024=== NAME TestNARDeduplicationMetadataUploadBug1025 metadata_upload_test.go:76: Retrieved narinfo from S3:1026 StorePath: /build/TestNARDeduplicationMetadataUploadBug36001168/001/store/p6awb1pqprrqf8fnfrwq7gh89nmvlxj3-file2.txt1027 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1028 Compression: zstd1029 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1030 NarSize: 1601031 References: 1032 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1033 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1034--- PASS: TestService_Rustfstest (0.56s)1035=== CONT TestClientSharedPathCommittedMidPush1036=== NAME TestNARDeduplicationMetadataUploadBug1037 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1038 {"version":1,"root":{"type":"regular","size":44}}10392026/11/12 00:23:05 WARN Force mode enabled - objects will be deleted immediately without grace period1040--- PASS: TestNARDeduplicationMetadataUploadBug (0.66s)1041=== CONT TestPush_RootsAreLockedInByteOrder10422026/11/12 00:23:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010432026/11/12 00:23:05 INFO Vacuumed table table=pending_closures10442026/11/12 00:23:05 INFO Vacuumed table table=pending_objects10452026/11/12 00:23:05 INFO Vacuumed table table=multipart_uploads10462026/11/12 00:23:05 INFO Vacuumed table table=closures10472026/11/12 00:23:05 INFO Vacuumed table table=objects1048--- PASS: TestSweepRechecksPendingAfterLocking (0.59s)1049=== CONT TestClientWithDependencies10502026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures1051--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.63s)1052=== CONT TestPush_CompleteWaitsForCleanupOfThePush10532026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10542026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10552026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10562026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10572026/11/12 00:23:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1058--- PASS: TestReadProxyInvalidPath (0.74s)1059=== CONT TestPush_RepeatedRootCommitsOnce10602026/11/12 00:23:05 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MjMwZTRiYjAtYzdlMy00NDAyLWI4YjYtZjYzMDVjZjMxZTkwLjE1ZWNkYWUxLTY5NTAtNGIxYi1iMDI2LTAzMzY1ZTQwNDFhNXgxNzk0NDQyOTg1MTEyNzI3MDgw parts=1010612026/11/12 00:23:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10622026/11/12 00:23:05 INFO Completed upload id=1 objects=2 duration=5.981578ms10632026/11/12 00:23:05 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010642026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10652026/11/12 00:23:05 INFO Starting cleanup of old closures method=DELETE path=/api/closures10662026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=01067--- PASS: TestGetExistingObjectsLocksRows (0.79s)1068=== CONT TestClientMultipleUploads10692026/11/12 00:23:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010702026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10712026/11/12 00:23:05 INFO Vacuumed table table=pending_closures10722026/11/12 00:23:05 INFO Vacuumed table table=pending_objects10732026/11/12 00:23:05 INFO Vacuumed table table=multipart_uploads10742026/11/12 00:23:05 INFO Vacuumed table table=closures10752026/11/12 00:23:05 INFO Vacuumed table table=objects10762026/11/12 00:23:05 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010772026/11/12 00:23:05 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10782026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures1079--- PASS: TestService_createPendingClosureHandler (0.92s)1080=== CONT TestPush_CompleteCommitsEveryRoot1081--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.83s)1082=== CONT TestClientIntegration10832026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10842026/11/12 00:23:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10852026/11/12 00:23:05 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MjMwZTRiYjAtYzdlMy00NDAyLWI4YjYtZjYzMDVjZjMxZTkwLjczNmVlYjMyLWMwNzAtNGRkZi05MjE5LTIzOGJmOTM5YTZhOXgxNzk0NDQyOTg1MjE1OTg0MDAx parts=1010862026/11/12 00:23:05 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10872026/11/12 00:23:05 INFO Completed upload id=1 objects=2 duration=3.013279ms10882026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10892026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10902026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures10912026/11/12 00:23:05 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10922026/11/12 00:23:05 WARN Found objects in DB but missing from S3, will re-upload count=11093--- PASS: TestService_verifyS3Integrity (0.96s)1094=== CONT TestPush_OverlappingRootsStoreOneRowPerKey1095--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.82s)1096=== CONT TestClientErrorHandling1097=== RUN TestClientErrorHandling/InvalidStorePath1098=== PAUSE TestClientErrorHandling/InvalidStorePath1099=== RUN TestClientErrorHandling/InvalidAuthToken1100=== PAUSE TestClientErrorHandling/InvalidAuthToken1101=== RUN TestClientErrorHandling/ServerNotAvailable1102=== PAUSE TestClientErrorHandling/ServerNotAvailable1103=== CONT TestReadRedirectUsesPublicS3URL11042026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures11052026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=011062026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=011072026/11/12 00:23:05 WARN Force mode enabled - objects will be deleted immediately without grace period11082026/11/12 00:23:05 WARN Force mode enabled - objects will be deleted immediately without grace period11092026/11/12 00:23:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011102026/11/12 00:23:05 INFO Vacuumed table table=pending_closures11112026/11/12 00:23:05 INFO Vacuumed table table=pending_objects11122026/11/12 00:23:05 INFO Vacuumed table table=multipart_uploads11132026/11/12 00:23:05 INFO Vacuumed table table=closures11142026/11/12 00:23:05 INFO Vacuumed table table=objects11152026/11/12 00:23:05 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."11162026/11/12 00:23:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1117--- PASS: TestFailedS3DeleteKeepsTombstone (0.95s)1118=== CONT TestClientCADerivations11192026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=011202026/11/12 00:23:05 WARN Force mode enabled - objects will be deleted immediately without grace period11212026/11/12 00:23:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11222026/11/12 00:23:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11232026/11/12 00:23:05 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MjMwZTRiYjAtYzdlMy00NDAyLWI4YjYtZjYzMDVjZjMxZTkwLmE3MzU1YTNkLWZmYTYtNDdlMC05NTM2LTBmMWU1NjdjYzk2NHgxNzk0NDQyOTg1NzUyODczMjc211242026/11/12 00:23:05 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MjMwZTRiYjAtYzdlMy00NDAyLWI4YjYtZjYzMDVjZjMxZTkwLmE3MzU1YTNkLWZmYTYtNDdlMC05NTM2LTBmMWU1NjdjYzk2NHgxNzk0NDQyOTg1NzUyODczMjc2 parts=11125--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.99s)1126=== CONT TestReadProxyRangeRequest11272026/11/12 00:23:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=011282026/11/12 00:23:05 INFO Vacuumed table table=pending_closures11292026/11/12 00:23:05 INFO Received push request method=POST path=/api/pushes11302026/11/12 00:23:05 INFO Vacuumed table table=pending_objects11312026/11/12 00:23:05 INFO Vacuumed table table=multipart_uploads11322026/11/12 00:23:05 INFO Vacuumed table table=closures11332026/11/12 00:23:05 INFO Vacuumed table table=objects1134--- PASS: TestSweepSkipsRowsLockedByPush (1.00s)1135=== CONT TestCacheStatsHandler11362026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=011372026/11/12 00:23:05 WARN Force mode enabled - objects will be deleted immediately without grace period11382026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11392026/11/12 00:23:05 INFO Signed narinfos id=1 count=111402026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11412026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11422026/11/12 00:23:05 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"11432026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11442026/11/12 00:23:05 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"11452026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11462026/11/12 00:23:05 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=111472026/11/12 00:23:05 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11482026/11/12 00:23:05 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1149--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.65s)1150=== CONT TestCacheConfigHandler1151=== RUN TestCacheConfigHandler/full_config,_no_issuer1152=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1153=== RUN TestCacheConfigHandler/no_cache_url_configured1154=== PAUSE TestCacheConfigHandler/no_cache_url_configured1155=== RUN TestCacheConfigHandler/no_signing_keys1156=== PAUSE TestCacheConfigHandler/no_signing_keys1157=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1158=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1159=== CONT TestService_ReadScope_PublicByDefault11602026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures1161--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.67s)1162=== CONT TestReadRedirectKeepsNarinfoProxied1163=== RUN TestPush_RejectsBadRequests/no_roots1164=== PAUSE TestPush_RejectsBadRequests/no_roots1165=== RUN TestPush_RejectsBadRequests/no_objects1166=== PAUSE TestPush_RejectsBadRequests/no_objects1167=== RUN TestPush_RejectsBadRequests/bad_root1168=== PAUSE TestPush_RejectsBadRequests/bad_root1169=== RUN TestPush_RejectsBadRequests/root_not_in_objects1170=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1171=== CONT TestClientPushesUseOnePush11722026/11/12 00:23:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=011732026/11/12 00:23:05 INFO Vacuumed table table=pending_closures11742026/11/12 00:23:05 INFO Vacuumed table table=pending_objects11752026/11/12 00:23:05 INFO Vacuumed table table=multipart_uploads11762026/11/12 00:23:05 INFO Vacuumed table table=closures11772026/11/12 00:23:05 INFO Vacuumed table table=objects1178--- PASS: TestGCSweepDeliversEachKeyOnce (0.77s)1179=== CONT TestService_RequireScope_OIDC1180--- PASS: TestPushWaitsForSweepHoldingRow (1.10s)1181=== CONT TestReadRedirectNar11822026/11/12 00:23:05 INFO Received uploads request method=POST path=/api/pending_closures11832026/11/12 00:23:05 INFO Received push request method=POST path=/api/pushes11842026/11/12 00:23:05 INFO Aborted multipart uploads count=0 kept=011852026/11/12 00:23:05 WARN Force mode enabled - objects will be deleted immediately without grace period11862026/11/12 00:23:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011872026/11/12 00:23:05 INFO Vacuumed table table=pending_closures11882026/11/12 00:23:05 INFO Received complete push request method=POST path=/api/pushes/1/complete11892026/11/12 00:23:05 INFO Vacuumed table table=pending_objects11902026/11/12 00:23:05 INFO Vacuumed table table=multipart_uploads11912026/11/12 00:23:05 INFO Vacuumed table table=closures11922026/11/12 00:23:05 INFO Vacuumed table table=objects11932026/11/12 00:23:05 INFO Completed push id=1 objects=4 duration=8.682226ms11942026/11/12 00:23:05 INFO Received push request method=POST path=/api/pushes1195--- PASS: TestGCSweepSkipsPendingObjects (0.71s)1196=== CONT TestService_AuthMiddleware_OIDC11972026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes11982026/11/12 00:23:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38307/oidc11992026/11/12 00:23:06 INFO Aborted multipart uploads count=0 kept=012002026/11/12 00:23:06 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12012026/11/12 00:23:06 WARN Force mode enabled - objects will be deleted immediately without grace period12022026-11-12 00:23:06.052 UTC [904] ERROR: could not obtain lock on row in relation "closures"12032026-11-12 00:23:06.052 UTC [904] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12042026-11-12 00:23:06.053 UTC [904] ERROR: could not obtain lock on row in relation "closures"12052026-11-12 00:23:06.053 UTC [904] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12062026-11-12 00:23:06.053 UTC [904] ERROR: could not obtain lock on row in relation "closures"12072026-11-12 00:23:06.053 UTC [904] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12082026-11-12 00:23:06.053 UTC [904] ERROR: could not obtain lock on row in relation "closures"12092026-11-12 00:23:06.053 UTC [904] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT12102026-11-12 00:23:06.054 UTC [904] ERROR: could not obtain lock on row in relation "closures"12112026-11-12 00:23:06.054 UTC [904] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1212--- PASS: TestPush_RootsAreLockedInByteOrder (0.68s)1213=== CONT TestReadProxyDisabled12142026/11/12 00:23:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=01215=== NAME TestPinProtectsFromGC1216 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC2515212105/001/store/2szgzild6pladcajgmv935siwjr9zjcp-pinned-file.txt1217 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC2515212105/001/store/myj4nnpsjg3lvn9p2gwjiw0pk57q3dqz-unpinned-file.txt12182026/11/12 00:23:06 INFO Vacuumed table table=pending_closures12192026/11/12 00:23:06 INFO Vacuumed table table=pending_objects12202026/11/12 00:23:06 INFO Vacuumed table table=multipart_uploads12212026/11/12 00:23:06 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MjMwZTRiYjAtYzdlMy00NDAyLWI4YjYtZjYzMDVjZjMxZTkwLjkyYjcxYTg1LWJmNTItNDg3MS1iMzE3LWYzMmQ2NTc4NjE2NngxNzk0NDQyOTg1NDcyMjUzOTgz parts=1212222026/11/12 00:23:06 INFO Vacuumed table table=closures12232026/11/12 00:23:06 INFO Vacuumed table table=objects12242026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/2/complete12252026/11/12 00:23:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35369/oidc12262026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes12272026/11/12 00:23:06 INFO Completed push id=2 objects=4 duration=17.787984ms1228--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.78s)1229=== CONT TestService_ReadAuthMiddleware12302026/11/12 00:23:06 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12312026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes12322026/11/12 00:23:06 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=MjMwZTRiYjAtYzdlMy00NDAyLWI4YjYtZjYzMDVjZjMxZTkwLjEyNWQyOGYzLTFhZDEtNDU1ZC05MWFkLWM1ODQ1Yzg2NGRkZngxNzk0NDQyOTg1NTM0MDY1NjQw parts=1212332026/11/12 00:23:06 INFO Received uploads request method=POST path=/api/pending_closures12342026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes1235--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.33s)1236=== CONT TestReadProxyRootRedirectsToIndexHTML12372026/11/12 00:23:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12382026/11/12 00:23:06 INFO Uploading 2szgzild6pladcajgmv935siwjr9zjcp-pinned-file.txt (128B)12392026/11/12 00:23:06 WARN Failed to register uploaded object key=2szgzild6pladcajgmv935siwjr9zjcp.ls error="server returned 404: 404 page not found\n"12402026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete1241=== NAME TestClientWithDependencies1242 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies4002580629/001/store/v07xc3rwmgk1c4pp32zzf7zpyq9987lk-test-script12432026/11/12 00:23:06 INFO Completed push id=1 objects=4 duration=10.414136ms1244--- PASS: TestPush_RepeatedRootCommitsOnce (0.62s)1245=== CONT TestService_AuthMiddleware_MTLSBoundSubjects12462026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12472026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"12482026/11/12 00:23:06 INFO Signed narinfos id=1 count=112492026/11/12 00:23:06 INFO Uploading 1 narinfos12502026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete12512026/11/12 00:23:06 WARN Failed to register uploaded object key=2szgzild6pladcajgmv935siwjr9zjcp.narinfo error="server returned 404: 404 page not found\n"1252=== NAME TestClientWithDependencies1253 client_integration_test.go:663: Found 1 dependencies (including self)12542026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes12552026/11/12 00:23:06 INFO Completed push id=1 objects=3 duration=22.587561ms12562026/11/12 00:23:06 INFO Upload complete. (113ms)12572026/11/12 00:23:06 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)12582026/11/12 00:23:06 INFO Uploading nsc2zlclwimdx23541qdzgws4pgw3r6k-top (224B)12592026/11/12 00:23:06 INFO Uploading x3xzd3ajg39nmrk39hhpq9nrj1carpf9-shared-dep (136B)12602026/11/12 00:23:06 WARN Failed to register uploaded object key=x3xzd3ajg39nmrk39hhpq9nrj1carpf9.ls error="server returned 404: 404 page not found\n"12612026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12622026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12632026/11/12 00:23:06 WARN Failed to register uploaded object key=nsc2zlclwimdx23541qdzgws4pgw3r6k.ls error="server returned 404: 404 page not found\n"12642026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/0mp6n1zqgmls83d00n926g37g8syrhzrfcpzl9k6gg834ll7811z.nar.zst error="server returned 404: 404 page not found\n"12652026/11/12 00:23:06 INFO Signed narinfos id=1 count=212662026/11/12 00:23:06 INFO Uploading 2 narinfos1267=== NAME TestClientMultipleUploads1268 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads3708801105/001/store/0dgmkgwgaysfkfl07vzc4zfzq5wqijgv-test-file-0.txt12692026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete12702026/11/12 00:23:06 WARN Failed to register uploaded object key=nsc2zlclwimdx23541qdzgws4pgw3r6k.narinfo error="server returned 404: 404 page not found\n"12712026/11/12 00:23:06 WARN Failed to register uploaded object key=x3xzd3ajg39nmrk39hhpq9nrj1carpf9.narinfo error="server returned 404: 404 page not found\n"12722026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes12732026/11/12 00:23:06 INFO Completed push id=1 objects=6 duration=10.773806ms12742026/11/12 00:23:06 INFO Upload complete. (82ms)1275=== NAME TestClientSharedPathCommittedMidPush1276 client_integration_test.go:728: Retrieved narinfo from S3:1277 StorePath: /build/TestClientSharedPathCommittedMidPush802911203/001/store/x3xzd3ajg39nmrk39hhpq9nrj1carpf9-shared-dep1278 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1279 Compression: zstd1280 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821281 NarSize: 1361282 References: 1283 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1284 client_integration_test.go:728: Retrieved narinfo from S3:1285 StorePath: /build/TestClientSharedPathCommittedMidPush802911203/001/store/nsc2zlclwimdx23541qdzgws4pgw3r6k-top1286 URL: nar/0mp6n1zqgmls83d00n926g37g8syrhzrfcpzl9k6gg834ll7811z.nar.zst1287 Compression: zstd1288 NarHash: sha256:0mp6n1zqgmls83d00n926g37g8syrhzrfcpzl9k6gg834ll7811z1289 NarSize: 2241290 References: /build/TestClientSharedPathCommittedMidPush802911203/001/store/x3xzd3ajg39nmrk39hhpq9nrj1carpf9-shared-dep1291 CA: text:sha256:1ifipg048dibwi149rpxynibs58b8b0p7mdl8sc6gn8qmj7x8dnc1292=== NAME TestClientIntegration1293 client_integration_test.go:292: Created store path: /build/TestClientIntegration1062530283/002/store/blpypn8svq4rrw0yi97zhz623v4wzarw-test-file.txt1294--- PASS: TestClientSharedPathCommittedMidPush (0.88s)1295=== CONT TestReadProxyConditionalGet12962026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete12972026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes12982026/11/12 00:23:06 INFO Completed push id=1 objects=6 duration=8.143137ms1299--- PASS: TestPush_CompleteCommitsEveryRoot (0.63s)1300=== CONT TestService_AuthMiddleware_MTLSProxyHeader1301=== NAME TestClientMultipleUploads1302 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads3708801105/001/store/v3q1i8si9lfyw78vkvlwin0swf13w37v-test-file-1.txt13032026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes13042026/11/12 00:23:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13052026/11/12 00:23:06 INFO Uploading myj4nnpsjg3lvn9p2gwjiw0pk57q3dqz-unpinned-file.txt (128B)13062026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes1307--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.60s)1308=== CONT TestService_AuthMiddleware13092026/11/12 00:23:06 WARN Failed to register uploaded object key=myj4nnpsjg3lvn9p2gwjiw0pk57q3dqz.ls error="server returned 404: 404 page not found\n"13102026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13112026/11/12 00:23:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13122026/11/12 00:23:06 INFO Uploading v07xc3rwmgk1c4pp32zzf7zpyq9987lk-test-script (136B)13132026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"13142026/11/12 00:23:06 INFO Signed narinfos id=2 count=113152026/11/12 00:23:06 INFO Uploading 1 narinfos13162026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/2/complete13172026/11/12 00:23:06 WARN Failed to register uploaded object key=v07xc3rwmgk1c4pp32zzf7zpyq9987lk.ls error="server returned 404: 404 page not found\n"13182026/11/12 00:23:06 WARN Failed to register uploaded object key=myj4nnpsjg3lvn9p2gwjiw0pk57q3dqz.narinfo error="server returned 404: 404 page not found\n"13192026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13202026/11/12 00:23:06 WARN Failed to register uploaded object key=log/gv2s3ypwzfsch0qnjfzfy41yfa9cszgc-test-script.drv error="server returned 404: 404 page not found\n"13212026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13222026/11/12 00:23:06 INFO Signed narinfos id=1 count=113232026/11/12 00:23:06 INFO Uploading 1 narinfos13242026/11/12 00:23:06 INFO Completed push id=2 objects=3 duration=2.878449ms13252026/11/12 00:23:06 INFO Upload complete. (55ms)13262026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete13272026/11/12 00:23:06 WARN Failed to register uploaded object key=v07xc3rwmgk1c4pp32zzf7zpyq9987lk.narinfo error="server returned 404: 404 page not found\n"1328--- PASS: TestReadRedirectUsesPublicS3URL (0.58s)1329=== CONT TestReadProxyOutlastsServerWriteTimeout1330=== NAME TestClientMultipleUploads1331 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads3708801105/001/store/rxsc80cazggddrdghp5xv5v9wlmvzi26-test-file-2.txt13322026/11/12 00:23:06 INFO Completed push id=1 objects=4 duration=10.264916ms13332026/11/12 00:23:06 INFO Upload complete. (82ms)1334=== NAME TestClientWithDependencies1335 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies4002580629/001/store) requires matching store prefix1336--- PASS: TestClientWithDependencies (0.92s)1337=== CONT TestDeduplicatedObjectsRecordedAsPending13382026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes13392026/11/12 00:23:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13402026/11/12 00:23:06 INFO Uploading blpypn8svq4rrw0yi97zhz623v4wzarw-test-file.txt (152B)13412026/11/12 00:23:06 INFO Starting cleanup of old closures method=DELETE path=/api/closures13422026/11/12 00:23:06 INFO Garbage collection started13432026/11/12 00:23:06 WARN Failed to register uploaded object key=blpypn8svq4rrw0yi97zhz623v4wzarw.ls error="server returned 404: 404 page not found\n"13442026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13452026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"13462026/11/12 00:23:06 INFO Signed narinfos id=1 count=113472026/11/12 00:23:06 INFO Uploading 1 narinfos13482026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete13492026/11/12 00:23:06 WARN Failed to register uploaded object key=blpypn8svq4rrw0yi97zhz623v4wzarw.narinfo error="server returned 404: 404 page not found\n"13502026/11/12 00:23:06 INFO Aborted multipart uploads count=0 kept=01351--- PASS: TestReadProxyRangeRequest (0.56s)1352=== CONT TestReadProxyHead13532026/11/12 00:23:06 INFO Completed push id=1 objects=3 duration=8.303357ms13542026/11/12 00:23:06 INFO Upload complete. (82ms)13552026/11/12 00:23:06 WARN Force mode enabled - objects will be deleted immediately without grace period13562026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes13572026/11/12 00:23:06 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)13582026/11/12 00:23:06 INFO Uploading 0dgmkgwgaysfkfl07vzc4zfzq5wqijgv-test-file-0.txt (160B)13592026/11/12 00:23:06 INFO Uploading rxsc80cazggddrdghp5xv5v9wlmvzi26-test-file-2.txt (160B)13602026/11/12 00:23:06 INFO Uploading v3q1i8si9lfyw78vkvlwin0swf13w37v-test-file-1.txt (160B)13612026/11/12 00:23:06 WARN Failed to register uploaded object key=v3q1i8si9lfyw78vkvlwin0swf13w37v.ls error="server returned 404: 404 page not found\n"13622026/11/12 00:23:06 WARN Failed to register uploaded object key=rxsc80cazggddrdghp5xv5v9wlmvzi26.ls error="server returned 404: 404 page not found\n"1363--- PASS: TestCacheStatsHandler (0.58s)13642026/11/12 00:23:06 WARN Failed to register uploaded object key=0dgmkgwgaysfkfl07vzc4zfzq5wqijgv.ls error="server returned 404: 404 page not found\n"1365=== CONT TestPushDedupSurvivesConcurrentGC13662026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"13672026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"13682026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13692026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"13702026/11/12 00:23:06 INFO Signed narinfos id=1 count=313712026/11/12 00:23:06 INFO All 1 paths already cached13722026/11/12 00:23:06 INFO Uploading 3 narinfos1373=== NAME TestClientIntegration1374 client_integration_test.go:318: Retrieved narinfo from S3:1375 StorePath: /build/TestClientIntegration1062530283/002/store/blpypn8svq4rrw0yi97zhz623v4wzarw-test-file.txt1376 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1377 Compression: zstd1378 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11379 NarSize: 1521380 References: 1381 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11382 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1383 client_integration_test.go:319: Decompressed .ls content (64 bytes):1384 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}13852026/11/12 00:23:06 WARN Failed to register uploaded object key=0dgmkgwgaysfkfl07vzc4zfzq5wqijgv.narinfo error="server returned 404: 404 page not found\n"13862026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete13872026/11/12 00:23:06 WARN Failed to register uploaded object key=rxsc80cazggddrdghp5xv5v9wlmvzi26.narinfo error="server returned 404: 404 page not found\n"13882026/11/12 00:23:06 WARN Failed to register uploaded object key=v3q1i8si9lfyw78vkvlwin0swf13w37v.narinfo error="server returned 404: 404 page not found\n"13892026/11/12 00:23:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01390--- PASS: TestService_ReadScope_PublicByDefault (0.58s)1391=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent1392=== NAME TestClientCADerivations1393 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations492197336/001/store/ibwzzglsq11j6nvyk7ibixndxc51pd35-ca-test13942026/11/12 00:23:06 INFO Vacuumed table table=pending_closures13952026/11/12 00:23:06 INFO Vacuumed table table=pending_objects13962026/11/12 00:23:06 INFO Completed push id=1 objects=9 duration=9.768616ms13972026/11/12 00:23:06 INFO Upload complete. (78ms)1398=== NAME TestClientMultipleUploads13992026/11/12 00:23:06 INFO Vacuumed table table=multipart_uploads1400 client_integration_test.go:417: Uploaded 3 paths in 111.879368ms14012026/11/12 00:23:06 INFO Vacuumed table table=closures14022026/11/12 00:23:06 INFO Vacuumed table table=objects14032026-11-12 00:23:06.427 UTC [1003] ERROR: Push does not exist: id=114042026-11-12 00:23:06.427 UTC [1003] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE14052026-11-12 00:23:06.427 UTC [1003] STATEMENT: SELECT commit_push($1)1406--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.99s)1407=== CONT TestCreatePin_ReservedPins1408--- PASS: TestClientMultipleUploads (0.83s)1409=== CONT TestLeadElectsOneAndHandsOver1410=== NAME TestClientCADerivations1411 client_ca_test.go:139: Found 1 dependencies (including self)1412--- PASS: TestReadRedirectKeepsNarinfoProxied (0.59s)1413=== CONT TestReadProxy40414142026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes14152026/11/12 00:23:06 INFO Object in database but missing from S3 key=blpypn8svq4rrw0yi97zhz623v4wzarw.ls14162026/11/12 00:23:06 WARN Found objects in DB but missing from S3, will re-upload count=114172026/11/12 00:23:06 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)14182026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/2/complete14192026/11/12 00:23:06 WARN Failed to register uploaded object key=blpypn8svq4rrw0yi97zhz623v4wzarw.ls error="server returned 404: 404 page not found\n"1420--- PASS: TestReadRedirectNar (0.60s)1421=== CONT TestGCMetrics14222026/11/12 00:23:06 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34097/oidc14232026/11/12 00:23:06 INFO Completed push id=2 objects=3 duration=14.894805ms14242026/11/12 00:23:06 INFO Upload complete. (62ms)1425=== NAME TestClientIntegration1426 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1427 client_integration_test.go:336: Decompressed .ls content (49 bytes):1428 {"version":1,"root":{"type":"regular","size":39}}1429=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1430=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1431=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1432=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1433=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1434=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1435=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1436=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1437=== CONT TestReadProxyNarStreaming14382026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes14392026/11/12 00:23:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14402026/11/12 00:23:06 INFO Uploading ibwzzglsq11j6nvyk7ibixndxc51pd35-ca-test (144B)1441--- PASS: TestReadProxyDisabled (0.51s)1442=== CONT TestGCBugBareHashReferences14432026/11/12 00:23:06 WARN Failed to register uploaded object key=ibwzzglsq11j6nvyk7ibixndxc51pd35.ls error="server returned 404: 404 page not found\n"14442026/11/12 00:23:06 WARN Failed to register uploaded object key=log/1033k30h6pym4hr2n709rlsaqijksg5i-ca-test.drv error="server returned 404: 404 page not found\n"14452026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"14462026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14472026/11/12 00:23:06 INFO Signed narinfos id=1 count=114482026/11/12 00:23:06 INFO Uploading 1 narinfos14492026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete14502026/11/12 00:23:06 WARN Failed to register uploaded object key=ibwzzglsq11j6nvyk7ibixndxc51pd35.narinfo error="server returned 404: 404 page not found\n"14512026/11/12 00:23:06 INFO Completed push id=1 objects=4 duration=9.982816ms14522026/11/12 00:23:06 INFO Upload complete. (105ms)1453=== NAME TestClientCADerivations1454 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations492197336/001/store/ibwzzglsq11j6nvyk7ibixndxc51pd35-ca-test1455 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1456 Compression: zstd1457 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1458 NarSize: 1441459 References: 1460 Deriver: /build/TestClientCADerivations492197336/001/store/1033k30h6pym4hr2n709rlsaqijksg5i-ca-test.drv1461 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1462 client_ca_test.go:185: Checking for realisation files in S3...1463 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1464 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1465--- PASS: TestService_ReadAuthMiddleware (0.49s)1466=== CONT TestDeleteClosuresRacingPins14672026/11/12 00:23:06 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14682026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes14692026/11/12 00:23:06 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)14702026/11/12 00:23:06 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MjMwZTRiYjAtYzdlMy00NDAyLWI4YjYtZjYzMDVjZjMxZTkwLjE5OGQzZTllLTNiMmYtNDE0Yy04NDkzLTlmNzA2YjQ4NWM1Y3gxNzk0NDQyOTg1NDgzMDg5Mjk5 parts=121471--- PASS: TestSharedNarBothUploadsComplete (1.75s)1472=== CONT TestReadProxyContentNegotiation14732026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/3/complete14742026/11/12 00:23:06 WARN Failed to register uploaded object key=blpypn8svq4rrw0yi97zhz623v4wzarw.ls error="server returned 404: 404 page not found\n"14752026/11/12 00:23:06 INFO Completed push id=3 objects=3 duration=2.982519ms14762026/11/12 00:23:06 INFO Upload complete. (50ms)1477=== RUN TestService_RequireScope_OIDC/builder_may_write1478=== PAUSE TestService_RequireScope_OIDC/builder_may_write1479=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1480=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1481=== RUN TestService_RequireScope_OIDC/ops_may_admin1482=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1483=== RUN TestService_RequireScope_OIDC/ops_may_not_write1484=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1485=== RUN TestService_RequireScope_OIDC/reader_may_not_write1486=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1487=== RUN TestService_RequireScope_OIDC/static_token_may_admin1488=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1489=== RUN TestService_RequireScope_OIDC/static_token_may_write1490=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1491=== RUN TestService_RequireScope_OIDC/reader_may_read1492=== PAUSE TestService_RequireScope_OIDC/reader_may_read1493=== RUN TestService_RequireScope_OIDC/writer_implies_read1494=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1495=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1496=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1497=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1498=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1499=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1500=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1501=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1502=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1503=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1504=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1505=== NAME TestClientIntegration1506 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1507=== CONT TestGCAdvisoryLockBlocksConcurrentRun1508=== NAME TestClientIntegration1509 client_integration_test.go:353: Decompressed .ls content (49 bytes):1510 {"version":1,"root":{"type":"regular","size":39}}1511 client_integration_test.go:364: Testing garbage collection...1512--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.51s)1513=== CONT TestReadProxyNarinfoAlreadyDecompressed15142026/11/12 00:23:06 INFO Received push request method=POST path=/api/pushes15152026/11/12 00:23:06 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)15162026/11/12 00:23:06 INFO Uploading jr5crylx2wzc9qhhgx4yx9lfgvk0si6d-shared-dep (136B)15172026/11/12 00:23:06 INFO Uploading i3rjjzk05yspl6pshsf0j9b9qj92c9h6-b (216B)15182026/11/12 00:23:06 INFO Starting cleanup of old closures method=DELETE path=/api/closures15192026/11/12 00:23:06 INFO Garbage collection started15202026/11/12 00:23:06 WARN Failed to register uploaded object key=i3rjjzk05yspl6pshsf0j9b9qj92c9h6.ls error="server returned 404: 404 page not found\n"15212026/11/12 00:23:06 WARN Failed to register uploaded object key=h1f928nwm2rq3bgzgh8kwjcy75wzwqxw.ls error="server returned 404: 404 page not found\n"15222026/11/12 00:23:06 WARN Failed to register uploaded object key=jr5crylx2wzc9qhhgx4yx9lfgvk0si6d.ls error="server returned 404: 404 page not found\n"15232026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/0rmgmwwqa8kxks99fnhyiaamivhziglaasd4fz9sysf210b0lxds.nar.zst error="server returned 404: 404 page not found\n"15242026/11/12 00:23:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15252026/11/12 00:23:06 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15262026/11/12 00:23:06 INFO Signed narinfos id=1 count=315272026/11/12 00:23:06 INFO Uploading 3 narinfos15282026/11/12 00:23:06 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"15292026/11/12 00:23:06 WARN mTLS auth: bound subjects configured but subject DN unavailable15302026/11/12 00:23:06 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1531--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.51s)1532=== CONT TestClientFallsBackToClosures15332026/11/12 00:23:06 WARN Failed to register uploaded object key=h1f928nwm2rq3bgzgh8kwjcy75wzwqxw.narinfo error="server returned 404: 404 page not found\n"15342026/11/12 00:23:06 WARN Failed to register uploaded object key=jr5crylx2wzc9qhhgx4yx9lfgvk0si6d.narinfo error="server returned 404: 404 page not found\n"15352026/11/12 00:23:06 INFO Received complete push request method=POST path=/api/pushes/1/complete15362026/11/12 00:23:06 WARN Failed to register uploaded object key=i3rjjzk05yspl6pshsf0j9b9qj92c9h6.narinfo error="server returned 404: 404 page not found\n"15372026/11/12 00:23:06 INFO Aborted multipart uploads count=0 kept=015382026/11/12 00:23:06 WARN Force mode enabled - objects will be deleted immediately without grace period15392026/11/12 00:23:06 INFO Completed push id=1 objects=8 duration=9.611267ms15402026/11/12 00:23:06 INFO Upload complete. (71ms)1541=== NAME TestClientPushesUseOnePush1542 client_pushes_test.go:97: Retrieved narinfo from S3:1543 StorePath: /build/TestClientPushesUseOnePush2623740605/001/store/jr5crylx2wzc9qhhgx4yx9lfgvk0si6d-shared-dep1544 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1545 Compression: zstd1546 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821547 NarSize: 1361548 References: 1549 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1550 client_pushes_test.go:97: Retrieved narinfo from S3:1551 StorePath: /build/TestClientPushesUseOnePush2623740605/001/store/h1f928nwm2rq3bgzgh8kwjcy75wzwqxw-a1552 URL: nar/0rmgmwwqa8kxks99fnhyiaamivhziglaasd4fz9sysf210b0lxds.nar.zst1553 Compression: zstd1554 NarHash: sha256:0rmgmwwqa8kxks99fnhyiaamivhziglaasd4fz9sysf210b0lxds1555 NarSize: 2161556 References: /build/TestClientPushesUseOnePush2623740605/001/store/jr5crylx2wzc9qhhgx4yx9lfgvk0si6d-shared-dep1557 CA: text:sha256:1099bd06dyc484jzbwrisdcr4gp8rxa9fdh3bplc06dwznv2p04y1558 client_pushes_test.go:97: Retrieved narinfo from S3:1559 StorePath: /build/TestClientPushesUseOnePush2623740605/001/store/i3rjjzk05yspl6pshsf0j9b9qj92c9h6-b1560 URL: nar/0rmgmwwqa8kxks99fnhyiaamivhziglaasd4fz9sysf210b0lxds.nar.zst1561 Compression: zstd1562 NarHash: sha256:0rmgmwwqa8kxks99fnhyiaamivhziglaasd4fz9sysf210b0lxds1563 NarSize: 2161564 References: /build/TestClientPushesUseOnePush2623740605/001/store/jr5crylx2wzc9qhhgx4yx9lfgvk0si6d-shared-dep1565 CA: text:sha256:1099bd06dyc484jzbwrisdcr4gp8rxa9fdh3bplc06dwznv2p04y1566--- PASS: TestClientPushesUseOnePush (0.83s)1567=== CONT TestLeadEndsOnShutdown1568=== NAME TestClientCADerivations1569 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1570 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1571 error: binary cache 's3://bucket45?endpoint=http://localhost:46549&region=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations492197336/001/store'1572 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11573--- PASS: TestClientCADerivations (0.95s)1574=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1575--- PASS: TestReadProxyConditionalGet (0.46s)1576=== CONT TestLeadGenerationGrowsWithEveryGrant1577--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.47s)1578=== CONT TestPresentNotReportedWhileGCDeletesClosure1579=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1580=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1581=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1582=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1583=== CONT TestLeadIncumbentAnnouncesAtOnce15842026/11/12 00:23:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=015852026/11/12 00:23:06 INFO Vacuumed table table=pending_closures15862026/11/12 00:23:06 INFO Vacuumed table table=pending_objects15872026/11/12 00:23:06 INFO Vacuumed table table=multipart_uploads15882026/11/12 00:23:06 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1589--- PASS: TestService_AuthMiddleware (0.48s)1590=== CONT TestReadProxyNarinfo15912026/11/12 00:23:06 INFO Vacuumed table table=closures15922026/11/12 00:23:06 INFO Vacuumed table table=objects15932026/11/12 00:23:06 INFO Received uploads request method=POST path=/api/pending_closures1594--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.53s)1595=== CONT TestConnectWaitsForAPeerMigration1596--- PASS: TestReadProxyHead (0.50s)1597=== CONT TestPresentRefreshesOnlyStaleClosures15982026/11/12 00:23:06 INFO Received uploads request method=POST path=/api/pending_closures15992026/11/12 00:23:06 INFO lead: acquired remote=192.0.2.1:1234 generation=116002026/11/12 00:23:06 INFO lead: acquired remote=192.0.2.1:1234 generation=116012026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=016022026/11/12 00:23:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016032026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=016042026/11/12 00:23:07 INFO Vacuumed table table=pending_closures16052026/11/12 00:23:07 INFO Vacuumed table table=pending_objects16062026/11/12 00:23:07 INFO Vacuumed table table=multipart_uploads16072026/11/12 00:23:07 WARN Force mode enabled - objects will be deleted immediately without grace period16082026/11/12 00:23:07 INFO Vacuumed table table=closures16092026/11/12 00:23:07 INFO Vacuumed table table=objects16102026/11/12 00:23:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016112026/11/12 00:23:07 INFO Vacuumed table table=pending_closures16122026/11/12 00:23:07 INFO Vacuumed table table=pending_objects16132026/11/12 00:23:07 INFO Vacuumed table table=multipart_uploads16142026/11/12 00:23:07 INFO Vacuumed table table=closures16152026/11/12 00:23:07 INFO Vacuumed table table=objects16162026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=016172026/11/12 00:23:07 WARN Force mode enabled - objects will be deleted immediately without grace period16182026/11/12 00:23:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016192026/11/12 00:23:07 INFO Vacuumed table table=pending_closures16202026/11/12 00:23:07 INFO Vacuumed table table=pending_objects16212026/11/12 00:23:07 INFO Vacuumed table table=multipart_uploads16222026/11/12 00:23:07 INFO Vacuumed table table=closures16232026/11/12 00:23:07 INFO Vacuumed table table=objects1624--- PASS: TestPushDedupSurvivesConcurrentGC (0.64s)1625=== CONT TestResolveDBConnectionString1626--- PASS: TestGCMetrics (0.52s)1627=== CONT TestIsValidCachePath1628=== RUN TestIsValidCachePath/narinfo1629=== PAUSE TestIsValidCachePath/narinfo1630=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1631=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1632=== RUN TestIsValidCachePath/nar_zst1633=== PAUSE TestIsValidCachePath/nar_zst1634=== RUN TestResolveDBConnectionString/flag_wins1635=== PAUSE TestResolveDBConnectionString/flag_wins1636=== RUN TestIsValidCachePath/nar_xz1637=== PAUSE TestIsValidCachePath/nar_xz1638=== RUN TestIsValidCachePath/nar_bz21639=== PAUSE TestIsValidCachePath/nar_bz21640=== RUN TestIsValidCachePath/nar_uncompressed1641=== PAUSE TestIsValidCachePath/nar_uncompressed1642=== RUN TestIsValidCachePath/ls1643=== PAUSE TestIsValidCachePath/ls1644=== RUN TestResolveDBConnectionString/file_when_flag_empty1645=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1646=== RUN TestResolveDBConnectionString/missing_file_is_an_error1647=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1648=== RUN TestIsValidCachePath/log1649=== PAUSE TestIsValidCachePath/log1650=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1651=== RUN TestIsValidCachePath/realisation1652=== PAUSE TestIsValidCachePath/realisation1653=== RUN TestIsValidCachePath/nix-cache-info1654=== PAUSE TestIsValidCachePath/nix-cache-info1655=== RUN TestIsValidCachePath/index.html1656=== PAUSE TestIsValidCachePath/index.html1657=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1658=== RUN TestIsValidCachePath/traversal_parent1659=== PAUSE TestIsValidCachePath/traversal_parent1660=== RUN TestResolveDBConnectionString/nothing_configured1661=== PAUSE TestResolveDBConnectionString/nothing_configured1662=== RUN TestIsValidCachePath/traversal_in_middle1663=== PAUSE TestIsValidCachePath/traversal_in_middle1664=== RUN TestIsValidCachePath/invalid_char_e1665=== PAUSE TestIsValidCachePath/invalid_char_e1666=== RUN TestIsValidCachePath/invalid_char_u1667=== PAUSE TestIsValidCachePath/invalid_char_u1668=== RUN TestIsValidCachePath/random_path1669=== PAUSE TestIsValidCachePath/random_path1670=== RUN TestIsValidCachePath/empty1671--- PASS: TestReadProxy404 (0.59s)1672=== CONT TestParseSingleRange1673=== RUN TestParseSingleRange/none1674=== PAUSE TestParseSingleRange/none1675=== CONT TestObjectStatsUpdatedOncePerCommit1676=== PAUSE TestIsValidCachePath/empty1677=== RUN TestIsValidCachePath/leading_slash1678=== PAUSE TestIsValidCachePath/leading_slash1679=== RUN TestIsValidCachePath/wrong_extension1680=== PAUSE TestIsValidCachePath/wrong_extension1681=== RUN TestIsValidCachePath/short_hash1682=== PAUSE TestIsValidCachePath/short_hash1683=== RUN TestParseSingleRange/unknown_unit1684=== CONT TestOrphanedObjectsGCStressTest1685=== PAUSE TestParseSingleRange/unknown_unit1686=== RUN TestParseSingleRange/multi-range_ignored1687=== PAUSE TestParseSingleRange/multi-range_ignored1688=== RUN TestParseSingleRange/malformed_no_dash1689=== PAUSE TestParseSingleRange/malformed_no_dash1690=== RUN TestParseSingleRange/malformed_both_empty1691=== PAUSE TestParseSingleRange/malformed_both_empty1692=== RUN TestParseSingleRange/malformed_end_before_start1693=== PAUSE TestParseSingleRange/malformed_end_before_start1694=== RUN TestParseSingleRange/closed1695=== PAUSE TestParseSingleRange/closed16962026/11/12 00:23:07 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1697=== RUN TestParseSingleRange/open-ended16982026/11/12 00:23:07 WARN Refused reserved pin name=worker-x86_64-linux1699=== PAUSE TestParseSingleRange/open-ended1700=== RUN TestParseSingleRange/end_clamped_to_size1701=== PAUSE TestParseSingleRange/end_clamped_to_size1702=== RUN TestParseSingleRange/suffix1703=== PAUSE TestParseSingleRange/suffix17042026/11/12 00:23:07 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1705=== RUN TestParseSingleRange/suffix_exceeds_size1706=== PAUSE TestParseSingleRange/suffix_exceeds_size1707=== RUN TestParseSingleRange/single_byte1708=== PAUSE TestParseSingleRange/single_byte17092026/11/12 00:23:07 INFO Received create pin request method=POST path=/api/pins/my-app1710=== RUN TestParseSingleRange/start_past_EOF17112026/11/12 00:23:07 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1712=== PAUSE TestParseSingleRange/start_past_EOF1713=== RUN TestParseSingleRange/start_far_past_EOF1714=== PAUSE TestParseSingleRange/start_far_past_EOF1715=== CONT TestOrphanedObjectsGC1716--- PASS: TestCreatePin_ReservedPins (0.62s)1717=== CONT TestObjectStatsTrigger1718--- PASS: TestReadProxyNarStreaming (0.53s)1719=== CONT TestPendingCleanupUsesOneCutoff17202026/11/12 00:23:07 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1721--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.51s)1722=== CONT TestPresentReportsOnlyClosureRoots1723--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.55s)1724=== CONT TestMetricsCommit1725--- PASS: TestReadProxyContentNegotiation (0.60s)1726=== CONT TestService_NativeMTLS17272026/11/12 00:23:07 INFO lead: acquired remote=192.0.2.1:1234 generation=11728--- PASS: TestGCBugBareHashReferences (0.75s)1729=== CONT TestMetricsInventory17302026/11/12 00:23:07 INFO lead: acquired remote=192.0.2.1:1234 generation=117312026/11/12 00:23:07 INFO lead: acquired remote=192.0.2.1:1234 generation=117322026/11/12 00:23:07 INFO lead: released remote=192.0.2.1:1234 generation=11733--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.63s)1734=== CONT TestMultipartCleanup17352026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures17362026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures17372026/11/12 00:23:07 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)17382026/11/12 00:23:07 INFO Uploading krn0h234g3574blqrir326h89qn3laay-shared-dep (136B)17392026/11/12 00:23:07 INFO Uploading w0fbrbxyix9lznq7ya535ryrzxk7xdi2-a (216B)1740--- PASS: TestPresentRefreshesOnlyStaleClosures (0.57s)1741=== CONT TestServerTLSConfig1742=== RUN TestServerTLSConfig/no_client_CA1743=== PAUSE TestServerTLSConfig/no_client_CA1744=== RUN TestServerTLSConfig/missing_CA_file1745=== PAUSE TestServerTLSConfig/missing_CA_file1746=== RUN TestServerTLSConfig/not_a_PEM_file1747=== PAUSE TestServerTLSConfig/not_a_PEM_file1748=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded17492026/11/12 00:23:07 WARN Failed to register uploaded object key=279bkxrkrmwmbmq6imizihppm2s37j21.ls error="server returned 404: 404 page not found\n"17502026/11/12 00:23:07 WARN Failed to register uploaded object key=w0fbrbxyix9lznq7ya535ryrzxk7xdi2.ls error="server returned 404: 404 page not found\n"17512026/11/12 00:23:07 WARN Failed to register uploaded object key=krn0h234g3574blqrir326h89qn3laay.ls error="server returned 404: 404 page not found\n"17522026/11/12 00:23:07 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"17532026/11/12 00:23:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign17542026/11/12 00:23:07 WARN Failed to register uploaded object key=nar/1qsdzp3am3b401lmc0x1y3qziaqxiar2z0aqs2xxvqgpipfzm3aq.nar.zst error="server returned 404: 404 page not found\n"17552026/11/12 00:23:07 INFO Signed narinfos id=1 count=217562026/11/12 00:23:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign17572026/11/12 00:23:07 INFO Signed narinfos id=2 count=217582026/11/12 00:23:07 INFO Uploading 4 narinfos17592026/11/12 00:23:07 WARN Failed to register uploaded object key=w0fbrbxyix9lznq7ya535ryrzxk7xdi2.narinfo error="server returned 404: 404 page not found\n"17602026/11/12 00:23:07 WARN Failed to register uploaded object key=krn0h234g3574blqrir326h89qn3laay.narinfo error="server returned 404: 404 page not found\n"17612026/11/12 00:23:07 WARN Failed to register uploaded object key=279bkxrkrmwmbmq6imizihppm2s37j21.narinfo error="server returned 404: 404 page not found\n"17622026/11/12 00:23:07 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17632026/11/12 00:23:07 WARN Failed to register uploaded object key=krn0h234g3574blqrir326h89qn3laay.narinfo error="server returned 404: 404 page not found\n"17642026/11/12 00:23:07 INFO Completed upload id=1 objects=6 duration=4.782248ms17652026/11/12 00:23:07 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17662026/11/12 00:23:07 INFO Completed upload id=2 objects=6 duration=1.465919ms17672026/11/12 00:23:07 INFO Upload complete. (71ms)1768=== NAME TestClientFallsBackToClosures1769 client_pushes_test.go:112: Retrieved narinfo from S3:1770 StorePath: /build/TestClientFallsBackToClosures3990685311/001/store/krn0h234g3574blqrir326h89qn3laay-shared-dep1771 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1772 Compression: zstd1773 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821774 NarSize: 1361775 References: 1776 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1777 client_pushes_test.go:112: Retrieved narinfo from S3:1778 StorePath: /build/TestClientFallsBackToClosures3990685311/001/store/w0fbrbxyix9lznq7ya535ryrzxk7xdi2-a1779 URL: nar/1qsdzp3am3b401lmc0x1y3qziaqxiar2z0aqs2xxvqgpipfzm3aq.nar.zst1780 Compression: zstd1781 NarHash: sha256:1qsdzp3am3b401lmc0x1y3qziaqxiar2z0aqs2xxvqgpipfzm3aq1782 NarSize: 2161783 References: /build/TestClientFallsBackToClosures3990685311/001/store/krn0h234g3574blqrir326h89qn3laay-shared-dep1784 CA: text:sha256:0r0z2dl90vv6wm5p3cf7lzparfb4nvpcgnwydsmqp2acylmp4h841785 client_pushes_test.go:112: Retrieved narinfo from S3:1786 StorePath: /build/TestClientFallsBackToClosures3990685311/001/store/279bkxrkrmwmbmq6imizihppm2s37j21-b1787 URL: nar/1qsdzp3am3b401lmc0x1y3qziaqxiar2z0aqs2xxvqgpipfzm3aq.nar.zst1788 Compression: zstd1789 NarHash: sha256:1qsdzp3am3b401lmc0x1y3qziaqxiar2z0aqs2xxvqgpipfzm3aq1790 NarSize: 2161791 References: /build/TestClientFallsBackToClosures3990685311/001/store/krn0h234g3574blqrir326h89qn3laay-shared-dep1792 CA: text:sha256:0r0z2dl90vv6wm5p3cf7lzparfb4nvpcgnwydsmqp2acylmp4h841793--- PASS: TestClientFallsBackToClosures (0.80s)1794=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info17952026/11/12 00:23:07 INFO Received uploads request method=POST path=/1796=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key17972026/11/12 00:23:07 INFO Received request for more parts method=POST path=/1798=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key17992026/11/12 00:23:07 INFO Received complete multipart upload request method=POST path=/1800=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18012026/11/12 00:23:07 INFO Received uploads request method=POST path=/1802--- PASS: TestUploadHandlersRejectInvalidKeys (0.09s)1803 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1804 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1805 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1806 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1807=== CONT TestPendingClosureWriteTimeout/empty1808=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1809--- PASS: TestObjectStatsUpdatedOncePerCommit (0.45s)1810=== CONT TestPendingClosureWriteTimeout/670k_objects1811=== CONT TestPendingClosureWriteTimeout/400_objects1812=== CONT TestProxyWriteTimeout/narinfo1813=== CONT TestGCEndsOnShutdown/before_the_run1814--- PASS: TestObjectStatsTrigger (0.45s)1815=== CONT TestProxyWriteTimeout/10_GiB_nar1816=== CONT TestProxyWriteTimeout/unknown_size1817=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18182026/11/12 00:23:07 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1819--- PASS: TestReadProxyNarinfo (0.73s)1820=== CONT TestProxyWriteTimeout/1_GiB_nar1821=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1822--- PASS: TestProxyWriteTimeout (0.09s)1823 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1824 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1825 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1826 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1827--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.83s)1828=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed18292026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures18302026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures18312026/11/12 00:23:07 INFO Received cleanup request method=DELETE path=/api/pending_closures1832--- PASS: TestPresentReportsOnlyClosureRoots (0.47s)1833=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed18342026/11/12 00:23:07 INFO Received push request method=POST path=/api/pushes18352026/11/12 00:23:07 INFO Received complete push request method=POST path=/api/pushes/1/complete18362026/11/12 00:23:07 INFO Completed push id=1 objects=4 duration=7.944757ms18372026/11/12 00:23:07 WARN mTLS auth: subject not in bound subjects subject="CN=reader"18382026/11/12 00:23:07 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1839--- PASS: TestService_NativeMTLS (0.42s)1840=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed18412026/11/12 00:23:07 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete18422026/11/12 00:23:07 INFO Completed upload id=2 objects=2 duration=1.382999ms1843--- PASS: TestMetricsCommit (0.45s)1844=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost18452026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures1846--- PASS: TestMetricsInventory (0.40s)1847=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed18482026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures1849--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.33s)1850=== CONT TestIsValidUploadKey/narinfo1851=== CONT TestIsValidUploadKey/realisation_plus_in_output1852=== CONT TestIsValidUploadKey/realisation1853=== CONT TestIsValidUploadKey/build_log_equals1854=== CONT TestIsValidUploadKey/build_log_question_mark1855=== CONT TestIsValidUploadKey/build_log_plus_in_name1856=== CONT TestIsValidUploadKey/build_log_home-manager_file1857=== CONT TestIsValidUploadKey/build_log1858=== CONT TestIsValidUploadKey/listing1859=== CONT TestIsValidUploadKey/nar_plain1860=== CONT TestIsValidUploadKey/nar_xz1861=== CONT TestIsValidUploadKey/nix-cache-info1862=== CONT TestIsValidUploadKey/nar_zst1863=== CONT TestIsValidUploadKey/traversal1864=== CONT TestIsValidUploadKey/unknown_type1865=== CONT TestIsValidUploadKey/empty_key1866=== CONT TestIsValidUploadKey/absolute1867=== CONT TestIsValidUploadKey/traversal_nar1868=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1869=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1870=== CONT TestIsValidUploadKey/index.html1871=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1872--- PASS: TestIsValidUploadKey (0.10s)1873 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1874 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1875 --- PASS: TestIsValidUploadKey/realisation (0.00s)1876 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1877 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1878 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1879 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1880 --- PASS: TestIsValidUploadKey/build_log (0.00s)1881 --- PASS: TestIsValidUploadKey/listing (0.00s)1882 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1883 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1884 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1885 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1886 --- PASS: TestIsValidUploadKey/traversal (0.00s)1887 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1888 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1889 --- PASS: TestIsValidUploadKey/absolute (0.00s)1890 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1891 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1892 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1893 --- PASS: TestIsValidUploadKey/index.html (0.00s)1894 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1895=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows18962026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures18972026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=018982026/11/12 00:23:07 INFO Received cleanup request method=DELETE path=/api/pending_closures18992026/11/12 00:23:07 WARN Force mode enabled - objects will be deleted immediately without grace period19002026/11/12 00:23:07 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket92/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""19012026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=119022026/11/12 00:23:07 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Post \"http://localhost:46549/bucket95/?delete=\": context canceled"1903=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check19042026/11/12 00:23:07 INFO Received cleanup request method=DELETE path=/api/pending_closures19052026/11/12 00:23:07 INFO Aborted multipart uploads count=1 kept=01906--- PASS: TestMultipartCleanup (0.48s)1907=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19082026/11/12 00:23:07 INFO Received uploads request method=POST path=/1909=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19102026/11/12 00:23:07 INFO Received complete multipart upload request method=POST path=/19112026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures19122026/11/12 00:23:07 INFO Aborted multipart uploads count=0 kept=01913=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19142026/11/12 00:23:07 INFO Received request for more parts method=POST path=/19152026/11/12 00:23:07 WARN Force mode enabled - objects will be deleted immediately without grace period19162026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures1917--- PASS: TestGCEndsOnShutdown (0.00s)1918 --- PASS: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (0.35s)1919 --- PASS: TestGCEndsOnShutdown/before_the_run (0.36s)1920 --- PASS: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (0.44s)1921=== CONT TestClientErrorHandling/InvalidStorePath1922=== NAME TestOrphanedObjectsGC1923 orphaned_objects_gc_test.go:287: GC Test Summary:19242026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures1925 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1926 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1927 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1928 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1929 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1930--- PASS: TestOrphanedObjectsGC (0.92s)1931=== CONT TestClientErrorHandling/ServerNotAvailable1932=== CONT TestClientErrorHandling/InvalidAuthToken19332026/11/12 00:23:07 INFO Received uploads request method=POST path=/api/pending_closures19342026/11/12 00:23:08 INFO Received uploads request method=POST path=/api/pending_closures19352026/11/12 00:23:08 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19362026/11/12 00:23:08 INFO lead: released remote=192.0.2.1:1234 generation=119372026/11/12 00:23:08 INFO Received uploads request method=POST path=/api/pending_closures19382026/11/12 00:23:08 INFO Aborted multipart uploads count=0 kept=019392026/11/12 00:23:08 INFO lead: acquired remote=192.0.2.1:1234 generation=219402026/11/12 00:23:08 WARN Force mode enabled - objects will be deleted immediately without grace period1941--- PASS: TestDeleteClosuresRacingPins (1.56s)1942=== CONT TestCacheConfigHandler/full_config,_no_issuer1943=== CONT TestCacheConfigHandler/no_signing_keys1944=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1945=== CONT TestCacheConfigHandler/no_cache_url_configured1946--- PASS: TestCacheConfigHandler (0.00s)1947 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1948 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1949 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1950 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1951=== CONT TestPush_RejectsBadRequests/no_roots19522026/11/12 00:23:08 INFO Received push request method=POST path=/api/pushes1953=== CONT TestPush_RejectsBadRequests/bad_root19542026/11/12 00:23:08 INFO Received push request method=POST path=/api/pushes1955=== CONT TestPush_RejectsBadRequests/root_not_in_objects19562026/11/12 00:23:08 INFO Received push request method=POST path=/api/pushes1957=== CONT TestPush_RejectsBadRequests/no_objects19582026/11/12 00:23:08 INFO Received push request method=POST path=/api/pushes1959=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1960--- PASS: TestPush_RejectsBadRequests (0.64s)1961 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1962 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1963 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1964 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1965=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected19662026/11/12 00:23:08 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1967=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1968=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected19692026/11/12 00:23:08 INFO Received uploads request method=POST path=/api/pending_closures19702026/11/12 00:23:08 WARN Authentication failed token_preview=eyJhbGciOi...sVGFFA5Aqw token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1971=== CONT TestService_RequireScope_OIDC/builder_may_write1972--- PASS: TestService_AuthMiddleware_OIDC (0.55s)1973 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1974 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1975 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1976 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)1977=== CONT TestService_RequireScope_OIDC/writer_implies_read1978=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1979=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1980=== CONT TestService_RequireScope_OIDC/reader_may_not_write1981=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1982=== CONT TestService_RequireScope_OIDC/reader_may_read1983=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1984=== CONT TestService_RequireScope_OIDC/static_token_may_write1985=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19862026/11/12 00:23:08 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1987=== CONT TestService_RequireScope_OIDC/static_token_may_admin1988=== CONT TestService_RequireScope_OIDC/ops_may_admin1989=== CONT TestService_RequireScope_OIDC/ops_may_not_write1990=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1991=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1992--- PASS: TestService_RequireScope_OIDC (0.74s)1993 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1994 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1995 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1996 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1997 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1998 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1999 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)2000 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)

loading 626 hidden lines…

26272026/11/12 00:23:21 INFO Received push request method=POST path=/26282026/11/12 00:23:21 INFO Vacuumed table table=pending_objects26292026/11/12 00:23:21 INFO Vacuumed table table=pending_objects26302026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads26312026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads26322026/11/12 00:23:21 INFO Vacuumed table table=closures26332026/11/12 00:23:21 INFO Vacuumed table table=closures26342026/11/12 00:23:21 INFO Vacuumed table table=objects26352026/11/12 00:23:21 INFO Received uploads request method=POST path=/26362026/11/12 00:23:21 INFO Vacuumed table table=objects26372026/11/12 00:23:21 INFO Vacuumed table table=closures26382026/11/12 00:23:21 INFO Received uploads request method=POST path=/26392026/11/12 00:23:21 INFO Vacuumed table table=objects26402026/11/12 00:23:21 INFO Received complete push request method=POST path=/26412026/11/12 00:23:21 INFO Completed push id=2 objects=25 duration=3.323229ms26422026/11/12 00:23:21 INFO Received push request method=POST path=/26432026/11/12 00:23:21 INFO Received complete push request method=POST path=/26442026/11/12 00:23:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."26452026/11/12 00:23:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."26462026/11/12 00:23:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."26472026/11/12 00:23:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."26482026/11/12 00:23:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."26492026/11/12 00:23:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."26502026/11/12 00:23:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."26512026/11/12 00:23:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."26522026/11/12 00:23:21 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."26532026/11/12 00:23:21 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."26542026/11/12 00:23:21 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."26552026/11/12 00:23:21 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."26562026/11/12 00:23:21 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."26572026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."26582026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."26592026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."26602026/11/12 00:23:21 INFO Received uploads request method=POST path=/26612026/11/12 00:23:21 INFO Completed push id=8 objects=6 duration=2.569989ms26622026/11/12 00:23:21 INFO Received push request method=POST path=/26632026/11/12 00:23:21 INFO Received uploads request method=POST path=/26642026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=026652026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=026662026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period26672026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026682026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period26692026/11/12 00:23:21 INFO Vacuumed table table=pending_closures26702026/11/12 00:23:21 INFO Vacuumed table table=pending_objects26712026/11/12 00:23:21 INFO Received complete push request method=POST path=/26722026/11/12 00:23:21 INFO Received complete upload request method=POST path=/26732026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads26742026/11/12 00:23:21 INFO Vacuumed table table=closures26752026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=026762026/11/12 00:23:21 INFO Received complete upload request method=POST path=/26772026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=026782026/11/12 00:23:21 INFO Vacuumed table table=objects26792026/11/12 00:23:21 INFO Received push request method=POST path=/26802026/11/12 00:23:21 INFO Completed upload id=6 objects=7 duration=2.711539ms26812026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=026822026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period26832026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026842026/11/12 00:23:21 INFO Vacuumed table table=pending_closures26852026/11/12 00:23:21 INFO Vacuumed table table=pending_objects26862026/11/12 00:23:21 INFO Completed upload id=5 objects=7 duration=3.288789ms26872026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads26882026/11/12 00:23:21 INFO Vacuumed table table=closures26892026/11/12 00:23:21 INFO Vacuumed table table=objects26902026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period26912026/11/12 00:23:21 INFO Completed push id=1 objects=10 duration=6.390927ms26922026/11/12 00:23:21 INFO Received complete upload request method=POST path=/26932026/11/12 00:23:21 INFO Received complete upload request method=POST path=/26942026/11/12 00:23:21 INFO Completed upload id=9 objects=3 duration=1.462399ms26952026/11/12 00:23:21 INFO Completed upload id=10 objects=2 duration=2.675019ms26962026/11/12 00:23:21 INFO Received push request method=POST path=/26972026/11/12 00:23:21 INFO Received push request method=POST path=/26982026/11/12 00:23:21 INFO Received push request method=POST path=/26992026/11/12 00:23:21 INFO Received complete push request method=POST path=/27002026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027012026/11/12 00:23:21 INFO Completed push id=4 objects=7 duration=2.070469ms27022026/11/12 00:23:21 INFO Received uploads request method=POST path=/27032026/11/12 00:23:21 INFO Received uploads request method=POST path=/27042026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period27052026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=027062026/11/12 00:23:21 INFO Received push request method=POST path=/27072026/11/12 00:23:21 INFO Vacuumed table table=pending_closures27082026/11/12 00:23:21 INFO Vacuumed table table=pending_objects27092026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads27102026/11/12 00:23:21 INFO Vacuumed table table=closures27112026/11/12 00:23:21 INFO Received complete upload request method=POST path=/27122026/11/12 00:23:21 INFO Received push request method=POST path=/27132026/11/12 00:23:21 INFO Vacuumed table table=objects27142026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=027152026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027162026/11/12 00:23:21 INFO Received push request method=POST path=/27172026/11/12 00:23:21 INFO Completed upload id=6 objects=9 duration=1.94082ms27182026/11/12 00:23:21 INFO Vacuumed table table=pending_closures27192026/11/12 00:23:21 INFO Received complete upload request method=POST path=/27202026/11/12 00:23:21 INFO Vacuumed table table=pending_objects27212026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads27222026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027232026/11/12 00:23:21 INFO Vacuumed table table=closures27242026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27252026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027262026/11/12 00:23:21 INFO Received push request method=POST path=/27272026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027282026/11/12 00:23:21 INFO Completed upload id=2 objects=26 duration=5.358378ms27292026/11/12 00:23:21 INFO Vacuumed table table=objects27302026/11/12 00:23:21 INFO Received complete upload request method=POST path=/27312026/11/12 00:23:21 INFO Vacuumed table table=pending_closures27322026/11/12 00:23:21 INFO Vacuumed table table=pending_objects27332026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads27342026/11/12 00:23:21 INFO Received uploads request method=POST path=/27352026/11/12 00:23:21 INFO Received complete upload request method=POST path=/27362026/11/12 00:23:21 INFO Completed upload id=10 objects=6 duration=2.779649ms27372026/11/12 00:23:21 INFO Vacuumed table table=closures27382026/11/12 00:23:21 INFO Vacuumed table table=objects27392026/11/12 00:23:21 INFO Completed upload id=9 objects=7 duration=2.401769ms27402026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027412026/11/12 00:23:21 INFO Received uploads request method=POST path=/27422026/11/12 00:23:21 INFO Received push request method=POST path=/27432026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027442026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=027452026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period27462026/11/12 00:23:21 INFO Vacuumed table table=pending_closures27472026/11/12 00:23:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27482026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27492026/11/12 00:23:21 INFO Vacuumed table table=pending_objects27502026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027512026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads27522026/11/12 00:23:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27532026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."27542026/11/12 00:23:21 INFO Vacuumed table table=pending_closures27552026/11/12 00:23:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."27562026/11/12 00:23:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."27572026/11/12 00:23:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27582026/11/12 00:23:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27592026/11/12 00:23:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27602026/11/12 00:23:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27612026/11/12 00:23:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27622026/11/12 00:23:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27632026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27642026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27652026/11/12 00:23:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."27662026/11/12 00:23:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."27672026/11/12 00:23:21 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."27682026/11/12 00:23:21 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27692026/11/12 00:23:21 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27702026/11/12 00:23:21 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27712026/11/12 00:23:21 INFO Vacuumed table table=closures27722026/11/12 00:23:21 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27732026/11/12 00:23:21 INFO Received push request method=POST path=/27742026/11/12 00:23:21 INFO Received complete push request method=POST path=/27752026/11/12 00:23:21 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."27762026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27772026/11/12 00:23:21 INFO Received uploads request method=POST path=/27782026/11/12 00:23:21 INFO Vacuumed table table=pending_objects27792026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27802026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."27812026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."27822026/11/12 00:23:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27832026/11/12 00:23:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27842026/11/12 00:23:21 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27852026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads27862026/11/12 00:23:21 INFO Vacuumed table table=objects27872026/11/12 00:23:21 INFO Vacuumed table table=closures27882026/11/12 00:23:21 INFO Vacuumed table table=objects27892026/11/12 00:23:21 INFO Received uploads request method=POST path=/27902026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027912026/11/12 00:23:21 INFO Completed push id=1 objects=7 duration=6.450468ms27922026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=027932026/11/12 00:23:21 INFO Received complete upload request method=POST path=/27942026/11/12 00:23:21 INFO Received uploads request method=POST path=/27952026/11/12 00:23:21 INFO Completed upload id=3 objects=5 duration=1.85523ms27962026/11/12 00:23:21 INFO Received uploads request method=POST path=/27972026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027982026/11/12 00:23:21 INFO Received complete upload request method=POST path=/27992026/11/12 00:23:21 INFO Vacuumed table table=pending_closures28002026/11/12 00:23:21 INFO Received push request method=POST path=/28012026/11/12 00:23:21 INFO Received complete upload request method=POST path=/28022026/11/12 00:23:21 INFO Completed upload id=3 objects=5 duration=1.511349ms28032026/11/12 00:23:21 INFO Vacuumed table table=pending_objects28042026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads28052026/11/12 00:23:21 INFO Vacuumed table table=closures28062026/11/12 00:23:21 INFO Received uploads request method=POST path=/28072026/11/12 00:23:21 INFO Completed upload id=3 objects=8 duration=2.654119ms28082026/11/12 00:23:21 INFO Received complete push request method=POST path=/28092026/11/12 00:23:21 INFO Vacuumed table table=objects28102026/11/12 00:23:21 INFO Received complete push request method=POST path=/28112026/11/12 00:23:21 INFO Received push request method=POST path=/28122026/11/12 00:23:21 INFO Completed push id=2 objects=7 duration=5.016338ms28132026/11/12 00:23:21 INFO Completed push id=3 objects=22 duration=6.916868ms28142026/11/12 00:23:21 INFO Received uploads request method=POST path=/28152026/11/12 00:23:21 INFO Received push request method=POST path=/28162026/11/12 00:23:21 INFO Received uploads request method=POST path=/28172026/11/12 00:23:21 INFO Received push request method=POST path=/28182026/11/12 00:23:21 INFO Received push request method=POST path=/28192026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=028202026/11/12 00:23:21 INFO Received push request method=POST path=/28212026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028222026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=028232026/11/12 00:23:21 INFO Vacuumed table table=pending_closures28242026/11/12 00:23:21 INFO Vacuumed table table=pending_objects28252026/11/12 00:23:21 INFO Received push request method=POST path=/28262026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads28272026/11/12 00:23:21 INFO Received push request method=POST path=/28282026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028292026/11/12 00:23:21 INFO Vacuumed table table=closures28302026/11/12 00:23:21 INFO Vacuumed table table=pending_closures28312026/11/12 00:23:21 INFO Received uploads request method=POST path=/28322026/11/12 00:23:21 INFO Vacuumed table table=objects28332026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028342026/11/12 00:23:21 INFO Vacuumed table table=pending_objects28352026/11/12 00:23:21 INFO Received complete push request method=POST path=/28362026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads28372026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=028382026/11/12 00:23:21 INFO Vacuumed table table=closures28392026/11/12 00:23:21 INFO Vacuumed table table=pending_closures28402026/11/12 00:23:21 INFO Vacuumed table table=objects28412026/11/12 00:23:21 INFO Completed push id=12 objects=3 duration=2.390749ms28422026/11/12 00:23:21 INFO Vacuumed table table=pending_objects28432026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads28442026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=028452026/11/12 00:23:21 INFO Received complete push request method=POST path=/28462026/11/12 00:23:21 INFO Received complete upload request method=POST path=/28472026/11/12 00:23:21 INFO Vacuumed table table=closures28482026/11/12 00:23:21 INFO Received uploads request method=POST path=/28492026/11/12 00:23:21 INFO Vacuumed table table=objects28502026/11/12 00:23:21 INFO Received complete upload request method=POST path=/28512026/11/12 00:23:21 INFO Completed upload id=2 objects=10 duration=2.19139ms28522026/11/12 00:23:21 INFO Received push request method=POST path=/28532026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=028542026/11/12 00:23:21 INFO Completed push id=11 objects=8 duration=3.171069ms28552026/11/12 00:23:21 INFO Completed upload id=3 objects=24 duration=2.093039ms28562026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028572026/11/12 00:23:21 INFO Vacuumed table table=pending_closures28582026/11/12 00:23:21 INFO Received uploads request method=POST path=/28592026/11/12 00:23:21 INFO Vacuumed table table=pending_objects28602026/11/12 00:23:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28612026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28622026/11/12 00:23:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28632026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period28642026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads28652026/11/12 00:23:21 INFO Vacuumed table table=closures28662026/11/12 00:23:21 INFO Vacuumed table table=objects28672026/11/12 00:23:21 INFO Received complete push request method=POST path=/28682026/11/12 00:23:21 INFO Received push request method=POST path=/28692026/11/12 00:23:21 INFO Completed push id=4 objects=3 duration=1.96926ms28702026/11/12 00:23:21 INFO Received uploads request method=POST path=/28712026/11/12 00:23:21 INFO Received push request method=POST path=/28722026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=028732026/11/12 00:23:21 INFO Received uploads request method=POST path=/28742026/11/12 00:23:21 INFO Received push request method=POST path=/28752026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028762026/11/12 00:23:21 INFO Vacuumed table table=pending_closures28772026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=028782026/11/12 00:23:21 INFO Vacuumed table table=pending_objects28792026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads28802026/11/12 00:23:21 INFO Vacuumed table table=closures28812026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period28822026/11/12 00:23:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28832026/11/12 00:23:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28842026/11/12 00:23:21 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28852026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28862026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28872026/11/12 00:23:21 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28882026/11/12 00:23:21 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28892026/11/12 00:23:21 INFO Vacuumed table table=objects28902026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28912026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28922026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28932026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."28942026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028952026/11/12 00:23:21 INFO Received complete push request method=POST path=/28962026/11/12 00:23:21 INFO Received push request method=POST path=/28972026/11/12 00:23:21 INFO Vacuumed table table=pending_closures28982026/11/12 00:23:21 INFO Vacuumed table table=pending_objects28992026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads29002026/11/12 00:23:21 INFO Completed push id=4 objects=4 duration=3.761539ms29012026/11/12 00:23:21 INFO Received uploads request method=POST path=/29022026/11/12 00:23:21 INFO Vacuumed table table=closures29032026/11/12 00:23:21 INFO Vacuumed table table=objects29042026/11/12 00:23:21 INFO Received push request method=POST path=/29052026/11/12 00:23:21 INFO Received push request method=POST path=/29062026/11/12 00:23:21 INFO Received push request method=POST path=/29072026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=029082026/11/12 00:23:21 INFO Received uploads request method=POST path=/29092026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period29102026/11/12 00:23:21 INFO Received complete push request method=POST path=/29112026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=029122026/11/12 00:23:21 INFO Completed push id=2 objects=10 duration=3.553009ms29132026/11/12 00:23:21 INFO Received complete upload request method=POST path=/29142026/11/12 00:23:21 INFO Received push request method=POST path=/29152026/11/12 00:23:21 INFO Completed upload id=11 objects=6 duration=2.115529ms29162026/11/12 00:23:21 INFO Received complete push request method=POST path=/29172026/11/12 00:23:21 INFO Received complete push request method=POST path=/29182026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=029192026/11/12 00:23:21 INFO Received complete push request method=POST path=/29202026/11/12 00:23:21 INFO Received complete push request method=POST path=/29212026/11/12 00:23:21 INFO Completed push id=3 objects=5 duration=2.263409ms29222026/11/12 00:23:21 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29232026/11/12 00:23:21 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29242026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29252026/11/12 00:23:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."29262026/11/12 00:23:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29272026/11/12 00:23:21 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29282026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29292026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29302026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29312026/11/12 00:23:21 INFO Completed push id=6 objects=6 duration=1.886429ms29322026/11/12 00:23:21 INFO Received complete push request method=POST path=/29332026/11/12 00:23:21 INFO Completed push id=13 objects=6 duration=2.783149ms29342026/11/12 00:23:21 INFO Received complete push request method=POST path=/29352026/11/12 00:23:21 INFO Completed push id=5 objects=9 duration=5.084838ms29362026/11/12 00:23:21 INFO Completed push id=8 objects=2 duration=2.295439ms29372026/11/12 00:23:21 INFO Received uploads request method=POST path=/29382026/11/12 00:23:21 INFO Completed push id=10 objects=2 duration=2.341409ms29392026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=029402026/11/12 00:23:21 INFO Received push request method=POST path=/29412026/11/12 00:23:21 INFO Vacuumed table table=pending_closures29422026/11/12 00:23:21 INFO Received complete upload request method=POST path=/29432026/11/12 00:23:21 INFO Vacuumed table table=pending_objects29442026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads29452026/11/12 00:23:21 INFO Completed upload id=2 objects=2 duration=1.945169ms29462026/11/12 00:23:21 INFO Vacuumed table table=closures29472026/11/12 00:23:21 INFO Received push request method=POST path=/29482026/11/12 00:23:21 INFO Vacuumed table table=objects29492026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=029502026/11/12 00:23:21 INFO Received uploads request method=POST path=/29512026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period29522026/11/12 00:23:21 INFO Received push request method=POST path=/29532026/11/12 00:23:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29542026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29552026/11/12 00:23:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."29562026/11/12 00:23:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."29572026/11/12 00:23:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."29582026/11/12 00:23:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."29592026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29602026/11/12 00:23:21 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29612026/11/12 00:23:21 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29622026/11/12 00:23:21 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."29632026/11/12 00:23:21 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."29642026/11/12 00:23:21 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29652026/11/12 00:23:21 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29662026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."29672026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."29682026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."29692026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29702026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29712026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."29722026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29732026/11/12 00:23:21 INFO Received push request method=POST path=/29742026/11/12 00:23:21 INFO Received complete push request method=POST path=/29752026/11/12 00:23:21 INFO Received uploads request method=POST path=/29762026/11/12 00:23:21 INFO Completed push id=7 objects=20 duration=2.809619ms29772026/11/12 00:23:21 INFO Received push request method=POST path=/29782026/11/12 00:23:21 INFO Received push request method=POST path=/29792026/11/12 00:23:21 INFO Received complete upload request method=POST path=/29802026/11/12 00:23:21 INFO Received push request method=POST path=/29812026/11/12 00:23:21 INFO Completed upload id=12 objects=21 duration=1.127739ms29822026/11/12 00:23:21 INFO Received push request method=POST path=/29832026/11/12 00:23:21 INFO Received push request method=POST path=/29842026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=029852026/11/12 00:23:21 INFO Received push request method=POST path=/29862026/11/12 00:23:21 INFO Received complete push request method=POST path=/29872026/11/12 00:23:21 INFO Received complete upload request method=POST path=/29882026/11/12 00:23:21 INFO Vacuumed table table=pending_closures29892026/11/12 00:23:21 INFO Completed upload id=16 objects=23 duration=1.914949ms29902026/11/12 00:23:21 INFO Completed push id=6 objects=5 duration=2.891869ms29912026/11/12 00:23:21 INFO Vacuumed table table=pending_objects29922026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads29932026/11/12 00:23:21 INFO Received uploads request method=POST path=/29942026/11/12 00:23:21 INFO Vacuumed table table=closures29952026/11/12 00:23:21 INFO Vacuumed table table=objects29962026/11/12 00:23:21 INFO Received push request method=POST path=/29972026/11/12 00:23:21 INFO Received complete push request method=POST path=/29982026/11/12 00:23:21 INFO Received push request method=POST path=/29992026/11/12 00:23:21 INFO Completed push id=8 objects=7 duration=2.570129ms30002026/11/12 00:23:21 INFO Received push request method=POST path=/30012026/11/12 00:23:21 INFO Received uploads request method=POST path=/30022026/11/12 00:23:21 INFO Received uploads request method=POST path=/30032026/11/12 00:23:21 INFO Received push request method=POST path=/30042026/11/12 00:23:21 INFO Received push request method=POST path=/30052026/11/12 00:23:21 INFO Received push request method=POST path=/30062026/11/12 00:23:21 INFO Received complete upload request method=POST path=/30072026/11/12 00:23:21 INFO Completed upload id=15 objects=6 duration=1.379539ms30082026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030092026/11/12 00:23:21 INFO Received complete push request method=POST path=/30102026/11/12 00:23:21 INFO Received complete upload request method=POST path=/30112026/11/12 00:23:21 INFO Completed push id=7 objects=27 duration=2.301599ms30122026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030132026/11/12 00:23:21 INFO Completed upload id=4 objects=14 duration=3.553869ms30142026/11/12 00:23:21 INFO Vacuumed table table=pending_closures30152026/11/12 00:23:21 INFO Received complete push request method=POST path=/30162026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030172026/11/12 00:23:21 INFO Received uploads request method=POST path=/30182026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period30192026/11/12 00:23:21 INFO Received uploads request method=POST path=/30202026/11/12 00:23:21 INFO Vacuumed table table=pending_objects30212026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads30222026/11/12 00:23:21 INFO Completed push id=9 objects=5 duration=2.600229ms30232026/11/12 00:23:21 INFO Vacuumed table table=closures30242026/11/12 00:23:21 INFO Received complete push request method=POST path=/30252026/11/12 00:23:21 INFO Vacuumed table table=objects30262026/11/12 00:23:21 INFO Received complete push request method=POST path=/30272026/11/12 00:23:21 INFO Received complete push request method=POST path=/30282026/11/12 00:23:21 INFO Received push request method=POST path=/30292026/11/12 00:23:21 INFO Completed push id=4 objects=4 duration=2.134239ms30302026/11/12 00:23:21 INFO Completed push id=13 objects=23 duration=1.79148ms30312026/11/12 00:23:21 INFO Received push request method=POST path=/30322026/11/12 00:23:21 INFO Completed push id=17 objects=6 duration=2.659999ms30332026/11/12 00:23:21 INFO Received uploads request method=POST path=/30342026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030352026/11/12 00:23:21 INFO Received uploads request method=POST path=/30362026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period30372026/11/12 00:23:21 INFO Received push request method=POST path=/30382026/11/12 00:23:21 INFO Received uploads request method=POST path=/30392026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030402026/11/12 00:23:21 INFO Received complete push request method=POST path=/30412026/11/12 00:23:21 INFO Vacuumed table table=pending_closures30422026/11/12 00:23:21 INFO Completed push id=8 objects=13 duration=2.029989ms30432026/11/12 00:23:21 INFO Vacuumed table table=pending_objects30442026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads30452026/11/12 00:23:21 INFO Vacuumed table table=closures30462026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=030472026/11/12 00:23:21 INFO Received push request method=POST path=/30482026/11/12 00:23:21 INFO Vacuumed table table=objects30492026/11/12 00:23:21 INFO Vacuumed table table=pending_closures30502026/11/12 00:23:21 INFO Vacuumed table table=pending_objects30512026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads30522026/11/12 00:23:21 INFO Received complete upload request method=POST path=/30532026/11/12 00:23:21 INFO Received complete push request method=POST path=/30542026/11/12 00:23:21 INFO Vacuumed table table=closures30552026/11/12 00:23:21 INFO Received complete upload request method=POST path=/30562026/11/12 00:23:21 INFO Vacuumed table table=objects30572026/11/12 00:23:21 INFO Completed upload id=3 objects=9 duration=1.66165ms30582026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030592026/11/12 00:23:21 INFO Completed upload id=9 objects=16 duration=1.462819ms30602026/11/12 00:23:21 INFO Completed push id=5 objects=3 duration=3.274779ms30612026/11/12 00:23:21 INFO Received push request method=POST path=/30622026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period30632026/11/12 00:23:21 INFO Received complete upload request method=POST path=/30642026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030652026/11/12 00:23:21 INFO Completed upload id=17 objects=7 duration=1.15358ms30662026/11/12 00:23:21 INFO Vacuumed table table=pending_closures30672026/11/12 00:23:21 INFO Vacuumed table table=pending_objects30682026/11/12 00:23:21 INFO Received uploads request method=POST path=/30692026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads30702026/11/12 00:23:21 INFO Received complete push request method=POST path=/30712026/11/12 00:23:21 INFO Vacuumed table table=closures30722026/11/12 00:23:21 INFO Vacuumed table table=objects30732026/11/12 00:23:21 INFO Completed push id=4 objects=7 duration=1.812459ms30742026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030752026/11/12 00:23:21 INFO Received push request method=POST path=/30762026/11/12 00:23:21 INFO Received push request method=POST path=/30772026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030782026/11/12 00:23:21 INFO Received push request method=POST path=/30792026/11/12 00:23:21 INFO Received push request method=POST path=/30802026/11/12 00:23:21 INFO Vacuumed table table=pending_closures30812026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030822026/11/12 00:23:21 INFO Vacuumed table table=pending_objects30832026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads30842026/11/12 00:23:21 INFO Received uploads request method=POST path=/30852026/11/12 00:23:21 INFO Vacuumed table table=closures30862026/11/12 00:23:21 INFO Vacuumed table table=objects30872026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030882026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=030892026/11/12 00:23:21 INFO Received push request method=POST path=/30902026/11/12 00:23:21 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30912026/11/12 00:23:21 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30922026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period30932026/11/12 00:23:21 INFO Received push request method=POST path=/30942026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030952026/11/12 00:23:21 INFO Vacuumed table table=pending_closures30962026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030972026/11/12 00:23:21 INFO Vacuumed table table=pending_objects30982026/11/12 00:23:21 INFO Received complete push request method=POST path=/30992026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads31002026/11/12 00:23:21 INFO Received uploads request method=POST path=/31012026/11/12 00:23:21 INFO Received uploads request method=POST path=/31022026/11/12 00:23:21 INFO Received push request method=POST path=/31032026/11/12 00:23:21 INFO Vacuumed table table=closures31042026/11/12 00:23:21 INFO Vacuumed table table=pending_closures31052026/11/12 00:23:21 INFO Completed push id=5 objects=6 duration=2.098459ms31062026/11/12 00:23:21 INFO Vacuumed table table=objects31072026/11/12 00:23:21 INFO Received uploads request method=POST path=/31082026/11/12 00:23:21 INFO Vacuumed table table=pending_objects31092026/11/12 00:23:21 INFO Received push request method=POST path=/31102026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads31112026/11/12 00:23:21 INFO Vacuumed table table=closures31122026/11/12 00:23:21 INFO Received push request method=POST path=/31132026/11/12 00:23:21 INFO Vacuumed table table=objects31142026/11/12 00:23:21 INFO Received push request method=POST path=/31152026/11/12 00:23:21 INFO Received complete push request method=POST path=/31162026/11/12 00:23:21 INFO Completed push id=10 objects=3 duration=1.79134ms31172026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=031182026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period31192026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031202026/11/12 00:23:21 INFO Vacuumed table table=pending_closures31212026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=031222026/11/12 00:23:21 INFO Received complete push request method=POST path=/31232026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=031242026/11/12 00:23:21 INFO Vacuumed table table=pending_objects31252026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads31262026/11/12 00:23:21 INFO Completed push id=5 objects=3 duration=2.097109ms31272026/11/12 00:23:21 INFO Vacuumed table table=closures31282026/11/12 00:23:21 INFO Received push request method=POST path=/31292026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period31302026/11/12 00:23:21 INFO Vacuumed table table=objects31312026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031322026/11/12 00:23:21 INFO Received push request method=POST path=/31332026/11/12 00:23:21 INFO Vacuumed table table=pending_closures31342026/11/12 00:23:21 INFO Received complete push request method=POST path=/31352026/11/12 00:23:21 INFO Vacuumed table table=pending_objects31362026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads31372026/11/12 00:23:21 INFO Vacuumed table table=closures31382026/11/12 00:23:21 INFO Received complete upload request method=POST path=/31392026/11/12 00:23:21 INFO Completed push id=8 objects=6 duration=1.759169ms31402026/11/12 00:23:21 INFO Vacuumed table table=objects31412026/11/12 00:23:21 INFO Received uploads request method=POST path=/31422026/11/12 00:23:21 INFO Received complete upload request method=POST path=/31432026/11/12 00:23:21 INFO Completed upload id=10 objects=4 duration=2.632509ms31442026/11/12 00:23:21 INFO Received push request method=POST path=/31452026/11/12 00:23:21 INFO Completed upload id=14 objects=6 duration=1.81445ms31462026/11/12 00:23:21 INFO Received complete upload request method=POST path=/31472026/11/12 00:23:21 INFO Completed upload id=10 objects=23 duration=2.726609ms31482026/11/12 00:23:21 INFO Received push request method=POST path=/31492026/11/12 00:23:21 INFO Received push request method=POST path=/31502026/11/12 00:23:21 INFO Received complete push request method=POST path=/31512026/11/12 00:23:21 INFO Received push request method=POST path=/31522026/11/12 00:23:21 INFO Completed push id=13 objects=12 duration=4.188899ms31532026/11/12 00:23:21 INFO Received complete push request method=POST path=/31542026/11/12 00:23:21 INFO Received uploads request method=POST path=/31552026/11/12 00:23:21 INFO Received push request method=POST path=/31562026/11/12 00:23:21 INFO Received push request method=POST path=/31572026/11/12 00:23:21 INFO Completed push id=5 objects=8 duration=3.665669ms31582026/11/12 00:23:21 INFO Received complete push request method=POST path=/31592026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=031602026/11/12 00:23:21 INFO Completed push id=7 objects=7 duration=1.560489ms31612026/11/12 00:23:21 INFO Received uploads request method=POST path=/31622026/11/12 00:23:21 INFO Received push request method=POST path=/31632026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=031642026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031652026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=031662026/11/12 00:23:21 INFO Received push request method=POST path=/31672026/11/12 00:23:21 INFO Received uploads request method=POST path=/31682026/11/12 00:23:21 INFO Vacuumed table table=pending_closures31692026/11/12 00:23:21 INFO Received push request method=POST path=/31702026/11/12 00:23:21 INFO Vacuumed table table=pending_objects31712026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads31722026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=031732026/11/12 00:23:21 INFO Received push request method=POST path=/31742026/11/12 00:23:21 INFO Vacuumed table table=closures31752026/11/12 00:23:21 INFO Received uploads request method=POST path=/31762026/11/12 00:23:21 INFO Vacuumed table table=objects31772026/11/12 00:23:21 INFO Vacuumed table table=pending_closures31782026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=031792026/11/12 00:23:21 INFO Vacuumed table table=pending_objects31802026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads31812026/11/12 00:23:21 INFO Received uploads request method=POST path=/31822026/11/12 00:23:21 INFO Vacuumed table table=closures31832026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period31842026/11/12 00:23:21 INFO Vacuumed table table=objects31852026/11/12 00:23:21 INFO Received complete push request method=POST path=/31862026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031872026/11/12 00:23:21 INFO Received push request method=POST path=/31882026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=031892026/11/12 00:23:21 INFO Received complete upload request method=POST path=/31902026/11/12 00:23:21 INFO Received uploads request method=POST path=/31912026/11/12 00:23:21 INFO Vacuumed table table=pending_closures31922026/11/12 00:23:21 INFO Vacuumed table table=pending_closures31932026/11/12 00:23:21 INFO Received uploads request method=POST path=/31942026/11/12 00:23:21 INFO Completed upload id=7 objects=3 duration=1.428339ms31952026/11/12 00:23:21 INFO Completed push id=11 objects=11 duration=4.369658ms31962026/11/12 00:23:21 INFO Vacuumed table table=pending_objects31972026/11/12 00:23:21 INFO Vacuumed table table=pending_objects31982026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads31992026/11/12 00:23:21 INFO Received push request method=POST path=/32002026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads32012026/11/12 00:23:21 INFO Vacuumed table table=closures32022026/11/12 00:23:21 INFO Vacuumed table table=closures32032026/11/12 00:23:21 INFO Received uploads request method=POST path=/32042026/11/12 00:23:21 INFO Vacuumed table table=objects32052026/11/12 00:23:21 INFO Vacuumed table table=objects32062026/11/12 00:23:21 INFO Received uploads request method=POST path=/32072026/11/12 00:23:21 INFO Received complete upload request method=POST path=/32082026/11/12 00:23:21 INFO Received uploads request method=POST path=/32092026/11/12 00:23:21 INFO Completed upload id=11 objects=6 duration=1.905609ms32102026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=032112026/11/12 00:23:21 INFO Received push request method=POST path=/32122026/11/12 00:23:21 INFO Received uploads request method=POST path=/32132026/11/12 00:23:21 INFO Vacuumed table table=pending_closures32142026/11/12 00:23:21 INFO Received push request method=POST path=/32152026/11/12 00:23:21 INFO Vacuumed table table=pending_objects32162026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads32172026/11/12 00:23:21 INFO Received complete upload request method=POST path=/32182026/11/12 00:23:21 INFO Received uploads request method=POST path=/32192026/11/12 00:23:21 INFO Vacuumed table table=closures32202026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=032212026/11/12 00:23:21 INFO Received uploads request method=POST path=/32222026/11/12 00:23:21 INFO Vacuumed table table=objects32232026/11/12 00:23:21 INFO Received push request method=POST path=/32242026/11/12 00:23:21 INFO Completed upload id=15 objects=4 duration=2.630569ms32252026/11/12 00:23:21 INFO Received complete push request method=POST path=/32262026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032272026/11/12 00:23:21 INFO Received push request method=POST path=/32282026/11/12 00:23:21 INFO Completed push id=18 objects=23 duration=2.394169ms32292026/11/12 00:23:21 INFO Vacuumed table table=pending_closures32302026/11/12 00:23:21 INFO Received complete upload request method=POST path=/32312026/11/12 00:23:21 INFO Received push request method=POST path=/32322026/11/12 00:23:21 INFO Vacuumed table table=pending_objects32332026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads32342026/11/12 00:23:21 INFO Completed upload id=18 objects=11 duration=1.458939ms32352026/11/12 00:23:21 INFO Vacuumed table table=closures32362026/11/12 00:23:21 INFO Vacuumed table table=objects32372026/11/12 00:23:21 INFO Received push request method=POST path=/32382026/11/12 00:23:21 INFO Received push request method=POST path=/32392026/11/12 00:23:21 INFO Received complete push request method=POST path=/32402026/11/12 00:23:21 INFO Received push request method=POST path=/32412026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=032422026/11/12 00:23:21 INFO Received push request method=POST path=/32432026/11/12 00:23:21 INFO Completed push id=15 objects=24 duration=4.205889ms32442026/11/12 00:23:21 INFO Received complete push request method=POST path=/32452026/11/12 00:23:21 WARN Force mode enabled - objects will be deleted immediately without grace period32462026/11/12 00:23:21 INFO Completed push id=4 objects=6 duration=2.546379ms32472026/11/12 00:23:21 INFO Received complete upload request method=POST path=/32482026/11/12 00:23:21 INFO Received uploads request method=POST path=/32492026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032502026/11/12 00:23:21 INFO Completed upload id=11 objects=14 duration=1.72263ms32512026/11/12 00:23:21 INFO Received uploads request method=POST path=/32522026/11/12 00:23:21 INFO Vacuumed table table=pending_closures32532026/11/12 00:23:21 INFO Vacuumed table table=pending_objects32542026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads32552026/11/12 00:23:21 INFO Vacuumed table table=closures32562026/11/12 00:23:21 INFO Vacuumed table table=objects32572026/11/12 00:23:21 INFO Received complete push request method=POST path=/32582026/11/12 00:23:21 INFO Completed push id=12 objects=15 duration=2.33235ms32592026/11/12 00:23:21 INFO Received uploads request method=POST path=/32602026/11/12 00:23:21 INFO Received uploads request method=POST path=/32612026/11/12 00:23:21 INFO Received uploads request method=POST path=/32622026/11/12 00:23:21 INFO Received push request method=POST path=/32632026/11/12 00:23:21 INFO Received complete upload request method=POST path=/32642026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=032652026/11/12 00:23:21 INFO Completed upload id=10 objects=17 duration=1.834959ms32662026/11/12 00:23:21 INFO Received push request method=POST path=/32672026/11/12 00:23:21 INFO Received uploads request method=POST path=/32682026/11/12 00:23:21 INFO Received push request method=POST path=/32692026/11/12 00:23:21 INFO Received push request method=POST path=/32702026/11/12 00:23:21 INFO Received uploads request method=POST path=/32712026/11/12 00:23:21 INFO Received push request method=POST path=/32722026/11/12 00:23:21 INFO Received complete push request method=POST path=/32732026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032742026/11/12 00:23:21 INFO Completed push id=11 objects=9 duration=2.062139ms32752026/11/12 00:23:21 INFO Received complete push request method=POST path=/32762026/11/12 00:23:21 INFO Received uploads request method=POST path=/32772026/11/12 00:23:21 INFO Vacuumed table table=pending_closures32782026/11/12 00:23:21 INFO Vacuumed table table=pending_objects32792026/11/12 00:23:21 INFO Received push request method=POST path=/32802026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads32812026/11/12 00:23:21 INFO Vacuumed table table=closures32822026/11/12 00:23:21 INFO Completed push id=19 objects=6 duration=3.915558ms32832026/11/12 00:23:21 INFO Received complete upload request method=POST path=/32842026/11/12 00:23:21 INFO Vacuumed table table=objects32852026/11/12 00:23:21 INFO Completed upload id=16 objects=3 duration=1.1242ms32862026/11/12 00:23:21 INFO Received uploads request method=POST path=/32872026/11/12 00:23:21 INFO Received complete upload request method=POST path=/32882026/11/12 00:23:21 INFO Received uploads request method=POST path=/32892026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=032902026/11/12 00:23:21 INFO Completed upload id=9 objects=2 duration=1.58584ms32912026/11/12 00:23:21 INFO Aborted multipart uploads count=0 kept=032922026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032932026/11/12 00:23:21 INFO Vacuumed table table=pending_closures32942026/11/12 00:23:21 INFO Vacuumed table table=pending_objects32952026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads32962026/11/12 00:23:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032972026/11/12 00:23:21 INFO Received push request method=POST path=/32982026/11/12 00:23:21 INFO Vacuumed table table=closures32992026/11/12 00:23:21 INFO Received push request method=POST path=/33002026/11/12 00:23:21 INFO Vacuumed table table=pending_closures33012026/11/12 00:23:21 INFO Vacuumed table table=objects33022026/11/12 00:23:21 INFO Vacuumed table table=pending_objects33032026/11/12 00:23:21 INFO Received push request method=POST path=/33042026/11/12 00:23:21 INFO Vacuumed table table=multipart_uploads33052026/11/12 00:23:21 INFO Vacuumed table table=closures33062026/11/12 00:23:22 INFO Received push request method=POST path=/33072026/11/12 00:23:22 INFO Vacuumed table table=objects33082026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033092026/11/12 00:23:22 INFO Received push request method=POST path=/33102026/11/12 00:23:22 INFO Received push request method=POST path=/33112026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033122026/11/12 00:23:22 INFO Received complete push request method=POST path=/33132026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period33142026/11/12 00:23:22 INFO Received uploads request method=POST path=/33152026/11/12 00:23:22 INFO Completed push id=8 objects=3 duration=2.083829ms33162026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033172026/11/12 00:23:22 INFO Received push request method=POST path=/33182026/11/12 00:23:22 INFO Vacuumed table table=pending_closures33192026/11/12 00:23:22 INFO Vacuumed table table=pending_objects33202026/11/12 00:23:22 INFO Received push request method=POST path=/33212026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads33222026/11/12 00:23:22 INFO Received uploads request method=POST path=/33232026/11/12 00:23:22 INFO Vacuumed table table=closures33242026/11/12 00:23:22 INFO Received push request method=POST path=/33252026/11/12 00:23:22 INFO Vacuumed table table=objects33262026/11/12 00:23:22 INFO Received push request method=POST path=/33272026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033282026/11/12 00:23:22 INFO Received complete upload request method=POST path=/33292026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period33302026/11/12 00:23:22 INFO Received push request method=POST path=/33312026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033322026/11/12 00:23:22 INFO Completed upload id=8 objects=3 duration=2.292949ms33332026/11/12 00:23:22 INFO Received push request method=POST path=/33342026/11/12 00:23:22 INFO Vacuumed table table=pending_closures33352026/11/12 00:23:22 INFO Vacuumed table table=pending_objects33362026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads33372026/11/12 00:23:22 INFO Received uploads request method=POST path=/33382026/11/12 00:23:22 INFO Vacuumed table table=closures33392026/11/12 00:23:22 INFO Vacuumed table table=objects33402026/11/12 00:23:22 INFO Received push request method=POST path=/33412026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033422026/11/12 00:23:22 INFO Received complete push request method=POST path=/33432026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period33442026/11/12 00:23:22 INFO Received uploads request method=POST path=/33452026/11/12 00:23:22 INFO Completed push id=8 objects=6 duration=3.976628ms33462026/11/12 00:23:22 INFO Received uploads request method=POST path=/33472026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33482026/11/12 00:23:22 INFO Received uploads request method=POST path=/33492026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033502026/11/12 00:23:22 INFO Received uploads request method=POST path=/33512026/11/12 00:23:22 INFO Received complete upload request method=POST path=/33522026/11/12 00:23:22 INFO Received uploads request method=POST path=/33532026/11/12 00:23:22 INFO Completed upload id=6 objects=6 duration=1.194379ms33542026/11/12 00:23:22 INFO Received complete push request method=POST path=/33552026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033562026/11/12 00:23:22 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33572026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33582026/11/12 00:23:22 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33592026/11/12 00:23:22 INFO Received uploads request method=POST path=/33602026/11/12 00:23:22 INFO Vacuumed table table=pending_closures33612026/11/12 00:23:22 INFO Completed push id=14 objects=6 duration=2.269709ms33622026/11/12 00:23:22 INFO Vacuumed table table=pending_objects33632026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads33642026/11/12 00:23:22 INFO Received push request method=POST path=/33652026/11/12 00:23:22 INFO Vacuumed table table=closures33662026/11/12 00:23:22 INFO Vacuumed table table=objects33672026/11/12 00:23:22 INFO Received push request method=POST path=/33682026/11/12 00:23:22 INFO Received uploads request method=POST path=/33692026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033702026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period33712026/11/12 00:23:22 INFO Received complete upload request method=POST path=/33722026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033732026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033742026/11/12 00:23:22 INFO Completed upload id=13 objects=10 duration=1.34611ms33752026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033762026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period33772026/11/12 00:23:22 INFO Received push request method=POST path=/33782026/11/12 00:23:22 INFO Received uploads request method=POST path=/33792026/11/12 00:23:22 INFO Vacuumed table table=pending_closures33802026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period33812026/11/12 00:23:22 INFO Received push request method=POST path=/33822026/11/12 00:23:22 INFO Vacuumed table table=pending_objects33832026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads33842026/11/12 00:23:22 INFO Vacuumed table table=closures33852026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033862026/11/12 00:23:22 INFO Vacuumed table table=objects33872026/11/12 00:23:22 INFO Received push request method=POST path=/33882026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=033892026/11/12 00:23:22 INFO Vacuumed table table=pending_closures33902026/11/12 00:23:22 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33912026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33922026/11/12 00:23:22 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33932026/11/12 00:23:22 INFO Received uploads request method=POST path=/33942026/11/12 00:23:22 INFO Vacuumed table table=pending_objects33952026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads33962026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period33972026/11/12 00:23:22 INFO Vacuumed table table=closures33982026/11/12 00:23:22 INFO Vacuumed table table=objects33992026/11/12 00:23:22 INFO Received complete upload request method=POST path=/34002026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034012026/11/12 00:23:22 INFO Completed upload id=18 objects=4 duration=2.252609ms34022026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34032026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034042026/11/12 00:23:22 INFO Received uploads request method=POST path=/34052026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34062026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34072026/11/12 00:23:22 INFO Received complete push request method=POST path=/34082026/11/12 00:23:22 INFO Vacuumed table table=closures34092026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period34102026/11/12 00:23:22 INFO Completed push id=15 objects=4 duration=1.37712ms34112026/11/12 00:23:22 INFO Vacuumed table table=objects34122026/11/12 00:23:22 INFO Received push request method=POST path=/34132026/11/12 00:23:22 INFO Received complete push request method=POST path=/34142026/11/12 00:23:22 INFO Received complete upload request method=POST path=/34152026/11/12 00:23:22 INFO Received uploads request method=POST path=/34162026/11/12 00:23:22 INFO Completed upload id=17 objects=6 duration=2.764659ms34172026/11/12 00:23:22 INFO Completed push id=14 objects=3 duration=3.383819ms34182026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034192026/11/12 00:23:22 INFO Received uploads request method=POST path=/34202026/11/12 00:23:22 INFO Received uploads request method=POST path=/34212026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period34222026/11/12 00:23:22 INFO Received push request method=POST path=/34232026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034242026/11/12 00:23:22 INFO Received complete push request method=POST path=/34252026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34262026/11/12 00:23:22 INFO Completed push id=6 objects=6 duration=2.613689ms34272026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34282026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34292026/11/12 00:23:22 INFO Received push request method=POST path=/34302026/11/12 00:23:22 INFO Vacuumed table table=closures34312026/11/12 00:23:22 INFO Received uploads request method=POST path=/34322026/11/12 00:23:22 INFO Vacuumed table table=objects34332026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034342026/11/12 00:23:22 INFO Received uploads request method=POST path=/34352026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=034362026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34372026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034382026/11/12 00:23:22 INFO Received push request method=POST path=/34392026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34402026/11/12 00:23:22 INFO Received uploads request method=POST path=/34412026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34422026/11/12 00:23:22 INFO Received complete push request method=POST path=/34432026/11/12 00:23:22 INFO Vacuumed table table=closures34442026/11/12 00:23:22 INFO Received uploads request method=POST path=/34452026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034462026/11/12 00:23:22 INFO Vacuumed table table=objects34472026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34482026/11/12 00:23:22 INFO Completed push id=17 objects=6 duration=2.320949ms34492026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34502026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034512026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34522026/11/12 00:23:22 INFO Vacuumed table table=closures34532026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034542026/11/12 00:23:22 INFO Received complete push request method=POST path=/34552026/11/12 00:23:22 INFO Vacuumed table table=objects34562026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period34572026/11/12 00:23:22 INFO Completed push id=5 objects=6 duration=1.864999ms34582026/11/12 00:23:22 INFO Received push request method=POST path=/34592026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034602026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034612026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34622026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34632026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34642026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34652026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34662026/11/12 00:23:22 INFO Vacuumed table table=closures34672026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34682026/11/12 00:23:22 INFO Vacuumed table table=closures34692026/11/12 00:23:22 INFO Vacuumed table table=objects34702026/11/12 00:23:22 INFO Received complete push request method=POST path=/34712026/11/12 00:23:22 INFO Vacuumed table table=objects34722026/11/12 00:23:22 INFO Received push request method=POST path=/34732026/11/12 00:23:22 INFO Completed push id=15 objects=7 duration=2.490959ms34742026/11/12 00:23:22 INFO Received push request method=POST path=/34752026/11/12 00:23:22 INFO Received complete push request method=POST path=/34762026/11/12 00:23:22 INFO Completed push id=9 objects=24 duration=2.09792ms34772026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=034782026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034792026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34802026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period34812026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34822026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034832026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34842026/11/12 00:23:22 INFO Vacuumed table table=pending_closures34852026/11/12 00:23:22 INFO Vacuumed table table=closures34862026/11/12 00:23:22 INFO Vacuumed table table=pending_objects34872026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads34882026/11/12 00:23:22 INFO Vacuumed table table=objects34892026/11/12 00:23:22 INFO Vacuumed table table=closures34902026/11/12 00:23:22 INFO Vacuumed table table=objects34912026/11/12 00:23:22 INFO Received push request method=POST path=/34922026/11/12 00:23:22 INFO Received uploads request method=POST path=/34932026/11/12 00:23:22 INFO Received complete upload request method=POST path=/34942026/11/12 00:23:22 INFO Received complete push request method=POST path=/34952026/11/12 00:23:22 INFO Completed push id=14 objects=3 duration=1.197109ms34962026/11/12 00:23:22 INFO Completed upload id=23 objects=15 duration=2.603159ms34972026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=034982026/11/12 00:23:22 INFO Received push request method=POST path=/34992026/11/12 00:23:22 INFO Received push request method=POST path=/35002026/11/12 00:23:22 INFO Received uploads request method=POST path=/35012026/11/12 00:23:22 INFO Received complete push request method=POST path=/35022026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period35032026/11/12 00:23:22 INFO Completed push id=10 objects=15 duration=2.354009ms35042026/11/12 00:23:22 INFO Received complete push request method=POST path=/35052026/11/12 00:23:22 INFO Received push request method=POST path=/35062026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035072026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35082026/11/12 00:23:22 INFO Vacuumed table table=pending_closures35092026/11/12 00:23:22 INFO Completed push id=9 objects=7 duration=4.191068ms35102026/11/12 00:23:22 INFO Vacuumed table table=pending_objects35112026/11/12 00:23:22 INFO Completed upload id=10 objects=6 duration=2.880869ms35122026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads35132026/11/12 00:23:22 INFO Received uploads request method=POST path=/35142026/11/12 00:23:22 INFO Vacuumed table table=closures35152026/11/12 00:23:22 INFO Vacuumed table table=objects35162026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35172026/11/12 00:23:22 INFO Received complete push request method=POST path=/35182026/11/12 00:23:22 INFO Completed upload id=15 objects=6 duration=1.076939ms35192026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=035202026/11/12 00:23:22 INFO Completed push id=9 objects=4 duration=3.479398ms35212026/11/12 00:23:22 INFO Received uploads request method=POST path=/35222026/11/12 00:23:22 INFO Received uploads request method=POST path=/35232026/11/12 00:23:22 INFO Received push request method=POST path=/35242026/11/12 00:23:22 INFO Received push request method=POST path=/35252026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035262026/11/12 00:23:22 INFO Vacuumed table table=pending_closures35272026/11/12 00:23:22 INFO Vacuumed table table=pending_objects35282026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads35292026/11/12 00:23:22 INFO Vacuumed table table=closures35302026/11/12 00:23:22 INFO Received push request method=POST path=/35312026/11/12 00:23:22 INFO Vacuumed table table=objects35322026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35332026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35342026/11/12 00:23:22 INFO Completed upload id=10 objects=7 duration=2.257069ms35352026/11/12 00:23:22 INFO Received uploads request method=POST path=/35362026/11/12 00:23:22 INFO Completed upload id=19 objects=6 duration=1.65463ms35372026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35382026/11/12 00:23:22 INFO Completed upload id=10 objects=10 duration=1.1634ms35392026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35402026/11/12 00:23:22 INFO Completed upload id=20 objects=4 duration=1.476189ms35412026/11/12 00:23:22 INFO Received uploads request method=POST path=/35422026/11/12 00:23:22 INFO Received complete push request method=POST path=/35432026/11/12 00:23:22 INFO Received push request method=POST path=/35442026/11/12 00:23:22 INFO Completed push id=14 objects=4 duration=3.860499ms35452026/11/12 00:23:22 INFO Received uploads request method=POST path=/35462026/11/12 00:23:22 INFO Received uploads request method=POST path=/35472026/11/12 00:23:22 INFO Received push request method=POST path=/35482026/11/12 00:23:22 INFO Received complete push request method=POST path=/35492026/11/12 00:23:22 INFO Received push request method=POST path=/35502026/11/12 00:23:22 INFO Completed push id=12 objects=2 duration=3.401698ms35512026/11/12 00:23:22 INFO Received complete push request method=POST path=/35522026/11/12 00:23:22 INFO Received push request method=POST path=/35532026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35542026/11/12 00:23:22 INFO Completed upload id=22 objects=7 duration=1.265299ms35552026/11/12 00:23:22 INFO Completed push id=3 objects=21 duration=4.483238ms35562026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=035572026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period35582026/11/12 00:23:22 INFO Received complete push request method=POST path=/35592026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35602026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035612026/11/12 00:23:22 INFO Completed push id=8 objects=6 duration=2.160509ms35622026/11/12 00:23:22 INFO Vacuumed table table=pending_closures35632026/11/12 00:23:22 INFO Received complete push request method=POST path=/35642026/11/12 00:23:22 INFO Completed upload id=22 objects=2 duration=1.937959ms35652026/11/12 00:23:22 INFO Received push request method=POST path=/35662026/11/12 00:23:22 INFO Received uploads request method=POST path=/35672026/11/12 00:23:22 INFO Received uploads request method=POST path=/35682026/11/12 00:23:22 INFO Vacuumed table table=pending_objects35692026/11/12 00:23:22 INFO Completed push id=16 objects=5 duration=1.626439ms35702026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads35712026/11/12 00:23:22 INFO Vacuumed table table=closures35722026/11/12 00:23:22 INFO Vacuumed table table=objects35732026/11/12 00:23:22 INFO Received uploads request method=POST path=/35742026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=035752026/11/12 00:23:22 INFO Received uploads request method=POST path=/35762026/11/12 00:23:22 INFO Received complete push request method=POST path=/35772026/11/12 00:23:22 INFO Received uploads request method=POST path=/35782026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period35792026/11/12 00:23:22 INFO Completed push id=16 objects=27 duration=2.250649ms35802026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=035812026/11/12 00:23:22 INFO Received uploads request method=POST path=/35822026/11/12 00:23:22 INFO Received uploads request method=POST path=/35832026/11/12 00:23:22 INFO Received complete push request method=POST path=/35842026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035852026/11/12 00:23:22 INFO Received push request method=POST path=/35862026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=035872026/11/12 00:23:22 INFO Completed push id=12 objects=7 duration=1.992599ms35882026/11/12 00:23:22 INFO Vacuumed table table=pending_closures35892026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period35902026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035912026/11/12 00:23:22 INFO Vacuumed table table=pending_objects35922026/11/12 00:23:22 INFO Received complete upload request method=POST path=/35932026/11/12 00:23:22 INFO Vacuumed table table=pending_closures35942026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads35952026/11/12 00:23:22 INFO Received push request method=POST path=/35962026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035972026/11/12 00:23:22 INFO Vacuumed table table=closures35982026/11/12 00:23:22 INFO Completed upload id=15 objects=9 duration=1.490989ms35992026/11/12 00:23:22 INFO Vacuumed table table=pending_closures36002026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36012026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36022026/11/12 00:23:22 INFO Vacuumed table table=objects36032026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36042026/11/12 00:23:22 INFO Vacuumed table table=closures36052026/11/12 00:23:22 INFO Received uploads request method=POST path=/36062026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36072026/11/12 00:23:22 INFO Received complete upload request method=POST path=/36082026/11/12 00:23:22 INFO Vacuumed table table=closures36092026/11/12 00:23:22 INFO Vacuumed table table=objects36102026/11/12 00:23:22 INFO Completed upload id=18 objects=4 duration=1.57529ms36112026/11/12 00:23:22 INFO Vacuumed table table=objects36122026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=036132026/11/12 00:23:22 INFO Received complete push request method=POST path=/36142026/11/12 00:23:22 INFO Received uploads request method=POST path=/36152026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=036162026/11/12 00:23:22 INFO Received push request method=POST path=/36172026/11/12 00:23:22 INFO Received complete push request method=POST path=/36182026/11/12 00:23:22 INFO Completed push id=6 objects=26 duration=5.163488ms36192026/11/12 00:23:22 INFO Completed push id=24 objects=9 duration=2.08945ms36202026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036212026/11/12 00:23:22 INFO Vacuumed table table=pending_closures36222026/11/12 00:23:22 INFO Received push request method=POST path=/36232026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36242026/11/12 00:23:22 INFO Received uploads request method=POST path=/36252026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36262026/11/12 00:23:22 INFO Received uploads request method=POST path=/36272026/11/12 00:23:22 INFO Vacuumed table table=closures36282026/11/12 00:23:22 INFO Vacuumed table table=objects36292026/11/12 00:23:22 INFO Received push request method=POST path=/36302026/11/12 00:23:22 INFO Received push request method=POST path=/36312026/11/12 00:23:22 INFO Received push request method=POST path=/36322026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=036332026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036342026/11/12 00:23:22 INFO Vacuumed table table=pending_closures36352026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36362026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36372026/11/12 00:23:22 INFO Vacuumed table table=closures36382026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036392026/11/12 00:23:22 INFO Vacuumed table table=objects36402026/11/12 00:23:22 INFO Received push request method=POST path=/36412026/11/12 00:23:22 INFO Vacuumed table table=pending_closures36422026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=036432026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36442026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36452026/11/12 00:23:22 INFO Vacuumed table table=closures36462026/11/12 00:23:22 INFO Received uploads request method=POST path=/36472026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=036482026/11/12 00:23:22 INFO Vacuumed table table=objects36492026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period36502026/11/12 00:23:22 INFO Received complete push request method=POST path=/36512026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036522026/11/12 00:23:22 INFO Vacuumed table table=pending_closures36532026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036542026/11/12 00:23:22 INFO Completed push id=21 objects=17 duration=2.207099ms36552026/11/12 00:23:22 INFO Received complete push request method=POST path=/36562026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36572026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36582026/11/12 00:23:22 INFO Vacuumed table table=pending_closures36592026/11/12 00:23:22 INFO Vacuumed table table=closures36602026/11/12 00:23:22 INFO Vacuumed table table=objects36612026/11/12 00:23:22 INFO Completed push id=17 objects=22 duration=3.029869ms36622026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36632026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36642026/11/12 00:23:22 INFO Vacuumed table table=closures36652026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=036662026/11/12 00:23:22 INFO Received complete push request method=POST path=/36672026/11/12 00:23:22 INFO Received uploads request method=POST path=/36682026/11/12 00:23:22 INFO Vacuumed table table=objects36692026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period36702026/11/12 00:23:22 INFO Received push request method=POST path=/36712026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036722026/11/12 00:23:22 INFO Completed push id=18 objects=10 duration=1.906819ms36732026/11/12 00:23:22 INFO Vacuumed table table=pending_closures36742026/11/12 00:23:22 INFO Received push request method=POST path=/36752026/11/12 00:23:22 INFO Vacuumed table table=pending_objects36762026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads36772026/11/12 00:23:22 INFO Vacuumed table table=closures36782026/11/12 00:23:22 INFO Vacuumed table table=objects36792026/11/12 00:23:22 INFO Received uploads request method=POST path=/36802026/11/12 00:23:22 INFO Received push request method=POST path=/36812026/11/12 00:23:22 INFO Received complete push request method=POST path=/36822026/11/12 00:23:22 INFO Completed push id=19 objects=7 duration=1.745739ms36832026/11/12 00:23:22 INFO Received push request method=POST path=/36842026/11/12 00:23:22 INFO Received uploads request method=POST path=/36852026/11/12 00:23:22 INFO Received push request method=POST path=/36862026/11/12 00:23:22 INFO Received complete push request method=POST path=/36872026/11/12 00:23:22 INFO Completed push id=14 objects=3 duration=1.38103ms36882026/11/12 00:23:22 INFO Received uploads request method=POST path=/36892026/11/12 00:23:22 INFO Received complete upload request method=POST path=/36902026/11/12 00:23:22 INFO Received complete push request method=POST path=/36912026/11/12 00:23:22 INFO Received uploads request method=POST path=/36922026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=036932026/11/12 00:23:22 INFO Completed push id=15 objects=5 duration=2.116129ms36942026/11/12 00:23:22 INFO Completed upload id=23 objects=16 duration=3.216138ms36952026/11/12 00:23:22 INFO Received push request method=POST path=/36962026/11/12 00:23:22 INFO Received uploads request method=POST path=/36972026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period36982026/11/12 00:23:22 INFO Received uploads request method=POST path=/36992026/11/12 00:23:22 INFO Received push request method=POST path=/37002026/11/12 00:23:22 INFO Received push request method=POST path=/37012026/11/12 00:23:22 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."37022026/11/12 00:23:22 INFO Received push request method=POST path=/37032026/11/12 00:23:22 INFO Received uploads request method=POST path=/37042026/11/12 00:23:22 INFO Received uploads request method=POST path=/37052026/11/12 00:23:22 INFO Received complete push request method=POST path=/37062026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037072026/11/12 00:23:22 INFO Received push request method=POST path=/37082026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037092026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period37102026/11/12 00:23:22 INFO Received uploads request method=POST path=/37112026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037122026/11/12 00:23:22 INFO Completed push id=16 objects=16 duration=4.126739ms37132026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037142026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037152026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37162026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37172026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037182026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37192026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37202026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37212026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37222026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period37232026/11/12 00:23:22 INFO Vacuumed table table=closures37242026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037252026/11/12 00:23:22 INFO Vacuumed table table=closures37262026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037272026/11/12 00:23:22 INFO Vacuumed table table=objects37282026/11/12 00:23:22 INFO Vacuumed table table=objects37292026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037302026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037312026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37322026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period37332026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37342026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037352026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37362026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37372026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37382026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37392026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037402026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37412026/11/12 00:23:22 INFO Vacuumed table table=closures37422026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037432026/11/12 00:23:22 INFO Vacuumed table table=closures37442026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37452026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037462026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37472026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037482026/11/12 00:23:22 INFO Vacuumed table table=objects37492026/11/12 00:23:22 INFO Vacuumed table table=objects37502026/11/12 00:23:22 INFO Received complete push request method=POST path=/37512026/11/12 00:23:22 INFO Vacuumed table table=closures37522026/11/12 00:23:22 INFO Received uploads request method=POST path=/37532026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37542026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period37552026/11/12 00:23:22 INFO Vacuumed table table=objects37562026/11/12 00:23:22 INFO Completed push id=1 objects=11 duration=2.460779ms37572026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37582026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37592026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37602026/11/12 00:23:22 INFO Received complete upload request method=POST path=/37612026/11/12 00:23:22 INFO Vacuumed table table=closures37622026/11/12 00:23:22 INFO Vacuumed table table=objects37632026/11/12 00:23:22 INFO Received complete push request method=POST path=/37642026/11/12 00:23:22 INFO Completed upload id=7 objects=10 duration=2.346899ms37652026/11/12 00:23:22 INFO Received uploads request method=POST path=/37662026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37672026/11/12 00:23:22 INFO Received complete upload request method=POST path=/37682026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37692026/11/12 00:23:22 INFO Completed push id=22 objects=3 duration=3.183409ms37702026/11/12 00:23:22 INFO Received push request method=POST path=/37712026/11/12 00:23:22 INFO Vacuumed table table=closures37722026/11/12 00:23:22 INFO Completed upload id=14 objects=8 duration=2.644269ms37732026/11/12 00:23:22 INFO Received push request method=POST path=/37742026/11/12 00:23:22 INFO Vacuumed table table=objects37752026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037762026/11/12 00:23:22 INFO Received push request method=POST path=/37772026/11/12 00:23:22 INFO Received uploads request method=POST path=/37782026/11/12 00:23:22 INFO Received push request method=POST path=/37792026/11/12 00:23:22 INFO Received uploads request method=POST path=/37802026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037812026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037822026/11/12 00:23:22 INFO Received complete upload request method=POST path=/37832026/11/12 00:23:22 INFO Vacuumed table table=pending_closures37842026/11/12 00:23:22 INFO Completed upload id=16 objects=3 duration=1.360189ms37852026/11/12 00:23:22 INFO Vacuumed table table=pending_objects37862026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads37872026/11/12 00:23:22 INFO Vacuumed table table=closures37882026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037892026/11/12 00:23:22 INFO Vacuumed table table=objects37902026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period37912026/11/12 00:23:22 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."37922026/11/12 00:23:22 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."37932026/11/12 00:23:22 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."37942026/11/12 00:23:22 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."37952026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."37962026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."37972026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37982026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=037992026/11/12 00:23:22 INFO Received uploads request method=POST path=/38002026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38012026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period38022026/11/12 00:23:22 INFO Received push request method=POST path=/38032026/11/12 00:23:22 INFO Received push request method=POST path=/38042026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038052026/11/12 00:23:22 INFO Vacuumed table table=pending_closures38062026/11/12 00:23:22 INFO Vacuumed table table=pending_objects38072026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads38082026/11/12 00:23:22 INFO Received push request method=POST path=/38092026/11/12 00:23:22 INFO Vacuumed table table=closures38102026/11/12 00:23:22 INFO Vacuumed table table=objects38112026/11/12 00:23:22 INFO Received uploads request method=POST path=/38122026/11/12 00:23:22 INFO Received complete push request method=POST path=/38132026/11/12 00:23:22 INFO Received complete upload request method=POST path=/38142026/11/12 00:23:22 INFO Completed push id=21 objects=3 duration=1.47755ms38152026/11/12 00:23:22 INFO Completed upload id=12 objects=26 duration=2.985309ms38162026/11/12 00:23:22 INFO Received push request method=POST path=/38172026/11/12 00:23:22 INFO Received uploads request method=POST path=/38182026/11/12 00:23:22 INFO Received uploads request method=POST path=/38192026/11/12 00:23:22 INFO Received push request method=POST path=/38202026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38212026/11/12 00:23:22 INFO Received push request method=POST path=/38222026/11/12 00:23:22 INFO Received push request method=POST path=/38232026/11/12 00:23:22 INFO Received uploads request method=POST path=/38242026/11/12 00:23:22 INFO Received uploads request method=POST path=/38252026/11/12 00:23:22 INFO Received push request method=POST path=/38262026/11/12 00:23:22 INFO Received uploads request method=POST path=/38272026/11/12 00:23:22 INFO Received uploads request method=POST path=/38282026/11/12 00:23:22 INFO Received push request method=POST path=/38292026/11/12 00:23:22 INFO Received complete push request method=POST path=/38302026/11/12 00:23:22 INFO Received push request method=POST path=/38312026/11/12 00:23:22 INFO Completed push id=26 objects=14 duration=3.027479ms38322026/11/12 00:23:22 INFO Received uploads request method=POST path=/38332026/11/12 00:23:22 INFO Received uploads request method=POST path=/38342026/11/12 00:23:22 INFO Received push request method=POST path=/38352026/11/12 00:23:22 INFO Received push request method=POST path=/38362026/11/12 00:23:22 INFO Received uploads request method=POST path=/38372026/11/12 00:23:22 INFO Received complete upload request method=POST path=/38382026/11/12 00:23:22 INFO Completed upload id=19 objects=25 duration=3.234509ms38392026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038402026/11/12 00:23:22 INFO Received push request method=POST path=/38412026/11/12 00:23:22 INFO Received uploads request method=POST path=/38422026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038432026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period38442026/11/12 00:23:22 INFO Received complete upload request method=POST path=/38452026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period38462026/11/12 00:23:22 INFO Completed upload id=11 objects=7 duration=1.904019ms38472026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038482026/11/12 00:23:22 INFO Vacuumed table table=pending_closures38492026/11/12 00:23:22 INFO Received uploads request method=POST path=/38502026/11/12 00:23:22 INFO Vacuumed table table=pending_objects38512026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads38522026/11/12 00:23:22 INFO Vacuumed table table=closures38532026/11/12 00:23:22 INFO Received uploads request method=POST path=/38542026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038552026/11/12 00:23:22 INFO Vacuumed table table=objects38562026/11/12 00:23:22 INFO Vacuumed table table=pending_closures38572026/11/12 00:23:22 INFO Vacuumed table table=pending_objects38582026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads38592026/11/12 00:23:22 INFO Received push request method=POST path=/38602026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038612026/11/12 00:23:22 INFO Received complete push request method=POST path=/38622026/11/12 00:23:22 INFO Vacuumed table table=closures38632026/11/12 00:23:22 INFO Vacuumed table table=objects38642026/11/12 00:23:22 INFO Received push request method=POST path=/38652026/11/12 00:23:22 INFO Completed push id=16 objects=8 duration=2.391229ms38662026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038672026/11/12 00:23:22 INFO Received complete push request method=POST path=/38682026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period38692026/11/12 00:23:22 INFO Received uploads request method=POST path=/38702026/11/12 00:23:22 INFO Received uploads request method=POST path=/38712026/11/12 00:23:22 INFO Completed push id=7 objects=15 duration=3.241349ms38722026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038732026/11/12 00:23:22 INFO Received uploads request method=POST path=/38742026/11/12 00:23:22 INFO Vacuumed table table=pending_closures38752026/11/12 00:23:22 INFO Received uploads request method=POST path=/38762026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038772026/11/12 00:23:22 INFO Vacuumed table table=pending_objects38782026/11/12 00:23:22 INFO Received push request method=POST path=/38792026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads38802026/11/12 00:23:22 INFO Vacuumed table table=pending_closures38812026/11/12 00:23:22 INFO Vacuumed table table=closures38822026/11/12 00:23:22 INFO Vacuumed table table=pending_objects38832026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads38842026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038852026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038862026/11/12 00:23:22 INFO Vacuumed table table=objects38872026/11/12 00:23:22 INFO Vacuumed table table=closures38882026/11/12 00:23:22 INFO Vacuumed table table=objects38892026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period38902026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period38912026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038922026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038932026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038942026/11/12 00:23:22 INFO Vacuumed table table=pending_closures38952026/11/12 00:23:22 INFO Vacuumed table table=pending_closures38962026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=038972026/11/12 00:23:22 INFO Vacuumed table table=pending_objects38982026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads38992026/11/12 00:23:22 INFO Vacuumed table table=closures39002026/11/12 00:23:22 INFO Received complete push request method=POST path=/39012026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39022026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39032026/11/12 00:23:22 INFO Vacuumed table table=objects39042026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039052026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period39062026/11/12 00:23:22 INFO Vacuumed table table=closures39072026/11/12 00:23:22 INFO Received uploads request method=POST path=/39082026/11/12 00:23:22 INFO Vacuumed table table=pending_closures39092026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039102026/11/12 00:23:22 INFO Vacuumed table table=objects39112026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039122026/11/12 00:23:22 INFO Completed push id=19 objects=8 duration=4.130158ms39132026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039142026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39152026/11/12 00:23:22 INFO Vacuumed table table=pending_closures39162026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39172026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period39182026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39192026/11/12 00:23:22 INFO Vacuumed table table=closures39202026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039212026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39222026/11/12 00:23:22 INFO Vacuumed table table=objects39232026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039242026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039252026/11/12 00:23:22 INFO Vacuumed table table=closures39262026/11/12 00:23:22 INFO Vacuumed table table=pending_closures39272026/11/12 00:23:22 INFO Received uploads request method=POST path=/39282026/11/12 00:23:22 INFO Vacuumed table table=pending_closures39292026/11/12 00:23:22 INFO Received push request method=POST path=/39302026/11/12 00:23:22 INFO Vacuumed table table=objects39312026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39322026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39332026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39342026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39352026/11/12 00:23:22 INFO Vacuumed table table=closures39362026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039372026/11/12 00:23:22 INFO Received push request method=POST path=/39382026/11/12 00:23:22 INFO Vacuumed table table=closures39392026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039402026/11/12 00:23:22 INFO Vacuumed table table=pending_closures39412026/11/12 00:23:22 INFO Vacuumed table table=objects39422026/11/12 00:23:22 INFO Vacuumed table table=objects39432026/11/12 00:23:22 INFO Received complete push request method=POST path=/39442026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39452026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39462026/11/12 00:23:22 INFO Vacuumed table table=closures39472026/11/12 00:23:22 INFO Received uploads request method=POST path=/39482026/11/12 00:23:22 INFO Completed push id=4 objects=6 duration=2.375449ms39492026/11/12 00:23:22 INFO Vacuumed table table=objects39502026/11/12 00:23:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."39512026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."39522026/11/12 00:23:22 INFO Received uploads request method=POST path=/39532026/11/12 00:23:22 INFO Received push request method=POST path=/39542026/11/12 00:23:22 INFO Received push request method=POST path=/39552026/11/12 00:23:22 INFO Received push request method=POST path=/39562026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039572026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039582026/11/12 00:23:22 INFO Received complete push request method=POST path=/39592026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period39602026/11/12 00:23:22 INFO Received uploads request method=POST path=/39612026/11/12 00:23:22 INFO Received uploads request method=POST path=/39622026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period39632026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039642026/11/12 00:23:22 INFO Received uploads request method=POST path=/39652026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039662026/11/12 00:23:22 INFO Completed push id=11 objects=6 duration=3.347809ms39672026/11/12 00:23:22 INFO Vacuumed table table=pending_closures39682026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39692026/11/12 00:23:22 INFO Received complete upload request method=POST path=/39702026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39712026/11/12 00:23:22 INFO Vacuumed table table=closures39722026/11/12 00:23:22 INFO Vacuumed table table=objects39732026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039742026/11/12 00:23:22 INFO Completed upload id=16 objects=17 duration=2.723709ms39752026/11/12 00:23:22 INFO Vacuumed table table=pending_closures39762026/11/12 00:23:22 INFO Received uploads request method=POST path=/39772026/11/12 00:23:22 INFO Vacuumed table table=pending_objects39782026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads39792026/11/12 00:23:22 INFO Received uploads request method=POST path=/39802026/11/12 00:23:22 INFO Vacuumed table table=closures39812026/11/12 00:23:22 INFO Vacuumed table table=objects39822026/11/12 00:23:22 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."39832026/11/12 00:23:22 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."39842026/11/12 00:23:22 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."39852026/11/12 00:23:22 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."39862026/11/12 00:23:22 INFO Received push request method=POST path=/39872026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."39882026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."39892026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39902026/11/12 00:23:22 INFO Received complete push request method=POST path=/39912026/11/12 00:23:22 INFO Received push request method=POST path=/39922026/11/12 00:23:22 INFO Completed push id=30 objects=4 duration=3.269468ms39932026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039942026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period39952026/11/12 00:23:22 INFO Received uploads request method=POST path=/39962026/11/12 00:23:22 INFO Received complete upload request method=POST path=/39972026/11/12 00:23:22 INFO Completed upload id=19 objects=2 duration=2.522389ms39982026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=039992026/11/12 00:23:22 INFO Received uploads request method=POST path=/40002026/11/12 00:23:22 INFO Received complete push request method=POST path=/40012026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period40022026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040032026/11/12 00:23:22 INFO Received complete push request method=POST path=/40042026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period40052026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=040062026/11/12 00:23:22 INFO Completed push id=18 objects=6 duration=1.60272ms40072026/11/12 00:23:22 INFO Completed push id=14 objects=3 duration=3.466809ms40082026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40092026/11/12 00:23:22 INFO Received push request method=POST path=/40102026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040112026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40122026/11/12 00:23:22 INFO Vacuumed table table=pending_objects40132026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads40142026/11/12 00:23:22 INFO Vacuumed table table=pending_objects40152026/11/12 00:23:22 INFO Vacuumed table table=closures40162026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads40172026/11/12 00:23:22 INFO Received uploads request method=POST path=/40182026/11/12 00:23:22 INFO Vacuumed table table=closures40192026/11/12 00:23:22 INFO Vacuumed table table=objects40202026/11/12 00:23:22 INFO Vacuumed table table=objects40212026/11/12 00:23:22 INFO Received uploads request method=POST path=/40222026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040232026/11/12 00:23:22 INFO Received uploads request method=POST path=/40242026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040252026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period40262026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040272026/11/12 00:23:22 INFO Received complete upload request method=POST path=/40282026/11/12 00:23:22 INFO Received uploads request method=POST path=/40292026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040302026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40312026/11/12 00:23:22 INFO Completed upload id=21 objects=3 duration=1.642269ms40322026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40332026/11/12 00:23:22 INFO Vacuumed table table=pending_objects40342026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040352026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads40362026/11/12 00:23:22 INFO Vacuumed table table=pending_objects40372026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads40382026/11/12 00:23:22 INFO Vacuumed table table=closures40392026/11/12 00:23:22 INFO Received push request method=POST path=/40402026/11/12 00:23:22 INFO Vacuumed table table=closures40412026/11/12 00:23:22 INFO Vacuumed table table=objects40422026/11/12 00:23:22 INFO Received complete upload request method=POST path=/40432026/11/12 00:23:22 INFO Vacuumed table table=objects40442026/11/12 00:23:22 INFO Received complete push request method=POST path=/40452026/11/12 00:23:22 INFO Received push request method=POST path=/40462026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040472026/11/12 00:23:22 INFO Received push request method=POST path=/40482026/11/12 00:23:22 INFO Received complete push request method=POST path=/40492026/11/12 00:23:22 INFO Completed upload id=21 objects=7 duration=3.185969ms40502026/11/12 00:23:22 INFO Completed push id=21 objects=9 duration=2.937689ms40512026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40522026/11/12 00:23:22 INFO Vacuumed table table=pending_objects40532026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads40542026/11/12 00:23:22 INFO Completed push id=33 objects=7 duration=2.557359ms40552026/11/12 00:23:22 INFO Vacuumed table table=closures40562026/11/12 00:23:22 INFO Vacuumed table table=objects40572026/11/12 00:23:22 INFO Received uploads request method=POST path=/40582026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040592026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period40602026/11/12 00:23:22 INFO Received push request method=POST path=/40612026/11/12 00:23:22 INFO Received uploads request method=POST path=/40622026/11/12 00:23:22 INFO Received complete push request method=POST path=/40632026/11/12 00:23:22 INFO Received uploads request method=POST path=/40642026/11/12 00:23:22 INFO Received complete upload request method=POST path=/40652026/11/12 00:23:22 INFO Completed upload id=26 objects=3 duration=1.50048ms40662026/11/12 00:23:22 INFO Completed push id=13 objects=15 duration=2.578419ms40672026/11/12 00:23:22 INFO Received push request method=POST path=/40682026/11/12 00:23:22 INFO Received uploads request method=POST path=/40692026/11/12 00:23:22 INFO Received complete upload request method=POST path=/40702026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=040712026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40722026/11/12 00:23:22 INFO Completed upload id=20 objects=3 duration=2.594719ms40732026/11/12 00:23:22 INFO Vacuumed table table=pending_objects40742026/11/12 00:23:22 INFO Received complete upload request method=POST path=/40752026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads40762026/11/12 00:23:22 INFO Vacuumed table table=closures40772026/11/12 00:23:22 INFO Vacuumed table table=objects40782026/11/12 00:23:22 INFO Received push request method=POST path=/40792026/11/12 00:23:22 INFO Completed upload id=23 objects=18 duration=2.744339ms40802026/11/12 00:23:22 INFO Received complete upload request method=POST path=/40812026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040822026/11/12 00:23:22 INFO Received push request method=POST path=/40832026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040842026/11/12 00:23:22 INFO Received complete push request method=POST path=/40852026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period40862026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period40872026/11/12 00:23:22 INFO Received uploads request method=POST path=/40882026/11/12 00:23:22 INFO Completed upload id=33 objects=5 duration=2.828128ms40892026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=040902026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040912026/11/12 00:23:22 INFO Received uploads request method=POST path=/40922026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040932026/11/12 00:23:22 INFO Completed push id=19 objects=14 duration=2.849369ms40942026/11/12 00:23:22 INFO Received complete push request method=POST path=/40952026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40962026/11/12 00:23:22 INFO Vacuumed table table=pending_closures40972026/11/12 00:23:22 INFO Completed push id=32 objects=7 duration=1.461689ms40982026/11/12 00:23:22 INFO Vacuumed table table=pending_objects40992026/11/12 00:23:22 INFO Vacuumed table table=pending_objects41002026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41012026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41022026/11/12 00:23:22 INFO Vacuumed table table=closures41032026/11/12 00:23:22 INFO Vacuumed table table=closures41042026/11/12 00:23:22 INFO Vacuumed table table=objects41052026/11/12 00:23:22 INFO Received push request method=POST path=/41062026/11/12 00:23:22 INFO Vacuumed table table=objects41072026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=041082026/11/12 00:23:22 INFO Vacuumed table table=pending_closures41092026/11/12 00:23:22 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41102026/11/12 00:23:22 INFO Vacuumed table table=pending_objects41112026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41122026/11/12 00:23:22 INFO Received uploads request method=POST path=/41132026/11/12 00:23:22 INFO Vacuumed table table=closures41142026/11/12 00:23:22 INFO Vacuumed table table=objects41152026/11/12 00:23:22 INFO Received uploads request method=POST path=/41162026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=041172026/11/12 00:23:22 INFO Received complete push request method=POST path=/41182026/11/12 00:23:22 INFO Completed push id=26 objects=4 duration=1.380329ms41192026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period41202026/11/12 00:23:22 INFO Received complete push request method=POST path=/41212026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=041222026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=041232026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period41242026/11/12 00:23:22 INFO Completed push id=18 objects=6 duration=3.933468ms41252026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period41262026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041272026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041282026/11/12 00:23:22 INFO Received uploads request method=POST path=/41292026/11/12 00:23:22 INFO Vacuumed table table=pending_closures41302026/11/12 00:23:22 INFO Vacuumed table table=pending_objects41312026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41322026/11/12 00:23:22 INFO Vacuumed table table=pending_closures41332026/11/12 00:23:22 INFO Received push request method=POST path=/41342026/11/12 00:23:22 INFO Vacuumed table table=closures41352026/11/12 00:23:22 INFO Vacuumed table table=pending_objects41362026/11/12 00:23:22 INFO Received push request method=POST path=/41372026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41382026/11/12 00:23:22 INFO Received uploads request method=POST path=/41392026/11/12 00:23:22 INFO Vacuumed table table=objects41402026/11/12 00:23:22 INFO Vacuumed table table=closures41412026/11/12 00:23:22 INFO Received push request method=POST path=/41422026/11/12 00:23:22 INFO Vacuumed table table=objects41432026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41442026/11/12 00:23:22 INFO Completed upload id=28 objects=13 duration=1.29401ms41452026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=041462026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41472026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period41482026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41492026/11/12 00:23:22 INFO Completed upload id=23 objects=9 duration=1.23423ms41502026/11/12 00:23:22 INFO Received complete push request method=POST path=/41512026/11/12 00:23:22 INFO Completed upload id=24 objects=7 duration=1.205599ms41522026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041532026/11/12 00:23:22 INFO Received push request method=POST path=/41542026/11/12 00:23:22 INFO Completed push id=20 objects=3 duration=2.08818ms41552026/11/12 00:23:22 INFO Received complete push request method=POST path=/41562026/11/12 00:23:22 INFO Vacuumed table table=pending_closures41572026/11/12 00:23:22 INFO Vacuumed table table=pending_objects41582026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41592026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41602026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41612026/11/12 00:23:22 INFO Vacuumed table table=closures41622026/11/12 00:23:22 INFO Received uploads request method=POST path=/41632026/11/12 00:23:22 INFO Completed upload id=21 objects=25 duration=1.986229ms41642026/11/12 00:23:22 INFO Completed push id=16 objects=4 duration=3.700438ms41652026/11/12 00:23:22 INFO Vacuumed table table=objects41662026/11/12 00:23:22 INFO Completed upload id=22 objects=22 duration=1.857709ms41672026/11/12 00:23:22 INFO Received uploads request method=POST path=/41682026/11/12 00:23:22 INFO Received push request method=POST path=/41692026/11/12 00:23:22 INFO Received complete push request method=POST path=/41702026/11/12 00:23:22 INFO Received complete push request method=POST path=/41712026/11/12 00:23:22 INFO Completed push id=29 objects=3 duration=1.24666ms41722026/11/12 00:23:22 INFO Completed push id=27 objects=25 duration=3.266339ms41732026/11/12 00:23:22 INFO Received complete push request method=POST path=/41742026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=041752026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41762026/11/12 00:23:22 INFO Vacuumed table table=pending_closures41772026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=041782026/11/12 00:23:22 INFO Completed push id=9 objects=9 duration=3.635739ms41792026/11/12 00:23:22 INFO Received complete push request method=POST path=/41802026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41812026/11/12 00:23:22 INFO Completed upload id=17 objects=25 duration=3.439349ms41822026/11/12 00:23:22 INFO Vacuumed table table=pending_objects41832026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41842026/11/12 00:23:22 INFO Received uploads request method=POST path=/41852026/11/12 00:23:22 INFO Completed upload id=15 objects=3 duration=2.114449ms41862026/11/12 00:23:22 INFO Received uploads request method=POST path=/41872026/11/12 00:23:22 INFO Completed push id=22 objects=3 duration=3.008679ms41882026/11/12 00:23:22 INFO Vacuumed table table=closures41892026/11/12 00:23:22 INFO Received complete upload request method=POST path=/41902026/11/12 00:23:22 INFO Received push request method=POST path=/41912026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041922026/11/12 00:23:22 INFO Vacuumed table table=objects41932026/11/12 00:23:22 INFO Completed upload id=27 objects=4 duration=1.47687ms41942026/11/12 00:23:22 INFO Vacuumed table table=pending_closures41952026/11/12 00:23:22 INFO Received push request method=POST path=/41962026/11/12 00:23:22 INFO Vacuumed table table=pending_objects41972026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads41982026/11/12 00:23:22 INFO Vacuumed table table=closures41992026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42002026/11/12 00:23:22 INFO Vacuumed table table=objects42012026/11/12 00:23:22 INFO Completed upload id=31 objects=2 duration=1.12808ms42022026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=042032026/11/12 00:23:22 INFO Received push request method=POST path=/42042026/11/12 00:23:22 INFO Received uploads request method=POST path=/42052026/11/12 00:23:22 INFO Received push request method=POST path=/42062026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42072026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042082026/11/12 00:23:22 INFO Vacuumed table table=pending_closures42092026/11/12 00:23:22 INFO Completed upload id=21 objects=8 duration=1.75784ms42102026/11/12 00:23:22 INFO Received complete push request method=POST path=/42112026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42122026/11/12 00:23:22 INFO Vacuumed table table=pending_objects42132026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads42142026/11/12 00:23:22 INFO Received push request method=POST path=/42152026/11/12 00:23:22 INFO Completed upload id=20 objects=5 duration=1.20355ms42162026/11/12 00:23:22 INFO Vacuumed table table=closures42172026/11/12 00:23:22 INFO Completed push id=13 objects=11 duration=2.118939ms42182026/11/12 00:23:22 INFO Received complete push request method=POST path=/42192026/11/12 00:23:22 INFO Vacuumed table table=objects42202026/11/12 00:23:22 INFO Completed push id=26 objects=5 duration=2.334919ms42212026/11/12 00:23:22 INFO Received uploads request method=POST path=/42222026/11/12 00:23:22 INFO Received complete push request method=POST path=/42232026/11/12 00:23:22 INFO Received push request method=POST path=/42242026/11/12 00:23:22 INFO Completed push id=33 objects=6 duration=1.23029ms42252026/11/12 00:23:22 INFO Received uploads request method=POST path=/42262026/11/12 00:23:22 INFO Received complete push request method=POST path=/42272026/11/12 00:23:22 INFO Completed push id=22 objects=6 duration=1.53449ms42282026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42292026/11/12 00:23:22 INFO Completed upload id=21 objects=2 duration=1.2561ms42302026/11/12 00:23:22 INFO Received uploads request method=POST path=/42312026/11/12 00:23:22 INFO Received uploads request method=POST path=/42322026/11/12 00:23:22 INFO Received complete push request method=POST path=/42332026/11/12 00:23:22 INFO Received uploads request method=POST path=/42342026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42352026/11/12 00:23:22 INFO Completed push id=24 objects=5 duration=3.190348ms42362026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42372026/11/12 00:23:22 INFO Completed upload id=24 objects=14 duration=1.251649ms42382026/11/12 00:23:22 INFO Completed upload id=12 objects=6 duration=1.50502ms42392026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=042402026/11/12 00:23:22 INFO Received push request method=POST path=/42412026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period42422026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=042432026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042442026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period42452026/11/12 00:23:22 INFO Received complete push request method=POST path=/42462026/11/12 00:23:22 INFO Received push request method=POST path=/42472026/11/12 00:23:22 INFO Vacuumed table table=pending_closures42482026/11/12 00:23:22 INFO Completed push id=24 objects=8 duration=1.202889ms42492026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042502026/11/12 00:23:22 INFO Vacuumed table table=pending_objects42512026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads42522026/11/12 00:23:22 INFO Vacuumed table table=closures42532026/11/12 00:23:22 INFO Received push request method=POST path=/42542026/11/12 00:23:22 INFO Vacuumed table table=pending_closures42552026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=042562026/11/12 00:23:22 INFO Vacuumed table table=objects42572026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period42582026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=042592026/11/12 00:23:22 INFO Vacuumed table table=pending_objects42602026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42612026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads42622026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=042632026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042642026/11/12 00:23:22 INFO Vacuumed table table=closures42652026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period42662026/11/12 00:23:22 INFO Vacuumed table table=pending_closures42672026/11/12 00:23:22 INFO Received complete push request method=POST path=/42682026/11/12 00:23:22 INFO Completed upload id=14 objects=16 duration=2.972519ms42692026/11/12 00:23:22 INFO Vacuumed table table=objects42702026/11/12 00:23:22 INFO Received complete push request method=POST path=/42712026/11/12 00:23:22 INFO Vacuumed table table=pending_objects42722026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042732026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads42742026/11/12 00:23:22 INFO Completed push id=19 objects=3 duration=1.587289ms42752026/11/12 00:23:22 INFO Received push request method=POST path=/42762026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042772026/11/12 00:23:22 INFO Completed push id=17 objects=3 duration=2.071789ms42782026/11/12 00:23:22 INFO Vacuumed table table=closures42792026/11/12 00:23:22 INFO Vacuumed table table=pending_closures42802026/11/12 00:23:22 INFO Received complete upload request method=POST path=/42812026/11/12 00:23:22 INFO Vacuumed table table=pending_closures42822026/11/12 00:23:22 INFO Completed upload id=25 objects=9 duration=1.540199ms42832026/11/12 00:23:22 INFO Vacuumed table table=objects42842026/11/12 00:23:22 INFO Vacuumed table table=pending_objects42852026/11/12 00:23:22 INFO Received complete push request method=POST path=/42862026/11/12 00:23:22 INFO Vacuumed table table=pending_objects42872026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads42882026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads42892026/11/12 00:23:22 INFO Vacuumed table table=closures42902026/11/12 00:23:22 INFO Received push request method=POST path=/42912026/11/12 00:23:22 INFO Vacuumed table table=closures42922026/11/12 00:23:22 INFO Completed push id=24 objects=3 duration=1.779809ms42932026/11/12 00:23:22 INFO Vacuumed table table=objects42942026/11/12 00:23:22 INFO Vacuumed table table=objects42952026/11/12 00:23:22 INFO Received push request method=POST path=/42962026/11/12 00:23:22 INFO Received uploads request method=POST path=/42972026/11/12 00:23:22 INFO Received complete push request method=POST path=/42982026/11/12 00:23:22 INFO Received complete push request method=POST path=/42992026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43002026/11/12 00:23:22 INFO Received push request method=POST path=/43012026/11/12 00:23:22 INFO Completed push id=23 objects=28 duration=3.564259ms43022026/11/12 00:23:22 INFO Completed upload id=27 objects=3 duration=1.52257ms43032026/11/12 00:23:22 INFO Completed push id=17 objects=22 duration=2.608749ms43042026/11/12 00:23:22 INFO Received push request method=POST path=/43052026/11/12 00:23:22 INFO Received complete push request method=POST path=/43062026/11/12 00:23:22 INFO Completed push id=23 objects=5 duration=1.62312ms43072026/11/12 00:23:22 INFO Received push request method=POST path=/43082026/11/12 00:23:22 INFO Received uploads request method=POST path=/43092026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43102026/11/12 00:23:22 INFO Completed upload id=28 objects=6 duration=1.10383ms43112026/11/12 00:23:22 INFO Received uploads request method=POST path=/43122026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43132026/11/12 00:23:22 INFO Completed upload id=22 objects=6 duration=1.38167ms43142026/11/12 00:23:22 INFO Received uploads request method=POST path=/43152026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43162026/11/12 00:23:22 INFO Completed upload id=24 objects=3 duration=1.157869ms43172026/11/12 00:23:22 INFO Received complete push request method=POST path=/43182026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43192026/11/12 00:23:22 INFO Completed push id=16 objects=26 duration=1.624719ms43202026/11/12 00:23:22 INFO Completed upload id=13 objects=8 duration=1.948159ms43212026/11/12 00:23:22 INFO Received complete push request method=POST path=/43222026/11/12 00:23:22 INFO Received complete push request method=POST path=/43232026/11/12 00:23:22 INFO Received complete push request method=POST path=/43242026/11/12 00:23:22 INFO Completed push id=21 objects=26 duration=1.589429ms43252026/11/12 00:23:22 INFO Completed push id=30 objects=4 duration=1.11347ms43262026/11/12 00:23:22 INFO Completed push id=17 objects=11 duration=2.549209ms43272026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=043282026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43292026/11/12 00:23:22 INFO Received uploads request method=POST path=/43302026/11/12 00:23:22 INFO Completed upload id=27 objects=8 duration=1.162389ms43312026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period43322026/11/12 00:23:22 INFO Received complete push request method=POST path=/43332026/11/12 00:23:22 INFO Completed push id=25 objects=4 duration=1.217659ms43342026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043352026/11/12 00:23:22 INFO Received complete push request method=POST path=/43362026/11/12 00:23:22 INFO Vacuumed table table=pending_closures43372026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=043382026/11/12 00:23:22 INFO Received complete push request method=POST path=/43392026/11/12 00:23:22 INFO Vacuumed table table=pending_objects43402026/11/12 00:23:22 INFO Received uploads request method=POST path=/43412026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads43422026/11/12 00:23:22 INFO Received complete push request method=POST path=/43432026/11/12 00:23:22 INFO Completed push id=16 objects=14 duration=3.126709ms43442026/11/12 00:23:22 INFO Completed push id=7 objects=24 duration=3.899519ms43452026/11/12 00:23:22 INFO Vacuumed table table=closures43462026/11/12 00:23:22 INFO Completed push id=31 objects=7 duration=1.13701ms43472026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period43482026/11/12 00:23:22 INFO Received push request method=POST path=/43492026/11/12 00:23:22 INFO Vacuumed table table=objects43502026/11/12 00:23:22 INFO Received uploads request method=POST path=/43512026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043522026/11/12 00:23:22 INFO Vacuumed table table=pending_closures43532026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43542026/11/12 00:23:22 INFO Vacuumed table table=pending_objects43552026/11/12 00:23:22 INFO Received push request method=POST path=/43562026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads43572026/11/12 00:23:22 INFO Completed upload id=28 objects=17 duration=1.163019ms43582026/11/12 00:23:22 INFO Vacuumed table table=closures43592026/11/12 00:23:22 INFO Vacuumed table table=objects43602026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43612026/11/12 00:23:22 INFO Completed upload id=13 objects=14 duration=2.984119ms43622026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=043632026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43642026/11/12 00:23:22 INFO Completed upload id=32 objects=5 duration=988.669µs43652026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=043662026/11/12 00:23:22 INFO Received uploads request method=POST path=/43672026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period43682026/11/12 00:23:22 INFO Received uploads request method=POST path=/43692026/11/12 00:23:22 INFO Received push request method=POST path=/43702026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period43712026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043722026/11/12 00:23:22 INFO Received complete upload request method=POST path=/43732026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043742026/11/12 00:23:22 INFO Vacuumed table table=pending_closures43752026/11/12 00:23:22 INFO Completed upload id=25 objects=4 duration=1.217ms43762026/11/12 00:23:22 INFO Received complete push request method=POST path=/43772026/11/12 00:23:22 INFO Vacuumed table table=pending_closures43782026/11/12 00:23:22 INFO Received uploads request method=POST path=/43792026/11/12 00:23:22 INFO Vacuumed table table=pending_objects43802026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads43812026/11/12 00:23:22 INFO Completed push id=29 objects=2 duration=1.46526ms43822026/11/12 00:23:22 INFO Vacuumed table table=pending_objects43832026/11/12 00:23:22 INFO Vacuumed table table=closures43842026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads43852026/11/12 00:23:22 INFO Vacuumed table table=closures43862026/11/12 00:23:22 INFO Received complete push request method=POST path=/43872026/11/12 00:23:22 INFO Vacuumed table table=objects43882026/11/12 00:23:22 INFO Vacuumed table table=objects43892026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=043902026/11/12 00:23:22 INFO Completed push id=27 objects=6 duration=3.427249ms43912026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=043922026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043932026/11/12 00:23:22 INFO Received push request method=POST path=/43942026/11/12 00:23:22 INFO Vacuumed table table=pending_closures43952026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043962026/11/12 00:23:22 INFO Received uploads request method=POST path=/43972026/11/12 00:23:22 INFO Received push request method=POST path=/43982026/11/12 00:23:22 INFO Vacuumed table table=pending_closures43992026/11/12 00:23:22 INFO Vacuumed table table=pending_objects44002026/11/12 00:23:22 INFO Received push request method=POST path=/44012026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads44022026/11/12 00:23:22 INFO Vacuumed table table=pending_objects44032026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads44042026/11/12 00:23:22 INFO Vacuumed table table=closures44052026/11/12 00:23:22 INFO Vacuumed table table=closures44062026/11/12 00:23:22 INFO Received complete push request method=POST path=/44072026/11/12 00:23:22 INFO Vacuumed table table=objects44082026/11/12 00:23:22 INFO Received push request method=POST path=/44092026/11/12 00:23:22 INFO Received complete push request method=POST path=/44102026/11/12 00:23:22 INFO Vacuumed table table=objects44112026/11/12 00:23:22 INFO Completed push id=31 objects=6 duration=1.188049ms44122026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=044132026/11/12 00:23:22 INFO Completed push id=26 objects=23 duration=1.684299ms44142026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=044152026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=044162026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044172026/11/12 00:23:22 INFO Vacuumed table table=pending_closures44182026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044192026/11/12 00:23:22 INFO Vacuumed table table=pending_objects44202026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads44212026/11/12 00:23:22 INFO Vacuumed table table=pending_closures44222026/11/12 00:23:22 INFO Vacuumed table table=closures44232026/11/12 00:23:22 INFO Received complete push request method=POST path=/44242026/11/12 00:23:22 INFO Vacuumed table table=pending_objects44252026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads44262026/11/12 00:23:22 INFO Vacuumed table table=objects44272026/11/12 00:23:22 INFO Vacuumed table table=closures44282026/11/12 00:23:22 INFO Completed push id=19 objects=14 duration=2.01736ms44292026/11/12 00:23:22 INFO Vacuumed table table=objects44302026/11/12 00:23:22 INFO Received uploads request method=POST path=/44312026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44322026/11/12 00:23:22 INFO Completed upload id=26 objects=6 duration=1.36848ms44332026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=044342026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=044352026/11/12 00:23:22 INFO Received push request method=POST path=/44362026/11/12 00:23:22 INFO Vacuumed table table=pending_closures44372026/11/12 00:23:22 INFO Received complete push request method=POST path=/44382026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044392026/11/12 00:23:22 INFO Completed push id=32 objects=9 duration=1.513109ms44402026/11/12 00:23:22 INFO Vacuumed table table=pending_objects44412026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44422026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads44432026/11/12 00:23:22 INFO Vacuumed table table=pending_closures44442026/11/12 00:23:22 INFO Received complete push request method=POST path=/44452026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44462026/11/12 00:23:22 INFO Vacuumed table table=closures44472026/11/12 00:23:22 INFO Completed upload id=15 objects=2 duration=1.119929ms44482026/11/12 00:23:22 INFO Vacuumed table table=pending_objects44492026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads44502026/11/12 00:23:22 INFO Vacuumed table table=objects44512026/11/12 00:23:22 INFO Completed upload id=34 objects=24 duration=1.692249ms44522026/11/12 00:23:22 INFO Vacuumed table table=closures44532026/11/12 00:23:22 INFO Completed push id=31 objects=6 duration=3.300188ms44542026/11/12 00:23:22 INFO Vacuumed table table=objects44552026/11/12 00:23:22 INFO Received uploads request method=POST path=/44562026/11/12 00:23:22 INFO Received push request method=POST path=/44572026/11/12 00:23:22 INFO Received push request method=POST path=/44582026/11/12 00:23:22 INFO Received uploads request method=POST path=/44592026/11/12 00:23:22 INFO Received complete push request method=POST path=/44602026/11/12 00:23:22 INFO Received push request method=POST path=/44612026/11/12 00:23:22 INFO Completed push id=20 objects=8 duration=1.26945ms44622026/11/12 00:23:22 INFO Received uploads request method=POST path=/44632026/11/12 00:23:22 INFO Received complete push request method=POST path=/44642026/11/12 00:23:22 INFO Completed push id=27 objects=15 duration=1.523649ms44652026/11/12 00:23:22 INFO Received complete push request method=POST path=/44662026/11/12 00:23:22 INFO Received uploads request method=POST path=/44672026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=044682026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44692026/11/12 00:23:22 INFO Completed push id=5 objects=28 duration=2.283629ms44702026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44712026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period44722026/11/12 00:23:22 INFO Completed upload id=33 objects=9 duration=1.53157ms44732026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044742026/11/12 00:23:22 INFO Received uploads request method=POST path=/44752026/11/12 00:23:22 INFO Completed upload id=25 objects=16 duration=2.635799ms44762026/11/12 00:23:22 INFO Vacuumed table table=pending_closures44772026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44782026/11/12 00:23:22 INFO Vacuumed table table=pending_objects44792026/11/12 00:23:22 INFO Completed upload id=29 objects=5 duration=1.27968ms44802026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads44812026/11/12 00:23:22 INFO Received uploads request method=POST path=/44822026/11/12 00:23:22 INFO Vacuumed table table=closures44832026/11/12 00:23:22 INFO Received uploads request method=POST path=/44842026/11/12 00:23:22 INFO Vacuumed table table=objects44852026/11/12 00:23:22 INFO Received uploads request method=POST path=/44862026/11/12 00:23:22 INFO Received uploads request method=POST path=/44872026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44882026/11/12 00:23:22 INFO Completed upload id=20 objects=6 duration=1.503779ms44892026/11/12 00:23:22 INFO Received complete push request method=POST path=/44902026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=044912026/11/12 00:23:22 INFO Completed push id=10 objects=2 duration=1.552949ms44922026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period44932026/11/12 00:23:22 INFO Received complete upload request method=POST path=/44942026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=044952026/11/12 00:23:22 INFO Received push request method=POST path=/44962026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period44972026/11/12 00:23:22 INFO Completed upload id=25 objects=6 duration=1.52181ms44982026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044992026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045002026/11/12 00:23:22 INFO Received complete push request method=POST path=/45012026/11/12 00:23:22 INFO Vacuumed table table=pending_closures45022026/11/12 00:23:22 INFO Received push request method=POST path=/45032026/11/12 00:23:22 INFO Vacuumed table table=pending_closures45042026/11/12 00:23:22 INFO Completed push id=13 objects=2 duration=1.437819ms45052026/11/12 00:23:22 INFO Vacuumed table table=pending_objects45062026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads45072026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45082026/11/12 00:23:22 INFO Vacuumed table table=closures45092026/11/12 00:23:22 INFO Vacuumed table table=pending_objects45102026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads45112026/11/12 00:23:22 INFO Received uploads request method=POST path=/45122026/11/12 00:23:22 INFO Vacuumed table table=objects45132026/11/12 00:23:22 INFO Vacuumed table table=closures45142026/11/12 00:23:22 INFO Completed upload id=26 objects=2 duration=2.337159ms45152026/11/12 00:23:22 INFO Vacuumed table table=objects45162026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=045172026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45182026/11/12 00:23:22 INFO Received complete push request method=POST path=/45192026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45202026/11/12 00:23:22 INFO Completed upload id=17 objects=5 duration=1.77245ms45212026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period45222026/11/12 00:23:22 INFO Completed push id=28 objects=19 duration=1.950519ms45232026/11/12 00:23:22 INFO Completed upload id=27 objects=13 duration=1.935209ms45242026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045252026/11/12 00:23:22 INFO Received uploads request method=POST path=/45262026/11/12 00:23:22 INFO Vacuumed table table=pending_closures45272026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=045282026/11/12 00:23:22 INFO Received push request method=POST path=/45292026/11/12 00:23:22 INFO Vacuumed table table=pending_objects45302026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads45312026/11/12 00:23:22 INFO Received push request method=POST path=/45322026/11/12 00:23:22 INFO Vacuumed table table=closures45332026/11/12 00:23:22 INFO Received uploads request method=POST path=/45342026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045352026/11/12 00:23:22 INFO Vacuumed table table=objects45362026/11/12 00:23:22 INFO Received push request method=POST path=/45372026/11/12 00:23:22 INFO Vacuumed table table=pending_closures45382026/11/12 00:23:22 INFO Vacuumed table table=pending_objects45392026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads45402026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=045412026/11/12 00:23:22 INFO Vacuumed table table=closures45422026/11/12 00:23:22 INFO Vacuumed table table=objects45432026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period45442026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=045452026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045462026/11/12 00:23:22 INFO Received complete push request method=POST path=/45472026/11/12 00:23:22 INFO Vacuumed table table=pending_closures45482026/11/12 00:23:22 INFO Completed push id=27 objects=11 duration=1.25631ms45492026/11/12 00:23:22 INFO Received complete push request method=POST path=/45502026/11/12 00:23:22 INFO Vacuumed table table=pending_objects45512026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads45522026/11/12 00:23:22 INFO Completed push id=19 objects=3 duration=1.410799ms45532026/11/12 00:23:22 INFO Vacuumed table table=closures45542026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045552026/11/12 00:23:22 INFO Received uploads request method=POST path=/45562026/11/12 00:23:22 INFO Received push request method=POST path=/45572026/11/12 00:23:22 INFO Received complete push request method=POST path=/45582026/11/12 00:23:22 INFO Vacuumed table table=objects45592026/11/12 00:23:22 INFO Received push request method=POST path=/45602026/11/12 00:23:22 INFO Vacuumed table table=pending_closures45612026/11/12 00:23:22 INFO Received uploads request method=POST path=/45622026/11/12 00:23:22 INFO Received uploads request method=POST path=/45632026/11/12 00:23:22 INFO Vacuumed table table=pending_objects45642026/11/12 00:23:22 INFO Completed push id=1 objects=9 duration=2.26373ms45652026/11/12 00:23:22 INFO Vacuumed table table=multipart_uploads45662026/11/12 00:23:22 INFO Received complete push request method=POST path=/45672026/11/12 00:23:22 INFO Vacuumed table table=closures45682026/11/12 00:23:22 INFO Completed push id=29 objects=10 duration=1.38745ms45692026/11/12 00:23:22 INFO Vacuumed table table=objects45702026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45712026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45722026/11/12 00:23:22 INFO Completed upload id=29 objects=26 duration=1.00787ms45732026/11/12 00:23:22 INFO Completed upload id=21 objects=2 duration=1.23406ms45742026/11/12 00:23:22 INFO Received push request method=POST path=/45752026/11/12 00:23:22 INFO Received push request method=POST path=/45762026/11/12 00:23:22 INFO Received complete push request method=POST path=/45772026/11/12 00:23:22 INFO Received uploads request method=POST path=/45782026/11/12 00:23:22 INFO Completed push id=11 objects=9 duration=1.652369ms45792026/11/12 00:23:22 INFO Received uploads request method=POST path=/45802026/11/12 00:23:22 INFO Received complete push request method=POST path=/45812026/11/12 00:23:22 INFO Completed push id=20 objects=4 duration=1.41378ms45822026/11/12 00:23:22 INFO Received complete push request method=POST path=/45832026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45842026/11/12 00:23:22 INFO Completed push id=30 objects=8 duration=1.18187ms45852026/11/12 00:23:22 INFO Received uploads request method=POST path=/45862026/11/12 00:23:22 INFO Completed upload id=30 objects=6 duration=1.58505ms45872026/11/12 00:23:22 INFO Received uploads request method=POST path=/45882026/11/12 00:23:22 INFO Received push request method=POST path=/45892026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45902026/11/12 00:23:22 INFO Completed upload id=26 objects=8 duration=1.990529ms45912026/11/12 00:23:22 INFO Received complete push request method=POST path=/45922026/11/12 00:23:22 INFO Received push request method=POST path=/45932026/11/12 00:23:22 INFO Completed push id=15 objects=24 duration=1.59821ms45942026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45952026/11/12 00:23:22 INFO Completed upload id=31 objects=5 duration=1.119899ms45962026/11/12 00:23:22 INFO Received complete upload request method=POST path=/45972026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=045982026/11/12 00:23:22 INFO Completed upload id=31 objects=5 duration=1.2039ms45992026/11/12 00:23:22 INFO Aborted multipart uploads count=0 kept=046002026/11/12 00:23:22 INFO Received push request method=POST path=/46012026/11/12 00:23:22 WARN Force mode enabled - objects will be deleted immediately without grace period46022026/11/12 00:23:22 INFO Received uploads request method=POST path=/46032026/11/12 00:23:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046042026/11/12 00:23:22 INFO Received complete upload request method=POST path=/46052026/11/12 00:23:22 INFO Vacuumed table table=pending_closures46062026/11/12 00:23:22 INFO Completed upload id=22 objects=8 duration=1.679159ms46072026/11/12 00:23:23 INFO Vacuumed table table=pending_objects46082026/11/12 00:23:23 INFO Received complete push request method=POST path=/46092026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads46102026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46112026/11/12 00:23:23 INFO Vacuumed table table=closures46122026/11/12 00:23:23 INFO Completed push id=28 objects=14 duration=1.498989ms46132026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46142026/11/12 00:23:23 INFO Completed upload id=2 objects=7 duration=1.592549ms46152026/11/12 00:23:23 INFO Vacuumed table table=objects46162026/11/12 00:23:23 INFO Completed upload id=19 objects=2 duration=1.15739ms46172026/11/12 00:23:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."46182026/11/12 00:23:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."46192026/11/12 00:23:23 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46202026/11/12 00:23:23 INFO Received push request method=POST path=/46212026/11/12 00:23:23 INFO Received complete push request method=POST path=/46222026/11/12 00:23:23 INFO Completed push id=32 objects=3 duration=1.128979ms46232026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46242026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46252026/11/12 00:23:23 INFO Completed upload id=34 objects=3 duration=1.18723ms46262026/11/12 00:23:23 INFO Completed upload id=31 objects=3 duration=2.559749ms46272026/11/12 00:23:23 INFO Received complete push request method=POST path=/46282026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46292026/11/12 00:23:23 INFO Completed push id=31 objects=14 duration=2.862109ms46302026/11/12 00:23:23 INFO Received complete push request method=POST path=/46312026/11/12 00:23:23 INFO Completed upload id=22 objects=5 duration=1.96645ms46322026/11/12 00:23:23 INFO Completed push id=23 objects=5 duration=1.280269ms46332026/11/12 00:23:23 INFO Received push request method=POST path=/46342026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46352026/11/12 00:23:23 INFO Received complete push request method=POST path=/46362026/11/12 00:23:23 INFO Completed upload id=24 objects=7 duration=1.00318ms46372026/11/12 00:23:23 INFO Received push request method=POST path=/46382026/11/12 00:23:23 INFO Completed push id=16 objects=7 duration=2.285539ms46392026/11/12 00:23:23 INFO Received uploads request method=POST path=/46402026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=046412026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46422026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=046432026/11/12 00:23:23 INFO Completed upload id=22 objects=2 duration=1.136199ms46442026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period46452026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period46462026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046472026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046482026/11/12 00:23:23 INFO Received complete push request method=POST path=/46492026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46502026/11/12 00:23:23 INFO Vacuumed table table=pending_closures46512026/11/12 00:23:23 INFO Completed upload id=24 objects=8 duration=1.555799ms46522026/11/12 00:23:23 INFO Completed push id=28 objects=3 duration=2.125149ms46532026/11/12 00:23:23 INFO Vacuumed table table=pending_closures46542026/11/12 00:23:23 INFO Vacuumed table table=pending_objects46552026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads46562026/11/12 00:23:23 INFO Vacuumed table table=pending_objects46572026/11/12 00:23:23 INFO Vacuumed table table=closures46582026/11/12 00:23:23 INFO Received complete push request method=POST path=/46592026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads46602026/11/12 00:23:23 INFO Received uploads request method=POST path=/46612026/11/12 00:23:23 INFO Received uploads request method=POST path=/46622026/11/12 00:23:23 INFO Vacuumed table table=objects46632026/11/12 00:23:23 INFO Completed push id=26 objects=3 duration=1.384359ms46642026/11/12 00:23:23 INFO Vacuumed table table=closures46652026/11/12 00:23:23 INFO Received uploads request method=POST path=/46662026/11/12 00:23:23 INFO Vacuumed table table=objects46672026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46682026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=046692026/11/12 00:23:23 INFO Completed upload id=8 objects=6 duration=1.24863ms46702026/11/12 00:23:23 INFO Received uploads request method=POST path=/46712026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period46722026/11/12 00:23:23 INFO Received complete push request method=POST path=/46732026/11/12 00:23:23 INFO Completed push id=23 objects=6 duration=1.47324ms46742026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046752026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46762026/11/12 00:23:23 INFO Received complete push request method=POST path=/46772026/11/12 00:23:23 INFO Vacuumed table table=pending_closures46782026/11/12 00:23:23 INFO Completed upload id=28 objects=5 duration=1.40531ms46792026/11/12 00:23:23 INFO Received push request method=POST path=/46802026/11/12 00:23:23 INFO Completed push id=29 objects=15 duration=1.34772ms46812026/11/12 00:23:23 INFO Vacuumed table table=pending_objects46822026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads46832026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46842026/11/12 00:23:23 INFO Vacuumed table table=closures46852026/11/12 00:23:23 INFO Completed upload id=28 objects=8 duration=980.41µs46862026/11/12 00:23:23 INFO Vacuumed table table=objects46872026/11/12 00:23:23 INFO Received push request method=POST path=/46882026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=046892026/11/12 00:23:23 INFO Received complete upload request method=POST path=/46902026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period46912026/11/12 00:23:23 INFO Completed upload id=29 objects=4 duration=2.082779ms46922026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046932026/11/12 00:23:23 INFO Received complete push request method=POST path=/46942026/11/12 00:23:23 INFO Vacuumed table table=pending_closures46952026/11/12 00:23:23 INFO Completed push id=24 objects=6 duration=1.548ms46962026/11/12 00:23:23 INFO Vacuumed table table=pending_objects46972026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads46982026/11/12 00:23:23 INFO Received complete push request method=POST path=/46992026/11/12 00:23:23 INFO Received complete push request method=POST path=/47002026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47012026/11/12 00:23:23 INFO Vacuumed table table=closures47022026/11/12 00:23:23 INFO Completed push id=30 objects=6 duration=1.279139ms47032026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47042026/11/12 00:23:23 INFO Completed upload id=29 objects=10 duration=1.0685ms47052026/11/12 00:23:23 INFO Vacuumed table table=objects47062026/11/12 00:23:23 INFO Completed push id=28 objects=11 duration=2.653069ms47072026/11/12 00:23:23 INFO Received complete push request method=POST path=/47082026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47092026/11/12 00:23:23 INFO Completed upload id=18 objects=2 duration=2.316299ms47102026/11/12 00:23:23 INFO Completed upload id=30 objects=5 duration=1.09799ms47112026/11/12 00:23:23 INFO Received uploads request method=POST path=/47122026/11/12 00:23:23 INFO Completed push id=24 objects=24 duration=3.715979ms47132026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47142026/11/12 00:23:23 INFO Completed upload id=31 objects=3 duration=1.10642ms47152026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47162026/11/12 00:23:23 INFO Completed upload id=30 objects=20 duration=1.09812ms47172026/11/12 00:23:23 INFO Received push request method=POST path=/47182026/11/12 00:23:23 INFO Received uploads request method=POST path=/47192026/11/12 00:23:23 INFO Received complete push request method=POST path=/47202026/11/12 00:23:23 INFO Received complete push request method=POST path=/47212026/11/12 00:23:23 INFO Completed push id=7 objects=7 duration=2.087959ms47222026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=047232026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47242026/11/12 00:23:23 INFO Completed push id=25 objects=7 duration=1.82597ms47252026/11/12 00:23:23 INFO Completed upload id=12 objects=9 duration=1.323659ms47262026/11/12 00:23:23 INFO Received uploads request method=POST path=/47272026/11/12 00:23:23 INFO Received complete push request method=POST path=/47282026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period47292026/11/12 00:23:23 INFO Completed push id=31 objects=3 duration=1.399229ms47302026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047312026/11/12 00:23:23 INFO Vacuumed table table=pending_closures47322026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=047332026/11/12 00:23:23 INFO Vacuumed table table=pending_objects47342026/11/12 00:23:23 INFO Received complete push request method=POST path=/47352026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads47362026/11/12 00:23:23 INFO Vacuumed table table=closures47372026/11/12 00:23:23 INFO Completed push id=31 objects=11 duration=1.33371ms47382026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period47392026/11/12 00:23:23 INFO Received complete push request method=POST path=/47402026/11/12 00:23:23 INFO Vacuumed table table=objects47412026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47422026/11/12 00:23:23 INFO Completed push id=12 objects=6 duration=1.4459ms47432026/11/12 00:23:23 INFO Received complete push request method=POST path=/47442026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047452026/11/12 00:23:23 INFO Completed push id=29 objects=24 duration=1.89096ms47462026/11/12 00:23:23 INFO Completed upload id=21 objects=9 duration=2.660399ms47472026/11/12 00:23:23 INFO Vacuumed table table=pending_closures47482026/11/12 00:23:23 INFO Vacuumed table table=pending_objects47492026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads47502026/11/12 00:23:23 INFO Vacuumed table table=closures47512026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47522026/11/12 00:23:23 INFO Vacuumed table table=objects47532026/11/12 00:23:23 INFO Received complete push request method=POST path=/47542026/11/12 00:23:23 INFO Completed upload id=13 objects=7 duration=1.197779ms47552026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47562026/11/12 00:23:23 INFO Completed push id=11 objects=6 duration=2.03177ms47572026/11/12 00:23:23 INFO Completed upload id=32 objects=2 duration=803.099µs47582026/11/12 00:23:23 INFO Received uploads request method=POST path=/47592026/11/12 00:23:23 INFO Received complete push request method=POST path=/47602026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=047612026/11/12 00:23:23 INFO Completed push id=32 objects=6 duration=1.28536ms47622026/11/12 00:23:23 INFO Received complete push request method=POST path=/47632026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period47642026/11/12 00:23:23 INFO Completed push id=25 objects=10 duration=1.868569ms47652026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47662026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47672026/11/12 00:23:23 INFO Completed upload id=26 objects=24 duration=1.569389ms47682026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047692026/11/12 00:23:23 INFO Completed upload id=20 objects=4 duration=1.58408ms47702026/11/12 00:23:23 INFO Received complete push request method=POST path=/47712026/11/12 00:23:23 INFO Vacuumed table table=pending_closures47722026/11/12 00:23:23 INFO Completed push id=33 objects=3 duration=987.92µs47732026/11/12 00:23:23 INFO Vacuumed table table=pending_objects47742026/11/12 00:23:23 INFO Received uploads request method=POST path=/47752026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads47762026/11/12 00:23:23 INFO Received complete push request method=POST path=/47772026/11/12 00:23:23 INFO Vacuumed table table=closures47782026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47792026/11/12 00:23:23 INFO Completed push id=20 objects=13 duration=1.530729ms47802026/11/12 00:23:23 INFO Vacuumed table table=objects47812026/11/12 00:23:23 INFO Completed upload id=33 objects=7 duration=1.302959ms47822026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47832026/11/12 00:23:23 INFO Completed upload id=29 objects=5 duration=980.859µs47842026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47852026/11/12 00:23:23 INFO Completed upload id=23 objects=7 duration=1.24273ms47862026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47872026/11/12 00:23:23 INFO Received complete push request method=POST path=/47882026/11/12 00:23:23 INFO Completed upload id=34 objects=3 duration=866.66µs47892026/11/12 00:23:23 INFO Completed push id=13 objects=22 duration=1.693009ms47902026/11/12 00:23:23 INFO Received complete push request method=POST path=/47912026/11/12 00:23:23 INFO Completed push id=22 objects=4 duration=1.6631ms47922026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47932026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47942026/11/12 00:23:23 INFO Completed upload id=34 objects=3 duration=1.08048ms47952026/11/12 00:23:23 INFO Completed upload id=30 objects=2 duration=1.05746ms47962026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=047972026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period47982026/11/12 00:23:23 INFO Received complete upload request method=POST path=/47992026/11/12 00:23:23 INFO Received complete push request method=POST path=/48002026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048012026/11/12 00:23:23 INFO Completed upload id=27 objects=26 duration=1.395469ms48022026/11/12 00:23:23 INFO Completed push id=21 objects=6 duration=1.361739ms48032026/11/12 00:23:23 INFO Received push request method=POST path=/48042026/11/12 00:23:23 INFO Vacuumed table table=pending_closures48052026/11/12 00:23:23 INFO Received complete push request method=POST path=/48062026/11/12 00:23:23 INFO Vacuumed table table=pending_objects48072026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads48082026/11/12 00:23:23 INFO Completed push id=35 objects=11 duration=1.38155ms48092026/11/12 00:23:23 INFO Vacuumed table table=closures48102026/11/12 00:23:23 INFO Received complete push request method=POST path=/48112026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48122026/11/12 00:23:23 INFO Vacuumed table table=objects48132026/11/12 00:23:23 INFO Completed push id=31 objects=20 duration=1.3293ms48142026/11/12 00:23:23 INFO Completed upload id=23 objects=7 duration=1.53987ms48152026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48162026/11/12 00:23:23 INFO Completed upload id=18 objects=22 duration=1.701489ms48172026/11/12 00:23:23 INFO Received complete push request method=POST path=/48182026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48192026/11/12 00:23:23 INFO Completed push id=28 objects=6 duration=1.34622ms48202026/11/12 00:23:23 INFO Completed upload id=22 objects=7 duration=1.67073ms48212026/11/12 00:23:23 INFO Received uploads request method=POST path=/48222026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48232026/11/12 00:23:23 INFO Completed upload id=24 objects=18 duration=1.603759ms48242026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=048252026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48262026/11/12 00:23:23 INFO Received complete push request method=POST path=/48272026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period48282026/11/12 00:23:23 INFO Completed upload id=32 objects=5 duration=986.7µs48292026/11/12 00:23:23 INFO Completed push id=24 objects=5 duration=1.31569ms48302026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48312026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048322026/11/12 00:23:23 INFO Completed upload id=21 objects=8 duration=1.00831ms48332026/11/12 00:23:23 INFO Vacuumed table table=pending_closures48342026/11/12 00:23:23 INFO Received complete push request method=POST path=/48352026/11/12 00:23:23 INFO Vacuumed table table=pending_objects48362026/11/12 00:23:23 INFO Received complete push request method=POST path=/48372026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads48382026/11/12 00:23:23 INFO Completed push id=29 objects=6 duration=1.224779ms48392026/11/12 00:23:23 INFO Vacuumed table table=closures48402026/11/12 00:23:23 INFO Completed push id=24 objects=4 duration=1.36833ms48412026/11/12 00:23:23 INFO Received complete push request method=POST path=/48422026/11/12 00:23:23 INFO Vacuumed table table=objects48432026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=048442026/11/12 00:23:23 INFO Completed push id=25 objects=2 duration=1.786019ms48452026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period48462026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048472026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48482026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48492026/11/12 00:23:23 INFO Completed upload id=33 objects=3 duration=866.019µs48502026/11/12 00:23:23 INFO Completed upload id=25 objects=2 duration=1.336609ms48512026/11/12 00:23:23 INFO Vacuumed table table=pending_closures48522026/11/12 00:23:23 INFO Vacuumed table table=pending_objects48532026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads48542026/11/12 00:23:23 INFO Vacuumed table table=closures48552026/11/12 00:23:23 INFO Vacuumed table table=objects48562026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48572026/11/12 00:23:23 INFO Received complete push request method=POST path=/48582026/11/12 00:23:23 INFO Completed upload id=22 objects=7 duration=1.10143ms48592026/11/12 00:23:23 INFO Completed push id=25 objects=9 duration=1.393759ms48602026/11/12 00:23:23 INFO Received complete push request method=POST path=/48612026/11/12 00:23:23 INFO Completed push id=27 objects=4 duration=1.11964ms48622026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=048632026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48642026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period48652026/11/12 00:23:23 INFO Completed upload id=26 objects=20 duration=1.204479ms48662026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048672026/11/12 00:23:23 INFO Received complete push request method=POST path=/48682026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48692026/11/12 00:23:23 INFO Vacuumed table table=pending_closures48702026/11/12 00:23:23 INFO Completed push id=24 objects=5 duration=1.147679ms48712026/11/12 00:23:23 INFO Completed upload id=26 objects=3 duration=1.43088ms48722026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48732026/11/12 00:23:23 INFO Vacuumed table table=pending_objects48742026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads48752026/11/12 00:23:23 INFO Completed upload id=30 objects=8 duration=1.314039ms48762026/11/12 00:23:23 INFO Vacuumed table table=closures48772026/11/12 00:23:23 INFO Vacuumed table table=objects48782026/11/12 00:23:23 INFO Received push request method=POST path=/48792026/11/12 00:23:23 INFO Received complete push request method=POST path=/48802026/11/12 00:23:23 INFO Received complete push request method=POST path=/48812026/11/12 00:23:23 INFO Completed push id=28 objects=7 duration=1.27969ms48822026/11/12 00:23:23 INFO Completed push id=28 objects=26 duration=1.560319ms48832026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48842026/11/12 00:23:23 INFO Completed upload id=25 objects=8 duration=1.07902ms48852026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48862026/11/12 00:23:23 INFO Completed upload id=27 objects=7 duration=1.02953ms48872026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48882026/11/12 00:23:23 INFO Completed upload id=31 objects=6 duration=1.27794ms48892026/11/12 00:23:23 INFO Received complete push request method=POST path=/48902026/11/12 00:23:23 INFO Completed push id=30 objects=3 duration=1.17678ms48912026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48922026/11/12 00:23:23 INFO Received complete push request method=POST path=/48932026/11/12 00:23:23 INFO Completed push id=26 objects=4 duration=1.10337ms48942026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48952026/11/12 00:23:23 INFO Received complete upload request method=POST path=/48962026/11/12 00:23:23 INFO Completed upload id=23 objects=7 duration=2.861258ms48972026/11/12 00:23:23 INFO Completed upload id=30 objects=6 duration=1.113119ms48982026/11/12 00:23:23 INFO Completed upload id=28 objects=3 duration=1.495259ms48992026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49002026/11/12 00:23:23 INFO Completed upload id=32 objects=4 duration=1.091519ms49012026/11/12 00:23:23 INFO Received complete push request method=POST path=/49022026/11/12 00:23:23 INFO Received complete push request method=POST path=/49032026/11/12 00:23:23 INFO Completed push id=32 objects=19 duration=1.332509ms49042026/11/12 00:23:23 INFO Completed push id=30 objects=2 duration=1.437109ms49052026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49062026/11/12 00:23:23 INFO Completed upload id=26 objects=5 duration=1.234059ms49072026/11/12 00:23:23 INFO Received complete push request method=POST path=/49082026/11/12 00:23:23 INFO Completed push id=31 objects=5 duration=1.18035ms49092026/11/12 00:23:23 INFO Received complete push request method=POST path=/49102026/11/12 00:23:23 INFO Completed push id=29 objects=23 duration=1.4438ms49112026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49122026/11/12 00:23:23 INFO Completed upload id=33 objects=10 duration=1.26689ms49132026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=049142026/11/12 00:23:23 INFO Received complete push request method=POST path=/49152026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period49162026/11/12 00:23:23 INFO Completed push id=27 objects=3 duration=2.022709ms49172026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049182026/11/12 00:23:23 INFO Vacuumed table table=pending_closures49192026/11/12 00:23:23 INFO Vacuumed table table=pending_objects49202026/11/12 00:23:23 INFO Received complete push request method=POST path=/49212026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads49222026/11/12 00:23:23 INFO Vacuumed table table=closures49232026/11/12 00:23:23 INFO Completed push id=30 objects=6 duration=1.727559ms49242026/11/12 00:23:23 INFO Received complete push request method=POST path=/49252026/11/12 00:23:23 INFO Vacuumed table table=objects49262026/11/12 00:23:23 INFO Completed push id=32 objects=7 duration=1.482249ms49272026/11/12 00:23:23 INFO Received complete push request method=POST path=/49282026/11/12 00:23:23 INFO Completed push id=32 objects=26 duration=1.564029ms49292026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49302026/11/12 00:23:23 INFO Completed upload id=28 objects=5 duration=1.220609ms49312026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49322026/11/12 00:23:23 INFO Completed upload id=34 objects=23 duration=1.347739ms49332026/11/12 00:23:23 INFO Received complete push request method=POST path=/49342026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49352026/11/12 00:23:23 INFO Completed upload id=33 objects=2 duration=1.076229ms49362026/11/12 00:23:23 INFO Completed push id=33 objects=3 duration=1.827299ms49372026/11/12 00:23:23 INFO Received complete push request method=POST path=/49382026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49392026/11/12 00:23:23 INFO Completed push id=29 objects=17 duration=1.358409ms49402026/11/12 00:23:23 INFO Completed upload id=33 objects=6 duration=1.149339ms49412026/11/12 00:23:23 INFO Received complete push request method=POST path=/49422026/11/12 00:23:23 INFO Received complete push request method=POST path=/49432026/11/12 00:23:23 INFO Completed push id=34 objects=6 duration=1.28943ms49442026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49452026/11/12 00:23:23 INFO Completed push id=35 objects=6 duration=1.56817ms49462026/11/12 00:23:23 INFO Completed upload id=34 objects=5 duration=1.06312ms49472026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49482026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=049492026/11/12 00:23:23 INFO Completed upload id=34 objects=3 duration=998.759µs49502026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period49512026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049522026/11/12 00:23:23 INFO Vacuumed table table=pending_closures49532026/11/12 00:23:23 INFO Vacuumed table table=pending_objects49542026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads49552026/11/12 00:23:23 INFO Vacuumed table table=closures49562026/11/12 00:23:23 INFO Vacuumed table table=objects49572026/11/12 00:23:23 INFO Received complete push request method=POST path=/49582026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=049592026/11/12 00:23:23 INFO Completed push id=35 objects=8 duration=1.25195ms49602026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49612026/11/12 00:23:23 INFO Completed upload id=35 objects=11 duration=1.550539ms49622026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period49632026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=049642026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period49652026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049662026/11/12 00:23:23 INFO Vacuumed table table=pending_closures49672026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049682026/11/12 00:23:23 INFO Vacuumed table table=pending_objects49692026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads49702026/11/12 00:23:23 INFO Vacuumed table table=pending_closures49712026/11/12 00:23:23 INFO Vacuumed table table=closures49722026/11/12 00:23:23 INFO Vacuumed table table=pending_objects49732026/11/12 00:23:23 INFO Vacuumed table table=objects49742026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads49752026/11/12 00:23:23 INFO Vacuumed table table=closures49762026/11/12 00:23:23 INFO Vacuumed table table=objects49772026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49782026/11/12 00:23:23 INFO Completed upload id=36 objects=9 duration=1.06996ms49792026/11/12 00:23:23 INFO Received complete upload request method=POST path=/49802026/11/12 00:23:23 INFO Completed upload id=36 objects=3 duration=1.038709ms49812026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=049822026/11/12 00:23:23 INFO Aborted multipart uploads count=0 kept=049832026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period49842026/11/12 00:23:23 WARN Force mode enabled - objects will be deleted immediately without grace period49852026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049862026/11/12 00:23:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049872026/11/12 00:23:23 INFO Vacuumed table table=pending_closures49882026/11/12 00:23:23 INFO Vacuumed table table=pending_closures49892026/11/12 00:23:23 INFO Vacuumed table table=pending_objects49902026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads49912026/11/12 00:23:23 INFO Vacuumed table table=pending_objects49922026/11/12 00:23:23 INFO Vacuumed table table=multipart_uploads49932026/11/12 00:23:23 INFO Vacuumed table table=closures49942026/11/12 00:23:23 INFO Vacuumed table table=closures49952026/11/12 00:23:23 INFO Vacuumed table table=objects49962026/11/12 00:23:23 INFO Vacuumed table table=objects4997--- PASS: FuzzPushGC (0.00s)4998 --- PASS: FuzzPushGC/seed#14 (2.06s)4999 --- PASS: FuzzPushGC/seed#9 (2.22s)5000 --- PASS: FuzzPushGC/seed#0 (2.24s)5001 --- PASS: FuzzPushGC/seed#3 (2.36s)5002 --- PASS: FuzzPushGC/seed#13 (2.36s)5003 --- PASS: FuzzPushGC/seed#15 (2.43s)5004 --- PASS: FuzzPushGC/seed#2 (2.45s)5005 --- PASS: FuzzPushGC/seed#5 (2.48s)5006 --- PASS: FuzzPushGC/seed#8 (2.51s)5007 --- PASS: FuzzPushGC/seed#6 (2.54s)5008 --- PASS: FuzzPushGC/seed#7 (2.61s)5009 --- PASS: FuzzPushGC/seed#1 (2.67s)5010 --- PASS: FuzzPushGC/seed#10 (2.69s)5011 --- PASS: FuzzPushGC/seed#12 (2.69s)5012 --- PASS: FuzzPushGC/seed#4 (2.72s)5013 --- PASS: FuzzPushGC/seed#11 (2.72s)5014PASS5015{"timestamp":"2026-11-12T00:23:23.416441164Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49092","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(395)"}5016{"timestamp":"2026-11-12T00:23:23.416446874Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:50496","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(399)"}5017{"timestamp":"2026-11-12T00:23:23.416729384Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49694","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(390)"}50182026-11-12 00:23:23.842 UTC [128] LOG: received smart shutdown request50192026-11-12 00:23:23.847 UTC [128] LOG: background worker "logical replication launcher" (PID 138) exited with exit code 150202026-11-12 00:23:24.463 UTC [133] LOG: checkpoint complete: wrote 4227 buffers (25.8%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=3.225 s, sync=0.449 s, total=3.749 s; sync files=42363, longest=0.009 s, average=0.001 s; distance=535367 kB, estimate=535367 kB; lsn=0/26D7A2C8, redo lsn=0/2223796850212026-11-12 00:23:24.467 UTC [133] LOG: shutting down50222026-11-12 00:23:24.467 UTC [133] LOG: checkpoint starting: shutdown immediate50232026-11-12 00:23:25.570 UTC [133] LOG: checkpoint complete: wrote 9511 buffers (58.1%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.188 s, sync=0.897 s, total=1.104 s; sync files=2844, longest=0.002 s, average=0.001 s; distance=77066 kB, estimate=489537 kB; lsn=0/26D7A340, redo lsn=0/26D7A34050242026-11-12 00:23:25.665 UTC [128] LOG: database system is shut down5025Running OIDC tests...5026=== RUN TestAudienceForIssuer5027=== PAUSE TestAudienceForIssuer5028=== RUN TestGlobMatch5029=== PAUSE TestGlobMatch5030=== RUN TestValidateToken_ValidToken5031=== PAUSE TestValidateToken_ValidToken5032=== RUN TestValidateToken_WrongAudience5033=== PAUSE TestValidateToken_WrongAudience5034=== RUN TestValidateToken_Expired5035=== PAUSE TestValidateToken_Expired5036=== RUN TestValidateToken_BoundClaimsMismatch5037=== PAUSE TestValidateToken_BoundClaimsMismatch5038=== RUN TestValidateToken_BoundSubjectMismatch5039=== PAUSE TestValidateToken_BoundSubjectMismatch5040=== RUN TestValidateToken_MultipleProviders5041=== PAUSE TestValidateToken_MultipleProviders5042=== RUN TestValidateToken_NoMatchingProvider5043=== PAUSE TestValidateToken_NoMatchingProvider5044=== RUN TestValidateToken_KubernetesServiceAccount5045=== PAUSE TestValidateToken_KubernetesServiceAccount5046=== RUN TestNewValidator_KubernetesRequiresCA5047=== PAUSE TestNewValidator_KubernetesRequiresCA5048=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5049=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5050=== RUN TestPins_ReservedForMatchingRule5051=== PAUSE TestPins_ReservedForMatchingRule5052=== RUN TestPins_TopLevelShorthand5053=== PAUSE TestPins_TopLevelShorthand5054=== RUN TestPins_ConfigValidation5055=== PAUSE TestPins_ConfigValidation5056=== RUN TestScopes_LegacyProviderDefaultsToWrite5057=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5058=== RUN TestScopes_Rules5059=== PAUSE TestScopes_Rules5060=== RUN TestScopes_ConfigValidation5061=== PAUSE TestScopes_ConfigValidation5062=== CONT TestAudienceForIssuer5063=== CONT TestValidateToken_KubernetesServiceAccount5064--- PASS: TestAudienceForIssuer (0.00s)5065=== CONT TestValidateToken_NoMatchingProvider5066=== CONT TestValidateToken_MultipleProviders5067=== CONT TestValidateToken_BoundSubjectMismatch5068=== CONT TestValidateToken_BoundClaimsMismatch5069=== CONT TestValidateToken_Expired5070=== CONT TestValidateToken_WrongAudience5071=== CONT TestValidateToken_ValidToken5072=== CONT TestGlobMatch5073=== RUN TestGlobMatch/foo_foo5074=== PAUSE TestGlobMatch/foo_foo5075=== RUN TestGlobMatch/foo_bar5076=== CONT TestPins_ConfigValidation5077=== CONT TestScopes_ConfigValidation5078=== CONT TestScopes_Rules5079=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5080=== CONT TestScopes_LegacyProviderDefaultsToWrite5081=== CONT TestPins_TopLevelShorthand5082=== CONT TestNewValidator_KubernetesRequiresCA5083=== CONT TestPins_ReservedForMatchingRule5084=== PAUSE TestGlobMatch/foo_bar5085=== RUN TestGlobMatch/*_5086=== PAUSE TestGlobMatch/*_5087=== RUN TestGlobMatch/*_anything5088=== PAUSE TestGlobMatch/*_anything5089=== RUN TestGlobMatch/foo*_foo5090=== PAUSE TestGlobMatch/foo*_foo5091=== RUN TestGlobMatch/foo*_foobar5092=== PAUSE TestGlobMatch/foo*_foobar5093=== RUN TestGlobMatch/foo*_bar5094=== PAUSE TestGlobMatch/foo*_bar5095=== RUN TestGlobMatch/*bar_bar5096=== PAUSE TestGlobMatch/*bar_bar5097=== RUN TestGlobMatch/*bar_foobar5098=== PAUSE TestGlobMatch/*bar_foobar5099=== RUN TestGlobMatch/*bar_foo5100=== PAUSE TestGlobMatch/*bar_foo5101=== RUN TestGlobMatch/foo*bar_foobar5102=== PAUSE TestGlobMatch/foo*bar_foobar5103=== RUN TestGlobMatch/foo*bar_foo123bar5104=== PAUSE TestGlobMatch/foo*bar_foo123bar5105=== RUN TestGlobMatch/foo*bar_foobarbaz5106=== PAUSE TestGlobMatch/foo*bar_foobarbaz5107=== RUN TestGlobMatch/*/*_foo/bar5108=== PAUSE TestGlobMatch/*/*_foo/bar5109=== RUN TestGlobMatch/*/*_foo5110=== PAUSE TestGlobMatch/*/*_foo5111=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5112=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5113=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05114=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05115=== RUN TestGlobMatch/refs/*/main_refs/heads/main5116=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5117=== RUN TestGlobMatch/fo?_foo5118=== PAUSE TestGlobMatch/fo?_foo5119=== RUN TestGlobMatch/fo?_fo5120=== PAUSE TestGlobMatch/fo?_fo5121=== RUN TestGlobMatch/fo?_fooo5122=== PAUSE TestGlobMatch/fo?_fooo5123=== RUN TestGlobMatch/?oo_foo5124=== PAUSE TestGlobMatch/?oo_foo5125=== RUN TestGlobMatch/?oo_boo5126=== PAUSE TestGlobMatch/?oo_boo5127=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5128=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5129=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5130=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5131=== CONT TestGlobMatch/foo_foo5132=== CONT TestGlobMatch/?oo_boo5133=== CONT TestGlobMatch/?oo_foo5134=== CONT TestGlobMatch/fo?_fooo5135=== CONT TestGlobMatch/fo?_fo5136=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5137=== CONT TestGlobMatch/foo*bar_foo123bar5138=== CONT TestGlobMatch/foo*bar_foobar5139=== CONT TestGlobMatch/*bar_foo5140=== CONT TestGlobMatch/*bar_foobar5141=== CONT TestGlobMatch/*bar_bar5142=== CONT TestGlobMatch/foo*_bar5143=== CONT TestGlobMatch/foo*_foobar5144=== CONT TestGlobMatch/foo*_foo5145=== CONT TestGlobMatch/*_anything5146=== CONT TestGlobMatch/*_5147=== CONT TestGlobMatch/foo_bar5148=== CONT TestGlobMatch/fo?_foo5149=== CONT TestGlobMatch/refs/*/main_refs/heads/main5150=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05151--- PASS: TestPins_ConfigValidation (0.00s)5152=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5153=== CONT TestGlobMatch/*/*_foo5154=== CONT TestGlobMatch/*/*_foo/bar5155=== CONT TestGlobMatch/foo*bar_foobarbaz5156=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5157--- PASS: TestScopes_ConfigValidation (0.00s)5158--- PASS: TestGlobMatch (0.00s)5159 --- PASS: TestGlobMatch/foo_foo (0.00s)5160 --- PASS: TestGlobMatch/?oo_boo (0.00s)5161 --- PASS: TestGlobMatch/?oo_foo (0.00s)5162 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5163 --- PASS: TestGlobMatch/fo?_fo (0.00s)5164 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5165 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5166 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5167 --- PASS: TestGlobMatch/*bar_foo (0.00s)5168 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5169 --- PASS: TestGlobMatch/*bar_bar (0.00s)5170 --- PASS: TestGlobMatch/foo*_bar (0.00s)5171 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5172 --- PASS: TestGlobMatch/foo*_foo (0.00s)5173 --- PASS: TestGlobMatch/*_anything (0.00s)5174 --- PASS: TestGlobMatch/*_ (0.00s)5175 --- PASS: TestGlobMatch/foo_bar (0.00s)5176 --- PASS: TestGlobMatch/fo?_foo (0.00s)5177 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5178 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5179 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5180 --- PASS: TestGlobMatch/*/*_foo (0.00s)5181 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5182 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5183 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)51842026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41221/oidc51852026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45301/oidc5186--- PASS: TestPins_TopLevelShorthand (0.02s)5187--- PASS: TestValidateToken_BoundSubjectMismatch (0.03s)51882026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34297/oidc5189--- PASS: TestValidateToken_BoundClaimsMismatch (0.05s)51902026/11/12 00:23:28 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235191--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.05s)51922026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:32929/oidc5193--- PASS: TestValidateToken_ValidToken (0.06s)51942026/11/12 00:23:28 http: TLS handshake error from 127.0.0.1:59108: remote error: tls: bad certificate5195--- PASS: TestNewValidator_KubernetesRequiresCA (0.06s)51962026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33487/oidc5197--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.07s)51982026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40951/oidc5199--- PASS: TestValidateToken_Expired (0.08s)52002026/11/12 00:23:28 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:3793552012026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42955/oidc5202--- PASS: TestValidateToken_KubernetesServiceAccount (0.09s)5203--- PASS: TestScopes_Rules (0.09s)52042026/11/12 00:23:28 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:40941/oidc52052026/11/12 00:23:28 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:38819/oidc5206--- PASS: TestValidateToken_MultipleProviders (0.13s)52072026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41633/oidc5208--- PASS: TestPins_ReservedForMatchingRule (0.13s)52092026/11/12 00:23:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42425/oidc5210--- PASS: TestValidateToken_WrongAudience (0.20s)52112026/11/12 00:23:28 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:44503/oidc5212--- PASS: TestValidateToken_NoMatchingProvider (0.24s)5213PASS5214Running hook tests...5215=== RUN TestSendPathsEmpty5216=== PAUSE TestSendPathsEmpty5217=== RUN TestQueueEnqueueAndFetch5218=== PAUSE TestQueueEnqueueAndFetch5219=== RUN TestQueueDeduplication5220=== PAUSE TestQueueDeduplication5221=== RUN TestQueueRemove5222=== PAUSE TestQueueRemove5223=== RUN TestQueueFetchBatchLimit5224=== PAUSE TestQueueFetchBatchLimit5225=== RUN TestQueueRetryMovesToBack5226=== PAUSE TestQueueRetryMovesToBack5227=== RUN TestQueueFetchRemoveLifecycle5228=== PAUSE TestQueueFetchRemoveLifecycle5229=== RUN TestQueueConcurrentWriters5230=== PAUSE TestQueueConcurrentWriters5231=== RUN TestQueueRemoveLargeClosure5232=== PAUSE TestQueueRemoveLargeClosure5233=== RUN TestServerClientIntegration5234--- PASS: TestServerClientIntegration (0.00s)5235=== RUN TestServerQueueError52362026/11/12 00:23:28 ERROR Failed to queue paths error="permission denied" count=15237--- PASS: TestServerQueueError (0.00s)5238=== RUN TestGetListenerSocketActivation5239 server_test.go:220: === RUN TestGetListenerSocketActivation5240 --- PASS: TestGetListenerSocketActivation (0.00s)5241 PASS5242 5243--- PASS: TestGetListenerSocketActivation (0.01s)5244=== RUN TestServerStalledClientDoesNotBlockShutdown5245=== PAUSE TestServerStalledClientDoesNotBlockShutdown5246=== RUN TestServerBacksOffOnAcceptErrors5247=== PAUSE TestServerBacksOffOnAcceptErrors5248=== RUN TestDrainIsolatesPoisonPath5249=== PAUSE TestDrainIsolatesPoisonPath5250=== RUN TestRunNotBlockedByPoisonHead5251=== PAUSE TestRunNotBlockedByPoisonHead5252=== RUN TestDrainGivesUpWhenServerDown5253=== PAUSE TestDrainGivesUpWhenServerDown5254=== RUN TestFailedPathPrunedByLaterClosure5255=== PAUSE TestFailedPathPrunedByLaterClosure5256=== RUN TestWorkerUploadsAndRemoves5257=== PAUSE TestWorkerUploadsAndRemoves5258=== RUN TestWorkerSkipsGCdPaths5259=== PAUSE TestWorkerSkipsGCdPaths5260=== RUN TestWorkerPrunesClosureDeps5261=== PAUSE TestWorkerPrunesClosureDeps5262=== RUN TestDrainTimeout5263=== PAUSE TestDrainTimeout5264=== RUN TestWorkerRemoveFailureIsNotProgress5265=== PAUSE TestWorkerRemoveFailureIsNotProgress5266=== RUN TestWorkerKeepsPathItCannotStat5267=== PAUSE TestWorkerKeepsPathItCannotStat5268=== CONT TestSendPathsEmpty5269=== CONT TestDrainIsolatesPoisonPath5270--- PASS: TestSendPathsEmpty (0.00s)5271=== CONT TestQueueRetryMovesToBack5272=== CONT TestQueueFetchBatchLimit5273=== CONT TestQueueFetchRemoveLifecycle5274=== CONT TestQueueRemove5275=== CONT TestQueueDeduplication5276=== CONT TestServerBacksOffOnAcceptErrors5277=== CONT TestServerStalledClientDoesNotBlockShutdown52782000/01/01 00:00:00 ERROR Accept failed error="too many open files"5279=== CONT TestQueueEnqueueAndFetch5280=== CONT TestWorkerSkipsGCdPaths52812000/01/01 00:00:00 ERROR Accept failed error="too many open files"5282=== CONT TestWorkerKeepsPathItCannotStat52832000/01/01 00:00:00 ERROR Accept failed error="too many open files"5284=== CONT TestWorkerRemoveFailureIsNotProgress52852000/01/01 00:00:00 ERROR Accept failed error="too many open files"52862000/01/01 00:00:00 ERROR Accept failed error="too many open files"52872000/01/01 00:00:00 ERROR Accept failed error="too many open files"52882000/01/01 00:00:00 ERROR Accept failed error="too many open files"52892000/01/01 00:00:00 ERROR Accept failed error="too many open files"5290=== CONT TestDrainTimeout5291=== CONT TestQueueConcurrentWriters5292=== CONT TestFailedPathPrunedByLaterClosure5293=== CONT TestWorkerPrunesClosureDeps5294=== CONT TestQueueRemoveLargeClosure5295=== CONT TestWorkerUploadsAndRemoves5296=== CONT TestDrainGivesUpWhenServerDown5297=== CONT TestRunNotBlockedByPoisonHead5298--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)52992000/01/01 00:00:00 INFO Uploading batch count=253002026/11/12 00:23:28 INFO Uploading batch count=453012026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=453022026/11/12 00:23:28 INFO Upload queue status pending=253032026/11/12 00:23:28 INFO Upload queue status pending=253042000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253052026/11/12 00:23:28 INFO Uploading batch count=253062026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=253072026/11/12 00:23:28 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3939717689/002/a53082000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453092026/11/12 00:23:28 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath1262104459/002/bbb53102026/11/12 00:23:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths547218152/002/nonexistent53112026/11/12 00:23:28 INFO Uploading batch count=153122026/11/12 00:23:28 INFO Uploading batch count=153132026/11/12 00:23:28 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3231271170/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3231271170/002/locked/aaa: permission denied"53142026/11/12 00:23:28 INFO Upload queue status pending=35315--- PASS: TestQueueEnqueueAndFetch (0.02s)53162026/11/12 00:23:28 INFO Upload queue status pending=253172026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=153182026/11/12 00:23:28 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3939717689/002/b5319--- PASS: TestQueueFetchRemoveLifecycle (0.02s)5320--- PASS: TestQueueFetchBatchLimit (0.02s)53212026/11/12 00:23:28 INFO Uploading batch count=153222026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=153232026/11/12 00:23:28 INFO Uploading batch count=25324--- PASS: TestQueueDeduplication (0.02s)53252026/11/12 00:23:28 INFO Uploading batch count=153262026/11/12 00:23:28 INFO Uploading batch count=153272026/11/12 00:23:28 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3231271170/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3231271170/002/locked/aaa: permission denied"5328--- PASS: TestDrainTimeout (0.02s)53292026/11/12 00:23:28 INFO Uploading batch count=153302026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=153312026/11/12 00:23:28 INFO Uploading batch count=153322026/11/12 00:23:28 INFO Uploading batch count=253332026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=253342026/11/12 00:23:28 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3939717689/002/c53352026/11/12 00:23:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress807793742/002/nonexistent5336--- PASS: TestQueueRemove (0.02s)53372026/11/12 00:23:28 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153382026/11/12 00:23:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress807793742/002/nonexistent53392026/11/12 00:23:28 INFO Uploading batch count=153402026/11/12 00:23:28 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat3231271170/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat3231271170/002/locked/aaa: permission denied"53412026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=15342--- PASS: TestQueueRetryMovesToBack (0.02s)53432026/11/12 00:23:28 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153442026/11/12 00:23:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress807793742/002/nonexistent53452026/11/12 00:23:28 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3939717689/002/d53462026/11/12 00:23:28 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153472026/11/12 00:23:28 ERROR Drain finished with paths left in queue remaining=153482026/11/12 00:23:28 ERROR Drain finished with paths left in queue remaining=153492026/11/12 00:23:28 INFO Uploading batch count=153502026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=153512026/11/12 00:23:28 ERROR Drain finished with paths left in queue remaining=15352--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)53532026/11/12 00:23:28 INFO Uploading batch count=253542026/11/12 00:23:28 ERROR Upload failed error="upload failed" count=253552026/11/12 00:23:28 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3939717689/002/e5356--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)53572026/11/12 00:23:28 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3939717689/002/f5358--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)53592026/11/12 00:23:28 ERROR Drain finished with paths left in queue remaining=105360--- PASS: TestDrainIsolatesPoisonPath (0.02s)5361--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5362--- PASS: TestWorkerSkipsGCdPaths (0.04s)5363--- PASS: TestWorkerPrunesClosureDeps (0.03s)5364--- PASS: TestWorkerUploadsAndRemoves (0.04s)5365--- PASS: TestQueueRemoveLargeClosure (0.05s)53662026/11/12 00:23:28 ERROR Failed to decode request error="read unix /build/hook3908676471/test.sock->@: i/o timeout"53672026/11/12 00:23:28 ERROR Failed to write response error="write unix /build/hook3908676471/test.sock->@: i/o timeout"5368--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)5369--- PASS: TestQueueConcurrentWriters (0.21s)53702026/11/12 00:23:29 INFO Uploading batch count=153712026/11/12 00:23:29 INFO Uploading batch count=153722026/11/12 00:23:29 INFO Uploading batch count=153732026/11/12 00:23:29 ERROR Upload failed error="upload failed" count=153742026/11/12 00:23:29 INFO Uploading batch count=153752026/11/12 00:23:29 ERROR Upload failed error="upload failed" count=153762026/11/12 00:23:29 INFO Uploading batch count=153772026/11/12 00:23:29 ERROR Upload failed error="upload failed" count=153782026/11/12 00:23:29 INFO Uploading batch count=153792026/11/12 00:23:29 ERROR Upload failed error="upload failed" count=153802026/11/12 00:23:29 ERROR Drain finished with paths left in queue remaining=15381--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5382PASS5383Running signing tests...5384=== RUN TestExternalSigner5385=== PAUSE TestExternalSigner5386=== RUN TestExternalSignerWrongPublicKey5387=== PAUSE TestExternalSignerWrongPublicKey5388=== RUN TestExternalSignerErrors5389=== PAUSE TestExternalSignerErrors5390=== RUN TestExternalSignerRestartsAfterCrash5391=== PAUSE TestExternalSignerRestartsAfterCrash5392=== RUN TestExternalSignerProgramErrorKeepsProcess5393=== PAUSE TestExternalSignerProgramErrorKeepsProcess5394=== RUN TestExternalSignerClose5395=== PAUSE TestExternalSignerClose5396=== RUN TestExternalSignerRejectsInvalidPublicKey5397=== PAUSE TestExternalSignerRejectsInvalidPublicKey5398=== RUN TestExternalSignerCancellation5399=== PAUSE TestExternalSignerCancellation5400=== RUN TestGenerateFingerprint5401=== PAUSE TestGenerateFingerprint5402=== RUN TestParseSigningKey5403=== PAUSE TestParseSigningKey5404=== RUN TestSign5405=== PAUSE TestSign5406=== RUN TestSignBatch5407=== PAUSE TestSignBatch5408=== RUN TestNilKeyPublicKey5409=== PAUSE TestNilKeyPublicKey5410=== RUN TestSignErrors5411=== PAUSE TestSignErrors5412=== CONT TestExternalSigner5413=== CONT TestExternalSignerCancellation5414=== CONT TestExternalSignerProgramErrorKeepsProcess5415=== CONT TestExternalSignerRejectsInvalidPublicKey5416=== CONT TestSignErrors5417=== CONT TestExternalSignerClose5418=== CONT TestNilKeyPublicKey5419--- PASS: TestNilKeyPublicKey (0.00s)5420=== CONT TestSignBatch5421=== CONT TestSign5422=== CONT TestExternalSignerWrongPublicKey5423=== CONT TestParseSigningKey5424=== RUN TestParseSigningKey/valid_32-byte_key5425=== CONT TestExternalSignerRestartsAfterCrash5426=== RUN TestSignErrors/nil_key5427=== CONT TestGenerateFingerprint5428=== CONT TestExternalSignerErrors5429=== RUN TestExternalSignerRejectsInvalidPublicKey/#005430=== PAUSE TestParseSigningKey/valid_32-byte_key5431=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5432=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5433=== RUN TestParseSigningKey/no_colon5434=== PAUSE TestParseSigningKey/no_colon5435=== RUN TestParseSigningKey/empty_name5436=== PAUSE TestParseSigningKey/empty_name5437=== RUN TestParseSigningKey/invalid_base645438--- PASS: TestSign (0.00s)5439=== PAUSE TestParseSigningKey/invalid_base645440=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005441=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5442=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5443=== RUN TestExternalSignerErrors/exit_at_startup5444=== PAUSE TestExternalSignerErrors/exit_at_startup5445=== RUN TestGenerateFingerprint/basic_with_references5446=== PAUSE TestGenerateFingerprint/basic_with_references5447=== RUN TestParseSigningKey/wrong_length5448=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5449--- PASS: TestSignBatch (0.00s)5450=== PAUSE TestSignErrors/nil_key5451=== RUN TestExternalSignerErrors/exit_on_request5452=== PAUSE TestExternalSignerErrors/exit_on_request5453=== RUN TestGenerateFingerprint/no_references5454=== PAUSE TestGenerateFingerprint/no_references5455=== PAUSE TestParseSigningKey/wrong_length5456=== CONT TestParseSigningKey/valid_32-byte_key5457=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5458=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5459=== RUN TestSignErrors/nil_narinfo5460=== PAUSE TestSignErrors/nil_narinfo5461=== RUN TestExternalSignerErrors/wrong_id5462=== RUN TestSignErrors/invalid_narinfo5463=== PAUSE TestSignErrors/invalid_narinfo5464=== RUN TestSignErrors/canceled5465=== PAUSE TestSignErrors/canceled5466=== CONT TestSignErrors/nil_key5467=== CONT TestSignErrors/invalid_narinfo5468=== PAUSE TestExternalSignerErrors/wrong_id5469=== CONT TestSignErrors/canceled5470=== CONT TestParseSigningKey/empty_name5471=== CONT TestParseSigningKey/wrong_length5472=== CONT TestParseSigningKey/invalid_base645473=== CONT TestParseSigningKey/no_colon5474=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5475=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5476=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5477=== CONT TestSignErrors/nil_narinfo5478=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5479=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5480=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5481=== RUN TestGenerateFingerprint/invalid_nar_hash_length5482=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5483=== RUN TestGenerateFingerprint/invalid_store_path_prefix5484=== RUN TestExternalSignerErrors/null_signatures5485=== PAUSE TestExternalSignerErrors/null_signatures5486--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5487=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5488=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5489=== CONT TestExternalSignerRejectsInvalidPublicKey/#005490=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5491=== RUN TestExternalSignerErrors/too_few_signatures5492=== PAUSE TestExternalSignerErrors/too_few_signatures5493--- PASS: TestParseSigningKey (0.00s)5494 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5495 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5496 --- PASS: TestParseSigningKey/empty_name (0.00s)5497 --- PASS: TestParseSigningKey/wrong_length (0.00s)5498 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5499 --- PASS: TestParseSigningKey/no_colon (0.00s)5500=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5501=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5502=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5503=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5504=== RUN TestExternalSignerErrors/wrong_signature_name5505=== PAUSE TestExternalSignerErrors/wrong_signature_name5506=== RUN TestExternalSignerErrors/invalid_signature_base645507=== PAUSE TestExternalSignerErrors/invalid_signature_base645508=== RUN TestExternalSignerErrors/wrong_shape5509=== PAUSE TestExternalSignerErrors/wrong_shape5510=== RUN TestExternalSignerErrors/missing_signatures5511=== PAUSE TestExternalSignerErrors/missing_signatures5512=== RUN TestExternalSignerErrors/wrong_signatures_shape5513=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5514=== RUN TestExternalSignerErrors/unrequested_key5515=== PAUSE TestExternalSignerErrors/unrequested_key5516--- PASS: TestSignErrors (0.00s)5517 --- PASS: TestSignErrors/nil_key (0.00s)5518 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5519 --- PASS: TestSignErrors/canceled (0.00s)5520 --- PASS: TestSignErrors/nil_narinfo (0.00s)5521=== RUN TestGenerateFingerprint/invalid_reference_prefix5522=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5523=== CONT TestGenerateFingerprint/basic_with_references5524=== RUN TestExternalSignerErrors/wrong_signature_length5525=== CONT TestGenerateFingerprint/invalid_reference_prefix5526=== CONT TestGenerateFingerprint/invalid_nar_hash_length5527=== PAUSE TestExternalSignerErrors/wrong_signature_length5528=== RUN TestExternalSignerErrors/null5529=== PAUSE TestExternalSignerErrors/null5530=== RUN TestExternalSignerErrors/program_error5531=== PAUSE TestExternalSignerErrors/program_error5532=== RUN TestExternalSignerErrors/invalid_JSON5533=== PAUSE TestExternalSignerErrors/invalid_JSON5534=== RUN TestExternalSignerErrors/missing_signature_name5535=== PAUSE TestExternalSignerErrors/missing_signature_name5536=== CONT TestExternalSignerErrors/exit_at_startup5537=== CONT TestExternalSignerErrors/wrong_signatures_shape5538=== CONT TestExternalSignerErrors/too_few_signatures5539=== CONT TestGenerateFingerprint/invalid_store_path_prefix5540=== CONT TestExternalSignerErrors/program_error5541=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5542--- PASS: TestExternalSignerClose (0.00s)5543=== CONT TestGenerateFingerprint/no_references5544--- PASS: TestExternalSignerWrongPublicKey (0.00s)5545=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5546=== CONT TestExternalSignerErrors/missing_signature_name5547=== CONT TestExternalSignerErrors/invalid_JSON5548=== CONT TestExternalSignerErrors/wrong_shape5549=== CONT TestExternalSignerErrors/wrong_signature_name5550=== CONT TestExternalSignerErrors/missing_signatures5551=== CONT TestExternalSignerErrors/invalid_signature_base645552=== CONT TestExternalSignerErrors/null_signatures5553=== CONT TestExternalSignerErrors/wrong_id5554=== CONT TestExternalSignerErrors/exit_on_request5555=== CONT TestExternalSignerErrors/wrong_signature_length5556=== CONT TestExternalSignerErrors/unrequested_key5557=== CONT TestExternalSignerErrors/null5558--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.01s)5559 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5560 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5561 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5562 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5563 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5564--- PASS: TestExternalSigner (0.01s)5565--- PASS: TestGenerateFingerprint (0.00s)5566 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5567 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5568 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5569 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5570 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5571 --- PASS: TestGenerateFingerprint/no_references (0.00s)5572 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5573--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5574--- PASS: TestExternalSignerErrors (0.00s)5575 --- PASS: TestExternalSignerErrors/exit_at_startup (0.00s)5576 --- PASS: TestExternalSignerErrors/missing_signature_name (0.00s)5577 --- PASS: TestExternalSignerErrors/program_error (0.00s)5578 --- PASS: TestExternalSignerErrors/invalid_JSON (0.00s)5579 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5580 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5581 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.00s)5582 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5583 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.00s)5584 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5585 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5586 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5587 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5588 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5589 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5590 --- PASS: TestExternalSignerErrors/null (0.01s)5591--- PASS: TestExternalSignerCancellation (0.02s)5592PASS5593Running OpenBao signer tests...5594=== RUN TestSign5595=== PAUSE TestSign5596=== RUN TestSignErrors5597=== PAUSE TestSignErrors5598=== RUN TestServe5599=== PAUSE TestServe5600=== RUN TestServeRejectsGarbage5601=== PAUSE TestServeRejectsGarbage5602=== CONT TestSign5603=== CONT TestServe5604=== CONT TestSignErrors5605=== CONT TestServeRejectsGarbage5606=== RUN TestSignErrors/http_error5607=== PAUSE TestSignErrors/http_error5608--- PASS: TestServeRejectsGarbage (0.00s)5609=== RUN TestSignErrors/invalid_json5610=== PAUSE TestSignErrors/invalid_json5611=== RUN TestSignErrors/no_prefix5612=== PAUSE TestSignErrors/no_prefix5613=== RUN TestSignErrors/invalid_base645614=== PAUSE TestSignErrors/invalid_base645615=== CONT TestSignErrors/http_error5616=== CONT TestSignErrors/invalid_json5617=== CONT TestSignErrors/no_prefix5618=== CONT TestSignErrors/invalid_base645619--- PASS: TestServe (0.00s)5620--- PASS: TestSign (0.00s)5621--- PASS: TestSignErrors (0.00s)5622 --- PASS: TestSignErrors/invalid_json (0.00s)5623 --- PASS: TestSignErrors/http_error (0.00s)5624 --- PASS: TestSignErrors/invalid_base64 (0.00s)5625 --- PASS: TestSignErrors/no_prefix (0.00s)5626PASS

succeeded niks3-go-unit-tests checks.x86_64-linux.go-unit-tests · build #498 · raw

succeeded `niks3-go-unit-tests` `checks.x86_64-linux.go-unit-tests` · build #498 · raw