niks3-go-unit-tests log

succeeded `niks3-go-unit-tests` `checks.aarch64-linux.go-unit-tests` · build #491 · raw

1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (1.08s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestDoServerRequestAttachesToken114=== CONT TestRateLimiterFeedback115=== CONT TestSetClientTLSDoesNotMutateDefaultTransport116=== RUN TestRateLimiterFeedback/429_enables_limiter117=== PAUSE TestRateLimiterFeedback/429_enables_limiter118=== RUN TestRateLimiterFeedback/503_enables_limiter119=== PAUSE TestRateLimiterFeedback/503_enables_limiter120=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter121=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter122=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter123=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter124=== CONT TestRateLimiterFeedback/429_enables_limiter125=== CONT TestShellSplitErrors126=== CONT TestStreamPushReportsEveryPath127=== CONT TestStreamPushStopsWhenCanceled128=== CONT TestStreamPushRequestLine129=== CONT TestStreamPushGivesUpOnDeadServer130=== CONT TestStreamPushIsolatesFailures131=== CONT TestStreamPushBatchesUnderLoad132=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false133=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false134=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true135=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true136=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false137=== CONT TestClientSignaturesByStorePath138=== CONT TestStreamPushReportsSkippedPaths139--- PASS: TestShellSplitErrors (0.00s)140--- PASS: TestStreamPushStopsWhenCanceled (0.00s)141--- PASS: TestClientSignaturesByStorePath (0.00s)142--- PASS: TestStreamPushReportsEveryPath (0.00s)1432026/11/11 23:38:51 ERROR Upload failed error=boom count=1144=== CONT TestPathInfoCACompatibility145=== RUN TestPathInfoCACompatibility/null_ca_field146=== PAUSE TestPathInfoCACompatibility/null_ca_field147=== RUN TestPathInfoCACompatibility/old_string_format_-_text148=== CONT TestDumpPathMatchesNix149=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text150=== CONT TestDoWithRetry_FinalResponseBodyReadable151=== CONT TestRetryTimeoutOutlastsMaxRetries1522026/11/11 23:38:53 ERROR Upload failed error="connection refused" count=201532026/11/11 23:38:53 ERROR Upload failed error="bad path" count=3154=== CONT TestDoWithRetry_BodyReplayedViaGetBody155=== CONT TestResolveStorePath1562026/11/11 23:38:53 ERROR Server seems unavailable, giving up on batch untried=17157=== CONT TestRegisterUploadedObject_OutlivesPush158=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer159=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess160=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter161=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter162=== CONT TestRateLimiterFeedback/503_enables_limiter163=== CONT TestStreamPushReportsSignatures164=== CONT TestSetClientTLS1652026/11/11 23:38:56 WARN Rate limiter enabled after throttle name=server-test rate=5166=== CONT TestPartSizeForNAR167=== RUN TestPartSizeForNAR/zero_stays_at_minimum1682026/11/11 23:38:56 ERROR Upload failed error=boom count=1169=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum170=== RUN TestPartSizeForNAR/small_stays_at_minimum171=== PAUSE TestPartSizeForNAR/small_stays_at_minimum172=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum173=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum174=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts175=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts176=== RUN TestPartSizeForNAR/1_TiB177=== PAUSE TestPartSizeForNAR/1_TiB178=== RUN TestPartSizeForNAR/5_TiB_S3_max_object179=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object180=== RUN TestPartSizeForNAR/capped_at_5_GiB181=== PAUSE TestPartSizeForNAR/capped_at_5_GiB182=== CONT TestShellSplit1832026/11/11 23:38:56 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:45765/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:45765/api/uploads/complete1842026/11/11 23:38:56 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"1852026/11/11 23:38:56 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:45765/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:45765/api/uploads/complete1862026/11/11 23:38:56 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"1872026/11/11 23:38:57 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:45765/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:45765/api/uploads/complete1882026/11/11 23:38:57 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"189=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica190=== CONT TestUploadMultipart_ReturnsAfterDroppedBody191=== CONT TestUploadMultipart_PartsInParallel192=== CONT TestScriptTokenEmptyCommand193=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive194=== CONT TestFilterOversizedClosures195=== CONT TestCaseHackSuffix196=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun197--- PASS: TestStreamPushReportsSkippedPaths (1.78s)198=== RUN TestSetClientTLS/rejects_connection_without_client_cert199=== CONT TestRegisterUploadedObjectReusesConnections200--- PASS: TestStreamPushIsolatesFailures (1.78s)201--- PASS: TestStreamPushGivesUpOnDeadServer (4.68s)202--- PASS: TestResolveStorePath (2.90s)203--- PASS: TestStreamPushReportsSignatures (0.00s)204--- PASS: TestShellSplit (1.10s)205--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (1.11s)206--- PASS: TestScriptTokenEmptyCommand (0.00s)207--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (5.79s)208=== CONT TestScriptTokenBadJSON2092026/11/11 23:38:57 WARN Rate limiter enabled after throttle name=server-test rate=5210=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive2112026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:41753212=== RUN TestPathInfoCACompatibility/new_structured_format_-_text2132026/11/11 23:38:57 WARN Rate limiter enabled after throttle name=server-test rate=5214=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text2152026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:45979216=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method217=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method218=== CONT TestScriptTokenEmptyToken219=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert220=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA221=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA222=== RUN TestSetClientTLS/preserves_debug_logging_transport223=== PAUSE TestSetClientTLS/preserves_debug_logging_transport2242026/11/11 23:38:57 WARN Rate limiter backed off name=server-test rate=5225=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry226=== CONT TestScriptTokenCachesUntilRefresh227=== CONT TestFileTokenEmpty2282026/11/11 23:38:57 WARN Rate limiter enabled after throttle name=server-test rate=52292026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:439932302026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:46821231=== CONT TestFileTokenMissing232=== CONT TestScriptTokenScriptFails233=== CONT TestScriptTokenNoExpiryRerunsEveryCall234--- PASS: TestFileTokenEmpty (0.00s)235=== CONT TestStaticToken236=== RUN TestFilterOversizedClosures/no_limit_keeps_everything237=== CONT TestSetClientTLSErrors238--- PASS: TestScriptTokenBadJSON (0.01s)239--- PASS: TestDoServerRequestAttachesToken (5.83s)2402026/11/11 23:38:57 INFO Garbage collection started241--- PASS: TestRegisterUploadedObject_OutlivesPush (1.11s)242--- PASS: TestStaticToken (0.00s)243=== CONT TestConvertHashToNix32244=== RUN TestConvertHashToNix32/SRI_format_to_Nix32245=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix322462026/11/11 23:38:57 WARN Rate limiter backed off name=server-test rate=5247=== RUN TestConvertHashToNix32/already_Nix32_format248=== PAUSE TestConvertHashToNix32/already_Nix32_format249=== RUN TestConvertHashToNix32/invalid_format250=== PAUSE TestConvertHashToNix32/invalid_format251=== CONT TestParsePathInfoJSONMultiplePaths252=== CONT TestParsePathInfoJSON253=== RUN TestParsePathInfoJSON/Nix_format254=== PAUSE TestParsePathInfoJSON/Nix_format255--- PASS: TestFileTokenMissing (0.00s)256=== CONT TestPathInfoHashCompatibility257--- PASS: TestRateLimiterFeedback (0.00s)258 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (1.10s)259 --- PASS: TestRateLimiterFeedback/503_enables_limiter (1.10s)260 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (1.14s)261 --- PASS: TestRateLimiterFeedback/429_enables_limiter (5.83s)262=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)2632026/11/11 23:38:57 WARN Rate limiter backed off name=server-test rate=5264=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)265=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon266=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon267=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI268=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI269=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512270=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512271=== CONT TestGetStorePathHash272=== RUN TestGetStorePathHash/valid_store_path273=== PAUSE TestGetStorePathHash/valid_store_path274=== RUN TestGetStorePathHash/basename_without_hyphen_should_error275=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error276=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error277=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error278=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error279=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error280=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths281=== RUN TestParsePathInfoJSON/Lix_format282=== PAUSE TestParsePathInfoJSON/Lix_format283=== RUN TestParsePathInfoJSON/empty_input284=== PAUSE TestParsePathInfoJSON/empty_input285=== RUN TestParsePathInfoJSON/whitespace_only286=== PAUSE TestParsePathInfoJSON/whitespace_only287=== RUN TestParsePathInfoJSON/invalid_JSON288--- PASS: TestDoWithRetry_FinalResponseBodyReadable (5.83s)289=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths290=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths291=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths292=== PAUSE TestParsePathInfoJSON/invalid_JSON293=== CONT TestEncodeNixBase32294=== CONT TestDumpPathSingleFile295=== RUN TestSetClientTLSErrors/missing_cert_file296=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything297=== CONT TestDumpPathWriterError298=== CONT TestEncodeNixBase32WithRealHash2992026/11/11 23:38:57 WARN Rate limiter enabled after throttle name=server-test rate=5300--- PASS: TestEncodeNixBase32WithRealHash (0.00s)3012026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:35515302=== CONT TestPartSizeForNAR/zero_stays_at_minimum303=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true304=== CONT TestPartSizeForNAR/5_TiB_S3_max_object305=== CONT TestPartSizeForNAR/capped_at_5_GiB306=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts307=== CONT TestPartSizeForNAR/1_TiB308=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum3092026/11/11 23:38:57 WARN Rate limiter backed off name=server-test rate=5310=== CONT TestPartSizeForNAR/small_stays_at_minimum3112026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:35515312=== CONT TestSetClientTLS/rejects_connection_without_client_cert313=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped314--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.04s)315--- PASS: TestPartSizeForNAR (0.00s)316 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)317 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)318 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)319 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)320 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)321 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)322 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)323--- PASS: TestStreamPushBatchesUnderLoad (0.00s)324 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (1.78s)325 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)326=== CONT TestPathInfoCACompatibility/new_structured_format_-_text327=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive328=== CONT TestPathInfoCACompatibility/old_string_format_-_text329=== CONT TestSetClientTLS/preserves_debug_logging_transport330=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method331--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (4.06s)332=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped333=== RUN TestFilterOversizedClosures/all_closures_skipped334=== PAUSE TestFilterOversizedClosures/all_closures_skipped335=== CONT TestPathInfoCACompatibility/null_ca_field336--- PASS: TestPathInfoCACompatibility (5.79s)337 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)338 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)339 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)340 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)341 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)342=== CONT TestConvertHashToNix32/SRI_format_to_Nix32343=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA344=== CONT TestConvertHashToNix32/already_Nix32_format345=== CONT TestConvertHashToNix32/invalid_format346--- PASS: TestConvertHashToNix32 (0.00s)347 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)348 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)349 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)350=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)351=== CONT TestGetStorePathHash/valid_store_path352=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512353=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI354=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error355=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error356=== CONT TestGetStorePathHash/basename_without_hyphen_should_error357--- PASS: TestGetStorePathHash (0.00s)358 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)359 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)360 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)361 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)362=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon363--- PASS: TestPathInfoHashCompatibility (0.00s)364 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)365 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)366 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)367 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)368=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths369=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths370=== PAUSE TestSetClientTLSErrors/missing_cert_file371=== RUN TestSetClientTLSErrors/missing_key_file372=== PAUSE TestSetClientTLSErrors/missing_key_file373=== RUN TestSetClientTLSErrors/missing_ca_file374=== PAUSE TestSetClientTLSErrors/missing_ca_file375=== RUN TestSetClientTLSErrors/invalid_ca_file376=== PAUSE TestSetClientTLSErrors/invalid_ca_file377=== CONT TestParsePathInfoJSON/Nix_format378--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)379 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)380 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)381=== CONT TestParsePathInfoJSON/whitespace_only382=== CONT TestParsePathInfoJSON/invalid_JSON383=== CONT TestFilterOversizedClosures/no_limit_keeps_everything384=== CONT TestFileTokenReadsAndCaches3852026/11/11 23:38:57 INFO Garbage collection started386=== CONT TestParsePathInfoJSON/empty_input3872026/11/11 23:38:57 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0388=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped389=== RUN TestEncodeNixBase32/test_string_hash3902026/11/11 23:38:57 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000391=== PAUSE TestEncodeNixBase32/test_string_hash3922026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:43993393=== RUN TestEncodeNixBase32/empty_input394=== PAUSE TestEncodeNixBase32/empty_input395=== CONT TestParsePathInfoJSON/Lix_format396=== CONT TestFilterOversizedClosures/all_closures_skipped3972026/11/11 23:38:57 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50398=== CONT TestSetClientTLSErrors/missing_cert_file399--- PASS: TestParsePathInfoJSON (0.00s)400 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)401 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)402 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)403 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)404 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)405--- PASS: TestFileTokenReadsAndCaches (0.00s)406=== CONT TestSetClientTLSErrors/missing_ca_file407=== CONT TestSetClientTLSErrors/invalid_ca_file408=== CONT TestSetClientTLSErrors/missing_key_file409=== CONT TestEncodeNixBase32/test_string_hash410=== CONT TestEncodeNixBase32/empty_input411--- PASS: TestEncodeNixBase32 (0.00s)412 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)413 --- PASS: TestEncodeNixBase32/empty_input (0.00s)414--- PASS: TestFilterOversizedClosures (0.04s)415 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)416 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)417 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)4182026/11/11 23:38:57 WARN Rate limiter enabled after throttle name=s3-test rate=5419--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.05s)4202026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36939421--- PASS: TestScriptTokenEmptyToken (0.05s)4222026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:439934232026/11/11 23:38:57 WARN Garbage collection finished on another replica. Its statistics are not available here424--- PASS: TestSetClientTLSErrors (0.01s)425 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)426 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)427 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)428 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)429--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.05s)430--- PASS: TestScriptTokenScriptFails (0.01s)431--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.05s)4322026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:439934332026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:439934342026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:439934352026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:439934362026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:43993437--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.02s)438--- PASS: TestScriptTokenCachesUntilRefresh (0.06s)4392026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:43993440--- PASS: TestStreamPushRequestLine (5.85s)4412026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:439934422026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:439934432026/11/11 23:38:57 http: TLS handshake error from 127.0.0.1:46510: remote error: tls: bad certificate444--- PASS: TestSetClientTLS (1.10s)445 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.01s)446 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)447 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4482026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:439934492026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:439934502026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:439934512026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:43993452--- PASS: TestRegisterUploadedObjectReusesConnections (0.07s)4532026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:439934542026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:439934552026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:439934562026/11/11 23:38:57 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:43993457--- PASS: TestRetryTimeoutOutlastsMaxRetries (4.09s)458--- PASS: TestDumpPathWriterError (0.04s)459--- PASS: TestDumpPathSingleFile (0.55s)460--- PASS: TestCaseHackSuffix (0.60s)461--- PASS: TestDumpPathMatchesNix (6.40s)462--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.95s)463--- PASS: TestUploadMultipart_PartsInParallel (1.14s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "nixbld".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /build/postgres1314532703/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /build/postgres1314532703/data -l logfile start492493/build/postgres1314532703:5432 - no response494/build/postgres1314532703:5432 - no response4952026-11-11 23:39:14.272 UTC [131] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit4962026-11-11 23:39:14.273 UTC [131] LOG: listening on Unix socket "/build/postgres1314532703/.s.PGSQL.5432"4972026-11-11 23:39:14.280 UTC [139] LOG: database system was shut down at 2026-11-11 23:39:13 UTC4982026-11-11 23:39:14.285 UTC [131] LOG: database system is ready to accept connections499/build/postgres1314532703:5432 - accepting connections500{"timestamp":"2026-11-11T23:39:21.333797869Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"3ee35379-b219-4f3d-87a9-edae8d72fbd7","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":12,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(200)"}501{"timestamp":"2026-11-11T23:39:21.483045972Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"8c5714ff-0037-4067-bd5f-c153af80c447","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(200)"}502{"timestamp":"2026-11-11T23:39:21.585701511Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c1c37e55-0f75-4da9-a70f-1e473f06ea57","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(201)"}503=== RUN TestAbortSucceeded504=== PAUSE TestAbortSucceeded505=== RUN TestService_AuthMiddleware506=== PAUSE TestService_AuthMiddleware507=== RUN TestService_AuthMiddleware_MTLSProxyHeader508=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader509=== RUN TestService_AuthMiddleware_MTLSBoundSubjects510=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects511=== RUN TestService_ReadAuthMiddleware512=== PAUSE TestService_ReadAuthMiddleware513=== RUN TestService_AuthMiddleware_OIDC514=== PAUSE TestService_AuthMiddleware_OIDC515=== RUN TestService_RequireScope_OIDC516=== PAUSE TestService_RequireScope_OIDC517=== RUN TestService_ReadScope_PublicByDefault518=== PAUSE TestService_ReadScope_PublicByDefault519=== RUN TestCacheConfigHandler520=== PAUSE TestCacheConfigHandler521=== RUN TestCacheStatsHandler522=== PAUSE TestCacheStatsHandler523=== RUN TestClientCADerivations524=== PAUSE TestClientCADerivations525=== RUN TestClientErrorHandling526=== PAUSE TestClientErrorHandling527=== RUN TestClientIntegration528=== PAUSE TestClientIntegration529=== RUN TestClientMultipleUploads530=== PAUSE TestClientMultipleUploads531=== RUN TestClientWithDependencies532=== PAUSE TestClientWithDependencies533=== RUN TestClientSharedPathCommittedMidPush534=== PAUSE TestClientSharedPathCommittedMidPush535=== RUN TestPinProtectsFromGC536=== PAUSE TestPinProtectsFromGC537=== RUN TestClientPushesUseOnePush538=== PAUSE TestClientPushesUseOnePush539=== RUN TestClientFallsBackToClosures540=== PAUSE TestClientFallsBackToClosures541=== RUN TestDeleteClosuresRacingPins542=== PAUSE TestDeleteClosuresRacingPins543=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock544=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock545=== RUN TestResolveDBConnectionString546=== PAUSE TestResolveDBConnectionString547=== RUN TestConnectWaitsForAPeerMigration548=== PAUSE TestConnectWaitsForAPeerMigration549=== RUN TestLeadElectsOneAndHandsOver550=== PAUSE TestLeadElectsOneAndHandsOver551=== RUN TestLeadIncumbentWinsAfterRestart5522026/11/11 23:39:23 INFO lead: acquired remote=192.0.2.1:1234 generation=15532026/11/11 23:39:23 INFO lead: released remote=192.0.2.1:1234 generation=15542026/11/11 23:39:24 INFO lead: acquired remote=192.0.2.1:1234 generation=25552026/11/11 23:39:25 INFO lead: released remote=192.0.2.1:1234 generation=2556--- PASS: TestLeadIncumbentWinsAfterRestart (3.42s)557=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent558=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent559=== RUN TestLeadIncumbentAnnouncesAtOnce560=== PAUSE TestLeadIncumbentAnnouncesAtOnce561=== RUN TestLeadGenerationGrowsWithEveryGrant562=== PAUSE TestLeadGenerationGrowsWithEveryGrant563=== RUN TestLeadEndsOnShutdown564=== PAUSE TestLeadEndsOnShutdown565=== RUN TestLeadEndsWhenItsConnectionHangs5662026/11/11 23:39:26 INFO lead: acquired remote=192.0.2.1:1234 generation=15672026-11-11 23:39:28.131 UTC [386] FATAL: terminating connection due to administrator command5682026/11/11 23:39:28 INFO lead: acquired remote=192.0.2.1:1234 generation=25692026/11/11 23:39:29 WARN lead: lock connection lost error="timeout: context deadline exceeded"5702026/11/11 23:39:29 INFO lead: released remote=192.0.2.1:1234 generation=15712026/11/11 23:39:29 INFO lead: released remote=192.0.2.1:1234 generation=2572--- PASS: TestLeadEndsWhenItsConnectionHangs (4.60s)573=== RUN TestGCAdvisoryLockBlocksConcurrentRun574=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun575=== RUN TestGCBugBareHashReferences576=== PAUSE TestGCBugBareHashReferences577=== RUN TestGCMetrics578=== PAUSE TestGCMetrics579=== RUN TestPushDedupSurvivesConcurrentGC580=== PAUSE TestPushDedupSurvivesConcurrentGC581=== RUN TestDeduplicatedObjectsRecordedAsPending582=== PAUSE TestDeduplicatedObjectsRecordedAsPending583=== RUN TestGCSweepSkipsPendingObjects584=== PAUSE TestGCSweepSkipsPendingObjects585=== RUN TestTombstonedObjectOfferedWithoutWaiting586=== PAUSE TestTombstonedObjectOfferedWithoutWaiting587=== RUN TestGCSweepDeliversEachKeyOnce588=== PAUSE TestGCSweepDeliversEachKeyOnce589=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection590=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection591=== RUN TestForceGCDuringPushOffersSweptObject592=== PAUSE TestForceGCDuringPushOffersSweptObject593=== RUN TestFailedS3DeleteKeepsTombstone594=== PAUSE TestFailedS3DeleteKeepsTombstone595=== RUN TestSweepSkipsRowsLockedByPush596=== PAUSE TestSweepSkipsRowsLockedByPush597=== RUN TestRegisterCompletedObjectNeedsPendingRow598=== PAUSE TestRegisterCompletedObjectNeedsPendingRow599=== RUN TestGetExistingObjectsLocksRows600=== PAUSE TestGetExistingObjectsLocksRows601=== RUN TestSweepRechecksPendingAfterLocking602=== PAUSE TestSweepRechecksPendingAfterLocking603=== RUN TestPushWaitsForSweepHoldingRow604=== PAUSE TestPushWaitsForSweepHoldingRow605=== RUN TestCommitRacingPendingCleanupKeepsObjects606=== PAUSE TestCommitRacingPendingCleanupKeepsObjects607=== RUN TestGCEndsOnShutdown608=== PAUSE TestGCEndsOnShutdown609=== RUN TestGCTaskStore_StartNew610=== PAUSE TestGCTaskStore_StartNew611=== RUN TestGCTaskStore_DeduplicateSameParams612=== PAUSE TestGCTaskStore_DeduplicateSameParams613=== RUN TestGCTaskStore_ConflictDifferentParams614=== PAUSE TestGCTaskStore_ConflictDifferentParams615=== RUN TestGCTaskStore_GetEmpty616=== PAUSE TestGCTaskStore_GetEmpty617=== RUN TestGCTaskStore_GetReturnsLatest618=== PAUSE TestGCTaskStore_GetReturnsLatest619=== RUN TestGCTaskStore_CompletedAllowsNewTask620=== PAUSE TestGCTaskStore_CompletedAllowsNewTask621=== RUN TestGCTaskStore_PhaseUpdates622=== PAUSE TestGCTaskStore_PhaseUpdates623=== RUN TestGCTaskStore_Fail624=== PAUSE TestGCTaskStore_Fail625=== RUN TestGracefulShutdownDrainsInflight626=== PAUSE TestGracefulShutdownDrainsInflight627=== RUN TestService_healthCheckHandler628=== PAUSE TestService_healthCheckHandler629=== RUN TestService_readinessHandler630=== PAUSE TestService_readinessHandler631=== RUN TestGenerateLandingPage632=== PAUSE TestGenerateLandingPage633=== RUN TestEnsureAbortUploadsRule634=== PAUSE TestEnsureAbortUploadsRule635=== RUN TestCacheConfigHandlerMaxNarSize636=== PAUSE TestCacheConfigHandlerMaxNarSize637=== RUN TestCreatePendingClosureRejectsOversizedNAR638=== PAUSE TestCreatePendingClosureRejectsOversizedNAR639=== RUN TestNARDeduplicationMetadataUploadBug640=== PAUSE TestNARDeduplicationMetadataUploadBug641=== RUN TestMetricsInventory642=== PAUSE TestMetricsInventory643=== RUN TestMetricsCommit644=== PAUSE TestMetricsCommit645=== RUN TestService_NativeMTLS646=== PAUSE TestService_NativeMTLS647=== RUN TestServerTLSConfig648=== PAUSE TestServerTLSConfig649=== RUN TestMultipartCleanup650=== PAUSE TestMultipartCleanup651=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded652=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded653=== RUN TestPendingCleanupUsesOneCutoff654=== PAUSE TestPendingCleanupUsesOneCutoff655=== RUN TestObjectStatsTrigger656=== PAUSE TestObjectStatsTrigger657=== RUN TestObjectStatsUpdatedOncePerCommit658=== PAUSE TestObjectStatsUpdatedOncePerCommit659=== RUN TestOrphanedObjectsGC660=== PAUSE TestOrphanedObjectsGC661=== RUN TestOrphanedObjectsGCStressTest662=== PAUSE TestOrphanedObjectsGCStressTest663=== RUN TestCreatePin_ReservedPins664=== PAUSE TestCreatePin_ReservedPins665=== RUN TestPresentReportsOnlyClosureRoots666=== PAUSE TestPresentReportsOnlyClosureRoots667=== RUN TestPresentRefreshesOnlyStaleClosures668=== PAUSE TestPresentRefreshesOnlyStaleClosures669=== RUN TestPresentNotReportedWhileGCDeletesClosure670=== PAUSE TestPresentNotReportedWhileGCDeletesClosure671=== RUN TestParseSingleRange672=== PAUSE TestParseSingleRange673=== RUN TestProxyHeadersOnlyTrustedOnSocket6742026/11/11 23:39:32 INFO Starting HTTP server address=proxy.sock6752026/11/11 23:39:32 INFO Starting HTTP server address=127.0.0.1:334936762026/11/11 23:39:32 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6772026/11/11 23:39:32 INFO Shutdown signal received, draining in-flight requests timeout=10s678--- PASS: TestProxyHeadersOnlyTrustedOnSocket (2.82s)679=== RUN TestIsValidCachePath680=== PAUSE TestIsValidCachePath681=== RUN TestReadProxyNarinfo682=== PAUSE TestReadProxyNarinfo683=== RUN TestReadProxyNarinfoAlreadyDecompressed684=== PAUSE TestReadProxyNarinfoAlreadyDecompressed685=== RUN TestReadProxyContentNegotiation686=== PAUSE TestReadProxyContentNegotiation687=== RUN TestReadProxyNarStreaming688=== PAUSE TestReadProxyNarStreaming689=== RUN TestReadProxy404690=== PAUSE TestReadProxy404691=== RUN TestReadProxyInvalidPath692=== PAUSE TestReadProxyInvalidPath693=== RUN TestReadProxyHead694=== PAUSE TestReadProxyHead695=== RUN TestReadProxyOutlastsServerWriteTimeout696=== PAUSE TestReadProxyOutlastsServerWriteTimeout697=== RUN TestReadProxyConditionalGet698=== PAUSE TestReadProxyConditionalGet699=== RUN TestReadProxyRootRedirectsToIndexHTML700=== PAUSE TestReadProxyRootRedirectsToIndexHTML701=== RUN TestReadProxyDisabled702=== PAUSE TestReadProxyDisabled703=== RUN TestReadRedirectNar704=== PAUSE TestReadRedirectNar705=== RUN TestReadRedirectKeepsNarinfoProxied706=== PAUSE TestReadRedirectKeepsNarinfoProxied707=== RUN TestReadProxyRangeRequest708=== PAUSE TestReadProxyRangeRequest709=== RUN TestReadRedirectUsesPublicS3URL710=== PAUSE TestReadRedirectUsesPublicS3URL711=== RUN TestPush_OverlappingRootsStoreOneRowPerKey712=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey713=== RUN TestPush_CompleteCommitsEveryRoot714=== PAUSE TestPush_CompleteCommitsEveryRoot715=== RUN TestPush_RepeatedRootCommitsOnce716=== PAUSE TestPush_RepeatedRootCommitsOnce717=== RUN TestPush_CompleteWaitsForCleanupOfThePush718=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush719=== RUN TestPush_RootsAreLockedInByteOrder720=== PAUSE TestPush_RootsAreLockedInByteOrder721=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit722=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit723=== RUN TestPush_RejectsBadRequests724=== PAUSE TestPush_RejectsBadRequests725=== RUN TestPush_SignsNarinfosOfItsPendingObjects726=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects727=== RUN TestSharedNarBothUploadsComplete728=== PAUSE TestSharedNarBothUploadsComplete729=== RUN TestCompleteMultipartUpload_ErrorButObjectExists730=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists731=== RUN TestCompletedNarNotReofferedAcrossClosures732=== PAUSE TestCompletedNarNotReofferedAcrossClosures733=== RUN TestPresignedUploadRegisteredBeforeCommit734=== PAUSE TestPresignedUploadRegisteredBeforeCommit735=== RUN TestService_Rustfstest736=== PAUSE TestService_Rustfstest737=== RUN TestParseSize738=== PAUSE TestParseSize739=== RUN TestSkippedUploadsHandler740=== PAUSE TestSkippedUploadsHandler741=== RUN TestSystemdListenerNotActivated742--- PASS: TestSystemdListenerNotActivated (0.00s)743=== RUN TestWatchdogBeatsWhenHealthy744--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)745=== RUN TestWatchdogSkipsWhenUnhealthy7462026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7472026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7482026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7492026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7532026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7542026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7552026/11/11 23:39:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"756--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)757=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle758=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle759=== RUN TestProxyWriteTimeout760=== PAUSE TestProxyWriteTimeout761=== RUN TestPendingClosureWriteTimeout762=== PAUSE TestPendingClosureWriteTimeout763=== RUN TestIsValidUploadKey764=== PAUSE TestIsValidUploadKey765=== RUN TestUploadHandlersRejectInvalidKeys766=== PAUSE TestUploadHandlersRejectInvalidKeys767=== RUN TestUploadHandlersRejectOversizedBody768=== PAUSE TestUploadHandlersRejectOversizedBody769=== RUN TestService_cleanupPendingClosuresHandler770=== PAUSE TestService_cleanupPendingClosuresHandler771=== RUN TestService_createPendingClosureHandler772=== PAUSE TestService_createPendingClosureHandler773=== RUN TestService_verifyS3Integrity774=== PAUSE TestService_verifyS3Integrity775=== RUN TestService_verifyS3ReuploadsStaleNarinfo776=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo777=== RUN TestCompleteMultipartUnregistered778=== PAUSE TestCompleteMultipartUnregistered779=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT780=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT781=== CONT TestProxyWriteTimeout782=== CONT TestPush_CompleteWaitsForCleanupOfThePush783=== CONT TestCompleteMultipartUnregistered784=== CONT TestService_createPendingClosureHandler785=== CONT TestGCTaskStore_GetEmpty786--- PASS: TestGCTaskStore_GetEmpty (0.00s)787=== CONT TestReadRedirectKeepsNarinfoProxied788=== CONT TestSkippedUploadsHandler789=== CONT TestParseSize790=== CONT TestService_Rustfstest791=== CONT TestPresignedUploadRegisteredBeforeCommit792=== CONT TestCompletedNarNotReofferedAcrossClosures7932026/11/11 23:39:32 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000794=== CONT TestCompleteMultipartUpload_ErrorButObjectExists795=== CONT TestSharedNarBothUploadsComplete796=== CONT TestReadProxyDisabled797=== CONT TestPush_SignsNarinfosOfItsPendingObjects798=== CONT TestPush_RejectsBadRequests799=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit800=== CONT TestPush_RootsAreLockedInByteOrder801=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT802=== CONT TestPush_RepeatedRootCommitsOnce803=== CONT TestPush_CompleteCommitsEveryRoot804=== CONT TestPush_OverlappingRootsStoreOneRowPerKey805=== CONT TestReadRedirectUsesPublicS3URL806=== CONT TestReadProxyRangeRequest807=== CONT TestAbortSucceeded808=== CONT TestReadProxyRootRedirectsToIndexHTML809=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle810=== RUN TestProxyWriteTimeout/narinfo811=== PAUSE TestProxyWriteTimeout/narinfo812=== RUN TestProxyWriteTimeout/1_GiB_nar813=== PAUSE TestProxyWriteTimeout/1_GiB_nar814=== RUN TestProxyWriteTimeout/10_GiB_nar815=== PAUSE TestProxyWriteTimeout/10_GiB_nar816=== RUN TestProxyWriteTimeout/unknown_size817=== PAUSE TestProxyWriteTimeout/unknown_size818--- PASS: TestParseSize (0.00s)819--- PASS: TestSkippedUploadsHandler (0.01s)820--- PASS: TestAbortSucceeded (0.00s)821=== CONT TestReadProxyConditionalGet822=== CONT TestReadRedirectNar8232026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures8242026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures8252026/11/11 23:39:34 INFO Received push request method=POST path=/api/pushes8262026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures8272026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures8282026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures8292026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures8302026/11/11 23:39:34 INFO Received complete push request method=POST path=/api/pushes/1/complete831--- PASS: TestReadProxyDisabled (1.38s)832=== CONT TestReadProxyOutlastsServerWriteTimeout8332026/11/11 23:39:34 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst8342026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures835--- PASS: TestPresignedUploadRegisteredBeforeCommit (1.40s)836=== CONT TestReadProxyHead8372026/11/11 23:39:34 INFO Received push request method=POST path=/api/pushes8382026/11/11 23:39:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete8392026/11/11 23:39:34 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst840--- PASS: TestCompleteMultipartUnregistered (1.45s)841=== CONT TestReadProxyInvalidPath8422026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures843=== RUN TestPush_RejectsBadRequests/no_roots844=== PAUSE TestPush_RejectsBadRequests/no_roots845=== RUN TestPush_RejectsBadRequests/no_objects846=== PAUSE TestPush_RejectsBadRequests/no_objects847=== RUN TestPush_RejectsBadRequests/bad_root848=== PAUSE TestPush_RejectsBadRequests/bad_root849=== RUN TestPush_RejectsBadRequests/root_not_in_objects850=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects851=== CONT TestReadProxy4048522026/11/11 23:39:34 INFO Received push request method=POST path=/api/pushes8532026/11/11 23:39:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8542026/11/11 23:39:34 INFO Signed narinfos id=1 count=18552026/11/11 23:39:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8562026/11/11 23:39:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8572026/11/11 23:39:34 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"8582026/11/11 23:39:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8592026/11/11 23:39:34 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"8602026/11/11 23:39:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8612026/11/11 23:39:34 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=18622026/11/11 23:39:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign8632026/11/11 23:39:34 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo864=== CONT TestReadProxyNarStreaming865--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (1.58s)866--- PASS: TestService_Rustfstest (1.64s)867=== CONT TestReadProxyContentNegotiation8682026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures869--- PASS: TestReadRedirectKeepsNarinfoProxied (1.69s)870=== CONT TestReadProxyNarinfoAlreadyDecompressed8712026-11-11 23:39:34.501 UTC [496] ERROR: Push does not exist: id=18722026-11-11 23:39:34.501 UTC [496] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE8732026-11-11 23:39:34.501 UTC [496] STATEMENT: SELECT commit_push($1)874--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (1.73s)875=== CONT TestReadProxyNarinfo8762026/11/11 23:39:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete8772026/11/11 23:39:34 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmJiYzA5YzA4LTMwNjktNDkxNC05ZWFkLTViMzk3MDk3ZGQxMXgxNzk0NDQwMzc0NDgyMDg3NTU58782026/11/11 23:39:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete8792026/11/11 23:39:34 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmJiYzA5YzA4LTMwNjktNDkxNC05ZWFkLTViMzk3MDk3ZGQxMXgxNzk0NDQwMzc0NDgyMDg3NTU5 parts=18802026/11/11 23:39:34 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LjRhZDBiMmY0LTZkYjAtNDNmMS1hNjRiLWY4MWYzMGI3NDMzOXgxNzk0NDQwMzc0MTQ5MDU5MTgy parts=108812026/11/11 23:39:34 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete8822026/11/11 23:39:34 INFO Completed upload id=1 objects=2 duration=4.750645ms8832026/11/11 23:39:34 INFO Received get closure request method=GET path=/api/closures/00000000000000000000000000000000884--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (1.82s)885=== CONT TestIsValidCachePath886=== RUN TestIsValidCachePath/narinfo887=== PAUSE TestIsValidCachePath/narinfo888=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars889=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars890=== RUN TestIsValidCachePath/nar_zst891=== PAUSE TestIsValidCachePath/nar_zst892=== RUN TestIsValidCachePath/nar_xz893=== PAUSE TestIsValidCachePath/nar_xz894=== RUN TestIsValidCachePath/nar_bz2895=== PAUSE TestIsValidCachePath/nar_bz2896=== RUN TestIsValidCachePath/nar_uncompressed897=== PAUSE TestIsValidCachePath/nar_uncompressed898=== RUN TestIsValidCachePath/ls899=== PAUSE TestIsValidCachePath/ls900=== RUN TestIsValidCachePath/log901=== PAUSE TestIsValidCachePath/log902=== RUN TestIsValidCachePath/realisation903=== PAUSE TestIsValidCachePath/realisation904=== RUN TestIsValidCachePath/nix-cache-info905=== PAUSE TestIsValidCachePath/nix-cache-info906=== RUN TestIsValidCachePath/index.html907=== PAUSE TestIsValidCachePath/index.html908=== RUN TestIsValidCachePath/traversal_parent9092026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures910=== PAUSE TestIsValidCachePath/traversal_parent911=== RUN TestIsValidCachePath/traversal_in_middle912=== PAUSE TestIsValidCachePath/traversal_in_middle913=== RUN TestIsValidCachePath/invalid_char_e914=== PAUSE TestIsValidCachePath/invalid_char_e915=== RUN TestIsValidCachePath/invalid_char_u916=== PAUSE TestIsValidCachePath/invalid_char_u917=== RUN TestIsValidCachePath/random_path918=== PAUSE TestIsValidCachePath/random_path919=== RUN TestIsValidCachePath/empty920=== PAUSE TestIsValidCachePath/empty921=== RUN TestIsValidCachePath/leading_slash922=== PAUSE TestIsValidCachePath/leading_slash923=== RUN TestIsValidCachePath/wrong_extension924=== PAUSE TestIsValidCachePath/wrong_extension925=== RUN TestIsValidCachePath/short_hash926=== PAUSE TestIsValidCachePath/short_hash927=== CONT TestParseSingleRange928=== RUN TestParseSingleRange/none929=== PAUSE TestParseSingleRange/none930=== RUN TestParseSingleRange/unknown_unit931=== PAUSE TestParseSingleRange/unknown_unit932=== RUN TestParseSingleRange/multi-range_ignored933=== PAUSE TestParseSingleRange/multi-range_ignored934=== RUN TestParseSingleRange/malformed_no_dash935=== PAUSE TestParseSingleRange/malformed_no_dash936=== RUN TestParseSingleRange/malformed_both_empty937=== PAUSE TestParseSingleRange/malformed_both_empty938=== RUN TestParseSingleRange/malformed_end_before_start939=== PAUSE TestParseSingleRange/malformed_end_before_start940=== RUN TestParseSingleRange/closed941=== PAUSE TestParseSingleRange/closed942=== RUN TestParseSingleRange/open-ended943=== PAUSE TestParseSingleRange/open-ended944=== RUN TestParseSingleRange/end_clamped_to_size945=== PAUSE TestParseSingleRange/end_clamped_to_size946=== RUN TestParseSingleRange/suffix947=== PAUSE TestParseSingleRange/suffix948=== RUN TestParseSingleRange/suffix_exceeds_size949=== PAUSE TestParseSingleRange/suffix_exceeds_size950=== RUN TestParseSingleRange/single_byte951=== PAUSE TestParseSingleRange/single_byte952=== RUN TestParseSingleRange/start_past_EOF953=== PAUSE TestParseSingleRange/start_past_EOF954=== RUN TestParseSingleRange/start_far_past_EOF955=== PAUSE TestParseSingleRange/start_far_past_EOF956=== CONT TestPresentNotReportedWhileGCDeletesClosure957=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale958=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale959=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh960=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh961=== CONT TestPresentRefreshesOnlyStaleClosures9622026/11/11 23:39:34 INFO Starting cleanup of old closures method=DELETE path=/api/closures9632026/11/11 23:39:34 INFO Aborted multipart uploads count=0 kept=09642026/11/11 23:39:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9652026/11/11 23:39:34 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmJkZWYxMDA4LTcxNWQtNDhiMS1iMDQ1LTEzYjVmYzA2YmU3MXgxNzk0NDQwMzc0MTI4MTEzNjAz parts=129662026/11/11 23:39:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9672026/11/11 23:39:34 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmZmZTAwYzJlLWRjZTYtNDM4Zi05YTgyLWU3YWMyMGQwNWE4MHgxNzk0NDQwMzc0MjgxOTUyNjUw parts=129682026/11/11 23:39:34 INFO Received uploads request method=POST path=/api/pending_closures969--- PASS: TestCompletedNarNotReofferedAcrossClosures (2.03s)970=== CONT TestPresentReportsOnlyClosureRoots9712026/11/11 23:39:35 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9722026/11/11 23:39:35 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmQwYzBjOGRiLTRmNDAtNDY3My1iNjk5LWVhMjJlZjdhZWUzYngxNzk0NDQwMzc0MTQxNjEyNTMx parts=12973--- PASS: TestSharedNarBothUploadsComplete (2.41s)974=== CONT TestCreatePin_ReservedPins9752026/11/11 23:39:35 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40703/oidc9762026/11/11 23:39:36 INFO Completed push id=1 objects=4 duration=2.125177753s9772026/11/11 23:39:36 INFO Received push request method=POST path=/api/pushes9782026/11/11 23:39:36 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09792026/11/11 23:39:36 INFO Vacuumed table table=pending_closures9802026/11/11 23:39:36 INFO Vacuumed table table=pending_objects9812026/11/11 23:39:36 INFO Vacuumed table table=multipart_uploads9822026/11/11 23:39:36 INFO Vacuumed table table=closures9832026/11/11 23:39:36 INFO Vacuumed table table=objects9842026/11/11 23:39:36 INFO Received get closure request method=GET path=/api/closures/00000000000000000000000000000000985--- PASS: TestService_createPendingClosureHandler (3.58s)986=== CONT TestOrphanedObjectsGCStressTest9872026/11/11 23:39:36 INFO Aborted multipart uploads count=0 kept=09882026/11/11 23:39:36 WARN Force mode enabled - objects will be deleted immediately without grace period989--- PASS: TestPresentRefreshesOnlyStaleClosures (1.91s)990=== CONT TestOrphanedObjectsGC991--- PASS: TestReadProxyRangeRequest (3.50s)992=== CONT TestUploadHandlersRejectInvalidKeys993=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info994=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info995=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal996=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal997=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key998=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key999=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1000=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1001=== CONT TestObjectStatsUpdatedOncePerCommit10022026/11/11 23:39:36 INFO Received push request method=POST path=/api/pushes10032026/11/11 23:39:36 INFO Received complete push request method=POST path=/api/pushes/1/complete1004--- PASS: TestReadProxy404 (2.28s)1005=== CONT TestObjectStatsTrigger1006--- PASS: TestReadRedirectNar (3.53s)1007=== CONT TestService_cleanupPendingClosuresHandler1008--- PASS: TestReadProxyConditionalGet (3.59s)1009=== CONT TestPendingCleanupUsesOneCutoff10102026/11/11 23:39:36 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10112026/11/11 23:39:36 WARN Refused reserved pin name=worker-x86_64-linux10122026/11/11 23:39:36 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10132026/11/11 23:39:36 INFO Received create pin request method=POST path=/api/pins/my-app10142026/11/11 23:39:36 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1015--- PASS: TestCreatePin_ReservedPins (1.47s)1016=== CONT TestUploadHandlersRejectOversizedBody10172026/11/11 23:39:36 INFO Received uploads request method=POST path=/api/pending_closures1018=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1019=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1020=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1021=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1022=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1023=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1024=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1025--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (3.86s)1026=== CONT TestService_verifyS3ReuploadsStaleNarinfo1027=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1028=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1029=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1030=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1031=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1032=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1033=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1034=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1035=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1036=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1037=== CONT TestMultipartCleanup10382026/11/11 23:39:36 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1039--- PASS: TestReadProxyNarinfo (2.39s)1040=== CONT TestIsValidUploadKey1041=== RUN TestIsValidUploadKey/narinfo1042=== PAUSE TestIsValidUploadKey/narinfo1043=== RUN TestIsValidUploadKey/nar_zst1044=== PAUSE TestIsValidUploadKey/nar_zst1045=== RUN TestIsValidUploadKey/nar_xz1046=== PAUSE TestIsValidUploadKey/nar_xz1047=== RUN TestIsValidUploadKey/nar_plain1048=== PAUSE TestIsValidUploadKey/nar_plain1049=== RUN TestIsValidUploadKey/listing1050=== PAUSE TestIsValidUploadKey/listing1051=== RUN TestIsValidUploadKey/build_log1052=== PAUSE TestIsValidUploadKey/build_log1053=== RUN TestIsValidUploadKey/build_log_home-manager_file1054=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1055=== RUN TestIsValidUploadKey/build_log_plus_in_name1056=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1057=== RUN TestIsValidUploadKey/build_log_question_mark1058=== PAUSE TestIsValidUploadKey/build_log_question_mark1059=== RUN TestIsValidUploadKey/build_log_equals1060=== PAUSE TestIsValidUploadKey/build_log_equals1061=== RUN TestIsValidUploadKey/realisation1062=== PAUSE TestIsValidUploadKey/realisation1063=== RUN TestIsValidUploadKey/realisation_plus_in_output1064=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1065=== RUN TestIsValidUploadKey/nix-cache-info1066=== PAUSE TestIsValidUploadKey/nix-cache-info1067=== RUN TestIsValidUploadKey/index.html1068=== PAUSE TestIsValidUploadKey/index.html1069=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1070=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1071=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1072=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1073=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1074=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1075=== RUN TestIsValidUploadKey/traversal1076=== PAUSE TestIsValidUploadKey/traversal1077=== RUN TestIsValidUploadKey/traversal_nar1078=== PAUSE TestIsValidUploadKey/traversal_nar1079=== RUN TestIsValidUploadKey/absolute1080=== PAUSE TestIsValidUploadKey/absolute1081=== RUN TestIsValidUploadKey/empty_key1082=== PAUSE TestIsValidUploadKey/empty_key1083=== RUN TestIsValidUploadKey/unknown_type1084=== PAUSE TestIsValidUploadKey/unknown_type1085=== CONT TestServerTLSConfig1086=== RUN TestServerTLSConfig/no_client_CA1087=== PAUSE TestServerTLSConfig/no_client_CA1088=== RUN TestServerTLSConfig/missing_CA_file1089=== PAUSE TestServerTLSConfig/missing_CA_file1090=== RUN TestServerTLSConfig/not_a_PEM_file1091=== PAUSE TestServerTLSConfig/not_a_PEM_file1092=== CONT TestService_NativeMTLS10932026/11/11 23:39:36 INFO Received push request method=POST path=/api/pushes1094--- PASS: TestReadProxyNarStreaming (2.57s)1095=== CONT TestGenerateLandingPage1096--- PASS: TestReadProxyContentNegotiation (2.53s)1097=== CONT TestService_readinessHandler1098--- PASS: TestGenerateLandingPage (0.01s)1099=== CONT TestMetricsCommit11002026/11/11 23:39:37 INFO Received push request method=POST path=/api/pushes1101--- PASS: TestReadProxyHead (2.84s)1102=== CONT TestService_healthCheckHandler1103--- PASS: TestReadProxyInvalidPath (2.82s)1104=== CONT TestMetricsInventory11052026/11/11 23:39:37 INFO Received complete push request method=POST path=/api/pushes/1/complete11062026/11/11 23:39:37 INFO Received uploads request method=POST path=/api/pending_closures1107--- PASS: TestPresentReportsOnlyClosureRoots (2.31s)1108=== CONT TestGracefulShutdownDrainsInflight11092026/11/11 23:39:37 INFO Starting HTTP server address=127.0.0.1:3370511102026/11/11 23:39:37 INFO Shutdown signal received, draining in-flight requests timeout=10s1111--- PASS: TestReadProxyNarinfoAlreadyDecompressed (2.67s)1112=== CONT TestNARDeduplicationMetadataUploadBug11132026/11/11 23:39:37 INFO Received push request method=POST path=/api/pushes1114--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (4.15s)1115=== CONT TestGCTaskStore_Fail1116--- PASS: TestGCTaskStore_Fail (0.00s)1117=== CONT TestLeadIncumbentAnnouncesAtOnce1118--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1119=== CONT TestGCTaskStore_PhaseUpdates1120--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1121=== CONT TestGCTaskStore_CompletedAllowsNewTask1122--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1123=== CONT TestGCTaskStore_ConflictDifferentParams1124--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1125=== CONT TestGCTaskStore_GetReturnsLatest1126--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1127=== CONT TestGCTaskStore_DeduplicateSameParams1128--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1129=== CONT TestGCTaskStore_StartNew1130--- PASS: TestGCTaskStore_StartNew (0.00s)1131=== CONT TestCreatePendingClosureRejectsOversizedNAR11322026/11/11 23:39:37 INFO Received uploads request method=POST path=/api/pending_closures1133--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1134=== CONT TestGCEndsOnShutdown1135=== RUN TestGCEndsOnShutdown/before_the_run1136=== PAUSE TestGCEndsOnShutdown/before_the_run1137=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1138=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1139=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1140=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1141=== CONT TestCacheConfigHandlerMaxNarSize1142--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1143=== CONT TestPendingClosureWriteTimeout1144=== RUN TestPendingClosureWriteTimeout/empty1145=== PAUSE TestPendingClosureWriteTimeout/empty1146=== RUN TestPendingClosureWriteTimeout/400_objects1147=== PAUSE TestPendingClosureWriteTimeout/400_objects1148=== RUN TestPendingClosureWriteTimeout/670k_objects1149=== PAUSE TestPendingClosureWriteTimeout/670k_objects1150=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1151=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1152=== CONT TestCommitRacingPendingCleanupKeepsObjects11532026/11/11 23:39:37 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1154--- PASS: TestReadRedirectUsesPublicS3URL (4.16s)1155=== CONT TestEnsureAbortUploadsRule1156--- PASS: TestReadProxyRootRedirectsToIndexHTML (4.17s)1157=== CONT TestPushWaitsForSweepHoldingRow11582026/11/11 23:39:37 INFO Completed push id=1 objects=6 duration=397.093548ms11592026/11/11 23:39:37 INFO Completed push id=1 objects=4 duration=882.899102ms1160--- PASS: TestPush_CompleteCommitsEveryRoot (4.43s)1161=== CONT TestService_verifyS3Integrity1162=== CONT TestSweepRechecksPendingAfterLocking1163--- PASS: TestPush_RepeatedRootCommitsOnce (4.43s)11642026-11-11 23:39:37.461 UTC [572] ERROR: could not obtain lock on row in relation "closures"11652026-11-11 23:39:37.461 UTC [572] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11662026-11-11 23:39:37.462 UTC [572] ERROR: could not obtain lock on row in relation "closures"11672026-11-11 23:39:37.462 UTC [572] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11682026-11-11 23:39:37.463 UTC [572] ERROR: could not obtain lock on row in relation "closures"11692026-11-11 23:39:37.463 UTC [572] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11702026-11-11 23:39:37.463 UTC [572] ERROR: could not obtain lock on row in relation "closures"11712026-11-11 23:39:37.463 UTC [572] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11722026-11-11 23:39:37.463 UTC [572] ERROR: could not obtain lock on row in relation "closures"11732026-11-11 23:39:37.463 UTC [572] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1174--- PASS: TestPush_RootsAreLockedInByteOrder (4.46s)1175=== CONT TestClientMultipleUploads11762026/11/11 23:39:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=011772026/11/11 23:39:37 INFO Vacuumed table table=pending_closures11782026/11/11 23:39:37 INFO Vacuumed table table=pending_objects11792026/11/11 23:39:37 INFO Vacuumed table table=multipart_uploads11802026/11/11 23:39:37 INFO Vacuumed table table=closures11812026/11/11 23:39:37 INFO Vacuumed table table=objects11822026/11/11 23:39:37 INFO Received complete push request method=POST path=/api/pushes/2/complete11832026/11/11 23:39:37 INFO Completed push id=2 objects=4 duration=7.877155ms1184--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (4.69s)1185=== CONT TestClientCADerivations11862026/11/11 23:39:37 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket39 days=21187--- PASS: TestEnsureAbortUploadsRule (0.38s)1188=== CONT TestClientIntegration11892026/11/11 23:39:37 INFO Received cleanup request method=DELETE path=/api/pending_closures11902026/11/11 23:39:37 INFO Aborted multipart uploads count=0 kept=011912026/11/11 23:39:37 INFO Received uploads request method=POST path=/api/pending_closures1192--- PASS: TestMetricsInventory (0.57s)1193=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent11942026/11/11 23:39:37 INFO Received cleanup request method=DELETE path=/api/pending_closures11952026/11/11 23:39:37 INFO Aborted multipart uploads count=1 kept=011962026/11/11 23:39:37 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11972026-11-11 23:39:37.634 UTC [560] ERROR: Closure does not exist: id=111982026-11-11 23:39:37.634 UTC [560] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE11992026-11-11 23:39:37.634 UTC [560] STATEMENT: -- name: CommitPendingClosure :one1200 SELECT commit_pending_closure($1::bigint)::bigint1201 1202--- PASS: TestService_cleanupPendingClosuresHandler (1.04s)1203=== CONT TestGetExistingObjectsLocksRows12042026/11/11 23:39:37 INFO Received uploads request method=POST path=/api/pending_closures12052026/11/11 23:39:37 INFO lead: acquired remote=192.0.2.1:1234 generation=112062026/11/11 23:39:37 INFO lead: released remote=192.0.2.1:1234 generation=11207--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.52s)1208=== CONT TestClientErrorHandling1209=== RUN TestClientErrorHandling/InvalidStorePath1210=== PAUSE TestClientErrorHandling/InvalidStorePath1211=== RUN TestClientErrorHandling/InvalidAuthToken1212=== PAUSE TestClientErrorHandling/InvalidAuthToken1213=== RUN TestClientErrorHandling/ServerNotAvailable1214=== PAUSE TestClientErrorHandling/ServerNotAvailable1215=== CONT TestLeadElectsOneAndHandsOver12162026/11/11 23:39:37 WARN readiness check failed error="closed pool"1217--- PASS: TestService_readinessHandler (0.76s)1218=== CONT TestRegisterCompletedObjectNeedsPendingRow12192026/11/11 23:39:37 INFO Aborted multipart uploads count=0 kept=01220=== NAME TestNARDeduplicationMetadataUploadBug1221 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3343055911/001/store/crg5cvzkykjwyrgh158g5sgyr1d0h94p-file1.txt12222026/11/11 23:39:37 WARN mTLS auth: subject not in bound subjects subject="CN=reader"12232026/11/11 23:39:37 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1224--- PASS: TestService_NativeMTLS (1.07s)1225=== CONT TestService_ReadAuthMiddleware1226--- PASS: TestObjectStatsUpdatedOncePerCommit (1.45s)1227=== CONT TestConnectWaitsForAPeerMigration1228--- PASS: TestGetExistingObjectsLocksRows (0.37s)1229=== CONT TestSweepSkipsRowsLockedByPush12302026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures12312026/11/11 23:39:38 INFO Aborted multipart uploads count=0 kept=01232--- PASS: TestObjectStatsTrigger (1.56s)1233=== CONT TestService_RequireScope_OIDC12342026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures1235--- PASS: TestPushWaitsForSweepHoldingRow (0.96s)1236=== CONT TestResolveDBConnectionString1237=== RUN TestResolveDBConnectionString/flag_wins1238=== PAUSE TestResolveDBConnectionString/flag_wins1239=== RUN TestResolveDBConnectionString/file_when_flag_empty1240=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1241=== RUN TestResolveDBConnectionString/missing_file_is_an_error1242=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1243=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1244=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1245=== RUN TestResolveDBConnectionString/nothing_configured1246=== PAUSE TestResolveDBConnectionString/nothing_configured1247=== CONT TestFailedS3DeleteKeepsTombstone12482026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures12492026/11/11 23:39:38 INFO Received push request method=POST path=/api/pushes1250--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (1.36s)1251=== CONT TestService_AuthMiddleware_OIDC12522026/11/11 23:39:38 INFO Received complete push request method=POST path=/api/pushes/1/complete12532026/11/11 23:39:38 WARN Force mode enabled - objects will be deleted immediately without grace period12542026/11/11 23:39:38 INFO Received cleanup request method=DELETE path=/api/pending_closures1255=== NAME TestClientMultipleUploads1256 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads2474262150/001/store/ibgprklqqn6zq7zm63snvhqv5g75w06q-test-file-0.txt12572026/11/11 23:39:38 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket57/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""12582026/11/11 23:39:38 INFO Aborted multipart uploads count=0 kept=112592026/11/11 23:39:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012602026/11/11 23:39:38 INFO Vacuumed table table=pending_closures12612026/11/11 23:39:38 INFO Received cleanup request method=DELETE path=/api/pending_closures12622026/11/11 23:39:38 INFO Vacuumed table table=pending_objects12632026/11/11 23:39:38 INFO Aborted multipart uploads count=1 kept=012642026/11/11 23:39:38 INFO Vacuumed table table=multipart_uploads12652026/11/11 23:39:38 INFO Vacuumed table table=closures12662026/11/11 23:39:38 INFO Vacuumed table table=objects1267--- PASS: TestSweepRechecksPendingAfterLocking (0.83s)1268=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1269--- PASS: TestMultipartCleanup (1.40s)1270=== CONT TestForceGCDuringPushOffersSweptObject1271=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1272=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1273=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1274=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1275=== CONT TestCacheStatsHandler12762026/11/11 23:39:38 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39173/oidc12772026/11/11 23:39:38 INFO lead: acquired remote=192.0.2.1:1234 generation=11278=== NAME TestClientCADerivations1279 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1290965359/001/store/6325rfnh4nz0fx09yy5n6lf69swb2iwj-ca-test1280--- PASS: TestService_healthCheckHandler (1.30s)1281=== CONT TestService_ReadScope_PublicByDefault1282=== NAME TestOrphanedObjectsGC1283 orphaned_objects_gc_test.go:287: GC Test Summary:1284 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1285 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1286 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1287 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1288 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1289--- PASS: TestOrphanedObjectsGC (1.80s)1290=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection12912026/11/11 23:39:38 WARN Force mode enabled - objects will be deleted immediately without grace period12922026/11/11 23:39:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012932026/11/11 23:39:38 INFO Completed push id=1 objects=4 duration=121.268656ms12942026/11/11 23:39:38 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12952026/11/11 23:39:38 INFO Vacuumed table table=pending_closures1296=== NAME TestClientIntegration1297 client_integration_test.go:292: Created store path: /build/TestClientIntegration1973485215/002/store/jf26p4k6jz0ps5y5lzc6zi0j6p9gz35s-test-file.txt12982026/11/11 23:39:38 INFO Completed upload id=2 objects=2 duration=4.930167ms12992026/11/11 23:39:38 INFO Vacuumed table table=pending_objects13002026/11/11 23:39:38 INFO Vacuumed table table=multipart_uploads1301--- PASS: TestMetricsCommit (1.43s)1302=== CONT TestDeleteClosuresRacingPins13032026/11/11 23:39:38 INFO Vacuumed table table=closures13042026/11/11 23:39:38 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35969/oidc13052026/11/11 23:39:38 INFO Vacuumed table table=objects13062026/11/11 23:39:38 INFO Aborted multipart uploads count=0 kept=01307--- PASS: TestService_ReadAuthMiddleware (0.44s)1308=== CONT TestGCSweepDeliversEachKeyOnce13092026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures13102026/11/11 23:39:38 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1311--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.83s)1312=== CONT TestCacheConfigHandler1313=== RUN TestCacheConfigHandler/full_config,_no_issuer1314=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1315=== RUN TestCacheConfigHandler/no_cache_url_configured1316=== PAUSE TestCacheConfigHandler/no_cache_url_configured1317=== RUN TestCacheConfigHandler/no_signing_keys1318=== PAUSE TestCacheConfigHandler/no_signing_keys1319=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1320=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1321=== CONT TestPinProtectsFromGC13222026/11/11 23:39:38 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LjNhOTYyMDY5LThjZDEtNDc2YS1iMjBhLWZmYWFjNDNhZjE3YXgxNzk0NDQwMzc4MDgxOTg3NzQy parts=1013232026/11/11 23:39:38 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete13242026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures13252026/11/11 23:39:38 INFO Completed upload id=1 objects=2 duration=4.284381ms13262026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures13272026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures13282026/11/11 23:39:38 INFO Received uploads request method=POST path=/api/pending_closures13292026/11/11 23:39:38 INFO Aborted multipart uploads count=0 kept=013302026/11/11 23:39:38 INFO lead: acquired remote=192.0.2.1:1234 generation=113312026/11/11 23:39:38 INFO Received cleanup request method=DELETE path=/api/pending_closures13322026/11/11 23:39:38 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo13332026/11/11 23:39:38 WARN Found objects in DB but missing from S3, will re-upload count=11334--- PASS: TestService_verifyS3Integrity (1.15s)1335=== CONT TestTombstonedObjectOfferedWithoutWaiting13362026/11/11 23:39:39 INFO lead: released remote=192.0.2.1:1234 generation=113372026/11/11 23:39:39 WARN Force mode enabled - objects will be deleted immediately without grace period13382026/11/11 23:39:39 WARN Force mode enabled - objects will be deleted immediately without grace period1339--- PASS: TestCommitRacingPendingCleanupKeepsObjects (2.40s)1340=== CONT TestClientFallsBackToClosures13412026/11/11 23:39:39 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01342=== NAME TestClientMultipleUploads1343 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads2474262150/001/store/qvkila1y98j27h4px3d0sl5xngrl8891-test-file-1.txt13442026/11/11 23:39:39 INFO Vacuumed table table=pending_closures13452026/11/11 23:39:39 INFO Received push request method=POST path=/api/pushes13462026/11/11 23:39:39 INFO Vacuumed table table=pending_objects13472026/11/11 23:39:39 INFO lead: acquired remote=192.0.2.1:1234 generation=21348=== NAME TestClientCADerivations1349 client_ca_test.go:139: Found 1 dependencies (including self)13502026/11/11 23:39:39 INFO Vacuumed table table=multipart_uploads13512026/11/11 23:39:39 INFO Vacuumed table table=closures13522026/11/11 23:39:39 INFO Vacuumed table table=objects13532026/11/11 23:39:39 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13542026/11/11 23:39:39 INFO Uploading crg5cvzkykjwyrgh158g5sgyr1d0h94p-file1.txt (160B)13552026/11/11 23:39:39 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."13562026/11/11 23:39:39 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."13572026/11/11 23:39:39 INFO Aborted multipart uploads count=0 kept=01358--- PASS: TestFailedS3DeleteKeepsTombstone (1.44s)1359=== CONT TestClientWithDependencies13602026/11/11 23:39:39 WARN Failed to register uploaded object key=crg5cvzkykjwyrgh158g5sgyr1d0h94p.ls error="server returned 404: 404 page not found\n"13612026/11/11 23:39:39 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"13622026/11/11 23:39:39 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13632026/11/11 23:39:39 INFO Signed narinfos id=1 count=113642026/11/11 23:39:39 INFO Uploading 1 narinfos13652026/11/11 23:39:39 INFO Received complete push request method=POST path=/api/pushes/1/complete13662026/11/11 23:39:39 WARN Failed to register uploaded object key=crg5cvzkykjwyrgh158g5sgyr1d0h94p.narinfo error="server returned 404: 404 page not found\n"13672026/11/11 23:39:39 INFO Completed push id=1 objects=3 duration=11.015225ms13682026/11/11 23:39:39 INFO Upload complete. (1.402s)1369=== NAME TestNARDeduplicationMetadataUploadBug1370 metadata_upload_test.go:54: Retrieved narinfo from S3:1371 StorePath: /build/TestNARDeduplicationMetadataUploadBug3343055911/001/store/crg5cvzkykjwyrgh158g5sgyr1d0h94p-file1.txt1372 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1373 Compression: zstd1374 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1375 NarSize: 1601376 References: 1377 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1378 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1379 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1380 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}13812026/11/11 23:39:39 INFO Received uploads request method=POST path=/api/pending_closures13822026/11/11 23:39:39 INFO lead: released remote=192.0.2.1:1234 generation=113832026/11/11 23:39:39 WARN Force mode enabled - objects will be deleted immediately without grace period13842026/11/11 23:39:39 INFO Received uploads request method=POST path=/api/pending_closures13852026/11/11 23:39:39 INFO lead: acquired remote=192.0.2.1:1234 generation=21386--- PASS: TestTombstonedObjectOfferedWithoutWaiting (1.14s)1387=== CONT TestClientSharedPathCommittedMidPush13882026/11/11 23:39:39 INFO Received push request method=POST path=/api/pushes1389=== NAME TestClientMultipleUploads1390 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads2474262150/001/store/qgrqypk2h1qpaxc40c695adq50wb9cwn-test-file-2.txt13912026/11/11 23:39:39 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13922026/11/11 23:39:39 INFO Uploading jf26p4k6jz0ps5y5lzc6zi0j6p9gz35s-test-file.txt (152B)13932026/11/11 23:39:39 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"13942026/11/11 23:39:39 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign1395=== RUN TestService_RequireScope_OIDC/builder_may_write1396=== PAUSE TestService_RequireScope_OIDC/builder_may_write1397=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1398=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1399=== RUN TestService_RequireScope_OIDC/ops_may_admin1400=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1401=== RUN TestService_RequireScope_OIDC/ops_may_not_write1402=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1403=== RUN TestService_RequireScope_OIDC/reader_may_not_write14042026/11/11 23:39:39 INFO Signed narinfos id=1 count=11405=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1406=== RUN TestService_RequireScope_OIDC/static_token_may_admin1407=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1408=== RUN TestService_RequireScope_OIDC/static_token_may_write1409=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1410=== RUN TestService_RequireScope_OIDC/reader_may_read1411=== PAUSE TestService_RequireScope_OIDC/reader_may_read1412=== RUN TestService_RequireScope_OIDC/writer_implies_read1413=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1414=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1415=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1416=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1417=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1418=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1419=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1420=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1421=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1422=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1423=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1424=== CONT TestGCSweepSkipsPendingObjects14252026/11/11 23:39:39 WARN Failed to register uploaded object key=jf26p4k6jz0ps5y5lzc6zi0j6p9gz35s.ls error="server returned 404: 404 page not found\n"14262026/11/11 23:39:39 INFO Uploading 1 narinfos14272026/11/11 23:39:39 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=014282026/11/11 23:39:39 INFO Vacuumed table table=pending_closures14292026/11/11 23:39:39 INFO Vacuumed table table=pending_objects14302026/11/11 23:39:39 INFO Vacuumed table table=multipart_uploads14312026/11/11 23:39:39 INFO Vacuumed table table=closures14322026/11/11 23:39:39 INFO Received complete push request method=POST path=/api/pushes/1/complete14332026/11/11 23:39:39 WARN Failed to register uploaded object key=jf26p4k6jz0ps5y5lzc6zi0j6p9gz35s.narinfo error="server returned 404: 404 page not found\n"14342026/11/11 23:39:39 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=21435=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1436=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1437=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1438=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected14392026/11/11 23:39:39 INFO Vacuumed table table=objects1440=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1441=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1442=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1443=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1444=== CONT TestGCBugBareHashReferences1445--- PASS: TestSweepSkipsRowsLockedByPush (1.79s)1446=== CONT TestGCAdvisoryLockBlocksConcurrentRun14472026/11/11 23:39:39 INFO lead: acquired remote=192.0.2.1:1234 generation=314482026/11/11 23:39:39 INFO Completed push id=1 objects=3 duration=12.334098ms14492026/11/11 23:39:39 INFO Upload complete. (207ms)1450--- PASS: TestCacheStatsHandler (1.56s)1451=== CONT TestLeadEndsOnShutdown1452--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (1.57s)1453=== CONT TestDeduplicatedObjectsRecordedAsPending1454--- PASS: TestService_ReadScope_PublicByDefault (1.60s)1455=== CONT TestLeadGenerationGrowsWithEveryGrant14562026/11/11 23:39:39 INFO Aborted multipart uploads count=0 kept=014572026/11/11 23:39:40 WARN Rate limiter enabled after throttle name=s3-test rate=514582026/11/11 23:39:40 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1459=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1460 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101461 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001462--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (7.21s)1463=== CONT TestPushDedupSurvivesConcurrentGC14642026/11/11 23:39:40 WARN Force mode enabled - objects will be deleted immediately without grace period1465=== NAME TestNARDeduplicationMetadataUploadBug1466 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug3343055911/001/store/62hyaxk8hdya9jpgjy0cdijy7qzbi1kk-file2.txt1467--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (2.13s)1468=== CONT TestGCMetrics1469=== NAME TestOrphanedObjectsGCStressTest1470 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains14712026/11/11 23:39:40 INFO All 1 paths already cached1472=== NAME TestClientIntegration1473 client_integration_test.go:318: Retrieved narinfo from S3:1474 StorePath: /build/TestClientIntegration1973485215/002/store/jf26p4k6jz0ps5y5lzc6zi0j6p9gz35s-test-file.txt1475 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1476 Compression: zstd1477 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11478 NarSize: 1521479 References: 1480 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11481 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1482 client_integration_test.go:319: Decompressed .ls content (64 bytes):1483 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}14842026/11/11 23:39:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014852026/11/11 23:39:40 INFO Vacuumed table table=pending_closures14862026/11/11 23:39:40 INFO Vacuumed table table=pending_objects14872026/11/11 23:39:40 INFO Vacuumed table table=multipart_uploads14882026/11/11 23:39:40 INFO Vacuumed table table=closures14892026/11/11 23:39:40 INFO Vacuumed table table=objects1490--- PASS: TestGCSweepDeliversEachKeyOnce (2.06s)1491=== CONT TestService_AuthMiddleware_MTLSBoundSubjects14922026/11/11 23:39:40 INFO Received uploads request method=POST path=/api/pending_closures14932026/11/11 23:39:40 INFO Aborted multipart uploads count=0 kept=014942026/11/11 23:39:40 INFO lead: acquired remote=192.0.2.1:1234 generation=114952026/11/11 23:39:40 INFO lead: acquired remote=192.0.2.1:1234 generation=114962026/11/11 23:39:40 INFO Received uploads request method=POST path=/api/pending_closures1497--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.68s)1498=== CONT TestService_AuthMiddleware_MTLSProxyHeader14992026/11/11 23:39:40 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1500--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.81s)1501=== CONT TestService_AuthMiddleware15022026/11/11 23:39:40 INFO Received uploads request method=POST path=/api/pending_closures15032026/11/11 23:39:40 INFO lead: released remote=192.0.2.1:1234 generation=21504--- PASS: TestLeadElectsOneAndHandsOver (3.01s)1505=== CONT TestClientPushesUseOnePush15062026/11/11 23:39:40 INFO Aborted multipart uploads count=0 kept=01507--- PASS: TestReadProxyOutlastsServerWriteTimeout (7.04s)1508=== CONT TestProxyWriteTimeout/narinfo1509=== CONT TestProxyWriteTimeout/10_GiB_nar1510=== CONT TestProxyWriteTimeout/unknown_size1511=== CONT TestProxyWriteTimeout/1_GiB_nar1512--- PASS: TestProxyWriteTimeout (0.27s)1513 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1514 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1515 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1516 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1517=== CONT TestPush_RejectsBadRequests/no_roots15182026/11/11 23:39:41 INFO Received push request method=POST path=/api/pushes1519=== CONT TestPush_RejectsBadRequests/bad_root15202026/11/11 23:39:41 INFO Received push request method=POST path=/api/pushes1521=== CONT TestPush_RejectsBadRequests/no_objects15222026/11/11 23:39:41 INFO Received push request method=POST path=/api/pushes1523=== CONT TestPush_RejectsBadRequests/root_not_in_objects15242026/11/11 23:39:41 INFO Received push request method=POST path=/api/pushes1525=== CONT TestIsValidCachePath/narinfo1526--- PASS: TestPush_RejectsBadRequests (1.46s)1527 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1528 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1529 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1530 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1531=== CONT TestIsValidCachePath/empty1532=== CONT TestIsValidCachePath/random_path1533=== CONT TestIsValidCachePath/invalid_char_u1534=== CONT TestIsValidCachePath/invalid_char_e1535=== CONT TestIsValidCachePath/traversal_in_middle1536=== CONT TestIsValidCachePath/traversal_parent1537=== CONT TestIsValidCachePath/index.html1538=== CONT TestIsValidCachePath/nix-cache-info1539=== CONT TestIsValidCachePath/realisation1540=== CONT TestIsValidCachePath/log1541=== CONT TestIsValidCachePath/ls1542=== CONT TestIsValidCachePath/nar_uncompressed1543=== CONT TestIsValidCachePath/nar_bz21544=== CONT TestIsValidCachePath/nar_xz1545=== CONT TestIsValidCachePath/nar_zst1546=== CONT TestIsValidCachePath/leading_slash1547=== CONT TestIsValidCachePath/wrong_extension1548=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1549=== CONT TestIsValidCachePath/short_hash1550--- PASS: TestIsValidCachePath (0.01s)1551 --- PASS: TestIsValidCachePath/narinfo (0.00s)1552 --- PASS: TestIsValidCachePath/empty (0.00s)1553 --- PASS: TestIsValidCachePath/random_path (0.00s)1554 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1555 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1556 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1557 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1558 --- PASS: TestIsValidCachePath/index.html (0.00s)1559 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1560 --- PASS: TestIsValidCachePath/realisation (0.00s)1561 --- PASS: TestIsValidCachePath/log (0.00s)1562 --- PASS: TestIsValidCachePath/ls (0.00s)1563 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1564 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1565 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1566 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1567 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1568 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1569 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1570 --- PASS: TestIsValidCachePath/short_hash (0.00s)1571=== CONT TestParseSingleRange/none1572=== CONT TestParseSingleRange/open-ended1573=== CONT TestParseSingleRange/start_far_past_EOF1574=== CONT TestParseSingleRange/malformed_both_empty1575=== CONT TestParseSingleRange/start_past_EOF1576=== CONT TestParseSingleRange/single_byte1577=== CONT TestParseSingleRange/closed1578=== CONT TestParseSingleRange/suffix_exceeds_size1579=== CONT TestParseSingleRange/malformed_end_before_start1580=== CONT TestParseSingleRange/suffix1581=== CONT TestParseSingleRange/end_clamped_to_size1582=== CONT TestParseSingleRange/multi-range_ignored1583=== CONT TestParseSingleRange/malformed_no_dash1584=== CONT TestParseSingleRange/unknown_unit1585=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1586--- PASS: TestParseSingleRange (0.00s)1587 --- PASS: TestParseSingleRange/none (0.00s)1588 --- PASS: TestParseSingleRange/open-ended (0.00s)1589 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1590 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1591 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1592 --- PASS: TestParseSingleRange/single_byte (0.00s)1593 --- PASS: TestParseSingleRange/closed (0.00s)1594 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1595 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1596 --- PASS: TestParseSingleRange/suffix (0.00s)1597 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1598 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1599 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1600 --- PASS: TestParseSingleRange/unknown_unit (0.00s)16012026/11/11 23:39:41 INFO lead: released remote=192.0.2.1:1234 generation=11602--- PASS: TestLeadEndsOnShutdown (1.79s)1603=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh16042026/11/11 23:39:41 INFO lead: released remote=192.0.2.1:1234 generation=31605--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (4.08s)1606=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info16072026/11/11 23:39:41 INFO Received uploads request method=POST path=/1608=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key16092026/11/11 23:39:41 INFO Received complete multipart upload request method=POST path=/1610=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal16112026/11/11 23:39:41 INFO Received uploads request method=POST path=/1612=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key16132026/11/11 23:39:41 INFO Received request for more parts method=POST path=/1614--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1615 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1616 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1617 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1618 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1619=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart16202026/11/11 23:39:41 INFO Received complete multipart upload request method=POST path=/16212026/11/11 23:39:41 INFO lead: released remote=192.0.2.1:1234 generation=116222026/11/11 23:39:41 INFO lead: acquired remote=192.0.2.1:1234 generation=21623=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure16242026/11/11 23:39:41 INFO Received uploads request method=POST path=/1625=== NAME TestClientWithDependencies1626 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies3663072921/001/store/lh6vh7xlwlp77xb6xfc8gs71dv1mimxj-test-script1627--- PASS: TestGCBugBareHashReferences (2.43s)1628=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts16292026/11/11 23:39:42 INFO Received request for more parts method=POST path=/16302026/11/11 23:39:42 WARN Force mode enabled - objects will be deleted immediately without grace period1631=== NAME TestOrphanedObjectsGCStressTest1632 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion16332026/11/11 23:39:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01634=== NAME TestPinProtectsFromGC1635 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC3721300907/001/store/qm4bcc04mc3kr6fvc2ga5llwwjyy4ipd-pinned-file.txt1636 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC3721300907/001/store/1f1by8ymlgqmd5hi03pmb8j6mjkafpcq-unpinned-file.txt16372026/11/11 23:39:42 INFO Received push request method=POST path=/api/pushes16382026/11/11 23:39:42 INFO Received push request method=POST path=/api/pushes16392026/11/11 23:39:42 INFO Vacuumed table table=pending_closures1640=== NAME TestClientWithDependencies1641 client_integration_test.go:663: Found 1 dependencies (including self)16422026/11/11 23:39:42 INFO Vacuumed table table=pending_objects16432026/11/11 23:39:42 INFO Vacuumed table table=multipart_uploads16442026/11/11 23:39:42 INFO Vacuumed table table=closures16452026/11/11 23:39:42 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)16462026/11/11 23:39:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016472026/11/11 23:39:42 INFO Uploading qvkila1y98j27h4px3d0sl5xngrl8891-test-file-1.txt (160B)16482026/11/11 23:39:42 INFO Uploading ibgprklqqn6zq7zm63snvhqv5g75w06q-test-file-0.txt (160B)16492026/11/11 23:39:42 INFO Uploading qgrqypk2h1qpaxc40c695adq50wb9cwn-test-file-2.txt (160B)16502026/11/11 23:39:42 INFO Vacuumed table table=pending_closures16512026/11/11 23:39:42 INFO Object in database but missing from S3 key=jf26p4k6jz0ps5y5lzc6zi0j6p9gz35s.ls16522026/11/11 23:39:42 WARN Found objects in DB but missing from S3, will re-upload count=116532026/11/11 23:39:42 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)16542026/11/11 23:39:42 INFO Vacuumed table table=objects16552026/11/11 23:39:42 INFO Vacuumed table table=pending_objects16562026/11/11 23:39:42 INFO Vacuumed table table=multipart_uploads16572026/11/11 23:39:42 INFO Vacuumed table table=closures16582026/11/11 23:39:42 INFO Vacuumed table table=objects1659=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1660--- PASS: TestGCSweepSkipsPendingObjects (2.49s)1661=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost16622026/11/11 23:39:42 INFO Aborted multipart uploads count=0 kept=016632026/11/11 23:39:42 WARN Force mode enabled - objects will be deleted immediately without grace period16642026/11/11 23:39:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016652026/11/11 23:39:42 INFO Vacuumed table table=pending_closures16662026/11/11 23:39:42 INFO Vacuumed table table=pending_objects16672026/11/11 23:39:42 INFO Vacuumed table table=multipart_uploads16682026/11/11 23:39:42 INFO Vacuumed table table=closures16692026/11/11 23:39:42 INFO Vacuumed table table=objects1670--- PASS: TestPushDedupSurvivesConcurrentGC (2.04s)1671=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed16722026/11/11 23:39:42 INFO Received complete push request method=POST path=/api/pushes/2/complete16732026/11/11 23:39:42 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"16742026/11/11 23:39:42 WARN Failed to register uploaded object key=ibgprklqqn6zq7zm63snvhqv5g75w06q.ls error="server returned 404: 404 page not found\n"16752026/11/11 23:39:42 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16762026/11/11 23:39:42 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"16772026/11/11 23:39:42 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"16782026/11/11 23:39:42 WARN Failed to register uploaded object key=jf26p4k6jz0ps5y5lzc6zi0j6p9gz35s.ls error="server returned 404: 404 page not found\n"16792026/11/11 23:39:42 WARN Failed to register uploaded object key=qgrqypk2h1qpaxc40c695adq50wb9cwn.ls error="server returned 404: 404 page not found\n"16802026/11/11 23:39:42 WARN Failed to register uploaded object key=qvkila1y98j27h4px3d0sl5xngrl8891.ls error="server returned 404: 404 page not found\n"16812026/11/11 23:39:42 INFO Signed narinfos id=1 count=316822026/11/11 23:39:42 INFO Uploading 3 narinfos16832026/11/11 23:39:42 INFO Completed push id=2 objects=3 duration=3.924417ms16842026/11/11 23:39:42 INFO Upload complete. (358ms)16852026/11/11 23:39:42 WARN Failed to register uploaded object key=qvkila1y98j27h4px3d0sl5xngrl8891.narinfo error="server returned 404: 404 page not found\n"16862026/11/11 23:39:42 INFO Received complete push request method=POST path=/api/pushes/1/complete16872026/11/11 23:39:42 WARN Failed to register uploaded object key=ibgprklqqn6zq7zm63snvhqv5g75w06q.narinfo error="server returned 404: 404 page not found\n"16882026/11/11 23:39:42 WARN Failed to register uploaded object key=qgrqypk2h1qpaxc40c695adq50wb9cwn.narinfo error="server returned 404: 404 page not found\n"1689=== NAME TestClientIntegration1690 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1691 client_integration_test.go:336: Decompressed .ls content (49 bytes):1692 {"version":1,"root":{"type":"regular","size":39}}16932026/11/11 23:39:42 INFO Completed push id=1 objects=9 duration=16.296416ms16942026/11/11 23:39:42 INFO Upload complete. (1.899s)1695=== NAME TestClientMultipleUploads1696 client_integration_test.go:417: Uploaded 3 paths in 2.552955354s1697--- PASS: TestClientMultipleUploads (4.87s)1698=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1699--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (1.76s)1700=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed17012026/11/11 23:39:42 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"17022026/11/11 23:39:42 WARN mTLS auth: bound subjects configured but subject DN unavailable17032026/11/11 23:39:42 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1704--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (1.91s)1705=== CONT TestIsValidUploadKey/narinfo1706=== CONT TestIsValidUploadKey/realisation_plus_in_output1707=== CONT TestIsValidUploadKey/unknown_type1708=== CONT TestIsValidUploadKey/empty_key1709=== CONT TestIsValidUploadKey/absolute1710=== CONT TestIsValidUploadKey/traversal_nar1711=== CONT TestIsValidUploadKey/traversal1712=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1713=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1714=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1715=== CONT TestIsValidUploadKey/nix-cache-info1716=== CONT TestIsValidUploadKey/build_log_home-manager_file1717=== CONT TestIsValidUploadKey/realisation1718=== CONT TestIsValidUploadKey/build_log_equals1719=== CONT TestIsValidUploadKey/build_log_question_mark1720=== CONT TestIsValidUploadKey/index.html1721=== CONT TestIsValidUploadKey/build_log_plus_in_name1722=== CONT TestIsValidUploadKey/nar_xz1723=== CONT TestIsValidUploadKey/nar_plain1724=== CONT TestIsValidUploadKey/nar_zst1725=== CONT TestIsValidUploadKey/build_log1726=== CONT TestIsValidUploadKey/listing1727--- PASS: TestIsValidUploadKey (0.00s)1728 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1729 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1730 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1731 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1732 --- PASS: TestIsValidUploadKey/absolute (0.00s)1733 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1734 --- PASS: TestIsValidUploadKey/traversal (0.00s)1735 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1736 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1737 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1738 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1739 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1740 --- PASS: TestIsValidUploadKey/realisation (0.00s)1741 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1742 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1743 --- PASS: TestIsValidUploadKey/index.html (0.00s)1744 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1745 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1746 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1747 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1748 --- PASS: TestIsValidUploadKey/build_log (0.00s)1749 --- PASS: TestIsValidUploadKey/listing (0.00s)1750=== CONT TestServerTLSConfig/no_client_CA1751=== CONT TestServerTLSConfig/not_a_PEM_file1752=== CONT TestServerTLSConfig/missing_CA_file1753--- PASS: TestServerTLSConfig (0.00s)1754 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1755 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1756 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1757=== CONT TestGCEndsOnShutdown/before_the_run1758=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark17592026/11/11 23:39:42 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1760--- PASS: TestService_AuthMiddleware (1.81s)1761=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete17622026/11/11 23:39:42 INFO Aborted multipart uploads count=0 kept=017632026/11/11 23:39:42 INFO lead: released remote=192.0.2.1:1234 generation=21764=== CONT TestPendingClosureWriteTimeout/empty1765=== CONT TestPendingClosureWriteTimeout/670k_objects1766=== CONT TestPendingClosureWriteTimeout/400_objects1767=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1768--- PASS: TestPresentNotReportedWhileGCDeletesClosure (0.00s)1769 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/stale (1.19s)1770 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/fresh (1.20s)17712026/11/11 23:39:42 INFO lead: acquired remote=192.0.2.1:1234 generation=317722026/11/11 23:39:42 INFO lead: released remote=192.0.2.1:1234 generation=31773--- PASS: TestLeadGenerationGrowsWithEveryGrant (2.91s)1774=== CONT TestClientErrorHandling/InvalidStorePath1775--- PASS: TestUploadHandlersRejectOversizedBody (0.20s)1776 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.07s)1777 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.05s)1778 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (1.07s)1779=== CONT TestClientErrorHandling/ServerNotAvailable1780=== NAME TestOrphanedObjectsGCStressTest1781 orphaned_objects_gc_test.go:503: Stress test completed successfully:1782 orphaned_objects_gc_test.go:504: - Active objects preserved: 201783 orphaned_objects_gc_test.go:505: - Objects deleted: 2101784 orphaned_objects_gc_test.go:506: - Total GC'd: 2101785--- PASS: TestOrphanedObjectsGCStressTest (6.81s)1786=== CONT TestClientErrorHandling/InvalidAuthToken1787--- PASS: TestConnectWaitsForAPeerMigration (5.76s)1788=== CONT TestResolveDBConnectionString/flag_wins1789=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1790=== CONT TestResolveDBConnectionString/nothing_configured1791=== CONT TestResolveDBConnectionString/missing_file_is_an_error1792=== CONT TestResolveDBConnectionString/file_when_flag_empty1793=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1794--- PASS: TestResolveDBConnectionString (0.00s)1795 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1796 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1797 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1798 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1799 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)18002026/11/11 23:39:43 WARN Force mode enabled - objects will be deleted immediately without grace period18012026/11/11 23:39:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=018022026/11/11 23:39:43 INFO Vacuumed table table=pending_closures18032026/11/11 23:39:43 INFO Vacuumed table table=pending_objects18042026/11/11 23:39:43 INFO Vacuumed table table=multipart_uploads18052026/11/11 23:39:43 INFO Vacuumed table table=closures18062026/11/11 23:39:43 INFO Vacuumed table table=objects18072026/11/11 23:39:43 INFO Received push request method=POST path=/api/pushes18082026/11/11 23:39:43 INFO Received push request method=POST path=/api/pushes1809--- PASS: TestGCMetrics (3.37s)1810=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check18112026/11/11 23:39:43 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18122026/11/11 23:39:43 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)18132026/11/11 23:39:43 INFO Uploading 6325rfnh4nz0fx09yy5n6lf69swb2iwj-ca-test (144B)18142026/11/11 23:39:43 WARN Failed to register uploaded object key=log/36bmqxbmb8y13lk4kiyfd9a2jggpz58v-ca-test.drv error="server returned 404: 404 page not found\n"18152026/11/11 23:39:43 WARN Failed to register uploaded object key=62hyaxk8hdya9jpgjy0cdijy7qzbi1kk.ls error="server returned 404: 404 page not found\n"18162026/11/11 23:39:43 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign18172026/11/11 23:39:43 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"18182026/11/11 23:39:43 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign18192026/11/11 23:39:43 INFO Signed narinfos id=2 count=118202026/11/11 23:39:43 WARN Failed to register uploaded object key=6325rfnh4nz0fx09yy5n6lf69swb2iwj.ls error="server returned 404: 404 page not found\n"18212026/11/11 23:39:43 INFO Uploading 1 narinfos18222026/11/11 23:39:43 INFO Signed narinfos id=1 count=118232026/11/11 23:39:43 INFO Uploading 1 narinfos18242026/11/11 23:39:43 INFO Received complete push request method=POST path=/api/pushes/2/complete18252026/11/11 23:39:43 INFO Received complete push request method=POST path=/api/pushes/1/complete18262026/11/11 23:39:43 WARN Failed to register uploaded object key=62hyaxk8hdya9jpgjy0cdijy7qzbi1kk.narinfo error="server returned 404: 404 page not found\n"18272026/11/11 23:39:43 WARN Failed to register uploaded object key=6325rfnh4nz0fx09yy5n6lf69swb2iwj.narinfo error="server returned 404: 404 page not found\n"18282026/11/11 23:39:43 INFO Completed push id=2 objects=3 duration=3.327852ms18292026/11/11 23:39:43 INFO Upload complete. (1.629s)1830=== NAME TestNARDeduplicationMetadataUploadBug1831 metadata_upload_test.go:76: Retrieved narinfo from S3:1832 StorePath: /build/TestNARDeduplicationMetadataUploadBug3343055911/001/store/62hyaxk8hdya9jpgjy0cdijy7qzbi1kk-file2.txt1833 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1834 Compression: zstd1835 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1836 NarSize: 1601837 References: 1838 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf18392026/11/11 23:39:43 INFO Completed push id=1 objects=4 duration=12.187916ms18402026/11/11 23:39:43 INFO Upload complete. (3.463s)1841 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1842 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1843 {"version":1,"root":{"type":"regular","size":44}}1844=== NAME TestClientCADerivations1845 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1290965359/001/store/6325rfnh4nz0fx09yy5n6lf69swb2iwj-ca-test1846 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1847 Compression: zstd1848 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1849 NarSize: 1441850 References: 1851 Deriver: /build/TestClientCADerivations1290965359/001/store/36bmqxbmb8y13lk4kiyfd9a2jggpz58v-ca-test.drv1852 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1853 client_ca_test.go:185: Checking for realisation files in S3...1854--- PASS: TestNARDeduplicationMetadataUploadBug (6.76s)1855=== CONT TestCacheConfigHandler/full_config,_no_issuer1856=== CONT TestCacheConfigHandler/no_signing_keys1857=== CONT TestCacheConfigHandler/no_cache_url_configured1858=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1859--- PASS: TestCacheConfigHandler (0.00s)1860 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1861 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1862 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1863 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1864=== CONT TestService_RequireScope_OIDC/builder_may_write1865=== CONT TestService_RequireScope_OIDC/writer_implies_read1866=== CONT TestService_RequireScope_OIDC/reader_may_read1867=== CONT TestService_RequireScope_OIDC/static_token_may_write1868=== CONT TestService_RequireScope_OIDC/static_token_may_admin1869=== CONT TestService_RequireScope_OIDC/reader_may_not_write1870=== CONT TestService_RequireScope_OIDC/ops_may_not_write1871=== CONT TestService_RequireScope_OIDC/ops_may_admin1872=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1873=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read18742026/11/11 23:39:43 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1875=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1876=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1877=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1878=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1879=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1880--- PASS: TestService_RequireScope_OIDC (1.65s)1881 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1882 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1883 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1884 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1885 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1886 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1887 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1888 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1889 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1890 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1891 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1892 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1893 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1894 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1895=== NAME TestClientCADerivations1896 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1897 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1898=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected18992026/11/11 23:39:43 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1900=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1901=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected19022026/11/11 23:39:43 WARN Authentication failed token_preview=eyJhbGciOi...l2LlaCmIPQ token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]19032026/11/11 23:39:43 INFO Aborted multipart uploads count=0 kept=01904--- PASS: TestService_AuthMiddleware_OIDC (1.57s)1905 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1906 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1907 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1908 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)19092026/11/11 23:39:43 INFO Received uploads request method=POST path=/api/pending_closures19102026/11/11 23:39:43 INFO Received uploads request method=POST path=/api/pending_closures19112026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19122026/11/11 23:39:44 INFO Aborted multipart uploads count=0 kept=019132026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19142026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19152026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19162026/11/11 23:39:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19172026/11/11 23:39:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19182026/11/11 23:39:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000200000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LjhjMTdjYTBiLWQ5NjMtNGQ3Ni05ZTI2LWViYjU5YTY5MDVmNngxNzk0NDQwMzgzOTUyOTEzNzg5 parts=1019192026/11/11 23:39:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmM0NGVkNzMzLTVmNWQtNDcyNy05YWZjLWFlZGE1MmJiZDQ2N3gxNzk0NDQwMzgzOTY4MTAxMTc0 parts=1019202026/11/11 23:39:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19212026/11/11 23:39:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19222026/11/11 23:39:44 INFO Completed upload id=1 objects=3 duration=2.856627ms19232026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19242026/11/11 23:39:44 INFO Completed upload id=1 objects=3 duration=2.826187ms19252026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19262026/11/11 23:39:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19272026/11/11 23:39:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LjIwMzVmZjA0LTFlNjEtNGUxNy1hNjI5LTNmMmY0ZWQ1YTZiNngxNzk0NDQwMzg0MDMwNzgyODkx parts=1019282026/11/11 23:39:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19292026/11/11 23:39:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19302026/11/11 23:39:44 INFO Completed upload id=1 objects=3 duration=2.937448ms19312026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19322026/11/11 23:39:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LjE4MGVhYzRkLWQ4M2UtNGQ5Mi05MjE5LTg1N2ZiNDljNTY5MHgxNzk0NDQwMzg0MDg5OTQ2MTU2 parts=1019332026/11/11 23:39:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19342026/11/11 23:39:44 INFO Completed upload id=1 objects=3 duration=5.325371ms19352026/11/11 23:39:44 WARN Force mode enabled - objects will be deleted immediately without grace period19362026/11/11 23:39:44 WARN Force mode enabled - objects will be deleted immediately without grace period19372026/11/11 23:39:44 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Post \"http://localhost:40069/bucket96/?delete=\": context canceled"19382026/11/11 23:39:44 INFO Aborted multipart uploads count=1 kept=01939--- PASS: TestGCEndsOnShutdown (0.00s)1940 --- PASS: TestGCEndsOnShutdown/before_the_run (1.68s)1941 --- PASS: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (2.20s)1942 --- PASS: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (2.23s)19432026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19442026/11/11 23:39:44 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst19452026/11/11 23:39:44 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0001.narinfo19462026/11/11 23:39:44 INFO Received cleanup request method=DELETE path=/api/pending_closures19472026/11/11 23:39:44 INFO Aborted multipart uploads count=1 kept=01948--- PASS: TestPendingCleanupUsesOneCutoff (8.01s)19492026/11/11 23:39:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19502026/11/11 23:39:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmU0NDI2MjFiLWJmNDAtNDg0OC05NjE5LTcwN2Y5OTMyOGU4NHgxNzk0NDQwMzg0MTk1MDk0OTU5 parts=1019512026/11/11 23:39:44 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete19522026/11/11 23:39:44 INFO Completed upload id=1 objects=3 duration=6.569723ms19532026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19542026/11/11 23:39:44 INFO Aborted multipart uploads count=0 kept=019552026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19562026/11/11 23:39:44 INFO Aborted multipart uploads count=0 kept=019572026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19582026/11/11 23:39:44 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst19592026/11/11 23:39:44 WARN Found objects in DB but missing from S3, will re-upload count=119602026/11/11 23:39:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19612026/11/11 23:39:44 INFO Received complete multipart upload request method=POST path=/api/multipart/complete19622026/11/11 23:39:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100400000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3Ljk4ODM1Yjg5LWJiZDktNDIzYi1hZTNhLTA4ZDJmM2YxNDZjMHgxNzk0NDQwMzg0NTEwMTYwNTg0 parts=1019632026/11/11 23:39:44 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete19642026/11/11 23:39:44 INFO Completed upload id=2 objects=3 duration=2.06172ms19652026/11/11 23:39:44 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100300000000000000000000.nar.zst upload_id=ZmVhZGFlYjQtYTc2ZC00ZTJmLWE1ZTItZDM3MWUxYWM3ZWI3LmQ3M2U2ZGY5LTk0ZjMtNDk2Zi1hZTkzLTYzY2EwMTIzMjlmZXgxNzk0NDQwMzg0NDU1OTEwNjY3 parts=1019662026/11/11 23:39:44 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete19672026/11/11 23:39:44 INFO Completed upload id=2 objects=3 duration=1.00669ms19682026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19692026/11/11 23:39:44 INFO Received uploads request method=POST path=/api/pending_closures19702026/11/11 23:39:44 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst19712026/11/11 23:39:44 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0004.narinfo19722026/11/11 23:39:44 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst19732026/11/11 23:39:44 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0003.narinfo1974--- PASS: TestService_verifyS3ReuploadsStaleNarinfo (0.00s)1975 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed (2.17s)1976 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed (2.36s)1977 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed (2.40s)1978 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed (2.59s)1979 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost (2.66s)1980--- PASS: TestPendingClosureWriteTimeout (0.00s)1981 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)1982 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)1983 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)1984 --- PASS: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (2.12s)19852026/11/11 23:39:45 INFO Received push request method=POST path=/api/pushes19862026/11/11 23:39:45 INFO Received push request method=POST path=/api/pushes19872026/11/11 23:39:45 INFO Received push request method=POST path=/api/pushes19882026/11/11 23:39:45 INFO Received push request method=POST path=/api/pushes19892026/11/11 23:39:45 WARN Authentication failed token_preview=invalid-token token_length=13 reason="static API token mismatch"19902026/11/11 23:39:45 WARN Force mode enabled - objects will be deleted immediately without grace period19912026/11/11 23:39:45 WARN Force mode enabled - objects will be deleted immediately without grace period19922026/11/11 23:39:45 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19932026/11/11 23:39:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019942026/11/11 23:39:45 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19952026/11/11 23:39:45 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)19962026/11/11 23:39:45 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19972026/11/11 23:39:45 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)19982026/11/11 23:39:45 INFO Uploading xz1ics0cmqxk17i0yrzr1v9a6qnxpb0f-top (224B)19992026/11/11 23:39:45 INFO Uploading qm4bcc04mc3kr6fvc2ga5llwwjyy4ipd-pinned-file.txt (128B)20002026/11/11 23:39:45 INFO Uploading lh6vh7xlwlp77xb6xfc8gs71dv1mimxj-test-script (136B)

loading 620 hidden lines…

26212026/11/11 23:40:02 INFO Received complete upload request method=POST path=/26222026/11/11 23:40:02 WARN Force mode enabled - objects will be deleted immediately without grace period26232026/11/11 23:40:02 INFO Completed upload id=7 objects=4 duration=1.899499ms26242026/11/11 23:40:02 INFO Aborted multipart uploads count=0 kept=026252026/11/11 23:40:02 INFO Aborted multipart uploads count=0 kept=026262026/11/11 23:40:02 WARN Force mode enabled - objects will be deleted immediately without grace period26272026/11/11 23:40:02 INFO Received complete upload request method=POST path=/26282026/11/11 23:40:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026292026/11/11 23:40:02 INFO Received complete upload request method=POST path=/26302026/11/11 23:40:02 INFO Vacuumed table table=pending_closures26312026/11/11 23:40:02 INFO Completed upload id=3 objects=24 duration=2.683565ms26322026/11/11 23:40:02 INFO Received push request method=POST path=/26332026/11/11 23:40:02 INFO Vacuumed table table=pending_objects26342026/11/11 23:40:02 INFO Completed upload id=6 objects=9 duration=1.426374ms26352026/11/11 23:40:02 INFO Vacuumed table table=multipart_uploads26362026/11/11 23:40:02 INFO Vacuumed table table=closures26372026/11/11 23:40:02 INFO Received push request method=POST path=/26382026/11/11 23:40:02 INFO Vacuumed table table=objects26392026/11/11 23:40:02 INFO Received push request method=POST path=/26402026/11/11 23:40:02 INFO Received push request method=POST path=/26412026/11/11 23:40:02 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."26422026/11/11 23:40:02 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."26432026/11/11 23:40:02 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."26442026/11/11 23:40:02 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."26452026/11/11 23:40:02 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."26462026/11/11 23:40:02 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."26472026/11/11 23:40:02 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."26482026/11/11 23:40:02 INFO Received complete upload request method=POST path=/26492026/11/11 23:40:02 INFO Completed upload id=10 objects=23 duration=1.992079ms26502026/11/11 23:40:02 INFO Received complete push request method=POST path=/26512026/11/11 23:40:02 INFO Received complete push request method=POST path=/26522026/11/11 23:40:02 INFO Completed push id=2 objects=25 duration=3.08913ms26532026/11/11 23:40:02 INFO Received uploads request method=POST path=/26542026/11/11 23:40:02 INFO Completed push id=2 objects=7 duration=4.447782ms26552026/11/11 23:40:02 INFO Aborted multipart uploads count=0 kept=026562026/11/11 23:40:02 WARN Force mode enabled - objects will be deleted immediately without grace period26572026/11/11 23:40:02 INFO Aborted multipart uploads count=0 kept=026582026/11/11 23:40:02 INFO Received complete upload request method=POST path=/26592026/11/11 23:40:02 INFO Aborted multipart uploads count=0 kept=026602026/11/11 23:40:02 INFO Completed upload id=2 objects=2 duration=1.555575ms26612026/11/11 23:40:02 INFO Received uploads request method=POST path=/26622026/11/11 23:40:02 INFO Received uploads request method=POST path=/26632026/11/11 23:40:02 INFO Received push request method=POST path=/26642026/11/11 23:40:02 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."26652026/11/11 23:40:02 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."26662026/11/11 23:40:02 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."26672026/11/11 23:40:02 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."26682026/11/11 23:40:02 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."26692026/11/11 23:40:02 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."26702026/11/11 23:40:02 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."26712026/11/11 23:40:02 INFO Received uploads request method=POST path=/26722026/11/11 23:40:02 INFO Aborted multipart uploads count=0 kept=026732026/11/11 23:40:02 WARN Force mode enabled - objects will be deleted immediately without grace period26742026/11/11 23:40:02 INFO Received push request method=POST path=/26752026/11/11 23:40:02 INFO Received complete push request method=POST path=/26762026/11/11 23:40:02 INFO Received complete push request method=POST path=/26772026/11/11 23:40:02 INFO Received complete upload request method=POST path=/26782026/11/11 23:40:02 INFO Completed upload id=3 objects=5 duration=1.13833ms26792026/11/11 23:40:02 INFO Completed push id=6 objects=5 duration=2.412103ms26802026/11/11 23:40:02 INFO Received push request method=POST path=/26812026/11/11 23:40:02 INFO Completed push id=14 objects=3 duration=3.178051ms26822026/11/11 23:40:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026832026/11/11 23:40:02 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026842026/11/11 23:40:02 INFO Vacuumed table table=pending_closures26852026/11/11 23:40:02 INFO Vacuumed table table=pending_closures26862026/11/11 23:40:02 INFO Received complete upload request method=POST path=/26872026/11/11 23:40:02 INFO Vacuumed table table=pending_objects26882026/11/11 23:40:02 INFO Vacuumed table table=multipart_uploads26892026/11/11 23:40:02 INFO Vacuumed table table=pending_objects26902026/11/11 23:40:02 INFO Vacuumed table table=multipart_uploads26912026/11/11 23:40:02 INFO Vacuumed table table=closures26922026/11/11 23:40:02 WARN Force mode enabled - objects will be deleted immediately without grace period26932026/11/11 23:40:02 INFO Vacuumed table table=closures26942026/11/11 23:40:02 INFO Completed upload id=11 objects=6 duration=2.305782ms26952026/11/11 23:40:02 WARN Force mode enabled - objects will be deleted immediately without grace period26962026/11/11 23:40:02 INFO Vacuumed table table=objects26972026/11/11 23:40:02 INFO Vacuumed table table=objects26982026/11/11 23:40:02 INFO Received push request method=POST path=/26992026/11/11 23:40:02 INFO Aborted multipart uploads count=0 kept=027002026/11/11 23:40:02 INFO Received push request method=POST path=/27012026/11/11 23:40:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027022026/11/11 23:40:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027032026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=027042026/11/11 23:40:03 INFO Received complete push request method=POST path=/27052026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027062026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27072026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27082026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27092026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27102026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27112026/11/11 23:40:03 INFO Vacuumed table table=closures27122026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27132026/11/11 23:40:03 INFO Vacuumed table table=objects27142026/11/11 23:40:03 INFO Vacuumed table table=closures27152026/11/11 23:40:03 INFO Vacuumed table table=objects27162026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27172026/11/11 23:40:03 INFO Received uploads request method=POST path=/27182026/11/11 23:40:03 INFO Completed push id=7 objects=20 duration=2.795767ms27192026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=027202026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27212026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27222026/11/11 23:40:03 INFO Received push request method=POST path=/27232026/11/11 23:40:03 INFO Vacuumed table table=closures27242026/11/11 23:40:03 INFO Received uploads request method=POST path=/27252026/11/11 23:40:03 INFO Vacuumed table table=objects27262026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=027272026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027282026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27292026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27302026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27312026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27322026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27332026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27342026/11/11 23:40:03 INFO Vacuumed table table=closures27352026/11/11 23:40:03 INFO Vacuumed table table=closures27362026/11/11 23:40:03 INFO Vacuumed table table=objects27372026/11/11 23:40:03 INFO Vacuumed table table=objects27382026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=027392026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period27402026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=027412026/11/11 23:40:03 INFO Received push request method=POST path=/27422026/11/11 23:40:03 INFO Received push request method=POST path=/27432026/11/11 23:40:03 INFO Received complete push request method=POST path=/27442026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027452026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27462026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27472026/11/11 23:40:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27482026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27492026/11/11 23:40:03 INFO Completed push id=5 objects=6 duration=4.257781ms27502026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27512026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27522026/11/11 23:40:03 INFO Received push request method=POST path=/27532026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27542026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27552026/11/11 23:40:03 INFO Vacuumed table table=closures27562026/11/11 23:40:03 INFO Received push request method=POST path=/27572026/11/11 23:40:03 INFO Vacuumed table table=closures27582026/11/11 23:40:03 INFO Received complete upload request method=POST path=/27592026/11/11 23:40:03 INFO Vacuumed table table=objects27602026/11/11 23:40:03 INFO Vacuumed table table=objects27612026/11/11 23:40:03 INFO Completed upload id=9 objects=7 duration=2.057839ms27622026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=027632026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period27642026/11/11 23:40:03 INFO Received push request method=POST path=/27652026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027662026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27672026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27682026/11/11 23:40:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27692026/11/11 23:40:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27702026/11/11 23:40:03 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27712026/11/11 23:40:03 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27722026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27732026/11/11 23:40:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27742026/11/11 23:40:03 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27752026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27762026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27772026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27782026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27792026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27802026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27812026/11/11 23:40:03 INFO Vacuumed table table=closures27822026/11/11 23:40:03 INFO Received complete push request method=POST path=/27832026/11/11 23:40:03 INFO Vacuumed table table=objects27842026/11/11 23:40:03 INFO Completed push id=12 objects=15 duration=2.112261ms27852026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=027862026/11/11 23:40:03 INFO Received push request method=POST path=/27872026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027882026/11/11 23:40:03 INFO Vacuumed table table=pending_closures27892026/11/11 23:40:03 INFO Received uploads request method=POST path=/27902026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=027912026/11/11 23:40:03 INFO Received uploads request method=POST path=/27922026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=027932026/11/11 23:40:03 INFO Received push request method=POST path=/27942026/11/11 23:40:03 INFO Vacuumed table table=pending_objects27952026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads27962026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27972026/11/11 23:40:03 INFO Vacuumed table table=closures27982026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period27992026/11/11 23:40:03 INFO Vacuumed table table=objects28002026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028012026/11/11 23:40:03 INFO Received push request method=POST path=/28022026/11/11 23:40:03 INFO Vacuumed table table=pending_closures28032026/11/11 23:40:03 INFO Vacuumed table table=pending_objects28042026/11/11 23:40:03 INFO Received complete push request method=POST path=/28052026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads28062026/11/11 23:40:03 INFO Received push request method=POST path=/28072026/11/11 23:40:03 INFO Vacuumed table table=closures28082026/11/11 23:40:03 INFO Received push request method=POST path=/28092026/11/11 23:40:03 INFO Vacuumed table table=objects28102026/11/11 23:40:03 INFO Completed push id=2 objects=10 duration=3.18831ms28112026/11/11 23:40:03 INFO Received push request method=POST path=/28122026/11/11 23:40:03 INFO Received complete upload request method=POST path=/28132026/11/11 23:40:03 INFO Received uploads request method=POST path=/28142026/11/11 23:40:03 INFO Completed upload id=3 objects=8 duration=2.05622ms28152026/11/11 23:40:03 INFO Received push request method=POST path=/28162026/11/11 23:40:03 INFO Received uploads request method=POST path=/28172026/11/11 23:40:03 INFO Received push request method=POST path=/28182026/11/11 23:40:03 INFO Received complete push request method=POST path=/28192026/11/11 23:40:03 INFO Received push request method=POST path=/28202026/11/11 23:40:03 INFO Completed push id=5 objects=3 duration=3.668135ms28212026/11/11 23:40:03 INFO Received uploads request method=POST path=/28222026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=028232026/11/11 23:40:03 INFO Received push request method=POST path=/28242026/11/11 23:40:03 INFO Received complete push request method=POST path=/28252026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=028262026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028272026/11/11 23:40:03 INFO Vacuumed table table=pending_closures28282026/11/11 23:40:03 INFO Vacuumed table table=pending_objects28292026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads28302026/11/11 23:40:03 INFO Vacuumed table table=closures28312026/11/11 23:40:03 INFO Vacuumed table table=objects28322026/11/11 23:40:03 INFO Completed push id=1 objects=5 duration=7.36473ms28332026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=028342026/11/11 23:40:03 INFO Received push request method=POST path=/28352026/11/11 23:40:03 INFO Received push request method=POST path=/28362026/11/11 23:40:03 INFO Received complete push request method=POST path=/28372026/11/11 23:40:03 INFO Received uploads request method=POST path=/28382026/11/11 23:40:03 INFO Received uploads request method=POST path=/28392026/11/11 23:40:03 INFO Completed push id=1 objects=7 duration=8.053177ms28402026/11/11 23:40:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28412026/11/11 23:40:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28422026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28432026/11/11 23:40:03 INFO Received complete push request method=POST path=/28442026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=028452026/11/11 23:40:03 INFO Completed push id=4 objects=4 duration=3.627955ms28462026/11/11 23:40:03 INFO Received push request method=POST path=/28472026/11/11 23:40:03 INFO Received complete push request method=POST path=/28482026/11/11 23:40:03 INFO Received uploads request method=POST path=/28492026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028502026/11/11 23:40:03 INFO Completed push id=8 objects=7 duration=1.623416ms28512026/11/11 23:40:03 INFO Received complete push request method=POST path=/28522026/11/11 23:40:03 INFO Vacuumed table table=pending_closures28532026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028542026/11/11 23:40:03 INFO Vacuumed table table=pending_objects28552026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads28562026/11/11 23:40:03 INFO Received complete upload request method=POST path=/28572026/11/11 23:40:03 INFO Vacuumed table table=pending_closures28582026/11/11 23:40:03 INFO Vacuumed table table=closures28592026/11/11 23:40:03 INFO Vacuumed table table=pending_objects28602026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads28612026/11/11 23:40:03 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28622026/11/11 23:40:03 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."28632026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28642026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."28652026/11/11 23:40:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."28662026/11/11 23:40:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."28672026/11/11 23:40:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28682026/11/11 23:40:03 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28692026/11/11 23:40:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28702026/11/11 23:40:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28712026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."28722026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."28732026/11/11 23:40:03 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."28742026/11/11 23:40:03 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."28752026/11/11 23:40:03 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."28762026/11/11 23:40:03 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28772026/11/11 23:40:03 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28782026/11/11 23:40:03 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."28792026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28802026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."28812026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."28822026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."28832026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."28842026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."28852026/11/11 23:40:03 INFO Completed upload id=4 objects=14 duration=3.280811ms28862026/11/11 23:40:03 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."28872026/11/11 23:40:03 INFO Completed push id=3 objects=5 duration=7.058867ms28882026/11/11 23:40:03 INFO Vacuumed table table=objects28892026/11/11 23:40:03 INFO Vacuumed table table=closures28902026/11/11 23:40:03 INFO Vacuumed table table=objects28912026/11/11 23:40:03 INFO Received push request method=POST path=/28922026/11/11 23:40:03 INFO Received uploads request method=POST path=/28932026/11/11 23:40:03 INFO Received uploads request method=POST path=/28942026/11/11 23:40:03 INFO Received push request method=POST path=/28952026/11/11 23:40:03 INFO Received complete push request method=POST path=/28962026/11/11 23:40:03 INFO Received push request method=POST path=/28972026/11/11 23:40:03 INFO Received push request method=POST path=/28982026/11/11 23:40:03 INFO Completed push id=6 objects=6 duration=2.397183ms28992026/11/11 23:40:03 INFO Received uploads request method=POST path=/29002026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=029012026/11/11 23:40:03 INFO Received complete push request method=POST path=/29022026/11/11 23:40:03 INFO Completed push id=8 objects=6 duration=1.488394ms29032026/11/11 23:40:03 INFO Received uploads request method=POST path=/29042026/11/11 23:40:03 INFO Received complete upload request method=POST path=/29052026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=029062026/11/11 23:40:03 INFO Received push request method=POST path=/29072026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period29082026/11/11 23:40:03 INFO Completed upload id=2 objects=10 duration=2.517504ms29092026/11/11 23:40:03 INFO Received complete upload request method=POST path=/29102026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=029112026/11/11 23:40:03 INFO Received push request method=POST path=/29122026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029132026/11/11 23:40:03 INFO Completed upload id=10 objects=2 duration=2.961689ms29142026/11/11 23:40:03 INFO Received complete push request method=POST path=/29152026/11/11 23:40:03 INFO Vacuumed table table=pending_closures29162026/11/11 23:40:03 INFO Received complete upload request method=POST path=/29172026/11/11 23:40:03 INFO Received uploads request method=POST path=/29182026/11/11 23:40:03 INFO Vacuumed table table=pending_objects29192026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads29202026/11/11 23:40:03 INFO Completed push id=10 objects=2 duration=2.736706ms29212026/11/11 23:40:03 INFO Completed upload id=9 objects=16 duration=2.1165ms29222026/11/11 23:40:03 INFO Vacuumed table table=closures29232026/11/11 23:40:03 INFO Vacuumed table table=objects29242026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029252026/11/11 23:40:03 INFO Vacuumed table table=pending_closures29262026/11/11 23:40:03 INFO Received push request method=POST path=/29272026/11/11 23:40:03 INFO Received push request method=POST path=/29282026/11/11 23:40:03 INFO Vacuumed table table=pending_objects29292026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads29302026/11/11 23:40:03 INFO Received push request method=POST path=/29312026/11/11 23:40:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29322026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29332026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."29342026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."29352026/11/11 23:40:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."29362026/11/11 23:40:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."29372026/11/11 23:40:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29382026/11/11 23:40:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29392026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29402026/11/11 23:40:03 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."29412026/11/11 23:40:03 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."29422026/11/11 23:40:03 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29432026/11/11 23:40:03 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29442026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."29452026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."29462026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."29472026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29482026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29492026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."29502026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29512026/11/11 23:40:03 INFO Vacuumed table table=closures29522026/11/11 23:40:03 INFO Received complete upload request method=POST path=/29532026/11/11 23:40:03 INFO Received push request method=POST path=/29542026/11/11 23:40:03 INFO Received complete upload request method=POST path=/29552026/11/11 23:40:03 INFO Received push request method=POST path=/29562026/11/11 23:40:03 INFO Vacuumed table table=objects29572026/11/11 23:40:03 INFO Completed upload id=5 objects=5 duration=3.544913ms29582026/11/11 23:40:03 INFO Completed upload id=4 objects=7 duration=2.470304ms29592026/11/11 23:40:03 INFO Received uploads request method=POST path=/29602026/11/11 23:40:03 INFO Received push request method=POST path=/29612026/11/11 23:40:03 INFO Received uploads request method=POST path=/29622026/11/11 23:40:03 INFO Received push request method=POST path=/29632026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=029642026/11/11 23:40:03 INFO Received complete upload request method=POST path=/29652026/11/11 23:40:03 INFO Received push request method=POST path=/29662026/11/11 23:40:03 INFO Received uploads request method=POST path=/29672026/11/11 23:40:03 INFO Completed upload id=12 objects=21 duration=1.631036ms29682026/11/11 23:40:03 INFO Received complete push request method=POST path=/29692026/11/11 23:40:03 INFO Received complete push request method=POST path=/29702026/11/11 23:40:03 INFO Received push request method=POST path=/29712026/11/11 23:40:03 INFO Received uploads request method=POST path=/29722026/11/11 23:40:03 INFO Received push request method=POST path=/29732026/11/11 23:40:03 INFO Received uploads request method=POST path=/29742026/11/11 23:40:03 INFO Received complete push request method=POST path=/29752026/11/11 23:40:03 INFO Completed push id=6 objects=26 duration=4.15164ms29762026/11/11 23:40:03 INFO Completed push id=5 objects=9 duration=5.032209ms29772026/11/11 23:40:03 INFO Completed push id=7 objects=27 duration=1.960959ms29782026/11/11 23:40:03 INFO Received push request method=POST path=/29792026/11/11 23:40:03 INFO Received uploads request method=POST path=/29802026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=029812026/11/11 23:40:03 INFO Received push request method=POST path=/29822026/11/11 23:40:03 INFO Vacuumed table table=pending_closures29832026/11/11 23:40:03 INFO Received complete push request method=POST path=/29842026/11/11 23:40:03 INFO Vacuumed table table=pending_objects29852026/11/11 23:40:03 INFO Received push request method=POST path=/29862026/11/11 23:40:03 INFO Received uploads request method=POST path=/29872026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads29882026/11/11 23:40:03 INFO Completed push id=3 objects=5 duration=2.426443ms29892026/11/11 23:40:03 INFO Vacuumed table table=closures29902026/11/11 23:40:03 INFO Vacuumed table table=objects29912026/11/11 23:40:03 INFO Received push request method=POST path=/29922026/11/11 23:40:03 INFO Received push request method=POST path=/29932026/11/11 23:40:03 INFO Received uploads request method=POST path=/29942026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=029952026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=029962026/11/11 23:40:03 INFO Received uploads request method=POST path=/29972026/11/11 23:40:03 INFO Received uploads request method=POST path=/29982026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period29992026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030002026/11/11 23:40:03 INFO Received complete push request method=POST path=/30012026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=030022026/11/11 23:40:03 INFO Vacuumed table table=pending_closures30032026/11/11 23:40:03 INFO Completed push id=10 objects=3 duration=1.738837ms30042026/11/11 23:40:03 INFO Vacuumed table table=pending_objects30052026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads30062026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period30072026/11/11 23:40:03 INFO Received push request method=POST path=/30082026/11/11 23:40:03 INFO Vacuumed table table=closures30092026/11/11 23:40:03 INFO Vacuumed table table=objects30102026/11/11 23:40:03 INFO Received push request method=POST path=/30112026/11/11 23:40:03 INFO Received push request method=POST path=/30122026/11/11 23:40:03 INFO Received push request method=POST path=/30132026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=030142026/11/11 23:40:03 INFO Received complete push request method=POST path=/30152026/11/11 23:40:03 INFO Received complete upload request method=POST path=/30162026/11/11 23:40:03 INFO Received push request method=POST path=/30172026/11/11 23:40:03 INFO Completed upload id=3 objects=9 duration=1.288433ms30182026/11/11 23:40:03 INFO Completed push id=13 objects=23 duration=2.134101ms30192026/11/11 23:40:03 INFO Received push request method=POST path=/30202026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period30212026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=030222026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030232026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period30242026/11/11 23:40:03 INFO Received uploads request method=POST path=/30252026/11/11 23:40:03 INFO Received uploads request method=POST path=/30262026/11/11 23:40:03 INFO Vacuumed table table=pending_closures30272026/11/11 23:40:03 INFO Vacuumed table table=pending_objects30282026/11/11 23:40:03 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."30292026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30302026/11/11 23:40:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30312026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."30322026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30332026/11/11 23:40:03 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30342026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30352026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30362026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30372026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads30382026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030392026/11/11 23:40:03 INFO Vacuumed table table=closures30402026/11/11 23:40:03 INFO Vacuumed table table=pending_closures30412026/11/11 23:40:03 INFO Vacuumed table table=objects30422026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=030432026/11/11 23:40:03 INFO Vacuumed table table=pending_objects30442026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads30452026/11/11 23:40:03 INFO Received uploads request method=POST path=/30462026/11/11 23:40:03 INFO Received uploads request method=POST path=/30472026/11/11 23:40:03 INFO Vacuumed table table=closures30482026/11/11 23:40:03 INFO Vacuumed table table=pending_closures30492026/11/11 23:40:03 INFO Vacuumed table table=objects30502026/11/11 23:40:03 INFO Received complete push request method=POST path=/30512026/11/11 23:40:03 INFO Vacuumed table table=pending_objects30522026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads30532026/11/11 23:40:03 INFO Vacuumed table table=closures30542026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=030552026/11/11 23:40:03 INFO Completed push id=5 objects=8 duration=3.635234ms30562026/11/11 23:40:03 INFO Vacuumed table table=objects30572026/11/11 23:40:03 INFO Received complete upload request method=POST path=/30582026/11/11 23:40:03 INFO Received push request method=POST path=/30592026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period30602026/11/11 23:40:03 INFO Completed upload id=15 objects=6 duration=933.348µs30612026/11/11 23:40:03 INFO Received push request method=POST path=/30622026/11/11 23:40:03 INFO Received push request method=POST path=/30632026/11/11 23:40:03 INFO Received push request method=POST path=/30642026/11/11 23:40:03 INFO Received uploads request method=POST path=/30652026/11/11 23:40:03 INFO Received complete upload request method=POST path=/30662026/11/11 23:40:03 INFO Completed upload id=10 objects=17 duration=1.316813ms30672026/11/11 23:40:03 INFO Received push request method=POST path=/30682026/11/11 23:40:03 INFO Received push request method=POST path=/30692026/11/11 23:40:03 INFO Received uploads request method=POST path=/30702026/11/11 23:40:03 INFO Received push request method=POST path=/30712026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=030722026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period30732026/11/11 23:40:03 INFO Received complete push request method=POST path=/30742026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030752026/11/11 23:40:03 INFO Received uploads request method=POST path=/30762026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=030772026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=030782026/11/11 23:40:03 INFO Vacuumed table table=pending_closures30792026/11/11 23:40:03 INFO Completed push id=11 objects=8 duration=4.597884ms30802026/11/11 23:40:03 INFO Vacuumed table table=pending_closures30812026/11/11 23:40:03 INFO Vacuumed table table=pending_objects30822026/11/11 23:40:03 INFO Received complete push request method=POST path=/30832026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads30842026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030852026/11/11 23:40:03 INFO Vacuumed table table=closures30862026/11/11 23:40:03 INFO Completed push id=5 objects=6 duration=2.309262ms30872026/11/11 23:40:03 INFO Vacuumed table table=pending_objects30882026/11/11 23:40:03 INFO Vacuumed table table=pending_closures30892026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads30902026/11/11 23:40:03 INFO Received complete push request method=POST path=/30912026/11/11 23:40:03 INFO Vacuumed table table=objects30922026/11/11 23:40:03 INFO Vacuumed table table=pending_objects30932026/11/11 23:40:03 INFO Vacuumed table table=closures30942026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads30952026/11/11 23:40:03 INFO Received uploads request method=POST path=/30962026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=030972026/11/11 23:40:03 INFO Vacuumed table table=closures30982026/11/11 23:40:03 INFO Received push request method=POST path=/30992026/11/11 23:40:03 INFO Vacuumed table table=objects31002026/11/11 23:40:03 INFO Completed push id=4 objects=4 duration=1.984219ms31012026/11/11 23:40:03 INFO Vacuumed table table=objects31022026/11/11 23:40:03 INFO Received push request method=POST path=/31032026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031042026/11/11 23:40:03 INFO Received push request method=POST path=/31052026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031062026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period31072026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031082026/11/11 23:40:03 INFO Received push request method=POST path=/31092026/11/11 23:40:03 INFO Received uploads request method=POST path=/31102026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031112026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031122026/11/11 23:40:03 INFO Vacuumed table table=pending_closures31132026/11/11 23:40:03 INFO Received push request method=POST path=/31142026/11/11 23:40:03 INFO Vacuumed table table=pending_closures31152026/11/11 23:40:03 INFO Vacuumed table table=pending_objects31162026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads31172026/11/11 23:40:03 INFO Vacuumed table table=pending_objects31182026/11/11 23:40:03 INFO Received push request method=POST path=/31192026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads31202026/11/11 23:40:03 INFO Vacuumed table table=closures31212026/11/11 23:40:03 INFO Vacuumed table table=closures31222026/11/11 23:40:03 INFO Vacuumed table table=objects31232026/11/11 23:40:03 INFO Vacuumed table table=objects31242026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031252026/11/11 23:40:03 INFO Received push request method=POST path=/31262026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031272026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031282026/11/11 23:40:03 INFO Received uploads request method=POST path=/31292026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period31302026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031312026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031322026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period31332026/11/11 23:40:03 INFO Vacuumed table table=pending_closures31342026/11/11 23:40:03 INFO Received uploads request method=POST path=/31352026/11/11 23:40:03 INFO Vacuumed table table=pending_closures31362026/11/11 23:40:03 INFO Vacuumed table table=pending_objects31372026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads31382026/11/11 23:40:03 INFO Vacuumed table table=pending_objects31392026/11/11 23:40:03 INFO Vacuumed table table=closures31402026/11/11 23:40:03 INFO Received push request method=POST path=/31412026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads31422026/11/11 23:40:03 INFO Vacuumed table table=objects31432026/11/11 23:40:03 INFO Vacuumed table table=closures31442026/11/11 23:40:03 INFO Vacuumed table table=objects31452026/11/11 23:40:03 INFO Received uploads request method=POST path=/31462026/11/11 23:40:03 INFO Received uploads request method=POST path=/31472026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031482026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031492026/11/11 23:40:03 INFO Received push request method=POST path=/31502026/11/11 23:40:03 INFO Received push request method=POST path=/31512026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period31522026/11/11 23:40:03 INFO Received uploads request method=POST path=/31532026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031542026/11/11 23:40:03 INFO Vacuumed table table=pending_closures31552026/11/11 23:40:03 INFO Received complete push request method=POST path=/31562026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031572026/11/11 23:40:03 INFO Vacuumed table table=pending_objects31582026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads31592026/11/11 23:40:03 INFO Vacuumed table table=pending_closures31602026/11/11 23:40:03 INFO Completed push id=16 objects=27 duration=1.938278ms31612026/11/11 23:40:03 INFO Received complete push request method=POST path=/31622026/11/11 23:40:03 INFO Received complete push request method=POST path=/31632026/11/11 23:40:03 INFO Received uploads request method=POST path=/31642026/11/11 23:40:03 INFO Vacuumed table table=closures31652026/11/11 23:40:03 INFO Completed push id=14 objects=6 duration=1.900458ms31662026/11/11 23:40:03 INFO Received push request method=POST path=/31672026/11/11 23:40:03 INFO Vacuumed table table=pending_objects31682026/11/11 23:40:03 INFO Vacuumed table table=objects31692026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads31702026/11/11 23:40:03 INFO Completed push id=13 objects=6 duration=3.709995ms31712026/11/11 23:40:03 INFO Vacuumed table table=closures31722026/11/11 23:40:03 INFO Vacuumed table table=objects31732026/11/11 23:40:03 INFO Received uploads request method=POST path=/31742026/11/11 23:40:03 INFO Received push request method=POST path=/31752026/11/11 23:40:03 INFO Received uploads request method=POST path=/31762026/11/11 23:40:03 INFO Received uploads request method=POST path=/31772026/11/11 23:40:03 INFO Received push request method=POST path=/31782026/11/11 23:40:03 INFO Received push request method=POST path=/31792026/11/11 23:40:03 INFO Received uploads request method=POST path=/31802026/11/11 23:40:03 INFO Received uploads request method=POST path=/31812026/11/11 23:40:03 INFO Received push request method=POST path=/31822026/11/11 23:40:03 INFO Received push request method=POST path=/31832026/11/11 23:40:03 INFO Received uploads request method=POST path=/31842026/11/11 23:40:03 INFO Received push request method=POST path=/31852026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=031862026/11/11 23:40:03 INFO Received push request method=POST path=/31872026/11/11 23:40:03 INFO Received uploads request method=POST path=/31882026/11/11 23:40:03 INFO Received push request method=POST path=/31892026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period31902026/11/11 23:40:03 INFO Received push request method=POST path=/31912026/11/11 23:40:03 INFO Received push request method=POST path=/31922026/11/11 23:40:03 INFO Received push request method=POST path=/31932026/11/11 23:40:03 INFO Received uploads request method=POST path=/31942026/11/11 23:40:03 INFO Received complete push request method=POST path=/31952026/11/11 23:40:03 INFO Received uploads request method=POST path=/31962026/11/11 23:40:03 INFO Completed push id=5 objects=3 duration=1.833597ms31972026/11/11 23:40:03 INFO Received push request method=POST path=/31982026/11/11 23:40:03 INFO Received push request method=POST path=/31992026/11/11 23:40:03 INFO Received uploads request method=POST path=/32002026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=032012026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32022026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32032026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32042026/11/11 23:40:03 INFO Vacuumed table table=closures32052026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032062026/11/11 23:40:03 INFO Vacuumed table table=objects32072026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=032082026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032092026/11/11 23:40:03 INFO Received uploads request method=POST path=/32102026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period32112026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32122026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32132026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32142026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32152026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032162026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032172026/11/11 23:40:03 INFO Received push request method=POST path=/32182026/11/11 23:40:03 INFO Vacuumed table table=closures32192026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32202026/11/11 23:40:03 INFO Received complete upload request method=POST path=/32212026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32222026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32232026/11/11 23:40:03 INFO Vacuumed table table=objects32242026/11/11 23:40:03 INFO Vacuumed table table=closures32252026/11/11 23:40:03 INFO Received complete upload request method=POST path=/32262026/11/11 23:40:03 INFO Received complete push request method=POST path=/32272026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32282026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32292026/11/11 23:40:03 INFO Vacuumed table table=objects32302026/11/11 23:40:03 INFO Completed upload id=8 objects=3 duration=2.699966ms32312026/11/11 23:40:03 INFO Vacuumed table table=closures32322026/11/11 23:40:03 INFO Completed push id=18 objects=10 duration=2.10722ms32332026/11/11 23:40:03 INFO Completed upload id=10 objects=6 duration=3.247731ms32342026/11/11 23:40:03 INFO Vacuumed table table=objects32352026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032362026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=032372026/11/11 23:40:03 INFO Received push request method=POST path=/32382026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32392026/11/11 23:40:03 INFO Received complete push request method=POST path=/32402026/11/11 23:40:03 INFO Received complete upload request method=POST path=/32412026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032422026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32432026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32442026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032452026/11/11 23:40:03 INFO Completed push id=17 objects=6 duration=2.304702ms32462026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32472026/11/11 23:40:03 INFO Vacuumed table table=closures32482026/11/11 23:40:03 INFO Completed upload id=15 objects=4 duration=2.481623ms32492026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32502026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032512026/11/11 23:40:03 INFO Vacuumed table table=objects32522026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032532026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32542026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period32552026/11/11 23:40:03 INFO Vacuumed table table=closures32562026/11/11 23:40:03 INFO Received uploads request method=POST path=/32572026/11/11 23:40:03 INFO Vacuumed table table=objects32582026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32592026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period32602026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032612026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32622026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032632026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32642026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32652026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32662026/11/11 23:40:03 INFO Vacuumed table table=closures32672026/11/11 23:40:03 INFO Vacuumed table table=objects32682026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32692026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32702026/11/11 23:40:03 INFO Vacuumed table table=closures32712026/11/11 23:40:03 INFO Vacuumed table table=objects32722026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032732026/11/11 23:40:03 INFO Received push request method=POST path=/32742026/11/11 23:40:03 INFO Received complete push request method=POST path=/32752026/11/11 23:40:03 INFO Received uploads request method=POST path=/32762026/11/11 23:40:03 INFO Completed push id=13 objects=12 duration=3.846897ms32772026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032782026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=032792026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period32802026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032812026/11/11 23:40:03 INFO Received complete push request method=POST path=/32822026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32832026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032842026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32852026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032862026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32872026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32882026/11/11 23:40:03 INFO Vacuumed table table=closures32892026/11/11 23:40:03 INFO Vacuumed table table=pending_closures32902026/11/11 23:40:03 INFO Vacuumed table table=objects32912026/11/11 23:40:03 INFO Completed push id=9 objects=7 duration=3.480653ms32922026/11/11 23:40:03 INFO Received complete push request method=POST path=/32932026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32942026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32952026/11/11 23:40:03 INFO Vacuumed table table=pending_objects32962026/11/11 23:40:03 INFO Vacuumed table table=closures32972026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads32982026/11/11 23:40:03 INFO Vacuumed table table=objects32992026/11/11 23:40:03 INFO Completed push id=4 objects=7 duration=2.440963ms33002026/11/11 23:40:03 INFO Vacuumed table table=closures33012026/11/11 23:40:03 INFO Received push request method=POST path=/33022026/11/11 23:40:03 INFO Vacuumed table table=objects33032026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=033042026/11/11 23:40:03 INFO Received uploads request method=POST path=/33052026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33062026/11/11 23:40:03 INFO Received uploads request method=POST path=/33072026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=033082026/11/11 23:40:03 INFO Completed upload id=6 objects=24 duration=4.061659ms33092026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period33102026/11/11 23:40:03 INFO Received push request method=POST path=/33112026/11/11 23:40:03 INFO Received complete push request method=POST path=/33122026/11/11 23:40:03 INFO Received uploads request method=POST path=/33132026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=033142026/11/11 23:40:03 INFO Received push request method=POST path=/33152026/11/11 23:40:03 INFO Completed push id=6 objects=6 duration=2.429223ms33162026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033172026/11/11 23:40:03 INFO Received uploads request method=POST path=/33182026/11/11 23:40:03 INFO Vacuumed table table=pending_closures33192026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33202026/11/11 23:40:03 INFO Completed upload id=16 objects=3 duration=1.09637ms33212026/11/11 23:40:03 INFO Vacuumed table table=pending_objects33222026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33232026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33242026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads33252026/11/11 23:40:03 INFO Received complete push request method=POST path=/33262026/11/11 23:40:03 INFO Received push request method=POST path=/33272026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33282026/11/11 23:40:03 INFO Vacuumed table table=closures33292026/11/11 23:40:03 INFO Completed upload id=10 objects=7 duration=1.531874ms33302026/11/11 23:40:03 INFO Vacuumed table table=objects33312026/11/11 23:40:03 INFO Completed upload id=17 objects=6 duration=2.224621ms33322026/11/11 23:40:03 INFO Received push request method=POST path=/33332026/11/11 23:40:03 INFO Completed upload id=19 objects=25 duration=2.845427ms33342026/11/11 23:40:03 INFO Completed push id=11 objects=11 duration=3.946838ms33352026/11/11 23:40:03 INFO Received uploads request method=POST path=/33362026/11/11 23:40:03 INFO Received push request method=POST path=/33372026/11/11 23:40:03 INFO Received push request method=POST path=/33382026/11/11 23:40:03 INFO Received uploads request method=POST path=/33392026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=033402026/11/11 23:40:03 INFO Received uploads request method=POST path=/33412026/11/11 23:40:03 INFO Received uploads request method=POST path=/33422026/11/11 23:40:03 INFO Received uploads request method=POST path=/33432026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033442026/11/11 23:40:03 INFO Vacuumed table table=pending_closures33452026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=033462026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period33472026/11/11 23:40:03 INFO Vacuumed table table=pending_objects33482026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads33492026/11/11 23:40:03 INFO Vacuumed table table=closures33502026/11/11 23:40:03 INFO Received complete push request method=POST path=/33512026/11/11 23:40:03 INFO Vacuumed table table=objects33522026/11/11 23:40:03 INFO Received push request method=POST path=/33532026/11/11 23:40:03 INFO Received complete push request method=POST path=/33542026/11/11 23:40:03 INFO Completed push id=8 objects=6 duration=3.484293ms33552026/11/11 23:40:03 INFO Completed push id=12 objects=2 duration=2.616345ms33562026/11/11 23:40:03 INFO Received uploads request method=POST path=/33572026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033582026/11/11 23:40:03 INFO Received push request method=POST path=/33592026/11/11 23:40:03 INFO Vacuumed table table=pending_closures33602026/11/11 23:40:03 INFO Vacuumed table table=pending_objects33612026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads33622026/11/11 23:40:03 INFO Vacuumed table table=closures33632026/11/11 23:40:03 INFO Vacuumed table table=objects33642026/11/11 23:40:03 INFO Received uploads request method=POST path=/33652026/11/11 23:40:03 INFO Received complete push request method=POST path=/33662026/11/11 23:40:03 INFO Received uploads request method=POST path=/33672026/11/11 23:40:03 INFO Completed push id=10 objects=15 duration=2.0984ms33682026/11/11 23:40:03 INFO Received complete push request method=POST path=/33692026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33702026/11/11 23:40:03 INFO Received uploads request method=POST path=/33712026/11/11 23:40:03 INFO Completed upload id=9 objects=3 duration=1.04689ms33722026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33732026/11/11 23:40:03 INFO Completed push id=4 objects=6 duration=2.204401ms33742026/11/11 23:40:03 INFO Completed upload id=11 objects=6 duration=1.941298ms33752026/11/11 23:40:03 INFO Received complete push request method=POST path=/33762026/11/11 23:40:03 INFO Received complete push request method=POST path=/33772026/11/11 23:40:03 INFO Completed push id=7 objects=7 duration=1.510895ms33782026/11/11 23:40:03 INFO Received uploads request method=POST path=/33792026/11/11 23:40:03 INFO Completed push id=3 objects=21 duration=5.807436ms33802026/11/11 23:40:03 INFO Received push request method=POST path=/33812026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33822026/11/11 23:40:03 INFO Received complete push request method=POST path=/33832026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=033842026/11/11 23:40:03 INFO Completed upload id=18 objects=4 duration=1.126891ms33852026/11/11 23:40:03 INFO Received uploads request method=POST path=/33862026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period33872026/11/11 23:40:03 INFO Completed push id=19 objects=6 duration=2.702386ms33882026/11/11 23:40:03 INFO Received push request method=POST path=/33892026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033902026/11/11 23:40:03 INFO Received uploads request method=POST path=/33912026/11/11 23:40:03 INFO Received complete upload request method=POST path=/33922026/11/11 23:40:03 INFO Vacuumed table table=pending_closures33932026/11/11 23:40:03 INFO Received uploads request method=POST path=/33942026/11/11 23:40:03 INFO Completed upload id=18 objects=4 duration=1.450334ms33952026/11/11 23:40:03 INFO Vacuumed table table=pending_objects33962026/11/11 23:40:03 INFO Received uploads request method=POST path=/33972026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads33982026/11/11 23:40:03 INFO Vacuumed table table=closures33992026/11/11 23:40:03 INFO Vacuumed table table=objects34002026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034012026/11/11 23:40:03 INFO Received push request method=POST path=/34022026/11/11 23:40:03 INFO Received complete upload request method=POST path=/34032026/11/11 23:40:03 INFO Completed upload id=9 objects=2 duration=1.251552ms34042026/11/11 23:40:03 INFO Received uploads request method=POST path=/34052026/11/11 23:40:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34062026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34072026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034082026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034092026/11/11 23:40:03 INFO Received push request method=POST path=/34102026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034112026/11/11 23:40:03 INFO Received push request method=POST path=/34122026/11/11 23:40:03 INFO Vacuumed table table=pending_closures34132026/11/11 23:40:03 INFO Vacuumed table table=pending_objects34142026/11/11 23:40:03 INFO Received complete push request method=POST path=/34152026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads34162026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034172026/11/11 23:40:03 INFO Vacuumed table table=closures34182026/11/11 23:40:03 INFO Received uploads request method=POST path=/34192026/11/11 23:40:03 INFO Vacuumed table table=pending_closures34202026/11/11 23:40:03 INFO Vacuumed table table=objects34212026/11/11 23:40:03 INFO Completed push id=8 objects=3 duration=1.690236ms34222026/11/11 23:40:03 INFO Vacuumed table table=pending_objects34232026/11/11 23:40:03 INFO Received complete upload request method=POST path=/34242026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads34252026/11/11 23:40:03 INFO Vacuumed table table=closures34262026/11/11 23:40:03 INFO Completed upload id=18 objects=11 duration=1.259532ms34272026/11/11 23:40:03 INFO Vacuumed table table=objects34282026/11/11 23:40:03 INFO Received complete push request method=POST path=/34292026/11/11 23:40:03 INFO Received uploads request method=POST path=/34302026/11/11 23:40:03 INFO Completed push id=17 objects=6 duration=1.680936ms34312026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034322026/11/11 23:40:03 INFO Received complete push request method=POST path=/34332026/11/11 23:40:03 INFO Received uploads request method=POST path=/34342026/11/11 23:40:03 INFO Completed push id=9 objects=5 duration=2.669725ms34352026/11/11 23:40:03 INFO Received push request method=POST path=/34362026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period34372026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034382026/11/11 23:40:03 INFO Received push request method=POST path=/34392026/11/11 23:40:03 INFO Received uploads request method=POST path=/34402026/11/11 23:40:03 INFO Received push request method=POST path=/34412026/11/11 23:40:03 INFO Received push request method=POST path=/34422026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period34432026/11/11 23:40:03 INFO Received uploads request method=POST path=/34442026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34452026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34462026/11/11 23:40:03 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34472026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034482026/11/11 23:40:03 INFO Received uploads request method=POST path=/34492026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period34502026/11/11 23:40:03 INFO Received uploads request method=POST path=/34512026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034522026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034532026/11/11 23:40:03 INFO Received push request method=POST path=/34542026/11/11 23:40:03 INFO Vacuumed table table=pending_closures34552026/11/11 23:40:03 INFO Vacuumed table table=pending_closures34562026/11/11 23:40:03 INFO Vacuumed table table=pending_objects34572026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads34582026/11/11 23:40:03 INFO Vacuumed table table=pending_objects34592026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads34602026/11/11 23:40:03 INFO Vacuumed table table=closures34612026/11/11 23:40:03 INFO Received uploads request method=POST path=/34622026/11/11 23:40:03 INFO Vacuumed table table=closures34632026/11/11 23:40:03 INFO Vacuumed table table=objects34642026/11/11 23:40:03 INFO Received uploads request method=POST path=/34652026/11/11 23:40:03 INFO Vacuumed table table=objects34662026/11/11 23:40:03 INFO Received push request method=POST path=/34672026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034682026/11/11 23:40:03 INFO Received push request method=POST path=/34692026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period34702026/11/11 23:40:03 INFO Received complete upload request method=POST path=/34712026/11/11 23:40:03 INFO Completed upload id=13 objects=10 duration=935.769µs34722026/11/11 23:40:03 INFO Received complete push request method=POST path=/34732026/11/11 23:40:03 INFO Received push request method=POST path=/34742026/11/11 23:40:03 INFO Completed push id=9 objects=4 duration=2.890028ms34752026/11/11 23:40:03 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34762026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34772026/11/11 23:40:03 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34782026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034792026/11/11 23:40:03 INFO Received uploads request method=POST path=/34802026/11/11 23:40:03 INFO Received push request method=POST path=/34812026/11/11 23:40:03 INFO Received complete upload request method=POST path=/34822026/11/11 23:40:03 INFO Completed upload id=15 objects=9 duration=1.184931ms34832026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034842026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034852026/11/11 23:40:03 INFO Received push request method=POST path=/34862026/11/11 23:40:03 INFO Vacuumed table table=pending_closures34872026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034882026/11/11 23:40:03 INFO Vacuumed table table=pending_objects34892026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads34902026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period34912026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=034922026/11/11 23:40:03 INFO Vacuumed table table=closures34932026/11/11 23:40:03 INFO Received complete push request method=POST path=/34942026/11/11 23:40:03 INFO Vacuumed table table=objects34952026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034962026/11/11 23:40:03 INFO Vacuumed table table=pending_closures34972026/11/11 23:40:03 INFO Completed push id=15 objects=5 duration=1.927719ms34982026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period34992026/11/11 23:40:03 INFO Vacuumed table table=pending_objects35002026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads35012026/11/11 23:40:03 INFO Vacuumed table table=closures35022026/11/11 23:40:03 INFO Vacuumed table table=objects35032026/11/11 23:40:03 INFO Received complete push request method=POST path=/35042026/11/11 23:40:03 INFO Completed push id=12 objects=3 duration=1.728977ms35052026/11/11 23:40:03 INFO Received complete push request method=POST path=/35062026/11/11 23:40:03 INFO Completed push id=1 objects=11 duration=2.353362ms35072026/11/11 23:40:03 INFO Received complete push request method=POST path=/35082026/11/11 23:40:03 INFO Received uploads request method=POST path=/35092026/11/11 23:40:03 INFO Completed push id=22 objects=3 duration=2.477744ms35102026/11/11 23:40:03 INFO Received uploads request method=POST path=/35112026/11/11 23:40:03 INFO Received uploads request method=POST path=/35122026/11/11 23:40:03 INFO Received uploads request method=POST path=/35132026/11/11 23:40:03 INFO Received complete push request method=POST path=/35142026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=035152026/11/11 23:40:03 INFO Completed push id=21 objects=17 duration=1.972499ms35162026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period35172026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35182026/11/11 23:40:03 INFO Completed upload id=16 objects=3 duration=1.132371ms35192026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."35202026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=035212026/11/11 23:40:03 INFO Received uploads request method=POST path=/35222026/11/11 23:40:03 INFO Received complete push request method=POST path=/35232026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period35242026/11/11 23:40:03 INFO Received complete push request method=POST path=/35252026/11/11 23:40:03 INFO Completed push id=15 objects=7 duration=2.647705ms35262026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=035272026/11/11 23:40:03 INFO Completed push id=16 objects=4 duration=2.446603ms35282026/11/11 23:40:03 INFO Vacuumed table table=pending_closures35292026/11/11 23:40:03 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."35302026/11/11 23:40:03 INFO Vacuumed table table=pending_objects35312026/11/11 23:40:03 INFO Received complete push request method=POST path=/35322026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads35332026/11/11 23:40:03 INFO Vacuumed table table=closures35342026/11/11 23:40:03 INFO Received push request method=POST path=/35352026/11/11 23:40:03 INFO Completed push id=19 objects=7 duration=1.436834ms35362026/11/11 23:40:03 INFO Vacuumed table table=objects35372026/11/11 23:40:03 INFO Received push request method=POST path=/35382026/11/11 23:40:03 INFO Received uploads request method=POST path=/35392026/11/11 23:40:03 INFO Received uploads request method=POST path=/35402026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35412026/11/11 23:40:03 INFO Received push request method=POST path=/35422026/11/11 23:40:03 INFO Received uploads request method=POST path=/35432026/11/11 23:40:03 INFO Received push request method=POST path=/35442026/11/11 23:40:03 INFO Received push request method=POST path=/35452026/11/11 23:40:03 INFO Received uploads request method=POST path=/35462026/11/11 23:40:03 INFO Received complete push request method=POST path=/35472026/11/11 23:40:03 INFO Completed push id=15 objects=4 duration=2.663505ms35482026/11/11 23:40:03 INFO Received uploads request method=POST path=/35492026/11/11 23:40:03 INFO Received push request method=POST path=/35502026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35512026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35522026/11/11 23:40:03 INFO Completed upload id=20 objects=5 duration=2.239461ms35532026/11/11 23:40:03 INFO Completed upload id=14 objects=8 duration=2.184541ms35542026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=035552026/11/11 23:40:03 INFO Received push request method=POST path=/35562026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period35572026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35582026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035592026/11/11 23:40:03 INFO Vacuumed table table=pending_closures35602026/11/11 23:40:03 INFO Completed upload id=16 objects=23 duration=1.921978ms35612026/11/11 23:40:03 INFO Vacuumed table table=pending_objects35622026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads35632026/11/11 23:40:03 INFO Vacuumed table table=closures35642026/11/11 23:40:03 INFO Vacuumed table table=objects35652026/11/11 23:40:03 INFO Received uploads request method=POST path=/35662026/11/11 23:40:03 INFO Received complete push request method=POST path=/35672026/11/11 23:40:03 INFO Received push request method=POST path=/35682026/11/11 23:40:03 INFO Received uploads request method=POST path=/35692026/11/11 23:40:03 INFO Completed push id=14 objects=4 duration=2.663545ms35702026/11/11 23:40:03 INFO Received uploads request method=POST path=/35712026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35722026/11/11 23:40:03 INFO Completed upload id=15 objects=6 duration=878.068µs35732026/11/11 23:40:03 INFO Received push request method=POST path=/35742026/11/11 23:40:03 INFO Received complete push request method=POST path=/35752026/11/11 23:40:03 INFO Received uploads request method=POST path=/35762026/11/11 23:40:03 INFO Received push request method=POST path=/35772026/11/11 23:40:03 INFO Received uploads request method=POST path=/35782026/11/11 23:40:03 INFO Completed push id=17 objects=22 duration=2.253821ms35792026/11/11 23:40:03 INFO Received uploads request method=POST path=/35802026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35812026/11/11 23:40:03 INFO Received push request method=POST path=/35822026/11/11 23:40:03 INFO Received uploads request method=POST path=/35832026/11/11 23:40:03 INFO Received complete push request method=POST path=/35842026/11/11 23:40:03 INFO Completed upload id=22 objects=2 duration=1.714657ms35852026/11/11 23:40:03 INFO Received push request method=POST path=/35862026/11/11 23:40:03 INFO Completed push id=15 objects=24 duration=4.18504ms35872026/11/11 23:40:03 INFO Received push request method=POST path=/35882026/11/11 23:40:03 INFO Received complete push request method=POST path=/35892026/11/11 23:40:03 INFO Completed push id=14 objects=3 duration=1.200811ms35902026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=035912026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35922026/11/11 23:40:03 INFO Received complete upload request method=POST path=/35932026/11/11 23:40:03 INFO Received uploads request method=POST path=/35942026/11/11 23:40:03 INFO Completed upload id=17 objects=7 duration=1.257932ms35952026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=035962026/11/11 23:40:03 INFO Completed upload id=12 objects=26 duration=2.691865ms35972026/11/11 23:40:03 INFO Received push request method=POST path=/35982026/11/11 23:40:03 INFO Received uploads request method=POST path=/35992026/11/11 23:40:03 INFO Received complete push request method=POST path=/36002026/11/11 23:40:03 INFO Completed push id=4 objects=6 duration=1.820478ms36012026/11/11 23:40:03 INFO Received push request method=POST path=/36022026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036032026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036042026/11/11 23:40:03 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36052026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36062026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period36072026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period36082026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036092026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036102026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36112026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036122026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36132026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36142026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36152026/11/11 23:40:03 INFO Received uploads request method=POST path=/36162026/11/11 23:40:03 INFO Vacuumed table table=closures36172026/11/11 23:40:03 INFO Received uploads request method=POST path=/36182026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36192026/11/11 23:40:03 INFO Vacuumed table table=objects36202026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36212026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036222026/11/11 23:40:03 INFO Vacuumed table table=closures36232026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36242026/11/11 23:40:03 INFO Vacuumed table table=objects36252026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36262026/11/11 23:40:03 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36272026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36282026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036292026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36302026/11/11 23:40:03 INFO Vacuumed table table=closures36312026/11/11 23:40:03 INFO Vacuumed table table=objects36322026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036332026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36342026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36352026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36362026/11/11 23:40:03 INFO Received uploads request method=POST path=/36372026/11/11 23:40:03 INFO Vacuumed table table=closures36382026/11/11 23:40:03 INFO Received uploads request method=POST path=/36392026/11/11 23:40:03 INFO Vacuumed table table=objects36402026/11/11 23:40:03 INFO Received complete upload request method=POST path=/36412026/11/11 23:40:03 INFO Received complete upload request method=POST path=/36422026/11/11 23:40:03 INFO Completed upload id=20 objects=4 duration=1.01235ms36432026/11/11 23:40:03 INFO Completed upload id=7 objects=10 duration=1.611396ms36442026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036452026/11/11 23:40:03 INFO Received push request method=POST path=/36462026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period36472026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036482026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36492026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36502026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36512026/11/11 23:40:03 INFO Vacuumed table table=closures36522026/11/11 23:40:03 INFO Vacuumed table table=objects36532026/11/11 23:40:03 INFO Received push request method=POST path=/36542026/11/11 23:40:03 INFO Received uploads request method=POST path=/36552026/11/11 23:40:03 INFO Received complete push request method=POST path=/36562026/11/11 23:40:03 INFO Received uploads request method=POST path=/36572026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036582026/11/11 23:40:03 INFO Completed push id=16 objects=8 duration=1.902358ms36592026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036602026/11/11 23:40:03 INFO Received uploads request method=POST path=/36612026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036622026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period36632026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period36642026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period36652026/11/11 23:40:03 INFO Received push request method=POST path=/36662026/11/11 23:40:03 INFO Received complete upload request method=POST path=/36672026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036682026/11/11 23:40:03 INFO Completed upload id=14 objects=6 duration=1.06005ms36692026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36702026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36712026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36722026/11/11 23:40:03 INFO Received complete push request method=POST path=/36732026/11/11 23:40:03 INFO Vacuumed table table=closures36742026/11/11 23:40:03 INFO Vacuumed table table=objects36752026/11/11 23:40:03 INFO Completed push id=16 objects=5 duration=1.361933ms36762026/11/11 23:40:03 INFO Received uploads request method=POST path=/36772026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=036782026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36792026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36802026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36812026/11/11 23:40:03 INFO Received complete upload request method=POST path=/36822026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036832026/11/11 23:40:03 INFO Vacuumed table table=closures36842026/11/11 23:40:03 INFO Received push request method=POST path=/36852026/11/11 23:40:03 INFO Completed upload id=6 objects=6 duration=1.237992ms36862026/11/11 23:40:03 INFO Vacuumed table table=objects36872026/11/11 23:40:03 INFO Received push request method=POST path=/36882026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period36892026/11/11 23:40:03 INFO Received uploads request method=POST path=/36902026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036912026/11/11 23:40:03 INFO Received complete push request method=POST path=/36922026/11/11 23:40:03 INFO Vacuumed table table=pending_closures36932026/11/11 23:40:03 INFO Received complete upload request method=POST path=/36942026/11/11 23:40:03 INFO Vacuumed table table=pending_objects36952026/11/11 23:40:03 INFO Completed push id=14 objects=3 duration=3.420213ms36962026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads36972026/11/11 23:40:03 INFO Vacuumed table table=closures36982026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=036992026/11/11 23:40:03 INFO Vacuumed table table=objects37002026/11/11 23:40:03 INFO Completed upload id=16 objects=17 duration=2.727106ms37012026/11/11 23:40:03 INFO Received push request method=POST path=/37022026/11/11 23:40:03 INFO Received complete upload request method=POST path=/37032026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037042026/11/11 23:40:03 INFO Received push request method=POST path=/37052026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037062026/11/11 23:40:03 INFO Vacuumed table table=pending_closures37072026/11/11 23:40:03 INFO Completed upload id=21 objects=7 duration=2.704326ms37082026/11/11 23:40:03 INFO Received complete push request method=POST path=/37092026/11/11 23:40:03 INFO Vacuumed table table=pending_objects37102026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads37112026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period37122026/11/11 23:40:03 INFO Vacuumed table table=closures37132026/11/11 23:40:03 INFO Vacuumed table table=objects37142026/11/11 23:40:03 INFO Completed push id=16 objects=16 duration=3.436133ms37152026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037162026/11/11 23:40:03 INFO Received complete upload request method=POST path=/37172026/11/11 23:40:03 INFO Completed upload id=7 objects=3 duration=986.849µs37182026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037192026/11/11 23:40:03 INFO Received complete push request method=POST path=/37202026/11/11 23:40:03 INFO Received complete push request method=POST path=/37212026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period37222026/11/11 23:40:03 INFO Completed push id=21 objects=9 duration=1.930318ms37232026/11/11 23:40:03 INFO Completed push id=26 objects=14 duration=3.083869ms37242026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037252026/11/11 23:40:03 INFO Received complete push request method=POST path=/37262026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037272026/11/11 23:40:03 INFO Received push request method=POST path=/37282026/11/11 23:40:03 INFO Vacuumed table table=pending_closures37292026/11/11 23:40:03 INFO Completed push id=11 objects=6 duration=3.267611ms37302026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037312026/11/11 23:40:03 INFO Vacuumed table table=pending_objects37322026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads37332026/11/11 23:40:03 INFO Vacuumed table table=closures37342026/11/11 23:40:03 INFO Vacuumed table table=pending_closures37352026/11/11 23:40:03 INFO Received complete upload request method=POST path=/37362026/11/11 23:40:03 INFO Vacuumed table table=pending_objects37372026/11/11 23:40:03 INFO Completed upload id=11 objects=7 duration=1.213392ms37382026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads37392026/11/11 23:40:03 INFO Vacuumed table table=closures37402026/11/11 23:40:03 INFO Vacuumed table table=objects37412026/11/11 23:40:03 INFO Vacuumed table table=objects37422026/11/11 23:40:03 INFO Received push request method=POST path=/37432026/11/11 23:40:03 INFO Received complete push request method=POST path=/37442026/11/11 23:40:03 INFO Received push request method=POST path=/37452026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037462026/11/11 23:40:03 INFO Completed push id=18 objects=23 duration=2.126661ms37472026/11/11 23:40:03 INFO Received uploads request method=POST path=/37482026/11/11 23:40:03 INFO Received complete push request method=POST path=/37492026/11/11 23:40:03 INFO Received push request method=POST path=/37502026/11/11 23:40:03 INFO Completed push id=18 objects=6 duration=1.228052ms37512026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037522026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037532026/11/11 23:40:03 INFO Vacuumed table table=pending_closures37542026/11/11 23:40:03 INFO Received uploads request method=POST path=/37552026/11/11 23:40:03 INFO Vacuumed table table=pending_objects37562026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads37572026/11/11 23:40:03 INFO Received uploads request method=POST path=/37582026/11/11 23:40:03 INFO Received uploads request method=POST path=/37592026/11/11 23:40:03 INFO Vacuumed table table=closures37602026/11/11 23:40:03 INFO Vacuumed table table=objects37612026/11/11 23:40:03 INFO Received push request method=POST path=/37622026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."37632026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."37642026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."37652026/11/11 23:40:03 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."37662026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."37672026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."37682026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37692026/11/11 23:40:03 INFO Received push request method=POST path=/37702026/11/11 23:40:03 INFO Received uploads request method=POST path=/37712026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037722026/11/11 23:40:03 INFO Received uploads request method=POST path=/37732026/11/11 23:40:03 INFO Received uploads request method=POST path=/37742026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037752026/11/11 23:40:03 INFO Received uploads request method=POST path=/37762026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037772026/11/11 23:40:03 INFO Received complete push request method=POST path=/37782026/11/11 23:40:03 INFO Vacuumed table table=pending_closures37792026/11/11 23:40:03 INFO Completed push id=11 objects=9 duration=1.930439ms37802026/11/11 23:40:03 INFO Vacuumed table table=pending_objects37812026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads37822026/11/11 23:40:03 INFO Vacuumed table table=closures37832026/11/11 23:40:03 INFO Received uploads request method=POST path=/37842026/11/11 23:40:03 INFO Vacuumed table table=objects37852026/11/11 23:40:03 INFO Received uploads request method=POST path=/37862026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=037872026/11/11 23:40:03 INFO Vacuumed table table=pending_closures37882026/11/11 23:40:03 INFO Vacuumed table table=pending_objects37892026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads37902026/11/11 23:40:03 INFO Received uploads request method=POST path=/37912026/11/11 23:40:03 INFO Vacuumed table table=closures37922026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=037932026/11/11 23:40:03 INFO Received uploads request method=POST path=/37942026/11/11 23:40:03 INFO Vacuumed table table=objects37952026/11/11 23:40:03 INFO Received push request method=POST path=/37962026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period37972026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037982026/11/11 23:40:03 INFO Received push request method=POST path=/37992026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38002026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38012026/11/11 23:40:03 INFO Received complete upload request method=POST path=/38022026/11/11 23:40:03 INFO Received push request method=POST path=/38032026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38042026/11/11 23:40:03 INFO Vacuumed table table=closures38052026/11/11 23:40:03 INFO Completed upload id=27 objects=4 duration=1.193071ms38062026/11/11 23:40:03 INFO Vacuumed table table=objects38072026/11/11 23:40:03 INFO Received push request method=POST path=/38082026/11/11 23:40:03 INFO Received push request method=POST path=/38092026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038102026/11/11 23:40:03 INFO Received push request method=POST path=/38112026/11/11 23:40:03 INFO Received complete push request method=POST path=/38122026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038132026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38142026/11/11 23:40:03 INFO Completed push id=13 objects=11 duration=1.899838ms38152026/11/11 23:40:03 INFO Received push request method=POST path=/38162026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38172026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38182026/11/11 23:40:03 INFO Vacuumed table table=closures38192026/11/11 23:40:03 INFO Received uploads request method=POST path=/38202026/11/11 23:40:03 INFO Vacuumed table table=objects38212026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038222026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period38232026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038242026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38252026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38262026/11/11 23:40:03 INFO Received uploads request method=POST path=/38272026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38282026/11/11 23:40:03 INFO Vacuumed table table=closures38292026/11/11 23:40:03 INFO Vacuumed table table=objects38302026/11/11 23:40:03 INFO Received push request method=POST path=/38312026/11/11 23:40:03 INFO Received uploads request method=POST path=/38322026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038332026/11/11 23:40:03 INFO Received push request method=POST path=/38342026/11/11 23:40:03 INFO Received push request method=POST path=/38352026/11/11 23:40:03 INFO Received uploads request method=POST path=/38362026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038372026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38382026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38392026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38402026/11/11 23:40:03 INFO Received complete upload request method=POST path=/38412026/11/11 23:40:03 INFO Vacuumed table table=closures38422026/11/11 23:40:03 INFO Vacuumed table table=objects38432026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038442026/11/11 23:40:03 INFO Completed upload id=15 objects=3 duration=3.710076ms38452026/11/11 23:40:03 INFO Received push request method=POST path=/38462026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period38472026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038482026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038492026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38502026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period38512026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038522026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38532026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38542026/11/11 23:40:03 INFO Received push request method=POST path=/38552026/11/11 23:40:03 INFO Vacuumed table table=closures38562026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period38572026/11/11 23:40:03 INFO Vacuumed table table=objects38582026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038592026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38602026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38612026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38622026/11/11 23:40:03 INFO Vacuumed table table=closures38632026/11/11 23:40:03 INFO Vacuumed table table=objects38642026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=038652026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038662026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038672026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38682026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38692026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38702026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38712026/11/11 23:40:03 INFO Vacuumed table table=closures38722026/11/11 23:40:03 INFO Vacuumed table table=pending_closures38732026/11/11 23:40:03 INFO Vacuumed table table=objects38742026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38752026/11/11 23:40:03 INFO Received push request method=POST path=/38762026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38772026/11/11 23:40:03 INFO Vacuumed table table=pending_objects38782026/11/11 23:40:03 INFO Received push request method=POST path=/38792026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads38802026/11/11 23:40:03 INFO Received complete push request method=POST path=/38812026/11/11 23:40:03 INFO Vacuumed table table=closures38822026/11/11 23:40:03 INFO Vacuumed table table=closures38832026/11/11 23:40:03 INFO Received uploads request method=POST path=/38842026/11/11 23:40:03 INFO Received complete push request method=POST path=/38852026/11/11 23:40:03 INFO Vacuumed table table=objects38862026/11/11 23:40:03 INFO Completed push id=31 objects=14 duration=3.202511ms38872026/11/11 23:40:03 INFO Vacuumed table table=objects38882026/11/11 23:40:03 INFO Completed push id=13 objects=15 duration=2.09874ms38892026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038902026/11/11 23:40:03 INFO Received push request method=POST path=/38912026/11/11 23:40:03 INFO Received push request method=POST path=/38922026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period38932026/11/11 23:40:03 INFO Received uploads request method=POST path=/38942026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=038952026/11/11 23:40:03 INFO Received uploads request method=POST path=/38962026/11/11 23:40:03 INFO Received push request method=POST path=/38972026/11/11 23:40:03 INFO Received uploads request method=POST path=/38982026/11/11 23:40:03 INFO Received uploads request method=POST path=/38992026/11/11 23:40:03 INFO Received push request method=POST path=/39002026/11/11 23:40:03 INFO Received push request method=POST path=/39012026/11/11 23:40:03 INFO Received complete upload request method=POST path=/39022026/11/11 23:40:03 INFO Received push request method=POST path=/39032026/11/11 23:40:03 INFO Completed upload id=13 objects=8 duration=1.326112ms39042026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."39052026/11/11 23:40:03 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."39062026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."39072026/11/11 23:40:03 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."39082026/11/11 23:40:03 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."39092026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."39102026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."39112026/11/11 23:40:03 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39122026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=039132026/11/11 23:40:03 INFO Received complete upload request method=POST path=/39142026/11/11 23:40:03 INFO Received push request method=POST path=/39152026/11/11 23:40:03 INFO Received complete push request method=POST path=/39162026/11/11 23:40:03 INFO Completed upload id=23 objects=15 duration=2.07978ms39172026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039182026/11/11 23:40:03 INFO Vacuumed table table=pending_closures39192026/11/11 23:40:03 INFO Completed push id=17 objects=22 duration=2.02962ms39202026/11/11 23:40:03 INFO Vacuumed table table=pending_objects39212026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads39222026/11/11 23:40:03 INFO Vacuumed table table=closures39232026/11/11 23:40:03 INFO Received complete push request method=POST path=/39242026/11/11 23:40:03 INFO Received uploads request method=POST path=/39252026/11/11 23:40:03 INFO Vacuumed table table=objects39262026/11/11 23:40:03 INFO Completed push id=27 objects=25 duration=3.385612ms39272026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=039282026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period39292026/11/11 23:40:03 INFO Received uploads request method=POST path=/39302026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039312026/11/11 23:40:03 INFO Received uploads request method=POST path=/39322026/11/11 23:40:03 INFO Vacuumed table table=pending_closures39332026/11/11 23:40:03 INFO Vacuumed table table=pending_objects39342026/11/11 23:40:03 INFO Received push request method=POST path=/39352026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads39362026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=039372026/11/11 23:40:03 INFO Vacuumed table table=closures39382026/11/11 23:40:03 INFO Received complete upload request method=POST path=/39392026/11/11 23:40:03 INFO Received complete push request method=POST path=/39402026/11/11 23:40:03 INFO Vacuumed table table=objects39412026/11/11 23:40:03 INFO Received uploads request method=POST path=/39422026/11/11 23:40:03 INFO Completed upload id=21 objects=3 duration=1.00485ms39432026/11/11 23:40:03 INFO Completed push id=7 objects=7 duration=1.528714ms39442026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039452026/11/11 23:40:03 INFO Vacuumed table table=pending_closures39462026/11/11 23:40:03 INFO Vacuumed table table=pending_objects39472026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads39482026/11/11 23:40:03 INFO Received complete push request method=POST path=/39492026/11/11 23:40:03 INFO Vacuumed table table=closures39502026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=039512026/11/11 23:40:03 INFO Completed push id=20 objects=3 duration=1.606155ms39522026/11/11 23:40:03 INFO Vacuumed table table=objects39532026/11/11 23:40:03 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039542026/11/11 23:40:03 INFO Vacuumed table table=pending_closures39552026/11/11 23:40:03 INFO Received complete upload request method=POST path=/39562026/11/11 23:40:03 INFO Received complete upload request method=POST path=/39572026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=039582026/11/11 23:40:03 INFO Vacuumed table table=pending_objects39592026/11/11 23:40:03 INFO Vacuumed table table=multipart_uploads39602026/11/11 23:40:03 INFO Received push request method=POST path=/39612026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period39622026/11/11 23:40:03 INFO Vacuumed table table=closures39632026/11/11 23:40:03 INFO Completed upload id=19 objects=6 duration=1.358093ms39642026/11/11 23:40:03 INFO Received push request method=POST path=/39652026/11/11 23:40:03 INFO Vacuumed table table=objects39662026/11/11 23:40:03 INFO Completed upload id=33 objects=5 duration=2.502904ms39672026/11/11 23:40:03 INFO Received push request method=POST path=/39682026/11/11 23:40:03 INFO Received complete push request method=POST path=/39692026/11/11 23:40:03 INFO Completed push id=12 objects=6 duration=1.925398ms39702026/11/11 23:40:03 INFO Aborted multipart uploads count=0 kept=039712026/11/11 23:40:03 WARN Force mode enabled - objects will be deleted immediately without grace period39722026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039732026/11/11 23:40:04 INFO Vacuumed table table=pending_closures39742026/11/11 23:40:04 INFO Vacuumed table table=pending_objects39752026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads39762026/11/11 23:40:04 INFO Vacuumed table table=closures39772026/11/11 23:40:04 INFO Vacuumed table table=objects39782026/11/11 23:40:04 INFO Received uploads request method=POST path=/39792026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=039802026/11/11 23:40:04 INFO Received complete push request method=POST path=/39812026/11/11 23:40:04 INFO Vacuumed table table=pending_closures39822026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=039832026/11/11 23:40:04 INFO Received complete upload request method=POST path=/39842026/11/11 23:40:04 INFO Completed upload id=22 objects=7 duration=1.112811ms39852026/11/11 23:40:04 INFO Vacuumed table table=pending_objects39862026/11/11 23:40:04 INFO Completed push id=24 objects=5 duration=2.732607ms39872026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads39882026/11/11 23:40:04 INFO Vacuumed table table=closures39892026/11/11 23:40:04 INFO Vacuumed table table=objects39902026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039912026/11/11 23:40:04 INFO Received complete upload request method=POST path=/39922026/11/11 23:40:04 INFO Vacuumed table table=pending_closures39932026/11/11 23:40:04 INFO Vacuumed table table=pending_objects39942026/11/11 23:40:04 INFO Completed upload id=19 objects=2 duration=2.841567ms39952026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads39962026/11/11 23:40:04 INFO Vacuumed table table=closures39972026/11/11 23:40:04 INFO Vacuumed table table=objects39982026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=039992026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period40002026/11/11 23:40:04 INFO Received push request method=POST path=/40012026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040022026/11/11 23:40:04 INFO Vacuumed table table=pending_closures40032026/11/11 23:40:04 INFO Vacuumed table table=pending_objects40042026/11/11 23:40:04 INFO Received uploads request method=POST path=/40052026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads40062026/11/11 23:40:04 INFO Vacuumed table table=closures40072026/11/11 23:40:04 INFO Vacuumed table table=objects40082026/11/11 23:40:04 INFO Received push request method=POST path=/40092026/11/11 23:40:04 INFO Received uploads request method=POST path=/40102026/11/11 23:40:04 INFO Received complete push request method=POST path=/40112026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40122026/11/11 23:40:04 INFO Received complete push request method=POST path=/40132026/11/11 23:40:04 INFO Completed push id=5 objects=28 duration=2.123341ms40142026/11/11 23:40:04 INFO Received push request method=POST path=/40152026/11/11 23:40:04 INFO Completed push id=13 objects=22 duration=1.410393ms40162026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=040172026/11/11 23:40:04 INFO Completed upload id=21 objects=25 duration=2.08828ms40182026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period40192026/11/11 23:40:04 INFO Received complete push request method=POST path=/40202026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040212026/11/11 23:40:04 INFO Vacuumed table table=pending_closures40222026/11/11 23:40:04 INFO Received uploads request method=POST path=/40232026/11/11 23:40:04 INFO Vacuumed table table=pending_objects40242026/11/11 23:40:04 INFO Completed push id=19 objects=14 duration=1.365813ms40252026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads40262026/11/11 23:40:04 INFO Vacuumed table table=closures40272026/11/11 23:40:04 INFO Vacuumed table table=objects40282026/11/11 23:40:04 INFO Received complete push request method=POST path=/40292026/11/11 23:40:04 INFO Completed push id=33 objects=7 duration=1.894438ms40302026/11/11 23:40:04 INFO Received uploads request method=POST path=/40312026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=040322026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period40332026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040342026/11/11 23:40:04 INFO Received complete push request method=POST path=/40352026/11/11 23:40:04 INFO Received push request method=POST path=/40362026/11/11 23:40:04 INFO Received complete push request method=POST path=/40372026/11/11 23:40:04 INFO Vacuumed table table=pending_closures40382026/11/11 23:40:04 INFO Completed push id=21 objects=6 duration=1.08349ms40392026/11/11 23:40:04 INFO Received uploads request method=POST path=/40402026/11/11 23:40:04 INFO Vacuumed table table=pending_objects40412026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads40422026/11/11 23:40:04 INFO Vacuumed table table=closures40432026/11/11 23:40:04 INFO Completed push id=10 objects=2 duration=2.476623ms40442026/11/11 23:40:04 INFO Vacuumed table table=objects40452026/11/11 23:40:04 INFO Received complete push request method=POST path=/40462026/11/11 23:40:04 INFO Received uploads request method=POST path=/40472026/11/11 23:40:04 INFO Completed push id=24 objects=9 duration=2.008159ms40482026/11/11 23:40:04 INFO Received push request method=POST path=/40492026/11/11 23:40:04 INFO Received uploads request method=POST path=/40502026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40512026/11/11 23:40:04 INFO Completed upload id=26 objects=3 duration=1.10781ms40522026/11/11 23:40:04 INFO Received push request method=POST path=/40532026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40542026/11/11 23:40:04 INFO Completed upload id=22 objects=7 duration=1.257332ms40552026/11/11 23:40:04 INFO Received push request method=POST path=/40562026/11/11 23:40:04 INFO Received uploads request method=POST path=/40572026/11/11 23:40:04 INFO Received push request method=POST path=/40582026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=040592026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period40602026/11/11 23:40:04 INFO Received complete push request method=POST path=/40612026/11/11 23:40:04 INFO Received complete push request method=POST path=/40622026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40632026/11/11 23:40:04 INFO Received uploads request method=POST path=/40642026/11/11 23:40:04 INFO Completed push id=22 objects=3 duration=2.161761ms40652026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040662026/11/11 23:40:04 INFO Completed upload id=17 objects=5 duration=1.162911ms40672026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40682026/11/11 23:40:04 INFO Completed push id=21 objects=26 duration=2.911288ms40692026/11/11 23:40:04 INFO Completed upload id=20 objects=6 duration=1.231192ms40702026/11/11 23:40:04 INFO Vacuumed table table=pending_closures40712026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40722026/11/11 23:40:04 INFO Vacuumed table table=pending_objects40732026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads40742026/11/11 23:40:04 INFO Vacuumed table table=closures40752026/11/11 23:40:04 INFO Completed upload id=22 objects=6 duration=1.624296ms40762026/11/11 23:40:04 INFO Received complete push request method=POST path=/40772026/11/11 23:40:04 INFO Vacuumed table table=objects40782026/11/11 23:40:04 INFO Completed push id=24 objects=4 duration=833.908µs40792026/11/11 23:40:04 INFO Received uploads request method=POST path=/40802026/11/11 23:40:04 INFO Received push request method=POST path=/40812026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40822026/11/11 23:40:04 INFO Completed upload id=12 objects=6 duration=1.256952ms40832026/11/11 23:40:04 INFO Received uploads request method=POST path=/40842026/11/11 23:40:04 INFO Received complete push request method=POST path=/40852026/11/11 23:40:04 INFO Completed push id=25 objects=9 duration=946.469µs40862026/11/11 23:40:04 INFO Received complete push request method=POST path=/40872026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40882026/11/11 23:40:04 INFO Completed push id=19 objects=3 duration=1.110371ms40892026/11/11 23:40:04 INFO Completed upload id=28 objects=17 duration=1.339513ms40902026/11/11 23:40:04 INFO Received uploads request method=POST path=/40912026/11/11 23:40:04 INFO Received uploads request method=POST path=/40922026/11/11 23:40:04 INFO Received complete push request method=POST path=/40932026/11/11 23:40:04 INFO Completed push id=17 objects=3 duration=2.316362ms40942026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40952026/11/11 23:40:04 INFO Received complete upload request method=POST path=/40962026/11/11 23:40:04 INFO Received push request method=POST path=/40972026/11/11 23:40:04 INFO Completed upload id=23 objects=16 duration=2.0994ms40982026/11/11 23:40:04 INFO Completed upload id=26 objects=3 duration=976.249µs40992026/11/11 23:40:04 INFO Received complete push request method=POST path=/41002026/11/11 23:40:04 INFO Completed push id=29 objects=2 duration=1.10291ms41012026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41022026/11/11 23:40:04 INFO Received uploads request method=POST path=/41032026/11/11 23:40:04 INFO Completed upload id=23 objects=18 duration=2.405043ms41042026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41052026/11/11 23:40:04 INFO Received complete push request method=POST path=/41062026/11/11 23:40:04 INFO Completed push id=20 objects=4 duration=1.205812ms41072026/11/11 23:40:04 INFO Received uploads request method=POST path=/41082026/11/11 23:40:04 INFO Completed upload id=10 objects=6 duration=2.749146ms41092026/11/11 23:40:04 INFO Received complete push request method=POST path=/41102026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=041112026/11/11 23:40:04 INFO Completed push id=24 objects=3 duration=1.05269ms41122026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period41132026/11/11 23:40:04 INFO Received complete push request method=POST path=/41142026/11/11 23:40:04 INFO Completed push id=23 objects=5 duration=1.267532ms41152026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041162026/11/11 23:40:04 INFO Vacuumed table table=pending_closures41172026/11/11 23:40:04 INFO Received complete push request method=POST path=/41182026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41192026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41202026/11/11 23:40:04 INFO Completed push id=31 objects=6 duration=1.536595ms41212026/11/11 23:40:04 INFO Vacuumed table table=pending_objects41222026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads41232026/11/11 23:40:04 INFO Completed upload id=21 objects=2 duration=1.766217ms41242026/11/11 23:40:04 INFO Received uploads request method=POST path=/41252026/11/11 23:40:04 INFO Vacuumed table table=closures41262026/11/11 23:40:04 INFO Vacuumed table table=objects41272026/11/11 23:40:04 INFO Completed upload id=27 objects=7 duration=4.113619ms41282026/11/11 23:40:04 INFO Received push request method=POST path=/41292026/11/11 23:40:04 INFO Received push request method=POST path=/41302026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41312026/11/11 23:40:04 INFO Completed upload id=24 objects=3 duration=815.388µs41322026/11/11 23:40:04 INFO Received complete push request method=POST path=/41332026/11/11 23:40:04 INFO Received uploads request method=POST path=/41342026/11/11 23:40:04 INFO Completed push id=32 objects=9 duration=1.106251ms41352026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41362026/11/11 23:40:04 INFO Received push request method=POST path=/41372026/11/11 23:40:04 INFO Completed upload id=22 objects=8 duration=1.175591ms41382026/11/11 23:40:04 INFO Received uploads request method=POST path=/41392026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41402026/11/11 23:40:04 INFO Completed upload id=28 objects=3 duration=835.228µs41412026/11/11 23:40:04 INFO Received push request method=POST path=/41422026/11/11 23:40:04 INFO Received complete push request method=POST path=/41432026/11/11 23:40:04 INFO Completed push id=25 objects=4 duration=1.017609ms41442026/11/11 23:40:04 INFO Received push request method=POST path=/41452026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=041462026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period41472026/11/11 23:40:04 INFO Received complete push request method=POST path=/41482026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041492026/11/11 23:40:04 INFO Completed push id=23 objects=5 duration=866.648µs41502026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=041512026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41522026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41532026/11/11 23:40:04 INFO Vacuumed table table=pending_closures41542026/11/11 23:40:04 INFO Completed upload id=33 objects=9 duration=993.149µs41552026/11/11 23:40:04 INFO Completed upload id=22 objects=22 duration=1.351153ms41562026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period41572026/11/11 23:40:04 INFO Vacuumed table table=pending_objects41582026/11/11 23:40:04 INFO Received uploads request method=POST path=/41592026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads41602026/11/11 23:40:04 INFO Vacuumed table table=closures41612026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041622026/11/11 23:40:04 INFO Received complete push request method=POST path=/41632026/11/11 23:40:04 INFO Vacuumed table table=pending_closures41642026/11/11 23:40:04 INFO Vacuumed table table=objects41652026/11/11 23:40:04 INFO Received complete push request method=POST path=/41662026/11/11 23:40:04 INFO Vacuumed table table=pending_objects41672026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads41682026/11/11 23:40:04 INFO Completed push id=8 objects=6 duration=2.309002ms41692026/11/11 23:40:04 INFO Vacuumed table table=closures41702026/11/11 23:40:04 INFO Completed push id=29 objects=23 duration=1.272393ms41712026/11/11 23:40:04 INFO Vacuumed table table=objects41722026/11/11 23:40:04 INFO Received complete push request method=POST path=/41732026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41742026/11/11 23:40:04 INFO Completed push id=21 objects=3 duration=1.368873ms41752026/11/11 23:40:04 INFO Completed upload id=24 objects=8 duration=1.458694ms41762026/11/11 23:40:04 INFO Received complete push request method=POST path=/41772026/11/11 23:40:04 INFO Received push request method=POST path=/41782026/11/11 23:40:04 INFO Received uploads request method=POST path=/41792026/11/11 23:40:04 INFO Completed push id=9 objects=9 duration=3.11949ms41802026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=041812026/11/11 23:40:04 INFO Received complete push request method=POST path=/41822026/11/11 23:40:04 INFO Completed push id=30 objects=6 duration=1.00921ms41832026/11/11 23:40:04 INFO Received complete push request method=POST path=/41842026/11/11 23:40:04 INFO Completed push id=12 objects=7 duration=2.798707ms41852026/11/11 23:40:04 INFO Received uploads request method=POST path=/41862026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period41872026/11/11 23:40:04 INFO Received uploads request method=POST path=/41882026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41892026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041902026/11/11 23:40:04 INFO Received complete push request method=POST path=/41912026/11/11 23:40:04 INFO Completed upload id=28 objects=5 duration=1.148631ms41922026/11/11 23:40:04 INFO Received uploads request method=POST path=/41932026/11/11 23:40:04 INFO Vacuumed table table=pending_closures41942026/11/11 23:40:04 INFO Completed push id=26 objects=5 duration=1.776796ms41952026/11/11 23:40:04 INFO Received complete upload request method=POST path=/41962026/11/11 23:40:04 INFO Completed upload id=21 objects=8 duration=1.354113ms41972026/11/11 23:40:04 INFO Vacuumed table table=pending_objects41982026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads41992026/11/11 23:40:04 INFO Received complete push request method=POST path=/42002026/11/11 23:40:04 INFO Vacuumed table table=closures42012026/11/11 23:40:04 INFO Completed push id=26 objects=23 duration=1.587015ms42022026/11/11 23:40:04 INFO Vacuumed table table=objects42032026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=042042026/11/11 23:40:04 INFO Received complete push request method=POST path=/42052026/11/11 23:40:04 INFO Completed push id=33 objects=3 duration=1.391974ms42062026/11/11 23:40:04 INFO Received uploads request method=POST path=/42072026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42082026/11/11 23:40:04 INFO Received complete push request method=POST path=/42092026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=042102026/11/11 23:40:04 INFO Completed upload id=24 objects=14 duration=1.10971ms42112026/11/11 23:40:04 INFO Completed push id=22 objects=6 duration=1.02023ms42122026/11/11 23:40:04 INFO Received push request method=POST path=/42132026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42142026/11/11 23:40:04 INFO Completed upload id=30 objects=5 duration=965.929µs42152026/11/11 23:40:04 INFO Received complete push request method=POST path=/42162026/11/11 23:40:04 INFO Completed push id=34 objects=6 duration=1.09469ms42172026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=042182026/11/11 23:40:04 INFO Received complete push request method=POST path=/42192026/11/11 23:40:04 INFO Completed push id=24 objects=8 duration=1.249852ms42202026/11/11 23:40:04 INFO Received complete push request method=POST path=/42212026/11/11 23:40:04 INFO Received uploads request method=POST path=/42222026/11/11 23:40:04 INFO Completed push id=19 objects=3 duration=1.10049ms42232026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042242026/11/11 23:40:04 INFO Received complete push request method=POST path=/42252026/11/11 23:40:04 INFO Vacuumed table table=pending_closures42262026/11/11 23:40:04 INFO Received complete push request method=POST path=/42272026/11/11 23:40:04 INFO Completed push id=24 objects=24 duration=3.063869ms42282026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=042292026/11/11 23:40:04 INFO Vacuumed table table=pending_objects42302026/11/11 23:40:04 INFO Completed push id=27 objects=15 duration=1.192752ms42312026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads42322026/11/11 23:40:04 INFO Vacuumed table table=closures42332026/11/11 23:40:04 INFO Vacuumed table table=pending_closures42342026/11/11 23:40:04 INFO Vacuumed table table=objects42352026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=042362026/11/11 23:40:04 INFO Received complete push request method=POST path=/42372026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42382026/11/11 23:40:04 INFO Vacuumed table table=pending_objects42392026/11/11 23:40:04 INFO Vacuumed table table=pending_closures42402026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads42412026/11/11 23:40:04 INFO Completed push id=35 objects=8 duration=1.01185ms42422026/11/11 23:40:04 INFO Completed upload id=25 objects=9 duration=1.083751ms42432026/11/11 23:40:04 INFO Vacuumed table table=closures42442026/11/11 23:40:04 INFO Received complete push request method=POST path=/42452026/11/11 23:40:04 INFO Received complete push request method=POST path=/42462026/11/11 23:40:04 INFO Vacuumed table table=pending_objects42472026/11/11 23:40:04 INFO Vacuumed table table=objects42482026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads42492026/11/11 23:40:04 INFO Completed push id=31 objects=3 duration=1.08817ms42502026/11/11 23:40:04 INFO Vacuumed table table=closures42512026/11/11 23:40:04 INFO Vacuumed table table=objects42522026/11/11 23:40:04 INFO Completed push id=7 objects=15 duration=3.982118ms42532026/11/11 23:40:04 INFO Received push request method=POST path=/42542026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=042552026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period42562026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42572026/11/11 23:40:04 INFO Completed upload id=27 objects=3 duration=1.09411ms42582026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042592026/11/11 23:40:04 INFO Received push request method=POST path=/42602026/11/11 23:40:04 INFO Vacuumed table table=pending_closures42612026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42622026/11/11 23:40:04 INFO Completed upload id=36 objects=9 duration=2.04094ms42632026/11/11 23:40:04 INFO Vacuumed table table=pending_objects42642026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads42652026/11/11 23:40:04 INFO Vacuumed table table=closures42662026/11/11 23:40:04 INFO Vacuumed table table=objects42672026/11/11 23:40:04 INFO Received uploads request method=POST path=/42682026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042692026/11/11 23:40:04 INFO Vacuumed table table=pending_closures42702026/11/11 23:40:04 INFO Vacuumed table table=pending_objects42712026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads42722026/11/11 23:40:04 INFO Vacuumed table table=closures42732026/11/11 23:40:04 INFO Vacuumed table table=objects42742026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42752026/11/11 23:40:04 INFO Completed upload id=28 objects=6 duration=939.249µs42762026/11/11 23:40:04 INFO Received complete push request method=POST path=/42772026/11/11 23:40:04 INFO Completed push id=32 objects=6 duration=995.609µs42782026/11/11 23:40:04 INFO Received uploads request method=POST path=/42792026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=042802026/11/11 23:40:04 INFO Received push request method=POST path=/42812026/11/11 23:40:04 INFO Received push request method=POST path=/42822026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period42832026/11/11 23:40:04 INFO Received uploads request method=POST path=/42842026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042852026/11/11 23:40:04 INFO Received push request method=POST path=/42862026/11/11 23:40:04 INFO Vacuumed table table=pending_closures42872026/11/11 23:40:04 INFO Received complete push request method=POST path=/42882026/11/11 23:40:04 INFO Received complete push request method=POST path=/42892026/11/11 23:40:04 INFO Completed push id=30 objects=4 duration=967.429µs42902026/11/11 23:40:04 INFO Vacuumed table table=pending_objects42912026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads42922026/11/11 23:40:04 INFO Completed push id=28 objects=19 duration=1.389213ms42932026/11/11 23:40:04 INFO Vacuumed table table=closures42942026/11/11 23:40:04 INFO Received uploads request method=POST path=/42952026/11/11 23:40:04 INFO Vacuumed table table=objects42962026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42972026/11/11 23:40:04 INFO Received complete upload request method=POST path=/42982026/11/11 23:40:04 INFO Completed upload id=33 objects=7 duration=787.607µs42992026/11/11 23:40:04 INFO Completed upload id=26 objects=2 duration=1.788517ms43002026/11/11 23:40:04 INFO Received push request method=POST path=/43012026/11/11 23:40:04 INFO Received complete push request method=POST path=/43022026/11/11 23:40:04 INFO Received uploads request method=POST path=/43032026/11/11 23:40:04 INFO Received uploads request method=POST path=/43042026/11/11 23:40:04 INFO Received complete push request method=POST path=/43052026/11/11 23:40:04 INFO Completed push id=30 objects=4 duration=2.961628ms43062026/11/11 23:40:04 INFO Received complete push request method=POST path=/43072026/11/11 23:40:04 INFO Completed push id=31 objects=7 duration=1.10777ms43082026/11/11 23:40:04 INFO Completed push id=8 objects=6 duration=2.834947ms43092026/11/11 23:40:04 INFO Received uploads request method=POST path=/43102026/11/11 23:40:04 INFO Received complete push request method=POST path=/43112026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43122026/11/11 23:40:04 INFO Completed push id=16 objects=7 duration=1.837038ms43132026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=043142026/11/11 23:40:04 INFO Completed upload id=34 objects=3 duration=862.029µs43152026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043162026/11/11 23:40:04 INFO Vacuumed table table=pending_closures43172026/11/11 23:40:04 INFO Received complete push request method=POST path=/43182026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43192026/11/11 23:40:04 INFO Received uploads request method=POST path=/43202026/11/11 23:40:04 INFO Completed upload id=32 objects=5 duration=984.029µs43212026/11/11 23:40:04 INFO Completed push id=29 objects=10 duration=1.266592ms43222026/11/11 23:40:04 INFO Vacuumed table table=pending_objects43232026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads43242026/11/11 23:40:04 INFO Vacuumed table table=closures43252026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43262026/11/11 23:40:04 INFO Vacuumed table table=objects43272026/11/11 23:40:04 INFO Received uploads request method=POST path=/43282026/11/11 23:40:04 INFO Completed upload id=25 objects=4 duration=1.02505ms43292026/11/11 23:40:04 INFO Received complete push request method=POST path=/43302026/11/11 23:40:04 INFO Completed push id=35 objects=11 duration=1.08323ms43312026/11/11 23:40:04 INFO Received push request method=POST path=/43322026/11/11 23:40:04 INFO Received uploads request method=POST path=/43332026/11/11 23:40:04 INFO Received complete push request method=POST path=/43342026/11/11 23:40:04 INFO Received push request method=POST path=/43352026/11/11 23:40:04 INFO Completed push id=23 objects=6 duration=1.318353ms43362026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43372026/11/11 23:40:04 INFO Completed upload id=30 objects=6 duration=893.669µs43382026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=043392026/11/11 23:40:04 INFO Received push request method=POST path=/43402026/11/11 23:40:04 INFO Received push request method=POST path=/43412026/11/11 23:40:04 INFO Received uploads request method=POST path=/43422026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period43432026/11/11 23:40:04 INFO Received push request method=POST path=/43442026/11/11 23:40:04 INFO Received complete push request method=POST path=/43452026/11/11 23:40:04 INFO Received push request method=POST path=/43462026/11/11 23:40:04 INFO Completed push id=24 objects=6 duration=1.341652ms43472026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43482026/11/11 23:40:04 INFO Completed upload id=31 objects=5 duration=928.469µs43492026/11/11 23:40:04 INFO Received push request method=POST path=/43502026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43512026/11/11 23:40:04 INFO Completed upload id=34 objects=24 duration=1.00763ms43522026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=043532026/11/11 23:40:04 INFO Received uploads request method=POST path=/43542026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period43552026/11/11 23:40:04 INFO Received complete push request method=POST path=/43562026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43572026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043582026/11/11 23:40:04 INFO Completed push id=32 objects=7 duration=1.180751ms43592026/11/11 23:40:04 INFO Completed upload id=34 objects=3 duration=827.608µs43602026/11/11 23:40:04 INFO Vacuumed table table=pending_closures43612026/11/11 23:40:04 INFO Vacuumed table table=pending_objects43622026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads43632026/11/11 23:40:04 INFO Vacuumed table table=closures43642026/11/11 23:40:04 INFO Vacuumed table table=objects43652026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=043662026/11/11 23:40:04 INFO Received complete push request method=POST path=/43672026/11/11 23:40:04 INFO Completed push id=25 objects=7 duration=1.290172ms43682026/11/11 23:40:04 INFO Received complete push request method=POST path=/43692026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=043702026/11/11 23:40:04 INFO Received complete push request method=POST path=/43712026/11/11 23:40:04 INFO Completed push id=20 objects=8 duration=963.829µs43722026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period43732026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043742026/11/11 23:40:04 INFO Completed push id=19 objects=14 duration=3.20933ms43752026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=043762026/11/11 23:40:04 INFO Vacuumed table table=pending_closures43772026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period43782026/11/11 23:40:04 INFO Vacuumed table table=pending_objects43792026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads43802026/11/11 23:40:04 INFO Vacuumed table table=closures43812026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043822026/11/11 23:40:04 INFO Vacuumed table table=objects43832026/11/11 23:40:04 INFO Vacuumed table table=pending_closures43842026/11/11 23:40:04 INFO Received complete push request method=POST path=/43852026/11/11 23:40:04 INFO Vacuumed table table=pending_objects43862026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads43872026/11/11 23:40:04 INFO Vacuumed table table=closures43882026/11/11 23:40:04 INFO Completed push id=26 objects=4 duration=2.000259ms43892026/11/11 23:40:04 INFO Vacuumed table table=objects43902026/11/11 23:40:04 INFO Received push request method=POST path=/43912026/11/11 23:40:04 INFO Received uploads request method=POST path=/43922026/11/11 23:40:04 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43932026/11/11 23:40:04 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."43942026/11/11 23:40:04 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43952026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=043962026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period43972026/11/11 23:40:04 INFO Received complete upload request method=POST path=/43982026/11/11 23:40:04 INFO Completed upload id=28 objects=13 duration=968.909µs43992026/11/11 23:40:04 INFO Received push request method=POST path=/44002026/11/11 23:40:04 INFO Received push request method=POST path=/44012026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=044022026/11/11 23:40:04 INFO Received push request method=POST path=/44032026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period44042026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44052026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044062026/11/11 23:40:04 INFO Completed upload id=26 objects=24 duration=1.245232ms44072026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44082026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44092026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44102026/11/11 23:40:04 INFO Vacuumed table table=closures44112026/11/11 23:40:04 INFO Vacuumed table table=objects44122026/11/11 23:40:04 INFO Received complete push request method=POST path=/44132026/11/11 23:40:04 INFO Completed push id=29 objects=3 duration=941.509µs44142026/11/11 23:40:04 INFO Received complete push request method=POST path=/44152026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44162026/11/11 23:40:04 INFO Completed push id=9 objects=24 duration=1.577715ms44172026/11/11 23:40:04 INFO Completed upload id=23 objects=7 duration=2.417923ms44182026/11/11 23:40:04 INFO Received push request method=POST path=/44192026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44202026/11/11 23:40:04 INFO Completed upload id=31 objects=2 duration=780.687µs44212026/11/11 23:40:04 INFO Received complete push request method=POST path=/44222026/11/11 23:40:04 INFO Completed push id=27 objects=11 duration=897.269µs44232026/11/11 23:40:04 INFO Received complete push request method=POST path=/44242026/11/11 23:40:04 INFO Received uploads request method=POST path=/44252026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44262026/11/11 23:40:04 INFO Completed upload id=26 objects=5 duration=941.649µs44272026/11/11 23:40:04 INFO Completed push id=19 objects=8 duration=6.593823ms44282026/11/11 23:40:04 INFO Received complete push request method=POST path=/44292026/11/11 23:40:04 INFO Received complete push request method=POST path=/44302026/11/11 23:40:04 INFO Completed push id=33 objects=6 duration=900.588µs44312026/11/11 23:40:04 INFO Completed push id=28 objects=11 duration=2.528264ms44322026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44332026/11/11 23:40:04 INFO Completed upload id=29 objects=26 duration=881.888µs44342026/11/11 23:40:04 INFO Received complete push request method=POST path=/44352026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44362026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=044372026/11/11 23:40:04 INFO Completed push id=27 objects=3 duration=1.716636ms44382026/11/11 23:40:04 INFO Received uploads request method=POST path=/44392026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44402026/11/11 23:40:04 INFO Received uploads request method=POST path=/44412026/11/11 23:40:04 INFO Completed upload id=27 objects=26 duration=1.292072ms44422026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44432026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44442026/11/11 23:40:04 INFO Vacuumed table table=closures44452026/11/11 23:40:04 INFO Vacuumed table table=objects44462026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=044472026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044482026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44492026/11/11 23:40:04 INFO Received complete push request method=POST path=/44502026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period44512026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44522026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44532026/11/11 23:40:04 INFO Completed push id=30 objects=8 duration=872.288µs44542026/11/11 23:40:04 INFO Vacuumed table table=closures44552026/11/11 23:40:04 INFO Vacuumed table table=objects44562026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044572026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44582026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44592026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44602026/11/11 23:40:04 INFO Completed upload id=28 objects=5 duration=866.088µs44612026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44622026/11/11 23:40:04 INFO Vacuumed table table=closures44632026/11/11 23:40:04 INFO Received complete push request method=POST path=/44642026/11/11 23:40:04 INFO Vacuumed table table=objects44652026/11/11 23:40:04 INFO Completed push id=28 objects=6 duration=1.08905ms44662026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=044672026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44682026/11/11 23:40:04 INFO Completed upload id=10 objects=10 duration=1.151131ms44692026/11/11 23:40:04 INFO Received push request method=POST path=/44702026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044712026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44722026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044732026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044742026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044752026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44762026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44772026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44782026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44792026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44802026/11/11 23:40:04 INFO Vacuumed table table=closures44812026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44822026/11/11 23:40:04 INFO Vacuumed table table=pending_closures44832026/11/11 23:40:04 INFO Received complete upload request method=POST path=/44842026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44852026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44862026/11/11 23:40:04 INFO Completed upload id=31 objects=5 duration=1.07587ms44872026/11/11 23:40:04 INFO Vacuumed table table=closures44882026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44892026/11/11 23:40:04 INFO Vacuumed table table=objects44902026/11/11 23:40:04 INFO Vacuumed table table=closures44912026/11/11 23:40:04 INFO Vacuumed table table=objects44922026/11/11 23:40:04 INFO Vacuumed table table=pending_objects44932026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads44942026/11/11 23:40:04 INFO Vacuumed table table=objects44952026/11/11 23:40:04 INFO Completed upload id=20 objects=3 duration=2.288402ms44962026/11/11 23:40:04 INFO Vacuumed table table=closures44972026/11/11 23:40:04 INFO Received complete push request method=POST path=/44982026/11/11 23:40:04 INFO Vacuumed table table=objects44992026/11/11 23:40:04 INFO Completed push id=29 objects=17 duration=1.360353ms45002026/11/11 23:40:04 INFO Received complete push request method=POST path=/45012026/11/11 23:40:04 INFO Completed push id=29 objects=6 duration=1.06825ms45022026/11/11 23:40:04 INFO Received complete push request method=POST path=/45032026/11/11 23:40:04 INFO Received uploads request method=POST path=/45042026/11/11 23:40:04 INFO Completed push id=29 objects=24 duration=1.724697ms45052026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045062026/11/11 23:40:04 INFO Received complete push request method=POST path=/45072026/11/11 23:40:04 INFO Completed push id=32 objects=3 duration=789.608µs45082026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period45092026/11/11 23:40:04 INFO Received push request method=POST path=/45102026/11/11 23:40:04 INFO Received uploads request method=POST path=/45112026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045122026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045132026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45142026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45152026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period45162026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45172026/11/11 23:40:04 INFO Vacuumed table table=closures45182026/11/11 23:40:04 INFO Vacuumed table table=objects45192026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045202026/11/11 23:40:04 INFO Received uploads request method=POST path=/45212026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45222026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045232026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period45242026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45252026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045262026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45272026/11/11 23:40:04 INFO Vacuumed table table=closures45282026/11/11 23:40:04 INFO Vacuumed table table=objects45292026/11/11 23:40:04 INFO Received uploads request method=POST path=/45302026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045312026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045322026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=045332026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45342026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045352026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period45362026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45372026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45382026/11/11 23:40:04 INFO Received push request method=POST path=/45392026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45402026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045412026/11/11 23:40:04 INFO Vacuumed table table=closures45422026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45432026/11/11 23:40:04 INFO Received complete upload request method=POST path=/45442026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45452026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period45462026/11/11 23:40:04 INFO Vacuumed table table=objects45472026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45482026/11/11 23:40:04 INFO Vacuumed table table=closures45492026/11/11 23:40:04 INFO Completed upload id=30 objects=8 duration=923.689µs45502026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45512026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45522026/11/11 23:40:04 INFO Vacuumed table table=objects45532026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045542026/11/11 23:40:04 INFO Vacuumed table table=closures45552026/11/11 23:40:04 INFO Vacuumed table table=objects45562026/11/11 23:40:04 INFO Received push request method=POST path=/45572026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45582026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45592026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45602026/11/11 23:40:04 INFO Vacuumed table table=closures45612026/11/11 23:40:04 INFO Vacuumed table table=objects45622026/11/11 23:40:04 INFO Received uploads request method=POST path=/45632026/11/11 23:40:04 INFO Received complete push request method=POST path=/45642026/11/11 23:40:04 INFO Received complete upload request method=POST path=/45652026/11/11 23:40:04 INFO Completed upload id=24 objects=7 duration=939.029µs45662026/11/11 23:40:04 INFO Completed push id=16 objects=14 duration=2.775127ms45672026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045682026/11/11 23:40:04 INFO Received push request method=POST path=/45692026/11/11 23:40:04 INFO Received complete upload request method=POST path=/45702026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period45712026/11/11 23:40:04 INFO Completed upload id=31 objects=6 duration=963.63µs45722026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045732026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45742026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45752026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45762026/11/11 23:40:04 INFO Vacuumed table table=closures45772026/11/11 23:40:04 INFO Vacuumed table table=objects45782026/11/11 23:40:04 INFO Received uploads request method=POST path=/45792026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045802026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=045812026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period45822026/11/11 23:40:04 INFO Received complete upload request method=POST path=/45832026/11/11 23:40:04 INFO Completed upload id=32 objects=4 duration=886.949µs45842026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045852026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045862026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45872026/11/11 23:40:04 INFO Vacuumed table table=pending_closures45882026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45892026/11/11 23:40:04 INFO Vacuumed table table=pending_objects45902026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45912026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads45922026/11/11 23:40:04 INFO Received uploads request method=POST path=/45932026/11/11 23:40:04 INFO Vacuumed table table=closures45942026/11/11 23:40:04 INFO Vacuumed table table=closures45952026/11/11 23:40:04 INFO Vacuumed table table=objects45962026/11/11 23:40:04 INFO Vacuumed table table=objects45972026/11/11 23:40:04 INFO Received uploads request method=POST path=/45982026/11/11 23:40:04 INFO Received complete upload request method=POST path=/45992026/11/11 23:40:04 INFO Completed upload id=17 objects=25 duration=2.913568ms46002026/11/11 23:40:04 INFO Received push request method=POST path=/46012026/11/11 23:40:04 INFO Received uploads request method=POST path=/46022026/11/11 23:40:04 INFO Received complete upload request method=POST path=/46032026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=046042026/11/11 23:40:04 INFO Received complete push request method=POST path=/46052026/11/11 23:40:04 INFO Completed upload id=33 objects=10 duration=993.09µs46062026/11/11 23:40:04 INFO Received complete upload request method=POST path=/46072026/11/11 23:40:04 INFO Completed push id=14 objects=3 duration=1.07905ms46082026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046092026/11/11 23:40:04 INFO Vacuumed table table=pending_closures46102026/11/11 23:40:04 INFO Completed upload id=13 objects=14 duration=2.934548ms46112026/11/11 23:40:04 INFO Vacuumed table table=pending_objects46122026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads46132026/11/11 23:40:04 INFO Vacuumed table table=closures46142026/11/11 23:40:04 INFO Vacuumed table table=objects46152026/11/11 23:40:04 INFO Received uploads request method=POST path=/46162026/11/11 23:40:04 INFO Received push request method=POST path=/46172026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=046182026/11/11 23:40:04 INFO Received push request method=POST path=/46192026/11/11 23:40:04 INFO Received push request method=POST path=/46202026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=046212026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period46222026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046232026/11/11 23:40:04 INFO Vacuumed table table=pending_closures46242026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046252026/11/11 23:40:04 INFO Received complete upload request method=POST path=/46262026/11/11 23:40:04 INFO Vacuumed table table=pending_objects46272026/11/11 23:40:04 INFO Vacuumed table table=pending_closures46282026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads46292026/11/11 23:40:04 INFO Completed upload id=21 objects=2 duration=1.00663ms46302026/11/11 23:40:04 INFO Vacuumed table table=closures46312026/11/11 23:40:04 INFO Vacuumed table table=pending_objects46322026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads46332026/11/11 23:40:04 INFO Vacuumed table table=objects46342026/11/11 23:40:04 INFO Vacuumed table table=closures46352026/11/11 23:40:04 INFO Vacuumed table table=objects46362026/11/11 23:40:04 INFO Received complete upload request method=POST path=/46372026/11/11 23:40:04 INFO Completed upload id=34 objects=23 duration=1.196172ms46382026/11/11 23:40:04 INFO Received uploads request method=POST path=/46392026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=046402026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period46412026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046422026/11/11 23:40:04 INFO Vacuumed table table=pending_closures46432026/11/11 23:40:04 INFO Vacuumed table table=pending_objects46442026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads46452026/11/11 23:40:04 INFO Received complete push request method=POST path=/46462026/11/11 23:40:04 INFO Received complete push request method=POST path=/46472026/11/11 23:40:04 INFO Vacuumed table table=closures46482026/11/11 23:40:04 INFO Received complete push request method=POST path=/46492026/11/11 23:40:04 INFO Vacuumed table table=objects46502026/11/11 23:40:04 INFO Completed push id=35 objects=6 duration=1.633136ms46512026/11/11 23:40:04 INFO Received push request method=POST path=/46522026/11/11 23:40:04 INFO Received complete upload request method=POST path=/46532026/11/11 23:40:04 INFO Completed push id=31 objects=6 duration=4.269941ms46542026/11/11 23:40:04 INFO Completed push id=23 objects=28 duration=3.356792ms46552026/11/11 23:40:04 INFO Completed upload id=25 objects=6 duration=1.884338ms46562026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=046572026/11/11 23:40:04 INFO Received uploads request method=POST path=/46582026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period46592026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=046602026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period46612026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046622026/11/11 23:40:04 INFO Received complete upload request method=POST path=/46632026/11/11 23:40:04 INFO Vacuumed table table=pending_closures46642026/11/11 23:40:04 INFO Vacuumed table table=pending_objects46652026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads46662026/11/11 23:40:04 INFO Completed upload id=14 objects=16 duration=2.963128ms46672026/11/11 23:40:04 INFO Vacuumed table table=closures46682026/11/11 23:40:04 INFO Vacuumed table table=objects46692026/11/11 23:40:04 INFO Received push request method=POST path=/46702026/11/11 23:40:04 INFO Received complete upload request method=POST path=/46712026/11/11 23:40:04 INFO Completed upload id=27 objects=8 duration=1.00911ms46722026/11/11 23:40:04 INFO Received push request method=POST path=/46732026/11/11 23:40:04 INFO Received uploads request method=POST path=/46742026/11/11 23:40:04 INFO Received push request method=POST path=/46752026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=046762026/11/11 23:40:04 INFO Received complete push request method=POST path=/46772026/11/11 23:40:04 INFO Vacuumed table table=pending_closures46782026/11/11 23:40:04 INFO Vacuumed table table=pending_objects46792026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads46802026/11/11 23:40:04 INFO Vacuumed table table=closures46812026/11/11 23:40:04 INFO Completed push id=7 objects=24 duration=4.587924ms46822026/11/11 23:40:04 INFO Vacuumed table table=objects46832026/11/11 23:40:04 INFO Received complete push request method=POST path=/46842026/11/11 23:40:04 INFO Completed push id=11 objects=9 duration=1.736416ms46852026/11/11 23:40:04 INFO Received uploads request method=POST path=/46862026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=046872026/11/11 23:40:04 INFO Received uploads request method=POST path=/46882026/11/11 23:40:04 INFO Received complete push request method=POST path=/46892026/11/11 23:40:04 INFO Completed push id=17 objects=11 duration=2.621625ms46902026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046912026/11/11 23:40:04 INFO Vacuumed table table=pending_closures46922026/11/11 23:40:04 INFO Vacuumed table table=pending_objects46932026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads46942026/11/11 23:40:04 INFO Received complete push request method=POST path=/46952026/11/11 23:40:04 INFO Vacuumed table table=closures46962026/11/11 23:40:04 INFO Received complete push request method=POST path=/46972026/11/11 23:40:04 INFO Vacuumed table table=objects46982026/11/11 23:40:04 INFO Completed push id=1 objects=9 duration=1.461294ms46992026/11/11 23:40:04 INFO Received uploads request method=POST path=/47002026/11/11 23:40:04 INFO Completed push id=15 objects=24 duration=2.403603ms47012026/11/11 23:40:04 INFO Received push request method=POST path=/47022026/11/11 23:40:04 INFO Received uploads request method=POST path=/47032026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=047042026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47052026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=047062026/11/11 23:40:04 INFO Completed upload id=19 objects=2 duration=939.589µs47072026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047082026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period47092026/11/11 23:40:04 INFO Vacuumed table table=pending_closures47102026/11/11 23:40:04 INFO Received push request method=POST path=/47112026/11/11 23:40:04 INFO Vacuumed table table=pending_objects47122026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads47132026/11/11 23:40:04 INFO Vacuumed table table=closures47142026/11/11 23:40:04 INFO Received uploads request method=POST path=/47152026/11/11 23:40:04 INFO Vacuumed table table=objects47162026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047172026/11/11 23:40:04 INFO Vacuumed table table=pending_closures47182026/11/11 23:40:04 INFO Vacuumed table table=pending_objects47192026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads47202026/11/11 23:40:04 INFO Vacuumed table table=closures47212026/11/11 23:40:04 INFO Vacuumed table table=objects47222026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47232026/11/11 23:40:04 INFO Completed upload id=24 objects=7 duration=936.489µs47242026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47252026/11/11 23:40:04 INFO Received push request method=POST path=/47262026/11/11 23:40:04 INFO Completed upload id=25 objects=16 duration=2.695205ms47272026/11/11 23:40:04 INFO Received complete push request method=POST path=/47282026/11/11 23:40:04 INFO Completed push id=26 objects=3 duration=1.02773ms47292026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47302026/11/11 23:40:04 INFO Completed upload id=26 objects=6 duration=1.124351ms47312026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47322026/11/11 23:40:04 INFO Received uploads request method=POST path=/47332026/11/11 23:40:04 INFO Completed upload id=2 objects=7 duration=1.687036ms47342026/11/11 23:40:04 INFO Received uploads request method=POST path=/47352026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=047362026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period47372026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047382026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47392026/11/11 23:40:04 INFO Vacuumed table table=pending_closures47402026/11/11 23:40:04 INFO Completed upload id=28 objects=8 duration=934.089µs47412026/11/11 23:40:04 INFO Vacuumed table table=pending_objects47422026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads47432026/11/11 23:40:04 INFO Vacuumed table table=closures47442026/11/11 23:40:04 INFO Received uploads request method=POST path=/47452026/11/11 23:40:04 INFO Vacuumed table table=objects47462026/11/11 23:40:04 INFO Received complete push request method=POST path=/47472026/11/11 23:40:04 INFO Completed push id=13 objects=2 duration=1.706656ms47482026/11/11 23:40:04 INFO Received uploads request method=POST path=/47492026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47502026/11/11 23:40:04 INFO Completed upload id=29 objects=10 duration=957.969µs47512026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47522026/11/11 23:40:04 INFO Received push request method=POST path=/47532026/11/11 23:40:04 INFO Completed upload id=29 objects=5 duration=1.08523ms47542026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47552026/11/11 23:40:04 INFO Completed upload id=27 objects=13 duration=1.134791ms47562026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47572026/11/11 23:40:04 INFO Received uploads request method=POST path=/47582026/11/11 23:40:04 INFO Completed upload id=30 objects=20 duration=1.10675ms47592026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47602026/11/11 23:40:04 INFO Received push request method=POST path=/47612026/11/11 23:40:04 INFO Completed upload id=8 objects=6 duration=1.344632ms47622026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=047632026/11/11 23:40:04 INFO Received complete push request method=POST path=/47642026/11/11 23:40:04 INFO Completed push id=31 objects=11 duration=1.116011ms47652026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=047662026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period47672026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047682026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047692026/11/11 23:40:04 INFO Vacuumed table table=pending_closures47702026/11/11 23:40:04 INFO Vacuumed table table=pending_closures47712026/11/11 23:40:04 INFO Vacuumed table table=pending_objects47722026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads47732026/11/11 23:40:04 INFO Vacuumed table table=pending_objects47742026/11/11 23:40:04 INFO Vacuumed table table=closures47752026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads47762026/11/11 23:40:04 INFO Vacuumed table table=closures47772026/11/11 23:40:04 INFO Vacuumed table table=objects47782026/11/11 23:40:04 INFO Vacuumed table table=objects47792026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47802026/11/11 23:40:04 INFO Completed upload id=32 objects=2 duration=884.169µs47812026/11/11 23:40:04 INFO Received uploads request method=POST path=/47822026/11/11 23:40:04 INFO Received push request method=POST path=/47832026/11/11 23:40:04 INFO Received uploads request method=POST path=/47842026/11/11 23:40:04 INFO Received complete push request method=POST path=/47852026/11/11 23:40:04 INFO Completed push id=33 objects=3 duration=965.489µs47862026/11/11 23:40:04 INFO Received uploads request method=POST path=/47872026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47882026/11/11 23:40:04 INFO Completed upload id=12 objects=9 duration=1.511434ms47892026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47902026/11/11 23:40:04 INFO Completed upload id=34 objects=3 duration=665.426µs47912026/11/11 23:40:04 INFO Received push request method=POST path=/47922026/11/11 23:40:04 INFO Received complete push request method=POST path=/47932026/11/11 23:40:04 INFO Completed push id=18 objects=6 duration=4.906186ms47942026/11/11 23:40:04 INFO Received complete upload request method=POST path=/47952026/11/11 23:40:04 INFO Received uploads request method=POST path=/47962026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=047972026/11/11 23:40:04 INFO Received push request method=POST path=/47982026/11/11 23:40:04 INFO Completed upload id=22 objects=5 duration=2.127841ms47992026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period48002026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048012026/11/11 23:40:04 INFO Vacuumed table table=pending_closures48022026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48032026/11/11 23:40:04 INFO Vacuumed table table=pending_objects48042026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads48052026/11/11 23:40:04 INFO Completed upload id=23 objects=9 duration=1.03047ms48062026/11/11 23:40:04 INFO Vacuumed table table=closures48072026/11/11 23:40:04 INFO Vacuumed table table=objects48082026/11/11 23:40:04 INFO Received complete push request method=POST path=/48092026/11/11 23:40:04 INFO Received uploads request method=POST path=/48102026/11/11 23:40:04 INFO Completed push id=28 objects=3 duration=1.913338ms48112026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48122026/11/11 23:40:04 INFO Completed upload id=31 objects=3 duration=3.09625ms48132026/11/11 23:40:04 INFO Received uploads request method=POST path=/48142026/11/11 23:40:04 INFO Received complete push request method=POST path=/48152026/11/11 23:40:04 INFO Completed push id=29 objects=15 duration=1.198351ms48162026/11/11 23:40:04 INFO Received push request method=POST path=/48172026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48182026/11/11 23:40:04 INFO Completed upload id=22 objects=2 duration=969.549µs48192026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48202026/11/11 23:40:04 INFO Received complete push request method=POST path=/48212026/11/11 23:40:04 INFO Completed push id=30 objects=6 duration=1.02525ms48222026/11/11 23:40:04 INFO Completed upload id=29 objects=4 duration=2.07836ms48232026/11/11 23:40:04 INFO Received uploads request method=POST path=/48242026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=048252026/11/11 23:40:04 INFO Received uploads request method=POST path=/48262026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48272026/11/11 23:40:04 INFO Completed upload id=31 objects=3 duration=1.118811ms48282026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period48292026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048302026/11/11 23:40:04 INFO Vacuumed table table=pending_closures48312026/11/11 23:40:04 INFO Vacuumed table table=pending_objects48322026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads48332026/11/11 23:40:04 INFO Vacuumed table table=closures48342026/11/11 23:40:04 INFO Vacuumed table table=objects48352026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48362026/11/11 23:40:04 INFO Completed upload id=18 objects=22 duration=1.483174ms48372026/11/11 23:40:04 INFO Received uploads request method=POST path=/48382026/11/11 23:40:04 INFO Aborted multipart uploads count=0 kept=048392026/11/11 23:40:04 WARN Force mode enabled - objects will be deleted immediately without grace period48402026/11/11 23:40:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048412026/11/11 23:40:04 INFO Vacuumed table table=pending_closures48422026/11/11 23:40:04 INFO Vacuumed table table=pending_objects48432026/11/11 23:40:04 INFO Vacuumed table table=multipart_uploads48442026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48452026/11/11 23:40:04 INFO Vacuumed table table=closures48462026/11/11 23:40:04 INFO Vacuumed table table=objects48472026/11/11 23:40:04 INFO Received push request method=POST path=/48482026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48492026/11/11 23:40:04 INFO Completed upload id=18 objects=2 duration=2.759027ms48502026/11/11 23:40:04 INFO Completed upload id=21 objects=8 duration=768.687µs48512026/11/11 23:40:04 INFO Received complete push request method=POST path=/48522026/11/11 23:40:04 INFO Completed push id=11 objects=6 duration=2.531985ms48532026/11/11 23:40:04 INFO Received push request method=POST path=/48542026/11/11 23:40:04 INFO Received complete upload request method=POST path=/48552026/11/11 23:40:04 INFO Completed upload id=22 objects=7 duration=954.089µs48562026/11/11 23:40:04 INFO Received complete push request method=POST path=/48572026/11/11 23:40:05 INFO Completed push id=16 objects=26 duration=1.435374ms48582026/11/11 23:40:05 INFO Received complete push request method=POST path=/48592026/11/11 23:40:05 INFO Received complete push request method=POST path=/48602026/11/11 23:40:05 INFO Completed push id=24 objects=5 duration=983.51µs48612026/11/11 23:40:05 INFO Completed push id=20 objects=13 duration=2.788627ms48622026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48632026/11/11 23:40:05 INFO Completed upload id=13 objects=7 duration=960.329µs48642026/11/11 23:40:05 INFO Received complete push request method=POST path=/48652026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48662026/11/11 23:40:05 INFO Completed upload id=20 objects=4 duration=904.369µs48672026/11/11 23:40:05 INFO Aborted multipart uploads count=0 kept=048682026/11/11 23:40:05 INFO Completed push id=22 objects=4 duration=1.704397ms48692026/11/11 23:40:05 WARN Force mode enabled - objects will be deleted immediately without grace period48702026/11/11 23:40:05 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048712026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48722026/11/11 23:40:05 INFO Vacuumed table table=pending_closures48732026/11/11 23:40:05 INFO Completed upload id=25 objects=8 duration=1.09971ms48742026/11/11 23:40:05 INFO Vacuumed table table=pending_objects48752026/11/11 23:40:05 INFO Vacuumed table table=multipart_uploads48762026/11/11 23:40:05 INFO Vacuumed table table=closures48772026/11/11 23:40:05 INFO Vacuumed table table=objects48782026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48792026/11/11 23:40:05 INFO Completed upload id=23 objects=7 duration=961.689µs48802026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48812026/11/11 23:40:05 INFO Completed upload id=23 objects=7 duration=1.310913ms48822026/11/11 23:40:05 INFO Received complete push request method=POST path=/48832026/11/11 23:40:05 INFO Completed push id=26 objects=4 duration=1.11401ms48842026/11/11 23:40:05 INFO Received complete push request method=POST path=/48852026/11/11 23:40:05 INFO Completed push id=27 objects=6 duration=3.13555ms48862026/11/11 23:40:05 INFO Received complete push request method=POST path=/48872026/11/11 23:40:05 INFO Received complete push request method=POST path=/48882026/11/11 23:40:05 INFO Completed push id=30 objects=2 duration=781.427µs48892026/11/11 23:40:05 INFO Completed push id=24 objects=5 duration=1.155231ms48902026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48912026/11/11 23:40:05 INFO Completed upload id=25 objects=2 duration=849.548µs48922026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48932026/11/11 23:40:05 INFO Completed upload id=15 objects=2 duration=1.01925ms48942026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48952026/11/11 23:40:05 INFO Completed upload id=24 objects=18 duration=1.327332ms48962026/11/11 23:40:05 INFO Received complete push request method=POST path=/48972026/11/11 23:40:05 INFO Completed push id=32 objects=7 duration=1.084951ms48982026/11/11 23:40:05 INFO Received complete upload request method=POST path=/48992026/11/11 23:40:05 INFO Completed upload id=26 objects=20 duration=928.269µs49002026/11/11 23:40:05 INFO Received complete push request method=POST path=/49012026/11/11 23:40:05 INFO Completed push id=25 objects=2 duration=1.960179ms49022026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49032026/11/11 23:40:05 INFO Completed upload id=33 objects=6 duration=1.06535ms49042026/11/11 23:40:05 INFO Received complete push request method=POST path=/49052026/11/11 23:40:05 INFO Completed push id=27 objects=4 duration=751.807µs49062026/11/11 23:40:05 INFO Received complete push request method=POST path=/49072026/11/11 23:40:05 INFO Completed push id=28 objects=7 duration=1.01775ms49082026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49092026/11/11 23:40:05 INFO Completed upload id=34 objects=3 duration=605.846µs49102026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49112026/11/11 23:40:05 INFO Completed upload id=30 objects=6 duration=963.649µs49122026/11/11 23:40:05 INFO Aborted multipart uploads count=0 kept=049132026/11/11 23:40:05 INFO Received complete push request method=POST path=/49142026/11/11 23:40:05 INFO Completed push id=28 objects=26 duration=1.949099ms49152026/11/11 23:40:05 WARN Force mode enabled - objects will be deleted immediately without grace period49162026/11/11 23:40:05 INFO Received push request method=POST path=/49172026/11/11 23:40:05 INFO Received complete push request method=POST path=/49182026/11/11 23:40:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049192026/11/11 23:40:05 INFO Completed push id=31 objects=5 duration=1.326113ms49202026/11/11 23:40:05 INFO Vacuumed table table=pending_closures49212026/11/11 23:40:05 INFO Vacuumed table table=pending_objects49222026/11/11 23:40:05 INFO Vacuumed table table=multipart_uploads49232026/11/11 23:40:05 INFO Vacuumed table table=closures49242026/11/11 23:40:05 INFO Received complete push request method=POST path=/49252026/11/11 23:40:05 INFO Completed push id=30 objects=3 duration=935.709µs49262026/11/11 23:40:05 INFO Vacuumed table table=objects49272026/11/11 23:40:05 INFO Received uploads request method=POST path=/49282026/11/11 23:40:05 INFO Received complete push request method=POST path=/49292026/11/11 23:40:05 INFO Completed push id=32 objects=19 duration=1.025969ms49302026/11/11 23:40:05 INFO Received complete push request method=POST path=/49312026/11/11 23:40:05 INFO Completed push id=32 objects=26 duration=1.785757ms49322026/11/11 23:40:05 INFO Aborted multipart uploads count=0 kept=049332026/11/11 23:40:05 INFO Received uploads request method=POST path=/49342026/11/11 23:40:05 WARN Force mode enabled - objects will be deleted immediately without grace period49352026/11/11 23:40:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049362026/11/11 23:40:05 INFO Vacuumed table table=pending_closures49372026/11/11 23:40:05 INFO Vacuumed table table=pending_objects49382026/11/11 23:40:05 INFO Vacuumed table table=multipart_uploads49392026/11/11 23:40:05 INFO Vacuumed table table=closures49402026/11/11 23:40:05 INFO Vacuumed table table=objects49412026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49422026/11/11 23:40:05 INFO Completed upload id=33 objects=2 duration=798.088µs49432026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49442026/11/11 23:40:05 INFO Completed upload id=26 objects=8 duration=1.10019ms49452026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49462026/11/11 23:40:05 INFO Completed upload id=34 objects=5 duration=921.349µs49472026/11/11 23:40:05 INFO Received complete push request method=POST path=/49482026/11/11 23:40:05 INFO Completed push id=28 objects=14 duration=1.260552ms49492026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49502026/11/11 23:40:05 INFO Completed upload id=35 objects=11 duration=721.707µs49512026/11/11 23:40:05 INFO Received push request method=POST path=/49522026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49532026/11/11 23:40:05 INFO Completed upload id=36 objects=3 duration=742.527µs49542026/11/11 23:40:05 INFO Received uploads request method=POST path=/49552026/11/11 23:40:05 INFO Aborted multipart uploads count=0 kept=049562026/11/11 23:40:05 WARN Force mode enabled - objects will be deleted immediately without grace period49572026/11/11 23:40:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049582026/11/11 23:40:05 INFO Vacuumed table table=pending_closures49592026/11/11 23:40:05 INFO Vacuumed table table=pending_objects49602026/11/11 23:40:05 INFO Vacuumed table table=multipart_uploads49612026/11/11 23:40:05 INFO Vacuumed table table=closures49622026/11/11 23:40:05 INFO Vacuumed table table=objects49632026/11/11 23:40:05 INFO Aborted multipart uploads count=0 kept=049642026/11/11 23:40:05 WARN Force mode enabled - objects will be deleted immediately without grace period49652026/11/11 23:40:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049662026/11/11 23:40:05 INFO Vacuumed table table=pending_closures49672026/11/11 23:40:05 INFO Vacuumed table table=pending_objects49682026/11/11 23:40:05 INFO Vacuumed table table=multipart_uploads49692026/11/11 23:40:05 INFO Vacuumed table table=closures49702026/11/11 23:40:05 INFO Vacuumed table table=objects49712026/11/11 23:40:05 INFO Received uploads request method=POST path=/49722026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49732026/11/11 23:40:05 INFO Completed upload id=21 objects=9 duration=1.755577ms49742026/11/11 23:40:05 INFO Received complete push request method=POST path=/49752026/11/11 23:40:05 INFO Completed push id=25 objects=10 duration=1.801138ms49762026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49772026/11/11 23:40:05 INFO Completed upload id=29 objects=5 duration=664.927µs49782026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49792026/11/11 23:40:05 INFO Completed upload id=30 objects=2 duration=715.567µs49802026/11/11 23:40:05 INFO Received complete push request method=POST path=/49812026/11/11 23:40:05 INFO Completed push id=31 objects=20 duration=1.134491ms49822026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49832026/11/11 23:40:05 INFO Completed upload id=32 objects=5 duration=773.208µs49842026/11/11 23:40:05 INFO Received complete upload request method=POST path=/49852026/11/11 23:40:05 INFO Completed upload id=33 objects=3 duration=661.266µs49862026/11/11 23:40:05 INFO Aborted multipart uploads count=0 kept=049872026/11/11 23:40:05 WARN Force mode enabled - objects will be deleted immediately without grace period49882026/11/11 23:40:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049892026/11/11 23:40:05 INFO Vacuumed table table=pending_closures49902026/11/11 23:40:05 INFO Vacuumed table table=pending_objects49912026/11/11 23:40:05 INFO Vacuumed table table=multipart_uploads49922026/11/11 23:40:05 INFO Vacuumed table table=closures49932026/11/11 23:40:05 INFO Vacuumed table table=objects4994--- PASS: FuzzPushGC (0.01s)4995 --- PASS: FuzzPushGC/seed#0 (3.63s)4996 --- PASS: FuzzPushGC/seed#4 (3.70s)4997 --- PASS: FuzzPushGC/seed#9 (3.78s)4998 --- PASS: FuzzPushGC/seed#13 (3.80s)4999 --- PASS: FuzzPushGC/seed#14 (3.88s)5000 --- PASS: FuzzPushGC/seed#8 (3.89s)5001 --- PASS: FuzzPushGC/seed#1 (3.91s)5002 --- PASS: FuzzPushGC/seed#2 (3.92s)5003 --- PASS: FuzzPushGC/seed#3 (3.93s)5004 --- PASS: FuzzPushGC/seed#12 (4.10s)5005 --- PASS: FuzzPushGC/seed#5 (4.30s)5006 --- PASS: FuzzPushGC/seed#15 (4.38s)5007 --- PASS: FuzzPushGC/seed#10 (4.56s)5008 --- PASS: FuzzPushGC/seed#7 (4.61s)5009 --- PASS: FuzzPushGC/seed#11 (4.68s)5010 --- PASS: FuzzPushGC/seed#6 (4.84s)5011PASS50122026-11-11 23:40:05.955 UTC [131] LOG: received smart shutdown request50132026-11-11 23:40:05.969 UTC [131] LOG: background worker "logical replication launcher" (PID 142) exited with exit code 150142026-11-11 23:40:08.138 UTC [137] LOG: checkpoint complete: wrote 3819 buffers (23.3%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=5.433 s, sync=0.419 s, total=7.474 s; sync files=42363, longest=0.016 s, average=0.001 s; distance=536409 kB, estimate=536409 kB; lsn=0/26DCB390, redo lsn=0/2233C4D850152026-11-11 23:40:08.145 UTC [137] LOG: shutting down50162026-11-11 23:40:08.146 UTC [137] LOG: checkpoint starting: shutdown immediate50172026-11-11 23:40:08.735 UTC [137] LOG: checkpoint complete: wrote 9010 buffers (55.0%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.180 s, sync=0.396 s, total=0.590 s; sync files=2778, longest=0.005 s, average=0.001 s; distance=76347 kB, estimate=490403 kB; lsn=0/26DCB408, redo lsn=0/26DCB40850182026-11-11 23:40:08.964 UTC [131] LOG: database system is shut down5019Running OIDC tests...5020=== RUN TestAudienceForIssuer5021=== PAUSE TestAudienceForIssuer5022=== RUN TestGlobMatch5023=== PAUSE TestGlobMatch5024=== RUN TestValidateToken_ValidToken5025=== PAUSE TestValidateToken_ValidToken5026=== RUN TestValidateToken_WrongAudience5027=== PAUSE TestValidateToken_WrongAudience5028=== RUN TestValidateToken_Expired5029=== PAUSE TestValidateToken_Expired5030=== RUN TestValidateToken_BoundClaimsMismatch5031=== PAUSE TestValidateToken_BoundClaimsMismatch5032=== RUN TestValidateToken_BoundSubjectMismatch5033=== PAUSE TestValidateToken_BoundSubjectMismatch5034=== RUN TestValidateToken_MultipleProviders5035=== PAUSE TestValidateToken_MultipleProviders5036=== RUN TestValidateToken_NoMatchingProvider5037=== PAUSE TestValidateToken_NoMatchingProvider5038=== RUN TestValidateToken_KubernetesServiceAccount5039=== PAUSE TestValidateToken_KubernetesServiceAccount5040=== RUN TestNewValidator_KubernetesRequiresCA5041=== PAUSE TestNewValidator_KubernetesRequiresCA5042=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5043=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5044=== RUN TestPins_ReservedForMatchingRule5045=== PAUSE TestPins_ReservedForMatchingRule5046=== RUN TestPins_TopLevelShorthand5047=== PAUSE TestPins_TopLevelShorthand5048=== RUN TestPins_ConfigValidation5049=== PAUSE TestPins_ConfigValidation5050=== RUN TestScopes_LegacyProviderDefaultsToWrite5051=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5052=== RUN TestScopes_Rules5053=== PAUSE TestScopes_Rules5054=== RUN TestScopes_ConfigValidation5055=== PAUSE TestScopes_ConfigValidation5056=== CONT TestAudienceForIssuer5057--- PASS: TestAudienceForIssuer (0.00s)5058=== CONT TestScopes_ConfigValidation5059--- PASS: TestScopes_ConfigValidation (0.00s)5060=== CONT TestScopes_Rules5061=== CONT TestScopes_LegacyProviderDefaultsToWrite5062=== CONT TestPins_ConfigValidation5063--- PASS: TestPins_ConfigValidation (0.00s)5064=== CONT TestPins_TopLevelShorthand5065=== CONT TestPins_ReservedForMatchingRule5066=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5067=== CONT TestValidateToken_BoundSubjectMismatch5068=== CONT TestGlobMatch5069=== CONT TestValidateToken_BoundClaimsMismatch5070=== CONT TestValidateToken_Expired5071=== CONT TestValidateToken_WrongAudience5072=== CONT TestValidateToken_NoMatchingProvider5073=== CONT TestValidateToken_KubernetesServiceAccount5074=== CONT TestNewValidator_KubernetesRequiresCA5075=== RUN TestGlobMatch/foo_foo50762026/11/11 23:40:12 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235077=== CONT TestValidateToken_ValidToken5078=== CONT TestValidateToken_MultipleProviders5079=== PAUSE TestGlobMatch/foo_foo5080=== RUN TestGlobMatch/foo_bar5081=== PAUSE TestGlobMatch/foo_bar5082=== RUN TestGlobMatch/*_5083=== PAUSE TestGlobMatch/*_5084=== RUN TestGlobMatch/*_anything5085=== PAUSE TestGlobMatch/*_anything5086=== RUN TestGlobMatch/foo*_foo5087=== PAUSE TestGlobMatch/foo*_foo5088=== RUN TestGlobMatch/foo*_foobar5089=== PAUSE TestGlobMatch/foo*_foobar5090=== RUN TestGlobMatch/foo*_bar5091=== PAUSE TestGlobMatch/foo*_bar5092=== RUN TestGlobMatch/*bar_bar5093=== PAUSE TestGlobMatch/*bar_bar5094=== RUN TestGlobMatch/*bar_foobar5095=== PAUSE TestGlobMatch/*bar_foobar5096=== RUN TestGlobMatch/*bar_foo5097=== PAUSE TestGlobMatch/*bar_foo5098=== RUN TestGlobMatch/foo*bar_foobar5099=== PAUSE TestGlobMatch/foo*bar_foobar5100=== RUN TestGlobMatch/foo*bar_foo123bar5101=== PAUSE TestGlobMatch/foo*bar_foo123bar5102=== RUN TestGlobMatch/foo*bar_foobarbaz5103=== PAUSE TestGlobMatch/foo*bar_foobarbaz5104=== RUN TestGlobMatch/*/*_foo/bar5105=== PAUSE TestGlobMatch/*/*_foo/bar5106=== RUN TestGlobMatch/*/*_foo5107=== PAUSE TestGlobMatch/*/*_foo5108=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5109=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5110=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05111=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05112=== RUN TestGlobMatch/refs/*/main_refs/heads/main5113=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5114=== RUN TestGlobMatch/fo?_foo5115=== PAUSE TestGlobMatch/fo?_foo5116=== RUN TestGlobMatch/fo?_fo5117=== PAUSE TestGlobMatch/fo?_fo5118=== RUN TestGlobMatch/fo?_fooo5119=== PAUSE TestGlobMatch/fo?_fooo5120=== RUN TestGlobMatch/?oo_foo5121=== PAUSE TestGlobMatch/?oo_foo5122=== RUN TestGlobMatch/?oo_boo5123=== PAUSE TestGlobMatch/?oo_boo5124=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5125=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5126=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5127=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5128=== CONT TestGlobMatch/foo_foo5129=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5130=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5131=== CONT TestGlobMatch/?oo_boo5132=== CONT TestGlobMatch/?oo_foo5133=== CONT TestGlobMatch/fo?_fooo5134=== CONT TestGlobMatch/fo?_fo5135=== CONT TestGlobMatch/fo?_foo5136=== CONT TestGlobMatch/refs/*/main_refs/heads/main5137=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05138=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5139=== CONT TestGlobMatch/*/*_foo5140=== CONT TestGlobMatch/*/*_foo/bar5141=== CONT TestGlobMatch/foo*bar_foobarbaz5142=== CONT TestGlobMatch/foo*bar_foo123bar5143=== CONT TestGlobMatch/foo*bar_foobar5144=== CONT TestGlobMatch/*bar_foo5145=== CONT TestGlobMatch/*bar_foobar5146=== CONT TestGlobMatch/*bar_bar5147=== CONT TestGlobMatch/foo*_bar5148=== CONT TestGlobMatch/foo*_foobar5149=== CONT TestGlobMatch/foo*_foo5150=== CONT TestGlobMatch/*_anything5151=== CONT TestGlobMatch/*_5152=== CONT TestGlobMatch/foo_bar5153--- PASS: TestGlobMatch (0.80s)5154 --- PASS: TestGlobMatch/foo_foo (0.00s)5155 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5156 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5157 --- PASS: TestGlobMatch/?oo_boo (0.00s)5158 --- PASS: TestGlobMatch/?oo_foo (0.00s)5159 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5160 --- PASS: TestGlobMatch/fo?_fo (0.00s)5161 --- PASS: TestGlobMatch/fo?_foo (0.00s)5162 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5163 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5164 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5165 --- PASS: TestGlobMatch/*/*_foo (0.00s)5166 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5167 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5168 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5169 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5170 --- PASS: TestGlobMatch/*bar_foo (0.00s)5171 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5172 --- PASS: TestGlobMatch/*bar_bar (0.00s)5173 --- PASS: TestGlobMatch/foo*_bar (0.00s)5174 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5175 --- PASS: TestGlobMatch/foo*_foo (0.00s)5176 --- PASS: TestGlobMatch/*_anything (0.00s)5177 --- PASS: TestGlobMatch/*_ (0.00s)5178 --- PASS: TestGlobMatch/foo_bar (0.00s)51792026/11/11 23:40:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40067/oidc51802026/11/11 23:40:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43103/oidc51812026/11/11 23:40:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43223/oidc51822026/11/11 23:40:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35659/oidc51832026/11/11 23:40:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38685/oidc5184--- PASS: TestValidateToken_BoundClaimsMismatch (3.54s)51852026/11/11 23:40:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43279/oidc5186--- PASS: TestPins_TopLevelShorthand (4.50s)51872026/11/11 23:40:16 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:414455188--- PASS: TestPins_ReservedForMatchingRule (4.37s)5189--- PASS: TestValidateToken_ValidToken (2.75s)5190--- PASS: TestScopes_Rules (4.80s)5191--- PASS: TestValidateToken_KubernetesServiceAccount (3.55s)5192--- PASS: TestScopes_LegacyProviderDefaultsToWrite (4.57s)51932026/11/11 23:40:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42365/oidc5194--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (4.36s)5195--- PASS: TestValidateToken_BoundSubjectMismatch (3.58s)51962026/11/11 23:40:16 http: TLS handshake error from 127.0.0.1:55008: remote error: tls: bad certificate5197--- PASS: TestNewValidator_KubernetesRequiresCA (3.59s)51982026/11/11 23:40:16 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:38659/oidc5199--- PASS: TestValidateToken_NoMatchingProvider (3.68s)52002026/11/11 23:40:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41125/oidc5201--- PASS: TestValidateToken_Expired (3.69s)52022026/11/11 23:40:16 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34891/oidc5203--- PASS: TestValidateToken_WrongAudience (3.86s)52042026/11/11 23:40:16 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:39655/oidc52052026/11/11 23:40:16 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:36793/oidc5206--- PASS: TestValidateToken_MultipleProviders (3.10s)5207PASS5208Running hook tests...5209=== RUN TestSendPathsEmpty5210=== PAUSE TestSendPathsEmpty5211=== RUN TestQueueEnqueueAndFetch5212=== PAUSE TestQueueEnqueueAndFetch5213=== RUN TestQueueDeduplication5214=== PAUSE TestQueueDeduplication5215=== RUN TestQueueRemove5216=== PAUSE TestQueueRemove5217=== RUN TestQueueFetchBatchLimit5218=== PAUSE TestQueueFetchBatchLimit5219=== RUN TestQueueRetryMovesToBack5220=== PAUSE TestQueueRetryMovesToBack5221=== RUN TestQueueFetchRemoveLifecycle5222=== PAUSE TestQueueFetchRemoveLifecycle5223=== RUN TestQueueConcurrentWriters5224=== PAUSE TestQueueConcurrentWriters5225=== RUN TestQueueRemoveLargeClosure5226=== PAUSE TestQueueRemoveLargeClosure5227=== RUN TestServerClientIntegration5228--- PASS: TestServerClientIntegration (0.00s)5229=== RUN TestServerQueueError52302026/11/11 23:40:18 ERROR Failed to queue paths error="permission denied" count=15231--- PASS: TestServerQueueError (0.00s)5232=== RUN TestGetListenerSocketActivation5233 server_test.go:220: === RUN TestGetListenerSocketActivation5234 --- PASS: TestGetListenerSocketActivation (0.00s)5235 PASS5236 5237--- PASS: TestGetListenerSocketActivation (1.37s)5238=== RUN TestServerStalledClientDoesNotBlockShutdown5239=== PAUSE TestServerStalledClientDoesNotBlockShutdown5240=== RUN TestServerBacksOffOnAcceptErrors5241=== PAUSE TestServerBacksOffOnAcceptErrors5242=== RUN TestDrainIsolatesPoisonPath5243=== PAUSE TestDrainIsolatesPoisonPath5244=== RUN TestRunNotBlockedByPoisonHead5245=== PAUSE TestRunNotBlockedByPoisonHead5246=== RUN TestDrainGivesUpWhenServerDown5247=== PAUSE TestDrainGivesUpWhenServerDown5248=== RUN TestFailedPathPrunedByLaterClosure5249=== PAUSE TestFailedPathPrunedByLaterClosure5250=== RUN TestWorkerUploadsAndRemoves5251=== PAUSE TestWorkerUploadsAndRemoves5252=== RUN TestWorkerSkipsGCdPaths5253=== PAUSE TestWorkerSkipsGCdPaths5254=== RUN TestWorkerPrunesClosureDeps5255=== PAUSE TestWorkerPrunesClosureDeps5256=== RUN TestDrainTimeout5257=== PAUSE TestDrainTimeout5258=== RUN TestWorkerRemoveFailureIsNotProgress5259=== PAUSE TestWorkerRemoveFailureIsNotProgress5260=== RUN TestWorkerKeepsPathItCannotStat5261=== PAUSE TestWorkerKeepsPathItCannotStat5262=== CONT TestSendPathsEmpty5263--- PASS: TestSendPathsEmpty (0.00s)5264=== CONT TestWorkerKeepsPathItCannotStat5265=== CONT TestWorkerRemoveFailureIsNotProgress5266=== CONT TestDrainTimeout5267=== CONT TestWorkerPrunesClosureDeps5268=== CONT TestWorkerSkipsGCdPaths5269=== CONT TestWorkerUploadsAndRemoves5270=== CONT TestFailedPathPrunedByLaterClosure5271=== CONT TestDrainGivesUpWhenServerDown5272=== CONT TestRunNotBlockedByPoisonHead5273=== CONT TestDrainIsolatesPoisonPath5274=== CONT TestServerBacksOffOnAcceptErrors5275=== CONT TestServerStalledClientDoesNotBlockShutdown5276=== CONT TestQueueRemoveLargeClosure5277=== CONT TestQueueConcurrentWriters5278=== CONT TestQueueFetchRemoveLifecycle5279=== CONT TestQueueRetryMovesToBack5280=== CONT TestQueueFetchBatchLimit5281=== CONT TestQueueRemove5282=== CONT TestQueueDeduplication5283=== CONT TestQueueEnqueueAndFetch52842026/11/11 23:40:20 INFO Upload queue status pending=252852026/11/11 23:40:20 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths897800010/002/nonexistent52862026/11/11 23:40:20 INFO Uploading batch count=152872000/01/01 00:00:00 INFO Uploading batch count=252882000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=252892000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=452902026/11/11 23:40:21 INFO Upload queue status pending=252912026/11/11 23:40:21 INFO Uploading batch count=152922026/11/11 23:40:21 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1817816885/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1817816885/002/locked/aaa: permission denied"52932000/01/01 00:00:00 ERROR Accept failed error="too many open files"52942000/01/01 00:00:00 ERROR Accept failed error="too many open files"52952000/01/01 00:00:00 ERROR Accept failed error="too many open files"52962000/01/01 00:00:00 ERROR Accept failed error="too many open files"52972000/01/01 00:00:00 ERROR Accept failed error="too many open files"52982000/01/01 00:00:00 ERROR Accept failed error="too many open files"52992000/01/01 00:00:00 ERROR Accept failed error="too many open files"53002000/01/01 00:00:00 ERROR Accept failed error="too many open files"5301--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5302--- PASS: TestWorkerSkipsGCdPaths (1.96s)53032026/11/11 23:40:21 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1817816885/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1817816885/002/locked/aaa: permission denied"53042026/11/11 23:40:21 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2372868230/002/nonexistent53052026/11/11 23:40:21 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1817816885/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1817816885/002/locked/aaa: permission denied"53062026/11/11 23:40:21 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153072026/11/11 23:40:21 ERROR Drain finished with paths left in queue remaining=153082026/11/11 23:40:21 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2372868230/002/nonexistent5309--- PASS: TestDrainTimeout (1.98s)53102026/11/11 23:40:21 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153112026/11/11 23:40:21 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress2372868230/002/nonexistent53122026/11/11 23:40:21 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153132026/11/11 23:40:21 ERROR Drain finished with paths left in queue remaining=153142026/11/11 23:40:21 INFO Upload queue status pending=253152026/11/11 23:40:21 INFO Uploading batch count=253162026/11/11 23:40:21 INFO Uploading batch count=253172026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=253182026/11/11 23:40:21 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3295390047/002/a5319--- PASS: TestWorkerRemoveFailureIsNotProgress (2.00s)53202026/11/11 23:40:21 INFO Uploading batch count=153212026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=153222026/11/11 23:40:21 INFO Uploading batch count=453232026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=453242026/11/11 23:40:21 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3295390047/002/b5325--- PASS: TestWorkerKeepsPathItCannotStat (2.01s)5326--- PASS: TestQueueRemove (0.02s)53272026/11/11 23:40:21 INFO Uploading batch count=153282026/11/11 23:40:21 INFO Upload queue status pending=353292026/11/11 23:40:21 INFO Uploading batch count=15330--- PASS: TestQueueEnqueueAndFetch (0.02s)53312026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=153322026/11/11 23:40:21 INFO Uploading batch count=253332026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=253342026/11/11 23:40:21 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3295390047/002/c53352026/11/11 23:40:21 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath111000723/002/bbb5336--- PASS: TestQueueRetryMovesToBack (0.02s)5337--- PASS: TestWorkerPrunesClosureDeps (1.98s)5338--- PASS: TestQueueDeduplication (0.02s)53392026/11/11 23:40:21 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3295390047/002/d5340--- PASS: TestQueueFetchBatchLimit (0.02s)53412026/11/11 23:40:21 INFO Uploading batch count=15342--- PASS: TestQueueFetchRemoveLifecycle (0.02s)53432026/11/11 23:40:21 INFO Uploading batch count=253442026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=253452026/11/11 23:40:21 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3295390047/002/e53462026/11/11 23:40:21 INFO Uploading batch count=153472026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=153482026/11/11 23:40:21 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown3295390047/002/f53492026/11/11 23:40:21 INFO Uploading batch count=153502026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=153512026/11/11 23:40:21 ERROR Drain finished with paths left in queue remaining=105352--- PASS: TestFailedPathPrunedByLaterClosure (1.93s)53532026/11/11 23:40:21 INFO Uploading batch count=153542026/11/11 23:40:21 ERROR Upload failed error="upload failed" count=153552026/11/11 23:40:21 ERROR Drain finished with paths left in queue remaining=15356--- PASS: TestDrainGivesUpWhenServerDown (1.93s)5357--- PASS: TestDrainIsolatesPoisonPath (0.03s)5358--- PASS: TestWorkerUploadsAndRemoves (1.94s)5359--- PASS: TestQueueRemoveLargeClosure (0.12s)5360--- PASS: TestQueueConcurrentWriters (0.13s)53612026/11/11 23:40:22 ERROR Failed to decode request error="read unix /build/hook3472838465/test.sock->@: i/o timeout"53622026/11/11 23:40:22 ERROR Failed to write response error="write unix /build/hook3472838465/test.sock->@: i/o timeout"5363--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)53642026/11/11 23:40:22 INFO Uploading batch count=153652026/11/11 23:40:22 INFO Uploading batch count=153662026/11/11 23:40:22 INFO Uploading batch count=153672026/11/11 23:40:22 ERROR Upload failed error="upload failed" count=153682026/11/11 23:40:22 INFO Uploading batch count=153692026/11/11 23:40:22 ERROR Upload failed error="upload failed" count=153702026/11/11 23:40:22 INFO Uploading batch count=153712026/11/11 23:40:22 ERROR Upload failed error="upload failed" count=153722026/11/11 23:40:22 INFO Uploading batch count=153732026/11/11 23:40:22 ERROR Upload failed error="upload failed" count=153742026/11/11 23:40:22 ERROR Drain finished with paths left in queue remaining=15375--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5376PASS5377Running signing tests...5378=== RUN TestExternalSigner5379=== PAUSE TestExternalSigner5380=== RUN TestExternalSignerWrongPublicKey5381=== PAUSE TestExternalSignerWrongPublicKey5382=== RUN TestExternalSignerErrors5383=== PAUSE TestExternalSignerErrors5384=== RUN TestExternalSignerRestartsAfterCrash5385=== PAUSE TestExternalSignerRestartsAfterCrash5386=== RUN TestExternalSignerProgramErrorKeepsProcess5387=== PAUSE TestExternalSignerProgramErrorKeepsProcess5388=== RUN TestExternalSignerClose5389=== PAUSE TestExternalSignerClose5390=== RUN TestExternalSignerRejectsInvalidPublicKey5391=== PAUSE TestExternalSignerRejectsInvalidPublicKey5392=== RUN TestExternalSignerCancellation5393=== PAUSE TestExternalSignerCancellation5394=== RUN TestGenerateFingerprint5395=== PAUSE TestGenerateFingerprint5396=== RUN TestParseSigningKey5397=== PAUSE TestParseSigningKey5398=== RUN TestSign5399=== PAUSE TestSign5400=== RUN TestSignBatch5401=== PAUSE TestSignBatch5402=== RUN TestNilKeyPublicKey5403=== PAUSE TestNilKeyPublicKey5404=== RUN TestSignErrors5405=== PAUSE TestSignErrors5406=== CONT TestExternalSigner5407=== CONT TestExternalSignerProgramErrorKeepsProcess5408=== CONT TestExternalSignerCancellation5409=== CONT TestSignErrors5410=== CONT TestNilKeyPublicKey5411--- PASS: TestNilKeyPublicKey (0.00s)5412=== CONT TestSignBatch5413=== CONT TestSign5414=== CONT TestParseSigningKey5415=== RUN TestParseSigningKey/valid_32-byte_key5416=== CONT TestGenerateFingerprint5417=== RUN TestGenerateFingerprint/basic_with_references5418=== PAUSE TestGenerateFingerprint/basic_with_references5419=== RUN TestGenerateFingerprint/no_references5420=== PAUSE TestGenerateFingerprint/no_references5421=== CONT TestExternalSignerRejectsInvalidPublicKey5422=== RUN TestExternalSignerRejectsInvalidPublicKey/#005423=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005424=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5425=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5426=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5427=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5428=== CONT TestExternalSignerErrors5429=== CONT TestExternalSignerRestartsAfterCrash5430=== CONT TestExternalSignerWrongPublicKey5431=== CONT TestExternalSignerClose5432=== PAUSE TestParseSigningKey/valid_32-byte_key5433=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5434=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5435=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5436=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5437=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5438=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5439=== CONT TestExternalSignerRejectsInvalidPublicKey/#005440=== RUN TestParseSigningKey/no_colon5441=== PAUSE TestParseSigningKey/no_colon5442=== RUN TestParseSigningKey/empty_name5443=== PAUSE TestParseSigningKey/empty_name5444=== RUN TestParseSigningKey/invalid_base645445=== PAUSE TestParseSigningKey/invalid_base645446=== RUN TestParseSigningKey/wrong_length5447=== PAUSE TestParseSigningKey/wrong_length5448=== CONT TestParseSigningKey/valid_32-byte_key5449=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5450=== CONT TestParseSigningKey/empty_name5451=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5452=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5453=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5454=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5455=== RUN TestGenerateFingerprint/invalid_nar_hash_length5456=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5457=== RUN TestGenerateFingerprint/invalid_store_path_prefix5458=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5459=== RUN TestGenerateFingerprint/invalid_reference_prefix5460=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5461=== CONT TestGenerateFingerprint/basic_with_references5462=== CONT TestParseSigningKey/wrong_length5463=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5464=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5465=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5466--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5467 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5468 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5469 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5470 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5471 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5472=== CONT TestGenerateFingerprint/invalid_reference_prefix5473=== CONT TestGenerateFingerprint/invalid_store_path_prefix5474=== CONT TestGenerateFingerprint/invalid_nar_hash_length5475=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5476=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5477=== CONT TestGenerateFingerprint/no_references5478--- PASS: TestGenerateFingerprint (0.00s)5479 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5480 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5481 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5482 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5483 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5484 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5485 --- PASS: TestGenerateFingerprint/no_references (0.00s)5486=== CONT TestParseSigningKey/no_colon5487=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5488=== CONT TestParseSigningKey/invalid_base645489=== RUN TestExternalSignerErrors/exit_at_startup5490=== PAUSE TestExternalSignerErrors/exit_at_startup5491=== RUN TestExternalSignerErrors/wrong_id5492=== PAUSE TestExternalSignerErrors/wrong_id5493=== RUN TestExternalSignerErrors/wrong_signature_name5494=== PAUSE TestExternalSignerErrors/wrong_signature_name5495=== RUN TestExternalSignerErrors/invalid_signature_base645496=== PAUSE TestExternalSignerErrors/invalid_signature_base645497=== RUN TestExternalSignerErrors/wrong_shape5498=== PAUSE TestExternalSignerErrors/wrong_shape5499=== RUN TestExternalSignerErrors/null5500=== PAUSE TestExternalSignerErrors/null5501=== RUN TestExternalSignerErrors/too_few_signatures5502=== PAUSE TestExternalSignerErrors/too_few_signatures5503=== RUN TestExternalSignerErrors/unrequested_key5504=== PAUSE TestExternalSignerErrors/unrequested_key5505=== RUN TestExternalSignerErrors/missing_signature_name5506=== PAUSE TestExternalSignerErrors/missing_signature_name5507=== RUN TestExternalSignerErrors/program_error5508=== PAUSE TestExternalSignerErrors/program_error5509=== RUN TestExternalSignerErrors/exit_on_request5510=== PAUSE TestExternalSignerErrors/exit_on_request5511=== RUN TestExternalSignerErrors/missing_signatures5512=== PAUSE TestExternalSignerErrors/missing_signatures5513=== RUN TestExternalSignerErrors/wrong_signatures_shape5514=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5515=== RUN TestExternalSignerErrors/wrong_signature_length5516=== PAUSE TestExternalSignerErrors/wrong_signature_length5517=== RUN TestExternalSignerErrors/invalid_JSON5518=== PAUSE TestExternalSignerErrors/invalid_JSON5519=== RUN TestExternalSignerErrors/null_signatures5520=== PAUSE TestExternalSignerErrors/null_signatures5521=== CONT TestExternalSignerErrors/exit_at_startup5522=== CONT TestExternalSignerErrors/missing_signature_name5523=== CONT TestExternalSignerErrors/wrong_signatures_shape5524=== RUN TestSignErrors/nil_key5525=== CONT TestExternalSignerErrors/null_signatures5526=== PAUSE TestSignErrors/nil_key5527=== RUN TestSignErrors/nil_narinfo5528=== CONT TestExternalSignerErrors/invalid_JSON5529=== PAUSE TestSignErrors/nil_narinfo5530=== RUN TestSignErrors/invalid_narinfo5531=== PAUSE TestSignErrors/invalid_narinfo5532=== RUN TestSignErrors/canceled5533=== PAUSE TestSignErrors/canceled5534=== CONT TestExternalSignerErrors/exit_on_request5535=== CONT TestExternalSignerErrors/wrong_signature_length5536--- PASS: TestSign (0.00s)5537=== CONT TestExternalSignerErrors/missing_signatures5538=== CONT TestExternalSignerErrors/wrong_shape5539=== CONT TestExternalSignerErrors/unrequested_key5540=== CONT TestExternalSignerErrors/too_few_signatures5541=== CONT TestExternalSignerErrors/null5542--- PASS: TestSignBatch (0.00s)5543=== CONT TestExternalSignerErrors/program_error5544=== CONT TestExternalSignerErrors/wrong_signature_name5545=== CONT TestExternalSignerErrors/invalid_signature_base645546=== CONT TestExternalSignerErrors/wrong_id5547=== CONT TestSignErrors/nil_key5548=== CONT TestSignErrors/invalid_narinfo5549=== CONT TestSignErrors/canceled5550=== CONT TestSignErrors/nil_narinfo5551--- PASS: TestParseSigningKey (0.00s)5552 --- PASS: TestParseSigningKey/empty_name (0.00s)5553 --- PASS: TestParseSigningKey/wrong_length (0.00s)5554 --- PASS: TestParseSigningKey/no_colon (0.00s)5555 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5556 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5557 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5558--- PASS: TestSignErrors (0.00s)5559 --- PASS: TestSignErrors/nil_key (0.00s)5560 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5561 --- PASS: TestSignErrors/canceled (0.00s)5562 --- PASS: TestSignErrors/nil_narinfo (0.00s)5563--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.02s)5564--- PASS: TestExternalSignerWrongPublicKey (0.02s)5565--- PASS: TestExternalSignerClose (0.02s)5566--- PASS: TestExternalSigner (0.02s)5567--- PASS: TestExternalSignerErrors (0.00s)5568 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5569 --- PASS: TestExternalSignerErrors/null (0.01s)5570 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5571 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5572 --- PASS: TestExternalSignerErrors/missing_signatures (0.02s)5573 --- PASS: TestExternalSignerErrors/null_signatures (0.02s)5574 --- PASS: TestExternalSignerErrors/program_error (0.02s)5575 --- PASS: TestExternalSignerErrors/exit_at_startup (0.02s)5576 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.02s)5577 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.02s)5578 --- PASS: TestExternalSignerErrors/exit_on_request (0.02s)5579 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.02s)5580 --- PASS: TestExternalSignerErrors/wrong_id (0.02s)5581 --- PASS: TestExternalSignerErrors/invalid_JSON (0.02s)5582 --- PASS: TestExternalSignerErrors/too_few_signatures (0.02s)5583 --- PASS: TestExternalSignerErrors/wrong_shape (0.02s)5584--- PASS: TestExternalSignerCancellation (0.03s)5585--- PASS: TestExternalSignerRestartsAfterCrash (0.02s)5586PASS5587Running OpenBao signer tests...5588=== RUN TestSign5589=== PAUSE TestSign5590=== RUN TestSignErrors5591=== PAUSE TestSignErrors5592=== RUN TestServe5593=== PAUSE TestServe5594=== RUN TestServeRejectsGarbage5595=== PAUSE TestServeRejectsGarbage5596=== CONT TestSign5597=== CONT TestServeRejectsGarbage5598=== CONT TestServe5599--- PASS: TestServeRejectsGarbage (0.00s)5600=== CONT TestSignErrors5601=== RUN TestSignErrors/no_prefix5602=== PAUSE TestSignErrors/no_prefix5603=== RUN TestSignErrors/invalid_base645604=== PAUSE TestSignErrors/invalid_base645605=== RUN TestSignErrors/http_error5606=== PAUSE TestSignErrors/http_error5607=== RUN TestSignErrors/invalid_json5608=== PAUSE TestSignErrors/invalid_json5609=== CONT TestSignErrors/no_prefix5610=== CONT TestSignErrors/invalid_json5611=== CONT TestSignErrors/http_error5612=== CONT TestSignErrors/invalid_base645613--- PASS: TestServe (0.00s)5614--- PASS: TestSign (1.79s)5615--- PASS: TestSignErrors (0.00s)5616 --- PASS: TestSignErrors/invalid_base64 (1.79s)5617 --- PASS: TestSignErrors/no_prefix (1.79s)5618 --- PASS: TestSignErrors/http_error (1.79s)5619 --- PASS: TestSignErrors/invalid_json (1.79s)5620PASS

succeeded niks3-go-unit-tests checks.aarch64-linux.go-unit-tests · build #491 · raw

succeeded `niks3-go-unit-tests` `checks.aarch64-linux.go-unit-tests` · build #491 · raw