niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #497
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestEncodeNixBase32119=== CONT TestRetryTimeoutOutlastsMaxRetries120=== CONT TestPathInfoHashCompatibility121=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)122=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)123=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon124=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon125=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI126=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI127=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512128=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512129=== CONT TestPathInfoCACompatibility130=== RUN TestPathInfoCACompatibility/null_ca_field131=== PAUSE TestPathInfoCACompatibility/null_ca_field132=== RUN TestPathInfoCACompatibility/old_string_format_-_text133=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text134=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive135=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive136=== RUN TestPathInfoCACompatibility/new_structured_format_-_text137=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text138=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method139=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method140=== CONT TestUploadMultipart_ReturnsAfterDroppedBody141=== CONT TestScriptTokenEmptyCommand142--- PASS: TestScriptTokenEmptyCommand (0.00s)143=== CONT TestStreamPushReportsSkippedPaths144=== CONT TestParsePathInfoJSONMultiplePaths145=== CONT TestFileTokenMissing146=== CONT TestDoWithRetry_BodyReplayedViaGetBody147=== CONT TestResolveStorePath148=== CONT TestScriptTokenScriptFails149=== CONT TestRegisterUploadedObject_OutlivesPush150=== CONT TestParsePathInfoJSON151=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer152=== CONT TestScriptTokenBadJSON153=== CONT TestUploadMultipart_PartsInParallel154=== CONT TestScriptTokenEmptyToken155=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess156=== CONT TestScriptTokenCachesUntilRefresh157=== CONT TestDumpPathWriterError158=== CONT TestScriptTokenNoExpiryRerunsEveryCall159=== CONT TestDumpPathSingleFile160=== CONT TestRateLimiterFeedback161=== CONT TestDumpPathMatchesNix162=== CONT TestFileTokenEmpty163=== RUN TestEncodeNixBase32/test_string_hash164=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths165=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths166=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths167--- PASS: TestFileTokenMissing (0.00s)168=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths169=== CONT TestFileTokenReadsAndCaches170=== PAUSE TestEncodeNixBase32/test_string_hash171=== RUN TestEncodeNixBase32/empty_input172=== PAUSE TestEncodeNixBase32/empty_input173=== CONT TestSetClientTLSErrors174=== RUN TestParsePathInfoJSON/Nix_format175=== CONT TestStaticToken176--- PASS: TestStaticToken (0.00s)177=== RUN TestRateLimiterFeedback/429_enables_limiter178=== CONT TestSetClientTLSDoesNotMutateDefaultTransport179--- PASS: TestResolveStorePath (0.00s)180=== CONT TestSetClientTLS1812026/11/12 00:20:20 WARN Rate limiter enabled after throttle name=server-test rate=5182--- PASS: TestScriptTokenScriptFails (0.00s)183=== CONT TestStreamPushStopsWhenCanceled184--- PASS: TestFileTokenReadsAndCaches (0.00s)185=== RUN TestSetClientTLSErrors/missing_cert_file186--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)187=== CONT TestStreamPushAnswersEveryLineWhenCanceled188=== RUN TestSetClientTLS/rejects_connection_without_client_cert189--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)190=== CONT TestStreamPushRequestLine191=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert192--- PASS: TestStreamPushStopsWhenCanceled (0.03s)193=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA194=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA195=== RUN TestSetClientTLS/preserves_debug_logging_transport196=== CONT TestClientSignaturesByStorePath1972000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2198--- PASS: TestClientSignaturesByStorePath (0.00s)199=== CONT TestStreamPushGivesUpOnDeadServer200=== PAUSE TestRateLimiterFeedback/429_enables_limiter201=== CONT TestStreamPushReportsSignatures202--- PASS: TestStreamPushReportsSkippedPaths (0.03s)203=== PAUSE TestSetClientTLS/preserves_debug_logging_transport204--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.02s)2052026/11/12 00:20:20 ERROR Upload failed error="connection refused" count=20206=== PAUSE TestSetClientTLSErrors/missing_cert_file2072026/11/12 00:20:20 ERROR Server seems unavailable, giving up on batch untried=172082026/11/12 00:20:20 ERROR Upload failed error=boom count=1209=== PAUSE TestParsePathInfoJSON/Nix_format2102026/11/12 00:20:20 ERROR Upload failed error=boom count=1211=== RUN TestRateLimiterFeedback/503_enables_limiter2122026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:42097213=== PAUSE TestRateLimiterFeedback/503_enables_limiter2142026/11/12 00:20:20 WARN Rate limiter enabled after throttle name=server-test rate=52152026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:44967216=== CONT TestPartSizeForNAR217=== CONT TestConvertHashToNix32218=== RUN TestConvertHashToNix32/SRI_format_to_Nix32219=== RUN TestPartSizeForNAR/zero_stays_at_minimum220=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun221=== CONT TestShellSplitErrors222=== CONT TestDoWithRetry_FinalResponseBodyReadable223--- PASS: TestScriptTokenBadJSON (0.03s)2242026/11/12 00:20:20 WARN Rate limiter backed off name=server-test rate=5225--- PASS: TestStreamPushReportsSignatures (0.00s)226--- PASS: TestScriptTokenEmptyToken (0.03s)227=== CONT TestFilterOversizedClosures2282026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:44967229=== CONT TestShellSplit230=== RUN TestFilterOversizedClosures/no_limit_keeps_everything231=== CONT TestUploadMultipart_FailedProducerIsNotCompleted232=== RUN TestSetClientTLSErrors/missing_key_file233=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything234=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part235=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped236=== RUN TestParsePathInfoJSON/Lix_format237=== CONT TestStreamPushIsolatesFailures238=== CONT TestEncodeNixBase32WithRealHash239=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry240=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter2412026/11/12 00:20:20 ERROR Upload failed error="bad path" count=3242=== CONT TestStreamPushBatchesUnderLoad243=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32244=== RUN TestConvertHashToNix32/already_Nix32_format245=== PAUSE TestConvertHashToNix32/already_Nix32_format246=== RUN TestConvertHashToNix32/invalid_format2472026/11/12 00:20:20 WARN Rate limiter enabled after throttle name=server-test rate=5248=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum249=== RUN TestPartSizeForNAR/small_stays_at_minimum2502026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:38041251=== CONT TestGetStorePathHash2522026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:42097253=== CONT TestStreamPushReportsEveryPath254=== RUN TestGetStorePathHash/valid_store_path255--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)2562026/11/12 00:20:20 INFO Garbage collection started257--- PASS: TestRegisterUploadedObject_OutlivesPush (0.03s)2582026/11/12 00:20:20 WARN Rate limiter backed off name=server-test rate=5259=== CONT TestCaseHackSuffix260=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part261=== PAUSE TestSetClientTLSErrors/missing_key_file262=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped263=== PAUSE TestParsePathInfoJSON/Lix_format264=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter265=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter266=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica267=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false268=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false269=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true270=== PAUSE TestConvertHashToNix32/invalid_format271=== PAUSE TestPartSizeForNAR/small_stays_at_minimum272=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum273=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum274=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts275=== PAUSE TestGetStorePathHash/valid_store_path276--- PASS: TestDoServerRequestAttachesToken (0.03s)277=== CONT TestRegisterUploadedObjectReusesConnections278=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary279=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary280=== RUN TestSetClientTLSErrors/missing_ca_file281=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha5122822026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:420972832026/11/12 00:20:20 INFO Garbage collection started284=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI285=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)286=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon287=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive288=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method2892026/11/12 00:20:20 WARN Rate limiter enabled after throttle name=s3-test rate=52902026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:36211291=== CONT TestPathInfoCACompatibility/old_string_format_-_text292=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths293=== CONT TestEncodeNixBase32/test_string_hash294=== CONT TestEncodeNixBase32/empty_input295=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths296=== CONT TestSetClientTLS/rejects_connection_without_client_cert297=== CONT TestPathInfoCACompatibility/new_structured_format_-_text298=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA299=== RUN TestParsePathInfoJSON/empty_input300=== PAUSE TestParsePathInfoJSON/empty_input301=== RUN TestParsePathInfoJSON/whitespace_only302=== PAUSE TestParsePathInfoJSON/whitespace_only303=== RUN TestParsePathInfoJSON/invalid_JSON3042026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:42097305=== PAUSE TestParsePathInfoJSON/invalid_JSON306=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter307=== CONT TestSetClientTLS/preserves_debug_logging_transport308=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true309=== CONT TestPathInfoCACompatibility/null_ca_field310=== CONT TestConvertHashToNix32/already_Nix32_format311=== CONT TestConvertHashToNix32/SRI_format_to_Nix32312=== CONT TestParsePathInfoJSON/Nix_format313=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary314=== CONT TestRateLimiterFeedback/429_enables_limiter3152026/11/12 00:20:20 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0316=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part317=== RUN TestGetStorePathHash/basename_without_hyphen_should_error318=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error3192026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:42097320=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error321=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error3222026/11/12 00:20:20 WARN Rate limiter enabled after throttle name=server-test rate=5323=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error3242026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:36377325--- PASS: TestFileTokenEmpty (0.03s)326--- PASS: TestShellSplitErrors (0.00s)327--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.03s)328--- PASS: TestShellSplit (0.00s)329--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.03s)330--- PASS: TestEncodeNixBase32WithRealHash (0.00s)331--- PASS: TestStreamPushIsolatesFailures (0.00s)332--- PASS: TestStreamPushReportsEveryPath (0.00s)333--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.00s)3342026/11/12 00:20:20 WARN Garbage collection finished on another replica. Its statistics are not available here335=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error3362026/11/12 00:20:20 WARN Rate limiter backed off name=server-test rate=5337=== PAUSE TestSetClientTLSErrors/missing_ca_file338=== RUN TestSetClientTLSErrors/invalid_ca_file339=== PAUSE TestSetClientTLSErrors/invalid_ca_file340=== CONT TestParsePathInfoJSON/invalid_JSON341=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter342--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.00s)343=== CONT TestParsePathInfoJSON/empty_input344=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter345=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts346=== RUN TestPartSizeForNAR/1_TiB3472026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:42097348=== PAUSE TestPartSizeForNAR/1_TiB349=== RUN TestPartSizeForNAR/5_TiB_S3_max_object350=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object351=== RUN TestPartSizeForNAR/capped_at_5_GiB352=== PAUSE TestPartSizeForNAR/capped_at_5_GiB353=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false354=== CONT TestRateLimiterFeedback/503_enables_limiter355=== CONT TestGetStorePathHash/valid_store_path356=== CONT TestParsePathInfoJSON/whitespace_only357=== RUN TestFilterOversizedClosures/all_closures_skipped358=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true359=== CONT TestSetClientTLSErrors/missing_cert_file360=== CONT TestSetClientTLSErrors/missing_key_file361=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error3622026/11/12 00:20:20 WARN Rate limiter enabled after throttle name=server-test rate=5363=== CONT TestPartSizeForNAR/zero_stays_at_minimum3642026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:34579365=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum366=== CONT TestPartSizeForNAR/small_stays_at_minimum3672026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:42097368=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts369--- PASS: TestEncodeNixBase32 (0.00s)370 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)371 --- PASS: TestEncodeNixBase32/empty_input (0.00s)372--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)3732026/11/12 00:20:20 WARN Rate limiter backed off name=server-test rate=5374=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error375=== CONT TestParsePathInfoJSON/Lix_format376=== CONT TestSetClientTLSErrors/invalid_ca_file377=== PAUSE TestFilterOversizedClosures/all_closures_skipped378=== CONT TestConvertHashToNix32/invalid_format379=== CONT TestSetClientTLSErrors/missing_ca_file3802026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:42097381=== CONT TestGetStorePathHash/basename_without_hyphen_should_error382=== CONT TestPartSizeForNAR/capped_at_5_GiB383--- PASS: TestPathInfoHashCompatibility (0.00s)384 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)385 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)386 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)387 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)388--- PASS: TestStreamPushBatchesUnderLoad (0.00s)389 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)390 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)391=== CONT TestPartSizeForNAR/5_TiB_S3_max_object392=== CONT TestPartSizeForNAR/1_TiB393=== CONT TestFilterOversizedClosures/no_limit_keeps_everything394=== CONT TestFilterOversizedClosures/all_closures_skipped3952026/11/12 00:20:20 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-small nar_size=1000 max_nar_size=50396--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.00s)397 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)398 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)399--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)400 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)401 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)402--- PASS: TestRateLimiterFeedback (0.03s)403 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)404 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)405 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)406 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)407--- PASS: TestPathInfoCACompatibility (0.00s)408 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)409 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)410 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)411 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)412 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)4132026/11/12 00:20:20 http: TLS handshake error from 127.0.0.1:54546: remote error: tls: bad certificate414--- PASS: TestConvertHashToNix32 (0.00s)415 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)416 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)417 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)418--- PASS: TestParsePathInfoJSON (0.03s)419 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)420 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)421 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)422 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)423 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)424=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped4252026/11/12 00:20:20 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000426--- PASS: TestPartSizeForNAR (0.01s)427 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)428 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)429 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)430 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)431 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)432 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)433 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)434--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)435--- PASS: TestGetStorePathHash (0.01s)436 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)437 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)438 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)439 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)440--- PASS: TestFilterOversizedClosures (0.01s)441 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)442 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)443 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)444--- PASS: TestSetClientTLSErrors (0.04s)445 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)446 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)447 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)448 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)449--- PASS: TestScriptTokenCachesUntilRefresh (0.04s)450--- PASS: TestSetClientTLS (0.03s)451 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)452 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)453 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)4542026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:42097455--- PASS: TestStreamPushRequestLine (0.03s)4562026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:420974572026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:420974582026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:420974592026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:420974602026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:420974612026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:42097462--- PASS: TestDumpPathSingleFile (0.06s)4632026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:420974642026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:420974652026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:42097466--- PASS: TestCaseHackSuffix (0.03s)4672026/11/12 00:20:20 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:42097468--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.07s)469--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)470--- PASS: TestDumpPathWriterError (0.07s)4712026/11/12 00:20:20 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:40595/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:40595/api/uploads/complete4722026/11/12 00:20:20 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4732026/11/12 00:20:20 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:40595/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:40595/api/uploads/complete4742026/11/12 00:20:20 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"475--- PASS: TestDumpPathMatchesNix (0.10s)4762026/11/12 00:20:20 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:40595/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:40595/api/uploads/complete4772026/11/12 00:20:20 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"478--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.13s)479--- PASS: TestUploadMultipart_PartsInParallel (0.65s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "nixbld".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /build/postgres3527985378/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /build/postgres3527985378/data -l logfile start509510/build/postgres3527985378:5432 - no response5112026-11-12 00:20:22.665 UTC [130] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit5122026-11-12 00:20:22.665 UTC [130] LOG: listening on Unix socket "/build/postgres3527985378/.s.PGSQL.5432"5132026-11-12 00:20:22.670 UTC [137] LOG: database system was shut down at 2026-11-12 00:20:22 UTC5142026-11-12 00:20:22.674 UTC [130] LOG: database system is ready to accept connections515/build/postgres3527985378:5432 - accepting connections516{"timestamp":"2026-11-12T00:20:22.969572747Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"0e2eaab0-6564-40b3-bc3c-cdc009dfc211","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(389)"}517=== RUN TestAbortSucceeded518=== PAUSE TestAbortSucceeded519=== RUN TestService_AuthMiddleware520=== PAUSE TestService_AuthMiddleware521=== RUN TestService_AuthMiddleware_MTLSProxyHeader522=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader523=== RUN TestService_AuthMiddleware_MTLSBoundSubjects524=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects525=== RUN TestService_ReadAuthMiddleware526=== PAUSE TestService_ReadAuthMiddleware527=== RUN TestService_AuthMiddleware_OIDC528=== PAUSE TestService_AuthMiddleware_OIDC529=== RUN TestService_RequireScope_OIDC530=== PAUSE TestService_RequireScope_OIDC531=== RUN TestService_ReadScope_PublicByDefault532=== PAUSE TestService_ReadScope_PublicByDefault533=== RUN TestCacheConfigHandler534=== PAUSE TestCacheConfigHandler535=== RUN TestCacheStatsHandler536=== PAUSE TestCacheStatsHandler537=== RUN TestClientCADerivations538=== PAUSE TestClientCADerivations539=== RUN TestClientErrorHandling540=== PAUSE TestClientErrorHandling541=== RUN TestClientIntegration542=== PAUSE TestClientIntegration543=== RUN TestClientMultipleUploads544=== PAUSE TestClientMultipleUploads545=== RUN TestClientWithDependencies546=== PAUSE TestClientWithDependencies547=== RUN TestClientSharedPathCommittedMidPush548=== PAUSE TestClientSharedPathCommittedMidPush549=== RUN TestPinProtectsFromGC550=== PAUSE TestPinProtectsFromGC551=== RUN TestClientPushesUseOnePush552=== PAUSE TestClientPushesUseOnePush553=== RUN TestClientFallsBackToClosures554=== PAUSE TestClientFallsBackToClosures555=== RUN TestDeleteClosuresRacingPins556=== PAUSE TestDeleteClosuresRacingPins557=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock558=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock559=== RUN TestResolveDBConnectionString560=== PAUSE TestResolveDBConnectionString561=== RUN TestConnectWaitsForAPeerMigration562=== PAUSE TestConnectWaitsForAPeerMigration563=== RUN TestLeadElectsOneAndHandsOver564=== PAUSE TestLeadElectsOneAndHandsOver565=== RUN TestLeadIncumbentWinsAfterRestart5662026/11/12 00:20:23 INFO lead: acquired remote=192.0.2.1:1234 generation=15672026/11/12 00:20:23 INFO lead: released remote=192.0.2.1:1234 generation=15682026/11/12 00:20:23 INFO lead: acquired remote=192.0.2.1:1234 generation=25692026/11/12 00:20:24 INFO lead: released remote=192.0.2.1:1234 generation=2570--- PASS: TestLeadIncumbentWinsAfterRestart (1.89s)571=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent572=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent573=== RUN TestLeadIncumbentAnnouncesAtOnce574=== PAUSE TestLeadIncumbentAnnouncesAtOnce575=== RUN TestLeadGenerationGrowsWithEveryGrant576=== PAUSE TestLeadGenerationGrowsWithEveryGrant577=== RUN TestLeadEndsOnShutdown578=== PAUSE TestLeadEndsOnShutdown579=== RUN TestLeadEndsWhenItsConnectionHangs5802026/11/12 00:20:25 INFO lead: acquired remote=192.0.2.1:1234 generation=15812026-11-12 00:20:26.482 UTC [574] FATAL: terminating connection due to administrator command5822026/11/12 00:20:26 INFO lead: acquired remote=192.0.2.1:1234 generation=25832026/11/12 00:20:27 WARN lead: lock connection lost error="timeout: context deadline exceeded"5842026/11/12 00:20:27 INFO lead: released remote=192.0.2.1:1234 generation=15852026/11/12 00:20:28 INFO lead: released remote=192.0.2.1:1234 generation=2586--- PASS: TestLeadEndsWhenItsConnectionHangs (3.11s)587=== RUN TestGCAdvisoryLockBlocksConcurrentRun588=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun589=== RUN TestGCBugBareHashReferences590=== PAUSE TestGCBugBareHashReferences591=== RUN TestGCMetrics592=== PAUSE TestGCMetrics593=== RUN TestPushDedupSurvivesConcurrentGC594=== PAUSE TestPushDedupSurvivesConcurrentGC595=== RUN TestDeduplicatedObjectsRecordedAsPending596=== PAUSE TestDeduplicatedObjectsRecordedAsPending597=== RUN TestGCSweepSkipsPendingObjects598=== PAUSE TestGCSweepSkipsPendingObjects599=== RUN TestTombstonedObjectOfferedWithoutWaiting600=== PAUSE TestTombstonedObjectOfferedWithoutWaiting601=== RUN TestGCSweepDeliversEachKeyOnce602=== PAUSE TestGCSweepDeliversEachKeyOnce603=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection604=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection605=== RUN TestForceGCDuringPushOffersSweptObject606=== PAUSE TestForceGCDuringPushOffersSweptObject607=== RUN TestFailedS3DeleteKeepsTombstone608=== PAUSE TestFailedS3DeleteKeepsTombstone609=== RUN TestSweepSkipsRowsLockedByPush610=== PAUSE TestSweepSkipsRowsLockedByPush611=== RUN TestRegisterCompletedObjectNeedsPendingRow612=== PAUSE TestRegisterCompletedObjectNeedsPendingRow613=== RUN TestGetExistingObjectsLocksRows614=== PAUSE TestGetExistingObjectsLocksRows615=== RUN TestSweepRechecksPendingAfterLocking616=== PAUSE TestSweepRechecksPendingAfterLocking617=== RUN TestPushWaitsForSweepHoldingRow618=== PAUSE TestPushWaitsForSweepHoldingRow619=== RUN TestCommitRacingPendingCleanupKeepsObjects620=== PAUSE TestCommitRacingPendingCleanupKeepsObjects621=== RUN TestGCEndsOnShutdown622=== PAUSE TestGCEndsOnShutdown623=== RUN TestGCTaskStore_StartNew624=== PAUSE TestGCTaskStore_StartNew625=== RUN TestGCTaskStore_DeduplicateSameParams626=== PAUSE TestGCTaskStore_DeduplicateSameParams627=== RUN TestGCTaskStore_ConflictDifferentParams628=== PAUSE TestGCTaskStore_ConflictDifferentParams629=== RUN TestGCTaskStore_GetEmpty630=== PAUSE TestGCTaskStore_GetEmpty631=== RUN TestGCTaskStore_GetReturnsLatest632=== PAUSE TestGCTaskStore_GetReturnsLatest633=== RUN TestGCTaskStore_CompletedAllowsNewTask634=== PAUSE TestGCTaskStore_CompletedAllowsNewTask635=== RUN TestGCTaskStore_PhaseUpdates636=== PAUSE TestGCTaskStore_PhaseUpdates637=== RUN TestGCTaskStore_Fail638=== PAUSE TestGCTaskStore_Fail639=== RUN TestGracefulShutdownDrainsInflight640=== PAUSE TestGracefulShutdownDrainsInflight641=== RUN TestService_healthCheckHandler642=== PAUSE TestService_healthCheckHandler643=== RUN TestService_readinessHandler644=== PAUSE TestService_readinessHandler645=== RUN TestGenerateLandingPage646=== PAUSE TestGenerateLandingPage647=== RUN TestEnsureAbortUploadsRule648=== PAUSE TestEnsureAbortUploadsRule649=== RUN TestCacheConfigHandlerMaxNarSize650=== PAUSE TestCacheConfigHandlerMaxNarSize651=== RUN TestCreatePendingClosureRejectsOversizedNAR652=== PAUSE TestCreatePendingClosureRejectsOversizedNAR653=== RUN TestNARDeduplicationMetadataUploadBug654=== PAUSE TestNARDeduplicationMetadataUploadBug655=== RUN TestMetricsInventory656=== PAUSE TestMetricsInventory657=== RUN TestMetricsCommit658=== PAUSE TestMetricsCommit659=== RUN TestService_NativeMTLS660=== PAUSE TestService_NativeMTLS661=== RUN TestServerTLSConfig662=== PAUSE TestServerTLSConfig663=== RUN TestMultipartCleanup664=== PAUSE TestMultipartCleanup665=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded666=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded667=== RUN TestPendingCleanupUsesOneCutoff668=== PAUSE TestPendingCleanupUsesOneCutoff669=== RUN TestObjectStatsTrigger670=== PAUSE TestObjectStatsTrigger671=== RUN TestObjectStatsUpdatedOncePerCommit672=== PAUSE TestObjectStatsUpdatedOncePerCommit673=== RUN TestOrphanedObjectsGC674=== PAUSE TestOrphanedObjectsGC675=== RUN TestOrphanedObjectsGCStressTest676=== PAUSE TestOrphanedObjectsGCStressTest677=== RUN TestCreatePin_ReservedPins678=== PAUSE TestCreatePin_ReservedPins679=== RUN TestPresentReportsOnlyClosureRoots680=== PAUSE TestPresentReportsOnlyClosureRoots681=== RUN TestPresentRefreshesOnlyStaleClosures682=== PAUSE TestPresentRefreshesOnlyStaleClosures683=== RUN TestPresentNotReportedWhileGCDeletesClosure684=== PAUSE TestPresentNotReportedWhileGCDeletesClosure685=== RUN TestParseSingleRange686=== PAUSE TestParseSingleRange687=== RUN TestProxyHeadersOnlyTrustedOnSocket6882026/11/12 00:20:28 INFO Starting HTTP server address=proxy.sock6892026/11/12 00:20:28 INFO Starting HTTP server address=127.0.0.1:430796902026/11/12 00:20:28 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6912026/11/12 00:20:28 INFO Shutdown signal received, draining in-flight requests timeout=10s692--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)693=== RUN TestIsValidCachePath694=== PAUSE TestIsValidCachePath695=== RUN TestReadProxyNarinfo696=== PAUSE TestReadProxyNarinfo697=== RUN TestReadProxyNarinfoAlreadyDecompressed698=== PAUSE TestReadProxyNarinfoAlreadyDecompressed699=== RUN TestReadProxyContentNegotiation700=== PAUSE TestReadProxyContentNegotiation701=== RUN TestReadProxyNarStreaming702=== PAUSE TestReadProxyNarStreaming703=== RUN TestReadProxy404704=== PAUSE TestReadProxy404705=== RUN TestReadProxyInvalidPath706=== PAUSE TestReadProxyInvalidPath707=== RUN TestReadProxyHead708=== PAUSE TestReadProxyHead709=== RUN TestReadProxyOutlastsServerWriteTimeout710=== PAUSE TestReadProxyOutlastsServerWriteTimeout711=== RUN TestReadProxyConditionalGet712=== PAUSE TestReadProxyConditionalGet713=== RUN TestReadProxyRootRedirectsToIndexHTML714=== PAUSE TestReadProxyRootRedirectsToIndexHTML715=== RUN TestReadProxyDisabled716=== PAUSE TestReadProxyDisabled717=== RUN TestReadRedirectNar718=== PAUSE TestReadRedirectNar719=== RUN TestReadRedirectKeepsNarinfoProxied720=== PAUSE TestReadRedirectKeepsNarinfoProxied721=== RUN TestReadProxyRangeRequest722=== PAUSE TestReadProxyRangeRequest723=== RUN TestReadRedirectUsesPublicS3URL724=== PAUSE TestReadRedirectUsesPublicS3URL725=== RUN TestPush_OverlappingRootsStoreOneRowPerKey726=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey727=== RUN TestPush_CompleteCommitsEveryRoot728=== PAUSE TestPush_CompleteCommitsEveryRoot729=== RUN TestPush_RepeatedRootCommitsOnce730=== PAUSE TestPush_RepeatedRootCommitsOnce731=== RUN TestPush_CompleteWaitsForCleanupOfThePush732=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush733=== RUN TestPush_RootsAreLockedInByteOrder734=== PAUSE TestPush_RootsAreLockedInByteOrder735=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit736=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit737=== RUN TestPush_RejectsBadRequests738=== PAUSE TestPush_RejectsBadRequests739=== RUN TestPush_SignsNarinfosOfItsPendingObjects740=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects741=== RUN TestSharedNarBothUploadsComplete742=== PAUSE TestSharedNarBothUploadsComplete743=== RUN TestCompleteMultipartUpload_ErrorButObjectExists744=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists745=== RUN TestCompletedNarNotReofferedAcrossClosures746=== PAUSE TestCompletedNarNotReofferedAcrossClosures747=== RUN TestPresignedUploadRegisteredBeforeCommit748=== PAUSE TestPresignedUploadRegisteredBeforeCommit749=== RUN TestService_Rustfstest750=== PAUSE TestService_Rustfstest751=== RUN TestParseSize752=== PAUSE TestParseSize753=== RUN TestSkippedUploadsHandler754=== PAUSE TestSkippedUploadsHandler755=== RUN TestSystemdListenerNotActivated756--- PASS: TestSystemdListenerNotActivated (0.00s)757=== RUN TestWatchdogBeatsWhenHealthy758--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)759=== RUN TestWatchdogSkipsWhenUnhealthy7602026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 00:20:28 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"770--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)771=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle772=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle773=== RUN TestProxyWriteTimeout774=== PAUSE TestProxyWriteTimeout775=== RUN TestPendingClosureWriteTimeout776=== PAUSE TestPendingClosureWriteTimeout777=== RUN TestIsValidUploadKey778=== PAUSE TestIsValidUploadKey779=== RUN TestUploadHandlersRejectInvalidKeys780=== PAUSE TestUploadHandlersRejectInvalidKeys781=== RUN TestUploadHandlersRejectOversizedBody782=== PAUSE TestUploadHandlersRejectOversizedBody783=== RUN TestService_cleanupPendingClosuresHandler784=== PAUSE TestService_cleanupPendingClosuresHandler785=== RUN TestService_createPendingClosureHandler786=== PAUSE TestService_createPendingClosureHandler787=== RUN TestService_verifyS3Integrity788=== PAUSE TestService_verifyS3Integrity789=== RUN TestService_verifyS3ReuploadsStaleNarinfo790=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo791=== RUN TestCompleteMultipartUnregistered792=== PAUSE TestCompleteMultipartUnregistered793=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT794=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT795=== CONT TestAbortSucceeded796=== CONT TestUploadHandlersRejectInvalidKeys797--- PASS: TestAbortSucceeded (0.00s)798=== CONT TestCreatePendingClosureRejectsOversizedNAR799=== CONT TestCompleteMultipartUnregistered800=== CONT TestService_verifyS3ReuploadsStaleNarinfo801=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed802=== CONT TestService_verifyS3Integrity803=== CONT TestService_createPendingClosureHandler804=== CONT TestService_cleanupPendingClosuresHandler8052026/11/12 00:20:28 INFO Received uploads request method=POST path=/api/pending_closures806=== CONT TestUploadHandlersRejectOversizedBody807=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT808=== CONT TestCacheConfigHandlerMaxNarSize809=== CONT TestGCBugBareHashReferences810=== CONT TestGCAdvisoryLockBlocksConcurrentRun811=== CONT TestLeadEndsOnShutdown812=== CONT TestLeadGenerationGrowsWithEveryGrant813=== CONT TestLeadIncumbentAnnouncesAtOnce814=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent815=== CONT TestLeadElectsOneAndHandsOver816=== CONT TestConnectWaitsForAPeerMigration817=== CONT TestResolveDBConnectionString818=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock819=== CONT TestDeleteClosuresRacingPins820=== CONT TestClientFallsBackToClosures821=== CONT TestClientPushesUseOnePush822=== CONT TestGCMetrics823=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info824=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed825=== CONT TestPinProtectsFromGC826--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)827--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)828=== RUN TestResolveDBConnectionString/flag_wins829=== PAUSE TestResolveDBConnectionString/flag_wins830=== RUN TestResolveDBConnectionString/file_when_flag_empty831=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed832=== PAUSE TestResolveDBConnectionString/file_when_flag_empty833=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info834=== CONT TestClientSharedPathCommittedMidPush835=== RUN TestResolveDBConnectionString/missing_file_is_an_error836=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal837=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed838=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error839=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost840=== RUN TestResolveDBConnectionString/PGHOST_allows_empty841=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty842=== RUN TestResolveDBConnectionString/nothing_configured843=== PAUSE TestResolveDBConnectionString/nothing_configured844=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost845=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal846=== CONT TestClientWithDependencies847=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed848=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key849=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key850=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key851=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key852=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed853=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed854=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed855=== CONT TestClientMultipleUploads856=== CONT TestClientIntegration857=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure858=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure859=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart860=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart861=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts862=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts863=== CONT TestClientErrorHandling864=== RUN TestClientErrorHandling/InvalidStorePath865=== PAUSE TestClientErrorHandling/InvalidStorePath866=== RUN TestClientErrorHandling/InvalidAuthToken867=== PAUSE TestClientErrorHandling/InvalidAuthToken868=== RUN TestClientErrorHandling/ServerNotAvailable869=== PAUSE TestClientErrorHandling/ServerNotAvailable870=== CONT TestClientCADerivations8712026/11/12 00:20:28 INFO Received uploads request method=POST path=/api/pending_closures8722026/11/12 00:20:28 INFO Received uploads request method=POST path=/api/pending_closures8732026/11/12 00:20:28 INFO Received uploads request method=POST path=/api/pending_closures8742026/11/12 00:20:28 INFO Received uploads request method=POST path=/api/pending_closures8752026/11/12 00:20:28 INFO Received complete multipart upload request method=POST path=/api/multipart/complete8762026/11/12 00:20:28 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst877--- PASS: TestCompleteMultipartUnregistered (0.54s)878=== CONT TestCacheStatsHandler8792026/11/12 00:20:29 INFO lead: acquired remote=192.0.2.1:1234 generation=18802026/11/12 00:20:29 INFO Received cleanup request method=DELETE path=/api/pending_closures8812026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=08822026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures8832026/11/12 00:20:29 INFO Received cleanup request method=DELETE path=/api/pending_closures8842026/11/12 00:20:29 INFO Aborted multipart uploads count=1 kept=08852026/11/12 00:20:29 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete8862026-11-12 00:20:29.053 UTC [648] ERROR: Closure does not exist: id=18872026-11-12 00:20:29.053 UTC [648] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE8882026-11-12 00:20:29.053 UTC [648] STATEMENT: -- name: CommitPendingClosure :one889 SELECT commit_pending_closure($1::bigint)::bigint890 891--- PASS: TestService_cleanupPendingClosuresHandler (0.63s)892=== CONT TestEnsureAbortUploadsRule8932026/11/12 00:20:29 INFO lead: acquired remote=192.0.2.1:1234 generation=18942026/11/12 00:20:29 INFO lead: acquired remote=192.0.2.1:1234 generation=18952026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=08962026/11/12 00:20:29 WARN Force mode enabled - objects will be deleted immediately without grace period8972026/11/12 00:20:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=08982026/11/12 00:20:29 INFO Vacuumed table table=pending_closures8992026/11/12 00:20:29 INFO Vacuumed table table=pending_objects9002026/11/12 00:20:29 INFO Vacuumed table table=multipart_uploads9012026/11/12 00:20:29 INFO Vacuumed table table=closures9022026/11/12 00:20:29 INFO Vacuumed table table=objects903--- PASS: TestGCBugBareHashReferences (0.79s)904=== CONT TestGenerateLandingPage905--- PASS: TestGCMetrics (0.80s)906=== CONT TestCacheConfigHandler907=== RUN TestCacheConfigHandler/full_config,_no_issuer908=== PAUSE TestCacheConfigHandler/full_config,_no_issuer909=== RUN TestCacheConfigHandler/no_cache_url_configured910=== PAUSE TestCacheConfigHandler/no_cache_url_configured911=== RUN TestCacheConfigHandler/no_signing_keys912=== PAUSE TestCacheConfigHandler/no_signing_keys913=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator914=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator915=== CONT TestService_ReadScope_PublicByDefault916--- PASS: TestGenerateLandingPage (0.01s)917=== CONT TestService_RequireScope_OIDC9182026/11/12 00:20:29 INFO lead: acquired remote=192.0.2.1:1234 generation=19192026/11/12 00:20:29 INFO lead: released remote=192.0.2.1:1234 generation=1920--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.81s)921=== CONT TestService_readinessHandler9222026/11/12 00:20:29 INFO lead: acquired remote=192.0.2.1:1234 generation=19232026/11/12 00:20:29 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:38877/oidc9242026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes9252026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures9262026/11/12 00:20:29 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)9272026/11/12 00:20:29 INFO Uploading n5lkx20bykvzyxbrghasqyjbz26c5ing-top (224B)9282026/11/12 00:20:29 INFO Uploading pwgq1ajfs942i6m3cnbnvmfcld59b4rx-shared-dep (136B)9292026/11/12 00:20:29 WARN Failed to register uploaded object key=n5lkx20bykvzyxbrghasqyjbz26c5ing.ls error="server returned 404: 404 page not found\n"9302026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"9312026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/1b0z0dy3kjgqmijjz9110ji8r38rxg8dapc34nxsghll3008j49j.nar.zst error="server returned 404: 404 page not found\n"9322026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9332026/11/12 00:20:29 WARN Failed to register uploaded object key=pwgq1ajfs942i6m3cnbnvmfcld59b4rx.ls error="server returned 404: 404 page not found\n"9342026/11/12 00:20:29 INFO Signed narinfos id=1 count=2935--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.90s)9362026/11/12 00:20:29 INFO Uploading 2 narinfos937=== CONT TestService_healthCheckHandler9382026/11/12 00:20:29 WARN Failed to register uploaded object key=n5lkx20bykvzyxbrghasqyjbz26c5ing.narinfo error="server returned 404: 404 page not found\n"9392026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete9402026/11/12 00:20:29 WARN Failed to register uploaded object key=pwgq1ajfs942i6m3cnbnvmfcld59b4rx.narinfo error="server returned 404: 404 page not found\n"9412026/11/12 00:20:29 INFO Completed push id=1 objects=6 duration=9.723176ms9422026/11/12 00:20:29 INFO Upload complete. (74ms)943=== NAME TestClientSharedPathCommittedMidPush944 client_integration_test.go:728: Retrieved narinfo from S3:945 StorePath: /build/TestClientSharedPathCommittedMidPush784369067/001/store/pwgq1ajfs942i6m3cnbnvmfcld59b4rx-shared-dep946 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst947 Compression: zstd948 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82949 NarSize: 136950 References: 951 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n952 client_integration_test.go:728: Retrieved narinfo from S3:953 StorePath: /build/TestClientSharedPathCommittedMidPush784369067/001/store/n5lkx20bykvzyxbrghasqyjbz26c5ing-top954 URL: nar/1b0z0dy3kjgqmijjz9110ji8r38rxg8dapc34nxsghll3008j49j.nar.zst955 Compression: zstd956 NarHash: sha256:1b0z0dy3kjgqmijjz9110ji8r38rxg8dapc34nxsghll3008j49j957 NarSize: 224958 References: /build/TestClientSharedPathCommittedMidPush784369067/001/store/pwgq1ajfs942i6m3cnbnvmfcld59b4rx-shared-dep959 CA: text:sha256:0a81jka225sqjyarn8phikcc7vzkhwbiyw5bvl1lfxzf14p8byhx9602026/11/12 00:20:29 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"961--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.92s)962=== CONT TestService_AuthMiddleware_OIDC963--- PASS: TestClientSharedPathCommittedMidPush (0.87s)964=== CONT TestGracefulShutdownDrainsInflight9652026/11/12 00:20:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9662026/11/12 00:20:29 INFO Starting HTTP server address=127.0.0.1:460239672026/11/12 00:20:29 INFO Shutdown signal received, draining in-flight requests timeout=10s9682026/11/12 00:20:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9692026/11/12 00:20:29 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ZmIxYTQ5NDItYjAyZC00ZGNhLTkyZDEtNzM0NTAzNmVjZmE4LmM4ZGE2ZTExLWE5YmYtNGE1Zi1hMGNjLTRmZjliYzRkZmYzY3gxNzk0NDQyODI4OTE0MTk2NDM4 parts=109702026/11/12 00:20:29 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9712026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures9722026/11/12 00:20:29 INFO Completed upload id=1 objects=2 duration=4.546398ms9732026/11/12 00:20:29 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000009742026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures9752026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures9762026/11/12 00:20:29 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)9772026/11/12 00:20:29 INFO Uploading g0r97mj950j445z773z30li1dw9m0a1c-shared-dep (136B)9782026/11/12 00:20:29 INFO Starting cleanup of old closures method=DELETE path=/api/closures9792026/11/12 00:20:29 INFO Uploading z9r4g5qbpvvq3j89smw3vwvrbx5apmpq-b (216B)9802026/11/12 00:20:29 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ZmIxYTQ5NDItYjAyZC00ZGNhLTkyZDEtNzM0NTAzNmVjZmE4LjVhNmUwYzRlLTEzYzgtNDhmZC05NGM4LWE1YzA2YzA2ODJhMHgxNzk0NDQyODI4OTM5ODI2MTU4 parts=109812026/11/12 00:20:29 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9822026/11/12 00:20:29 WARN Failed to register uploaded object key=g0r97mj950j445z773z30li1dw9m0a1c.ls error="server returned 404: 404 page not found\n"9832026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"9842026/11/12 00:20:29 WARN Failed to register uploaded object key=0mrpqrvrz8c27gfwfvgwf44clp812ysk.ls error="server returned 404: 404 page not found\n"9852026/11/12 00:20:29 WARN Failed to register uploaded object key=z9r4g5qbpvvq3j89smw3vwvrbx5apmpq.ls error="server returned 404: 404 page not found\n"9862026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=09872026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign9882026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/0savrb3b55mgrkdj30fh9q2kagzq13aagpm8hrnf9xfkackfp1xj.nar.zst error="server returned 404: 404 page not found\n"9892026/11/12 00:20:29 INFO Signed narinfos id=1 count=29902026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign9912026/11/12 00:20:29 INFO Signed narinfos id=2 count=29922026/11/12 00:20:29 INFO Uploading 4 narinfos9932026/11/12 00:20:29 INFO Completed upload id=1 objects=2 duration=3.789129ms9942026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures9952026/11/12 00:20:29 WARN Failed to register uploaded object key=g0r97mj950j445z773z30li1dw9m0a1c.narinfo error="server returned 404: 404 page not found\n"9962026/11/12 00:20:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09972026/11/12 00:20:29 WARN Failed to register uploaded object key=0mrpqrvrz8c27gfwfvgwf44clp812ysk.narinfo error="server returned 404: 404 page not found\n"9982026/11/12 00:20:29 WARN Failed to register uploaded object key=z9r4g5qbpvvq3j89smw3vwvrbx5apmpq.narinfo error="server returned 404: 404 page not found\n"9992026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures10002026/11/12 00:20:29 INFO Vacuumed table table=pending_closures10012026/11/12 00:20:29 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10022026/11/12 00:20:29 WARN Failed to register uploaded object key=g0r97mj950j445z773z30li1dw9m0a1c.narinfo error="server returned 404: 404 page not found\n"10032026/11/12 00:20:29 INFO Vacuumed table table=pending_objects10042026/11/12 00:20:29 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10052026/11/12 00:20:29 WARN Found objects in DB but missing from S3, will re-upload count=11006--- PASS: TestService_verifyS3Integrity (0.99s)1007=== CONT TestService_ReadAuthMiddleware10082026/11/12 00:20:29 INFO Vacuumed table table=multipart_uploads10092026/11/12 00:20:29 INFO Vacuumed table table=closures10102026/11/12 00:20:29 INFO Completed upload id=1 objects=6 duration=6.149418ms10112026/11/12 00:20:29 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete10122026/11/12 00:20:29 INFO Vacuumed table table=objects10132026/11/12 00:20:29 INFO Completed upload id=2 objects=6 duration=2.32817ms10142026/11/12 00:20:29 INFO Upload complete. (76ms)1015=== NAME TestClientFallsBackToClosures1016 client_pushes_test.go:112: Retrieved narinfo from S3:1017 StorePath: /build/TestClientFallsBackToClosures1267974143/001/store/g0r97mj950j445z773z30li1dw9m0a1c-shared-dep1018 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1019 Compression: zstd1020 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821021 NarSize: 1361022 References: 1023 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1024 client_pushes_test.go:112: Retrieved narinfo from S3:1025 StorePath: /build/TestClientFallsBackToClosures1267974143/001/store/0mrpqrvrz8c27gfwfvgwf44clp812ysk-a1026 URL: nar/0savrb3b55mgrkdj30fh9q2kagzq13aagpm8hrnf9xfkackfp1xj.nar.zst1027 Compression: zstd1028 NarHash: sha256:0savrb3b55mgrkdj30fh9q2kagzq13aagpm8hrnf9xfkackfp1xj1029 NarSize: 2161030 References: /build/TestClientFallsBackToClosures1267974143/001/store/g0r97mj950j445z773z30li1dw9m0a1c-shared-dep1031 CA: text:sha256:1d9qhf8wdng67q17z0w5ff54d2xnw56liqzh3f531l0klbaphjf01032 client_pushes_test.go:112: Retrieved narinfo from S3:1033 StorePath: /build/TestClientFallsBackToClosures1267974143/001/store/z9r4g5qbpvvq3j89smw3vwvrbx5apmpq-b1034 URL: nar/0savrb3b55mgrkdj30fh9q2kagzq13aagpm8hrnf9xfkackfp1xj.nar.zst1035 Compression: zstd1036 NarHash: sha256:0savrb3b55mgrkdj30fh9q2kagzq13aagpm8hrnf9xfkackfp1xj1037 NarSize: 2161038 References: /build/TestClientFallsBackToClosures1267974143/001/store/g0r97mj950j445z773z30li1dw9m0a1c-shared-dep1039 CA: text:sha256:1d9qhf8wdng67q17z0w5ff54d2xnw56liqzh3f531l0klbaphjf01040--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1041=== CONT TestService_AuthMiddleware_MTLSBoundSubjects1042--- PASS: TestClientFallsBackToClosures (1.00s)1043=== CONT TestGCTaskStore_Fail1044--- PASS: TestGCTaskStore_Fail (0.00s)1045=== CONT TestService_AuthMiddleware_MTLSProxyHeader10462026/11/12 00:20:29 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001047--- PASS: TestService_createPendingClosureHandler (1.02s)1048=== CONT TestService_AuthMiddleware10492026/11/12 00:20:29 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:34637/oidc1050=== NAME TestPinProtectsFromGC1051 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC2658795018/001/store/mm4lcv1lxyiqsaxv3rn7imbj13n4jsq6-pinned-file.txt1052 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC2658795018/001/store/gwxs8xr3l0ssh6z7rz81h62v8gx4acjc-unpinned-file.txt1053=== NAME TestClientIntegration1054 client_integration_test.go:292: Created store path: /build/TestClientIntegration2906287627/002/store/xibclk3l2hx200fjck2iv8dkgqi514p8-test-file.txt1055=== NAME TestClientWithDependencies1056 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies2936269864/001/store/zalxayywidpvikp379g62iw9klz0lvsf-test-script1057=== NAME TestClientMultipleUploads1058 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads325149975/001/store/gaqzpmv7zm6f06zqqiv4pbpcly60hm8j-test-file-0.txt1059=== NAME TestClientWithDependencies1060 client_integration_test.go:663: Found 1 dependencies (including self)10612026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes10622026/11/12 00:20:29 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)1063=== NAME TestClientMultipleUploads10642026/11/12 00:20:29 INFO Uploading mm4lcv1lxyiqsaxv3rn7imbj13n4jsq6-pinned-file.txt (128B)1065 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads325149975/001/store/cwgwdfpxbjicywyha5z4zznpxgijry2d-test-file-1.txt10662026/11/12 00:20:29 WARN Failed to register uploaded object key=mm4lcv1lxyiqsaxv3rn7imbj13n4jsq6.ls error="server returned 404: 404 page not found\n"10672026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10682026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"10692026/11/12 00:20:29 INFO Signed narinfos id=1 count=110702026/11/12 00:20:29 INFO Uploading 1 narinfos10712026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete10722026/11/12 00:20:29 WARN Failed to register uploaded object key=mm4lcv1lxyiqsaxv3rn7imbj13n4jsq6.narinfo error="server returned 404: 404 page not found\n"10732026/11/12 00:20:29 INFO Completed push id=1 objects=3 duration=10.658726ms10742026/11/12 00:20:29 INFO Upload complete. (69ms)1075--- PASS: TestCacheStatsHandler (0.60s)1076=== CONT TestGCTaskStore_PhaseUpdates10772026/11/12 00:20:29 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket28 days=21078--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1079=== CONT TestGCTaskStore_CompletedAllowsNewTask1080--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1081=== CONT TestReadProxyConditionalGet1082--- PASS: TestEnsureAbortUploadsRule (0.51s)1083=== CONT TestGCTaskStore_GetReturnsLatest1084--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1085=== CONT TestGCTaskStore_GetEmpty1086--- PASS: TestGCTaskStore_GetEmpty (0.00s)1087=== CONT TestIsValidUploadKey1088=== RUN TestIsValidUploadKey/narinfo1089=== PAUSE TestIsValidUploadKey/narinfo1090=== RUN TestIsValidUploadKey/nar_zst1091=== PAUSE TestIsValidUploadKey/nar_zst1092=== RUN TestIsValidUploadKey/nar_xz1093=== PAUSE TestIsValidUploadKey/nar_xz1094=== RUN TestIsValidUploadKey/nar_plain1095=== PAUSE TestIsValidUploadKey/nar_plain1096=== RUN TestIsValidUploadKey/listing1097=== PAUSE TestIsValidUploadKey/listing1098=== RUN TestIsValidUploadKey/build_log1099=== PAUSE TestIsValidUploadKey/build_log1100=== RUN TestIsValidUploadKey/build_log_home-manager_file1101=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1102=== RUN TestIsValidUploadKey/build_log_plus_in_name1103=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1104=== RUN TestIsValidUploadKey/build_log_question_mark1105=== PAUSE TestIsValidUploadKey/build_log_question_mark1106=== RUN TestIsValidUploadKey/build_log_equals1107=== PAUSE TestIsValidUploadKey/build_log_equals1108=== RUN TestIsValidUploadKey/realisation1109=== PAUSE TestIsValidUploadKey/realisation1110=== RUN TestIsValidUploadKey/realisation_plus_in_output1111=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1112=== RUN TestIsValidUploadKey/nix-cache-info1113=== PAUSE TestIsValidUploadKey/nix-cache-info1114=== NAME TestClientCADerivations1115 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations1410889338/001/store/qnhnxbpb8dswfgf8qv2y2gd5yw1wiijf-ca-test1116=== RUN TestIsValidUploadKey/index.html1117=== PAUSE TestIsValidUploadKey/index.html1118=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1119=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1120=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1121=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1122=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1123=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1124=== RUN TestIsValidUploadKey/traversal1125=== PAUSE TestIsValidUploadKey/traversal1126=== RUN TestIsValidUploadKey/traversal_nar1127=== PAUSE TestIsValidUploadKey/traversal_nar1128=== RUN TestIsValidUploadKey/absolute1129=== PAUSE TestIsValidUploadKey/absolute1130--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (1.14s)1131=== CONT TestGCTaskStore_ConflictDifferentParams1132--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1133=== CONT TestGCTaskStore_DeduplicateSameParams1134--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)11352026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes1136=== RUN TestIsValidUploadKey/empty_key1137=== PAUSE TestIsValidUploadKey/empty_key1138=== CONT TestPendingClosureWriteTimeout1139=== RUN TestPendingClosureWriteTimeout/empty1140=== PAUSE TestPendingClosureWriteTimeout/empty1141=== RUN TestPendingClosureWriteTimeout/400_objects1142=== PAUSE TestPendingClosureWriteTimeout/400_objects1143=== RUN TestPendingClosureWriteTimeout/670k_objects1144=== RUN TestIsValidUploadKey/unknown_type1145=== PAUSE TestIsValidUploadKey/unknown_type1146=== PAUSE TestPendingClosureWriteTimeout/670k_objects1147=== CONT TestGCTaskStore_StartNew1148--- PASS: TestGCTaskStore_StartNew (0.00s)1149=== CONT TestProxyWriteTimeout1150=== RUN TestProxyWriteTimeout/narinfo1151=== PAUSE TestProxyWriteTimeout/narinfo1152=== RUN TestProxyWriteTimeout/1_GiB_nar1153=== PAUSE TestProxyWriteTimeout/1_GiB_nar1154=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1155=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1156=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1157=== RUN TestProxyWriteTimeout/10_GiB_nar1158=== PAUSE TestProxyWriteTimeout/10_GiB_nar1159=== RUN TestProxyWriteTimeout/unknown_size1160=== PAUSE TestProxyWriteTimeout/unknown_size1161=== CONT TestGCEndsOnShutdown1162=== RUN TestGCEndsOnShutdown/before_the_run1163=== PAUSE TestGCEndsOnShutdown/before_the_run1164=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1165=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1166=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1167=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1168=== CONT TestSkippedUploadsHandler11692026/11/12 00:20:29 INFO Client skipped oversized paths paths=3 nar_bytes=500000000011702026/11/12 00:20:29 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11712026/11/12 00:20:29 INFO Uploading xibclk3l2hx200fjck2iv8dkgqi514p8-test-file.txt (152B)1172=== NAME TestClientMultipleUploads1173 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads325149975/001/store/86x81v4840gy03v45g26hdgc10ln49gw-test-file-2.txt1174--- PASS: TestService_ReadScope_PublicByDefault (0.35s)1175=== CONT TestCommitRacingPendingCleanupKeepsObjects1176--- PASS: TestSkippedUploadsHandler (0.01s)1177=== CONT TestParseSize1178--- PASS: TestParseSize (0.00s)1179=== CONT TestPushWaitsForSweepHoldingRow11802026/11/12 00:20:29 WARN Failed to register uploaded object key=xibclk3l2hx200fjck2iv8dkgqi514p8.ls error="server returned 404: 404 page not found\n"11812026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11822026/11/12 00:20:29 INFO Signed narinfos id=1 count=111832026/11/12 00:20:29 INFO Uploading 1 narinfos11842026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"11852026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete11862026/11/12 00:20:29 WARN Failed to register uploaded object key=xibclk3l2hx200fjck2iv8dkgqi514p8.narinfo error="server returned 404: 404 page not found\n"11872026/11/12 00:20:29 INFO Completed push id=1 objects=3 duration=9.985236ms11882026/11/12 00:20:29 INFO Upload complete. (66ms)1189=== NAME TestClientCADerivations1190 client_ca_test.go:139: Found 1 dependencies (including self)11912026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes11922026/11/12 00:20:29 WARN readiness check failed error="closed pool"1193--- PASS: TestService_readinessHandler (0.37s)1194=== CONT TestSweepRechecksPendingAfterLocking11952026/11/12 00:20:29 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)11962026/11/12 00:20:29 INFO Uploading zalxayywidpvikp379g62iw9klz0lvsf-test-script (136B)11972026/11/12 00:20:29 WARN Failed to register uploaded object key=zalxayywidpvikp379g62iw9klz0lvsf.ls error="server returned 404: 404 page not found\n"11982026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"11992026/11/12 00:20:29 WARN Failed to register uploaded object key=log/b04g5kffx9kj6sc5cdn3gmzvdd52ys4v-test-script.drv error="server returned 404: 404 page not found\n"12002026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12012026/11/12 00:20:29 INFO Signed narinfos id=1 count=112022026/11/12 00:20:29 INFO Uploading 1 narinfos12032026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete12042026/11/12 00:20:29 WARN Failed to register uploaded object key=zalxayywidpvikp379g62iw9klz0lvsf.narinfo error="server returned 404: 404 page not found\n"12052026/11/12 00:20:29 INFO All 1 paths already cached12062026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes1207=== NAME TestClientIntegration1208 client_integration_test.go:318: Retrieved narinfo from S3:1209 StorePath: /build/TestClientIntegration2906287627/002/store/xibclk3l2hx200fjck2iv8dkgqi514p8-test-file.txt1210 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1211 Compression: zstd1212 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11213 NarSize: 1521214 References: 12152026/11/12 00:20:29 INFO Completed push id=1 objects=4 duration=11.207116ms1216 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk112172026/11/12 00:20:29 INFO Upload complete. (72ms)12182026/11/12 00:20:29 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12192026/11/12 00:20:29 INFO Uploading gwxs8xr3l0ssh6z7rz81h62v8gx4acjc-unpinned-file.txt (128B)1220 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1221 client_integration_test.go:319: Decompressed .ls content (64 bytes):1222 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1223=== NAME TestClientWithDependencies1224 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies2936269864/001/store) requires matching store prefix1225=== RUN TestService_RequireScope_OIDC/builder_may_write1226=== PAUSE TestService_RequireScope_OIDC/builder_may_write1227=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1228=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1229=== RUN TestService_RequireScope_OIDC/ops_may_admin1230=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1231=== RUN TestService_RequireScope_OIDC/ops_may_not_write1232=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1233=== RUN TestService_RequireScope_OIDC/reader_may_not_write1234=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1235=== RUN TestService_RequireScope_OIDC/static_token_may_admin1236=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1237=== RUN TestService_RequireScope_OIDC/static_token_may_write1238=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1239=== RUN TestService_RequireScope_OIDC/reader_may_read1240=== PAUSE TestService_RequireScope_OIDC/reader_may_read1241=== RUN TestService_RequireScope_OIDC/writer_implies_read1242=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1243=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1244=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1245=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1246=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1247=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1248=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc12492026/11/12 00:20:29 WARN Failed to register uploaded object key=gwxs8xr3l0ssh6z7rz81h62v8gx4acjc.ls error="server returned 404: 404 page not found\n"1250=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read12512026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign1252=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1253=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1254=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1255=== CONT TestService_Rustfstest12562026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"12572026/11/12 00:20:29 INFO Signed narinfos id=2 count=112582026/11/12 00:20:29 INFO Uploading 1 narinfos1259--- PASS: TestClientWithDependencies (1.12s)1260=== CONT TestGetExistingObjectsLocksRows12612026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/2/complete12622026/11/12 00:20:29 WARN Failed to register uploaded object key=gwxs8xr3l0ssh6z7rz81h62v8gx4acjc.narinfo error="server returned 404: 404 page not found\n"12632026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes1264--- PASS: TestService_healthCheckHandler (0.32s)1265=== CONT TestPresignedUploadRegisteredBeforeCommit12662026/11/12 00:20:29 INFO Completed push id=2 objects=3 duration=16.828504ms12672026/11/12 00:20:29 INFO Upload complete. (64ms)12682026/11/12 00:20:29 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)12692026/11/12 00:20:29 INFO Uploading gaqzpmv7zm6f06zqqiv4pbpcly60hm8j-test-file-0.txt (160B)12702026/11/12 00:20:29 INFO Uploading cwgwdfpxbjicywyha5z4zznpxgijry2d-test-file-1.txt (160B)12712026/11/12 00:20:29 INFO Uploading 86x81v4840gy03v45g26hdgc10ln49gw-test-file-2.txt (160B)12722026/11/12 00:20:29 WARN Failed to register uploaded object key=86x81v4840gy03v45g26hdgc10ln49gw.ls error="server returned 404: 404 page not found\n"12732026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"12742026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"12752026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"12762026/11/12 00:20:29 WARN Failed to register uploaded object key=gaqzpmv7zm6f06zqqiv4pbpcly60hm8j.ls error="server returned 404: 404 page not found\n"12772026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12782026/11/12 00:20:29 WARN Failed to register uploaded object key=cwgwdfpxbjicywyha5z4zznpxgijry2d.ls error="server returned 404: 404 page not found\n"12792026/11/12 00:20:29 INFO Signed narinfos id=1 count=312802026/11/12 00:20:29 INFO Uploading 3 narinfos12812026/11/12 00:20:29 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"12822026/11/12 00:20:29 WARN mTLS auth: bound subjects configured but subject DN unavailable12832026/11/12 00:20:29 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1284--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.25s)1285=== CONT TestCompletedNarNotReofferedAcrossClosures12862026/11/12 00:20:29 WARN Failed to register uploaded object key=cwgwdfpxbjicywyha5z4zznpxgijry2d.narinfo error="server returned 404: 404 page not found\n"12872026/11/12 00:20:29 WARN Failed to register uploaded object key=86x81v4840gy03v45g26hdgc10ln49gw.narinfo error="server returned 404: 404 page not found\n"12882026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete12892026/11/12 00:20:29 WARN Failed to register uploaded object key=gaqzpmv7zm6f06zqqiv4pbpcly60hm8j.narinfo error="server returned 404: 404 page not found\n"12902026/11/12 00:20:29 INFO Completed push id=1 objects=9 duration=11.226695ms12912026/11/12 00:20:29 INFO Upload complete. (82ms)1292=== NAME TestClientMultipleUploads1293 client_integration_test.go:417: Uploaded 3 paths in 116.186985ms12942026/11/12 00:20:29 INFO Starting cleanup of old closures method=DELETE path=/api/closures12952026/11/12 00:20:29 INFO Garbage collection started12962026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes1297--- PASS: TestClientMultipleUploads (1.18s)1298=== CONT TestRegisterCompletedObjectNeedsPendingRow1299--- PASS: TestService_ReadAuthMiddleware (0.29s)1300=== CONT TestCompleteMultipartUpload_ErrorButObjectExists13012026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes13022026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=013032026/11/12 00:20:29 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)13042026/11/12 00:20:29 INFO Uploading c990hmfn1l4700cf5awd07cyh04pp50n-b (216B)13052026/11/12 00:20:29 INFO Uploading n0w4n1qaskrdhqkadni3pr9mbl8p7wpk-shared-dep (136B)13062026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes13072026/11/12 00:20:29 WARN Force mode enabled - objects will be deleted immediately without grace period13082026/11/12 00:20:29 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13092026/11/12 00:20:29 INFO Uploading qnhnxbpb8dswfgf8qv2y2gd5yw1wiijf-ca-test (144B)13102026/11/12 00:20:29 WARN Failed to register uploaded object key=3gx2ijzrcfcf3xfamhr829z16pvjcpwc.ls error="server returned 404: 404 page not found\n"13112026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/07fh8ji51s9zv3087s2rphfp0q06gh87hxn1r4y7mqn4r098nkcx.nar.zst error="server returned 404: 404 page not found\n"13122026/11/12 00:20:29 WARN Failed to register uploaded object key=n0w4n1qaskrdhqkadni3pr9mbl8p7wpk.ls error="server returned 404: 404 page not found\n"13132026/11/12 00:20:29 WARN Failed to register uploaded object key=c990hmfn1l4700cf5awd07cyh04pp50n.ls error="server returned 404: 404 page not found\n"13142026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13152026/11/12 00:20:29 INFO Object in database but missing from S3 key=xibclk3l2hx200fjck2iv8dkgqi514p8.ls13162026/11/12 00:20:29 WARN Found objects in DB but missing from S3, will re-upload count=113172026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13182026/11/12 00:20:29 INFO Signed narinfos id=1 count=313192026/11/12 00:20:29 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)13202026/11/12 00:20:29 INFO Uploading 3 narinfos13212026/11/12 00:20:29 WARN Failed to register uploaded object key=qnhnxbpb8dswfgf8qv2y2gd5yw1wiijf.ls error="server returned 404: 404 page not found\n"13222026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"13232026/11/12 00:20:29 WARN Failed to register uploaded object key=log/cwqkg3p6q43k8qjbfzn3bc4z650s6pj4-ca-test.drv error="server returned 404: 404 page not found\n"13242026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13252026/11/12 00:20:29 INFO Signed narinfos id=1 count=113262026/11/12 00:20:29 INFO Uploading 1 narinfos13272026/11/12 00:20:29 WARN Failed to register uploaded object key=3gx2ijzrcfcf3xfamhr829z16pvjcpwc.narinfo error="server returned 404: 404 page not found\n"1328--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.30s)1329=== CONT TestSweepSkipsRowsLockedByPush13302026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/2/complete13312026/11/12 00:20:29 WARN Failed to register uploaded object key=n0w4n1qaskrdhqkadni3pr9mbl8p7wpk.narinfo error="server returned 404: 404 page not found\n"13322026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete13332026/11/12 00:20:29 WARN Failed to register uploaded object key=xibclk3l2hx200fjck2iv8dkgqi514p8.ls error="server returned 404: 404 page not found\n"13342026/11/12 00:20:29 WARN Failed to register uploaded object key=c990hmfn1l4700cf5awd07cyh04pp50n.narinfo error="server returned 404: 404 page not found\n"13352026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete13362026/11/12 00:20:29 WARN Failed to register uploaded object key=qnhnxbpb8dswfgf8qv2y2gd5yw1wiijf.narinfo error="server returned 404: 404 page not found\n"13372026/11/12 00:20:29 INFO Completed push id=2 objects=3 duration=3.936409ms13382026/11/12 00:20:29 INFO Upload complete. (52ms)1339=== NAME TestClientIntegration1340 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1341 client_integration_test.go:336: Decompressed .ls content (49 bytes):1342 {"version":1,"root":{"type":"regular","size":39}}13432026/11/12 00:20:29 INFO Completed push id=1 objects=8 duration=11.741826ms13442026/11/12 00:20:29 INFO Upload complete. (85ms)1345=== NAME TestClientPushesUseOnePush1346 client_pushes_test.go:97: Retrieved narinfo from S3:1347 StorePath: /build/TestClientPushesUseOnePush2323145864/001/store/n0w4n1qaskrdhqkadni3pr9mbl8p7wpk-shared-dep1348 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1349 Compression: zstd1350 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821351 NarSize: 1361352 References: 1353 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n13542026/11/12 00:20:29 INFO Completed push id=1 objects=4 duration=13.094715ms13552026/11/12 00:20:29 INFO Upload complete. (114ms)1356 client_pushes_test.go:97: Retrieved narinfo from S3:1357 StorePath: /build/TestClientPushesUseOnePush2323145864/001/store/3gx2ijzrcfcf3xfamhr829z16pvjcpwc-a1358 URL: nar/07fh8ji51s9zv3087s2rphfp0q06gh87hxn1r4y7mqn4r098nkcx.nar.zst1359 Compression: zstd1360 NarHash: sha256:07fh8ji51s9zv3087s2rphfp0q06gh87hxn1r4y7mqn4r098nkcx1361 NarSize: 2161362 References: /build/TestClientPushesUseOnePush2323145864/001/store/n0w4n1qaskrdhqkadni3pr9mbl8p7wpk-shared-dep1363 CA: text:sha256:0nbm2fn6m12n0577ypafwaspvadwmhyam5714rjcsf1hfzk9xjin1364 client_pushes_test.go:97: Retrieved narinfo from S3:1365 StorePath: /build/TestClientPushesUseOnePush2323145864/001/store/c990hmfn1l4700cf5awd07cyh04pp50n-b1366 URL: nar/07fh8ji51s9zv3087s2rphfp0q06gh87hxn1r4y7mqn4r098nkcx.nar.zst1367 Compression: zstd1368 NarHash: sha256:07fh8ji51s9zv3087s2rphfp0q06gh87hxn1r4y7mqn4r098nkcx1369 NarSize: 2161370 References: /build/TestClientPushesUseOnePush2323145864/001/store/n0w4n1qaskrdhqkadni3pr9mbl8p7wpk-shared-dep1371 CA: text:sha256:0nbm2fn6m12n0577ypafwaspvadwmhyam5714rjcsf1hfzk9xjin1372=== NAME TestClientCADerivations1373 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations1410889338/001/store/qnhnxbpb8dswfgf8qv2y2gd5yw1wiijf-ca-test1374 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1375 Compression: zstd1376 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1377 NarSize: 1441378 References: 1379 Deriver: /build/TestClientCADerivations1410889338/001/store/cwqkg3p6q43k8qjbfzn3bc4z650s6pj4-ca-test.drv1380 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1381 client_ca_test.go:185: Checking for realisation files in S3...1382 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1383 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1384--- PASS: TestClientPushesUseOnePush (1.33s)1385=== CONT TestSharedNarBothUploadsComplete1386=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1387=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1388=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1389=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1390=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1391=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1392=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1393=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1394=== CONT TestPush_SignsNarinfosOfItsPendingObjects13952026/11/12 00:20:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=013962026/11/12 00:20:29 INFO Vacuumed table table=pending_closures13972026/11/12 00:20:29 INFO Vacuumed table table=pending_objects13982026/11/12 00:20:29 INFO Vacuumed table table=multipart_uploads13992026/11/12 00:20:29 INFO Vacuumed table table=closures14002026/11/12 00:20:29 INFO Vacuumed table table=objects14012026/11/12 00:20:29 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1402--- PASS: TestService_AuthMiddleware (0.37s)1403=== CONT TestFailedS3DeleteKeepsTombstone14042026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures14052026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes14062026/11/12 00:20:29 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1407--- PASS: TestReadProxyConditionalGet (0.31s)1408=== CONT TestPush_RejectsBadRequests14092026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/3/complete14102026/11/12 00:20:29 WARN Failed to register uploaded object key=xibclk3l2hx200fjck2iv8dkgqi514p8.ls error="server returned 404: 404 page not found\n"14112026/11/12 00:20:29 INFO Completed push id=3 objects=3 duration=2.293279ms14122026/11/12 00:20:29 INFO Upload complete. (50ms)1413=== NAME TestClientIntegration1414 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1415 client_integration_test.go:353: Decompressed .ls content (49 bytes):1416 {"version":1,"root":{"type":"regular","size":39}}1417 client_integration_test.go:364: Testing garbage collection...1418=== NAME TestClientCADerivations1419 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1420 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1421 error: binary cache 's3://bucket25?endpoint=http://localhost:32833®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations1410889338/001/store'1422 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11423--- PASS: TestClientCADerivations (1.30s)1424=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit14252026/11/12 00:20:29 INFO Received uploads request method=POST path=/api/pending_closures14262026/11/12 00:20:29 INFO Starting cleanup of old closures method=DELETE path=/api/closures14272026/11/12 00:20:29 INFO Garbage collection started14282026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=014292026/11/12 00:20:29 WARN Force mode enabled - objects will be deleted immediately without grace period14302026/11/12 00:20:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14312026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=014322026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=014332026/11/12 00:20:29 WARN Force mode enabled - objects will be deleted immediately without grace period14342026/11/12 00:20:29 WARN Force mode enabled - objects will be deleted immediately without grace period14352026/11/12 00:20:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014362026/11/12 00:20:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01437--- PASS: TestService_Rustfstest (0.34s)1438=== CONT TestForceGCDuringPushOffersSweptObject1439=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1440=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1441=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1442=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1443=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection14442026/11/12 00:20:29 INFO Vacuumed table table=pending_closures14452026/11/12 00:20:29 INFO Vacuumed table table=pending_closures14462026/11/12 00:20:29 INFO Vacuumed table table=pending_objects14472026/11/12 00:20:29 INFO Vacuumed table table=multipart_uploads14482026/11/12 00:20:29 INFO Vacuumed table table=pending_objects14492026/11/12 00:20:29 INFO Vacuumed table table=multipart_uploads14502026/11/12 00:20:29 INFO Vacuumed table table=closures14512026/11/12 00:20:29 INFO Vacuumed table table=objects14522026/11/12 00:20:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014532026/11/12 00:20:29 INFO Vacuumed table table=closures14542026/11/12 00:20:29 INFO Vacuumed table table=pending_closures1455--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.41s)1456=== CONT TestGCSweepDeliversEachKeyOnce14572026/11/12 00:20:29 INFO Vacuumed table table=objects14582026/11/12 00:20:29 INFO Vacuumed table table=pending_objects14592026/11/12 00:20:29 INFO Vacuumed table table=multipart_uploads14602026/11/12 00:20:29 INFO Vacuumed table table=closures14612026/11/12 00:20:29 INFO Vacuumed table table=objects1462--- PASS: TestSweepRechecksPendingAfterLocking (0.39s)1463=== CONT TestPush_RootsAreLockedInByteOrder1464--- PASS: TestGetExistingObjectsLocksRows (0.38s)1465=== CONT TestPush_CompleteWaitsForCleanupOfThePush14662026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures14672026/11/12 00:20:30 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst14682026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1469--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.41s)1470=== CONT TestPush_RepeatedRootCommitsOnce14712026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1472--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.40s)1473=== CONT TestTombstonedObjectOfferedWithoutWaiting14742026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures14752026/11/12 00:20:30 INFO lead: released remote=192.0.2.1:1234 generation=11476--- PASS: TestLeadEndsOnShutdown (1.73s)1477=== CONT TestGCSweepSkipsPendingObjects14782026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures14792026/11/12 00:20:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14802026/11/12 00:20:30 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZmIxYTQ5NDItYjAyZC00ZGNhLTkyZDEtNzM0NTAzNmVjZmE4LjJiNDM3MmM0LTdjOTgtNGVmMi04ZmZkLTg2YTAzOGFlOWFmNHgxNzk0NDQyODMwMTQ4MDI4Njgz14812026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1482--- PASS: TestDeleteClosuresRacingPins (1.76s)1483=== CONT TestReadRedirectKeepsNarinfoProxied14842026/11/12 00:20:30 INFO lead: released remote=192.0.2.1:1234 generation=114852026/11/12 00:20:30 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZmIxYTQ5NDItYjAyZC00ZGNhLTkyZDEtNzM0NTAzNmVjZmE4LjJiNDM3MmM0LTdjOTgtNGVmMi04ZmZkLTg2YTAzOGFlOWFmNHgxNzk0NDQyODMwMTQ4MDI4Njgz parts=11486--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.50s)1487=== CONT TestReadRedirectNar14882026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes14892026/11/12 00:20:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14902026/11/12 00:20:30 INFO Signed narinfos id=1 count=114912026/11/12 00:20:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14922026/11/12 00:20:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14932026/11/12 00:20:30 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"14942026/11/12 00:20:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14952026/11/12 00:20:30 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"14962026/11/12 00:20:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14972026/11/12 00:20:30 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=114982026/11/12 00:20:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14992026/11/12 00:20:30 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1500--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.45s)1501=== CONT TestDeduplicatedObjectsRecordedAsPending15022026/11/12 00:20:30 INFO lead: acquired remote=192.0.2.1:1234 generation=215032026/11/12 00:20:30 INFO lead: released remote=192.0.2.1:1234 generation=115042026/11/12 00:20:30 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=215052026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures15062026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=015072026/11/12 00:20:30 INFO lead: acquired remote=192.0.2.1:1234 generation=315082026/11/12 00:20:30 WARN Force mode enabled - objects will be deleted immediately without grace period15092026/11/12 00:20:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015102026/11/12 00:20:30 INFO lead: acquired remote=192.0.2.1:1234 generation=215112026/11/12 00:20:30 INFO Vacuumed table table=pending_closures15122026/11/12 00:20:30 INFO Vacuumed table table=pending_objects15132026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads15142026/11/12 00:20:30 INFO Vacuumed table table=closures15152026/11/12 00:20:30 INFO Vacuumed table table=objects15162026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=015172026/11/12 00:20:30 WARN Force mode enabled - objects will be deleted immediately without grace period1518=== RUN TestPush_RejectsBadRequests/no_roots1519=== PAUSE TestPush_RejectsBadRequests/no_roots1520=== RUN TestPush_RejectsBadRequests/no_objects1521=== PAUSE TestPush_RejectsBadRequests/no_objects1522=== RUN TestPush_RejectsBadRequests/bad_root1523=== PAUSE TestPush_RejectsBadRequests/bad_root1524=== RUN TestPush_RejectsBadRequests/root_not_in_objects1525=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1526=== CONT TestReadProxyDisabled15272026/11/12 00:20:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=015282026/11/12 00:20:30 INFO Vacuumed table table=pending_closures15292026/11/12 00:20:30 INFO Vacuumed table table=pending_objects15302026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads15312026/11/12 00:20:30 INFO Vacuumed table table=closures15322026/11/12 00:20:30 INFO Vacuumed table table=objects15332026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes1534--- PASS: TestSweepSkipsRowsLockedByPush (0.63s)1535=== CONT TestPushDedupSurvivesConcurrentGC15362026/11/12 00:20:30 INFO Received complete push request method=POST path=/api/pushes/1/complete15372026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=015382026/11/12 00:20:30 INFO Completed push id=1 objects=4 duration=8.954536ms15392026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes15402026/11/12 00:20:30 WARN Force mode enabled - objects will be deleted immediately without grace period15412026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures15422026/11/12 00:20:30 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."15432026/11/12 00:20:30 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1544--- PASS: TestPushWaitsForSweepHoldingRow (0.82s)1545=== CONT TestReadProxyRootRedirectsToIndexHTML1546--- PASS: TestFailedS3DeleteKeepsTombstone (0.59s)1547=== CONT TestPush_OverlappingRootsStoreOneRowPerKey15482026/11/12 00:20:30 INFO lead: released remote=192.0.2.1:1234 generation=115492026/11/12 00:20:30 INFO lead: acquired remote=192.0.2.1:1234 generation=215502026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=015512026/11/12 00:20:30 WARN Force mode enabled - objects will be deleted immediately without grace period15522026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes15532026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=015542026/11/12 00:20:30 WARN Force mode enabled - objects will be deleted immediately without grace period15552026/11/12 00:20:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=015562026/11/12 00:20:30 INFO Vacuumed table table=pending_closures15572026/11/12 00:20:30 INFO Vacuumed table table=pending_objects15582026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads15592026/11/12 00:20:30 INFO Vacuumed table table=closures15602026/11/12 00:20:30 INFO Vacuumed table table=objects15612026/11/12 00:20:30 INFO Received complete push request method=POST path=/api/pushes/2/complete15622026/11/12 00:20:30 INFO Completed push id=2 objects=4 duration=2.746419ms15632026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes1564--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.61s)1565=== CONT TestPush_CompleteCommitsEveryRoot15662026-11-12 00:20:30.497 UTC [1884] ERROR: could not obtain lock on row in relation "closures"15672026-11-12 00:20:30.497 UTC [1884] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15682026-11-12 00:20:30.498 UTC [1884] ERROR: could not obtain lock on row in relation "closures"15692026-11-12 00:20:30.498 UTC [1884] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15702026-11-12 00:20:30.498 UTC [1884] ERROR: could not obtain lock on row in relation "closures"15712026-11-12 00:20:30.498 UTC [1884] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15722026-11-12 00:20:30.499 UTC [1884] ERROR: could not obtain lock on row in relation "closures"15732026-11-12 00:20:30.499 UTC [1884] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT15742026-11-12 00:20:30.499 UTC [1884] ERROR: could not obtain lock on row in relation "closures"15752026-11-12 00:20:30.499 UTC [1884] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1576--- PASS: TestPush_RootsAreLockedInByteOrder (0.51s)1577=== CONT TestPresentReportsOnlyClosureRoots15782026/11/12 00:20:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=015792026/11/12 00:20:30 INFO Vacuumed table table=pending_closures15802026/11/12 00:20:30 INFO Vacuumed table table=pending_objects15812026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads15822026/11/12 00:20:30 INFO Vacuumed table table=closures15832026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes15842026/11/12 00:20:30 INFO Vacuumed table table=objects1585--- PASS: TestGCSweepDeliversEachKeyOnce (0.54s)1586=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded15872026/11/12 00:20:30 INFO Received complete push request method=POST path=/api/pushes/1/complete15882026/11/12 00:20:30 INFO Completed push id=1 objects=4 duration=15.536704ms1589--- PASS: TestPush_RepeatedRootCommitsOnce (0.50s)1590=== CONT TestReadProxyOutlastsServerWriteTimeout15912026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1592--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.47s)1593=== CONT TestCreatePin_ReservedPins15942026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1595--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.63s)1596=== CONT TestReadProxyRangeRequest15972026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=015982026/11/12 00:20:30 WARN Force mode enabled - objects will be deleted immediately without grace period15992026/11/12 00:20:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016002026/11/12 00:20:30 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45365/oidc16012026/11/12 00:20:30 INFO Vacuumed table table=pending_closures16022026/11/12 00:20:30 INFO Vacuumed table table=pending_objects16032026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads16042026/11/12 00:20:30 INFO Vacuumed table table=closures1605--- PASS: TestReadRedirectKeepsNarinfoProxied (0.45s)1606=== CONT TestOrphanedObjectsGCStressTest16072026/11/12 00:20:30 INFO Vacuumed table table=objects1608--- PASS: TestGCSweepSkipsPendingObjects (0.49s)1609=== CONT TestReadRedirectUsesPublicS3URL16102026/11/12 00:20:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1611--- PASS: TestReadRedirectNar (0.47s)1612=== CONT TestOrphanedObjectsGC16132026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures16142026/11/12 00:20:30 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ZmIxYTQ5NDItYjAyZC00ZGNhLTkyZDEtNzM0NTAzNmVjZmE4LjJmYjI5ZDlmLTQwZjItNGExYy1hZjE4LTFlZmJmYWNmNzZhYXgxNzk0NDQyODMwMDc0MjMxNTkx parts=1216152026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1616--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.48s)1617=== CONT TestObjectStatsUpdatedOncePerCommit1618--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.03s)1619=== CONT TestReadProxyHead1620--- PASS: TestReadProxyDisabled (0.41s)1621=== CONT TestObjectStatsTrigger16222026/11/12 00:20:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16232026-11-12 00:20:30.807 UTC [1887] ERROR: Push does not exist: id=116242026-11-12 00:20:30.807 UTC [1887] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE16252026-11-12 00:20:30.807 UTC [1887] STATEMENT: SELECT commit_push($1)1626--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (0.79s)1627=== CONT TestService_NativeMTLS16282026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes16292026/11/12 00:20:30 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZmIxYTQ5NDItYjAyZC00ZGNhLTkyZDEtNzM0NTAzNmVjZmE4LmYzNmVmOWM3LTM3MzgtNDg2OC1iNDRmLWUxNGQ4MDM3ZGNiMXgxNzk0NDQyODMwMTgwMDUyNDEw parts=1216302026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1631--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.44s)1632=== CONT TestReadProxyInvalidPath1633--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.45s)1634=== CONT TestMultipartCleanup16352026/11/12 00:20:30 INFO Received push request method=POST path=/api/pushes16362026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=016372026/11/12 00:20:30 INFO Received complete push request method=POST path=/api/pushes/1/complete16382026/11/12 00:20:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016392026/11/12 00:20:30 INFO Vacuumed table table=pending_closures16402026/11/12 00:20:30 INFO Completed push id=1 objects=6 duration=7.617027ms1641--- PASS: TestPush_CompleteCommitsEveryRoot (0.41s)1642=== CONT TestServerTLSConfig1643=== RUN TestServerTLSConfig/no_client_CA1644=== PAUSE TestServerTLSConfig/no_client_CA1645=== RUN TestServerTLSConfig/missing_CA_file1646=== PAUSE TestServerTLSConfig/missing_CA_file1647=== RUN TestServerTLSConfig/not_a_PEM_file1648=== PAUSE TestServerTLSConfig/not_a_PEM_file1649=== CONT TestPendingCleanupUsesOneCutoff16502026/11/12 00:20:30 INFO Vacuumed table table=pending_objects16512026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads16522026/11/12 00:20:30 INFO Vacuumed table table=closures16532026/11/12 00:20:30 INFO Vacuumed table table=objects16542026/11/12 00:20:30 INFO Aborted multipart uploads count=0 kept=016552026/11/12 00:20:30 WARN Force mode enabled - objects will be deleted immediately without grace period16562026/11/12 00:20:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01657--- PASS: TestPresentReportsOnlyClosureRoots (0.42s)1658=== CONT TestReadProxyNarinfo16592026/11/12 00:20:30 INFO Vacuumed table table=pending_closures16602026/11/12 00:20:30 INFO Vacuumed table table=pending_objects16612026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads16622026/11/12 00:20:30 INFO Vacuumed table table=closures16632026/11/12 00:20:30 INFO Vacuumed table table=objects1664--- PASS: TestPushDedupSurvivesConcurrentGC (0.58s)1665=== CONT TestIsValidCachePath1666=== RUN TestIsValidCachePath/narinfo1667=== PAUSE TestIsValidCachePath/narinfo1668=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1669=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1670=== RUN TestIsValidCachePath/nar_zst1671=== PAUSE TestIsValidCachePath/nar_zst1672=== RUN TestIsValidCachePath/nar_xz1673=== PAUSE TestIsValidCachePath/nar_xz1674=== RUN TestIsValidCachePath/nar_bz21675=== PAUSE TestIsValidCachePath/nar_bz21676=== RUN TestIsValidCachePath/nar_uncompressed1677=== PAUSE TestIsValidCachePath/nar_uncompressed1678=== RUN TestIsValidCachePath/ls1679=== PAUSE TestIsValidCachePath/ls1680=== RUN TestIsValidCachePath/log1681=== PAUSE TestIsValidCachePath/log1682=== RUN TestIsValidCachePath/realisation1683=== PAUSE TestIsValidCachePath/realisation1684=== RUN TestIsValidCachePath/nix-cache-info1685=== PAUSE TestIsValidCachePath/nix-cache-info1686=== RUN TestIsValidCachePath/index.html1687=== PAUSE TestIsValidCachePath/index.html1688=== RUN TestIsValidCachePath/traversal_parent1689=== PAUSE TestIsValidCachePath/traversal_parent1690=== RUN TestIsValidCachePath/traversal_in_middle1691=== PAUSE TestIsValidCachePath/traversal_in_middle1692=== RUN TestIsValidCachePath/invalid_char_e1693=== PAUSE TestIsValidCachePath/invalid_char_e1694=== RUN TestIsValidCachePath/invalid_char_u1695=== PAUSE TestIsValidCachePath/invalid_char_u1696=== RUN TestIsValidCachePath/random_path1697=== PAUSE TestIsValidCachePath/random_path1698=== RUN TestIsValidCachePath/empty1699=== PAUSE TestIsValidCachePath/empty1700=== RUN TestIsValidCachePath/leading_slash1701=== PAUSE TestIsValidCachePath/leading_slash1702=== RUN TestIsValidCachePath/wrong_extension1703=== PAUSE TestIsValidCachePath/wrong_extension1704=== RUN TestIsValidCachePath/short_hash1705=== PAUSE TestIsValidCachePath/short_hash1706=== CONT TestPresentNotReportedWhileGCDeletesClosure1707=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1708=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1709=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1710=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1711=== CONT TestReadProxy40417122026/11/12 00:20:30 INFO Received uploads request method=POST path=/api/pending_closures1713--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.44s)1714=== CONT TestParseSingleRange1715=== RUN TestParseSingleRange/none1716=== PAUSE TestParseSingleRange/none1717=== RUN TestParseSingleRange/unknown_unit1718=== PAUSE TestParseSingleRange/unknown_unit1719=== RUN TestParseSingleRange/multi-range_ignored1720=== PAUSE TestParseSingleRange/multi-range_ignored1721=== RUN TestParseSingleRange/malformed_no_dash1722=== PAUSE TestParseSingleRange/malformed_no_dash1723=== RUN TestParseSingleRange/malformed_both_empty1724=== PAUSE TestParseSingleRange/malformed_both_empty1725=== RUN TestParseSingleRange/malformed_end_before_start1726=== PAUSE TestParseSingleRange/malformed_end_before_start1727=== RUN TestParseSingleRange/closed1728=== PAUSE TestParseSingleRange/closed1729=== RUN TestParseSingleRange/open-ended1730=== PAUSE TestParseSingleRange/open-ended1731=== RUN TestParseSingleRange/end_clamped_to_size1732=== PAUSE TestParseSingleRange/end_clamped_to_size1733=== RUN TestParseSingleRange/suffix1734=== PAUSE TestParseSingleRange/suffix1735=== RUN TestParseSingleRange/suffix_exceeds_size1736=== PAUSE TestParseSingleRange/suffix_exceeds_size1737=== RUN TestParseSingleRange/single_byte1738=== PAUSE TestParseSingleRange/single_byte1739=== RUN TestParseSingleRange/start_past_EOF1740=== PAUSE TestParseSingleRange/start_past_EOF1741=== RUN TestParseSingleRange/start_far_past_EOF1742=== PAUSE TestParseSingleRange/start_far_past_EOF1743=== CONT TestReadProxyNarStreaming1744--- PASS: TestReadProxyRangeRequest (0.41s)1745=== CONT TestPresentRefreshesOnlyStaleClosures17462026/11/12 00:20:31 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux17472026/11/12 00:20:31 WARN Refused reserved pin name=worker-x86_64-linux17482026/11/12 00:20:31 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux17492026/11/12 00:20:31 INFO Received create pin request method=POST path=/api/pins/my-app17502026/11/12 00:20:31 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1751--- PASS: TestCreatePin_ReservedPins (0.47s)1752=== CONT TestReadProxyContentNegotiation1753--- PASS: TestReadRedirectUsesPublicS3URL (0.48s)1754=== CONT TestReadProxyNarinfoAlreadyDecompressed1755--- PASS: TestObjectStatsUpdatedOncePerCommit (0.51s)1756=== CONT TestMetricsInventory1757--- PASS: TestReadProxyHead (0.51s)1758=== CONT TestMetricsCommit1759--- PASS: TestObjectStatsTrigger (0.53s)1760=== CONT TestNARDeduplicationMetadataUploadBug17612026/11/12 00:20:31 WARN mTLS auth: subject not in bound subjects subject="CN=reader"17622026/11/12 00:20:31 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1763--- PASS: TestService_NativeMTLS (0.45s)1764=== CONT TestResolveDBConnectionString/flag_wins1765=== CONT TestResolveDBConnectionString/nothing_configured1766=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1767=== CONT TestResolveDBConnectionString/missing_file_is_an_error1768=== CONT TestResolveDBConnectionString/file_when_flag_empty1769=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info17702026/11/12 00:20:31 INFO Received uploads request method=POST path=/1771=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key17722026/11/12 00:20:31 INFO Received request for more parts method=POST path=/1773=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal17742026/11/12 00:20:31 INFO Received uploads request method=POST path=/1775--- PASS: TestResolveDBConnectionString (0.09s)1776 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1777 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1778 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1779 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1780 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1781=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key17822026/11/12 00:20:31 INFO Received complete multipart upload request method=POST path=/1783--- PASS: TestUploadHandlersRejectInvalidKeys (0.09s)1784 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1785 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1786 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1787 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1788=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1789--- PASS: TestReadProxyInvalidPath (0.46s)1790=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed17912026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures17922026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures17932026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures17942026/11/12 00:20:31 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17952026/11/12 00:20:31 INFO Received cleanup request method=DELETE path=/api/pending_closures17962026/11/12 00:20:31 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZmIxYTQ5NDItYjAyZC00ZGNhLTkyZDEtNzM0NTAzNmVjZmE4LjhiMTAwZGRjLWI1MzgtNDJmNy05NzQxLTA2ZWVmNGY2MGUyOHgxNzk0NDQyODMwMTkwNDA4Njg3 parts=121797--- PASS: TestSharedNarBothUploadsComplete (1.64s)1798=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed17992026/11/12 00:20:31 INFO lead: released remote=192.0.2.1:1234 generation=21800--- PASS: TestLeadElectsOneAndHandsOver (2.98s)1801=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost18022026/11/12 00:20:31 INFO Received cleanup request method=DELETE path=/api/pending_closures18032026/11/12 00:20:31 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket83/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""18042026/11/12 00:20:31 INFO Aborted multipart uploads count=0 kept=118052026/11/12 00:20:31 INFO Received cleanup request method=DELETE path=/api/pending_closures18062026/11/12 00:20:31 INFO Aborted multipart uploads count=1 kept=01807--- PASS: TestMultipartCleanup (0.61s)1808=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1809--- PASS: TestReadProxy404 (0.53s)1810=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure18112026/11/12 00:20:31 INFO Received uploads request method=POST path=/1812--- PASS: TestReadProxyNarStreaming (0.50s)1813=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts18142026/11/12 00:20:31 INFO Received request for more parts method=POST path=/1815--- PASS: TestPresentRefreshesOnlyStaleClosures (0.48s)1816=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart18172026/11/12 00:20:31 INFO Received complete multipart upload request method=POST path=/18182026/11/12 00:20:31 INFO lead: released remote=192.0.2.1:1234 generation=218192026/11/12 00:20:31 INFO lead: acquired remote=192.0.2.1:1234 generation=318202026/11/12 00:20:31 INFO lead: released remote=192.0.2.1:1234 generation=31821--- PASS: TestLeadGenerationGrowsWithEveryGrant (3.10s)1822=== CONT TestClientErrorHandling/InvalidStorePath18232026/11/12 00:20:31 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1824--- PASS: TestReadProxyNarinfo (0.62s)1825=== CONT TestClientErrorHandling/ServerNotAvailable1826=== CONT TestClientErrorHandling/InvalidAuthToken1827=== NAME TestOrphanedObjectsGC1828 orphaned_objects_gc_test.go:287: GC Test Summary:1829 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1830 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1831 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1832 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1833 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1834--- PASS: TestOrphanedObjectsGC (0.87s)1835=== CONT TestCacheConfigHandler/full_config,_no_issuer1836=== CONT TestCacheConfigHandler/no_signing_keys1837=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1838=== CONT TestCacheConfigHandler/no_cache_url_configured1839--- PASS: TestCacheConfigHandler (0.00s)1840 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1841 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1842 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1843 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1844=== CONT TestIsValidUploadKey/narinfo1845=== CONT TestIsValidUploadKey/unknown_type1846=== CONT TestIsValidUploadKey/empty_key1847=== CONT TestIsValidUploadKey/build_log_equals1848=== CONT TestIsValidUploadKey/build_log_question_mark1849=== CONT TestIsValidUploadKey/build_log_plus_in_name1850=== CONT TestIsValidUploadKey/realisation1851=== CONT TestIsValidUploadKey/build_log_home-manager_file1852=== CONT TestIsValidUploadKey/absolute1853=== CONT TestIsValidUploadKey/build_log1854=== CONT TestIsValidUploadKey/traversal_nar1855=== CONT TestIsValidUploadKey/listing1856=== CONT TestIsValidUploadKey/traversal1857=== CONT TestIsValidUploadKey/nar_zst1858=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1859=== CONT TestIsValidUploadKey/index.html1860=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1861=== CONT TestIsValidUploadKey/nix-cache-info1862=== CONT TestIsValidUploadKey/nar_xz1863=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1864=== CONT TestIsValidUploadKey/realisation_plus_in_output1865=== CONT TestIsValidUploadKey/nar_plain1866--- PASS: TestIsValidUploadKey (0.00s)1867 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1868 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1869 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1870 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1871 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1872 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1873 --- PASS: TestIsValidUploadKey/realisation (0.00s)1874 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1875 --- PASS: TestIsValidUploadKey/absolute (0.00s)1876 --- PASS: TestIsValidUploadKey/build_log (0.00s)1877 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1878 --- PASS: TestIsValidUploadKey/listing (0.00s)1879 --- PASS: TestIsValidUploadKey/traversal (0.00s)1880 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1881 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1882 --- PASS: TestIsValidUploadKey/index.html (0.00s)1883 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1884 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1885 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1886 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1887 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1888 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1889=== CONT TestPendingClosureWriteTimeout/empty1890=== CONT TestPendingClosureWriteTimeout/670k_objects1891=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1892--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.42s)1893=== CONT TestPendingClosureWriteTimeout/400_objects1894=== CONT TestProxyWriteTimeout/narinfo1895=== CONT TestProxyWriteTimeout/10_GiB_nar1896=== CONT TestProxyWriteTimeout/unknown_size1897=== CONT TestProxyWriteTimeout/1_GiB_nar1898--- PASS: TestProxyWriteTimeout (0.00s)1899 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1900 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1901 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1902 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1903=== CONT TestGCEndsOnShutdown/before_the_run1904=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1905--- PASS: TestReadProxyContentNegotiation (0.53s)1906=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1907--- PASS: TestMetricsInventory (0.38s)1908=== CONT TestService_RequireScope_OIDC/builder_may_write1909=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1910=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19112026/11/12 00:20:31 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1912=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1913=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1914=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1915=== CONT TestService_RequireScope_OIDC/writer_implies_read1916=== CONT TestService_RequireScope_OIDC/reader_may_read1917=== CONT TestService_RequireScope_OIDC/static_token_may_write1918=== CONT TestService_RequireScope_OIDC/static_token_may_admin1919=== CONT TestService_RequireScope_OIDC/reader_may_not_write1920=== CONT TestService_RequireScope_OIDC/ops_may_not_write1921=== CONT TestService_RequireScope_OIDC/ops_may_admin1922=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1923=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1924--- PASS: TestService_RequireScope_OIDC (0.41s)1925 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1926 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1927 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1928 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1929 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1930 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1931 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1932 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1933 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1934 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1935 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1936 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1937 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1938 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)19392026/11/12 00:20:31 INFO Received push request method=POST path=/api/pushes1940=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected19412026/11/12 00:20:31 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1942=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected19432026/11/12 00:20:31 WARN Authentication failed token_preview=eyJhbGciOi...cUVsnh307Q token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1944=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1945=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1946--- PASS: TestService_AuthMiddleware_OIDC (0.43s)1947 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1948 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1949 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)1950 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)19512026/11/12 00:20:31 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19522026/11/12 00:20:31 INFO Received complete push request method=POST path=/api/pushes/1/complete19532026/11/12 00:20:31 INFO Completed push id=1 objects=4 duration=9.427546ms19542026/11/12 00:20:31 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete19552026/11/12 00:20:31 INFO Completed upload id=2 objects=2 duration=1.84608ms1956--- PASS: TestMetricsCommit (0.44s)1957=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check19582026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures1959=== NAME TestNARDeduplicationMetadataUploadBug1960 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug3546075453/001/store/li9fv6yc2jh9s6x3yqrz92m16i6ak60z-file1.txt19612026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures19622026/11/12 00:20:31 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=019632026/11/12 00:20:31 INFO Received create pin request method=POST path=/api/pins/myapp19642026/11/12 00:20:31 INFO Created/updated pin name=myapp store_path=/build/TestPinProtectsFromGC2658795018/001/store/mm4lcv1lxyiqsaxv3rn7imbj13n4jsq6-pinned-file.txt narinfo_key=mm4lcv1lxyiqsaxv3rn7imbj13n4jsq6.narinfo19652026/11/12 00:20:31 WARN Request failed, retrying attempt=2 backoff=211.07606ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19662026/11/12 00:20:31 INFO Starting cleanup of old closures method=DELETE path=/api/closures19672026/11/12 00:20:31 INFO Garbage collection started19682026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures19692026/11/12 00:20:31 INFO Aborted multipart uploads count=0 kept=019702026/11/12 00:20:31 WARN Force mode enabled - objects will be deleted immediately without grace period19712026/11/12 00:20:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019722026/11/12 00:20:31 INFO Vacuumed table table=pending_closures19732026/11/12 00:20:31 INFO Vacuumed table table=pending_objects19742026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures19752026/11/12 00:20:31 INFO Vacuumed table table=multipart_uploads19762026/11/12 00:20:31 INFO Vacuumed table table=closures19772026/11/12 00:20:31 INFO Vacuumed table table=objects19782026/11/12 00:20:31 INFO Received push request method=POST path=/api/pushes19792026/11/12 00:20:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19802026/11/12 00:20:31 INFO Uploading li9fv6yc2jh9s6x3yqrz92m16i6ak60z-file1.txt (160B)19812026/11/12 00:20:31 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"19822026/11/12 00:20:31 WARN Failed to register uploaded object key=li9fv6yc2jh9s6x3yqrz92m16i6ak60z.ls error="server returned 404: 404 page not found\n"19832026/11/12 00:20:31 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19842026/11/12 00:20:31 INFO Signed narinfos id=1 count=119852026/11/12 00:20:31 INFO Uploading 1 narinfos19862026/11/12 00:20:31 INFO Received uploads request method=POST path=/api/pending_closures19872026/11/12 00:20:31 INFO Received complete push request method=POST path=/api/pushes/1/complete19882026/11/12 00:20:31 WARN Failed to register uploaded object key=li9fv6yc2jh9s6x3yqrz92m16i6ak60z.narinfo error="server returned 404: 404 page not found\n"19892026/11/12 00:20:31 INFO Completed push id=1 objects=3 duration=3.901568ms19902026/11/12 00:20:31 INFO Upload complete. (71ms)1991 metadata_upload_test.go:54: Retrieved narinfo from S3:1992 StorePath: /build/TestNARDeduplicationMetadataUploadBug3546075453/001/store/li9fv6yc2jh9s6x3yqrz92m16i6ak60z-file1.txt1993 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1994 Compression: zstd1995 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1996 NarSize: 1601997 References: 1998 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf19992026/11/12 00:20:31 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=020002026/11/12 00:20:31 WARN Request failed, retrying attempt=3 backoff=433.365352ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present
loading 635 hidden lines…
26362026/11/12 00:20:44 INFO Vacuumed table table=pending_objects26372026/11/12 00:20:44 INFO Vacuumed table table=multipart_uploads26382026/11/12 00:20:44 INFO Aborted multipart uploads count=0 kept=026392026/11/12 00:20:44 INFO Vacuumed table table=closures26402026/11/12 00:20:44 INFO Completed push id=3 objects=5 duration=6.700988ms26412026/11/12 00:20:44 WARN Force mode enabled - objects will be deleted immediately without grace period26422026/11/12 00:20:44 INFO Received push request method=POST path=/26432026/11/12 00:20:44 INFO Vacuumed table table=objects26442026/11/12 00:20:44 INFO Received push request method=POST path=/26452026/11/12 00:20:44 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."26462026/11/12 00:20:44 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."26472026/11/12 00:20:44 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."26482026/11/12 00:20:44 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."26492026/11/12 00:20:44 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."26502026/11/12 00:20:44 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."26512026/11/12 00:20:44 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."26522026/11/12 00:20:44 INFO Received complete push request method=POST path=/26532026/11/12 00:20:44 INFO Received push request method=POST path=/26542026/11/12 00:20:44 INFO Completed push id=2 objects=4 duration=2.211639ms26552026/11/12 00:20:44 INFO Received push request method=POST path=/26562026/11/12 00:20:44 INFO Received uploads request method=POST path=/26572026/11/12 00:20:44 INFO Aborted multipart uploads count=0 kept=026582026/11/12 00:20:44 INFO Received push request method=POST path=/26592026/11/12 00:20:44 INFO Received uploads request method=POST path=/26602026/11/12 00:20:44 INFO Received uploads request method=POST path=/26612026/11/12 00:20:44 WARN Force mode enabled - objects will be deleted immediately without grace period26622026/11/12 00:20:44 INFO Aborted multipart uploads count=0 kept=026632026/11/12 00:20:44 INFO Received complete upload request method=POST path=/26642026/11/12 00:20:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=026652026/11/12 00:20:44 INFO Vacuumed table table=pending_closures26662026/11/12 00:20:44 INFO Vacuumed table table=pending_objects26672026/11/12 00:20:44 INFO Vacuumed table table=multipart_uploads26682026/11/12 00:20:44 INFO Vacuumed table table=closures26692026/11/12 00:20:44 INFO Completed upload id=2 objects=26 duration=5.454277ms26702026/11/12 00:20:44 INFO Vacuumed table table=objects26712026/11/12 00:20:44 INFO Received push request method=POST path=/26722026/11/12 00:20:44 INFO Received push request method=POST path=/26732026/11/12 00:20:44 INFO Received uploads request method=POST path=/26742026/11/12 00:20:44 INFO Received complete push request method=POST path=/26752026/11/12 00:20:44 INFO Aborted multipart uploads count=0 kept=026762026/11/12 00:20:44 INFO Received push request method=POST path=/26772026/11/12 00:20:44 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."26782026/11/12 00:20:44 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."26792026/11/12 00:20:44 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."26802026/11/12 00:20:44 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."26812026/11/12 00:20:44 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."26822026/11/12 00:20:44 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."26832026/11/12 00:20:44 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."26842026/11/12 00:20:44 INFO Aborted multipart uploads count=0 kept=026852026/11/12 00:20:44 INFO Completed push id=1 objects=7 duration=7.190827ms26862026/11/12 00:20:44 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026872026/11/12 00:20:44 INFO Received push request method=POST path=/26882026/11/12 00:20:44 INFO Vacuumed table table=pending_closures26892026/11/12 00:20:44 INFO Received complete upload request method=POST path=/26902026/11/12 00:20:44 INFO Vacuumed table table=pending_objects26912026/11/12 00:20:44 INFO Vacuumed table table=multipart_uploads26922026/11/12 00:20:44 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."26932026/11/12 00:20:44 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."26942026/11/12 00:20:44 INFO Received complete push request method=POST path=/26952026/11/12 00:20:44 INFO Received complete push request method=POST path=/26962026/11/12 00:20:44 INFO Vacuumed table table=closures26972026/11/12 00:20:44 INFO Completed upload id=9 objects=7 duration=2.629299ms26982026/11/12 00:20:44 INFO Vacuumed table table=objects26992026/11/12 00:20:44 INFO Completed push id=3 objects=5 duration=2.203439ms27002026/11/12 00:20:44 INFO Completed push id=4 objects=7 duration=2.062849ms27012026/11/12 00:20:44 INFO Aborted multipart uploads count=0 kept=027022026/11/12 00:20:44 INFO Received complete upload request method=POST path=/27032026/11/12 00:20:44 INFO Completed upload id=4 objects=7 duration=2.191999ms27042026/11/12 00:20:44 INFO Received uploads request method=POST path=/27052026/11/12 00:20:44 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027062026/11/12 00:20:44 INFO Vacuumed table table=pending_closures27072026/11/12 00:20:44 INFO Vacuumed table table=pending_objects27082026/11/12 00:20:44 INFO Vacuumed table table=multipart_uploads27092026/11/12 00:20:44 INFO Vacuumed table table=closures27102026/11/12 00:20:44 INFO Vacuumed table table=objects27112026/11/12 00:20:44 INFO Received uploads request method=POST path=/27122026/11/12 00:20:44 INFO Received complete push request method=POST path=/27132026/11/12 00:20:44 INFO Received push request method=POST path=/27142026/11/12 00:20:44 INFO Received push request method=POST path=/27152026/11/12 00:20:44 INFO Completed push id=4 objects=4 duration=3.653988ms27162026/11/12 00:20:44 INFO Received complete push request method=POST path=/27172026/11/12 00:20:44 INFO Received uploads request method=POST path=/27182026/11/12 00:20:45 INFO Completed push id=8 objects=13 duration=2.273089ms27192026/11/12 00:20:45 INFO Received complete upload request method=POST path=/27202026/11/12 00:20:45 INFO Completed upload id=3 objects=6 duration=4.954688ms27212026/11/12 00:20:45 INFO Received uploads request method=POST path=/27222026/11/12 00:20:45 INFO Received push request method=POST path=/27232026/11/12 00:20:45 INFO Received uploads request method=POST path=/27242026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=027252026/11/12 00:20:45 INFO Received uploads request method=POST path=/27262026/11/12 00:20:45 INFO Received push request method=POST path=/27272026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=027282026/11/12 00:20:45 INFO Received uploads request method=POST path=/27292026/11/12 00:20:45 INFO Received complete push request method=POST path=/27302026/11/12 00:20:45 INFO Received uploads request method=POST path=/27312026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."27322026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."27332026/11/12 00:20:45 INFO Received uploads request method=POST path=/27342026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27352026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27362026/11/12 00:20:45 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27372026/11/12 00:20:45 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27382026/11/12 00:20:45 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27392026/11/12 00:20:45 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27402026/11/12 00:20:45 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27412026/11/12 00:20:45 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27422026/11/12 00:20:45 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."27432026/11/12 00:20:45 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."27442026/11/12 00:20:45 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."27452026/11/12 00:20:45 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27462026/11/12 00:20:45 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27472026/11/12 00:20:45 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27482026/11/12 00:20:45 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27492026/11/12 00:20:45 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."27502026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27512026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27522026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."27532026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."27542026/11/12 00:20:45 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27552026/11/12 00:20:45 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27562026/11/12 00:20:45 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27572026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027582026/11/12 00:20:45 INFO Completed push id=6 objects=6 duration=2.296129ms27592026/11/12 00:20:45 INFO Received uploads request method=POST path=/27602026/11/12 00:20:45 INFO Vacuumed table table=pending_closures27612026/11/12 00:20:45 INFO Received push request method=POST path=/27622026/11/12 00:20:45 INFO Vacuumed table table=pending_objects27632026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads27642026/11/12 00:20:45 INFO Received push request method=POST path=/27652026/11/12 00:20:45 INFO Vacuumed table table=closures27662026/11/12 00:20:45 INFO Vacuumed table table=objects27672026/11/12 00:20:45 INFO Received complete push request method=POST path=/27682026/11/12 00:20:45 INFO Received push request method=POST path=/27692026/11/12 00:20:45 INFO Completed push id=12 objects=3 duration=1.775639ms27702026/11/12 00:20:45 INFO Received uploads request method=POST path=/27712026/11/12 00:20:45 INFO Received push request method=POST path=/27722026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=027732026/11/12 00:20:45 INFO Received push request method=POST path=/27742026/11/12 00:20:45 INFO Received push request method=POST path=/27752026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=027762026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period27772026/11/12 00:20:45 INFO Received uploads request method=POST path=/27782026/11/12 00:20:45 INFO Received uploads request method=POST path=/27792026/11/12 00:20:45 INFO Received complete push request method=POST path=/27802026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027812026/11/12 00:20:45 INFO Vacuumed table table=pending_closures27822026/11/12 00:20:45 INFO Received uploads request method=POST path=/27832026/11/12 00:20:45 INFO Completed push id=4 objects=4 duration=2.349249ms27842026/11/12 00:20:45 INFO Vacuumed table table=pending_objects27852026/11/12 00:20:45 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."27862026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."27872026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."27882026/11/12 00:20:45 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27892026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads27902026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27912026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27922026/11/12 00:20:45 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27932026/11/12 00:20:45 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27942026/11/12 00:20:45 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."27952026/11/12 00:20:45 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."27962026/11/12 00:20:45 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27972026/11/12 00:20:45 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27982026/11/12 00:20:45 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27992026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."28002026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28012026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."28022026/11/12 00:20:45 INFO Vacuumed table table=closures28032026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028042026/11/12 00:20:45 INFO Received uploads request method=POST path=/28052026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28062026/11/12 00:20:45 INFO Vacuumed table table=objects28072026/11/12 00:20:45 INFO Received uploads request method=POST path=/28082026/11/12 00:20:45 INFO Received complete upload request method=POST path=/28092026/11/12 00:20:45 INFO Received push request method=POST path=/28102026/11/12 00:20:45 INFO Completed upload id=5 objects=7 duration=3.119419ms28112026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028122026/11/12 00:20:45 INFO Received push request method=POST path=/28132026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period28142026/11/12 00:20:45 INFO Received push request method=POST path=/28152026/11/12 00:20:45 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28162026/11/12 00:20:45 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28172026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28182026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028192026/11/12 00:20:45 INFO Vacuumed table table=pending_closures28202026/11/12 00:20:45 INFO Vacuumed table table=pending_objects28212026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads28222026/11/12 00:20:45 INFO Received push request method=POST path=/28232026/11/12 00:20:45 INFO Received uploads request method=POST path=/28242026/11/12 00:20:45 INFO Vacuumed table table=closures28252026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028262026/11/12 00:20:45 INFO Received push request method=POST path=/28272026/11/12 00:20:45 INFO Received push request method=POST path=/28282026/11/12 00:20:45 INFO Vacuumed table table=objects28292026/11/12 00:20:45 INFO Received uploads request method=POST path=/28302026/11/12 00:20:45 INFO Received push request method=POST path=/28312026/11/12 00:20:45 INFO Received push request method=POST path=/28322026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028332026/11/12 00:20:45 INFO Received complete upload request method=POST path=/28342026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period28352026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028362026/11/12 00:20:45 INFO Completed upload id=10 objects=4 duration=3.156528ms28372026/11/12 00:20:45 INFO Vacuumed table table=pending_closures28382026/11/12 00:20:45 INFO Vacuumed table table=pending_objects28392026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads28402026/11/12 00:20:45 INFO Vacuumed table table=closures28412026/11/12 00:20:45 INFO Received complete upload request method=POST path=/28422026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028432026/11/12 00:20:45 INFO Received push request method=POST path=/28442026/11/12 00:20:45 INFO Vacuumed table table=objects28452026/11/12 00:20:45 INFO Completed upload id=6 objects=9 duration=1.536209ms28462026/11/12 00:20:45 INFO Received push request method=POST path=/28472026/11/12 00:20:45 INFO Received uploads request method=POST path=/28482026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period28492026/11/12 00:20:45 INFO Received push request method=POST path=/28502026/11/12 00:20:45 INFO Received push request method=POST path=/28512026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028522026/11/12 00:20:45 INFO Received complete push request method=POST path=/28532026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028542026/11/12 00:20:45 INFO Completed push id=7 objects=27 duration=2.619779ms28552026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028562026/11/12 00:20:45 INFO Received uploads request method=POST path=/28572026/11/12 00:20:45 INFO Vacuumed table table=pending_closures28582026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=028592026/11/12 00:20:45 INFO Vacuumed table table=pending_objects28602026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads28612026/11/12 00:20:45 INFO Received push request method=POST path=/28622026/11/12 00:20:45 INFO Vacuumed table table=pending_closures28632026/11/12 00:20:45 INFO Vacuumed table table=closures28642026/11/12 00:20:45 INFO Received uploads request method=POST path=/28652026/11/12 00:20:45 INFO Vacuumed table table=pending_objects28662026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads28672026/11/12 00:20:45 INFO Vacuumed table table=objects28682026/11/12 00:20:45 INFO Received push request method=POST path=/28692026/11/12 00:20:45 INFO Vacuumed table table=closures28702026/11/12 00:20:45 INFO Received uploads request method=POST path=/28712026/11/12 00:20:45 INFO Vacuumed table table=objects28722026/11/12 00:20:45 INFO Received uploads request method=POST path=/28732026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28742026/11/12 00:20:45 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28752026/11/12 00:20:45 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28762026/11/12 00:20:45 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28772026/11/12 00:20:45 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28782026/11/12 00:20:45 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28792026/11/12 00:20:45 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28802026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28812026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28822026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28832026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."28842026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028852026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period28862026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=028872026/11/12 00:20:45 INFO Received push request method=POST path=/28882026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028892026/11/12 00:20:45 INFO Vacuumed table table=pending_closures28902026/11/12 00:20:45 INFO Vacuumed table table=pending_objects28912026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads28922026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=028932026/11/12 00:20:45 INFO Vacuumed table table=closures28942026/11/12 00:20:45 INFO Received push request method=POST path=/28952026/11/12 00:20:45 INFO Vacuumed table table=pending_closures28962026/11/12 00:20:45 INFO Vacuumed table table=objects28972026/11/12 00:20:45 INFO Received push request method=POST path=/28982026/11/12 00:20:45 INFO Received push request method=POST path=/28992026/11/12 00:20:45 INFO Vacuumed table table=pending_objects29002026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads29012026/11/12 00:20:45 INFO Received push request method=POST path=/29022026/11/12 00:20:45 INFO Received uploads request method=POST path=/29032026/11/12 00:20:45 INFO Vacuumed table table=closures29042026/11/12 00:20:45 INFO Received uploads request method=POST path=/29052026/11/12 00:20:45 INFO Vacuumed table table=objects29062026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=029072026/11/12 00:20:45 INFO Received complete push request method=POST path=/29082026/11/12 00:20:45 INFO Received complete upload request method=POST path=/29092026/11/12 00:20:45 INFO Received push request method=POST path=/29102026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period29112026/11/12 00:20:45 INFO Completed upload id=3 objects=9 duration=1.66879ms29122026/11/12 00:20:45 INFO Completed push id=10 objects=2 duration=2.408679ms29132026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=029142026/11/12 00:20:45 INFO Vacuumed table table=pending_closures29152026/11/12 00:20:45 INFO Vacuumed table table=pending_objects29162026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads29172026/11/12 00:20:45 INFO Vacuumed table table=closures29182026/11/12 00:20:45 INFO Received complete upload request method=POST path=/29192026/11/12 00:20:45 INFO Vacuumed table table=objects29202026/11/12 00:20:45 INFO Completed upload id=16 objects=23 duration=1.917769ms29212026/11/12 00:20:45 INFO Received complete push request method=POST path=/29222026/11/12 00:20:45 INFO Received push request method=POST path=/29232026/11/12 00:20:45 INFO Received uploads request method=POST path=/29242026/11/12 00:20:45 INFO Completed push id=6 objects=5 duration=4.499098ms29252026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=029262026/11/12 00:20:45 INFO Received complete upload request method=POST path=/29272026/11/12 00:20:45 INFO Received push request method=POST path=/29282026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029292026/11/12 00:20:45 INFO Completed upload id=3 objects=5 duration=1.430029ms29302026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29312026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29322026/11/12 00:20:45 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29332026/11/12 00:20:45 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."29342026/11/12 00:20:45 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29352026/11/12 00:20:45 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29362026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29372026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29382026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29392026/11/12 00:20:45 INFO Vacuumed table table=pending_closures29402026/11/12 00:20:45 INFO Vacuumed table table=pending_objects29412026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads29422026/11/12 00:20:45 INFO Received complete push request method=POST path=/29432026/11/12 00:20:45 INFO Vacuumed table table=closures29442026/11/12 00:20:45 INFO Vacuumed table table=objects29452026/11/12 00:20:45 INFO Completed push id=8 objects=2 duration=2.224009ms29462026/11/12 00:20:45 INFO Received complete push request method=POST path=/29472026/11/12 00:20:45 INFO Completed push id=5 objects=3 duration=1.889149ms29482026/11/12 00:20:45 INFO Received uploads request method=POST path=/29492026/11/12 00:20:45 INFO Received push request method=POST path=/29502026/11/12 00:20:45 INFO Received complete upload request method=POST path=/29512026/11/12 00:20:45 INFO Received uploads request method=POST path=/29522026/11/12 00:20:45 INFO Received complete upload request method=POST path=/29532026/11/12 00:20:45 INFO Received complete upload request method=POST path=/29542026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=029552026/11/12 00:20:45 INFO Completed upload id=6 objects=7 duration=3.237909ms29562026/11/12 00:20:45 INFO Completed upload id=15 objects=6 duration=1.270339ms29572026/11/12 00:20:45 INFO Completed upload id=12 objects=21 duration=1.698739ms29582026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=029592026/11/12 00:20:45 INFO Received uploads request method=POST path=/29602026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period29612026/11/12 00:20:45 INFO Received complete push request method=POST path=/29622026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=029632026/11/12 00:20:45 INFO Received push request method=POST path=/29642026/11/12 00:20:45 INFO Completed push id=2 objects=7 duration=5.070978ms29652026/11/12 00:20:45 INFO Received push request method=POST path=/29662026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029672026/11/12 00:20:45 INFO Vacuumed table table=pending_closures29682026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=029692026/11/12 00:20:45 INFO Vacuumed table table=pending_objects29702026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads29712026/11/12 00:20:45 INFO Vacuumed table table=closures29722026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period29732026/11/12 00:20:45 INFO Vacuumed table table=objects29742026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029752026/11/12 00:20:45 INFO Received uploads request method=POST path=/29762026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=029772026/11/12 00:20:45 INFO Received complete push request method=POST path=/29782026/11/12 00:20:45 INFO Vacuumed table table=pending_closures29792026/11/12 00:20:45 INFO Received push request method=POST path=/29802026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=029812026/11/12 00:20:45 INFO Vacuumed table table=pending_closures29822026/11/12 00:20:45 INFO Completed push id=5 objects=6 duration=1.92797ms29832026/11/12 00:20:45 INFO Vacuumed table table=pending_objects29842026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads29852026/11/12 00:20:45 INFO Vacuumed table table=pending_objects29862026/11/12 00:20:45 INFO Vacuumed table table=closures29872026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads29882026/11/12 00:20:45 INFO Vacuumed table table=objects29892026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029902026/11/12 00:20:45 INFO Vacuumed table table=closures29912026/11/12 00:20:45 INFO Vacuumed table table=objects29922026/11/12 00:20:45 INFO Received push request method=POST path=/29932026/11/12 00:20:45 INFO Vacuumed table table=pending_closures29942026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=029952026/11/12 00:20:45 INFO Vacuumed table table=pending_objects29962026/11/12 00:20:45 INFO Vacuumed table table=pending_closures29972026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads29982026/11/12 00:20:45 INFO Received push request method=POST path=/29992026/11/12 00:20:45 INFO Received complete upload request method=POST path=/30002026/11/12 00:20:45 INFO Vacuumed table table=closures30012026/11/12 00:20:45 INFO Received push request method=POST path=/30022026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30032026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30042026/11/12 00:20:45 INFO Vacuumed table table=objects30052026/11/12 00:20:45 INFO Received uploads request method=POST path=/30062026/11/12 00:20:45 INFO Vacuumed table table=closures30072026/11/12 00:20:45 INFO Completed upload id=11 objects=14 duration=2.187519ms30082026/11/12 00:20:45 INFO Vacuumed table table=objects30092026/11/12 00:20:45 INFO Received push request method=POST path=/30102026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030112026/11/12 00:20:45 INFO Received uploads request method=POST path=/30122026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030132026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period30142026/11/12 00:20:45 INFO Received complete upload request method=POST path=/30152026/11/12 00:20:45 INFO Received push request method=POST path=/30162026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=030172026/11/12 00:20:45 INFO Received complete upload request method=POST path=/30182026/11/12 00:20:45 INFO Received push request method=POST path=/30192026/11/12 00:20:45 INFO Received complete push request method=POST path=/30202026/11/12 00:20:45 INFO Completed upload id=17 objects=7 duration=1.41658ms30212026/11/12 00:20:45 INFO Completed upload id=10 objects=6 duration=3.082299ms30222026/11/12 00:20:45 INFO Received complete push request method=POST path=/30232026/11/12 00:20:45 INFO Vacuumed table table=pending_closures30242026/11/12 00:20:45 INFO Completed push id=13 objects=23 duration=2.452149ms30252026/11/12 00:20:45 INFO Received uploads request method=POST path=/30262026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30272026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30282026/11/12 00:20:45 INFO Completed push id=5 objects=9 duration=5.740318ms30292026/11/12 00:20:45 INFO Vacuumed table table=closures30302026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030312026/11/12 00:20:45 INFO Vacuumed table table=objects30322026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period30332026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030342026/11/12 00:20:45 INFO Vacuumed table table=pending_closures30352026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30362026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30372026/11/12 00:20:45 INFO Received uploads request method=POST path=/30382026/11/12 00:20:45 INFO Vacuumed table table=closures30392026/11/12 00:20:45 INFO Vacuumed table table=objects30402026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030412026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period30422026/11/12 00:20:45 INFO Received push request method=POST path=/30432026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030442026/11/12 00:20:45 INFO Received push request method=POST path=/30452026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030462026/11/12 00:20:45 INFO Vacuumed table table=pending_closures30472026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030482026/11/12 00:20:45 INFO Received push request method=POST path=/30492026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30502026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30512026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period30522026/11/12 00:20:45 INFO Received complete push request method=POST path=/30532026/11/12 00:20:45 INFO Vacuumed table table=closures30542026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030552026/11/12 00:20:45 INFO Vacuumed table table=objects30562026/11/12 00:20:45 INFO Completed push id=2 objects=10 duration=3.500719ms30572026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030582026/11/12 00:20:45 INFO Received uploads request method=POST path=/30592026/11/12 00:20:45 INFO Vacuumed table table=pending_closures30602026/11/12 00:20:45 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30612026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30622026/11/12 00:20:45 INFO Received uploads request method=POST path=/30632026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30642026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30652026/11/12 00:20:45 INFO Vacuumed table table=closures30662026/11/12 00:20:45 INFO Vacuumed table table=objects30672026/11/12 00:20:45 INFO Received push request method=POST path=/30682026/11/12 00:20:45 INFO Received push request method=POST path=/30692026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030702026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030712026/11/12 00:20:45 INFO Received push request method=POST path=/30722026/11/12 00:20:45 INFO Vacuumed table table=pending_closures30732026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30742026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30752026/11/12 00:20:45 INFO Vacuumed table table=closures30762026/11/12 00:20:45 INFO Vacuumed table table=objects30772026/11/12 00:20:45 INFO Received uploads request method=POST path=/30782026/11/12 00:20:45 INFO Received uploads request method=POST path=/30792026/11/12 00:20:45 INFO Received push request method=POST path=/30802026/11/12 00:20:45 INFO Received push request method=POST path=/30812026/11/12 00:20:45 INFO Received uploads request method=POST path=/30822026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030832026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period30842026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=030852026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030862026/11/12 00:20:45 INFO Vacuumed table table=pending_closures30872026/11/12 00:20:45 INFO Received complete push request method=POST path=/30882026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30892026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30902026/11/12 00:20:45 INFO Received push request method=POST path=/30912026/11/12 00:20:45 INFO Vacuumed table table=closures30922026/11/12 00:20:45 INFO Vacuumed table table=objects30932026/11/12 00:20:45 INFO Completed push id=9 objects=7 duration=4.538158ms30942026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030952026/11/12 00:20:45 INFO Vacuumed table table=pending_closures30962026/11/12 00:20:45 INFO Vacuumed table table=pending_objects30972026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads30982026/11/12 00:20:45 INFO Vacuumed table table=closures30992026/11/12 00:20:45 INFO Vacuumed table table=objects31002026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=031012026/11/12 00:20:45 INFO Received push request method=POST path=/31022026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period31032026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31042026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031052026/11/12 00:20:45 INFO Completed upload id=14 objects=6 duration=1.713319ms31062026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=031072026/11/12 00:20:45 INFO Vacuumed table table=pending_closures31082026/11/12 00:20:45 INFO Vacuumed table table=pending_closures31092026/11/12 00:20:45 INFO Vacuumed table table=pending_objects31102026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads31112026/11/12 00:20:45 INFO Vacuumed table table=closures31122026/11/12 00:20:45 INFO Vacuumed table table=pending_objects31132026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads31142026/11/12 00:20:45 INFO Vacuumed table table=objects31152026/11/12 00:20:45 INFO Received push request method=POST path=/31162026/11/12 00:20:45 INFO Vacuumed table table=closures31172026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31182026/11/12 00:20:45 INFO Vacuumed table table=objects31192026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."31202026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."31212026/11/12 00:20:45 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."31222026/11/12 00:20:45 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."31232026/11/12 00:20:45 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."31242026/11/12 00:20:45 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."31252026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31262026/11/12 00:20:45 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."31272026/11/12 00:20:45 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."31282026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31292026/11/12 00:20:45 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."31302026/11/12 00:20:45 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."31312026/11/12 00:20:45 INFO Received complete push request method=POST path=/31322026/11/12 00:20:45 INFO Completed upload id=3 objects=24 duration=2.126049ms31332026/11/12 00:20:45 INFO Received uploads request method=POST path=/31342026/11/12 00:20:45 INFO Completed push id=8 objects=7 duration=1.984639ms31352026/11/12 00:20:45 INFO Received push request method=POST path=/31362026/11/12 00:20:45 INFO Received uploads request method=POST path=/31372026/11/12 00:20:45 INFO Received push request method=POST path=/31382026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31392026/11/12 00:20:45 INFO Received push request method=POST path=/31402026/11/12 00:20:45 INFO Completed upload id=10 objects=7 duration=1.58958ms31412026/11/12 00:20:45 INFO Received uploads request method=POST path=/31422026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=031432026/11/12 00:20:45 INFO Received uploads request method=POST path=/31442026/11/12 00:20:45 INFO Received uploads request method=POST path=/31452026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031462026/11/12 00:20:45 INFO Vacuumed table table=pending_closures31472026/11/12 00:20:45 INFO Received push request method=POST path=/31482026/11/12 00:20:45 INFO Vacuumed table table=pending_objects31492026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31502026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads31512026/11/12 00:20:45 INFO Vacuumed table table=closures31522026/11/12 00:20:45 INFO Received uploads request method=POST path=/31532026/11/12 00:20:45 INFO Vacuumed table table=objects31542026/11/12 00:20:45 INFO Completed upload id=4 objects=14 duration=3.821879ms31552026/11/12 00:20:45 INFO Received push request method=POST path=/31562026/11/12 00:20:45 INFO Received uploads request method=POST path=/31572026/11/12 00:20:45 INFO Received uploads request method=POST path=/31582026/11/12 00:20:45 INFO Received push request method=POST path=/31592026/11/12 00:20:45 INFO Received complete push request method=POST path=/31602026/11/12 00:20:45 INFO Received complete push request method=POST path=/31612026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31622026/11/12 00:20:45 INFO Completed push id=14 objects=3 duration=3.177619ms31632026/11/12 00:20:45 INFO Completed upload id=7 objects=3 duration=1.279199ms31642026/11/12 00:20:45 INFO Received uploads request method=POST path=/31652026/11/12 00:20:45 INFO Completed push id=13 objects=12 duration=3.985549ms31662026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31672026/11/12 00:20:45 INFO Received uploads request method=POST path=/31682026/11/12 00:20:45 INFO Completed upload id=10 objects=23 duration=2.776559ms31692026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31702026/11/12 00:20:45 INFO Received uploads request method=POST path=/31712026/11/12 00:20:45 INFO Completed upload id=11 objects=6 duration=2.382799ms31722026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=031732026/11/12 00:20:45 INFO Received complete upload request method=POST path=/31742026/11/12 00:20:45 INFO Received push request method=POST path=/31752026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period31762026/11/12 00:20:45 INFO Completed upload id=9 objects=16 duration=1.335259ms31772026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031782026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=031792026/11/12 00:20:45 INFO Vacuumed table table=pending_closures31802026/11/12 00:20:45 INFO Vacuumed table table=pending_objects31812026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads31822026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=031832026/11/12 00:20:45 INFO Vacuumed table table=closures31842026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period31852026/11/12 00:20:45 INFO Vacuumed table table=objects31862026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031872026/11/12 00:20:45 INFO Vacuumed table table=pending_closures31882026/11/12 00:20:45 INFO Vacuumed table table=pending_objects31892026/11/12 00:20:45 INFO Received uploads request method=POST path=/31902026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads31912026/11/12 00:20:45 INFO Vacuumed table table=closures31922026/11/12 00:20:45 INFO Received push request method=POST path=/31932026/11/12 00:20:45 INFO Vacuumed table table=objects31942026/11/12 00:20:45 INFO Received uploads request method=POST path=/31952026/11/12 00:20:45 INFO Received uploads request method=POST path=/31962026/11/12 00:20:45 INFO Received complete push request method=POST path=/31972026/11/12 00:20:45 INFO Received push request method=POST path=/31982026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=031992026/11/12 00:20:45 INFO Received push request method=POST path=/32002026/11/12 00:20:45 INFO Completed push id=18 objects=23 duration=2.554209ms32012026/11/12 00:20:45 INFO Received complete push request method=POST path=/32022026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032032026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032042026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32052026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32062026/11/12 00:20:45 INFO Received complete push request method=POST path=/32072026/11/12 00:20:45 INFO Vacuumed table table=pending_objects32082026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=032092026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads32102026/11/12 00:20:45 INFO Completed push id=9 objects=4 duration=3.370819ms32112026/11/12 00:20:45 INFO Vacuumed table table=pending_objects32122026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads32132026/11/12 00:20:45 INFO Vacuumed table table=closures32142026/11/12 00:20:45 INFO Completed push id=5 objects=6 duration=2.272619ms32152026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period32162026/11/12 00:20:45 INFO Received complete push request method=POST path=/32172026/11/12 00:20:45 INFO Vacuumed table table=closures32182026/11/12 00:20:45 INFO Vacuumed table table=objects32192026/11/12 00:20:45 INFO Vacuumed table table=objects32202026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032212026/11/12 00:20:45 INFO Received uploads request method=POST path=/32222026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32232026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=032242026/11/12 00:20:45 INFO Received uploads request method=POST path=/32252026/11/12 00:20:45 INFO Completed push id=7 objects=20 duration=3.054979ms32262026/11/12 00:20:45 INFO Vacuumed table table=pending_objects32272026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads32282026/11/12 00:20:45 INFO Vacuumed table table=closures32292026/11/12 00:20:45 INFO Vacuumed table table=objects32302026/11/12 00:20:45 INFO Received push request method=POST path=/32312026/11/12 00:20:45 INFO Received complete upload request method=POST path=/32322026/11/12 00:20:45 INFO Completed upload id=15 objects=4 duration=2.508929ms32332026/11/12 00:20:45 INFO Received complete upload request method=POST path=/32342026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032352026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32362026/11/12 00:20:45 INFO Completed upload id=18 objects=4 duration=1.82357ms32372026/11/12 00:20:45 INFO Received uploads request method=POST path=/32382026/11/12 00:20:45 INFO Received push request method=POST path=/32392026/11/12 00:20:45 INFO Vacuumed table table=pending_objects32402026/11/12 00:20:45 INFO Received push request method=POST path=/32412026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads32422026/11/12 00:20:45 INFO Vacuumed table table=closures32432026/11/12 00:20:45 INFO Vacuumed table table=objects32442026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=032452026/11/12 00:20:45 INFO Received push request method=POST path=/32462026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period32472026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032482026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32492026/11/12 00:20:45 INFO Vacuumed table table=pending_objects32502026/11/12 00:20:45 INFO Received push request method=POST path=/32512026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads32522026/11/12 00:20:45 INFO Vacuumed table table=closures32532026/11/12 00:20:45 INFO Vacuumed table table=objects32542026/11/12 00:20:45 INFO Received uploads request method=POST path=/32552026/11/12 00:20:45 INFO Received push request method=POST path=/32562026/11/12 00:20:45 INFO Received complete upload request method=POST path=/32572026/11/12 00:20:45 INFO Completed upload id=11 objects=6 duration=1.79038ms32582026/11/12 00:20:45 INFO Received push request method=POST path=/32592026/11/12 00:20:45 INFO Received push request method=POST path=/32602026/11/12 00:20:45 INFO Received push request method=POST path=/32612026/11/12 00:20:45 INFO Received complete push request method=POST path=/32622026/11/12 00:20:45 INFO Completed push id=10 objects=3 duration=1.666489ms32632026/11/12 00:20:45 INFO Received complete push request method=POST path=/32642026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=032652026/11/12 00:20:45 INFO Received complete push request method=POST path=/32662026/11/12 00:20:45 INFO Completed push id=8 objects=6 duration=3.693798ms32672026/11/12 00:20:45 INFO Completed push id=11 objects=9 duration=1.896359ms32682026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032692026/11/12 00:20:45 INFO Received uploads request method=POST path=/32702026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32712026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=032722026/11/12 00:20:45 INFO Vacuumed table table=pending_objects32732026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads32742026/11/12 00:20:45 INFO Vacuumed table table=closures32752026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032762026/11/12 00:20:45 INFO Vacuumed table table=objects32772026/11/12 00:20:45 INFO Received push request method=POST path=/32782026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32792026/11/12 00:20:45 INFO Received complete push request method=POST path=/32802026/11/12 00:20:45 INFO Vacuumed table table=pending_objects32812026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads32822026/11/12 00:20:45 INFO Vacuumed table table=closures32832026/11/12 00:20:45 INFO Completed push id=9 objects=5 duration=2.819549ms32842026/11/12 00:20:45 INFO Vacuumed table table=objects32852026/11/12 00:20:45 INFO Received push request method=POST path=/32862026/11/12 00:20:45 INFO Received uploads request method=POST path=/32872026/11/12 00:20:45 INFO Received complete upload request method=POST path=/32882026/11/12 00:20:45 INFO Received uploads request method=POST path=/32892026/11/12 00:20:45 INFO Completed upload id=10 objects=2 duration=3.267059ms32902026/11/12 00:20:45 INFO Received push request method=POST path=/32912026/11/12 00:20:45 INFO Received complete push request method=POST path=/32922026/11/12 00:20:45 INFO Received uploads request method=POST path=/32932026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=032942026/11/12 00:20:45 INFO Completed push id=11 objects=11 duration=3.958128ms32952026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period32962026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032972026/11/12 00:20:45 INFO Vacuumed table table=pending_closures32982026/11/12 00:20:45 INFO Received push request method=POST path=/32992026/11/12 00:20:45 INFO Vacuumed table table=pending_objects33002026/11/12 00:20:45 INFO Received uploads request method=POST path=/33012026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads33022026/11/12 00:20:45 INFO Vacuumed table table=closures33032026/11/12 00:20:45 INFO Vacuumed table table=objects33042026/11/12 00:20:45 INFO Received complete push request method=POST path=/33052026/11/12 00:20:45 INFO Received complete upload request method=POST path=/33062026/11/12 00:20:45 INFO Received push request method=POST path=/33072026/11/12 00:20:45 INFO Completed upload id=16 objects=3 duration=1.0522ms33082026/11/12 00:20:45 INFO Received push request method=POST path=/33092026/11/12 00:20:45 INFO Received complete push request method=POST path=/33102026/11/12 00:20:45 INFO Completed push id=5 objects=3 duration=3.754178ms33112026/11/12 00:20:45 INFO Completed push id=5 objects=8 duration=3.849388ms33122026/11/12 00:20:45 INFO Received complete push request method=POST path=/33132026/11/12 00:20:45 INFO Completed push id=12 objects=15 duration=2.412429ms33142026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=033152026/11/12 00:20:45 INFO Received complete push request method=POST path=/33162026/11/12 00:20:45 INFO Received push request method=POST path=/33172026/11/12 00:20:45 INFO Completed push id=19 objects=6 duration=3.757169ms33182026/11/12 00:20:45 INFO Received uploads request method=POST path=/33192026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=033202026/11/12 00:20:45 INFO Received push request method=POST path=/33212026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033222026/11/12 00:20:45 INFO Vacuumed table table=pending_closures33232026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=033242026/11/12 00:20:45 INFO Vacuumed table table=pending_objects33252026/11/12 00:20:45 INFO Received push request method=POST path=/33262026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads33272026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033282026/11/12 00:20:45 INFO Received push request method=POST path=/33292026/11/12 00:20:45 INFO Vacuumed table table=closures33302026/11/12 00:20:45 INFO Vacuumed table table=pending_closures33312026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period33322026/11/12 00:20:45 INFO Vacuumed table table=objects33332026/11/12 00:20:45 INFO Vacuumed table table=pending_objects33342026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads33352026/11/12 00:20:45 INFO Received uploads request method=POST path=/33362026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033372026/11/12 00:20:45 INFO Vacuumed table table=closures33382026/11/12 00:20:45 INFO Vacuumed table table=pending_closures33392026/11/12 00:20:45 INFO Received push request method=POST path=/33402026/11/12 00:20:45 INFO Received uploads request method=POST path=/33412026/11/12 00:20:45 INFO Vacuumed table table=objects33422026/11/12 00:20:45 INFO Vacuumed table table=pending_objects33432026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads33442026/11/12 00:20:45 INFO Received complete push request method=POST path=/33452026/11/12 00:20:45 INFO Vacuumed table table=closures33462026/11/12 00:20:45 INFO Vacuumed table table=objects33472026/11/12 00:20:45 INFO Completed push id=4 objects=6 duration=2.530839ms33482026/11/12 00:20:45 INFO Received uploads request method=POST path=/33492026/11/12 00:20:45 INFO Received complete upload request method=POST path=/33502026/11/12 00:20:45 INFO Completed upload id=10 objects=6 duration=2.962248ms33512026/11/12 00:20:45 INFO Received push request method=POST path=/33522026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=033532026/11/12 00:20:45 INFO Received push request method=POST path=/33542026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=033552026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033562026/11/12 00:20:45 INFO Vacuumed table table=pending_closures33572026/11/12 00:20:45 INFO Received push request method=POST path=/33582026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period33592026/11/12 00:20:45 INFO Received uploads request method=POST path=/33602026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=033612026/11/12 00:20:45 INFO Vacuumed table table=pending_objects33622026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads33632026/11/12 00:20:45 INFO Received complete push request method=POST path=/33642026/11/12 00:20:45 INFO Received complete push request method=POST path=/33652026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033662026/11/12 00:20:45 INFO Vacuumed table table=closures33672026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period33682026/11/12 00:20:45 INFO Vacuumed table table=objects33692026/11/12 00:20:45 INFO Vacuumed table table=pending_closures33702026/11/12 00:20:45 INFO Completed push id=1 objects=11 duration=2.412979ms33712026/11/12 00:20:45 INFO Received complete push request method=POST path=/33722026/11/12 00:20:45 INFO Completed push id=17 objects=22 duration=3.001909ms33732026/11/12 00:20:45 INFO Vacuumed table table=pending_objects33742026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads33752026/11/12 00:20:45 INFO Completed push id=9 objects=24 duration=2.132429ms33762026/11/12 00:20:45 INFO Vacuumed table table=closures33772026/11/12 00:20:45 INFO Vacuumed table table=objects33782026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33792026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33802026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33812026/11/12 00:20:45 INFO Received complete push request method=POST path=/33822026/11/12 00:20:45 INFO Completed push id=8 objects=6 duration=1.754359ms33832026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=033842026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period33852026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033862026/11/12 00:20:45 INFO Received complete upload request method=POST path=/33872026/11/12 00:20:45 INFO Vacuumed table table=pending_closures33882026/11/12 00:20:45 INFO Received complete push request method=POST path=/33892026/11/12 00:20:45 INFO Completed upload id=18 objects=4 duration=2.005389ms33902026/11/12 00:20:45 INFO Vacuumed table table=pending_objects33912026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads33922026/11/12 00:20:45 INFO Vacuumed table table=closures33932026/11/12 00:20:45 INFO Received complete upload request method=POST path=/33942026/11/12 00:20:45 INFO Received uploads request method=POST path=/33952026/11/12 00:20:45 INFO Vacuumed table table=objects33962026/11/12 00:20:45 INFO Completed upload id=9 objects=2 duration=1.518309ms33972026/11/12 00:20:45 INFO Received uploads request method=POST path=/33982026/11/12 00:20:45 INFO Completed push id=6 objects=26 duration=4.104429ms33992026/11/12 00:20:45 INFO Received complete upload request method=POST path=/34002026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=034012026/11/12 00:20:45 INFO Completed upload id=16 objects=3 duration=1.38204ms34022026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period34032026/11/12 00:20:45 INFO Received uploads request method=POST path=/34042026/11/12 00:20:45 INFO Received complete push request method=POST path=/34052026/11/12 00:20:45 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34062026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34072026/11/12 00:20:45 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34082026/11/12 00:20:45 INFO Received push request method=POST path=/34092026/11/12 00:20:45 INFO Completed push id=11 objects=8 duration=3.539379ms34102026/11/12 00:20:45 INFO Received uploads request method=POST path=/34112026/11/12 00:20:45 INFO Received complete push request method=POST path=/34122026/11/12 00:20:45 INFO Completed push id=8 objects=3 duration=1.86342ms34132026/11/12 00:20:45 INFO Received push request method=POST path=/34142026/11/12 00:20:45 INFO Received uploads request method=POST path=/34152026/11/12 00:20:45 INFO Received push request method=POST path=/34162026/11/12 00:20:45 INFO Received uploads request method=POST path=/34172026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=034182026/11/12 00:20:45 INFO Received complete push request method=POST path=/34192026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034202026/11/12 00:20:45 INFO Vacuumed table table=pending_closures34212026/11/12 00:20:45 INFO Completed push id=8 objects=6 duration=2.486269ms34222026/11/12 00:20:45 INFO Vacuumed table table=pending_objects34232026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads34242026/11/12 00:20:45 INFO Vacuumed table table=closures34252026/11/12 00:20:45 INFO Received push request method=POST path=/34262026/11/12 00:20:45 INFO Vacuumed table table=objects34272026/11/12 00:20:45 INFO Received complete push request method=POST path=/34282026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=034292026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=034302026/11/12 00:20:45 INFO Completed push id=17 objects=6 duration=2.038199ms34312026/11/12 00:20:45 INFO Received uploads request method=POST path=/34322026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period34332026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period34342026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034352026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034362026/11/12 00:20:45 INFO Received uploads request method=POST path=/34372026/11/12 00:20:45 INFO Vacuumed table table=pending_closures34382026/11/12 00:20:45 INFO Vacuumed table table=pending_closures34392026/11/12 00:20:45 INFO Received push request method=POST path=/34402026/11/12 00:20:45 INFO Vacuumed table table=pending_objects34412026/11/12 00:20:45 INFO Received push request method=POST path=/34422026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads34432026/11/12 00:20:45 INFO Received complete upload request method=POST path=/34442026/11/12 00:20:45 INFO Vacuumed table table=pending_objects34452026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads34462026/11/12 00:20:45 INFO Vacuumed table table=closures34472026/11/12 00:20:45 INFO Vacuumed table table=closures34482026/11/12 00:20:45 INFO Received push request method=POST path=/34492026/11/12 00:20:45 INFO Completed upload id=10 objects=10 duration=1.47767ms34502026/11/12 00:20:45 INFO Vacuumed table table=objects34512026/11/12 00:20:45 INFO Vacuumed table table=objects34522026/11/12 00:20:45 INFO Received uploads request method=POST path=/34532026/11/12 00:20:45 INFO Received complete upload request method=POST path=/34542026/11/12 00:20:45 INFO Received complete push request method=POST path=/34552026/11/12 00:20:45 INFO Received push request method=POST path=/34562026/11/12 00:20:45 INFO Completed upload id=7 objects=10 duration=2.460849ms34572026/11/12 00:20:45 INFO Completed push id=12 objects=7 duration=2.776849ms34582026/11/12 00:20:45 INFO Received push request method=POST path=/34592026/11/12 00:20:45 INFO Received uploads request method=POST path=/34602026/11/12 00:20:45 INFO Received push request method=POST path=/34612026/11/12 00:20:45 INFO Received complete push request method=POST path=/34622026/11/12 00:20:45 INFO Received complete upload request method=POST path=/34632026/11/12 00:20:45 INFO Received complete upload request method=POST path=/34642026/11/12 00:20:45 INFO Received uploads request method=POST path=/34652026/11/12 00:20:45 INFO Completed upload id=8 objects=3 duration=2.748549ms34662026/11/12 00:20:45 INFO Completed push id=13 objects=6 duration=3.197289ms34672026/11/12 00:20:45 INFO Received complete upload request method=POST path=/34682026/11/12 00:20:45 INFO Completed upload id=23 objects=15 duration=2.964639ms34692026/11/12 00:20:45 INFO Received complete push request method=POST path=/34702026/11/12 00:20:45 INFO Completed upload id=13 objects=10 duration=1.57923ms34712026/11/12 00:20:45 INFO Received push request method=POST path=/34722026/11/12 00:20:45 INFO Completed push id=15 objects=7 duration=2.570219ms34732026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=034742026/11/12 00:20:45 INFO Received push request method=POST path=/34752026/11/12 00:20:45 INFO Received push request method=POST path=/34762026/11/12 00:20:45 INFO Received complete push request method=POST path=/34772026/11/12 00:20:45 INFO Received uploads request method=POST path=/34782026/11/12 00:20:45 INFO Completed push id=15 objects=24 duration=3.840828ms34792026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=034802026/11/12 00:20:45 INFO Received uploads request method=POST path=/34812026/11/12 00:20:45 INFO Received uploads request method=POST path=/34822026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period34832026/11/12 00:20:45 INFO Received push request method=POST path=/34842026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=034852026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=034862026/11/12 00:20:45 INFO Vacuumed table table=pending_closures34872026/11/12 00:20:45 INFO Received uploads request method=POST path=/34882026/11/12 00:20:45 INFO Vacuumed table table=pending_objects34892026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads34902026/11/12 00:20:45 INFO Vacuumed table table=closures34912026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034922026/11/12 00:20:45 INFO Vacuumed table table=objects34932026/11/12 00:20:45 INFO Vacuumed table table=pending_closures34942026/11/12 00:20:45 INFO Vacuumed table table=pending_objects34952026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads34962026/11/12 00:20:45 INFO Vacuumed table table=closures34972026/11/12 00:20:45 INFO Received uploads request method=POST path=/34982026/11/12 00:20:45 INFO Vacuumed table table=objects34992026/11/12 00:20:45 INFO Received uploads request method=POST path=/35002026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35012026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=035022026/11/12 00:20:45 INFO Completed upload id=10 objects=17 duration=1.01292ms35032026/11/12 00:20:45 INFO Received push request method=POST path=/35042026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period35052026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35062026/11/12 00:20:45 INFO Completed upload id=19 objects=6 duration=1.493289ms35072026/11/12 00:20:45 INFO Received uploads request method=POST path=/35082026/11/12 00:20:45 INFO Received push request method=POST path=/35092026/11/12 00:20:45 INFO Received complete push request method=POST path=/35102026/11/12 00:20:45 INFO Received complete push request method=POST path=/35112026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35122026/11/12 00:20:45 INFO Received complete push request method=POST path=/35132026/11/12 00:20:45 INFO Completed push id=4 objects=6 duration=2.170659ms35142026/11/12 00:20:45 INFO Completed upload id=15 objects=6 duration=1.15089ms35152026/11/12 00:20:45 INFO Completed push id=3 objects=21 duration=4.550388ms35162026/11/12 00:20:45 INFO Completed push id=6 objects=6 duration=2.239599ms35172026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35182026/11/12 00:20:45 INFO Received complete push request method=POST path=/35192026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35202026/11/12 00:20:45 INFO Completed upload id=22 objects=7 duration=1.21171ms35212026/11/12 00:20:45 INFO Completed push id=17 objects=6 duration=2.866938ms35222026/11/12 00:20:45 INFO Completed upload id=17 objects=6 duration=3.510948ms35232026/11/12 00:20:45 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35242026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=035252026/11/12 00:20:45 INFO Vacuumed table table=pending_closures35262026/11/12 00:20:45 INFO Received push request method=POST path=/35272026/11/12 00:20:45 INFO Vacuumed table table=pending_objects35282026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads35292026/11/12 00:20:45 INFO Vacuumed table table=closures35302026/11/12 00:20:45 INFO Received uploads request method=POST path=/35312026/11/12 00:20:45 INFO Vacuumed table table=objects35322026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35332026/11/12 00:20:45 INFO Completed upload id=11 objects=7 duration=1.59506ms35342026/11/12 00:20:45 INFO Received push request method=POST path=/35352026/11/12 00:20:45 INFO Received uploads request method=POST path=/35362026/11/12 00:20:45 INFO Received uploads request method=POST path=/35372026/11/12 00:20:45 INFO Received push request method=POST path=/35382026/11/12 00:20:45 INFO Received uploads request method=POST path=/35392026/11/12 00:20:45 INFO Received push request method=POST path=/35402026/11/12 00:20:45 INFO Received push request method=POST path=/35412026/11/12 00:20:45 INFO Received uploads request method=POST path=/35422026/11/12 00:20:45 INFO Received uploads request method=POST path=/35432026/11/12 00:20:45 INFO Received push request method=POST path=/35442026/11/12 00:20:45 INFO Received uploads request method=POST path=/35452026/11/12 00:20:45 INFO Received complete push request method=POST path=/35462026/11/12 00:20:45 INFO Received push request method=POST path=/35472026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35482026/11/12 00:20:45 INFO Received push request method=POST path=/35492026/11/12 00:20:45 INFO Completed push id=4 objects=7 duration=2.197199ms35502026/11/12 00:20:45 INFO Completed upload id=22 objects=2 duration=1.62969ms35512026/11/12 00:20:45 INFO Received push request method=POST path=/35522026/11/12 00:20:45 INFO Received push request method=POST path=/35532026/11/12 00:20:45 INFO Received complete push request method=POST path=/35542026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35552026/11/12 00:20:45 INFO Received uploads request method=POST path=/35562026/11/12 00:20:45 INFO Completed push id=14 objects=6 duration=2.025279ms35572026/11/12 00:20:45 INFO Completed upload id=6 objects=6 duration=1.573119ms35582026/11/12 00:20:45 INFO Received uploads request method=POST path=/35592026/11/12 00:20:45 INFO Received uploads request method=POST path=/35602026/11/12 00:20:45 INFO Received complete push request method=POST path=/35612026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=035622026/11/12 00:20:45 INFO Completed push id=14 objects=3 duration=1.497479ms35632026/11/12 00:20:45 INFO Received uploads request method=POST path=/35642026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=035652026/11/12 00:20:45 INFO Received complete push request method=POST path=/35662026/11/12 00:20:45 INFO Completed push id=24 objects=9 duration=2.775289ms35672026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period35682026/11/12 00:20:45 INFO Received complete upload request method=POST path=/35692026/11/12 00:20:45 INFO Received complete push request method=POST path=/35702026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035712026/11/12 00:20:45 INFO Vacuumed table table=pending_closures35722026/11/12 00:20:45 INFO Received push request method=POST path=/35732026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035742026/11/12 00:20:45 INFO Completed upload id=19 objects=25 duration=2.816058ms35752026/11/12 00:20:45 INFO Completed push id=10 objects=15 duration=2.281859ms35762026/11/12 00:20:45 INFO Vacuumed table table=pending_objects35772026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads35782026/11/12 00:20:45 INFO Vacuumed table table=pending_closures35792026/11/12 00:20:45 INFO Received uploads request method=POST path=/35802026/11/12 00:20:45 INFO Received complete push request method=POST path=/35812026/11/12 00:20:45 INFO Vacuumed table table=closures35822026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=035832026/11/12 00:20:45 INFO Vacuumed table table=pending_objects35842026/11/12 00:20:45 INFO Vacuumed table table=objects35852026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads35862026/11/12 00:20:45 INFO Received uploads request method=POST path=/35872026/11/12 00:20:45 INFO Vacuumed table table=closures35882026/11/12 00:20:45 INFO Completed push id=14 objects=4 duration=3.754568ms35892026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period35902026/11/12 00:20:45 INFO Vacuumed table table=objects35912026/11/12 00:20:45 INFO Received push request method=POST path=/35922026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035932026/11/12 00:20:45 INFO Received push request method=POST path=/35942026/11/12 00:20:45 INFO Vacuumed table table=pending_closures35952026/11/12 00:20:45 INFO Vacuumed table table=pending_objects35962026/11/12 00:20:45 INFO Received complete push request method=POST path=/35972026/11/12 00:20:45 INFO Received uploads request method=POST path=/35982026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads35992026/11/12 00:20:45 INFO Vacuumed table table=closures36002026/11/12 00:20:45 INFO Completed push id=16 objects=27 duration=2.338529ms36012026/11/12 00:20:45 INFO Vacuumed table table=objects36022026/11/12 00:20:45 INFO Received push request method=POST path=/36032026/11/12 00:20:45 INFO Received uploads request method=POST path=/36042026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=036052026/11/12 00:20:45 INFO Received push request method=POST path=/36062026/11/12 00:20:45 WARN Force mode enabled - objects will be deleted immediately without grace period36072026/11/12 00:20:45 INFO Received push request method=POST path=/36082026/11/12 00:20:45 INFO Received uploads request method=POST path=/36092026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036102026/11/12 00:20:45 INFO Vacuumed table table=pending_closures36112026/11/12 00:20:45 INFO Vacuumed table table=pending_objects36122026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads36132026/11/12 00:20:45 INFO Vacuumed table table=closures36142026/11/12 00:20:45 INFO Vacuumed table table=objects36152026/11/12 00:20:45 INFO Received uploads request method=POST path=/36162026/11/12 00:20:45 INFO Received push request method=POST path=/36172026/11/12 00:20:45 INFO Received push request method=POST path=/36182026/11/12 00:20:45 INFO Received push request method=POST path=/36192026/11/12 00:20:45 INFO Received push request method=POST path=/36202026/11/12 00:20:45 INFO Received uploads request method=POST path=/36212026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=036222026/11/12 00:20:45 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=036232026/11/12 00:20:45 INFO Vacuumed table table=pending_closures36242026/11/12 00:20:45 INFO Received push request method=POST path=/36252026/11/12 00:20:45 INFO Vacuumed table table=pending_objects36262026/11/12 00:20:45 INFO Vacuumed table table=multipart_uploads36272026/11/12 00:20:45 INFO Aborted multipart uploads count=0 kept=036282026/11/12 00:20:45 INFO Vacuumed table table=closures36292026/11/12 00:20:45 INFO Vacuumed table table=objects36302026/11/12 00:20:46 INFO Received push request method=POST path=/36312026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period36322026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036332026/11/12 00:20:46 INFO Vacuumed table table=pending_closures36342026/11/12 00:20:46 INFO Received complete push request method=POST path=/36352026/11/12 00:20:46 INFO Vacuumed table table=pending_objects36362026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads36372026/11/12 00:20:46 INFO Received push request method=POST path=/36382026/11/12 00:20:46 INFO Completed push id=15 objects=5 duration=2.202179ms36392026/11/12 00:20:46 INFO Received uploads request method=POST path=/36402026/11/12 00:20:46 INFO Vacuumed table table=closures36412026/11/12 00:20:46 INFO Received push request method=POST path=/36422026/11/12 00:20:46 INFO Vacuumed table table=objects36432026/11/12 00:20:46 INFO Received uploads request method=POST path=/36442026/11/12 00:20:46 INFO Received uploads request method=POST path=/36452026/11/12 00:20:46 INFO Received push request method=POST path=/36462026/11/12 00:20:46 INFO Received complete push request method=POST path=/36472026/11/12 00:20:46 INFO Completed push id=7 objects=7 duration=1.958319ms36482026/11/12 00:20:46 INFO Received push request method=POST path=/36492026/11/12 00:20:46 INFO Received push request method=POST path=/36502026/11/12 00:20:46 INFO Received push request method=POST path=/36512026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=036522026/11/12 00:20:46 INFO Received uploads request method=POST path=/36532026/11/12 00:20:46 INFO Received push request method=POST path=/36542026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=036552026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period36562026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period36572026/11/12 00:20:46 INFO Received uploads request method=POST path=/36582026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036592026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=036602026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036612026/11/12 00:20:46 INFO Vacuumed table table=pending_closures36622026/11/12 00:20:46 INFO Received uploads request method=POST path=/36632026/11/12 00:20:46 INFO Vacuumed table table=pending_closures36642026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period36652026/11/12 00:20:46 INFO Vacuumed table table=pending_objects36662026/11/12 00:20:46 INFO Vacuumed table table=pending_objects36672026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads36682026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads36692026/11/12 00:20:46 INFO Received push request method=POST path=/36702026/11/12 00:20:46 INFO Vacuumed table table=closures36712026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036722026/11/12 00:20:46 INFO Vacuumed table table=closures36732026/11/12 00:20:46 INFO Received complete push request method=POST path=/36742026/11/12 00:20:46 INFO Vacuumed table table=objects36752026/11/12 00:20:46 INFO Vacuumed table table=objects36762026/11/12 00:20:46 INFO Vacuumed table table=pending_closures36772026/11/12 00:20:46 INFO Completed push id=18 objects=10 duration=2.10276ms36782026/11/12 00:20:46 INFO Vacuumed table table=pending_objects36792026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads36802026/11/12 00:20:46 INFO Vacuumed table table=closures36812026/11/12 00:20:46 INFO Vacuumed table table=objects36822026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=036832026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=036842026/11/12 00:20:46 INFO Received uploads request method=POST path=/36852026/11/12 00:20:46 INFO Received push request method=POST path=/36862026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=036872026/11/12 00:20:46 INFO Received uploads request method=POST path=/36882026/11/12 00:20:46 INFO Received complete push request method=POST path=/36892026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period36902026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036912026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period36922026/11/12 00:20:46 INFO Vacuumed table table=pending_closures36932026/11/12 00:20:46 INFO Completed push id=13 objects=15 duration=2.509219ms36942026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036952026/11/12 00:20:46 INFO Vacuumed table table=pending_objects36962026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads36972026/11/12 00:20:46 INFO Vacuumed table table=pending_closures36982026/11/12 00:20:46 INFO Vacuumed table table=closures36992026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37002026/11/12 00:20:46 INFO Vacuumed table table=objects37012026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37022026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037032026/11/12 00:20:46 INFO Vacuumed table table=closures37042026/11/12 00:20:46 INFO Vacuumed table table=objects37052026/11/12 00:20:46 INFO Vacuumed table table=pending_closures37062026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37072026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37082026/11/12 00:20:46 INFO Vacuumed table table=closures37092026/11/12 00:20:46 INFO Vacuumed table table=objects37102026/11/12 00:20:46 INFO Received push request method=POST path=/37112026/11/12 00:20:46 INFO Received uploads request method=POST path=/37122026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=037132026/11/12 00:20:46 INFO Received push request method=POST path=/37142026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=037152026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period37162026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037172026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=037182026/11/12 00:20:46 INFO Vacuumed table table=pending_closures37192026/11/12 00:20:46 INFO Received complete upload request method=POST path=/37202026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037212026/11/12 00:20:46 INFO Received complete upload request method=POST path=/37222026/11/12 00:20:46 INFO Completed upload id=15 objects=9 duration=1.424519ms37232026/11/12 00:20:46 INFO Vacuumed table table=pending_closures37242026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37252026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37262026/11/12 00:20:46 INFO Completed upload id=17 objects=25 duration=3.015869ms37272026/11/12 00:20:46 INFO Received complete push request method=POST path=/37282026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37292026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37302026/11/12 00:20:46 INFO Vacuumed table table=closures37312026/11/12 00:20:46 INFO Received complete upload request method=POST path=/37322026/11/12 00:20:46 INFO Received push request method=POST path=/37332026/11/12 00:20:46 INFO Vacuumed table table=closures37342026/11/12 00:20:46 INFO Vacuumed table table=objects37352026/11/12 00:20:46 INFO Received push request method=POST path=/37362026/11/12 00:20:46 INFO Completed upload id=18 objects=11 duration=1.69949ms37372026/11/12 00:20:46 INFO Vacuumed table table=objects37382026/11/12 00:20:46 INFO Received push request method=POST path=/37392026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."37402026/11/12 00:20:46 INFO Completed push id=19 objects=8 duration=5.003938ms37412026/11/12 00:20:46 INFO Received uploads request method=POST path=/37422026/11/12 00:20:46 INFO Received uploads request method=POST path=/37432026/11/12 00:20:46 INFO Received uploads request method=POST path=/37442026/11/12 00:20:46 INFO Received push request method=POST path=/37452026/11/12 00:20:46 INFO Received push request method=POST path=/37462026/11/12 00:20:46 INFO Received uploads request method=POST path=/37472026/11/12 00:20:46 INFO Received uploads request method=POST path=/37482026/11/12 00:20:46 INFO Received uploads request method=POST path=/37492026/11/12 00:20:46 INFO Received uploads request method=POST path=/37502026/11/12 00:20:46 INFO Received uploads request method=POST path=/37512026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=037522026/11/12 00:20:46 INFO Received complete push request method=POST path=/37532026/11/12 00:20:46 INFO Received push request method=POST path=/37542026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period37552026/11/12 00:20:46 INFO Completed push id=12 objects=2 duration=3.369869ms37562026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037572026/11/12 00:20:46 INFO Received push request method=POST path=/37582026/11/12 00:20:46 INFO Vacuumed table table=pending_closures37592026/11/12 00:20:46 INFO Received push request method=POST path=/37602026/11/12 00:20:46 INFO Received complete upload request method=POST path=/37612026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37622026/11/12 00:20:46 INFO Received complete push request method=POST path=/37632026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37642026/11/12 00:20:46 INFO Completed upload id=21 objects=2 duration=1.529239ms37652026/11/12 00:20:46 INFO Received complete push request method=POST path=/37662026/11/12 00:20:46 INFO Completed push id=21 objects=3 duration=1.76044ms37672026/11/12 00:20:46 INFO Received complete push request method=POST path=/37682026/11/12 00:20:46 INFO Received uploads request method=POST path=/37692026/11/12 00:20:46 INFO Vacuumed table table=closures37702026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=037712026/11/12 00:20:46 INFO Received push request method=POST path=/37722026/11/12 00:20:46 INFO Completed push id=20 objects=3 duration=2.704698ms37732026/11/12 00:20:46 INFO Completed push id=21 objects=17 duration=2.903738ms37742026/11/12 00:20:46 INFO Vacuumed table table=objects37752026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=037762026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037772026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period37782026/11/12 00:20:46 INFO Vacuumed table table=pending_closures37792026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=037802026/11/12 00:20:46 INFO Received uploads request method=POST path=/37812026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37822026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037832026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37842026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period37852026/11/12 00:20:46 INFO Received complete push request method=POST path=/37862026/11/12 00:20:46 INFO Vacuumed table table=closures37872026/11/12 00:20:46 INFO Vacuumed table table=pending_closures37882026/11/12 00:20:46 INFO Received uploads request method=POST path=/37892026/11/12 00:20:46 INFO Vacuumed table table=objects37902026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037912026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37922026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37932026/11/12 00:20:46 INFO Vacuumed table table=pending_closures37942026/11/12 00:20:46 INFO Completed push id=16 objects=16 duration=4.235598ms37952026/11/12 00:20:46 INFO Vacuumed table table=closures37962026/11/12 00:20:46 INFO Received push request method=POST path=/37972026/11/12 00:20:46 INFO Vacuumed table table=pending_objects37982026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads37992026/11/12 00:20:46 INFO Vacuumed table table=objects38002026/11/12 00:20:46 INFO Vacuumed table table=closures38012026/11/12 00:20:46 INFO Vacuumed table table=objects38022026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038032026/11/12 00:20:46 INFO Received uploads request method=POST path=/38042026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period38052026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038062026/11/12 00:20:46 INFO Vacuumed table table=pending_closures38072026/11/12 00:20:46 INFO Vacuumed table table=pending_objects38082026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038092026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads38102026/11/12 00:20:46 INFO Vacuumed table table=closures38112026/11/12 00:20:46 INFO Received complete push request method=POST path=/38122026/11/12 00:20:46 INFO Received complete push request method=POST path=/38132026/11/12 00:20:46 INFO Vacuumed table table=objects38142026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period38152026/11/12 00:20:46 INFO Completed push id=19 objects=7 duration=2.013229ms38162026/11/12 00:20:46 INFO Completed push id=26 objects=14 duration=3.037588ms38172026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038182026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period38192026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038202026/11/12 00:20:46 INFO Received uploads request method=POST path=/38212026/11/12 00:20:46 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."38222026/11/12 00:20:46 INFO Vacuumed table table=pending_closures38232026/11/12 00:20:46 INFO Vacuumed table table=pending_objects38242026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038252026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads38262026/11/12 00:20:46 INFO Vacuumed table table=closures38272026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038282026/11/12 00:20:46 INFO Vacuumed table table=objects38292026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period38302026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period38312026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038322026/11/12 00:20:46 INFO Vacuumed table table=pending_closures38332026/11/12 00:20:46 INFO Vacuumed table table=pending_objects38342026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads38352026/11/12 00:20:46 INFO Vacuumed table table=closures38362026/11/12 00:20:46 INFO Vacuumed table table=objects38372026/11/12 00:20:46 INFO Received push request method=POST path=/38382026/11/12 00:20:46 INFO Received push request method=POST path=/38392026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038402026/11/12 00:20:46 INFO Received uploads request method=POST path=/38412026/11/12 00:20:46 INFO Received push request method=POST path=/38422026/11/12 00:20:46 INFO Received uploads request method=POST path=/38432026/11/12 00:20:46 INFO Received uploads request method=POST path=/38442026/11/12 00:20:46 INFO Received push request method=POST path=/38452026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038462026/11/12 00:20:46 INFO Received push request method=POST path=/38472026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038482026/11/12 00:20:46 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."38492026/11/12 00:20:46 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."38502026/11/12 00:20:46 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."38512026/11/12 00:20:46 INFO Received uploads request method=POST path=/38522026/11/12 00:20:46 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."38532026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."38542026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."38552026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38562026/11/12 00:20:46 INFO Vacuumed table table=pending_closures38572026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."38582026/11/12 00:20:46 INFO Received uploads request method=POST path=/38592026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038602026/11/12 00:20:46 INFO Vacuumed table table=pending_objects38612026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads38622026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038632026/11/12 00:20:46 INFO Vacuumed table table=closures38642026/11/12 00:20:46 INFO Received uploads request method=POST path=/38652026/11/12 00:20:46 INFO Vacuumed table table=objects38662026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038672026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period38682026/11/12 00:20:46 INFO Received complete push request method=POST path=/38692026/11/12 00:20:46 INFO Received complete upload request method=POST path=/38702026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038712026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period38722026/11/12 00:20:46 INFO Received push request method=POST path=/38732026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038742026/11/12 00:20:46 INFO Vacuumed table table=pending_closures38752026/11/12 00:20:46 INFO Completed push id=16 objects=4 duration=2.932689ms38762026/11/12 00:20:46 INFO Completed upload id=14 objects=8 duration=2.579609ms38772026/11/12 00:20:46 INFO Vacuumed table table=pending_closures38782026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038792026/11/12 00:20:46 INFO Vacuumed table table=pending_objects38802026/11/12 00:20:46 INFO Vacuumed table table=pending_objects38812026/11/12 00:20:46 INFO Received uploads request method=POST path=/38822026/11/12 00:20:46 INFO Vacuumed table table=pending_closures38832026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads38842026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads38852026/11/12 00:20:46 INFO Vacuumed table table=closures38862026/11/12 00:20:46 INFO Vacuumed table table=pending_objects38872026/11/12 00:20:46 INFO Vacuumed table table=closures38882026/11/12 00:20:46 INFO Vacuumed table table=objects38892026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads38902026/11/12 00:20:46 INFO Vacuumed table table=closures38912026/11/12 00:20:46 INFO Vacuumed table table=objects38922026/11/12 00:20:46 INFO Vacuumed table table=objects38932026/11/12 00:20:46 INFO Received push request method=POST path=/38942026/11/12 00:20:46 INFO Received push request method=POST path=/38952026/11/12 00:20:46 INFO Received uploads request method=POST path=/38962026/11/12 00:20:46 INFO Received uploads request method=POST path=/38972026/11/12 00:20:46 INFO Received uploads request method=POST path=/38982026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=038992026/11/12 00:20:46 INFO Received complete push request method=POST path=/39002026/11/12 00:20:46 INFO Received push request method=POST path=/39012026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039022026/11/12 00:20:46 INFO Completed push id=7 objects=24 duration=3.868708ms39032026/11/12 00:20:46 INFO Vacuumed table table=pending_closures39042026/11/12 00:20:46 INFO Received uploads request method=POST path=/39052026/11/12 00:20:46 INFO Vacuumed table table=pending_objects39062026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads39072026/11/12 00:20:46 INFO Received push request method=POST path=/39082026/11/12 00:20:46 INFO Vacuumed table table=closures39092026/11/12 00:20:46 INFO Vacuumed table table=objects39102026/11/12 00:20:46 INFO Received push request method=POST path=/39112026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=039122026/11/12 00:20:46 INFO Received uploads request method=POST path=/39132026/11/12 00:20:46 INFO Received uploads request method=POST path=/39142026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period39152026/11/12 00:20:46 INFO Received uploads request method=POST path=/39162026/11/12 00:20:46 INFO Received push request method=POST path=/39172026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039182026/11/12 00:20:46 INFO Received uploads request method=POST path=/39192026/11/12 00:20:46 INFO Vacuumed table table=pending_closures39202026/11/12 00:20:46 INFO Vacuumed table table=pending_objects39212026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads39222026/11/12 00:20:46 INFO Received complete push request method=POST path=/39232026/11/12 00:20:46 INFO Vacuumed table table=closures39242026/11/12 00:20:46 INFO Completed push id=15 objects=4 duration=1.383879ms39252026/11/12 00:20:46 INFO Received push request method=POST path=/39262026/11/12 00:20:46 INFO Vacuumed table table=objects39272026/11/12 00:20:46 INFO Received uploads request method=POST path=/39282026/11/12 00:20:46 INFO Received complete upload request method=POST path=/39292026/11/12 00:20:46 INFO Received complete upload request method=POST path=/39302026/11/12 00:20:46 INFO Received push request method=POST path=/39312026/11/12 00:20:46 INFO Completed upload id=23 objects=16 duration=2.758179ms39322026/11/12 00:20:46 INFO Completed upload id=20 objects=5 duration=1.23163ms39332026/11/12 00:20:46 INFO Received uploads request method=POST path=/39342026/11/12 00:20:46 INFO Received uploads request method=POST path=/39352026/11/12 00:20:46 INFO Received push request method=POST path=/39362026/11/12 00:20:46 INFO Received push request method=POST path=/39372026/11/12 00:20:46 INFO Received push request method=POST path=/39382026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=039392026/11/12 00:20:46 INFO Received uploads request method=POST path=/39402026/11/12 00:20:46 INFO Received uploads request method=POST path=/39412026/11/12 00:20:46 INFO Received complete push request method=POST path=/39422026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039432026/11/12 00:20:46 INFO Received uploads request method=POST path=/39442026/11/12 00:20:46 INFO Received complete upload request method=POST path=/39452026/11/12 00:20:46 INFO Received push request method=POST path=/39462026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=039472026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period39482026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039492026/11/12 00:20:46 INFO Completed upload id=22 objects=6 duration=9.414917ms39502026/11/12 00:20:46 INFO Completed push id=30 objects=4 duration=12.612046ms39512026/11/12 00:20:46 INFO Vacuumed table table=pending_closures39522026/11/12 00:20:46 INFO Vacuumed table table=pending_closures39532026/11/12 00:20:46 INFO Received uploads request method=POST path=/39542026/11/12 00:20:46 INFO Received uploads request method=POST path=/39552026/11/12 00:20:46 INFO Received uploads request method=POST path=/39562026/11/12 00:20:46 INFO Received uploads request method=POST path=/39572026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=039582026/11/12 00:20:46 INFO Vacuumed table table=pending_objects39592026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads39602026/11/12 00:20:46 INFO Vacuumed table table=pending_objects39612026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039622026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads39632026/11/12 00:20:46 INFO Vacuumed table table=closures39642026/11/12 00:20:46 INFO Vacuumed table table=pending_closures39652026/11/12 00:20:46 INFO Vacuumed table table=closures39662026/11/12 00:20:46 INFO Vacuumed table table=objects39672026/11/12 00:20:46 INFO Vacuumed table table=pending_objects39682026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads39692026/11/12 00:20:46 INFO Vacuumed table table=objects39702026/11/12 00:20:46 INFO Vacuumed table table=closures39712026/11/12 00:20:46 INFO Received complete upload request method=POST path=/39722026/11/12 00:20:46 INFO Vacuumed table table=objects39732026/11/12 00:20:46 INFO Completed upload id=12 objects=26 duration=3.077359ms39742026/11/12 00:20:46 INFO Received uploads request method=POST path=/39752026/11/12 00:20:46 INFO Received uploads request method=POST path=/39762026/11/12 00:20:46 INFO Received uploads request method=POST path=/39772026/11/12 00:20:46 INFO Received push request method=POST path=/39782026/11/12 00:20:46 INFO Received push request method=POST path=/39792026/11/12 00:20:46 INFO Received complete push request method=POST path=/39802026/11/12 00:20:46 INFO Received push request method=POST path=/39812026/11/12 00:20:46 INFO Received uploads request method=POST path=/39822026/11/12 00:20:46 INFO Completed push id=14 objects=3 duration=2.040179ms39832026/11/12 00:20:46 INFO Received uploads request method=POST path=/39842026/11/12 00:20:46 INFO Received complete push request method=POST path=/39852026/11/12 00:20:46 INFO Completed push id=22 objects=3 duration=2.089149ms39862026/11/12 00:20:46 INFO Received uploads request method=POST path=/39872026/11/12 00:20:46 INFO Received push request method=POST path=/39882026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=039892026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period39902026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=039912026/11/12 00:20:46 INFO Received uploads request method=POST path=/39922026/11/12 00:20:46 INFO Received push request method=POST path=/39932026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=039942026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039952026/11/12 00:20:46 INFO Vacuumed table table=pending_closures39962026/11/12 00:20:46 INFO Vacuumed table table=pending_closures39972026/11/12 00:20:46 INFO Received complete push request method=POST path=/39982026/11/12 00:20:46 INFO Received push request method=POST path=/39992026/11/12 00:20:46 INFO Vacuumed table table=pending_objects40002026/11/12 00:20:46 INFO Received push request method=POST path=/40012026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads40022026/11/12 00:20:46 INFO Vacuumed table table=closures40032026/11/12 00:20:46 INFO Vacuumed table table=pending_objects40042026/11/12 00:20:46 INFO Completed push id=18 objects=6 duration=3.098159ms40052026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads40062026/11/12 00:20:46 INFO Vacuumed table table=objects40072026/11/12 00:20:46 INFO Received complete push request method=POST path=/40082026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=040092026/11/12 00:20:46 INFO Vacuumed table table=closures40102026/11/12 00:20:46 INFO Received complete push request method=POST path=/40112026/11/12 00:20:46 INFO Completed push id=14 objects=3 duration=1.410819ms40122026/11/12 00:20:46 INFO Vacuumed table table=objects40132026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period40142026/11/12 00:20:46 INFO Completed push id=21 objects=9 duration=1.943689ms40152026/11/12 00:20:46 INFO Received uploads request method=POST path=/40162026/11/12 00:20:46 INFO Received uploads request method=POST path=/40172026/11/12 00:20:46 INFO Received push request method=POST path=/40182026/11/12 00:20:46 INFO Received push request method=POST path=/40192026/11/12 00:20:46 INFO Received complete push request method=POST path=/40202026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40212026/11/12 00:20:46 INFO Received uploads request method=POST path=/40222026/11/12 00:20:46 INFO Completed upload id=23 objects=9 duration=796.31µs40232026/11/12 00:20:46 INFO Completed push id=32 objects=7 duration=1.51186ms40242026/11/12 00:20:46 INFO Received complete push request method=POST path=/40252026/11/12 00:20:46 INFO Completed push id=16 objects=8 duration=1.82436ms40262026/11/12 00:20:46 INFO Received complete push request method=POST path=/40272026/11/12 00:20:46 INFO Completed push id=13 objects=2 duration=1.56187ms40282026/11/12 00:20:46 INFO Received push request method=POST path=/40292026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40302026/11/12 00:20:46 INFO Received uploads request method=POST path=/40312026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40322026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=040332026/11/12 00:20:46 INFO Received complete push request method=POST path=/40342026/11/12 00:20:46 INFO Completed upload id=33 objects=5 duration=2.755068ms40352026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period40362026/11/12 00:20:46 INFO Completed push id=26 objects=4 duration=1.233339ms40372026/11/12 00:20:46 INFO Received uploads request method=POST path=/40382026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40392026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=040402026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40412026/11/12 00:20:46 INFO Completed upload id=14 objects=16 duration=2.664258ms40422026/11/12 00:20:46 INFO Completed upload id=27 objects=13 duration=1.277109ms40432026/11/12 00:20:46 INFO Received uploads request method=POST path=/40442026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period40452026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40462026/11/12 00:20:46 INFO Completed upload id=20 objects=4 duration=981.619µs40472026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040482026/11/12 00:20:46 INFO Received uploads request method=POST path=/40492026/11/12 00:20:46 INFO Vacuumed table table=pending_closures40502026/11/12 00:20:46 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40512026/11/12 00:20:46 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40522026/11/12 00:20:46 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40532026/11/12 00:20:46 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40542026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40552026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40562026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40572026/11/12 00:20:46 INFO Vacuumed table table=pending_objects40582026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads40592026/11/12 00:20:46 INFO Vacuumed table table=closures40602026/11/12 00:20:46 INFO Vacuumed table table=objects40612026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40622026/11/12 00:20:46 INFO Completed upload id=28 objects=13 duration=1.18099ms40632026/11/12 00:20:46 INFO Received push request method=POST path=/40642026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40652026/11/12 00:20:46 INFO Received uploads request method=POST path=/40662026/11/12 00:20:46 INFO Completed upload id=16 objects=17 duration=2.644499ms40672026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=040682026/11/12 00:20:46 INFO Received uploads request method=POST path=/40692026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040702026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=040712026/11/12 00:20:46 INFO Vacuumed table table=pending_closures40722026/11/12 00:20:46 INFO Vacuumed table table=pending_objects40732026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads40742026/11/12 00:20:46 INFO Vacuumed table table=closures40752026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period40762026/11/12 00:20:46 INFO Vacuumed table table=objects40772026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040782026/11/12 00:20:46 INFO Received push request method=POST path=/40792026/11/12 00:20:46 INFO Received complete push request method=POST path=/40802026/11/12 00:20:46 INFO Received complete push request method=POST path=/40812026/11/12 00:20:46 INFO Vacuumed table table=pending_closures40822026/11/12 00:20:46 INFO Completed push id=29 objects=3 duration=1.710729ms40832026/11/12 00:20:46 INFO Received uploads request method=POST path=/40842026/11/12 00:20:46 INFO Vacuumed table table=pending_objects40852026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads40862026/11/12 00:20:46 INFO Completed push id=11 objects=6 duration=3.762419ms40872026/11/12 00:20:46 INFO Received complete push request method=POST path=/40882026/11/12 00:20:46 INFO Vacuumed table table=closures40892026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=040902026/11/12 00:20:46 INFO Received complete push request method=POST path=/40912026/11/12 00:20:46 INFO Completed push id=16 objects=5 duration=1.307489ms40922026/11/12 00:20:46 INFO Vacuumed table table=objects40932026/11/12 00:20:46 INFO Received complete upload request method=POST path=/40942026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period40952026/11/12 00:20:46 INFO Received uploads request method=POST path=/40962026/11/12 00:20:46 INFO Completed push id=7 objects=15 duration=3.125699ms40972026/11/12 00:20:46 INFO Completed upload id=23 objects=18 duration=3.000689ms40982026/11/12 00:20:46 INFO Received uploads request method=POST path=/40992026/11/12 00:20:46 INFO Received complete upload request method=POST path=/41002026/11/12 00:20:46 INFO Received complete upload request method=POST path=/41012026/11/12 00:20:46 INFO Completed upload id=21 objects=3 duration=1.19269ms41022026/11/12 00:20:46 INFO Completed upload id=21 objects=25 duration=1.571769ms41032026/11/12 00:20:46 INFO Received uploads request method=POST path=/41042026/11/12 00:20:46 INFO Received uploads request method=POST path=/41052026/11/12 00:20:46 INFO Received complete upload request method=POST path=/41062026/11/12 00:20:46 INFO Received push request method=POST path=/41072026/11/12 00:20:46 INFO Completed upload id=31 objects=2 duration=1.25655ms41082026/11/12 00:20:46 INFO Received complete push request method=POST path=/41092026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041102026/11/12 00:20:46 INFO Received complete push request method=POST path=/41112026/11/12 00:20:46 INFO Completed push id=17 objects=11 duration=2.605839ms41122026/11/12 00:20:46 INFO Received uploads request method=POST path=/41132026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041142026/11/12 00:20:46 INFO Received uploads request method=POST path=/41152026/11/12 00:20:46 INFO Completed push id=27 objects=25 duration=3.235379ms41162026/11/12 00:20:46 INFO Received complete push request method=POST path=/41172026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41182026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041192026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41202026/11/12 00:20:46 INFO Completed push id=18 objects=6 duration=1.65544ms41212026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period41222026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41232026/11/12 00:20:46 INFO Received uploads request method=POST path=/41242026/11/12 00:20:46 INFO Vacuumed table table=closures41252026/11/12 00:20:46 INFO Vacuumed table table=objects41262026/11/12 00:20:46 INFO Received push request method=POST path=/41272026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041282026/11/12 00:20:46 INFO Received complete push request method=POST path=/41292026/11/12 00:20:46 INFO Received push request method=POST path=/41302026/11/12 00:20:46 INFO Received push request method=POST path=/41312026/11/12 00:20:46 INFO Completed push id=33 objects=6 duration=1.315449ms41322026/11/12 00:20:46 INFO Received push request method=POST path=/41332026/11/12 00:20:46 INFO Received push request method=POST path=/41342026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=041352026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041362026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41372026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041382026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041392026/11/12 00:20:46 INFO Received uploads request method=POST path=/41402026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41412026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41422026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41432026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41442026/11/12 00:20:46 INFO Vacuumed table table=closures41452026/11/12 00:20:46 INFO Received push request method=POST path=/41462026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41472026/11/12 00:20:46 INFO Vacuumed table table=objects41482026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041492026/11/12 00:20:46 INFO Vacuumed table table=closures41502026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041512026/11/12 00:20:46 INFO Vacuumed table table=objects41522026/11/12 00:20:46 INFO Received push request method=POST path=/41532026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41542026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41552026/11/12 00:20:46 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."41562026/11/12 00:20:46 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."41572026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41582026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041592026/11/12 00:20:46 INFO Vacuumed table table=closures41602026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41612026/11/12 00:20:46 INFO Vacuumed table table=objects41622026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41632026/11/12 00:20:46 INFO Received push request method=POST path=/41642026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41652026/11/12 00:20:46 INFO Vacuumed table table=closures41662026/11/12 00:20:46 INFO Vacuumed table table=objects41672026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041682026/11/12 00:20:46 INFO Received push request method=POST path=/41692026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period41702026/11/12 00:20:46 INFO Received push request method=POST path=/41712026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041722026/11/12 00:20:46 INFO Received push request method=POST path=/41732026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=041742026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41752026/11/12 00:20:46 INFO Received complete upload request method=POST path=/41762026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41772026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41782026/11/12 00:20:46 INFO Completed upload id=22 objects=22 duration=1.6028ms41792026/11/12 00:20:46 INFO Vacuumed table table=closures41802026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041812026/11/12 00:20:46 INFO Vacuumed table table=objects41822026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41832026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=041842026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41852026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41862026/11/12 00:20:46 INFO Received complete push request method=POST path=/41872026/11/12 00:20:46 INFO Vacuumed table table=pending_closures41882026/11/12 00:20:46 INFO Vacuumed table table=closures41892026/11/12 00:20:46 INFO Vacuumed table table=pending_objects41902026/11/12 00:20:46 INFO Received complete upload request method=POST path=/41912026/11/12 00:20:46 INFO Vacuumed table table=objects41922026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads41932026/11/12 00:20:46 INFO Completed push id=24 objects=5 duration=2.890479ms41942026/11/12 00:20:46 INFO Received uploads request method=POST path=/41952026/11/12 00:20:46 INFO Received complete upload request method=POST path=/41962026/11/12 00:20:46 INFO Completed upload id=12 objects=6 duration=1.968099ms41972026/11/12 00:20:46 INFO Vacuumed table table=closures41982026/11/12 00:20:46 INFO Vacuumed table table=objects41992026/11/12 00:20:46 INFO Completed upload id=20 objects=3 duration=2.694419ms42002026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=042012026/11/12 00:20:46 INFO Received complete push request method=POST path=/42022026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period42032026/11/12 00:20:46 INFO Completed push id=16 objects=26 duration=1.417779ms42042026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042052026/11/12 00:20:46 INFO Vacuumed table table=pending_closures42062026/11/12 00:20:46 INFO Vacuumed table table=pending_objects42072026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads42082026/11/12 00:20:46 INFO Vacuumed table table=closures42092026/11/12 00:20:46 INFO Received push request method=POST path=/42102026/11/12 00:20:46 INFO Vacuumed table table=objects42112026/11/12 00:20:46 INFO Received push request method=POST path=/42122026/11/12 00:20:46 INFO Received uploads request method=POST path=/42132026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=042142026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period42152026/11/12 00:20:46 INFO Received complete push request method=POST path=/42162026/11/12 00:20:46 INFO Received uploads request method=POST path=/42172026/11/12 00:20:46 INFO Completed push id=17 objects=3 duration=2.01661ms42182026/11/12 00:20:46 INFO Received push request method=POST path=/42192026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42202026/11/12 00:20:46 INFO Completed upload id=26 objects=6 duration=1.222829ms42212026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=042222026/11/12 00:20:46 INFO Vacuumed table table=pending_closures42232026/11/12 00:20:46 INFO Vacuumed table table=pending_objects42242026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=042252026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads42262026/11/12 00:20:46 INFO Received push request method=POST path=/42272026/11/12 00:20:46 INFO Vacuumed table table=closures42282026/11/12 00:20:46 INFO Vacuumed table table=objects42292026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period42302026/11/12 00:20:46 INFO Received uploads request method=POST path=/42312026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042322026/11/12 00:20:46 INFO Received complete push request method=POST path=/42332026/11/12 00:20:46 INFO Vacuumed table table=pending_closures42342026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=042352026/11/12 00:20:46 INFO Completed push id=26 objects=5 duration=2.163589ms42362026/11/12 00:20:46 INFO Received uploads request method=POST path=/42372026/11/12 00:20:46 INFO Vacuumed table table=pending_objects42382026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads42392026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period42402026/11/12 00:20:46 INFO Vacuumed table table=closures42412026/11/12 00:20:46 INFO Vacuumed table table=objects42422026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042432026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42442026/11/12 00:20:46 INFO Vacuumed table table=pending_closures42452026/11/12 00:20:46 INFO Vacuumed table table=pending_objects42462026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads42472026/11/12 00:20:46 INFO Completed upload id=29 objects=4 duration=1.92822ms42482026/11/12 00:20:46 INFO Vacuumed table table=closures42492026/11/12 00:20:46 INFO Received uploads request method=POST path=/42502026/11/12 00:20:46 INFO Vacuumed table table=objects42512026/11/12 00:20:46 INFO Received complete push request method=POST path=/42522026/11/12 00:20:46 INFO Completed push id=23 objects=5 duration=1.746609ms42532026/11/12 00:20:46 INFO Received push request method=POST path=/42542026/11/12 00:20:46 INFO Received complete push request method=POST path=/42552026/11/12 00:20:46 INFO Received complete push request method=POST path=/42562026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42572026/11/12 00:20:46 INFO Completed push id=16 objects=7 duration=2.077639ms42582026/11/12 00:20:46 INFO Completed push id=22 objects=3 duration=4.009859ms42592026/11/12 00:20:46 INFO Completed upload id=21 objects=7 duration=3.132489ms42602026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=042612026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42622026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period42632026/11/12 00:20:46 INFO Completed upload id=24 objects=14 duration=1.33734ms42642026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042652026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42662026/11/12 00:20:46 INFO Vacuumed table table=pending_closures42672026/11/12 00:20:46 INFO Completed upload id=15 objects=3 duration=1.946989ms42682026/11/12 00:20:46 INFO Vacuumed table table=pending_objects42692026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads42702026/11/12 00:20:46 INFO Received complete push request method=POST path=/42712026/11/12 00:20:46 INFO Vacuumed table table=closures42722026/11/12 00:20:46 INFO Received uploads request method=POST path=/42732026/11/12 00:20:46 INFO Vacuumed table table=objects42742026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42752026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42762026/11/12 00:20:46 INFO Completed push id=27 objects=6 duration=3.230469ms42772026/11/12 00:20:46 INFO Completed upload id=29 objects=5 duration=1.513599ms42782026/11/12 00:20:46 INFO Completed upload id=24 objects=7 duration=1.16206ms42792026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42802026/11/12 00:20:46 INFO Completed upload id=24 objects=3 duration=1.059769ms42812026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=042822026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042832026/11/12 00:20:46 INFO Received uploads request method=POST path=/42842026/11/12 00:20:46 INFO Vacuumed table table=pending_closures42852026/11/12 00:20:46 INFO Vacuumed table table=pending_objects42862026/11/12 00:20:46 INFO Received complete push request method=POST path=/42872026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads42882026/11/12 00:20:46 INFO Vacuumed table table=closures42892026/11/12 00:20:46 INFO Completed push id=21 objects=26 duration=1.487839ms42902026/11/12 00:20:46 INFO Vacuumed table table=objects42912026/11/12 00:20:46 INFO Received complete push request method=POST path=/42922026/11/12 00:20:46 INFO Completed push id=19 objects=3 duration=1.354559ms42932026/11/12 00:20:46 INFO Received complete push request method=POST path=/42942026/11/12 00:20:46 INFO Completed push id=23 objects=6 duration=1.336579ms42952026/11/12 00:20:46 INFO Received complete upload request method=POST path=/42962026/11/12 00:20:46 INFO Received uploads request method=POST path=/42972026/11/12 00:20:46 INFO Completed upload id=13 objects=8 duration=1.76672ms42982026/11/12 00:20:46 INFO Received complete push request method=POST path=/42992026/11/12 00:20:46 INFO Received complete push request method=POST path=/43002026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=043012026/11/12 00:20:46 INFO Completed push id=25 objects=4 duration=1.31307ms43022026/11/12 00:20:46 INFO Completed push id=19 objects=14 duration=2.371609ms43032026/11/12 00:20:46 INFO Received push request method=POST path=/43042026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043052026/11/12 00:20:46 INFO Vacuumed table table=pending_closures43062026/11/12 00:20:46 INFO Received push request method=POST path=/43072026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=043082026/11/12 00:20:46 INFO Vacuumed table table=pending_objects43092026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads43102026/11/12 00:20:46 INFO Vacuumed table table=closures43112026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043122026/11/12 00:20:46 INFO Vacuumed table table=objects43132026/11/12 00:20:46 INFO Vacuumed table table=pending_closures43142026/11/12 00:20:46 INFO Vacuumed table table=pending_objects43152026/11/12 00:20:46 INFO Received complete push request method=POST path=/43162026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads43172026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=043182026/11/12 00:20:46 INFO Received complete upload request method=POST path=/43192026/11/12 00:20:46 INFO Received push request method=POST path=/43202026/11/12 00:20:46 INFO Vacuumed table table=closures43212026/11/12 00:20:46 INFO Completed push id=24 objects=6 duration=1.548219ms43222026/11/12 00:20:46 INFO Completed upload id=28 objects=17 duration=1.607049ms43232026/11/12 00:20:46 INFO Vacuumed table table=objects43242026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period43252026/11/12 00:20:46 INFO Received uploads request method=POST path=/43262026/11/12 00:20:46 INFO Received complete upload request method=POST path=/43272026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043282026/11/12 00:20:46 INFO Completed upload id=15 objects=2 duration=1.683609ms43292026/11/12 00:20:46 INFO Vacuumed table table=pending_closures43302026/11/12 00:20:46 INFO Vacuumed table table=pending_objects43312026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads43322026/11/12 00:20:46 INFO Vacuumed table table=closures43332026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=043342026/11/12 00:20:46 INFO Received push request method=POST path=/43352026/11/12 00:20:46 INFO Vacuumed table table=objects43362026/11/12 00:20:46 INFO Received complete push request method=POST path=/43372026/11/12 00:20:46 INFO Completed push id=11 objects=6 duration=2.016549ms43382026/11/12 00:20:46 INFO Received uploads request method=POST path=/43392026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period43402026/11/12 00:20:46 INFO Received uploads request method=POST path=/43412026/11/12 00:20:46 INFO Received push request method=POST path=/43422026/11/12 00:20:46 INFO Received complete push request method=POST path=/43432026/11/12 00:20:46 INFO Completed push id=29 objects=2 duration=1.309619ms43442026/11/12 00:20:46 INFO Received uploads request method=POST path=/43452026/11/12 00:20:46 INFO Received complete push request method=POST path=/43462026/11/12 00:20:46 INFO Received push request method=POST path=/43472026/11/12 00:20:46 INFO Completed push id=24 objects=3 duration=1.37905ms43482026/11/12 00:20:46 INFO Received uploads request method=POST path=/43492026/11/12 00:20:46 INFO Received complete push request method=POST path=/43502026/11/12 00:20:46 INFO Completed push id=25 objects=7 duration=1.66715ms43512026/11/12 00:20:46 INFO Received push request method=POST path=/43522026/11/12 00:20:46 INFO Received complete push request method=POST path=/43532026/11/12 00:20:46 INFO Completed push id=20 objects=13 duration=1.771129ms43542026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=043552026/11/12 00:20:46 INFO Received complete push request method=POST path=/43562026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period43572026/11/12 00:20:46 INFO Completed push id=26 objects=23 duration=1.612579ms43582026/11/12 00:20:46 INFO Received complete push request method=POST path=/43592026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043602026/11/12 00:20:46 INFO Received complete push request method=POST path=/43612026/11/12 00:20:46 INFO Vacuumed table table=pending_closures43622026/11/12 00:20:46 INFO Completed push id=31 objects=6 duration=1.19685ms43632026/11/12 00:20:46 INFO Completed push id=31 objects=14 duration=3.199319ms43642026/11/12 00:20:46 INFO Vacuumed table table=pending_objects43652026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads43662026/11/12 00:20:46 INFO Vacuumed table table=closures43672026/11/12 00:20:46 INFO Received complete push request method=POST path=/43682026/11/12 00:20:46 INFO Vacuumed table table=objects43692026/11/12 00:20:46 INFO Received push request method=POST path=/43702026/11/12 00:20:46 INFO Received push request method=POST path=/43712026/11/12 00:20:46 INFO Completed push id=17 objects=22 duration=2.933069ms43722026/11/12 00:20:46 INFO Received uploads request method=POST path=/43732026/11/12 00:20:46 INFO Received uploads request method=POST path=/43742026/11/12 00:20:46 INFO Received push request method=POST path=/43752026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=043762026/11/12 00:20:46 INFO Received complete push request method=POST path=/43772026/11/12 00:20:46 INFO Vacuumed table table=pending_closures43782026/11/12 00:20:46 INFO Completed push id=22 objects=4 duration=1.33476ms43792026/11/12 00:20:46 INFO Vacuumed table table=pending_objects43802026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads43812026/11/12 00:20:46 INFO Vacuumed table table=closures43822026/11/12 00:20:46 INFO Received complete push request method=POST path=/43832026/11/12 00:20:46 INFO Vacuumed table table=objects43842026/11/12 00:20:46 INFO Completed push id=32 objects=9 duration=1.117049ms43852026/11/12 00:20:46 INFO Received uploads request method=POST path=/43862026/11/12 00:20:46 INFO Received uploads request method=POST path=/43872026/11/12 00:20:46 INFO Received complete upload request method=POST path=/43882026/11/12 00:20:46 INFO Completed upload id=27 objects=4 duration=1.445599ms43892026/11/12 00:20:46 INFO Received uploads request method=POST path=/43902026/11/12 00:20:46 INFO Received complete push request method=POST path=/43912026/11/12 00:20:46 INFO Received complete upload request method=POST path=/43922026/11/12 00:20:46 INFO Completed upload id=26 objects=24 duration=1.498079ms43932026/11/12 00:20:46 INFO Received complete upload request method=POST path=/43942026/11/12 00:20:46 INFO Received complete push request method=POST path=/43952026/11/12 00:20:46 INFO Completed push id=23 objects=28 duration=3.754088ms43962026/11/12 00:20:46 INFO Completed upload id=25 objects=4 duration=1.344569ms43972026/11/12 00:20:46 INFO Completed push id=27 objects=15 duration=1.43674ms43982026/11/12 00:20:46 INFO Received uploads request method=POST path=/43992026/11/12 00:20:46 INFO Received push request method=POST path=/44002026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44012026/11/12 00:20:46 INFO Completed upload id=23 objects=7 duration=1.635899ms44022026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044032026/11/12 00:20:46 INFO Received push request method=POST path=/44042026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period44052026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44062026/11/12 00:20:46 INFO Completed upload id=33 objects=9 duration=988.37µs44072026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044082026/11/12 00:20:46 INFO Received complete push request method=POST path=/44092026/11/12 00:20:46 INFO Received push request method=POST path=/44102026/11/12 00:20:46 INFO Vacuumed table table=pending_closures44112026/11/12 00:20:46 INFO Completed push id=13 objects=11 duration=1.944159ms44122026/11/12 00:20:46 INFO Vacuumed table table=pending_objects44132026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads44142026/11/12 00:20:46 INFO Vacuumed table table=closures44152026/11/12 00:20:46 INFO Vacuumed table table=objects44162026/11/12 00:20:46 INFO Received push request method=POST path=/44172026/11/12 00:20:46 INFO Received complete push request method=POST path=/44182026/11/12 00:20:46 INFO Completed push id=5 objects=28 duration=2.190849ms44192026/11/12 00:20:46 INFO Received complete push request method=POST path=/44202026/11/12 00:20:46 INFO Received uploads request method=POST path=/44212026/11/12 00:20:46 INFO Completed push id=24 objects=5 duration=1.319219ms44222026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44232026/11/12 00:20:46 INFO Received uploads request method=POST path=/44242026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044252026/11/12 00:20:46 INFO Completed upload id=31 objects=3 duration=2.582579ms44262026/11/12 00:20:46 INFO Received uploads request method=POST path=/44272026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period44282026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44292026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044302026/11/12 00:20:46 INFO Completed upload id=27 objects=8 duration=1.18329ms44312026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44322026/11/12 00:20:46 INFO Vacuumed table table=pending_closures44332026/11/12 00:20:46 INFO Completed upload id=27 objects=26 duration=1.685369ms44342026/11/12 00:20:46 INFO Vacuumed table table=pending_objects44352026/11/12 00:20:46 INFO Received complete push request method=POST path=/44362026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads44372026/11/12 00:20:46 INFO Vacuumed table table=closures44382026/11/12 00:20:46 INFO Completed push id=10 objects=2 duration=1.708549ms44392026/11/12 00:20:46 INFO Vacuumed table table=objects44402026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44412026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044422026/11/12 00:20:46 INFO Completed upload id=25 objects=2 duration=1.13778ms44432026/11/12 00:20:46 INFO Received complete push request method=POST path=/44442026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044452026/11/12 00:20:46 INFO Completed push id=28 objects=19 duration=1.525129ms44462026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44472026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044482026/11/12 00:20:46 INFO Completed upload id=22 objects=2 duration=1.050569ms44492026/11/12 00:20:46 INFO Vacuumed table table=pending_closures44502026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044512026/11/12 00:20:46 INFO Received uploads request method=POST path=/44522026/11/12 00:20:46 INFO Vacuumed table table=pending_closures44532026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044542026/11/12 00:20:46 INFO Vacuumed table table=pending_objects44552026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads44562026/11/12 00:20:46 INFO Received complete push request method=POST path=/44572026/11/12 00:20:46 INFO Vacuumed table table=pending_objects44582026/11/12 00:20:46 INFO Vacuumed table table=closures44592026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads44602026/11/12 00:20:46 INFO Received push request method=POST path=/44612026/11/12 00:20:46 INFO Vacuumed table table=closures44622026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period44632026/11/12 00:20:46 INFO Vacuumed table table=objects44642026/11/12 00:20:46 INFO Completed push id=7 objects=7 duration=2.046849ms44652026/11/12 00:20:46 INFO Vacuumed table table=objects44662026/11/12 00:20:46 INFO Received complete push request method=POST path=/44672026/11/12 00:20:46 INFO Completed push id=20 objects=8 duration=1.417399ms44682026/11/12 00:20:46 INFO Received complete push request method=POST path=/44692026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044702026/11/12 00:20:46 INFO Completed push id=28 objects=6 duration=1.389259ms44712026/11/12 00:20:46 INFO Vacuumed table table=pending_closures44722026/11/12 00:20:46 INFO Received push request method=POST path=/44732026/11/12 00:20:46 INFO Vacuumed table table=pending_objects44742026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads44752026/11/12 00:20:46 INFO Vacuumed table table=closures44762026/11/12 00:20:46 INFO Vacuumed table table=objects44772026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44782026/11/12 00:20:46 INFO Completed upload id=26 objects=20 duration=1.0865ms44792026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44802026/11/12 00:20:46 INFO Completed upload id=17 objects=5 duration=1.211419ms44812026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044822026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044832026/11/12 00:20:46 INFO Received complete upload request method=POST path=/44842026/11/12 00:20:46 INFO Completed upload id=26 objects=8 duration=1.58478ms44852026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044862026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period44872026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period44882026/11/12 00:20:46 INFO Received complete push request method=POST path=/44892026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044902026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044912026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=044922026/11/12 00:20:46 INFO Completed push id=12 objects=6 duration=1.182349ms44932026/11/12 00:20:46 INFO Vacuumed table table=pending_closures44942026/11/12 00:20:46 INFO Vacuumed table table=pending_closures44952026/11/12 00:20:46 INFO Received complete push request method=POST path=/44962026/11/12 00:20:46 INFO Received uploads request method=POST path=/44972026/11/12 00:20:46 INFO Vacuumed table table=pending_objects44982026/11/12 00:20:46 INFO Vacuumed table table=pending_objects44992026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads45002026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads45012026/11/12 00:20:46 INFO Completed push id=29 objects=6 duration=1.213089ms45022026/11/12 00:20:46 INFO Received complete push request method=POST path=/45032026/11/12 00:20:46 INFO Vacuumed table table=closures45042026/11/12 00:20:46 INFO Received complete push request method=POST path=/45052026/11/12 00:20:46 INFO Vacuumed table table=closures45062026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045072026/11/12 00:20:46 INFO Completed push id=29 objects=10 duration=1.383079ms45082026/11/12 00:20:46 INFO Vacuumed table table=objects45092026/11/12 00:20:46 INFO Vacuumed table table=objects45102026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=045112026/11/12 00:20:46 INFO Vacuumed table table=pending_closures45122026/11/12 00:20:46 INFO Completed push id=19 objects=14 duration=2.178509ms45132026/11/12 00:20:46 INFO Vacuumed table table=pending_closures45142026/11/12 00:20:46 INFO Vacuumed table table=pending_objects45152026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads45162026/11/12 00:20:46 INFO Vacuumed table table=closures45172026/11/12 00:20:46 INFO Vacuumed table table=pending_objects45182026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads45192026/11/12 00:20:46 INFO Vacuumed table table=objects45202026/11/12 00:20:46 INFO Vacuumed table table=closures45212026/11/12 00:20:46 INFO Vacuumed table table=objects45222026/11/12 00:20:46 INFO Received complete push request method=POST path=/45232026/11/12 00:20:46 INFO Received complete push request method=POST path=/45242026/11/12 00:20:46 INFO Completed push id=28 objects=14 duration=1.373679ms45252026/11/12 00:20:46 INFO Completed push id=28 objects=7 duration=1.386659ms45262026/11/12 00:20:46 INFO Received complete push request method=POST path=/45272026/11/12 00:20:46 INFO Completed push id=19 objects=3 duration=1.27769ms45282026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45292026/11/12 00:20:46 INFO Received push request method=POST path=/45302026/11/12 00:20:46 INFO Completed upload id=19 objects=2 duration=2.896338ms45312026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45322026/11/12 00:20:46 INFO Received uploads request method=POST path=/45332026/11/12 00:20:46 INFO Received uploads request method=POST path=/45342026/11/12 00:20:46 INFO Completed upload id=18 objects=2 duration=2.290239ms45352026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=045362026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45372026/11/12 00:20:46 INFO Completed upload id=30 objects=6 duration=1.186049ms45382026/11/12 00:20:46 INFO Received push request method=POST path=/45392026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045402026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45412026/11/12 00:20:46 INFO Vacuumed table table=pending_closures45422026/11/12 00:20:46 INFO Completed upload id=30 objects=8 duration=2.430119ms45432026/11/12 00:20:46 INFO Vacuumed table table=pending_objects45442026/11/12 00:20:46 INFO Received uploads request method=POST path=/45452026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads45462026/11/12 00:20:46 INFO Vacuumed table table=closures45472026/11/12 00:20:46 INFO Vacuumed table table=objects45482026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45492026/11/12 00:20:46 INFO Completed upload id=30 objects=6 duration=1.32129ms45502026/11/12 00:20:46 INFO Received complete push request method=POST path=/45512026/11/12 00:20:46 INFO Received push request method=POST path=/45522026/11/12 00:20:46 INFO Completed push id=13 objects=22 duration=1.696999ms45532026/11/12 00:20:46 INFO Received uploads request method=POST path=/45542026/11/12 00:20:46 INFO Received complete push request method=POST path=/45552026/11/12 00:20:46 INFO Completed push id=20 objects=4 duration=1.442059ms45562026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45572026/11/12 00:20:46 INFO Completed upload id=31 objects=5 duration=1.28396ms45582026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45592026/11/12 00:20:46 INFO Received complete push request method=POST path=/45602026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45612026/11/12 00:20:46 INFO Completed upload id=20 objects=6 duration=1.516699ms45622026/11/12 00:20:46 INFO Completed upload id=13 objects=14 duration=3.444949ms45632026/11/12 00:20:46 INFO Received uploads request method=POST path=/45642026/11/12 00:20:46 INFO Completed push id=16 objects=14 duration=2.864449ms45652026/11/12 00:20:46 INFO Received complete push request method=POST path=/45662026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45672026/11/12 00:20:46 INFO Received complete push request method=POST path=/45682026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45692026/11/12 00:20:46 INFO Completed push id=33 objects=7 duration=1.971659ms45702026/11/12 00:20:46 INFO Completed upload id=31 objects=6 duration=1.00232ms45712026/11/12 00:20:46 INFO Completed push id=31 objects=5 duration=1.27166ms45722026/11/12 00:20:46 INFO Received complete push request method=POST path=/45732026/11/12 00:20:46 INFO Completed upload id=25 objects=16 duration=2.341239ms45742026/11/12 00:20:46 INFO Completed push id=27 objects=11 duration=1.50562ms45752026/11/12 00:20:46 INFO Received complete push request method=POST path=/45762026/11/12 00:20:46 INFO Completed push id=21 objects=6 duration=1.337079ms45772026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45782026/11/12 00:20:46 INFO Completed upload id=26 objects=2 duration=1.91533ms45792026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45802026/11/12 00:20:46 INFO Completed upload id=34 objects=3 duration=822.249µs45812026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=045822026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45832026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=045842026/11/12 00:20:46 INFO Completed upload id=26 objects=3 duration=1.1831ms45852026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=045862026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period45872026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45882026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045892026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45902026/11/12 00:20:46 INFO Completed upload id=32 objects=4 duration=1.601069ms45912026/11/12 00:20:46 INFO Vacuumed table table=pending_closures45922026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045932026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period45942026/11/12 00:20:46 INFO Received complete upload request method=POST path=/45952026/11/12 00:20:46 INFO Completed upload id=13 objects=7 duration=1.42933ms45962026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=045972026/11/12 00:20:46 INFO Vacuumed table table=pending_objects45982026/11/12 00:20:46 INFO Vacuumed table table=pending_closures45992026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period46002026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46012026/11/12 00:20:46 INFO Received complete upload request method=POST path=/46022026/11/12 00:20:46 INFO Completed upload id=22 objects=8 duration=1.199059ms46032026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046042026/11/12 00:20:46 INFO Vacuumed table table=closures46052026/11/12 00:20:46 INFO Vacuumed table table=pending_objects46062026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046072026/11/12 00:20:46 INFO Completed upload id=29 objects=26 duration=1.61747ms46082026/11/12 00:20:46 INFO Received complete upload request method=POST path=/46092026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46102026/11/12 00:20:46 INFO Vacuumed table table=objects46112026/11/12 00:20:46 INFO Vacuumed table table=pending_closures46122026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=046132026/11/12 00:20:46 INFO Vacuumed table table=pending_closures46142026/11/12 00:20:46 INFO Completed upload id=22 objects=7 duration=1.650679ms46152026/11/12 00:20:46 INFO Vacuumed table table=closures46162026/11/12 00:20:46 INFO Vacuumed table table=pending_objects46172026/11/12 00:20:46 INFO Vacuumed table table=pending_objects46182026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46192026/11/12 00:20:46 INFO Vacuumed table table=objects46202026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46212026/11/12 00:20:46 INFO Vacuumed table table=closures46222026/11/12 00:20:46 INFO Vacuumed table table=closures46232026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046242026/11/12 00:20:46 INFO Vacuumed table table=objects46252026/11/12 00:20:46 INFO Vacuumed table table=objects46262026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=046272026/11/12 00:20:46 INFO Received complete push request method=POST path=/46282026/11/12 00:20:46 INFO Vacuumed table table=pending_closures46292026/11/12 00:20:46 INFO Received uploads request method=POST path=/46302026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period46312026/11/12 00:20:46 INFO Completed push id=32 objects=26 duration=1.608029ms46322026/11/12 00:20:46 INFO Vacuumed table table=pending_objects46332026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46342026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=046352026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046362026/11/12 00:20:46 INFO Vacuumed table table=closures46372026/11/12 00:20:46 WARN Force mode enabled - objects will be deleted immediately without grace period46382026/11/12 00:20:46 INFO Vacuumed table table=objects46392026/11/12 00:20:46 INFO Vacuumed table table=pending_closures46402026/11/12 00:20:46 INFO Received complete upload request method=POST path=/46412026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046422026/11/12 00:20:46 INFO Completed upload id=20 objects=4 duration=1.055549ms46432026/11/12 00:20:46 INFO Vacuumed table table=pending_closures46442026/11/12 00:20:46 INFO Vacuumed table table=pending_objects46452026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46462026/11/12 00:20:46 INFO Vacuumed table table=pending_objects46472026/11/12 00:20:46 INFO Vacuumed table table=closures46482026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46492026/11/12 00:20:46 INFO Vacuumed table table=closures46502026/11/12 00:20:46 INFO Vacuumed table table=objects46512026/11/12 00:20:46 INFO Vacuumed table table=objects46522026/11/12 00:20:46 INFO Received complete push request method=POST path=/46532026/11/12 00:20:46 INFO Received complete push request method=POST path=/46542026/11/12 00:20:46 INFO Completed push id=23 objects=5 duration=1.21676ms46552026/11/12 00:20:46 INFO Completed push id=30 objects=8 duration=1.27165ms46562026/11/12 00:20:46 INFO Received complete push request method=POST path=/46572026/11/12 00:20:46 INFO Completed push id=24 objects=4 duration=1.15401ms46582026/11/12 00:20:46 INFO Aborted multipart uploads count=0 kept=046592026/11/12 00:20:46 INFO Received uploads request method=POST path=/46602026/11/12 00:20:46 INFO Received complete upload request method=POST path=/46612026/11/12 00:20:46 INFO Received push request method=POST path=/46622026/11/12 00:20:46 INFO Received complete push request method=POST path=/46632026/11/12 00:20:46 INFO Completed upload id=33 objects=10 duration=1.083109ms46642026/11/12 00:20:46 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046652026/11/12 00:20:46 INFO Vacuumed table table=pending_closures46662026/11/12 00:20:46 INFO Received complete upload request method=POST path=/46672026/11/12 00:20:46 INFO Completed push id=31 objects=6 duration=3.247149ms46682026/11/12 00:20:46 INFO Vacuumed table table=pending_objects46692026/11/12 00:20:46 INFO Completed upload id=33 objects=2 duration=1.25464ms46702026/11/12 00:20:46 INFO Vacuumed table table=multipart_uploads46712026/11/12 00:20:46 INFO Vacuumed table table=closures46722026/11/12 00:20:46 INFO Received complete upload request method=POST path=/46732026/11/12 00:20:46 INFO Vacuumed table table=objects46742026/11/12 00:20:46 INFO Completed upload id=23 objects=7 duration=1.479749ms46752026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46762026/11/12 00:20:47 INFO Completed upload id=24 objects=8 duration=1.107729ms46772026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46782026/11/12 00:20:47 INFO Completed upload id=31 objects=5 duration=1.176169ms46792026/11/12 00:20:47 INFO Received complete push request method=POST path=/46802026/11/12 00:20:47 INFO Completed push id=25 objects=9 duration=1.34552ms46812026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46822026/11/12 00:20:47 INFO Received uploads request method=POST path=/46832026/11/12 00:20:47 INFO Received push request method=POST path=/46842026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46852026/11/12 00:20:47 INFO Completed upload id=21 objects=9 duration=2.112809ms46862026/11/12 00:20:47 INFO Received push request method=POST path=/46872026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46882026/11/12 00:20:47 INFO Completed upload id=21 objects=2 duration=1.75151ms46892026/11/12 00:20:47 INFO Received push request method=POST path=/46902026/11/12 00:20:47 INFO Completed upload id=34 objects=5 duration=1.63058ms46912026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46922026/11/12 00:20:47 INFO Completed upload id=34 objects=23 duration=1.20068ms46932026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46942026/11/12 00:20:47 INFO Completed upload id=28 objects=5 duration=1.098969ms46952026/11/12 00:20:47 INFO Received complete push request method=POST path=/46962026/11/12 00:20:47 INFO Completed push id=32 objects=3 duration=1.134309ms46972026/11/12 00:20:47 INFO Received complete upload request method=POST path=/46982026/11/12 00:20:47 INFO Completed upload id=26 objects=3 duration=1.05001ms46992026/11/12 00:20:47 INFO Received complete push request method=POST path=/47002026/11/12 00:20:47 INFO Received uploads request method=POST path=/47012026/11/12 00:20:47 INFO Completed push id=25 objects=10 duration=1.906129ms47022026/11/12 00:20:47 INFO Received uploads request method=POST path=/47032026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47042026/11/12 00:20:47 INFO Completed upload id=35 objects=11 duration=1.19038ms47052026/11/12 00:20:47 INFO Received complete push request method=POST path=/47062026/11/12 00:20:47 INFO Completed push id=9 objects=9 duration=3.219359ms47072026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=047082026/11/12 00:20:47 INFO Received complete push request method=POST path=/47092026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47102026/11/12 00:20:47 INFO Completed push id=35 objects=6 duration=1.22552ms47112026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47122026/11/12 00:20:47 INFO Completed upload id=24 objects=18 duration=1.721339ms47132026/11/12 00:20:47 INFO Completed upload id=30 objects=5 duration=958.38µs47142026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47152026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period47162026/11/12 00:20:47 INFO Completed upload id=29 objects=5 duration=1.04644ms47172026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=047182026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047192026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47202026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period47212026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=047222026/11/12 00:20:47 INFO Vacuumed table table=pending_closures47232026/11/12 00:20:47 INFO Completed upload id=27 objects=7 duration=1.479349ms47242026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047252026/11/12 00:20:47 INFO Vacuumed table table=pending_closures47262026/11/12 00:20:47 INFO Vacuumed table table=pending_objects47272026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads47282026/11/12 00:20:47 INFO Vacuumed table table=pending_objects47292026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads47302026/11/12 00:20:47 INFO Vacuumed table table=closures47312026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47322026/11/12 00:20:47 INFO Vacuumed table table=closures47332026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47342026/11/12 00:20:47 INFO Vacuumed table table=objects47352026/11/12 00:20:47 INFO Vacuumed table table=objects47362026/11/12 00:20:47 INFO Completed upload id=36 objects=3 duration=1.229219ms47372026/11/12 00:20:47 INFO Completed upload id=25 objects=6 duration=2.151849ms47382026/11/12 00:20:47 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."47392026/11/12 00:20:47 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."47402026/11/12 00:20:47 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."47412026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47422026/11/12 00:20:47 INFO Received complete push request method=POST path=/47432026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=047442026/11/12 00:20:47 INFO Completed upload id=21 objects=8 duration=1.356319ms47452026/11/12 00:20:47 INFO Completed push id=25 objects=2 duration=2.323809ms47462026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period47472026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47482026/11/12 00:20:47 INFO Completed upload id=30 objects=2 duration=1.187619ms47492026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47502026/11/12 00:20:47 INFO Received uploads request method=POST path=/47512026/11/12 00:20:47 INFO Received complete push request method=POST path=/47522026/11/12 00:20:47 INFO Completed upload id=28 objects=3 duration=1.046369ms47532026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047542026/11/12 00:20:47 INFO Completed push id=31 objects=3 duration=1.28618ms47552026/11/12 00:20:47 INFO Vacuumed table table=pending_closures47562026/11/12 00:20:47 INFO Vacuumed table table=pending_objects47572026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads47582026/11/12 00:20:47 INFO Vacuumed table table=closures47592026/11/12 00:20:47 INFO Vacuumed table table=objects47602026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=047612026/11/12 00:20:47 INFO Received complete push request method=POST path=/47622026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period47632026/11/12 00:20:47 INFO Received push request method=POST path=/47642026/11/12 00:20:47 INFO Completed push id=22 objects=6 duration=1.2734ms47652026/11/12 00:20:47 INFO Received complete push request method=POST path=/47662026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047672026/11/12 00:20:47 INFO Completed push id=27 objects=4 duration=1.04185ms47682026/11/12 00:20:47 INFO Vacuumed table table=pending_closures47692026/11/12 00:20:47 INFO Received complete push request method=POST path=/47702026/11/12 00:20:47 INFO Vacuumed table table=pending_objects47712026/11/12 00:20:47 INFO Completed push id=31 objects=20 duration=1.56646ms47722026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads47732026/11/12 00:20:47 INFO Received complete push request method=POST path=/47742026/11/12 00:20:47 INFO Vacuumed table table=closures47752026/11/12 00:20:47 INFO Vacuumed table table=objects47762026/11/12 00:20:47 INFO Completed push id=1 objects=9 duration=2.277349ms47772026/11/12 00:20:47 INFO Received uploads request method=POST path=/47782026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47792026/11/12 00:20:47 INFO Received push request method=POST path=/47802026/11/12 00:20:47 INFO Received complete push request method=POST path=/47812026/11/12 00:20:47 INFO Completed push id=29 objects=23 duration=1.427909ms47822026/11/12 00:20:47 INFO Completed upload id=22 objects=5 duration=2.306379ms47832026/11/12 00:20:47 INFO Received complete push request method=POST path=/47842026/11/12 00:20:47 INFO Completed push id=24 objects=8 duration=1.443039ms47852026/11/12 00:20:47 INFO Received complete push request method=POST path=/47862026/11/12 00:20:47 INFO Completed push id=32 objects=6 duration=1.154199ms47872026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47882026/11/12 00:20:47 INFO Completed upload id=32 objects=5 duration=919.57µs47892026/11/12 00:20:47 INFO Received uploads request method=POST path=/47902026/11/12 00:20:47 INFO Received complete push request method=POST path=/47912026/11/12 00:20:47 INFO Received push request method=POST path=/47922026/11/12 00:20:47 INFO Completed push id=30 objects=6 duration=959.089µs47932026/11/12 00:20:47 INFO Received push request method=POST path=/47942026/11/12 00:20:47 INFO Received complete upload request method=POST path=/47952026/11/12 00:20:47 INFO Received complete push request method=POST path=/47962026/11/12 00:20:47 INFO Received complete push request method=POST path=/47972026/11/12 00:20:47 INFO Completed upload id=25 objects=9 duration=1.835309ms47982026/11/12 00:20:47 INFO Completed push id=28 objects=26 duration=2.007789ms47992026/11/12 00:20:47 INFO Completed push id=28 objects=3 duration=2.304799ms48002026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48012026/11/12 00:20:47 INFO Completed upload id=33 objects=7 duration=824.389µs48022026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48032026/11/12 00:20:47 INFO Completed upload id=33 objects=3 duration=979.349µs48042026/11/12 00:20:47 INFO Received complete push request method=POST path=/48052026/11/12 00:20:47 INFO Completed push id=11 objects=9 duration=1.76055ms48062026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48072026/11/12 00:20:47 INFO Received complete push request method=POST path=/48082026/11/12 00:20:47 INFO Received complete push request method=POST path=/48092026/11/12 00:20:47 INFO Completed upload id=27 objects=3 duration=1.3923ms48102026/11/12 00:20:47 INFO Received complete push request method=POST path=/48112026/11/12 00:20:47 INFO Received complete push request method=POST path=/48122026/11/12 00:20:47 INFO Completed push id=30 objects=3 duration=1.060269ms48132026/11/12 00:20:47 INFO Completed push id=24 objects=24 duration=3.510729ms48142026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48152026/11/12 00:20:47 INFO Completed push id=29 objects=15 duration=1.59739ms48162026/11/12 00:20:47 INFO Completed push id=28 objects=11 duration=2.837669ms48172026/11/12 00:20:47 INFO Received complete push request method=POST path=/48182026/11/12 00:20:47 INFO Completed upload id=34 objects=3 duration=816.039µs48192026/11/12 00:20:47 INFO Completed push id=33 objects=3 duration=1.151989ms48202026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=048212026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period48222026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048232026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48242026/11/12 00:20:47 INFO Vacuumed table table=pending_closures48252026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48262026/11/12 00:20:47 INFO Completed upload id=28 objects=6 duration=1.16827ms48272026/11/12 00:20:47 INFO Completed upload id=2 objects=7 duration=1.616189ms48282026/11/12 00:20:47 INFO Received complete push request method=POST path=/48292026/11/12 00:20:47 INFO Vacuumed table table=pending_objects48302026/11/12 00:20:47 INFO Received complete push request method=POST path=/48312026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads48322026/11/12 00:20:47 INFO Completed push id=32 objects=19 duration=1.260629ms48332026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=048342026/11/12 00:20:47 INFO Vacuumed table table=closures48352026/11/12 00:20:47 INFO Completed push id=15 objects=24 duration=1.897379ms48362026/11/12 00:20:47 INFO Vacuumed table table=objects48372026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period48382026/11/12 00:20:47 INFO Received complete push request method=POST path=/48392026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048402026/11/12 00:20:47 INFO Completed push id=30 objects=6 duration=1.24545ms48412026/11/12 00:20:47 INFO Received complete push request method=POST path=/48422026/11/12 00:20:47 INFO Received complete push request method=POST path=/48432026/11/12 00:20:47 INFO Vacuumed table table=pending_closures48442026/11/12 00:20:47 INFO Completed push id=35 objects=11 duration=1.021049ms48452026/11/12 00:20:47 INFO Completed push id=34 objects=6 duration=1.065379ms48462026/11/12 00:20:47 INFO Vacuumed table table=pending_objects48472026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads48482026/11/12 00:20:47 INFO Vacuumed table table=closures48492026/11/12 00:20:47 INFO Vacuumed table table=objects48502026/11/12 00:20:47 INFO Received complete push request method=POST path=/48512026/11/12 00:20:47 INFO Completed push id=30 objects=4 duration=1.21505ms48522026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=048532026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48542026/11/12 00:20:47 INFO Completed upload id=19 objects=2 duration=1.147089ms48552026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period48562026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048572026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48582026/11/12 00:20:47 INFO Completed upload id=31 objects=3 duration=1.17589ms48592026/11/12 00:20:47 INFO Received complete push request method=POST path=/48602026/11/12 00:20:47 INFO Vacuumed table table=pending_closures48612026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=048622026/11/12 00:20:47 INFO Completed push id=35 objects=8 duration=1.424469ms48632026/11/12 00:20:47 INFO Vacuumed table table=pending_objects48642026/11/12 00:20:47 INFO Received uploads request method=POST path=/48652026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads48662026/11/12 00:20:47 INFO Received complete push request method=POST path=/48672026/11/12 00:20:47 INFO Received complete push request method=POST path=/48682026/11/12 00:20:47 INFO Vacuumed table table=closures48692026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period48702026/11/12 00:20:47 INFO Completed push id=31 objects=7 duration=1.41273ms48712026/11/12 00:20:47 INFO Vacuumed table table=objects48722026/11/12 00:20:47 INFO Completed push id=29 objects=24 duration=2.273079ms48732026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048742026/11/12 00:20:47 INFO Vacuumed table table=pending_closures48752026/11/12 00:20:47 INFO Vacuumed table table=pending_objects48762026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48772026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads48782026/11/12 00:20:47 INFO Vacuumed table table=closures48792026/11/12 00:20:47 INFO Completed upload id=24 objects=7 duration=1.5212ms48802026/11/12 00:20:47 INFO Vacuumed table table=objects48812026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48822026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=048832026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48842026/11/12 00:20:47 INFO Completed upload id=8 objects=6 duration=1.42519ms48852026/11/12 00:20:47 INFO Completed upload id=36 objects=9 duration=1.007409ms48862026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period48872026/11/12 00:20:47 INFO Received complete upload request method=POST path=/48882026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048892026/11/12 00:20:47 INFO Completed upload id=32 objects=5 duration=1.1428ms48902026/11/12 00:20:47 INFO Received uploads request method=POST path=/48912026/11/12 00:20:47 INFO Vacuumed table table=pending_closures48922026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=048932026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period48942026/11/12 00:20:47 INFO Vacuumed table table=pending_objects48952026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads48962026/11/12 00:20:47 INFO Vacuumed table table=closures48972026/11/12 00:20:47 INFO Vacuumed table table=objects48982026/11/12 00:20:47 INFO Received complete push request method=POST path=/48992026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049002026/11/12 00:20:47 INFO Completed push id=26 objects=3 duration=1.605109ms49012026/11/12 00:20:47 INFO Vacuumed table table=pending_closures49022026/11/12 00:20:47 INFO Vacuumed table table=pending_objects49032026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads49042026/11/12 00:20:47 INFO Vacuumed table table=closures49052026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=049062026/11/12 00:20:47 INFO Vacuumed table table=objects49072026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period49082026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049092026/11/12 00:20:47 INFO Vacuumed table table=pending_closures49102026/11/12 00:20:47 INFO Vacuumed table table=pending_objects49112026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads49122026/11/12 00:20:47 INFO Vacuumed table table=closures49132026/11/12 00:20:47 INFO Vacuumed table table=objects49142026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49152026/11/12 00:20:47 INFO Completed upload id=28 objects=8 duration=1.16785ms49162026/11/12 00:20:47 INFO Received push request method=POST path=/49172026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49182026/11/12 00:20:47 INFO Completed upload id=29 objects=10 duration=1.132259ms49192026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49202026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49212026/11/12 00:20:47 INFO Completed upload id=12 objects=9 duration=1.3931ms49222026/11/12 00:20:47 INFO Received uploads request method=POST path=/49232026/11/12 00:20:47 INFO Completed upload id=34 objects=24 duration=1.505039ms49242026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49252026/11/12 00:20:47 INFO Completed upload id=30 objects=20 duration=1.08871ms49262026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=049272026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period49282026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=049292026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049302026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period49312026/11/12 00:20:47 INFO Vacuumed table table=pending_closures49322026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049332026/11/12 00:20:47 INFO Vacuumed table table=pending_closures49342026/11/12 00:20:47 INFO Vacuumed table table=pending_objects49352026/11/12 00:20:47 INFO Vacuumed table table=pending_objects49362026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads49372026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads49382026/11/12 00:20:47 INFO Vacuumed table table=closures49392026/11/12 00:20:47 INFO Vacuumed table table=closures49402026/11/12 00:20:47 INFO Vacuumed table table=objects49412026/11/12 00:20:47 INFO Vacuumed table table=objects49422026/11/12 00:20:47 INFO Received complete push request method=POST path=/49432026/11/12 00:20:47 INFO Completed push id=31 objects=11 duration=1.382329ms49442026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49452026/11/12 00:20:47 INFO Completed upload id=32 objects=2 duration=1.051949ms49462026/11/12 00:20:47 INFO Received push request method=POST path=/49472026/11/12 00:20:47 INFO Received complete push request method=POST path=/49482026/11/12 00:20:47 INFO Completed push id=33 objects=3 duration=1.07318ms49492026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49502026/11/12 00:20:47 INFO Completed upload id=34 objects=3 duration=1.05335ms49512026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49522026/11/12 00:20:47 INFO Completed upload id=23 objects=7 duration=2.792509ms49532026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49542026/11/12 00:20:47 INFO Completed upload id=18 objects=22 duration=1.983269ms49552026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=049562026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period49572026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049582026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49592026/11/12 00:20:47 INFO Completed upload id=26 objects=5 duration=1.185849ms49602026/11/12 00:20:47 INFO Vacuumed table table=pending_closures49612026/11/12 00:20:47 INFO Vacuumed table table=pending_objects49622026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads49632026/11/12 00:20:47 INFO Vacuumed table table=closures49642026/11/12 00:20:47 INFO Vacuumed table table=objects49652026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49662026/11/12 00:20:47 INFO Completed upload id=21 objects=8 duration=1.2889ms49672026/11/12 00:20:47 INFO Received complete push request method=POST path=/49682026/11/12 00:20:47 INFO Completed push id=27 objects=3 duration=1.834499ms49692026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49702026/11/12 00:20:47 INFO Completed upload id=22 objects=7 duration=1.19672ms49712026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49722026/11/12 00:20:47 INFO Completed upload id=28 objects=5 duration=1.068099ms49732026/11/12 00:20:47 INFO Received complete push request method=POST path=/49742026/11/12 00:20:47 INFO Completed push id=24 objects=5 duration=1.32027ms49752026/11/12 00:20:47 INFO Received complete push request method=POST path=/49762026/11/12 00:20:47 INFO Completed push id=29 objects=17 duration=1.546979ms49772026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=049782026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49792026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period49802026/11/12 00:20:47 INFO Completed upload id=25 objects=8 duration=1.25685ms49812026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049822026/11/12 00:20:47 INFO Vacuumed table table=pending_closures49832026/11/12 00:20:47 INFO Vacuumed table table=pending_objects49842026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads49852026/11/12 00:20:47 INFO Vacuumed table table=closures49862026/11/12 00:20:47 INFO Vacuumed table table=objects49872026/11/12 00:20:47 INFO Received complete push request method=POST path=/49882026/11/12 00:20:47 INFO Completed push id=26 objects=4 duration=1.39179ms49892026/11/12 00:20:47 INFO Received complete push request method=POST path=/49902026/11/12 00:20:47 INFO Completed push id=30 objects=2 duration=1.305189ms49912026/11/12 00:20:47 INFO Received complete push request method=POST path=/49922026/11/12 00:20:47 INFO Completed push id=32 objects=7 duration=1.57861ms49932026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49942026/11/12 00:20:47 INFO Completed upload id=33 objects=6 duration=1.27264ms49952026/11/12 00:20:47 INFO Received complete upload request method=POST path=/49962026/11/12 00:20:47 INFO Completed upload id=34 objects=3 duration=1.252859ms49972026/11/12 00:20:47 INFO Aborted multipart uploads count=0 kept=049982026/11/12 00:20:47 WARN Force mode enabled - objects will be deleted immediately without grace period49992026/11/12 00:20:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050002026/11/12 00:20:47 INFO Vacuumed table table=pending_closures50012026/11/12 00:20:47 INFO Vacuumed table table=pending_objects50022026/11/12 00:20:47 INFO Vacuumed table table=multipart_uploads50032026/11/12 00:20:47 INFO Vacuumed table table=closures50042026/11/12 00:20:47 INFO Vacuumed table table=objects5005--- PASS: FuzzPushGC (0.00s)5006 --- PASS: FuzzPushGC/seed#14 (2.58s)5007 --- PASS: FuzzPushGC/seed#0 (2.83s)5008 --- PASS: FuzzPushGC/seed#13 (2.97s)5009 --- PASS: FuzzPushGC/seed#3 (3.05s)5010 --- PASS: FuzzPushGC/seed#12 (3.08s)5011 --- PASS: FuzzPushGC/seed#11 (3.09s)5012 --- PASS: FuzzPushGC/seed#6 (3.15s)5013 --- PASS: FuzzPushGC/seed#7 (3.18s)5014 --- PASS: FuzzPushGC/seed#8 (3.18s)5015 --- PASS: FuzzPushGC/seed#15 (3.20s)5016 --- PASS: FuzzPushGC/seed#2 (3.20s)5017 --- PASS: FuzzPushGC/seed#4 (3.22s)5018 --- PASS: FuzzPushGC/seed#9 (3.27s)5019 --- PASS: FuzzPushGC/seed#5 (3.37s)5020 --- PASS: FuzzPushGC/seed#1 (3.46s)5021 --- PASS: FuzzPushGC/seed#10 (3.62s)5022PASS5023{"timestamp":"2026-11-12T00:20:47.650479081Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:57228","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(394)"}5024{"timestamp":"2026-11-12T00:20:47.650480121Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:55820","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(395)"}5025{"timestamp":"2026-11-12T00:20:47.650649271Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:56144","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(397)"}5026{"timestamp":"2026-11-12T00:20:47.65111601Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:57212","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(398)"}50272026-11-12 00:20:48.047 UTC [130] LOG: received smart shutdown request50282026-11-12 00:20:48.053 UTC [130] LOG: background worker "logical replication launcher" (PID 140) exited with exit code 150292026-11-12 00:20:48.970 UTC [135] LOG: checkpoint complete: wrote 4332 buffers (26.4%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=4.084 s, sync=0.771 s, total=4.922 s; sync files=42363, longest=0.007 s, average=0.001 s; distance=534763 kB, estimate=534763 kB; lsn=0/26D712F0, redo lsn=0/221A09A850302026-11-12 00:20:48.973 UTC [135] LOG: shutting down50312026-11-12 00:20:48.973 UTC [135] LOG: checkpoint starting: shutdown immediate50322026-11-12 00:20:49.980 UTC [135] LOG: checkpoint complete: wrote 9396 buffers (57.3%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.155 s, sync=0.835 s, total=1.007 s; sync files=2832, longest=0.002 s, average=0.001 s; distance=77634 kB, estimate=489050 kB; lsn=0/26D71368, redo lsn=0/26D7136850332026-11-12 00:20:50.079 UTC [130] LOG: database system is shut down5034Running OIDC tests...5035=== RUN TestAudienceForIssuer5036=== PAUSE TestAudienceForIssuer5037=== RUN TestGlobMatch5038=== PAUSE TestGlobMatch5039=== RUN TestValidateToken_ValidToken5040=== PAUSE TestValidateToken_ValidToken5041=== RUN TestValidateToken_WrongAudience5042=== PAUSE TestValidateToken_WrongAudience5043=== RUN TestValidateToken_Expired5044=== PAUSE TestValidateToken_Expired5045=== RUN TestValidateToken_BoundClaimsMismatch5046=== PAUSE TestValidateToken_BoundClaimsMismatch5047=== RUN TestValidateToken_BoundSubjectMismatch5048=== PAUSE TestValidateToken_BoundSubjectMismatch5049=== RUN TestValidateToken_MultipleProviders5050=== PAUSE TestValidateToken_MultipleProviders5051=== RUN TestValidateToken_NoMatchingProvider5052=== PAUSE TestValidateToken_NoMatchingProvider5053=== RUN TestValidateToken_KubernetesServiceAccount5054=== PAUSE TestValidateToken_KubernetesServiceAccount5055=== RUN TestNewValidator_KubernetesRequiresCA5056=== PAUSE TestNewValidator_KubernetesRequiresCA5057=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5058=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5059=== RUN TestPins_ReservedForMatchingRule5060=== PAUSE TestPins_ReservedForMatchingRule5061=== RUN TestPins_TopLevelShorthand5062=== PAUSE TestPins_TopLevelShorthand5063=== RUN TestPins_ConfigValidation5064=== PAUSE TestPins_ConfigValidation5065=== RUN TestScopes_LegacyProviderDefaultsToWrite5066=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5067=== RUN TestScopes_Rules5068=== PAUSE TestScopes_Rules5069=== RUN TestScopes_ConfigValidation5070=== PAUSE TestScopes_ConfigValidation5071=== CONT TestAudienceForIssuer5072=== CONT TestValidateToken_BoundClaimsMismatch5073=== CONT TestValidateToken_KubernetesServiceAccount5074=== CONT TestScopes_ConfigValidation5075=== CONT TestScopes_Rules5076=== CONT TestScopes_LegacyProviderDefaultsToWrite5077=== CONT TestPins_ConfigValidation5078=== CONT TestPins_TopLevelShorthand5079=== CONT TestPins_ReservedForMatchingRule5080=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5081=== CONT TestNewValidator_KubernetesRequiresCA5082--- PASS: TestAudienceForIssuer (0.00s)5083--- PASS: TestScopes_ConfigValidation (0.00s)5084--- PASS: TestPins_ConfigValidation (0.00s)5085=== CONT TestValidateToken_ValidToken5086=== CONT TestValidateToken_MultipleProviders5087=== CONT TestGlobMatch5088=== RUN TestGlobMatch/foo_foo5089=== PAUSE TestGlobMatch/foo_foo5090=== RUN TestGlobMatch/foo_bar5091=== PAUSE TestGlobMatch/foo_bar5092=== RUN TestGlobMatch/*_5093=== PAUSE TestGlobMatch/*_5094=== RUN TestGlobMatch/*_anything5095=== PAUSE TestGlobMatch/*_anything5096=== RUN TestGlobMatch/foo*_foo5097=== PAUSE TestGlobMatch/foo*_foo5098=== RUN TestGlobMatch/foo*_foobar5099=== CONT TestValidateToken_NoMatchingProvider5100=== CONT TestValidateToken_BoundSubjectMismatch5101=== CONT TestValidateToken_Expired5102=== CONT TestValidateToken_WrongAudience5103=== PAUSE TestGlobMatch/foo*_foobar5104=== RUN TestGlobMatch/foo*_bar5105=== PAUSE TestGlobMatch/foo*_bar5106=== RUN TestGlobMatch/*bar_bar5107=== PAUSE TestGlobMatch/*bar_bar5108=== RUN TestGlobMatch/*bar_foobar5109=== PAUSE TestGlobMatch/*bar_foobar5110=== RUN TestGlobMatch/*bar_foo5111=== PAUSE TestGlobMatch/*bar_foo5112=== RUN TestGlobMatch/foo*bar_foobar5113=== PAUSE TestGlobMatch/foo*bar_foobar5114=== RUN TestGlobMatch/foo*bar_foo123bar5115=== PAUSE TestGlobMatch/foo*bar_foo123bar5116=== RUN TestGlobMatch/foo*bar_foobarbaz5117=== PAUSE TestGlobMatch/foo*bar_foobarbaz5118=== RUN TestGlobMatch/*/*_foo/bar5119=== PAUSE TestGlobMatch/*/*_foo/bar5120=== RUN TestGlobMatch/*/*_foo5121=== PAUSE TestGlobMatch/*/*_foo5122=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5123=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5124=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05125=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05126=== RUN TestGlobMatch/refs/*/main_refs/heads/main5127=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5128=== RUN TestGlobMatch/fo?_foo5129=== PAUSE TestGlobMatch/fo?_foo5130=== RUN TestGlobMatch/fo?_fo5131=== PAUSE TestGlobMatch/fo?_fo5132=== RUN TestGlobMatch/fo?_fooo5133=== PAUSE TestGlobMatch/fo?_fooo5134=== RUN TestGlobMatch/?oo_foo5135=== PAUSE TestGlobMatch/?oo_foo5136=== RUN TestGlobMatch/?oo_boo5137=== PAUSE TestGlobMatch/?oo_boo5138=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5139=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5140=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5141=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5142=== CONT TestGlobMatch/foo_foo5143=== CONT TestGlobMatch/*/*_foo5144=== CONT TestGlobMatch/*/*_foo/bar5145=== CONT TestGlobMatch/foo*bar_foobarbaz5146=== CONT TestGlobMatch/foo*bar_foo123bar5147=== CONT TestGlobMatch/foo*bar_foobar5148=== CONT TestGlobMatch/*bar_foo5149=== CONT TestGlobMatch/*bar_foobar5150=== CONT TestGlobMatch/*bar_bar5151=== CONT TestGlobMatch/foo*_bar5152=== CONT TestGlobMatch/foo*_foobar5153=== CONT TestGlobMatch/foo*_foo5154=== CONT TestGlobMatch/*_anything5155=== CONT TestGlobMatch/*_5156=== CONT TestGlobMatch/foo_bar5157=== CONT TestGlobMatch/fo?_fooo5158=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5159=== CONT TestGlobMatch/fo?_fo5160=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5161=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5162=== CONT TestGlobMatch/?oo_boo5163=== CONT TestGlobMatch/?oo_foo5164=== CONT TestGlobMatch/refs/*/main_refs/heads/main5165=== CONT TestGlobMatch/fo?_foo5166=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05167--- PASS: TestGlobMatch (0.00s)5168 --- PASS: TestGlobMatch/foo_foo (0.00s)5169 --- PASS: TestGlobMatch/*/*_foo (0.00s)5170 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5171 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5172 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5173 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5174 --- PASS: TestGlobMatch/*bar_foo (0.00s)5175 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5176 --- PASS: TestGlobMatch/*bar_bar (0.00s)5177 --- PASS: TestGlobMatch/foo*_bar (0.00s)5178 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5179 --- PASS: TestGlobMatch/foo*_foo (0.00s)5180 --- PASS: TestGlobMatch/*_anything (0.00s)5181 --- PASS: TestGlobMatch/*_ (0.00s)5182 --- PASS: TestGlobMatch/foo_bar (0.00s)5183 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5184 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5185 --- PASS: TestGlobMatch/fo?_fo (0.00s)5186 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5187 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5188 --- PASS: TestGlobMatch/?oo_boo (0.00s)5189 --- PASS: TestGlobMatch/?oo_foo (0.00s)5190 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5191 --- PASS: TestGlobMatch/fo?_foo (0.00s)5192 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)51932026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40931/oidc51942026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40613/oidc5195--- PASS: TestPins_TopLevelShorthand (0.02s)5196--- PASS: TestValidateToken_ValidToken (0.02s)51972026/11/12 00:20:52 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:3320551982026/11/12 00:20:52 http: TLS handshake error from 127.0.0.1:32984: remote error: tls: bad certificate5199--- PASS: TestNewValidator_KubernetesRequiresCA (0.03s)5200--- PASS: TestValidateToken_KubernetesServiceAccount (0.03s)52012026/11/12 00:20:52 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352022026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43657/oidc5203--- PASS: TestValidateToken_WrongAudience (0.04s)5204--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.05s)52052026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:39139/oidc52062026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37453/oidc5207--- PASS: TestValidateToken_Expired (0.05s)5208--- PASS: TestScopes_Rules (0.06s)52092026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41407/oidc5210--- PASS: TestPins_ReservedForMatchingRule (0.07s)52112026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45383/oidc5212--- PASS: TestValidateToken_BoundClaimsMismatch (0.09s)52132026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40345/oidc5214--- PASS: TestValidateToken_BoundSubjectMismatch (0.09s)52152026/11/12 00:20:52 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:44727/oidc52162026/11/12 00:20:52 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:44329/oidc5217--- PASS: TestValidateToken_MultipleProviders (0.13s)52182026/11/12 00:20:52 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:33041/oidc5219--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.14s)52202026/11/12 00:20:52 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:40129/oidc5221--- PASS: TestValidateToken_NoMatchingProvider (0.16s)5222PASS5223Running hook tests...5224=== RUN TestSendPathsEmpty5225=== PAUSE TestSendPathsEmpty5226=== RUN TestQueueEnqueueAndFetch5227=== PAUSE TestQueueEnqueueAndFetch5228=== RUN TestQueueDeduplication5229=== PAUSE TestQueueDeduplication5230=== RUN TestQueueRemove5231=== PAUSE TestQueueRemove5232=== RUN TestQueueFetchBatchLimit5233=== PAUSE TestQueueFetchBatchLimit5234=== RUN TestQueueRetryMovesToBack5235=== PAUSE TestQueueRetryMovesToBack5236=== RUN TestQueueFetchRemoveLifecycle5237=== PAUSE TestQueueFetchRemoveLifecycle5238=== RUN TestQueueConcurrentWriters5239=== PAUSE TestQueueConcurrentWriters5240=== RUN TestQueueRemoveLargeClosure5241=== PAUSE TestQueueRemoveLargeClosure5242=== RUN TestServerClientIntegration5243--- PASS: TestServerClientIntegration (0.00s)5244=== RUN TestServerQueueError52452026/11/12 00:20:52 ERROR Failed to queue paths error="permission denied" count=15246--- PASS: TestServerQueueError (0.00s)5247=== RUN TestGetListenerSocketActivation5248 server_test.go:220: === RUN TestGetListenerSocketActivation5249 --- PASS: TestGetListenerSocketActivation (0.00s)5250 PASS5251 5252--- PASS: TestGetListenerSocketActivation (0.01s)5253=== RUN TestServerStalledClientDoesNotBlockShutdown5254=== PAUSE TestServerStalledClientDoesNotBlockShutdown5255=== RUN TestServerBacksOffOnAcceptErrors5256=== PAUSE TestServerBacksOffOnAcceptErrors5257=== RUN TestDrainIsolatesPoisonPath5258=== PAUSE TestDrainIsolatesPoisonPath5259=== RUN TestRunNotBlockedByPoisonHead5260=== PAUSE TestRunNotBlockedByPoisonHead5261=== RUN TestDrainGivesUpWhenServerDown5262=== PAUSE TestDrainGivesUpWhenServerDown5263=== RUN TestFailedPathPrunedByLaterClosure5264=== PAUSE TestFailedPathPrunedByLaterClosure5265=== RUN TestWorkerUploadsAndRemoves5266=== PAUSE TestWorkerUploadsAndRemoves5267=== RUN TestWorkerSkipsGCdPaths5268=== PAUSE TestWorkerSkipsGCdPaths5269=== RUN TestWorkerPrunesClosureDeps5270=== PAUSE TestWorkerPrunesClosureDeps5271=== RUN TestDrainTimeout5272=== PAUSE TestDrainTimeout5273=== RUN TestWorkerRemoveFailureIsNotProgress5274=== PAUSE TestWorkerRemoveFailureIsNotProgress5275=== RUN TestWorkerKeepsPathItCannotStat5276=== PAUSE TestWorkerKeepsPathItCannotStat5277=== CONT TestSendPathsEmpty5278=== CONT TestDrainIsolatesPoisonPath5279--- PASS: TestSendPathsEmpty (0.00s)5280=== CONT TestQueueRetryMovesToBack5281=== CONT TestQueueFetchBatchLimit5282=== CONT TestQueueRemove5283=== CONT TestQueueDeduplication5284=== CONT TestQueueFetchRemoveLifecycle5285=== CONT TestQueueEnqueueAndFetch5286=== CONT TestServerBacksOffOnAcceptErrors5287=== CONT TestWorkerSkipsGCdPaths52882000/01/01 00:00:00 ERROR Accept failed error="too many open files"52892000/01/01 00:00:00 ERROR Accept failed error="too many open files"52902000/01/01 00:00:00 ERROR Accept failed error="too many open files"52912000/01/01 00:00:00 ERROR Accept failed error="too many open files"52922000/01/01 00:00:00 ERROR Accept failed error="too many open files"52932000/01/01 00:00:00 ERROR Accept failed error="too many open files"52942000/01/01 00:00:00 ERROR Accept failed error="too many open files"52952000/01/01 00:00:00 ERROR Accept failed error="too many open files"5296--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5297=== CONT TestWorkerKeepsPathItCannotStat5298=== CONT TestWorkerRemoveFailureIsNotProgress5299=== CONT TestDrainTimeout5300=== CONT TestServerStalledClientDoesNotBlockShutdown5301=== CONT TestQueueRemoveLargeClosure5302=== CONT TestWorkerPrunesClosureDeps5303=== CONT TestQueueConcurrentWriters5304=== CONT TestWorkerUploadsAndRemoves5305=== CONT TestFailedPathPrunedByLaterClosure5306=== CONT TestRunNotBlockedByPoisonHead5307=== CONT TestDrainGivesUpWhenServerDown53082000/01/01 00:00:00 INFO Uploading batch count=253092026/11/12 00:20:52 INFO Uploading batch count=153102026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=153112026/11/12 00:20:52 INFO Upload queue status pending=253122026/11/12 00:20:52 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat4034692678/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat4034692678/002/locked/aaa: permission denied"53132000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253142026/11/12 00:20:52 INFO Uploading batch count=153152026/11/12 00:20:52 INFO Upload queue status pending=253162026/11/12 00:20:52 INFO Uploading batch count=153172000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45318--- PASS: TestQueueFetchBatchLimit (0.02s)53192026/11/12 00:20:52 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths2674694398/002/nonexistent53202026/11/12 00:20:52 INFO Upload queue status pending=253212026/11/12 00:20:52 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat4034692678/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat4034692678/002/locked/aaa: permission denied"53222026/11/12 00:20:52 INFO Uploading batch count=253232026/11/12 00:20:52 INFO Uploading batch count=453242026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=453252026/11/12 00:20:52 INFO Upload queue status pending=353262026/11/12 00:20:52 INFO Uploading batch count=153272026/11/12 00:20:52 INFO Uploading batch count=15328--- PASS: TestQueueEnqueueAndFetch (0.02s)53292026/11/12 00:20:52 INFO Uploading batch count=153302026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=153312026/11/12 00:20:52 INFO Uploading batch count=253322026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=253332026/11/12 00:20:52 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1491878921/002/a53342026/11/12 00:20:52 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1851942618/002/nonexistent53352026/11/12 00:20:52 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153362026/11/12 00:20:52 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath1463882110/002/bbb5337--- PASS: TestQueueRemove (0.02s)53382026/11/12 00:20:52 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat4034692678/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat4034692678/002/locked/aaa: permission denied"53392026/11/12 00:20:52 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1851942618/002/nonexistent53402026/11/12 00:20:52 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153412026/11/12 00:20:52 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1851942618/002/nonexistent53422026/11/12 00:20:52 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1491878921/002/b5343--- PASS: TestQueueRetryMovesToBack (0.02s)5344--- PASS: TestQueueDeduplication (0.02s)53452026/11/12 00:20:52 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15346--- PASS: TestQueueFetchRemoveLifecycle (0.02s)53472026/11/12 00:20:52 ERROR Drain finished with paths left in queue remaining=153482026/11/12 00:20:52 ERROR Drain finished with paths left in queue remaining=15349--- PASS: TestDrainTimeout (0.02s)53502026/11/12 00:20:52 INFO Uploading batch count=25351--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)53522026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=253532026/11/12 00:20:52 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1491878921/002/c53542026/11/12 00:20:52 INFO Uploading batch count=153552026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=15356--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)53572026/11/12 00:20:52 INFO Uploading batch count=153582026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=153592026/11/12 00:20:52 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1491878921/002/d5360--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)53612026/11/12 00:20:52 INFO Uploading batch count=153622026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=153632026/11/12 00:20:52 INFO Uploading batch count=253642026/11/12 00:20:52 ERROR Upload failed error="upload failed" count=253652026/11/12 00:20:52 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1491878921/002/e53662026/11/12 00:20:52 ERROR Drain finished with paths left in queue remaining=153672026/11/12 00:20:52 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1491878921/002/f53682026/11/12 00:20:52 ERROR Drain finished with paths left in queue remaining=105369--- PASS: TestDrainIsolatesPoisonPath (0.03s)5370--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5371--- PASS: TestWorkerPrunesClosureDeps (0.04s)5372--- PASS: TestWorkerSkipsGCdPaths (0.04s)5373--- PASS: TestWorkerUploadsAndRemoves (0.04s)5374--- PASS: TestQueueRemoveLargeClosure (0.07s)53752026/11/12 00:20:52 ERROR Failed to decode request error="read unix /build/hook3684218874/test.sock->@: i/o timeout"53762026/11/12 00:20:52 ERROR Failed to write response error="write unix /build/hook3684218874/test.sock->@: i/o timeout"5377--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)5378--- PASS: TestQueueConcurrentWriters (0.21s)53792026/11/12 00:20:53 INFO Uploading batch count=153802026/11/12 00:20:53 INFO Uploading batch count=153812026/11/12 00:20:53 INFO Uploading batch count=153822026/11/12 00:20:53 ERROR Upload failed error="upload failed" count=153832026/11/12 00:20:53 INFO Uploading batch count=153842026/11/12 00:20:53 ERROR Upload failed error="upload failed" count=153852026/11/12 00:20:53 INFO Uploading batch count=153862026/11/12 00:20:53 ERROR Upload failed error="upload failed" count=153872026/11/12 00:20:53 INFO Uploading batch count=153882026/11/12 00:20:53 ERROR Upload failed error="upload failed" count=153892026/11/12 00:20:53 ERROR Drain finished with paths left in queue remaining=15390--- PASS: TestRunNotBlockedByPoisonHead (1.03s)5391PASS5392Running signing tests...5393=== RUN TestExternalSigner5394=== PAUSE TestExternalSigner5395=== RUN TestExternalSignerWrongPublicKey5396=== PAUSE TestExternalSignerWrongPublicKey5397=== RUN TestExternalSignerErrors5398=== PAUSE TestExternalSignerErrors5399=== RUN TestExternalSignerRestartsAfterCrash5400=== PAUSE TestExternalSignerRestartsAfterCrash5401=== RUN TestExternalSignerProgramErrorKeepsProcess5402=== PAUSE TestExternalSignerProgramErrorKeepsProcess5403=== RUN TestExternalSignerClose5404=== PAUSE TestExternalSignerClose5405=== RUN TestExternalSignerRejectsInvalidPublicKey5406=== PAUSE TestExternalSignerRejectsInvalidPublicKey5407=== RUN TestExternalSignerCancellation5408=== PAUSE TestExternalSignerCancellation5409=== RUN TestGenerateFingerprint5410=== PAUSE TestGenerateFingerprint5411=== RUN TestParseSigningKey5412=== PAUSE TestParseSigningKey5413=== RUN TestSign5414=== PAUSE TestSign5415=== RUN TestSignBatch5416=== PAUSE TestSignBatch5417=== RUN TestNilKeyPublicKey5418=== PAUSE TestNilKeyPublicKey5419=== RUN TestSignErrors5420=== PAUSE TestSignErrors5421=== CONT TestExternalSigner5422=== CONT TestSignBatch5423=== CONT TestExternalSignerCancellation5424=== CONT TestSign5425=== CONT TestParseSigningKey5426=== RUN TestParseSigningKey/valid_32-byte_key5427=== CONT TestGenerateFingerprint5428=== CONT TestExternalSignerRejectsInvalidPublicKey5429=== RUN TestExternalSignerRejectsInvalidPublicKey/#005430=== RUN TestGenerateFingerprint/basic_with_references5431=== PAUSE TestGenerateFingerprint/basic_with_references5432=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005433=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5434=== CONT TestNilKeyPublicKey5435=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5436=== CONT TestExternalSignerClose5437--- PASS: TestNilKeyPublicKey (0.00s)5438=== CONT TestExternalSignerErrors5439=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5440=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5441=== CONT TestExternalSignerRestartsAfterCrash5442=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5443=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5444=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5445=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5446=== CONT TestExternalSignerRejectsInvalidPublicKey/#005447=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5448=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5449=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5450=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5451--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5452 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5453 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5454 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5455 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5456 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5457=== CONT TestExternalSignerWrongPublicKey5458=== CONT TestExternalSignerProgramErrorKeepsProcess5459=== PAUSE TestParseSigningKey/valid_32-byte_key5460=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5461=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5462=== CONT TestSignErrors5463=== RUN TestSignErrors/nil_key5464=== PAUSE TestSignErrors/nil_key5465=== RUN TestSignErrors/nil_narinfo5466=== RUN TestGenerateFingerprint/no_references5467=== RUN TestExternalSignerErrors/null_signatures5468--- PASS: TestSignBatch (0.00s)5469--- PASS: TestSign (0.00s)5470=== PAUSE TestGenerateFingerprint/no_references5471=== PAUSE TestSignErrors/nil_narinfo5472=== PAUSE TestExternalSignerErrors/null_signatures5473=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5474=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5475=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5476=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5477=== RUN TestParseSigningKey/no_colon5478=== PAUSE TestParseSigningKey/no_colon5479=== RUN TestParseSigningKey/empty_name5480=== PAUSE TestParseSigningKey/empty_name5481=== RUN TestParseSigningKey/invalid_base645482=== PAUSE TestParseSigningKey/invalid_base645483=== RUN TestSignErrors/invalid_narinfo5484=== PAUSE TestSignErrors/invalid_narinfo5485=== RUN TestExternalSignerErrors/unrequested_key5486=== PAUSE TestExternalSignerErrors/unrequested_key5487=== RUN TestGenerateFingerprint/invalid_nar_hash_length5488=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5489=== RUN TestExternalSignerErrors/missing_signature_name5490=== PAUSE TestExternalSignerErrors/missing_signature_name5491=== RUN TestParseSigningKey/wrong_length5492=== RUN TestExternalSignerErrors/program_error5493=== RUN TestGenerateFingerprint/invalid_store_path_prefix5494=== PAUSE TestParseSigningKey/wrong_length5495=== PAUSE TestExternalSignerErrors/program_error5496=== CONT TestParseSigningKey/invalid_base645497=== CONT TestParseSigningKey/no_colon5498=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5499=== CONT TestParseSigningKey/empty_name5500=== CONT TestParseSigningKey/valid_32-byte_key5501=== RUN TestExternalSignerErrors/null5502=== PAUSE TestExternalSignerErrors/null5503=== RUN TestExternalSignerErrors/wrong_id5504=== PAUSE TestExternalSignerErrors/wrong_id5505=== RUN TestExternalSignerErrors/wrong_signatures_shape5506=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5507=== RUN TestSignErrors/canceled5508=== PAUSE TestSignErrors/canceled5509=== CONT TestSignErrors/nil_key5510=== CONT TestSignErrors/canceled5511=== CONT TestParseSigningKey/wrong_length5512=== CONT TestSignErrors/invalid_narinfo5513=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5514=== RUN TestGenerateFingerprint/invalid_reference_prefix5515=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5516=== RUN TestExternalSignerErrors/exit_at_startup5517=== PAUSE TestExternalSignerErrors/exit_at_startup5518=== CONT TestSignErrors/nil_narinfo5519--- PASS: TestParseSigningKey (0.00s)5520 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5521 --- PASS: TestParseSigningKey/no_colon (0.00s)5522 --- PASS: TestParseSigningKey/empty_name (0.00s)5523 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5524 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5525 --- PASS: TestParseSigningKey/wrong_length (0.00s)5526--- PASS: TestSignErrors (0.00s)5527 --- PASS: TestSignErrors/nil_key (0.00s)5528 --- PASS: TestSignErrors/canceled (0.00s)5529 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5530 --- PASS: TestSignErrors/nil_narinfo (0.00s)5531=== CONT TestGenerateFingerprint/basic_with_references5532=== CONT TestGenerateFingerprint/invalid_reference_prefix5533=== CONT TestGenerateFingerprint/invalid_store_path_prefix5534=== CONT TestGenerateFingerprint/invalid_nar_hash_length5535=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5536=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5537=== CONT TestGenerateFingerprint/no_references5538--- PASS: TestGenerateFingerprint (0.00s)5539 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5540 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5541 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5542 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5543 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5544 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5545 --- PASS: TestGenerateFingerprint/no_references (0.00s)5546=== RUN TestExternalSignerErrors/invalid_JSON5547=== PAUSE TestExternalSignerErrors/invalid_JSON5548=== RUN TestExternalSignerErrors/missing_signatures5549=== PAUSE TestExternalSignerErrors/missing_signatures5550=== RUN TestExternalSignerErrors/wrong_signature_name5551=== PAUSE TestExternalSignerErrors/wrong_signature_name5552=== RUN TestExternalSignerErrors/exit_on_request5553=== PAUSE TestExternalSignerErrors/exit_on_request5554=== RUN TestExternalSignerErrors/too_few_signatures5555=== PAUSE TestExternalSignerErrors/too_few_signatures5556--- PASS: TestExternalSignerClose (0.00s)5557=== RUN TestExternalSignerErrors/invalid_signature_base645558=== PAUSE TestExternalSignerErrors/invalid_signature_base645559=== RUN TestExternalSignerErrors/wrong_signature_length5560=== PAUSE TestExternalSignerErrors/wrong_signature_length5561=== RUN TestExternalSignerErrors/wrong_shape5562=== PAUSE TestExternalSignerErrors/wrong_shape5563=== CONT TestExternalSignerErrors/null_signatures5564=== CONT TestExternalSignerErrors/exit_on_request5565=== CONT TestExternalSignerErrors/missing_signature_name5566=== CONT TestExternalSignerErrors/wrong_id5567=== CONT TestExternalSignerErrors/wrong_shape5568=== CONT TestExternalSignerErrors/wrong_signature_length5569=== CONT TestExternalSignerErrors/invalid_signature_base645570=== CONT TestExternalSignerErrors/too_few_signatures5571=== CONT TestExternalSignerErrors/invalid_JSON5572=== CONT TestExternalSignerErrors/exit_at_startup5573=== CONT TestExternalSignerErrors/wrong_signatures_shape5574=== CONT TestExternalSignerErrors/program_error5575=== CONT TestExternalSignerErrors/null5576=== CONT TestExternalSignerErrors/wrong_signature_name5577=== CONT TestExternalSignerErrors/unrequested_key5578=== CONT TestExternalSignerErrors/missing_signatures5579--- PASS: TestExternalSignerWrongPublicKey (0.01s)5580--- PASS: TestExternalSigner (0.01s)5581--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5582--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5583--- PASS: TestExternalSignerErrors (0.00s)5584 --- PASS: TestExternalSignerErrors/exit_on_request (0.00s)5585 --- PASS: TestExternalSignerErrors/missing_signature_name (0.00s)5586 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5587 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5588 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5589 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5590 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5591 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5592 --- PASS: TestExternalSignerErrors/null (0.00s)5593 --- PASS: TestExternalSignerErrors/unrequested_key (0.00s)5594 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5595 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5596 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5597 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5598 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5599 --- PASS: TestExternalSignerErrors/program_error (0.01s)5600--- PASS: TestExternalSignerCancellation (0.02s)5601PASS5602Running OpenBao signer tests...5603=== RUN TestSign5604=== PAUSE TestSign5605=== RUN TestSignErrors5606=== PAUSE TestSignErrors5607=== RUN TestServe5608=== PAUSE TestServe5609=== RUN TestServeRejectsGarbage5610=== PAUSE TestServeRejectsGarbage5611=== CONT TestSign5612=== CONT TestServe5613=== CONT TestServeRejectsGarbage5614--- PASS: TestServeRejectsGarbage (0.00s)5615=== CONT TestSignErrors5616=== RUN TestSignErrors/http_error5617=== PAUSE TestSignErrors/http_error5618=== RUN TestSignErrors/invalid_json5619=== PAUSE TestSignErrors/invalid_json5620=== RUN TestSignErrors/no_prefix5621=== PAUSE TestSignErrors/no_prefix5622=== RUN TestSignErrors/invalid_base645623=== PAUSE TestSignErrors/invalid_base645624=== CONT TestSignErrors/http_error5625=== CONT TestSignErrors/no_prefix5626--- PASS: TestServe (0.00s)5627=== CONT TestSignErrors/invalid_base645628=== CONT TestSignErrors/invalid_json5629--- PASS: TestSign (0.00s)5630--- PASS: TestSignErrors (0.00s)5631 --- PASS: TestSignErrors/no_prefix (0.00s)5632 --- PASS: TestSignErrors/http_error (0.00s)5633 --- PASS: TestSignErrors/invalid_base64 (0.00s)5634 --- PASS: TestSignErrors/invalid_json (0.00s)5635PASS