niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #497
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestUploadMultipart_FailedProducerIsNotCompleted22=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted23=== RUN TestPartSizeForNAR24=== PAUSE TestPartSizeForNAR25=== RUN TestDumpPathCaseHackMatchesNix26--- PASS: TestDumpPathCaseHackMatchesNix (0.08s)27=== RUN TestDumpPathCaseHackCollision28--- PASS: TestDumpPathCaseHackCollision (0.00s)29=== RUN TestDumpPathMatchesNix30=== PAUSE TestDumpPathMatchesNix31=== RUN TestDumpPathSingleFile32=== PAUSE TestDumpPathSingleFile33=== RUN TestDumpPathWriterError34=== PAUSE TestDumpPathWriterError35=== RUN TestEncodeNixBase3236=== PAUSE TestEncodeNixBase3237=== RUN TestEncodeNixBase32WithRealHash38=== PAUSE TestEncodeNixBase32WithRealHash39=== RUN TestConvertHashToNix3240=== PAUSE TestConvertHashToNix3241=== RUN TestGetStorePathHash42=== PAUSE TestGetStorePathHash43=== RUN TestPathInfoHashCompatibility44=== PAUSE TestPathInfoHashCompatibility45=== RUN TestParsePathInfoJSON46=== PAUSE TestParsePathInfoJSON47=== RUN TestParsePathInfoJSONMultiplePaths48=== PAUSE TestParsePathInfoJSONMultiplePaths49=== RUN TestPathInfoCACompatibility50=== PAUSE TestPathInfoCACompatibility51=== RUN TestRateLimiterFeedback52=== PAUSE TestRateLimiterFeedback53=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess54=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess55=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer56=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer57=== RUN TestRegisterUploadedObject_OutlivesPush58=== PAUSE TestRegisterUploadedObject_OutlivesPush59=== RUN TestResolveStorePath60=== PAUSE TestResolveStorePath61=== RUN TestDoWithRetry_BodyReplayedViaGetBody62=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody63=== RUN TestRetryTimeoutOutlastsMaxRetries64=== PAUSE TestRetryTimeoutOutlastsMaxRetries65=== RUN TestDoWithRetry_FinalResponseBodyReadable66=== PAUSE TestDoWithRetry_FinalResponseBodyReadable67=== RUN TestShellSplit68=== PAUSE TestShellSplit69=== RUN TestShellSplitErrors70=== PAUSE TestShellSplitErrors71=== RUN TestStreamPushReportsEveryPath72=== PAUSE TestStreamPushReportsEveryPath73=== RUN TestStreamPushBatchesUnderLoad74=== PAUSE TestStreamPushBatchesUnderLoad75=== RUN TestStreamPushIsolatesFailures76=== PAUSE TestStreamPushIsolatesFailures77=== RUN TestStreamPushGivesUpOnDeadServer78=== PAUSE TestStreamPushGivesUpOnDeadServer79=== RUN TestStreamPushRequestLine80=== PAUSE TestStreamPushRequestLine81=== RUN TestStreamPushReportsSignatures82=== PAUSE TestStreamPushReportsSignatures83=== RUN TestStreamPushReportsSkippedPaths84=== PAUSE TestStreamPushReportsSkippedPaths85=== RUN TestClientSignaturesByStorePath86=== PAUSE TestClientSignaturesByStorePath87=== RUN TestStreamPushAnswersEveryLineWhenCanceled88=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled89=== RUN TestStreamPushStopsWhenCanceled90=== PAUSE TestStreamPushStopsWhenCanceled91=== RUN TestSetClientTLS92=== PAUSE TestSetClientTLS93=== RUN TestSetClientTLSDoesNotMutateDefaultTransport94=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport95=== RUN TestSetClientTLSErrors96=== PAUSE TestSetClientTLSErrors97=== RUN TestStaticToken98=== PAUSE TestStaticToken99=== RUN TestFileTokenReadsAndCaches100=== PAUSE TestFileTokenReadsAndCaches101=== RUN TestFileTokenMissing102=== PAUSE TestFileTokenMissing103=== RUN TestFileTokenEmpty104=== PAUSE TestFileTokenEmpty105=== RUN TestScriptTokenNoExpiryRerunsEveryCall106=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall107=== RUN TestScriptTokenCachesUntilRefresh108=== PAUSE TestScriptTokenCachesUntilRefresh109=== RUN TestScriptTokenEmptyToken110=== PAUSE TestScriptTokenEmptyToken111=== RUN TestScriptTokenBadJSON112=== PAUSE TestScriptTokenBadJSON113=== RUN TestScriptTokenScriptFails114=== PAUSE TestScriptTokenScriptFails115=== RUN TestScriptTokenEmptyCommand116=== PAUSE TestScriptTokenEmptyCommand117=== CONT TestDoServerRequestAttachesToken118=== CONT TestRetryTimeoutOutlastsMaxRetries119=== CONT TestEncodeNixBase32120=== RUN TestEncodeNixBase32/test_string_hash121=== PAUSE TestEncodeNixBase32/test_string_hash122=== CONT TestParsePathInfoJSONMultiplePaths123=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths124=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths125=== CONT TestParsePathInfoJSON126=== RUN TestParsePathInfoJSON/Nix_format127=== CONT TestPathInfoHashCompatibility128=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)129=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)130=== CONT TestGetStorePathHash131=== RUN TestGetStorePathHash/valid_store_path132=== PAUSE TestGetStorePathHash/valid_store_path133=== RUN TestGetStorePathHash/basename_without_hyphen_should_error134=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error135=== CONT TestConvertHashToNix32136=== CONT TestEncodeNixBase32WithRealHash137--- PASS: TestEncodeNixBase32WithRealHash (0.00s)138=== CONT TestRegisterUploadedObject_OutlivesPush139=== CONT TestDoWithRetry_BodyReplayedViaGetBody140=== CONT TestResolveStorePath1412026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:650571422026/11/12 00:21:06 WARN Rate limiter enabled after throttle name=server-test rate=51432026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:65061144=== CONT TestSetClientTLS145=== CONT TestScriptTokenEmptyCommand146=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer147=== CONT TestScriptTokenScriptFails1482026/11/12 00:21:06 WARN Rate limiter backed off name=server-test rate=5149=== CONT TestScriptTokenBadJSON1502026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:650611512026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:65057152=== CONT TestScriptTokenEmptyToken153=== CONT TestScriptTokenCachesUntilRefresh154=== CONT TestScriptTokenNoExpiryRerunsEveryCall1552026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:65057156=== CONT TestFileTokenEmpty157=== CONT TestFileTokenMissing1582026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:65057159=== CONT TestFileTokenReadsAndCaches160=== CONT TestStaticToken161=== CONT TestRateLimiterFeedback162=== CONT TestPathInfoCACompatibility163=== RUN TestEncodeNixBase32/empty_input164=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths165=== PAUSE TestParsePathInfoJSON/Nix_format166=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon167=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error168=== CONT TestSetClientTLSErrors169=== RUN TestConvertHashToNix32/SRI_format_to_Nix32170--- PASS: TestDoServerRequestAttachesToken (0.00s)171=== CONT TestSetClientTLSDoesNotMutateDefaultTransport172=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess173=== CONT TestStreamPushGivesUpOnDeadServer174=== CONT TestStreamPushStopsWhenCanceled175=== RUN TestRateLimiterFeedback/429_enables_limiter176=== RUN TestPathInfoCACompatibility/null_ca_field177=== PAUSE TestPathInfoCACompatibility/null_ca_field178=== RUN TestPathInfoCACompatibility/old_string_format_-_text179=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text1802026/11/12 00:21:06 ERROR Upload failed error="connection refused" count=201812026/11/12 00:21:06 ERROR Server seems unavailable, giving up on batch untried=171822026/11/12 00:21:06 WARN Rate limiter enabled after throttle name=server-test rate=5183=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32184=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon185=== RUN TestConvertHashToNix32/already_Nix32_format186=== PAUSE TestConvertHashToNix32/already_Nix32_format187=== RUN TestConvertHashToNix32/invalid_format188=== CONT TestStreamPushRequestLine189=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI190=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI191=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512192=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512193=== PAUSE TestRateLimiterFeedback/429_enables_limiter194=== RUN TestParsePathInfoJSON/Lix_format195=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error196=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths1972026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:65057198=== CONT TestStreamPushBatchesUnderLoad199=== CONT TestStreamPushAnswersEveryLineWhenCanceled200=== CONT TestClientSignaturesByStorePath201=== PAUSE TestEncodeNixBase32/empty_input202--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)203=== CONT TestStreamPushReportsSignatures204=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive205=== RUN TestSetClientTLSErrors/missing_cert_file206=== PAUSE TestConvertHashToNix32/invalid_format207=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error208=== PAUSE TestParsePathInfoJSON/Lix_format209=== CONT TestStreamPushReportsSkippedPaths210=== CONT TestStreamPushReportsEveryPath211=== CONT TestStreamPushIsolatesFailures212=== RUN TestRateLimiterFeedback/503_enables_limiter213=== PAUSE TestRateLimiterFeedback/503_enables_limiter214=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter215=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter216=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter217=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter218=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false219=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false220=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true221=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true222=== CONT TestDumpPathSingleFile223--- PASS: TestScriptTokenEmptyCommand (0.00s)224--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)2252026/11/12 00:21:06 ERROR Upload failed error=boom count=12262026/11/12 00:21:06 ERROR Upload failed error=boom count=1227--- PASS: TestScriptTokenScriptFails (0.01s)228--- PASS: TestStreamPushReportsSignatures (0.00s)229=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error230=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive231=== PAUSE TestSetClientTLSErrors/missing_cert_file232=== CONT TestShellSplitErrors233=== RUN TestParsePathInfoJSON/empty_input234=== CONT TestShellSplit235=== CONT TestDumpPathMatchesNix236=== CONT TestUploadMultipart_PartsInParallel237=== CONT TestDumpPathWriterError238=== CONT TestUploadMultipart_ReturnsAfterDroppedBody239=== RUN TestSetClientTLS/rejects_connection_without_client_cert240=== RUN TestPathInfoCACompatibility/new_structured_format_-_text241=== RUN TestSetClientTLSErrors/missing_key_file2422000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2243=== PAUSE TestSetClientTLSErrors/missing_key_file244=== RUN TestSetClientTLSErrors/missing_ca_file245=== PAUSE TestSetClientTLSErrors/missing_ca_file246=== RUN TestSetClientTLSErrors/invalid_ca_file247=== PAUSE TestSetClientTLSErrors/invalid_ca_file248=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun249=== CONT TestPartSizeForNAR250=== RUN TestPartSizeForNAR/zero_stays_at_minimum251=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum252=== RUN TestPartSizeForNAR/small_stays_at_minimum253=== PAUSE TestPartSizeForNAR/small_stays_at_minimum2542026/11/12 00:21:06 ERROR Upload failed error="bad path" count=3255=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum2562026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:65057257=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text258=== CONT TestFilterOversizedClosures259=== CONT TestCaseHackSuffix260=== PAUSE TestParsePathInfoJSON/empty_input261--- PASS: TestStaticToken (0.00s)262=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert263=== CONT TestUploadMultipart_FailedProducerIsNotCompleted264=== CONT TestRegisterUploadedObjectReusesConnections265=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method266=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA267=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA268--- PASS: TestStreamPushStopsWhenCanceled (0.00s)269--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)270--- PASS: TestFileTokenEmpty (0.01s)271--- PASS: TestFileTokenReadsAndCaches (0.00s)272--- PASS: TestFileTokenMissing (0.01s)273--- PASS: TestResolveStorePath (0.02s)274--- PASS: TestClientSignaturesByStorePath (0.00s)275--- PASS: TestStreamPushReportsSkippedPaths (0.00s)276--- PASS: TestStreamPushReportsEveryPath (0.00s)277--- PASS: TestShellSplitErrors (0.00s)278--- PASS: TestShellSplit (0.00s)279--- PASS: TestStreamPushIsolatesFailures (0.00s)280--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)281=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica282=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry2832026/11/12 00:21:06 INFO Garbage collection started2842026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:650572852026/11/12 00:21:06 INFO Garbage collection started286=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method287=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part288=== RUN TestParsePathInfoJSON/whitespace_only289=== RUN TestSetClientTLS/preserves_debug_logging_transport290=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum291--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.01s)292--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.01s)293--- PASS: TestScriptTokenEmptyToken (0.03s)294--- PASS: TestScriptTokenBadJSON (0.03s)295=== RUN TestFilterOversizedClosures/no_limit_keeps_everything296=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything297=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped298=== CONT TestDoWithRetry_FinalResponseBodyReadable299=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)300=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI301=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512302=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon303=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts304=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts305=== RUN TestPartSizeForNAR/1_TiB306=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths307=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped308=== RUN TestFilterOversizedClosures/all_closures_skipped309=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths310=== CONT TestRateLimiterFeedback/429_enables_limiter311=== PAUSE TestFilterOversizedClosures/all_closures_skipped312=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false313=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part314=== RUN TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary315=== PAUSE TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary316=== CONT TestEncodeNixBase32/test_string_hash317=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter318=== CONT TestConvertHashToNix32/already_Nix32_format319=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter320=== PAUSE TestSetClientTLS/preserves_debug_logging_transport321=== PAUSE TestParsePathInfoJSON/whitespace_only322=== RUN TestParsePathInfoJSON/invalid_JSON323=== PAUSE TestParsePathInfoJSON/invalid_JSON324=== CONT TestSetClientTLSErrors/missing_cert_file325=== CONT TestGetStorePathHash/valid_store_path326=== CONT TestSetClientTLSErrors/missing_ca_file327=== CONT TestSetClientTLSErrors/invalid_ca_file328=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error329=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error330=== CONT TestGetStorePathHash/basename_without_hyphen_should_error331=== CONT TestSetClientTLSErrors/missing_key_file332=== CONT TestPathInfoCACompatibility/null_ca_field333=== CONT TestPathInfoCACompatibility/new_structured_format_-_text334=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive335=== CONT TestPathInfoCACompatibility/old_string_format_-_text336=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method337=== CONT TestFilterOversizedClosures/no_limit_keeps_everything338=== CONT TestFilterOversizedClosures/all_closures_skipped3392026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:650573402026/11/12 00:21:06 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=03412026/11/12 00:21:06 WARN Rate limiter enabled after throttle name=server-test rate=53422026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:651463432026/11/12 00:21:06 WARN Rate limiter enabled after throttle name=s3-test rate=53442026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:65143345=== CONT TestEncodeNixBase32/empty_input3462026/11/12 00:21:06 WARN Rate limiter enabled after throttle name=server-test rate=5347=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3482026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:651403492026/11/12 00:21:06 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50350--- PASS: TestParsePathInfoJSONMultiplePaths (0.02s)351 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)352 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)353--- PASS: TestPathInfoHashCompatibility (0.02s)354 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)355 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)356 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)357 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)358=== CONT TestConvertHashToNix32/invalid_format359=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary360--- PASS: TestEncodeNixBase32 (0.02s)361 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)362 --- PASS: TestEncodeNixBase32/empty_input (0.00s)363--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.02s)364--- PASS: TestStreamPushRequestLine (0.03s)365=== CONT TestConvertHashToNix32/SRI_format_to_Nix323662026/11/12 00:21:06 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=20003672026/11/12 00:21:06 WARN Rate limiter backed off name=server-test rate=53682026/11/12 00:21:06 WARN Rate limiter backed off name=server-test rate=5369--- PASS: TestGetStorePathHash (0.03s)370 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)371 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)372 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)373 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)374--- PASS: TestConvertHashToNix32 (0.02s)375 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)376 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)377 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)378=== CONT TestSetClientTLS/rejects_connection_without_client_cert379=== CONT TestRateLimiterFeedback/503_enables_limiter380--- PASS: TestPathInfoCACompatibility (0.02s)381 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)382 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)383 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)384 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)385 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)386--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)387--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)388=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true389=== CONT TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part390--- PASS: TestStreamPushBatchesUnderLoad (0.00s)391 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)392 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)393=== CONT TestParsePathInfoJSON/whitespace_only394=== CONT TestParsePathInfoJSON/invalid_JSON395=== CONT TestParsePathInfoJSON/empty_input396=== CONT TestParsePathInfoJSON/Lix_format397=== PAUSE TestPartSizeForNAR/1_TiB398--- PASS: TestFilterOversizedClosures (0.01s)399 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)400 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)401 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)402=== CONT TestSetClientTLS/preserves_debug_logging_transport403=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA404=== CONT TestParsePathInfoJSON/Nix_format405--- PASS: TestParsePathInfoJSON (0.05s)406 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)407 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)408 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)409 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)410 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)411=== RUN TestPartSizeForNAR/5_TiB_S3_max_object412=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object413=== RUN TestPartSizeForNAR/capped_at_5_GiB414=== PAUSE TestPartSizeForNAR/capped_at_5_GiB415=== CONT TestPartSizeForNAR/zero_stays_at_minimum416=== CONT TestPartSizeForNAR/capped_at_5_GiB417=== CONT TestPartSizeForNAR/5_TiB_S3_max_object418=== CONT TestPartSizeForNAR/1_TiB419=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts420=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum421=== CONT TestPartSizeForNAR/small_stays_at_minimum422--- PASS: TestPartSizeForNAR (0.03s)423 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)424 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)425 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)426 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)427 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)428 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)429 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)4302026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:650574312026/11/12 00:21:06 WARN Rate limiter enabled after throttle name=server-test rate=54322026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:651604332026/11/12 00:21:06 WARN Garbage collection finished on another replica. Its statistics are not available here434--- PASS: TestSetClientTLSErrors (0.01s)435 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)436 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)437 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)438 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.01s)439--- PASS: TestUploadMultipart_FailedProducerIsNotCompleted (0.02s)440 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/on_a_part_boundary (0.00s)441 --- PASS: TestUploadMultipart_FailedProducerIsNotCompleted/inside_a_part (0.00s)442--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.03s)4432026/11/12 00:21:06 WARN Rate limiter backed off name=server-test rate=54442026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:65057445--- PASS: TestRateLimiterFeedback (0.01s)446 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)447 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)448 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.01s)449 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)4502026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:650574512026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:650574522026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:65057453--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)4542026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:650574552026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:650574562026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:650574572026/11/12 00:21:06 http: TLS handshake error from 127.0.0.1:65159: remote error: tls: bad certificate4582026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:65057459--- PASS: TestSetClientTLS (0.04s)460 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)461 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)462 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4632026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:650574642026/11/12 00:21:06 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:65057465--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.09s)466--- PASS: TestScriptTokenCachesUntilRefresh (0.09s)4672026/11/12 00:21:06 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:65064/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:65064/api/uploads/complete4682026/11/12 00:21:06 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4692026/11/12 00:21:06 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:65064/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:65064/api/uploads/complete4702026/11/12 00:21:06 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4712026/11/12 00:21:06 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:65064/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:65064/api/uploads/complete4722026/11/12 00:21:06 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"473--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.10s)474--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.11s)475--- PASS: TestDumpPathSingleFile (0.10s)476--- PASS: TestCaseHackSuffix (0.09s)477--- PASS: TestDumpPathWriterError (0.16s)478--- PASS: TestDumpPathMatchesNix (0.18s)479--- PASS: TestUploadMultipart_PartsInParallel (0.62s)480--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)481PASS482Running server tests...483The files belonging to this database system will be owned by user "_tribuchetbld1".484This user must also own the server process.485486The database cluster will be initialized with locale "C".487The default database encoding has accordingly been set to "SQL_ASCII".488The default text search configuration will be set to "english".489490Data page checksums are enabled.491492creating directory /private/var/tb/1/scratch/d3a820deccb6fcfd/build/postgres3198171574/data ... ok493creating subdirectories ... ok494selecting dynamic shared memory implementation ... posix495selecting default "max_connections" ... 100496selecting default "shared_buffers" ... 128MB497selecting default time zone ... UTC498creating configuration files ... ok499running bootstrap script ... ok500performing post-bootstrap initialization ... ok501syncing data to disk ... ok502503initdb: warning: enabling "trust" authentication for local connections504initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.505506Success. You can now start the database server using:507508 pg_ctl -D /private/var/tb/1/scratch/d3a820deccb6fcfd/build/postgres3198171574/data -l logfile start509510/private/var/tb/1/scratch/d3a820deccb6fcfd/build/postgres3198171574:5432 - no response5112026-11-12 00:21:08.912 UTC [58585] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5122026-11-12 00:21:08.912 UTC [58585] LOG: listening on Unix socket "/private/var/tb/1/scratch/d3a820deccb6fcfd/build/postgres3198171574/.s.PGSQL.5432"5132026-11-12 00:21:08.916 UTC [58592] LOG: database system was shut down at 2026-11-12 00:21:08 UTC5142026-11-12 00:21:08.918 UTC [58585] LOG: database system is ready to accept connections515/private/var/tb/1/scratch/d3a820deccb6fcfd/build/postgres3198171574:5432 - accepting connections516{"timestamp":"2026-11-12T00:21:09.143068Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"2ae035a2-9d17-4278-9043-1c45d4480132","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}517=== RUN TestAbortSucceeded518=== PAUSE TestAbortSucceeded519=== RUN TestService_AuthMiddleware520=== PAUSE TestService_AuthMiddleware521=== RUN TestService_AuthMiddleware_MTLSProxyHeader522=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader523=== RUN TestService_AuthMiddleware_MTLSBoundSubjects524=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects525=== RUN TestService_ReadAuthMiddleware526=== PAUSE TestService_ReadAuthMiddleware527=== RUN TestService_AuthMiddleware_OIDC528=== PAUSE TestService_AuthMiddleware_OIDC529=== RUN TestService_RequireScope_OIDC530=== PAUSE TestService_RequireScope_OIDC531=== RUN TestService_ReadScope_PublicByDefault532=== PAUSE TestService_ReadScope_PublicByDefault533=== RUN TestCacheConfigHandler534=== PAUSE TestCacheConfigHandler535=== RUN TestCacheStatsHandler536=== PAUSE TestCacheStatsHandler537=== RUN TestClientCADerivations538=== PAUSE TestClientCADerivations539=== RUN TestClientErrorHandling540=== PAUSE TestClientErrorHandling541=== RUN TestClientIntegration542=== PAUSE TestClientIntegration543=== RUN TestClientMultipleUploads544=== PAUSE TestClientMultipleUploads545=== RUN TestClientWithDependencies546=== PAUSE TestClientWithDependencies547=== RUN TestClientSharedPathCommittedMidPush548=== PAUSE TestClientSharedPathCommittedMidPush549=== RUN TestPinProtectsFromGC550=== PAUSE TestPinProtectsFromGC551=== RUN TestClientPushesUseOnePush552=== PAUSE TestClientPushesUseOnePush553=== RUN TestClientFallsBackToClosures554=== PAUSE TestClientFallsBackToClosures555=== RUN TestDeleteClosuresRacingPins556=== PAUSE TestDeleteClosuresRacingPins557=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock558=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock559=== RUN TestResolveDBConnectionString560=== PAUSE TestResolveDBConnectionString561=== RUN TestConnectWaitsForAPeerMigration562=== PAUSE TestConnectWaitsForAPeerMigration563=== RUN TestLeadElectsOneAndHandsOver564=== PAUSE TestLeadElectsOneAndHandsOver565=== RUN TestLeadIncumbentWinsAfterRestart5662026/11/12 00:21:10 INFO lead: acquired remote=192.0.2.1:1234 generation=15672026/11/12 00:21:10 INFO lead: released remote=192.0.2.1:1234 generation=15682026/11/12 00:21:10 INFO lead: acquired remote=192.0.2.1:1234 generation=25692026/11/12 00:21:11 INFO lead: released remote=192.0.2.1:1234 generation=2570--- PASS: TestLeadIncumbentWinsAfterRestart (2.57s)571=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent572=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent573=== RUN TestLeadIncumbentAnnouncesAtOnce574=== PAUSE TestLeadIncumbentAnnouncesAtOnce575=== RUN TestLeadGenerationGrowsWithEveryGrant576=== PAUSE TestLeadGenerationGrowsWithEveryGrant577=== RUN TestLeadEndsOnShutdown578=== PAUSE TestLeadEndsOnShutdown579=== RUN TestLeadEndsWhenItsConnectionHangs5802026/11/12 00:21:12 INFO lead: acquired remote=192.0.2.1:1234 generation=15812026-11-12 00:21:13.462 UTC [58699] FATAL: terminating connection due to administrator command5822026/11/12 00:21:13 INFO lead: acquired remote=192.0.2.1:1234 generation=25832026/11/12 00:21:14 WARN lead: lock connection lost error="timeout: context deadline exceeded"5842026/11/12 00:21:14 INFO lead: released remote=192.0.2.1:1234 generation=15852026/11/12 00:21:15 INFO lead: released remote=192.0.2.1:1234 generation=2586--- PASS: TestLeadEndsWhenItsConnectionHangs (3.25s)587=== RUN TestGCAdvisoryLockBlocksConcurrentRun588=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun589=== RUN TestGCBugBareHashReferences590=== PAUSE TestGCBugBareHashReferences591=== RUN TestGCMetrics592=== PAUSE TestGCMetrics593=== RUN TestPushDedupSurvivesConcurrentGC594=== PAUSE TestPushDedupSurvivesConcurrentGC595=== RUN TestDeduplicatedObjectsRecordedAsPending596=== PAUSE TestDeduplicatedObjectsRecordedAsPending597=== RUN TestGCSweepSkipsPendingObjects598=== PAUSE TestGCSweepSkipsPendingObjects599=== RUN TestTombstonedObjectOfferedWithoutWaiting600=== PAUSE TestTombstonedObjectOfferedWithoutWaiting601=== RUN TestGCSweepDeliversEachKeyOnce602=== PAUSE TestGCSweepDeliversEachKeyOnce603=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection604=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection605=== RUN TestForceGCDuringPushOffersSweptObject606=== PAUSE TestForceGCDuringPushOffersSweptObject607=== RUN TestFailedS3DeleteKeepsTombstone608=== PAUSE TestFailedS3DeleteKeepsTombstone609=== RUN TestSweepSkipsRowsLockedByPush610=== PAUSE TestSweepSkipsRowsLockedByPush611=== RUN TestRegisterCompletedObjectNeedsPendingRow612=== PAUSE TestRegisterCompletedObjectNeedsPendingRow613=== RUN TestGetExistingObjectsLocksRows614=== PAUSE TestGetExistingObjectsLocksRows615=== RUN TestSweepRechecksPendingAfterLocking616=== PAUSE TestSweepRechecksPendingAfterLocking617=== RUN TestPushWaitsForSweepHoldingRow618=== PAUSE TestPushWaitsForSweepHoldingRow619=== RUN TestCommitRacingPendingCleanupKeepsObjects620=== PAUSE TestCommitRacingPendingCleanupKeepsObjects621=== RUN TestGCEndsOnShutdown622=== PAUSE TestGCEndsOnShutdown623=== RUN TestGCTaskStore_StartNew624=== PAUSE TestGCTaskStore_StartNew625=== RUN TestGCTaskStore_DeduplicateSameParams626=== PAUSE TestGCTaskStore_DeduplicateSameParams627=== RUN TestGCTaskStore_ConflictDifferentParams628=== PAUSE TestGCTaskStore_ConflictDifferentParams629=== RUN TestGCTaskStore_GetEmpty630=== PAUSE TestGCTaskStore_GetEmpty631=== RUN TestGCTaskStore_GetReturnsLatest632=== PAUSE TestGCTaskStore_GetReturnsLatest633=== RUN TestGCTaskStore_CompletedAllowsNewTask634=== PAUSE TestGCTaskStore_CompletedAllowsNewTask635=== RUN TestGCTaskStore_PhaseUpdates636=== PAUSE TestGCTaskStore_PhaseUpdates637=== RUN TestGCTaskStore_Fail638=== PAUSE TestGCTaskStore_Fail639=== RUN TestGracefulShutdownDrainsInflight640=== PAUSE TestGracefulShutdownDrainsInflight641=== RUN TestService_healthCheckHandler642=== PAUSE TestService_healthCheckHandler643=== RUN TestService_readinessHandler644=== PAUSE TestService_readinessHandler645=== RUN TestGenerateLandingPage646=== PAUSE TestGenerateLandingPage647=== RUN TestEnsureAbortUploadsRule648=== PAUSE TestEnsureAbortUploadsRule649=== RUN TestCacheConfigHandlerMaxNarSize650=== PAUSE TestCacheConfigHandlerMaxNarSize651=== RUN TestCreatePendingClosureRejectsOversizedNAR652=== PAUSE TestCreatePendingClosureRejectsOversizedNAR653=== RUN TestNARDeduplicationMetadataUploadBug654=== PAUSE TestNARDeduplicationMetadataUploadBug655=== RUN TestMetricsInventory656=== PAUSE TestMetricsInventory657=== RUN TestMetricsCommit658=== PAUSE TestMetricsCommit659=== RUN TestService_NativeMTLS660=== PAUSE TestService_NativeMTLS661=== RUN TestServerTLSConfig662=== PAUSE TestServerTLSConfig663=== RUN TestMultipartCleanup664=== PAUSE TestMultipartCleanup665=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded666=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded667=== RUN TestPendingCleanupUsesOneCutoff668=== PAUSE TestPendingCleanupUsesOneCutoff669=== RUN TestObjectStatsTrigger670=== PAUSE TestObjectStatsTrigger671=== RUN TestObjectStatsUpdatedOncePerCommit672=== PAUSE TestObjectStatsUpdatedOncePerCommit673=== RUN TestOrphanedObjectsGC674=== PAUSE TestOrphanedObjectsGC675=== RUN TestOrphanedObjectsGCStressTest676=== PAUSE TestOrphanedObjectsGCStressTest677=== RUN TestCreatePin_ReservedPins678=== PAUSE TestCreatePin_ReservedPins679=== RUN TestPresentReportsOnlyClosureRoots680=== PAUSE TestPresentReportsOnlyClosureRoots681=== RUN TestPresentRefreshesOnlyStaleClosures682=== PAUSE TestPresentRefreshesOnlyStaleClosures683=== RUN TestPresentNotReportedWhileGCDeletesClosure684=== PAUSE TestPresentNotReportedWhileGCDeletesClosure685=== RUN TestParseSingleRange686=== PAUSE TestParseSingleRange687=== RUN TestProxyHeadersOnlyTrustedOnSocket6882026/11/12 00:21:15 INFO Starting HTTP server address=127.0.0.1:492156892026/11/12 00:21:15 INFO Starting HTTP server address=proxy.sock6902026/11/12 00:21:15 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6912026/11/12 00:21:15 INFO Shutdown signal received, draining in-flight requests timeout=10s692--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.38s)693=== RUN TestIsValidCachePath694=== PAUSE TestIsValidCachePath695=== RUN TestReadProxyNarinfo696=== PAUSE TestReadProxyNarinfo697=== RUN TestReadProxyNarinfoAlreadyDecompressed698=== PAUSE TestReadProxyNarinfoAlreadyDecompressed699=== RUN TestReadProxyContentNegotiation700=== PAUSE TestReadProxyContentNegotiation701=== RUN TestReadProxyNarStreaming702=== PAUSE TestReadProxyNarStreaming703=== RUN TestReadProxy404704=== PAUSE TestReadProxy404705=== RUN TestReadProxyInvalidPath706=== PAUSE TestReadProxyInvalidPath707=== RUN TestReadProxyHead708=== PAUSE TestReadProxyHead709=== RUN TestReadProxyOutlastsServerWriteTimeout710=== PAUSE TestReadProxyOutlastsServerWriteTimeout711=== RUN TestReadProxyConditionalGet712=== PAUSE TestReadProxyConditionalGet713=== RUN TestReadProxyRootRedirectsToIndexHTML714=== PAUSE TestReadProxyRootRedirectsToIndexHTML715=== RUN TestReadProxyDisabled716=== PAUSE TestReadProxyDisabled717=== RUN TestReadRedirectNar718=== PAUSE TestReadRedirectNar719=== RUN TestReadRedirectKeepsNarinfoProxied720=== PAUSE TestReadRedirectKeepsNarinfoProxied721=== RUN TestReadProxyRangeRequest722=== PAUSE TestReadProxyRangeRequest723=== RUN TestReadRedirectUsesPublicS3URL724=== PAUSE TestReadRedirectUsesPublicS3URL725=== RUN TestPush_OverlappingRootsStoreOneRowPerKey726=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey727=== RUN TestPush_CompleteCommitsEveryRoot728=== PAUSE TestPush_CompleteCommitsEveryRoot729=== RUN TestPush_RepeatedRootCommitsOnce730=== PAUSE TestPush_RepeatedRootCommitsOnce731=== RUN TestPush_CompleteWaitsForCleanupOfThePush732=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush733=== RUN TestPush_RootsAreLockedInByteOrder734=== PAUSE TestPush_RootsAreLockedInByteOrder735=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit736=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit737=== RUN TestPush_RejectsBadRequests738=== PAUSE TestPush_RejectsBadRequests739=== RUN TestPush_SignsNarinfosOfItsPendingObjects740=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects741=== RUN TestSharedNarBothUploadsComplete742=== PAUSE TestSharedNarBothUploadsComplete743=== RUN TestCompleteMultipartUpload_ErrorButObjectExists744=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists745=== RUN TestCompletedNarNotReofferedAcrossClosures746=== PAUSE TestCompletedNarNotReofferedAcrossClosures747=== RUN TestPresignedUploadRegisteredBeforeCommit748=== PAUSE TestPresignedUploadRegisteredBeforeCommit749=== RUN TestService_Rustfstest750=== PAUSE TestService_Rustfstest751=== RUN TestParseSize752=== PAUSE TestParseSize753=== RUN TestSkippedUploadsHandler754=== PAUSE TestSkippedUploadsHandler755=== RUN TestSystemdListenerNotActivated756--- PASS: TestSystemdListenerNotActivated (0.00s)757=== RUN TestWatchdogBeatsWhenHealthy758--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)759=== RUN TestWatchdogSkipsWhenUnhealthy7602026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7662026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7672026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7682026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7692026/11/12 00:21:15 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"770--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)771=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle772=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle773=== RUN TestProxyWriteTimeout774=== PAUSE TestProxyWriteTimeout775=== RUN TestPendingClosureWriteTimeout776=== PAUSE TestPendingClosureWriteTimeout777=== RUN TestIsValidUploadKey778=== PAUSE TestIsValidUploadKey779=== RUN TestUploadHandlersRejectInvalidKeys780=== PAUSE TestUploadHandlersRejectInvalidKeys781=== RUN TestUploadHandlersRejectOversizedBody782=== PAUSE TestUploadHandlersRejectOversizedBody783=== RUN TestService_cleanupPendingClosuresHandler784=== PAUSE TestService_cleanupPendingClosuresHandler785=== RUN TestService_createPendingClosureHandler786=== PAUSE TestService_createPendingClosureHandler787=== RUN TestService_verifyS3Integrity788=== PAUSE TestService_verifyS3Integrity789=== RUN TestService_verifyS3ReuploadsStaleNarinfo790=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo791=== RUN TestCompleteMultipartUnregistered792=== PAUSE TestCompleteMultipartUnregistered793=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT794=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT795=== CONT TestPush_CompleteCommitsEveryRoot796=== CONT TestService_verifyS3Integrity797=== CONT TestReadRedirectNar798=== CONT TestPush_OverlappingRootsStoreOneRowPerKey799=== CONT TestSkippedUploadsHandler800=== CONT TestReadRedirectUsesPublicS3URL801=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT802=== CONT TestReadProxyRangeRequest803=== CONT TestCompleteMultipartUnregistered804=== CONT TestService_verifyS3ReuploadsStaleNarinfo805=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed8062026/11/12 00:21:15 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000807=== CONT TestReadRedirectKeepsNarinfoProxied808=== CONT TestReadProxyDisabled809=== CONT TestReadProxyRootRedirectsToIndexHTML810=== CONT TestReadProxyConditionalGet811=== CONT TestReadProxyOutlastsServerWriteTimeout812=== CONT TestReadProxyHead813=== CONT TestReadProxyInvalidPath814=== CONT TestReadProxy404815=== CONT TestReadProxyNarStreaming816=== CONT TestReadProxyContentNegotiation817=== CONT TestReadProxyNarinfoAlreadyDecompressed818=== CONT TestReadProxyNarinfo819=== CONT TestIsValidCachePath820=== RUN TestIsValidCachePath/narinfo821=== CONT TestAbortSucceeded822=== PAUSE TestIsValidCachePath/narinfo823=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars824=== CONT TestParseSingleRange825=== RUN TestParseSingleRange/none826=== PAUSE TestParseSingleRange/none827=== RUN TestParseSingleRange/unknown_unit828=== PAUSE TestParseSingleRange/unknown_unit829=== RUN TestParseSingleRange/multi-range_ignored830--- PASS: TestAbortSucceeded (0.00s)831=== PAUSE TestParseSingleRange/multi-range_ignored832=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars833=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed834=== RUN TestParseSingleRange/malformed_no_dash835=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed836=== PAUSE TestParseSingleRange/malformed_no_dash837=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed838=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost839--- PASS: TestSkippedUploadsHandler (0.00s)840=== RUN TestIsValidCachePath/nar_zst841=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost842=== RUN TestParseSingleRange/malformed_both_empty843=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed844=== PAUSE TestParseSingleRange/malformed_both_empty845=== PAUSE TestIsValidCachePath/nar_zst846=== RUN TestIsValidCachePath/nar_xz847=== CONT TestPresentNotReportedWhileGCDeletesClosure848=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale849=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale850=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh851=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed852=== RUN TestParseSingleRange/malformed_end_before_start853=== PAUSE TestParseSingleRange/malformed_end_before_start854=== RUN TestParseSingleRange/closed855=== PAUSE TestIsValidCachePath/nar_xz856=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh857=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed858=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed859=== PAUSE TestParseSingleRange/closed860=== CONT TestPresentReportsOnlyClosureRoots861=== RUN TestParseSingleRange/open-ended862=== PAUSE TestParseSingleRange/open-ended863=== RUN TestIsValidCachePath/nar_bz2864=== PAUSE TestIsValidCachePath/nar_bz2865=== RUN TestIsValidCachePath/nar_uncompressed866=== PAUSE TestIsValidCachePath/nar_uncompressed867=== RUN TestIsValidCachePath/ls868=== PAUSE TestIsValidCachePath/ls869=== RUN TestIsValidCachePath/log870=== CONT TestPresentRefreshesOnlyStaleClosures871=== RUN TestParseSingleRange/end_clamped_to_size872=== PAUSE TestParseSingleRange/end_clamped_to_size873=== PAUSE TestIsValidCachePath/log874=== RUN TestParseSingleRange/suffix875=== PAUSE TestParseSingleRange/suffix876=== RUN TestIsValidCachePath/realisation877=== RUN TestParseSingleRange/suffix_exceeds_size878=== PAUSE TestParseSingleRange/suffix_exceeds_size879=== RUN TestParseSingleRange/single_byte880=== PAUSE TestParseSingleRange/single_byte881=== PAUSE TestIsValidCachePath/realisation882=== RUN TestParseSingleRange/start_past_EOF883=== PAUSE TestParseSingleRange/start_past_EOF884=== RUN TestParseSingleRange/start_far_past_EOF885=== PAUSE TestParseSingleRange/start_far_past_EOF886=== RUN TestIsValidCachePath/nix-cache-info887=== CONT TestCreatePin_ReservedPins888=== PAUSE TestIsValidCachePath/nix-cache-info889=== RUN TestIsValidCachePath/index.html890=== PAUSE TestIsValidCachePath/index.html891=== RUN TestIsValidCachePath/traversal_parent892=== PAUSE TestIsValidCachePath/traversal_parent893=== RUN TestIsValidCachePath/traversal_in_middle894=== PAUSE TestIsValidCachePath/traversal_in_middle895=== RUN TestIsValidCachePath/invalid_char_e896=== PAUSE TestIsValidCachePath/invalid_char_e897=== RUN TestIsValidCachePath/invalid_char_u898=== PAUSE TestIsValidCachePath/invalid_char_u899=== RUN TestIsValidCachePath/random_path900=== PAUSE TestIsValidCachePath/random_path901=== RUN TestIsValidCachePath/empty902=== PAUSE TestIsValidCachePath/empty903=== RUN TestIsValidCachePath/leading_slash904=== PAUSE TestIsValidCachePath/leading_slash905=== RUN TestIsValidCachePath/wrong_extension906=== PAUSE TestIsValidCachePath/wrong_extension907=== RUN TestIsValidCachePath/short_hash908=== PAUSE TestIsValidCachePath/short_hash909=== CONT TestOrphanedObjectsGCStressTest9102026/11/12 00:21:15 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49223/oidc9112026/11/12 00:21:18 INFO Received uploads request method=POST path=/api/pending_closures912--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (3.08s)913=== CONT TestOrphanedObjectsGC9142026/11/12 00:21:18 INFO Received uploads request method=POST path=/api/pending_closures9152026/11/12 00:21:19 INFO Received push request method=POST path=/api/pushes9162026/11/12 00:21:19 INFO Received complete push request method=POST path=/api/pushes/1/complete9172026/11/12 00:21:19 INFO Completed push id=1 objects=6 duration=42.877458ms918--- PASS: TestPush_CompleteCommitsEveryRoot (3.65s)919=== CONT TestObjectStatsUpdatedOncePerCommit9202026/11/12 00:21:19 INFO Received push request method=POST path=/api/pushes921--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (3.87s)922=== CONT TestObjectStatsTrigger923--- PASS: TestReadRedirectNar (4.10s)924=== CONT TestPendingCleanupUsesOneCutoff925--- PASS: TestReadProxyRangeRequest (4.39s)926=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded927--- PASS: TestReadProxyNarStreaming (4.65s)928=== CONT TestMultipartCleanup9292026/11/12 00:21:20 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9302026/11/12 00:21:20 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=YTk3NjVkMjgtYWZhMS00OGQ0LTk3NTEtNGIyNjNmZjZiNmRkLjJmY2Y3YmVlLTdhMmQtNGYzZC1hNjljLTExMjllZjlkYzA4NXgxNzk0NDQyODc4OTM1OTE4MDAw parts=109312026/11/12 00:21:20 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9322026/11/12 00:21:20 INFO Completed upload id=1 objects=2 duration=8.884125ms9332026/11/12 00:21:20 INFO Received uploads request method=POST path=/api/pending_closures9342026/11/12 00:21:20 INFO Received uploads request method=POST path=/api/pending_closures9352026/11/12 00:21:20 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo9362026/11/12 00:21:20 WARN Found objects in DB but missing from S3, will re-upload count=1937--- PASS: TestService_verifyS3Integrity (4.77s)938=== CONT TestUploadHandlersRejectInvalidKeys939=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info940=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info941=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal942=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal943=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key944=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key945=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key946=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key947=== CONT TestService_createPendingClosureHandler948--- PASS: TestPresentReportsOnlyClosureRoots (5.09s)949=== CONT TestService_cleanupPendingClosuresHandler950--- PASS: TestReadProxyRootRedirectsToIndexHTML (5.82s)951=== CONT TestUploadHandlersRejectOversizedBody952=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure953=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure954=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart955=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart956=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts957=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts958=== CONT TestPendingClosureWriteTimeout959=== RUN TestPendingClosureWriteTimeout/empty960=== PAUSE TestPendingClosureWriteTimeout/empty961=== RUN TestPendingClosureWriteTimeout/400_objects962=== PAUSE TestPendingClosureWriteTimeout/400_objects963=== RUN TestPendingClosureWriteTimeout/670k_objects964=== PAUSE TestPendingClosureWriteTimeout/670k_objects965=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline966=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline967=== CONT TestIsValidUploadKey968=== RUN TestIsValidUploadKey/narinfo969=== PAUSE TestIsValidUploadKey/narinfo970=== RUN TestIsValidUploadKey/nar_zst971=== PAUSE TestIsValidUploadKey/nar_zst972=== RUN TestIsValidUploadKey/nar_xz973=== PAUSE TestIsValidUploadKey/nar_xz974=== RUN TestIsValidUploadKey/nar_plain975=== PAUSE TestIsValidUploadKey/nar_plain976=== RUN TestIsValidUploadKey/listing977=== PAUSE TestIsValidUploadKey/listing978=== RUN TestIsValidUploadKey/build_log979=== PAUSE TestIsValidUploadKey/build_log980=== RUN TestIsValidUploadKey/build_log_home-manager_file981=== PAUSE TestIsValidUploadKey/build_log_home-manager_file982=== RUN TestIsValidUploadKey/build_log_plus_in_name983=== PAUSE TestIsValidUploadKey/build_log_plus_in_name984=== RUN TestIsValidUploadKey/build_log_question_mark985=== PAUSE TestIsValidUploadKey/build_log_question_mark986=== RUN TestIsValidUploadKey/build_log_equals987=== PAUSE TestIsValidUploadKey/build_log_equals988=== RUN TestIsValidUploadKey/realisation989=== PAUSE TestIsValidUploadKey/realisation990=== RUN TestIsValidUploadKey/realisation_plus_in_output991=== PAUSE TestIsValidUploadKey/realisation_plus_in_output992=== RUN TestIsValidUploadKey/nix-cache-info993=== PAUSE TestIsValidUploadKey/nix-cache-info994=== RUN TestIsValidUploadKey/index.html995=== PAUSE TestIsValidUploadKey/index.html996=== RUN TestIsValidUploadKey/narinfo_key,_nar_type997=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type998=== RUN TestIsValidUploadKey/nar_key,_narinfo_type999=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1000=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1001=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1002=== RUN TestIsValidUploadKey/traversal1003=== PAUSE TestIsValidUploadKey/traversal1004=== RUN TestIsValidUploadKey/traversal_nar1005=== PAUSE TestIsValidUploadKey/traversal_nar1006=== RUN TestIsValidUploadKey/absolute1007=== PAUSE TestIsValidUploadKey/absolute1008=== RUN TestIsValidUploadKey/empty_key1009=== PAUSE TestIsValidUploadKey/empty_key1010=== RUN TestIsValidUploadKey/unknown_type1011=== PAUSE TestIsValidUploadKey/unknown_type1012=== CONT TestGCSweepSkipsPendingObjects1013--- PASS: TestReadProxy404 (5.97s)1014=== CONT TestService_NativeMTLS1015--- PASS: TestReadRedirectUsesPublicS3URL (6.04s)1016=== CONT TestMetricsCommit10172026/11/12 00:21:21 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10182026/11/12 00:21:21 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1019--- PASS: TestCompleteMultipartUnregistered (6.20s)1020=== CONT TestMetricsInventory1021--- PASS: TestReadProxyHead (6.40s)1022=== CONT TestSharedNarBothUploadsComplete1023--- PASS: TestReadProxyNarinfoAlreadyDecompressed (6.55s)1024=== CONT TestParseSize1025--- PASS: TestParseSize (0.00s)1026=== CONT TestService_Rustfstest1027--- PASS: TestReadProxyContentNegotiation (6.77s)1028=== CONT TestPresignedUploadRegisteredBeforeCommit10292026/11/12 00:21:22 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1030--- PASS: TestReadProxyNarinfo (6.91s)1031=== CONT TestCompletedNarNotReofferedAcrossClosures1032--- PASS: TestReadProxyConditionalGet (7.02s)1033=== CONT TestCompleteMultipartUpload_ErrorButObjectExists1034--- PASS: TestReadRedirectKeepsNarinfoProxied (7.20s)1035=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit1036--- PASS: TestReadProxyInvalidPath (7.32s)1037=== CONT TestPush_SignsNarinfosOfItsPendingObjects1038--- PASS: TestReadProxyDisabled (7.49s)1039=== CONT TestPush_RejectsBadRequests10402026/11/12 00:21:23 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10412026/11/12 00:21:23 WARN Refused reserved pin name=worker-x86_64-linux10422026/11/12 00:21:23 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux10432026/11/12 00:21:23 INFO Received create pin request method=POST path=/api/pins/my-app10442026/11/12 00:21:23 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1045--- PASS: TestCreatePin_ReservedPins (7.71s)1046=== CONT TestServerTLSConfig1047=== RUN TestServerTLSConfig/no_client_CA1048=== PAUSE TestServerTLSConfig/no_client_CA1049=== RUN TestServerTLSConfig/missing_CA_file1050=== PAUSE TestServerTLSConfig/missing_CA_file1051=== RUN TestServerTLSConfig/not_a_PEM_file1052=== PAUSE TestServerTLSConfig/not_a_PEM_file1053=== CONT TestGCTaskStore_DeduplicateSameParams1054--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1055=== CONT TestCreatePendingClosureRejectsOversizedNAR10562026/11/12 00:21:23 INFO Received uploads request method=POST path=/api/pending_closures1057--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1058=== CONT TestCacheConfigHandlerMaxNarSize1059--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1060=== CONT TestDeduplicatedObjectsRecordedAsPending1061--- PASS: TestPresentRefreshesOnlyStaleClosures (7.97s)1062=== CONT TestEnsureAbortUploadsRule1063--- PASS: TestObjectStatsTrigger (4.29s)1064=== CONT TestGenerateLandingPage1065--- PASS: TestGenerateLandingPage (0.00s)1066=== CONT TestNARDeduplicationMetadataUploadBug1067--- PASS: TestObjectStatsUpdatedOncePerCommit (5.04s)1068=== CONT TestGCTaskStore_StartNew1069--- PASS: TestGCTaskStore_StartNew (0.00s)1070=== CONT TestGCEndsOnShutdown1071=== RUN TestGCEndsOnShutdown/before_the_run1072=== PAUSE TestGCEndsOnShutdown/before_the_run1073=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1074=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1075=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1076=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1077=== CONT TestService_readinessHandler10782026/11/12 00:21:24 INFO Received uploads request method=POST path=/api/pending_closures10792026/11/12 00:21:24 INFO Received uploads request method=POST path=/api/pending_closures10802026/11/12 00:21:24 INFO Received cleanup request method=DELETE path=/api/pending_closures10812026/11/12 00:21:25 INFO Received uploads request method=POST path=/api/pending_closures1082--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (5.05s)1083=== CONT TestService_healthCheckHandler10842026/11/12 00:21:25 INFO Received uploads request method=POST path=/api/pending_closures1085--- PASS: TestReadProxyOutlastsServerWriteTimeout (9.62s)1086=== CONT TestPushDedupSurvivesConcurrentGC10872026/11/12 00:21:25 INFO Received cleanup request method=DELETE path=/api/pending_closures10882026/11/12 00:21:25 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket33/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""10892026/11/12 00:21:25 INFO Aborted multipart uploads count=0 kept=110902026/11/12 00:21:25 INFO Received cleanup request method=DELETE path=/api/pending_closures10912026/11/12 00:21:25 INFO Aborted multipart uploads count=1 kept=01092--- PASS: TestMultipartCleanup (5.16s)1093=== CONT TestGCMetrics10942026/11/12 00:21:25 INFO Received uploads request method=POST path=/api/pending_closures10952026/11/12 00:21:25 INFO Received uploads request method=POST path=/api/pending_closures10962026/11/12 00:21:25 INFO Received uploads request method=POST path=/api/pending_closures10972026/11/12 00:21:25 INFO Received cleanup request method=DELETE path=/api/pending_closures10982026/11/12 00:21:25 INFO Aborted multipart uploads count=0 kept=010992026/11/12 00:21:25 INFO Received uploads request method=POST path=/api/pending_closures11002026/11/12 00:21:26 INFO Received cleanup request method=DELETE path=/api/pending_closures1101=== NAME TestOrphanedObjectsGC1102 orphaned_objects_gc_test.go:287: GC Test Summary:1103 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1104 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B11052026/11/12 00:21:26 INFO Aborted multipart uploads count=1 kept=01106 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1107 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1108 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1109--- PASS: TestOrphanedObjectsGC (7.33s)1110=== CONT TestGCBugBareHashReferences11112026/11/12 00:21:26 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11122026-11-12 00:21:26.143 UTC [58918] ERROR: Closure does not exist: id=111132026-11-12 00:21:26.143 UTC [58918] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE11142026-11-12 00:21:26.143 UTC [58918] STATEMENT: -- name: CommitPendingClosure :one1115 SELECT commit_pending_closure($1::bigint)::bigint1116 1117--- PASS: TestService_cleanupPendingClosuresHandler (5.36s)1118=== CONT TestGCAdvisoryLockBlocksConcurrentRun11192026/11/12 00:21:26 INFO Received uploads request method=POST path=/api/pending_closures11202026/11/12 00:21:26 INFO Aborted multipart uploads count=0 kept=011212026/11/12 00:21:26 WARN Force mode enabled - objects will be deleted immediately without grace period11222026/11/12 00:21:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011232026/11/12 00:21:26 INFO Vacuumed table table=pending_closures11242026/11/12 00:21:26 INFO Vacuumed table table=pending_objects11252026/11/12 00:21:26 INFO Vacuumed table table=multipart_uploads11262026/11/12 00:21:26 INFO Vacuumed table table=closures11272026/11/12 00:21:26 INFO Vacuumed table table=objects11282026/11/12 00:21:26 WARN mTLS auth: subject not in bound subjects subject="CN=reader"11292026/11/12 00:21:26 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1130--- PASS: TestService_NativeMTLS (5.12s)1131=== CONT TestLeadEndsOnShutdown1132--- PASS: TestGCSweepSkipsPendingObjects (5.20s)1133=== CONT TestLeadGenerationGrowsWithEveryGrant11342026/11/12 00:21:27 INFO Received push request method=POST path=/api/pushes11352026/11/12 00:21:27 INFO Received complete push request method=POST path=/api/pushes/1/complete11362026/11/12 00:21:27 INFO Completed push id=1 objects=4 duration=73.465709ms11372026/11/12 00:21:27 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete11382026/11/12 00:21:27 INFO Completed upload id=2 objects=2 duration=723.25µs1139--- PASS: TestMetricsCommit (5.63s)1140=== CONT TestLeadIncumbentAnnouncesAtOnce1141--- PASS: TestMetricsInventory (5.65s)1142=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent11432026/11/12 00:21:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11442026/11/12 00:21:27 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=YTk3NjVkMjgtYWZhMS00OGQ0LTk3NTEtNGIyNjNmZjZiNmRkLjYyMWQ0NjBjLWIxMzAtNDE5Zi1iNjE4LTk5NWMyYmQ0ODUxMngxNzk0NDQyODg1NjMzMTQ5MDAw parts=1011452026/11/12 00:21:27 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete11462026/11/12 00:21:27 INFO Completed upload id=1 objects=2 duration=8.423542ms11472026/11/12 00:21:27 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000011482026/11/12 00:21:27 INFO Received uploads request method=POST path=/api/pending_closures11492026/11/12 00:21:27 INFO Starting cleanup of old closures method=DELETE path=/api/closures11502026/11/12 00:21:27 INFO Aborted multipart uploads count=0 kept=011512026/11/12 00:21:27 INFO Received uploads request method=POST path=/api/pending_closures11522026/11/12 00:21:27 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011532026/11/12 00:21:27 INFO Vacuumed table table=pending_closures11542026/11/12 00:21:27 INFO Vacuumed table table=pending_objects11552026/11/12 00:21:27 INFO Received uploads request method=POST path=/api/pending_closures11562026/11/12 00:21:27 INFO Vacuumed table table=multipart_uploads11572026/11/12 00:21:27 INFO Vacuumed table table=closures11582026/11/12 00:21:27 INFO Vacuumed table table=objects11592026/11/12 00:21:27 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001160--- PASS: TestService_createPendingClosureHandler (7.40s)1161=== CONT TestLeadElectsOneAndHandsOver1162--- PASS: TestService_Rustfstest (5.65s)1163=== CONT TestConnectWaitsForAPeerMigration11642026/11/12 00:21:28 INFO Received uploads request method=POST path=/api/pending_closures11652026/11/12 00:21:28 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst11662026/11/12 00:21:28 INFO Received uploads request method=POST path=/api/pending_closures1167--- PASS: TestPresignedUploadRegisteredBeforeCommit (5.75s)1168=== CONT TestCommitRacingPendingCleanupKeepsObjects11692026/11/12 00:21:28 INFO Received uploads request method=POST path=/api/pending_closures11702026/11/12 00:21:28 INFO Received uploads request method=POST path=/api/pending_closures11712026/11/12 00:21:28 INFO Received push request method=POST path=/api/pushes11722026/11/12 00:21:28 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11732026/11/12 00:21:28 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YTk3NjVkMjgtYWZhMS00OGQ0LTk3NTEtNGIyNjNmZjZiNmRkLmExODA3ZWYwLTFjZGMtNDQxZi04ODEyLWRlMTZlNzNlMTg0OHgxNzk0NDQyODg4NTgxNzI1MDAw11742026/11/12 00:21:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11752026/11/12 00:21:28 INFO Signed narinfos id=1 count=111762026/11/12 00:21:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11772026/11/12 00:21:28 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YTk3NjVkMjgtYWZhMS00OGQ0LTk3NTEtNGIyNjNmZjZiNmRkLmExODA3ZWYwLTFjZGMtNDQxZi04ODEyLWRlMTZlNzNlMTg0OHgxNzk0NDQyODg4NTgxNzI1MDAw parts=111782026/11/12 00:21:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11792026/11/12 00:21:28 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"1180--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (6.20s)1181=== CONT TestPushWaitsForSweepHoldingRow11822026/11/12 00:21:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11832026/11/12 00:21:28 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"11842026/11/12 00:21:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11852026/11/12 00:21:28 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=111862026/11/12 00:21:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11872026/11/12 00:21:28 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1188--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (5.91s)1189=== CONT TestGracefulShutdownDrainsInflight11902026/11/12 00:21:28 INFO Starting HTTP server address=127.0.0.1:4934311912026/11/12 00:21:28 INFO Shutdown signal received, draining in-flight requests timeout=10s1192--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1193=== CONT TestSweepRechecksPendingAfterLocking11942026/11/12 00:21:29 INFO Received push request method=POST path=/api/pushes11952026/11/12 00:21:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11962026/11/12 00:21:29 INFO Received complete push request method=POST path=/api/pushes/1/complete11972026/11/12 00:21:29 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=YTk3NjVkMjgtYWZhMS00OGQ0LTk3NTEtNGIyNjNmZjZiNmRkLjU5MWYyN2Q3LWYyOTUtNGQ3My1hYmRiLTE5ZWYwODMzYjVkZngxNzk0NDQyODg3NzM0NjUxMDAw parts=1211982026/11/12 00:21:29 INFO Completed push id=1 objects=4 duration=28.814167ms11992026/11/12 00:21:29 INFO Received push request method=POST path=/api/pushes12002026/11/12 00:21:29 INFO Aborted multipart uploads count=0 kept=012012026/11/12 00:21:29 WARN Force mode enabled - objects will be deleted immediately without grace period12022026/11/12 00:21:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=01203=== RUN TestPush_RejectsBadRequests/root_not_in_objects1204=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1205=== RUN TestPush_RejectsBadRequests/no_roots1206=== PAUSE TestPush_RejectsBadRequests/no_roots1207=== RUN TestPush_RejectsBadRequests/no_objects1208=== PAUSE TestPush_RejectsBadRequests/no_objects1209=== RUN TestPush_RejectsBadRequests/bad_root1210=== PAUSE TestPush_RejectsBadRequests/bad_root1211=== CONT TestGCTaskStore_Fail1212--- PASS: TestGCTaskStore_Fail (0.00s)1213=== CONT TestGetExistingObjectsLocksRows12142026/11/12 00:21:29 INFO Vacuumed table table=pending_closures12152026/11/12 00:21:29 INFO Vacuumed table table=pending_objects12162026/11/12 00:21:29 INFO Vacuumed table table=multipart_uploads12172026/11/12 00:21:29 INFO Vacuumed table table=closures12182026/11/12 00:21:29 INFO Vacuumed table table=objects12192026/11/12 00:21:29 INFO Received complete push request method=POST path=/api/pushes/2/complete12202026/11/12 00:21:29 INFO Completed push id=2 objects=4 duration=4.398958ms1221--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (6.78s)1222=== CONT TestGCTaskStore_PhaseUpdates1223--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1224=== CONT TestGCTaskStore_CompletedAllowsNewTask1225--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1226=== CONT TestGCTaskStore_GetReturnsLatest1227--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1228=== CONT TestGCTaskStore_GetEmpty1229--- PASS: TestGCTaskStore_GetEmpty (0.00s)1230=== CONT TestGCTaskStore_ConflictDifferentParams1231--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1232=== CONT TestRegisterCompletedObjectNeedsPendingRow12332026/11/12 00:21:29 INFO Received uploads request method=POST path=/api/pending_closures1234=== NAME TestOrphanedObjectsGCStressTest1235 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains1236--- PASS: TestDeduplicatedObjectsRecordedAsPending (6.40s)1237=== CONT TestProxyWriteTimeout1238=== RUN TestProxyWriteTimeout/narinfo1239=== PAUSE TestProxyWriteTimeout/narinfo1240=== RUN TestProxyWriteTimeout/1_GiB_nar1241=== PAUSE TestProxyWriteTimeout/1_GiB_nar1242=== RUN TestProxyWriteTimeout/10_GiB_nar1243=== PAUSE TestProxyWriteTimeout/10_GiB_nar1244=== RUN TestProxyWriteTimeout/unknown_size1245=== PAUSE TestProxyWriteTimeout/unknown_size1246=== CONT TestSweepSkipsRowsLockedByPush1247=== NAME TestOrphanedObjectsGCStressTest1248 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion12492026/11/12 00:21:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12502026/11/12 00:21:30 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=YTk3NjVkMjgtYWZhMS00OGQ0LTk3NTEtNGIyNjNmZjZiNmRkLmU0Nzc0YWU5LWUwYTUtNGZkMy05ZTYxLWEwNGUzZDY3ZjAzNngxNzk0NDQyODg4MzQ4NjY5MDAw parts=1212512026/11/12 00:21:30 INFO Received uploads request method=POST path=/api/pending_closures12522026/11/12 00:21:30 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket49 days=21253--- PASS: TestEnsureAbortUploadsRule (6.48s)1254--- PASS: TestCompletedNarNotReofferedAcrossClosures (7.54s)1255=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1256=== CONT TestFailedS3DeleteKeepsTombstone12572026/11/12 00:21:30 WARN readiness check failed error="closed pool"1258--- PASS: TestService_readinessHandler (6.10s)1259=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection1260=== NAME TestNARDeduplicationMetadataUploadBug1261 metadata_upload_test.go:48: First store path: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestNARDeduplicationMetadataUploadBug1187711098/001/store/syf13kmqabi7q53gp074nghidg31gd3n-file1.txt12622026/11/12 00:21:30 INFO Received push request method=POST path=/api/pushes1263--- PASS: TestService_healthCheckHandler (5.64s)1264=== CONT TestGCSweepDeliversEachKeyOnce12652026/11/12 00:21:30 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12662026/11/12 00:21:30 INFO Uploading syf13kmqabi7q53gp074nghidg31gd3n-file1.txt (160B)12672026/11/12 00:21:30 WARN Failed to register uploaded object key=syf13kmqabi7q53gp074nghidg31gd3n.ls error="server returned 404: 404 page not found\n"12682026/11/12 00:21:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12692026/11/12 00:21:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12702026/11/12 00:21:30 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"12712026/11/12 00:21:30 INFO Signed narinfos id=1 count=112722026/11/12 00:21:30 INFO Uploading 1 narinfos12732026/11/12 00:21:30 INFO Received complete push request method=POST path=/api/pushes/1/complete12742026/11/12 00:21:30 WARN Failed to register uploaded object key=syf13kmqabi7q53gp074nghidg31gd3n.narinfo error="server returned 404: 404 page not found\n"12752026/11/12 00:21:30 INFO Aborted multipart uploads count=1 kept=012762026/11/12 00:21:30 INFO Completed push id=1 objects=3 duration=23.7385ms12772026/11/12 00:21:30 INFO Upload complete. (356ms)1278=== NAME TestNARDeduplicationMetadataUploadBug12792026/11/12 00:21:31 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=YTk3NjVkMjgtYWZhMS00OGQ0LTk3NTEtNGIyNjNmZjZiNmRkLjIyNTA5YjI1LTk5MTctNGU5Zi04MGY3LWIwMDA4OWQ2YWFjNXgxNzk0NDQyODg3Nzc4MTYyMDAw parts=121280 metadata_upload_test.go:54: Retrieved narinfo from S3:1281 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestNARDeduplicationMetadataUploadBug1187711098/001/store/syf13kmqabi7q53gp074nghidg31gd3n-file1.txt1282 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1283 Compression: zstd1284 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1285 NarSize: 1601286 References: 1287 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1288--- PASS: TestSharedNarBothUploadsComplete (8.90s)12892026/11/12 00:21:31 INFO Received cleanup request method=DELETE path=/api/pending_closures1290=== CONT TestForceGCDuringPushOffersSweptObject1291=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1292=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1293=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1294=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1295=== CONT TestTombstonedObjectOfferedWithoutWaiting1296=== NAME TestNARDeduplicationMetadataUploadBug1297 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1298 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1299 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}13002026/11/12 00:21:31 INFO Aborted multipart uploads count=1 kept=01301--- PASS: TestPendingCleanupUsesOneCutoff (11.26s)1302=== CONT TestResolveDBConnectionString1303=== RUN TestResolveDBConnectionString/flag_wins1304=== PAUSE TestResolveDBConnectionString/flag_wins1305=== RUN TestResolveDBConnectionString/file_when_flag_empty1306=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1307=== RUN TestResolveDBConnectionString/missing_file_is_an_error1308=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1309=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1310=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1311=== RUN TestResolveDBConnectionString/nothing_configured1312=== PAUSE TestResolveDBConnectionString/nothing_configured1313=== CONT TestClientErrorHandling1314=== RUN TestClientErrorHandling/InvalidStorePath1315=== PAUSE TestClientErrorHandling/InvalidStorePath1316=== RUN TestClientErrorHandling/InvalidAuthToken1317=== PAUSE TestClientErrorHandling/InvalidAuthToken1318=== RUN TestClientErrorHandling/ServerNotAvailable1319=== PAUSE TestClientErrorHandling/ServerNotAvailable1320=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1321=== NAME TestNARDeduplicationMetadataUploadBug1322 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestNARDeduplicationMetadataUploadBug1187711098/001/store/f995rbx5v4f11v77dfgcxskhsvsr228j-file2.txt13232026/11/12 00:21:31 INFO Received uploads request method=POST path=/api/pending_closures13242026/11/12 00:21:31 INFO Received push request method=POST path=/api/pushes13252026/11/12 00:21:31 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)13262026/11/12 00:21:31 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13272026/11/12 00:21:31 WARN Failed to register uploaded object key=f995rbx5v4f11v77dfgcxskhsvsr228j.ls error="server returned 404: 404 page not found\n"13282026/11/12 00:21:31 INFO Signed narinfos id=2 count=113292026/11/12 00:21:31 INFO Uploading 1 narinfos13302026/11/12 00:21:31 INFO Received complete push request method=POST path=/api/pushes/2/complete13312026/11/12 00:21:31 WARN Failed to register uploaded object key=f995rbx5v4f11v77dfgcxskhsvsr228j.narinfo error="server returned 404: 404 page not found\n"13322026/11/12 00:21:31 INFO Aborted multipart uploads count=0 kept=013332026/11/12 00:21:31 INFO Completed push id=2 objects=3 duration=4.276417ms13342026/11/12 00:21:31 INFO Aborted multipart uploads count=0 kept=013352026/11/12 00:21:31 INFO Upload complete. (231ms)1336 metadata_upload_test.go:76: Retrieved narinfo from S3:1337 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestNARDeduplicationMetadataUploadBug1187711098/001/store/f995rbx5v4f11v77dfgcxskhsvsr228j-file2.txt1338 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1339 Compression: zstd1340 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1341 NarSize: 1601342 References: 1343 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1344 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1345 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1346 {"version":1,"root":{"type":"regular","size":44}}13472026/11/12 00:21:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013482026/11/12 00:21:31 WARN Force mode enabled - objects will be deleted immediately without grace period13492026/11/12 00:21:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013502026/11/12 00:21:31 INFO Vacuumed table table=pending_closures13512026/11/12 00:21:31 INFO Vacuumed table table=pending_objects13522026/11/12 00:21:31 INFO Vacuumed table table=multipart_uploads13532026/11/12 00:21:31 INFO Vacuumed table table=closures13542026/11/12 00:21:31 INFO Vacuumed table table=pending_closures13552026/11/12 00:21:31 INFO Vacuumed table table=objects1356--- PASS: TestGCMetrics (6.06s)1357=== CONT TestDeleteClosuresRacingPins13582026/11/12 00:21:31 INFO Vacuumed table table=pending_objects13592026/11/12 00:21:31 INFO Vacuumed table table=multipart_uploads1360--- PASS: TestNARDeduplicationMetadataUploadBug (7.74s)1361=== CONT TestClientFallsBackToClosures13622026/11/12 00:21:31 INFO Vacuumed table table=closures13632026/11/12 00:21:31 INFO Vacuumed table table=objects13642026/11/12 00:21:31 INFO Aborted multipart uploads count=0 kept=013652026/11/12 00:21:31 WARN Force mode enabled - objects will be deleted immediately without grace period13662026/11/12 00:21:31 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013672026/11/12 00:21:31 INFO Vacuumed table table=pending_closures13682026/11/12 00:21:31 INFO Vacuumed table table=pending_objects13692026/11/12 00:21:31 INFO Vacuumed table table=multipart_uploads13702026/11/12 00:21:31 INFO Vacuumed table table=closures13712026/11/12 00:21:31 INFO Vacuumed table table=objects1372--- PASS: TestPushDedupSurvivesConcurrentGC (6.45s)1373=== CONT TestClientPushesUseOnePush13742026/11/12 00:21:31 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld1 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1375--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (5.72s)1376=== CONT TestPinProtectsFromGC13772026/11/12 00:21:32 INFO lead: acquired remote=192.0.2.1:1234 generation=11378--- PASS: TestGCBugBareHashReferences (6.22s)1379=== CONT TestService_RequireScope_OIDC13802026/11/12 00:21:32 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49367/oidc13812026/11/12 00:21:32 INFO lead: acquired remote=192.0.2.1:1234 generation=113822026/11/12 00:21:32 INFO lead: acquired remote=192.0.2.1:1234 generation=113832026/11/12 00:21:32 INFO lead: released remote=192.0.2.1:1234 generation=11384--- PASS: TestLeadIncumbentAnnouncesAtOnce (5.51s)1385=== CONT TestClientCADerivations13862026/11/12 00:21:33 INFO lead: acquired remote=192.0.2.1:1234 generation=113872026/11/12 00:21:33 INFO lead: acquired remote=192.0.2.1:1234 generation=113882026/11/12 00:21:33 INFO lead: released remote=192.0.2.1:1234 generation=11389--- PASS: TestLeadEndsOnShutdown (6.63s)1390=== CONT TestService_AuthMiddleware_OIDC13912026/11/12 00:21:33 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49374/oidc13922026/11/12 00:21:33 INFO Aborted multipart uploads count=0 kept=013932026/11/12 00:21:33 INFO lead: released remote=192.0.2.1:1234 generation=113942026/11/12 00:21:33 INFO lead: acquired remote=192.0.2.1:1234 generation=213952026/11/12 00:21:33 WARN Force mode enabled - objects will be deleted immediately without grace period13962026/11/12 00:21:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013972026/11/12 00:21:33 INFO Vacuumed table table=pending_closures13982026/11/12 00:21:33 INFO Vacuumed table table=pending_objects13992026/11/12 00:21:33 INFO Vacuumed table table=multipart_uploads14002026/11/12 00:21:33 INFO Vacuumed table table=closures14012026/11/12 00:21:33 INFO Vacuumed table table=objects1402--- PASS: TestCommitRacingPendingCleanupKeepsObjects (5.65s)1403=== CONT TestCacheStatsHandler14042026/11/12 00:21:33 INFO Aborted multipart uploads count=0 kept=014052026/11/12 00:21:33 WARN Force mode enabled - objects will be deleted immediately without grace period14062026/11/12 00:21:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014072026/11/12 00:21:33 INFO Vacuumed table table=pending_closures14082026/11/12 00:21:34 INFO Vacuumed table table=pending_objects14092026/11/12 00:21:34 INFO Vacuumed table table=multipart_uploads14102026/11/12 00:21:34 INFO Vacuumed table table=closures14112026/11/12 00:21:34 INFO Vacuumed table table=objects1412--- PASS: TestSweepRechecksPendingAfterLocking (5.04s)1413=== CONT TestCacheConfigHandler1414=== RUN TestCacheConfigHandler/full_config,_no_issuer1415=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1416=== RUN TestCacheConfigHandler/no_cache_url_configured1417=== PAUSE TestCacheConfigHandler/no_cache_url_configured1418=== RUN TestCacheConfigHandler/no_signing_keys1419=== PAUSE TestCacheConfigHandler/no_signing_keys1420=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1421=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1422=== CONT TestService_ReadScope_PublicByDefault14232026/11/12 00:21:34 INFO Received uploads request method=POST path=/api/pending_closures14242026/11/12 00:21:34 INFO lead: released remote=192.0.2.1:1234 generation=114252026/11/12 00:21:34 INFO lead: acquired remote=192.0.2.1:1234 generation=21426--- PASS: TestGetExistingObjectsLocksRows (4.77s)1427=== CONT TestPush_CompleteWaitsForCleanupOfThePush14282026/11/12 00:21:34 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=214292026/11/12 00:21:34 INFO lead: acquired remote=192.0.2.1:1234 generation=31430--- PASS: TestRegisterCompletedObjectNeedsPendingRow (4.73s)1431=== CONT TestPush_RootsAreLockedInByteOrder14322026/11/12 00:21:34 INFO lead: released remote=192.0.2.1:1234 generation=11433--- PASS: TestPushWaitsForSweepHoldingRow (5.67s)1434=== CONT TestPush_RepeatedRootCommitsOnce14352026/11/12 00:21:34 INFO lead: acquired remote=192.0.2.1:1234 generation=214362026/11/12 00:21:34 INFO Aborted multipart uploads count=0 kept=014372026/11/12 00:21:34 WARN Force mode enabled - objects will be deleted immediately without grace period14382026/11/12 00:21:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014392026/11/12 00:21:34 INFO Vacuumed table table=pending_closures14402026/11/12 00:21:34 INFO Vacuumed table table=pending_objects14412026/11/12 00:21:34 INFO Vacuumed table table=multipart_uploads14422026/11/12 00:21:34 INFO Vacuumed table table=closures14432026/11/12 00:21:34 INFO Vacuumed table table=objects14442026/11/12 00:21:34 INFO Aborted multipart uploads count=0 kept=014452026/11/12 00:21:34 INFO Received uploads request method=POST path=/api/pending_closures14462026/11/12 00:21:34 WARN Force mode enabled - objects will be deleted immediately without grace period14472026/11/12 00:21:34 INFO lead: released remote=192.0.2.1:1234 generation=214482026/11/12 00:21:34 INFO lead: acquired remote=192.0.2.1:1234 generation=314492026/11/12 00:21:34 INFO lead: released remote=192.0.2.1:1234 generation=31450--- PASS: TestLeadGenerationGrowsWithEveryGrant (8.07s)1451=== CONT TestClientSharedPathCommittedMidPush14522026/11/12 00:21:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=014532026/11/12 00:21:34 INFO Vacuumed table table=pending_closures14542026/11/12 00:21:34 INFO Vacuumed table table=pending_objects14552026/11/12 00:21:34 INFO Vacuumed table table=multipart_uploads14562026/11/12 00:21:34 INFO Vacuumed table table=closures14572026/11/12 00:21:34 INFO Vacuumed table table=objects1458--- PASS: TestSweepSkipsRowsLockedByPush (5.09s)1459=== CONT TestClientWithDependencies14602026/11/12 00:21:34 INFO Received uploads request method=POST path=/api/pending_closures14612026/11/12 00:21:35 INFO Received complete multipart upload request method=POST path=/api/multipart/complete14622026/11/12 00:21:35 INFO Aborted multipart uploads count=0 kept=014632026/11/12 00:21:35 WARN Force mode enabled - objects will be deleted immediately without grace period14642026/11/12 00:21:35 INFO Received uploads request method=POST path=/api/pending_closures14652026/11/12 00:21:35 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."14662026/11/12 00:21:35 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1467--- PASS: TestFailedS3DeleteKeepsTombstone (5.04s)1468=== CONT TestClientMultipleUploads1469--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (4.90s)1470=== CONT TestClientIntegration14712026/11/12 00:21:35 INFO Aborted multipart uploads count=0 kept=014722026/11/12 00:21:35 WARN Force mode enabled - objects will be deleted immediately without grace period14732026/11/12 00:21:35 INFO Received uploads request method=POST path=/api/pending_closures1474--- PASS: TestTombstonedObjectOfferedWithoutWaiting (4.62s)1475=== CONT TestService_ReadAuthMiddleware14762026/11/12 00:21:35 INFO lead: released remote=192.0.2.1:1234 generation=21477--- PASS: TestLeadElectsOneAndHandsOver (7.78s)1478=== CONT TestService_AuthMiddleware1479--- PASS: TestConnectWaitsForAPeerMigration (7.80s)1480=== CONT TestService_AuthMiddleware_MTLSProxyHeader14812026/11/12 00:21:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014822026/11/12 00:21:35 INFO Vacuumed table table=pending_closures14832026/11/12 00:21:35 INFO Vacuumed table table=pending_objects14842026/11/12 00:21:35 INFO Vacuumed table table=multipart_uploads14852026/11/12 00:21:35 INFO Vacuumed table table=closures14862026/11/12 00:21:35 INFO Vacuumed table table=objects1487--- PASS: TestGCSweepDeliversEachKeyOnce (5.08s)1488=== CONT TestService_AuthMiddleware_MTLSBoundSubjects14892026/11/12 00:21:36 INFO lead: released remote=192.0.2.1:1234 generation=31490--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (8.63s)1491=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale14922026/11/12 00:21:36 INFO Received uploads request method=POST path=/api/pending_closures14932026/11/12 00:21:36 INFO Received uploads request method=POST path=/api/pending_closures14942026/11/12 00:21:36 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)14952026/11/12 00:21:36 INFO Uploading ava68a3ws9k42c0mb32icn5ajf3hqa7s-shared-dep (136B)14962026/11/12 00:21:36 INFO Uploading 7wm4q8naj77611ksmmaakywixqh6rnlx-a (256B)14972026/11/12 00:21:36 WARN Failed to register uploaded object key=3vc8p0aq01a43g1gjz8zydh4a6szm5hy.ls error="server returned 404: 404 page not found\n"14982026/11/12 00:21:36 WARN Failed to register uploaded object key=7wm4q8naj77611ksmmaakywixqh6rnlx.ls error="server returned 404: 404 page not found\n"14992026/11/12 00:21:36 WARN Failed to register uploaded object key=ava68a3ws9k42c0mb32icn5ajf3hqa7s.ls error="server returned 404: 404 page not found\n"15002026/11/12 00:21:36 WARN Failed to register uploaded object key=nar/0z5220nb449hwirigc2s34lw3rhb46x1751g34mq6975j7whnv5a.nar.zst error="server returned 404: 404 page not found\n"15012026/11/12 00:21:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign15022026/11/12 00:21:36 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15032026/11/12 00:21:36 INFO Signed narinfos id=1 count=215042026/11/12 00:21:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign15052026/11/12 00:21:37 INFO Signed narinfos id=2 count=215062026/11/12 00:21:37 INFO Uploading 4 narinfos15072026/11/12 00:21:37 WARN Failed to register uploaded object key=ava68a3ws9k42c0mb32icn5ajf3hqa7s.narinfo error="server returned 404: 404 page not found\n"15082026/11/12 00:21:37 WARN Failed to register uploaded object key=7wm4q8naj77611ksmmaakywixqh6rnlx.narinfo error="server returned 404: 404 page not found\n"1509--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (5.98s)1510=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh15112026/11/12 00:21:37 WARN Failed to register uploaded object key=3vc8p0aq01a43g1gjz8zydh4a6szm5hy.narinfo error="server returned 404: 404 page not found\n"15122026/11/12 00:21:37 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15132026/11/12 00:21:37 WARN Failed to register uploaded object key=ava68a3ws9k42c0mb32icn5ajf3hqa7s.narinfo error="server returned 404: 404 page not found\n"15142026/11/12 00:21:37 INFO Completed upload id=1 objects=6 duration=68.932375ms15152026/11/12 00:21:37 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete15162026/11/12 00:21:37 INFO Completed upload id=2 objects=6 duration=1.8755ms15172026/11/12 00:21:37 INFO Upload complete. (485ms)1518=== NAME TestClientFallsBackToClosures1519 client_pushes_test.go:112: Retrieved narinfo from S3:1520 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientFallsBackToClosures3143878806/001/store/ava68a3ws9k42c0mb32icn5ajf3hqa7s-shared-dep1521 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1522 Compression: zstd1523 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821524 NarSize: 1361525 References: 1526 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1527 client_pushes_test.go:112: Retrieved narinfo from S3:1528 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientFallsBackToClosures3143878806/001/store/7wm4q8naj77611ksmmaakywixqh6rnlx-a1529 URL: nar/0z5220nb449hwirigc2s34lw3rhb46x1751g34mq6975j7whnv5a.nar.zst1530 Compression: zstd1531 NarHash: sha256:0z5220nb449hwirigc2s34lw3rhb46x1751g34mq6975j7whnv5a1532 NarSize: 2561533 References: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientFallsBackToClosures3143878806/001/store/ava68a3ws9k42c0mb32icn5ajf3hqa7s-shared-dep1534 CA: text:sha256:0k95wyk0vbfq0yjzphpahn4s6lp75rhqbk0m9kky4n25pibdrjlz1535 client_pushes_test.go:112: Retrieved narinfo from S3:1536 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientFallsBackToClosures3143878806/001/store/3vc8p0aq01a43g1gjz8zydh4a6szm5hy-b1537 URL: nar/0z5220nb449hwirigc2s34lw3rhb46x1751g34mq6975j7whnv5a.nar.zst1538 Compression: zstd1539 NarHash: sha256:0z5220nb449hwirigc2s34lw3rhb46x1751g34mq6975j7whnv5a1540 NarSize: 2561541 References: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientFallsBackToClosures3143878806/001/store/ava68a3ws9k42c0mb32icn5ajf3hqa7s-shared-dep1542 CA: text:sha256:0k95wyk0vbfq0yjzphpahn4s6lp75rhqbk0m9kky4n25pibdrjlz1543--- PASS: TestClientFallsBackToClosures (5.71s)1544=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1545=== RUN TestService_RequireScope_OIDC/builder_may_write1546=== PAUSE TestService_RequireScope_OIDC/builder_may_write1547=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1548=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1549=== RUN TestService_RequireScope_OIDC/ops_may_admin1550=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1551=== RUN TestService_RequireScope_OIDC/ops_may_not_write1552=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1553=== RUN TestService_RequireScope_OIDC/reader_may_not_write1554=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1555=== RUN TestService_RequireScope_OIDC/static_token_may_admin1556=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1557=== RUN TestService_RequireScope_OIDC/static_token_may_write1558=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1559=== RUN TestService_RequireScope_OIDC/reader_may_read1560=== PAUSE TestService_RequireScope_OIDC/reader_may_read1561=== RUN TestService_RequireScope_OIDC/writer_implies_read1562=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1563=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1564=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1565=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1566=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1567=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1568=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1569=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1570=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1571=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1572=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1573=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1574=== NAME TestPinProtectsFromGC1575 client_integration_test.go:779: Pinned store path: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestPinProtectsFromGC3283227060/001/store/i1gpyhpv6v11hxmrn0pmwky7z9ag3j8z-pinned-file.txt1576 client_integration_test.go:780: Unpinned store path: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestPinProtectsFromGC3283227060/001/store/bni8g9pvsidxwg57s346ljwzh86r67n8-unpinned-file.txt15772026/11/12 00:21:37 INFO Received push request method=POST path=/api/pushes15782026/11/12 00:21:37 INFO Received push request method=POST path=/api/pushes15792026/11/12 00:21:37 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)15802026/11/12 00:21:37 INFO Uploading s91r990n04ihp5k7c9w8y4iycysk3mja-b (256B)15812026/11/12 00:21:37 INFO Uploading z403a7n2mf748vflmxrql3nk6zidj1wp-shared-dep (136B)15822026/11/12 00:21:37 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15832026/11/12 00:21:37 INFO Uploading i1gpyhpv6v11hxmrn0pmwky7z9ag3j8z-pinned-file.txt (128B)15842026/11/12 00:21:37 WARN Failed to register uploaded object key=z403a7n2mf748vflmxrql3nk6zidj1wp.ls error="server returned 404: 404 page not found\n"15852026/11/12 00:21:37 WARN Failed to register uploaded object key=crfgi9lfdfihyzgv7m5iw1cfndlgribm.ls error="server returned 404: 404 page not found\n"15862026/11/12 00:21:37 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"15872026/11/12 00:21:37 WARN Failed to register uploaded object key=s91r990n04ihp5k7c9w8y4iycysk3mja.ls error="server returned 404: 404 page not found\n"15882026/11/12 00:21:37 WARN Failed to register uploaded object key=nar/0a2nhgrhc9zras4krx6x94hi7i1m2aqpz9yfxw5l7gqwmf66813j.nar.zst error="server returned 404: 404 page not found\n"15892026/11/12 00:21:37 WARN Failed to register uploaded object key=i1gpyhpv6v11hxmrn0pmwky7z9ag3j8z.ls error="server returned 404: 404 page not found\n"15902026/11/12 00:21:37 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15912026/11/12 00:21:37 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15922026/11/12 00:21:37 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"15932026/11/12 00:21:37 INFO Signed narinfos id=1 count=315942026/11/12 00:21:37 INFO Uploading 3 narinfos15952026/11/12 00:21:37 INFO Signed narinfos id=1 count=115962026/11/12 00:21:37 INFO Uploading 1 narinfos15972026/11/12 00:21:37 WARN Failed to register uploaded object key=crfgi9lfdfihyzgv7m5iw1cfndlgribm.narinfo error="server returned 404: 404 page not found\n"15982026/11/12 00:21:37 WARN Failed to register uploaded object key=z403a7n2mf748vflmxrql3nk6zidj1wp.narinfo error="server returned 404: 404 page not found\n"15992026/11/12 00:21:37 INFO Received complete push request method=POST path=/api/pushes/1/complete16002026/11/12 00:21:37 WARN Failed to register uploaded object key=i1gpyhpv6v11hxmrn0pmwky7z9ag3j8z.narinfo error="server returned 404: 404 page not found\n"16012026/11/12 00:21:37 INFO Received complete push request method=POST path=/api/pushes/1/complete16022026/11/12 00:21:37 WARN Failed to register uploaded object key=s91r990n04ihp5k7c9w8y4iycysk3mja.narinfo error="server returned 404: 404 page not found\n"16032026/11/12 00:21:37 INFO Completed push id=1 objects=3 duration=12.601959ms16042026/11/12 00:21:38 INFO Upload complete. (404ms)16052026/11/12 00:21:38 INFO Completed push id=1 objects=8 duration=16.355333ms16062026/11/12 00:21:38 INFO Upload complete. (421ms)1607=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1608=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1609=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1610=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1611=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1612=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1613=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1614=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1615=== NAME TestClientPushesUseOnePush1616 client_pushes_test.go:97: Retrieved narinfo from S3:1617 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientPushesUseOnePush3699827256/001/store/z403a7n2mf748vflmxrql3nk6zidj1wp-shared-dep1618 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1619 Compression: zstd1620 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821621 NarSize: 1361622 References: 1623 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1624=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1625=== NAME TestClientPushesUseOnePush1626 client_pushes_test.go:97: Retrieved narinfo from S3:1627 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientPushesUseOnePush3699827256/001/store/crfgi9lfdfihyzgv7m5iw1cfndlgribm-a1628 URL: nar/0a2nhgrhc9zras4krx6x94hi7i1m2aqpz9yfxw5l7gqwmf66813j.nar.zst1629 Compression: zstd1630 NarHash: sha256:0a2nhgrhc9zras4krx6x94hi7i1m2aqpz9yfxw5l7gqwmf66813j1631 NarSize: 2561632 References: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientPushesUseOnePush3699827256/001/store/z403a7n2mf748vflmxrql3nk6zidj1wp-shared-dep1633 CA: text:sha256:1q8hhpg1pivzfg9d50jfqhnlma2kf6k9g0qmlszcy5jwaqni1dz91634 client_pushes_test.go:97: Retrieved narinfo from S3:1635 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientPushesUseOnePush3699827256/001/store/s91r990n04ihp5k7c9w8y4iycysk3mja-b1636 URL: nar/0a2nhgrhc9zras4krx6x94hi7i1m2aqpz9yfxw5l7gqwmf66813j.nar.zst1637 Compression: zstd1638 NarHash: sha256:0a2nhgrhc9zras4krx6x94hi7i1m2aqpz9yfxw5l7gqwmf66813j1639 NarSize: 2561640 References: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientPushesUseOnePush3699827256/001/store/z403a7n2mf748vflmxrql3nk6zidj1wp-shared-dep1641 CA: text:sha256:1q8hhpg1pivzfg9d50jfqhnlma2kf6k9g0qmlszcy5jwaqni1dz91642--- PASS: TestClientPushesUseOnePush (6.30s)1643=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost16442026/11/12 00:21:38 INFO Received push request method=POST path=/api/pushes16452026/11/12 00:21:38 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16462026/11/12 00:21:38 INFO Uploading bni8g9pvsidxwg57s346ljwzh86r67n8-unpinned-file.txt (128B)16472026/11/12 00:21:38 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"16482026/11/12 00:21:38 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign16492026/11/12 00:21:38 WARN Failed to register uploaded object key=bni8g9pvsidxwg57s346ljwzh86r67n8.ls error="server returned 404: 404 page not found\n"16502026/11/12 00:21:38 INFO Signed narinfos id=2 count=116512026/11/12 00:21:38 INFO Uploading 1 narinfos16522026/11/12 00:21:38 INFO Received complete push request method=POST path=/api/pushes/2/complete16532026/11/12 00:21:38 WARN Failed to register uploaded object key=bni8g9pvsidxwg57s346ljwzh86r67n8.narinfo error="server returned 404: 404 page not found\n"16542026/11/12 00:21:38 INFO Completed push id=2 objects=3 duration=1.402959ms16552026/11/12 00:21:38 INFO Upload complete. (197ms)1656=== NAME TestClientCADerivations1657 client_ca_test.go:136: Built CA derivation: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientCADerivations2993108417/001/store/h1lnld4n6k9i3bk3nxvvj5cxg018vbqv-ca-test1658--- PASS: TestCacheStatsHandler (4.47s)1659=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed16602026/11/12 00:21:38 INFO Starting cleanup of old closures method=DELETE path=/api/closures16612026/11/12 00:21:38 INFO Garbage collection started1662=== NAME TestClientCADerivations1663 client_ca_test.go:139: Found 1 dependencies (including self)16642026/11/12 00:21:38 INFO Aborted multipart uploads count=0 kept=016652026/11/12 00:21:38 WARN Force mode enabled - objects will be deleted immediately without grace period1666--- PASS: TestService_ReadScope_PublicByDefault (4.45s)1667=== CONT TestParseSingleRange/none1668=== CONT TestParseSingleRange/start_far_past_EOF1669=== CONT TestParseSingleRange/start_past_EOF1670=== CONT TestParseSingleRange/single_byte1671=== CONT TestParseSingleRange/suffix_exceeds_size1672=== CONT TestParseSingleRange/suffix1673=== CONT TestParseSingleRange/end_clamped_to_size1674=== CONT TestParseSingleRange/open-ended1675=== CONT TestParseSingleRange/closed1676=== CONT TestParseSingleRange/malformed_end_before_start1677=== CONT TestParseSingleRange/malformed_both_empty1678=== CONT TestParseSingleRange/malformed_no_dash1679=== CONT TestParseSingleRange/multi-range_ignored1680=== CONT TestParseSingleRange/unknown_unit1681--- PASS: TestParseSingleRange (0.00s)1682 --- PASS: TestParseSingleRange/none (0.00s)1683 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1684 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1685 --- PASS: TestParseSingleRange/single_byte (0.00s)1686 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1687 --- PASS: TestParseSingleRange/suffix (0.00s)1688 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1689 --- PASS: TestParseSingleRange/open-ended (0.00s)1690 --- PASS: TestParseSingleRange/closed (0.00s)1691 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1692 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1693 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1694 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1695 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1696=== CONT TestIsValidCachePath/traversal_parent1697=== CONT TestIsValidCachePath/narinfo1698=== CONT TestIsValidCachePath/index.html1699=== CONT TestIsValidCachePath/nix-cache-info1700=== CONT TestIsValidCachePath/realisation1701=== CONT TestIsValidCachePath/nar_xz1702=== CONT TestIsValidCachePath/nar_zst1703=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1704=== CONT TestIsValidCachePath/empty1705=== CONT TestIsValidCachePath/short_hash1706=== CONT TestIsValidCachePath/wrong_extension1707=== CONT TestIsValidCachePath/leading_slash1708=== CONT TestIsValidCachePath/ls1709=== CONT TestIsValidCachePath/log1710=== CONT TestIsValidCachePath/invalid_char_u1711=== CONT TestIsValidCachePath/random_path1712=== CONT TestIsValidCachePath/invalid_char_e1713=== CONT TestIsValidCachePath/traversal_in_middle1714=== CONT TestIsValidCachePath/nar_uncompressed1715=== CONT TestIsValidCachePath/nar_bz21716--- PASS: TestIsValidCachePath (0.00s)1717 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1718 --- PASS: TestIsValidCachePath/narinfo (0.00s)1719 --- PASS: TestIsValidCachePath/index.html (0.00s)1720 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1721 --- PASS: TestIsValidCachePath/realisation (0.00s)1722 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1723 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1724 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1725 --- PASS: TestIsValidCachePath/empty (0.00s)1726 --- PASS: TestIsValidCachePath/short_hash (0.00s)1727 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1728 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1729 --- PASS: TestIsValidCachePath/ls (0.00s)1730 --- PASS: TestIsValidCachePath/log (0.00s)1731 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1732 --- PASS: TestIsValidCachePath/random_path (0.00s)1733 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1734 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1735 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1736 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1737=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info17382026/11/12 00:21:38 INFO Received uploads request method=POST path=/1739=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key17402026/11/12 00:21:38 INFO Received complete multipart upload request method=POST path=/1741=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key17422026/11/12 00:21:38 INFO Received request for more parts method=POST path=/1743=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal17442026/11/12 00:21:38 INFO Received uploads request method=POST path=/1745--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1746 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1747 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1748 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1749 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1750=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure17512026/11/12 00:21:38 INFO Received uploads request method=POST path=/17522026/11/12 00:21:38 WARN Rate limiter enabled after throttle name=s3-test rate=517532026/11/12 00:21:38 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1754=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1755 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101756 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001757--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (8.41s)1758=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts17592026/11/12 00:21:38 INFO Received request for more parts method=POST path=/1760=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart17612026/11/12 00:21:38 INFO Received complete multipart upload request method=POST path=/1762=== CONT TestPendingClosureWriteTimeout/empty1763=== CONT TestPendingClosureWriteTimeout/670k_objects1764=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline17652026/11/12 00:21:38 INFO Received push request method=POST path=/api/pushes17662026/11/12 00:21:38 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17672026/11/12 00:21:38 INFO Uploading h1lnld4n6k9i3bk3nxvvj5cxg018vbqv-ca-test (144B)17682026/11/12 00:21:38 WARN Failed to register uploaded object key=h1lnld4n6k9i3bk3nxvvj5cxg018vbqv.ls error="server returned 404: 404 page not found\n"17692026/11/12 00:21:38 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"17702026/11/12 00:21:38 INFO Received push request method=POST path=/api/pushes17712026/11/12 00:21:38 WARN Failed to register uploaded object key=log/473df2vparpz3iz84jl2iln8nljjk6pi-ca-test.drv error="server returned 404: 404 page not found\n"17722026/11/12 00:21:38 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17732026/11/12 00:21:38 INFO Signed narinfos id=1 count=117742026/11/12 00:21:38 INFO Uploading 1 narinfos17752026/11/12 00:21:38 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=017762026/11/12 00:21:38 INFO Received complete push request method=POST path=/api/pushes/1/complete17772026/11/12 00:21:38 WARN Failed to register uploaded object key=h1lnld4n6k9i3bk3nxvvj5cxg018vbqv.narinfo error="server returned 404: 404 page not found\n"17782026/11/12 00:21:38 INFO Vacuumed table table=pending_closures17792026/11/12 00:21:38 INFO Completed push id=1 objects=4 duration=83.447167ms17802026/11/12 00:21:38 INFO Upload complete. (445ms)17812026/11/12 00:21:38 INFO Vacuumed table table=pending_objects1782=== NAME TestClientCADerivations1783 client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientCADerivations2993108417/001/store/h1lnld4n6k9i3bk3nxvvj5cxg018vbqv-ca-test1784 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1785 Compression: zstd1786 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1787 NarSize: 1441788 References: 1789 Deriver: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientCADerivations2993108417/001/store/473df2vparpz3iz84jl2iln8nljjk6pi-ca-test.drv1790 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1791 client_ca_test.go:185: Checking for realisation files in S3...17922026/11/12 00:21:38 INFO Vacuumed table table=multipart_uploads1793 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1794 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache17952026/11/12 00:21:38 INFO Vacuumed table table=closures17962026/11/12 00:21:38 INFO Vacuumed table table=objects17972026/11/12 00:21:39 INFO Received push request method=POST path=/api/pushes1798--- PASS: TestUploadHandlersRejectOversizedBody (0.04s)1799 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.03s)1800 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.05s)1801 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (0.58s)1802=== CONT TestPendingClosureWriteTimeout/400_objects1803=== CONT TestIsValidUploadKey/narinfo1804=== CONT TestIsValidUploadKey/realisation_plus_in_output1805=== CONT TestIsValidUploadKey/unknown_type1806=== CONT TestIsValidUploadKey/empty_key1807=== CONT TestIsValidUploadKey/absolute1808=== CONT TestIsValidUploadKey/traversal_nar1809=== CONT TestIsValidUploadKey/traversal1810=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1811=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1812=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1813=== CONT TestIsValidUploadKey/index.html1814=== CONT TestIsValidUploadKey/nix-cache-info1815=== CONT TestIsValidUploadKey/build_log_home-manager_file1816=== CONT TestIsValidUploadKey/realisation1817=== CONT TestIsValidUploadKey/build_log_equals1818=== CONT TestIsValidUploadKey/build_log_question_mark1819=== CONT TestIsValidUploadKey/build_log_plus_in_name1820=== CONT TestIsValidUploadKey/nar_plain1821=== CONT TestIsValidUploadKey/build_log1822=== CONT TestIsValidUploadKey/listing1823=== CONT TestIsValidUploadKey/nar_xz1824=== CONT TestIsValidUploadKey/nar_zst1825--- PASS: TestIsValidUploadKey (0.01s)1826 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1827 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1828 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1829 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1830 --- PASS: TestIsValidUploadKey/absolute (0.00s)1831 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1832 --- PASS: TestIsValidUploadKey/traversal (0.00s)1833 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1834 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1835 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1836 --- PASS: TestIsValidUploadKey/index.html (0.00s)1837 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1838 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1839 --- PASS: TestIsValidUploadKey/realisation (0.00s)1840 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1841 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1842 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1843 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1844 --- PASS: TestIsValidUploadKey/build_log (0.00s)1845 --- PASS: TestIsValidUploadKey/listing (0.00s)1846 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1847 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1848=== CONT TestServerTLSConfig/no_client_CA1849=== CONT TestServerTLSConfig/not_a_PEM_file1850=== NAME TestClientCADerivations1851 client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket79?endpoint=http://localhost:65322®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientCADerivations2993108417/001/store'1852 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11853=== CONT TestServerTLSConfig/missing_CA_file1854--- PASS: TestServerTLSConfig (0.00s)1855 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1856 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.02s)1857 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1858=== CONT TestGCEndsOnShutdown/before_the_run1859--- PASS: TestClientCADerivations (6.33s)1860=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark18612026-11-12 00:21:39.211 UTC [59254] ERROR: Push does not exist: id=118622026-11-12 00:21:39.211 UTC [59254] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE18632026-11-12 00:21:39.211 UTC [59254] STATEMENT: SELECT commit_push($1)1864--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (4.97s)1865=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18662026-11-12 00:21:39.242 UTC [59266] ERROR: could not obtain lock on row in relation "closures"18672026-11-12 00:21:39.242 UTC [59266] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18682026-11-12 00:21:39.245 UTC [59266] ERROR: could not obtain lock on row in relation "closures"18692026-11-12 00:21:39.245 UTC [59266] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18702026-11-12 00:21:39.250 UTC [59266] ERROR: could not obtain lock on row in relation "closures"18712026-11-12 00:21:39.250 UTC [59266] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18722026-11-12 00:21:39.255 UTC [59266] ERROR: could not obtain lock on row in relation "closures"18732026-11-12 00:21:39.255 UTC [59266] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18742026-11-12 00:21:39.259 UTC [59266] ERROR: could not obtain lock on row in relation "closures"18752026-11-12 00:21:39.259 UTC [59266] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1876--- PASS: TestPush_RootsAreLockedInByteOrder (4.87s)1877=== CONT TestPush_RejectsBadRequests/root_not_in_objects18782026/11/12 00:21:39 INFO Received push request method=POST path=/api/pushes1879=== CONT TestPush_RejectsBadRequests/no_objects18802026/11/12 00:21:39 INFO Received push request method=POST path=/api/pushes1881=== CONT TestPush_RejectsBadRequests/bad_root18822026/11/12 00:21:39 INFO Received push request method=POST path=/api/pushes1883=== CONT TestPush_RejectsBadRequests/no_roots18842026/11/12 00:21:39 INFO Received push request method=POST path=/api/pushes1885=== CONT TestProxyWriteTimeout/narinfo1886=== CONT TestProxyWriteTimeout/10_GiB_nar1887=== CONT TestProxyWriteTimeout/unknown_size1888=== CONT TestProxyWriteTimeout/1_GiB_nar1889--- PASS: TestProxyWriteTimeout (0.01s)1890 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1891 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1892 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1893 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1894=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1895--- PASS: TestPush_RejectsBadRequests (6.29s)1896 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1897 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1898 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1899 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)19002026/11/12 00:21:39 INFO Received push request method=POST path=/api/pushes19012026/11/12 00:21:39 INFO Received complete push request method=POST path=/api/pushes/1/complete19022026/11/12 00:21:39 INFO Completed push id=1 objects=4 duration=49.452625ms1903--- PASS: TestPush_RepeatedRootCommitsOnce (4.97s)1904=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1905=== NAME TestClientMultipleUploads1906 client_integration_test.go:406: Created store path 0: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientMultipleUploads3851592461/001/store/n0h1njmrhcp7srxk9ppcxc7qgzf5llbq-test-file-0.txt1907 client_integration_test.go:406: Created store path 1: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientMultipleUploads3851592461/001/store/j90kg1h449m5dp0gbmnzh06p7mjiph8b-test-file-1.txt19082026/11/12 00:21:40 INFO Received push request method=POST path=/api/pushes19092026/11/12 00:21:40 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)19102026/11/12 00:21:40 INFO Uploading sh40jc2q878sxbgj1rjwn2ij5xcl2hb1-top (264B)19112026/11/12 00:21:40 INFO Uploading g4s5rvyb6apygqv502g5pyn7dzw6j4r7-shared-dep (136B)19122026/11/12 00:21:40 WARN Failed to register uploaded object key=sh40jc2q878sxbgj1rjwn2ij5xcl2hb1.ls error="server returned 404: 404 page not found\n"19132026/11/12 00:21:40 WARN Failed to register uploaded object key=g4s5rvyb6apygqv502g5pyn7dzw6j4r7.ls error="server returned 404: 404 page not found\n"19142026/11/12 00:21:40 WARN Failed to register uploaded object key=nar/1zzrgvgl2bk3h2qlq5k2brq949cxdbz1ccs7vvm31y25gb1hkmfk.nar.zst error="server returned 404: 404 page not found\n"19152026/11/12 00:21:40 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19162026/11/12 00:21:40 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19172026/11/12 00:21:40 INFO Signed narinfos id=1 count=219182026/11/12 00:21:40 INFO Uploading 2 narinfos1919 client_integration_test.go:406: Created store path 2: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientMultipleUploads3851592461/001/store/k68qrv3z0jcf2d15y7gk16ardmmlxq4p-test-file-2.txt19202026/11/12 00:21:40 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=019212026/11/12 00:21:40 INFO Received complete push request method=POST path=/api/pushes/1/complete19222026/11/12 00:21:40 WARN Failed to register uploaded object key=g4s5rvyb6apygqv502g5pyn7dzw6j4r7.narinfo error="server returned 404: 404 page not found\n"19232026/11/12 00:21:40 WARN Failed to register uploaded object key=sh40jc2q878sxbgj1rjwn2ij5xcl2hb1.narinfo error="server returned 404: 404 page not found\n"19242026/11/12 00:21:40 INFO Received create pin request method=POST path=/api/pins/myapp19252026/11/12 00:21:40 INFO Completed push id=1 objects=6 duration=26.390542ms19262026/11/12 00:21:40 INFO Upload complete. (366ms)1927=== NAME TestClientSharedPathCommittedMidPush1928 client_integration_test.go:728: Retrieved narinfo from S3:1929 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientSharedPathCommittedMidPush1475973420/001/store/g4s5rvyb6apygqv502g5pyn7dzw6j4r7-shared-dep1930 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1931 Compression: zstd1932 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821933 NarSize: 1361934 References: 1935 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1936 client_integration_test.go:728: Retrieved narinfo from S3:1937 StorePath: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientSharedPathCommittedMidPush1475973420/001/store/sh40jc2q878sxbgj1rjwn2ij5xcl2hb1-top1938 URL: nar/1zzrgvgl2bk3h2qlq5k2brq949cxdbz1ccs7vvm31y25gb1hkmfk.nar.zst1939 Compression: zstd1940 NarHash: sha256:1zzrgvgl2bk3h2qlq5k2brq949cxdbz1ccs7vvm31y25gb1hkmfk1941 NarSize: 2641942 References: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientSharedPathCommittedMidPush1475973420/001/store/g4s5rvyb6apygqv502g5pyn7dzw6j4r7-shared-dep1943 CA: text:sha256:1dgq6bk4a2by2yy59k5y97bil5hpfm130rjk6a08y6fp7malx24k19442026/11/12 00:21:40 INFO Created/updated pin name=myapp store_path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestPinProtectsFromGC3283227060/001/store/i1gpyhpv6v11hxmrn0pmwky7z9ag3j8z-pinned-file.txt narinfo_key=i1gpyhpv6v11hxmrn0pmwky7z9ag3j8z.narinfo19452026/11/12 00:21:40 INFO Starting cleanup of old closures method=DELETE path=/api/closures19462026/11/12 00:21:40 INFO Garbage collection started19472026/11/12 00:21:40 INFO Aborted multipart uploads count=0 kept=019482026/11/12 00:21:40 WARN Force mode enabled - objects will be deleted immediately without grace period19492026/11/12 00:21:40 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019502026/11/12 00:21:40 INFO Vacuumed table table=pending_closures19512026/11/12 00:21:40 INFO Vacuumed table table=pending_objects19522026/11/12 00:21:40 INFO Vacuumed table table=multipart_uploads19532026/11/12 00:21:40 INFO Vacuumed table table=closures1954--- PASS: TestClientSharedPathCommittedMidPush (5.65s)19552026/11/12 00:21:40 INFO Vacuumed table table=objects1956=== CONT TestResolveDBConnectionString/flag_wins1957=== CONT TestClientErrorHandling/InvalidStorePath1958--- PASS: TestDeleteClosuresRacingPins (8.97s)1959=== CONT TestResolveDBConnectionString/nothing_configured1960=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1961=== CONT TestResolveDBConnectionString/missing_file_is_an_error1962=== CONT TestResolveDBConnectionString/file_when_flag_empty1963--- PASS: TestResolveDBConnectionString (0.01s)1964 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1965 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1966 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1967 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1968 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1969=== CONT TestClientErrorHandling/ServerNotAvailable19702026/11/12 00:21:40 INFO Received push request method=POST path=/api/pushes1971=== NAME TestClientIntegration1972 client_integration_test.go:292: Created store path: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientIntegration102136193/002/store/bzcsfrzv51rdx762mycjqjkvi7ry3f3k-test-file.txt1973=== NAME TestClientWithDependencies1974 client_integration_test.go:661: Built derivation: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestClientWithDependencies3690839975/001/store/0n8chcxch0h56mvzwzpm4c54s7ymj5fi-test-script1975--- PASS: TestService_ReadAuthMiddleware (4.99s)1976=== CONT TestClientErrorHandling/InvalidAuthToken19772026/11/12 00:21:40 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)19782026/11/12 00:21:40 INFO Uploading n0h1njmrhcp7srxk9ppcxc7qgzf5llbq-test-file-0.txt (160B)19792026/11/12 00:21:40 INFO Uploading k68qrv3z0jcf2d15y7gk16ardmmlxq4p-test-file-2.txt (160B)19802026/11/12 00:21:40 INFO Uploading j90kg1h449m5dp0gbmnzh06p7mjiph8b-test-file-1.txt (160B)19812026/11/12 00:21:40 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"19822026/11/12 00:21:40 WARN Failed to register uploaded object key=k68qrv3z0jcf2d15y7gk16ardmmlxq4p.ls error="server returned 404: 404 page not found\n"19832026/11/12 00:21:40 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"19842026/11/12 00:21:40 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"19852026/11/12 00:21:40 WARN Failed to register uploaded object key=n0h1njmrhcp7srxk9ppcxc7qgzf5llbq.ls error="server returned 404: 404 page not found\n"19862026/11/12 00:21:40 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19872026/11/12 00:21:40 WARN Failed to register uploaded object key=j90kg1h449m5dp0gbmnzh06p7mjiph8b.ls error="server returned 404: 404 page not found\n"19882026/11/12 00:21:40 INFO Signed narinfos id=1 count=319892026/11/12 00:21:40 INFO Uploading 3 narinfos1990=== NAME TestClientWithDependencies1991 client_integration_test.go:663: Found 1 dependencies (including self)19922026/11/12 00:21:40 WARN Failed to register uploaded object key=k68qrv3z0jcf2d15y7gk16ardmmlxq4p.narinfo error="server returned 404: 404 page not found\n"19932026/11/12 00:21:40 INFO Received complete push request method=POST path=/api/pushes/1/complete19942026/11/12 00:21:40 WARN Failed to register uploaded object key=j90kg1h449m5dp0gbmnzh06p7mjiph8b.narinfo error="server returned 404: 404 page not found\n"19952026/11/12 00:21:40 WARN Failed to register uploaded object key=n0h1njmrhcp7srxk9ppcxc7qgzf5llbq.narinfo error="server returned 404: 404 page not found\n"19962026/11/12 00:21:40 INFO Completed push id=1 objects=9 duration=21.286708ms19972026/11/12 00:21:40 INFO Upload complete. (404ms)1998=== NAME TestClientMultipleUploads1999 client_integration_test.go:417: Uploaded 3 paths in 494.94925ms20002026/11/12 00:21:40 INFO Received push request method=POST path=/api/pushes
loading 675 hidden lines…
26762026/11/12 00:21:59 INFO Vacuumed table table=pending_objects26772026/11/12 00:21:59 INFO Received uploads request method=POST path=/26782026/11/12 00:21:59 INFO Vacuumed table table=multipart_uploads26792026/11/12 00:21:59 INFO Vacuumed table table=multipart_uploads26802026/11/12 00:21:59 INFO Vacuumed table table=closures26812026/11/12 00:21:59 INFO Received uploads request method=POST path=/26822026/11/12 00:21:59 INFO Vacuumed table table=closures26832026/11/12 00:21:59 INFO Vacuumed table table=objects26842026/11/12 00:21:59 INFO Received uploads request method=POST path=/26852026/11/12 00:21:59 INFO Received uploads request method=POST path=/26862026/11/12 00:21:59 INFO Vacuumed table table=objects26872026/11/12 00:21:59 INFO Received uploads request method=POST path=/26882026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=026892026/11/12 00:21:59 INFO Received complete upload request method=POST path=/26902026/11/12 00:21:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026912026/11/12 00:21:59 INFO Completed upload id=9 objects=7 duration=8.29925ms26922026/11/12 00:21:59 INFO Vacuumed table table=pending_closures26932026/11/12 00:21:59 INFO Received push request method=POST path=/26942026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=026952026/11/12 00:21:59 INFO Vacuumed table table=pending_objects26962026/11/12 00:21:59 INFO Received complete upload request method=POST path=/26972026/11/12 00:21:59 INFO Vacuumed table table=multipart_uploads26982026/11/12 00:21:59 INFO Received push request method=POST path=/26992026/11/12 00:21:59 INFO Vacuumed table table=closures27002026/11/12 00:21:59 INFO Completed upload id=10 objects=17 duration=3.668541ms27012026/11/12 00:21:59 WARN Force mode enabled - objects will be deleted immediately without grace period27022026/11/12 00:21:59 INFO Vacuumed table table=objects27032026/11/12 00:21:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027042026/11/12 00:21:59 INFO Vacuumed table table=pending_closures27052026/11/12 00:21:59 INFO Vacuumed table table=pending_objects27062026/11/12 00:21:59 INFO Received uploads request method=POST path=/27072026/11/12 00:21:59 INFO Vacuumed table table=multipart_uploads27082026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=027092026/11/12 00:21:59 INFO Vacuumed table table=closures27102026/11/12 00:21:59 INFO Vacuumed table table=objects27112026/11/12 00:21:59 INFO Received complete push request method=POST path=/27122026/11/12 00:21:59 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027132026/11/12 00:21:59 INFO Vacuumed table table=pending_closures27142026/11/12 00:21:59 INFO Completed push id=1 objects=10 duration=15.630458ms27152026/11/12 00:21:59 INFO Vacuumed table table=pending_objects27162026/11/12 00:21:59 INFO Received uploads request method=POST path=/27172026/11/12 00:21:59 INFO Vacuumed table table=multipart_uploads27182026/11/12 00:21:59 INFO Received push request method=POST path=/27192026/11/12 00:21:59 INFO Received uploads request method=POST path=/27202026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=027212026/11/12 00:21:59 INFO Vacuumed table table=closures27222026/11/12 00:21:59 INFO Received complete upload request method=POST path=/27232026/11/12 00:21:59 INFO Completed upload id=10 objects=6 duration=4.257667ms27242026/11/12 00:21:59 INFO Vacuumed table table=objects27252026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=027262026/11/12 00:21:59 WARN Force mode enabled - objects will be deleted immediately without grace period27272026/11/12 00:21:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027282026/11/12 00:21:59 INFO Received push request method=POST path=/27292026/11/12 00:21:59 INFO Received push request method=POST path=/27302026/11/12 00:21:59 INFO Received complete upload request method=POST path=/27312026/11/12 00:21:59 INFO Vacuumed table table=pending_closures27322026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=027332026/11/12 00:21:59 INFO Vacuumed table table=pending_objects27342026/11/12 00:21:59 INFO Vacuumed table table=multipart_uploads27352026/11/12 00:21:59 INFO Completed upload id=10 objects=23 duration=2.611458ms27362026/11/12 00:21:59 INFO Vacuumed table table=closures27372026/11/12 00:21:59 INFO Vacuumed table table=objects27382026/11/12 00:21:59 INFO Received push request method=POST path=/27392026/11/12 00:21:59 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27402026/11/12 00:21:59 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27412026/11/12 00:21:59 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27422026/11/12 00:21:59 INFO Received uploads request method=POST path=/27432026/11/12 00:21:59 INFO Received complete push request method=POST path=/27442026/11/12 00:21:59 INFO Received uploads request method=POST path=/27452026/11/12 00:21:59 INFO Completed push id=14 objects=6 duration=890.417µs27462026/11/12 00:21:59 INFO Received push request method=POST path=/27472026/11/12 00:21:59 INFO Received uploads request method=POST path=/27482026/11/12 00:21:59 INFO Received complete upload request method=POST path=/27492026/11/12 00:21:59 INFO Completed upload id=3 objects=5 duration=6.053542ms27502026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=027512026/11/12 00:21:59 WARN Force mode enabled - objects will be deleted immediately without grace period27522026/11/12 00:21:59 INFO Received push request method=POST path=/27532026/11/12 00:21:59 INFO Received complete push request method=POST path=/27542026/11/12 00:21:59 INFO Received push request method=POST path=/27552026/11/12 00:21:59 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027562026/11/12 00:21:59 INFO Completed push id=8 objects=6 duration=1.762166ms27572026/11/12 00:21:59 INFO Received complete upload request method=POST path=/27582026/11/12 00:21:59 INFO Received push request method=POST path=/27592026/11/12 00:21:59 INFO Received uploads request method=POST path=/27602026/11/12 00:21:59 INFO Vacuumed table table=pending_closures27612026/11/12 00:21:59 INFO Completed upload id=17 objects=6 duration=2.951958ms27622026/11/12 00:21:59 INFO Vacuumed table table=pending_objects27632026/11/12 00:21:59 INFO Aborted multipart uploads count=0 kept=027642026/11/12 00:21:59 INFO Received push request method=POST path=/27652026/11/12 00:21:59 INFO Vacuumed table table=multipart_uploads27662026/11/12 00:21:59 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027672026/11/12 00:21:59 INFO Vacuumed table table=closures27682026/11/12 00:21:59 INFO Vacuumed table table=objects27692026/11/12 00:22:00 INFO Received complete upload request method=POST path=/27702026/11/12 00:22:00 INFO Vacuumed table table=pending_closures27712026/11/12 00:22:00 INFO Completed upload id=11 objects=6 duration=841µs27722026/11/12 00:22:00 INFO Vacuumed table table=pending_objects27732026/11/12 00:22:00 INFO Received complete push request method=POST path=/27742026/11/12 00:22:00 INFO Completed push id=12 objects=7 duration=1.159458ms27752026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads27762026/11/12 00:22:00 INFO Vacuumed table table=closures27772026/11/12 00:22:00 INFO Received complete push request method=POST path=/27782026/11/12 00:22:00 INFO Vacuumed table table=objects27792026/11/12 00:22:00 INFO Completed push id=10 objects=2 duration=1.906167ms27802026/11/12 00:22:00 INFO Received push request method=POST path=/27812026/11/12 00:22:00 INFO Received push request method=POST path=/27822026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=027832026/11/12 00:22:00 INFO Received push request method=POST path=/27842026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=027852026/11/12 00:22:00 INFO Vacuumed table table=pending_closures27862026/11/12 00:22:00 INFO Vacuumed table table=pending_objects27872026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads27882026/11/12 00:22:00 INFO Vacuumed table table=closures27892026/11/12 00:22:00 INFO Vacuumed table table=objects27902026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=027912026/11/12 00:22:00 INFO Received push request method=POST path=/27922026/11/12 00:22:00 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27932026/11/12 00:22:00 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27942026/11/12 00:22:00 INFO Received complete upload request method=POST path=/27952026/11/12 00:22:00 INFO Received uploads request method=POST path=/27962026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=027972026/11/12 00:22:00 INFO Completed upload id=12 objects=21 duration=2.902125ms27982026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=027992026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028002026/11/12 00:22:00 INFO Vacuumed table table=pending_closures28012026/11/12 00:22:00 INFO Vacuumed table table=pending_objects28022026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads28032026/11/12 00:22:00 INFO Vacuumed table table=pending_closures28042026/11/12 00:22:00 INFO Received complete upload request method=POST path=/28052026/11/12 00:22:00 INFO Vacuumed table table=closures28062026/11/12 00:22:00 INFO Vacuumed table table=objects28072026/11/12 00:22:00 INFO Completed upload id=1 objects=9 duration=10.058792ms28082026/11/12 00:22:00 INFO Received push request method=POST path=/28092026/11/12 00:22:00 INFO Received push request method=POST path=/28102026/11/12 00:22:00 INFO Received push request method=POST path=/28112026/11/12 00:22:00 INFO Vacuumed table table=pending_objects28122026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads28132026/11/12 00:22:00 INFO Received complete push request method=POST path=/28142026/11/12 00:22:00 INFO Vacuumed table table=closures28152026/11/12 00:22:00 INFO Vacuumed table table=objects28162026/11/12 00:22:00 INFO Completed push id=4 objects=3 duration=5.428791ms28172026/11/12 00:22:00 INFO Received push request method=POST path=/28182026/11/12 00:22:00 INFO Received push request method=POST path=/28192026/11/12 00:22:00 INFO Received complete push request method=POST path=/28202026/11/12 00:22:00 INFO Completed push id=12 objects=15 duration=3.198625ms28212026/11/12 00:22:00 INFO Received push request method=POST path=/28222026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=028232026/11/12 00:22:00 WARN Force mode enabled - objects will be deleted immediately without grace period28242026/11/12 00:22:00 INFO Received uploads request method=POST path=/28252026/11/12 00:22:00 INFO Received uploads request method=POST path=/28262026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=028272026/11/12 00:22:00 INFO Received complete upload request method=POST path=/28282026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028292026/11/12 00:22:00 INFO Vacuumed table table=pending_closures28302026/11/12 00:22:00 INFO Completed upload id=3 objects=8 duration=1.010708ms28312026/11/12 00:22:00 INFO Vacuumed table table=pending_objects28322026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads28332026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=028342026/11/12 00:22:00 INFO Vacuumed table table=closures28352026/11/12 00:22:00 INFO Vacuumed table table=objects28362026/11/12 00:22:00 INFO Vacuumed table table=pending_closures28372026/11/12 00:22:00 INFO Received complete push request method=POST path=/28382026/11/12 00:22:00 INFO Completed push id=13 objects=23 duration=19.822125ms28392026/11/12 00:22:00 INFO Vacuumed table table=pending_objects28402026/11/12 00:22:00 INFO Received uploads request method=POST path=/28412026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads28422026/11/12 00:22:00 INFO Vacuumed table table=closures28432026/11/12 00:22:00 INFO Received push request method=POST path=/28442026/11/12 00:22:00 INFO Vacuumed table table=objects28452026/11/12 00:22:00 INFO Received uploads request method=POST path=/28462026/11/12 00:22:00 INFO Received uploads request method=POST path=/28472026/11/12 00:22:00 INFO Received uploads request method=POST path=/28482026/11/12 00:22:00 INFO Received push request method=POST path=/28492026/11/12 00:22:00 INFO Received complete push request method=POST path=/28502026/11/12 00:22:00 INFO Completed push id=12 objects=2 duration=7.555917ms28512026/11/12 00:22:00 INFO Received push request method=POST path=/28522026/11/12 00:22:00 INFO Received push request method=POST path=/28532026/11/12 00:22:00 INFO Received push request method=POST path=/28542026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=028552026/11/12 00:22:00 WARN Force mode enabled - objects will be deleted immediately without grace period28562026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028572026/11/12 00:22:00 INFO Vacuumed table table=pending_closures28582026/11/12 00:22:00 INFO Vacuumed table table=pending_objects28592026/11/12 00:22:00 INFO Received uploads request method=POST path=/28602026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads28612026/11/12 00:22:00 INFO Vacuumed table table=closures28622026/11/12 00:22:00 INFO Vacuumed table table=objects28632026/11/12 00:22:00 INFO Received uploads request method=POST path=/28642026/11/12 00:22:00 INFO Received uploads request method=POST path=/28652026/11/12 00:22:00 INFO Received complete upload request method=POST path=/28662026/11/12 00:22:00 INFO Received push request method=POST path=/28672026/11/12 00:22:00 INFO Received push request method=POST path=/28682026/11/12 00:22:00 INFO Completed upload id=4 objects=14 duration=23.014209ms28692026/11/12 00:22:00 INFO Received complete upload request method=POST path=/28702026/11/12 00:22:00 INFO Received uploads request method=POST path=/28712026/11/12 00:22:00 INFO Completed upload id=2 objects=26 duration=7.522792ms28722026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=028732026/11/12 00:22:00 INFO Received push request method=POST path=/28742026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=028752026/11/12 00:22:00 INFO Received uploads request method=POST path=/28762026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=028772026/11/12 00:22:00 INFO Received uploads request method=POST path=/28782026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028792026/11/12 00:22:00 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028802026/11/12 00:22:00 INFO Vacuumed table table=pending_closures28812026/11/12 00:22:00 INFO Vacuumed table table=pending_closures28822026/11/12 00:22:00 INFO Vacuumed table table=pending_objects28832026/11/12 00:22:00 INFO Vacuumed table table=pending_objects28842026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads28852026/11/12 00:22:00 INFO Received uploads request method=POST path=/28862026/11/12 00:22:00 INFO Vacuumed table table=closures28872026/11/12 00:22:00 INFO Vacuumed table table=multipart_uploads28882026/11/12 00:22:00 INFO Vacuumed table table=objects28892026/11/12 00:22:00 INFO Vacuumed table table=closures28902026/11/12 00:22:00 INFO Vacuumed table table=objects28912026/11/12 00:22:00 INFO Received push request method=POST path=/28922026/11/12 00:22:00 INFO Received uploads request method=POST path=/28932026/11/12 00:22:00 INFO Received complete push request method=POST path=/28942026/11/12 00:22:00 INFO Completed push id=1 objects=5 duration=4.051292ms28952026/11/12 00:22:00 INFO Aborted multipart uploads count=0 kept=028962026/11/12 00:22:01 INFO Received uploads request method=POST path=/28972026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028982026/11/12 00:22:01 INFO Vacuumed table table=pending_closures28992026/11/12 00:22:01 INFO Vacuumed table table=pending_objects29002026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads29012026/11/12 00:22:01 INFO Vacuumed table table=closures29022026/11/12 00:22:01 INFO Vacuumed table table=objects29032026/11/12 00:22:01 INFO Received push request method=POST path=/29042026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=029052026/11/12 00:22:01 WARN Force mode enabled - objects will be deleted immediately without grace period29062026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029072026/11/12 00:22:01 INFO Vacuumed table table=pending_closures29082026/11/12 00:22:01 INFO Vacuumed table table=pending_objects29092026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads29102026/11/12 00:22:01 INFO Vacuumed table table=closures29112026/11/12 00:22:01 INFO Vacuumed table table=objects29122026/11/12 00:22:01 INFO Received push request method=POST path=/29132026/11/12 00:22:01 INFO Received push request method=POST path=/29142026/11/12 00:22:01 INFO Received complete upload request method=POST path=/29152026/11/12 00:22:01 INFO Completed upload id=15 objects=9 duration=8.648ms29162026/11/12 00:22:01 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29172026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29182026/11/12 00:22:01 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."29192026/11/12 00:22:01 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."29202026/11/12 00:22:01 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."29212026/11/12 00:22:01 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."29222026/11/12 00:22:01 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29232026/11/12 00:22:01 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29242026/11/12 00:22:01 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29252026/11/12 00:22:01 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."29262026/11/12 00:22:01 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."29272026/11/12 00:22:01 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29282026/11/12 00:22:01 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29292026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."29302026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."29312026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."29322026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29332026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29342026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."29352026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29362026/11/12 00:22:01 INFO Received push request method=POST path=/29372026/11/12 00:22:01 INFO Received uploads request method=POST path=/29382026/11/12 00:22:01 INFO Received push request method=POST path=/29392026/11/12 00:22:01 INFO Received complete upload request method=POST path=/29402026/11/12 00:22:01 INFO Received push request method=POST path=/29412026/11/12 00:22:01 INFO Completed upload id=1 objects=15 duration=63.854083ms29422026/11/12 00:22:01 INFO Received push request method=POST path=/29432026/11/12 00:22:01 INFO Received push request method=POST path=/29442026/11/12 00:22:01 INFO Received uploads request method=POST path=/29452026/11/12 00:22:01 INFO Received complete push request method=POST path=/29462026/11/12 00:22:01 INFO Completed push id=9 objects=7 duration=1.7755ms29472026/11/12 00:22:01 INFO Received uploads request method=POST path=/29482026/11/12 00:22:01 INFO Received uploads request method=POST path=/29492026/11/12 00:22:01 INFO Received uploads request method=POST path=/29502026/11/12 00:22:01 INFO Received uploads request method=POST path=/29512026/11/12 00:22:01 INFO Received push request method=POST path=/29522026/11/12 00:22:01 INFO Received push request method=POST path=/29532026/11/12 00:22:01 INFO Received push request method=POST path=/29542026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=029552026/11/12 00:22:01 WARN Force mode enabled - objects will be deleted immediately without grace period29562026/11/12 00:22:01 INFO Received uploads request method=POST path=/29572026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=029582026/11/12 00:22:01 WARN Force mode enabled - objects will be deleted immediately without grace period29592026/11/12 00:22:01 INFO Received uploads request method=POST path=/29602026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=029612026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029622026/11/12 00:22:01 INFO Received complete push request method=POST path=/29632026/11/12 00:22:01 INFO Received complete push request method=POST path=/29642026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029652026/11/12 00:22:01 INFO Completed push id=21 objects=17 duration=1.1195ms29662026/11/12 00:22:01 INFO Completed push id=8 objects=13 duration=1.328875ms29672026/11/12 00:22:01 INFO Vacuumed table table=pending_closures29682026/11/12 00:22:01 INFO Vacuumed table table=pending_closures29692026/11/12 00:22:01 INFO Vacuumed table table=pending_objects29702026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads29712026/11/12 00:22:01 INFO Vacuumed table table=pending_objects29722026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads29732026/11/12 00:22:01 INFO Vacuumed table table=closures29742026/11/12 00:22:01 INFO Vacuumed table table=closures29752026/11/12 00:22:01 INFO Vacuumed table table=objects29762026/11/12 00:22:01 INFO Vacuumed table table=objects29772026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=029782026/11/12 00:22:01 INFO Received complete upload request method=POST path=/29792026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029802026/11/12 00:22:01 INFO Completed upload id=10 objects=7 duration=1.573875ms29812026/11/12 00:22:01 INFO Vacuumed table table=pending_closures29822026/11/12 00:22:01 INFO Received uploads request method=POST path=/29832026/11/12 00:22:01 INFO Vacuumed table table=pending_objects29842026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads29852026/11/12 00:22:01 INFO Vacuumed table table=closures29862026/11/12 00:22:01 INFO Received push request method=POST path=/29872026/11/12 00:22:01 INFO Vacuumed table table=objects29882026/11/12 00:22:01 INFO Received complete push request method=POST path=/29892026/11/12 00:22:01 INFO Completed push id=19 objects=7 duration=1.622458ms29902026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=029912026/11/12 00:22:01 INFO Received uploads request method=POST path=/29922026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029932026/11/12 00:22:01 INFO Received complete push request method=POST path=/29942026/11/12 00:22:01 INFO Completed push id=22 objects=3 duration=2.209333ms29952026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=029962026/11/12 00:22:01 INFO Vacuumed table table=pending_closures29972026/11/12 00:22:01 INFO Vacuumed table table=pending_closures29982026/11/12 00:22:01 INFO Vacuumed table table=pending_objects29992026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads30002026/11/12 00:22:01 INFO Vacuumed table table=pending_objects30012026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads30022026/11/12 00:22:01 INFO Vacuumed table table=closures30032026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=030042026/11/12 00:22:01 INFO Vacuumed table table=closures30052026/11/12 00:22:01 INFO Vacuumed table table=objects30062026/11/12 00:22:01 INFO Vacuumed table table=objects30072026/11/12 00:22:01 INFO Received uploads request method=POST path=/30082026/11/12 00:22:01 INFO Received push request method=POST path=/30092026/11/12 00:22:01 INFO Received push request method=POST path=/30102026/11/12 00:22:01 INFO Received complete upload request method=POST path=/30112026/11/12 00:22:01 INFO Completed upload id=15 objects=4 duration=2.648875ms30122026/11/12 00:22:01 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30132026/11/12 00:22:01 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."30142026/11/12 00:22:01 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30152026/11/12 00:22:01 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30162026/11/12 00:22:01 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."30172026/11/12 00:22:01 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30182026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30192026/11/12 00:22:01 INFO Received complete push request method=POST path=/30202026/11/12 00:22:01 INFO Received push request method=POST path=/30212026/11/12 00:22:01 INFO Received complete upload request method=POST path=/30222026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=030232026/11/12 00:22:01 INFO Received uploads request method=POST path=/30242026/11/12 00:22:01 INFO Completed upload id=5 objects=5 duration=5.461708ms30252026/11/12 00:22:01 INFO Received complete push request method=POST path=/30262026/11/12 00:22:01 INFO Completed push id=1 objects=6 duration=24.095125ms30272026/11/12 00:22:01 WARN Force mode enabled - objects will be deleted immediately without grace period30282026/11/12 00:22:01 INFO Completed push id=5 objects=8 duration=12.48675ms30292026/11/12 00:22:01 INFO Received push request method=POST path=/30302026/11/12 00:22:01 INFO Received complete push request method=POST path=/30312026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=030322026/11/12 00:22:01 INFO Received complete push request method=POST path=/30332026/11/12 00:22:01 INFO Completed push id=3 objects=5 duration=7.170542ms30342026/11/12 00:22:01 INFO Completed push id=2 objects=25 duration=5.724542ms30352026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=030362026/11/12 00:22:01 INFO Aborted multipart uploads count=0 kept=030372026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=030382026/11/12 00:22:01 INFO Received complete push request method=POST path=/30392026/11/12 00:22:01 INFO Received uploads request method=POST path=/30402026/11/12 00:22:01 INFO Completed push id=19 objects=8 duration=10.871292ms30412026/11/12 00:22:01 INFO Vacuumed table table=pending_closures30422026/11/12 00:22:01 INFO Received push request method=POST path=/30432026/11/12 00:22:01 INFO Received uploads request method=POST path=/30442026/11/12 00:22:01 INFO Vacuumed table table=pending_objects30452026/11/12 00:22:01 WARN Force mode enabled - objects will be deleted immediately without grace period30462026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads30472026/11/12 00:22:01 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030482026/11/12 00:22:01 INFO Vacuumed table table=pending_closures30492026/11/12 00:22:01 INFO Vacuumed table table=pending_objects30502026/11/12 00:22:01 INFO Vacuumed table table=closures30512026/11/12 00:22:01 INFO Vacuumed table table=multipart_uploads30522026/11/12 00:22:01 INFO Vacuumed table table=objects30532026/11/12 00:22:01 INFO Vacuumed table table=closures30542026/11/12 00:22:01 INFO Vacuumed table table=objects30552026/11/12 00:22:01 INFO Received uploads request method=POST path=/30562026/11/12 00:22:01 INFO Received complete push request method=POST path=/30572026/11/12 00:22:01 INFO Completed push id=1 objects=7 duration=13.643583ms30582026/11/12 00:22:01 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."30592026/11/12 00:22:01 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."30602026/11/12 00:22:01 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30612026/11/12 00:22:01 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30622026/11/12 00:22:01 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."30632026/11/12 00:22:01 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30642026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."30652026/11/12 00:22:01 INFO Received complete upload request method=POST path=/30662026/11/12 00:22:01 INFO Received uploads request method=POST path=/30672026/11/12 00:22:01 INFO Received push request method=POST path=/30682026/11/12 00:22:01 INFO Completed upload id=10 objects=4 duration=2.583ms30692026/11/12 00:22:01 INFO Received uploads request method=POST path=/30702026/11/12 00:22:01 INFO Received uploads request method=POST path=/30712026/11/12 00:22:01 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30722026/11/12 00:22:01 INFO Received complete upload request method=POST path=/30732026/11/12 00:22:01 INFO Received complete upload request method=POST path=/30742026/11/12 00:22:01 INFO Completed upload id=18 objects=4 duration=2.065708ms30752026/11/12 00:22:01 INFO Completed upload id=15 objects=6 duration=3.358167ms30762026/11/12 00:22:01 INFO Received complete upload request method=POST path=/30772026/11/12 00:22:01 INFO Received uploads request method=POST path=/30782026/11/12 00:22:01 INFO Completed upload id=16 objects=3 duration=1.35225ms30792026/11/12 00:22:02 INFO Received push request method=POST path=/30802026/11/12 00:22:02 INFO Received complete push request method=POST path=/30812026/11/12 00:22:02 INFO Completed push id=2 objects=4 duration=1.088041ms30822026/11/12 00:22:02 INFO Received push request method=POST path=/30832026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=030842026/11/12 00:22:02 INFO Received uploads request method=POST path=/30852026/11/12 00:22:02 INFO Received push request method=POST path=/30862026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030872026/11/12 00:22:02 INFO Vacuumed table table=pending_closures30882026/11/12 00:22:02 INFO Vacuumed table table=pending_objects30892026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads30902026/11/12 00:22:02 INFO Vacuumed table table=closures30912026/11/12 00:22:02 INFO Vacuumed table table=objects30922026/11/12 00:22:02 INFO Received push request method=POST path=/30932026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=030942026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=030952026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030962026/11/12 00:22:02 INFO Received push request method=POST path=/30972026/11/12 00:22:02 INFO Vacuumed table table=pending_closures30982026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030992026/11/12 00:22:02 INFO Received push request method=POST path=/31002026/11/12 00:22:02 INFO Vacuumed table table=pending_closures31012026/11/12 00:22:02 INFO Vacuumed table table=pending_objects31022026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads31032026/11/12 00:22:02 INFO Vacuumed table table=pending_objects31042026/11/12 00:22:02 INFO Vacuumed table table=closures31052026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads31062026/11/12 00:22:02 INFO Vacuumed table table=objects31072026/11/12 00:22:02 INFO Vacuumed table table=closures31082026/11/12 00:22:02 INFO Vacuumed table table=objects31092026/11/12 00:22:02 INFO Received push request method=POST path=/31102026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031112026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031122026/11/12 00:22:02 INFO Vacuumed table table=pending_closures31132026/11/12 00:22:02 INFO Vacuumed table table=pending_objects31142026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads31152026/11/12 00:22:02 INFO Vacuumed table table=closures31162026/11/12 00:22:02 INFO Vacuumed table table=objects31172026/11/12 00:22:02 INFO Received uploads request method=POST path=/31182026/11/12 00:22:02 INFO Received push request method=POST path=/31192026/11/12 00:22:02 INFO Received complete upload request method=POST path=/31202026/11/12 00:22:02 INFO Received complete upload request method=POST path=/31212026/11/12 00:22:02 INFO Completed upload id=4 objects=7 duration=1.589084ms31222026/11/12 00:22:02 INFO Completed upload id=12 objects=26 duration=2.118292ms31232026/11/12 00:22:02 INFO Received uploads request method=POST path=/31242026/11/12 00:22:02 INFO Received push request method=POST path=/31252026/11/12 00:22:02 INFO Received push request method=POST path=/31262026/11/12 00:22:02 INFO Received uploads request method=POST path=/31272026/11/12 00:22:02 INFO Received uploads request method=POST path=/31282026/11/12 00:22:02 INFO Received push request method=POST path=/31292026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031302026/11/12 00:22:02 INFO Received uploads request method=POST path=/31312026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031322026/11/12 00:22:02 INFO Vacuumed table table=pending_closures31332026/11/12 00:22:02 INFO Vacuumed table table=pending_objects31342026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads31352026/11/12 00:22:02 INFO Received uploads request method=POST path=/31362026/11/12 00:22:02 INFO Received push request method=POST path=/31372026/11/12 00:22:02 INFO Vacuumed table table=closures31382026/11/12 00:22:02 INFO Vacuumed table table=objects31392026/11/12 00:22:02 INFO Received push request method=POST path=/31402026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031412026/11/12 00:22:02 WARN Force mode enabled - objects will be deleted immediately without grace period31422026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031432026/11/12 00:22:02 INFO Vacuumed table table=pending_closures31442026/11/12 00:22:02 INFO Vacuumed table table=pending_objects31452026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads31462026/11/12 00:22:02 INFO Vacuumed table table=closures31472026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031482026/11/12 00:22:02 INFO Vacuumed table table=objects31492026/11/12 00:22:02 INFO Received push request method=POST path=/31502026/11/12 00:22:02 INFO Received uploads request method=POST path=/31512026/11/12 00:22:02 INFO Received push request method=POST path=/31522026/11/12 00:22:02 INFO Received uploads request method=POST path=/31532026/11/12 00:22:02 INFO Received push request method=POST path=/31542026/11/12 00:22:02 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31552026/11/12 00:22:02 INFO Received uploads request method=POST path=/31562026/11/12 00:22:02 INFO Received push request method=POST path=/31572026/11/12 00:22:02 INFO Received push request method=POST path=/31582026/11/12 00:22:02 INFO Received complete upload request method=POST path=/31592026/11/12 00:22:02 INFO Completed upload id=18 objects=4 duration=1.43825ms31602026/11/12 00:22:02 INFO Received push request method=POST path=/31612026/11/12 00:22:02 INFO Received push request method=POST path=/31622026/11/12 00:22:02 INFO Received uploads request method=POST path=/31632026/11/12 00:22:02 INFO Received uploads request method=POST path=/31642026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031652026/11/12 00:22:02 INFO Received push request method=POST path=/31662026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031672026/11/12 00:22:02 INFO Vacuumed table table=pending_closures31682026/11/12 00:22:02 INFO Vacuumed table table=pending_objects31692026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads31702026/11/12 00:22:02 INFO Vacuumed table table=closures31712026/11/12 00:22:02 INFO Vacuumed table table=objects31722026/11/12 00:22:02 INFO Received uploads request method=POST path=/31732026/11/12 00:22:02 INFO Received uploads request method=POST path=/31742026/11/12 00:22:02 INFO Received uploads request method=POST path=/31752026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031762026/11/12 00:22:02 INFO Received uploads request method=POST path=/31772026/11/12 00:22:02 INFO Received complete upload request method=POST path=/31782026/11/12 00:22:02 INFO Completed upload id=11 objects=14 duration=4.264875ms31792026/11/12 00:22:02 INFO Received push request method=POST path=/31802026/11/12 00:22:02 INFO Received push request method=POST path=/31812026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031822026/11/12 00:22:02 INFO Received complete push request method=POST path=/31832026/11/12 00:22:02 INFO Completed push id=16 objects=27 duration=2.68775ms31842026/11/12 00:22:02 INFO Received push request method=POST path=/31852026/11/12 00:22:02 INFO Received push request method=POST path=/31862026/11/12 00:22:02 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."31872026/11/12 00:22:02 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31882026/11/12 00:22:02 INFO Received push request method=POST path=/31892026/11/12 00:22:02 INFO Received uploads request method=POST path=/31902026/11/12 00:22:02 INFO Received push request method=POST path=/31912026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031922026/11/12 00:22:02 INFO Received complete push request method=POST path=/31932026/11/12 00:22:02 INFO Completed push id=16 objects=8 duration=4.748625ms31942026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=031952026/11/12 00:22:02 INFO Received push request method=POST path=/31962026/11/12 00:22:02 WARN Force mode enabled - objects will be deleted immediately without grace period31972026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031982026/11/12 00:22:02 INFO Vacuumed table table=pending_closures31992026/11/12 00:22:02 INFO Received complete push request method=POST path=/32002026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32012026/11/12 00:22:02 INFO Completed push id=17 objects=6 duration=1.535209ms32022026/11/12 00:22:02 INFO Received complete upload request method=POST path=/32032026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32042026/11/12 00:22:02 INFO Received uploads request method=POST path=/32052026/11/12 00:22:02 INFO Completed upload id=8 objects=3 duration=2.315875ms32062026/11/12 00:22:02 INFO Vacuumed table table=closures32072026/11/12 00:22:02 INFO Received push request method=POST path=/32082026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032092026/11/12 00:22:02 INFO Vacuumed table table=objects32102026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32112026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32122026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32132026/11/12 00:22:02 INFO Vacuumed table table=closures32142026/11/12 00:22:02 INFO Vacuumed table table=objects32152026/11/12 00:22:02 INFO Received uploads request method=POST path=/32162026/11/12 00:22:02 INFO Received push request method=POST path=/32172026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=032182026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=032192026/11/12 00:22:02 WARN Force mode enabled - objects will be deleted immediately without grace period32202026/11/12 00:22:02 WARN Force mode enabled - objects will be deleted immediately without grace period32212026/11/12 00:22:02 INFO Received uploads request method=POST path=/32222026/11/12 00:22:02 INFO Received push request method=POST path=/32232026/11/12 00:22:02 INFO Received push request method=POST path=/32242026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032252026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=032262026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032272026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32282026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=032292026/11/12 00:22:02 INFO Received push request method=POST path=/32302026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032312026/11/12 00:22:02 INFO Received push request method=POST path=/32322026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32332026/11/12 00:22:02 INFO Received uploads request method=POST path=/32342026/11/12 00:22:02 INFO Received push request method=POST path=/32352026/11/12 00:22:02 INFO Received uploads request method=POST path=/32362026/11/12 00:22:02 WARN Force mode enabled - objects will be deleted immediately without grace period32372026/11/12 00:22:02 INFO Received push request method=POST path=/32382026/11/12 00:22:02 INFO Received push request method=POST path=/32392026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32402026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32412026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32422026/11/12 00:22:02 INFO Vacuumed table table=closures32432026/11/12 00:22:02 INFO Vacuumed table table=objects32442026/11/12 00:22:02 INFO Received uploads request method=POST path=/32452026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32462026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32472026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32482026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32492026/11/12 00:22:02 INFO Vacuumed table table=closures32502026/11/12 00:22:02 INFO Vacuumed table table=objects32512026/11/12 00:22:02 INFO Vacuumed table table=closures32522026/11/12 00:22:02 INFO Vacuumed table table=objects32532026/11/12 00:22:02 INFO Received uploads request method=POST path=/32542026/11/12 00:22:02 INFO Received push request method=POST path=/32552026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=032562026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32572026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32582026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32592026/11/12 00:22:02 INFO Vacuumed table table=closures32602026/11/12 00:22:02 INFO Received uploads request method=POST path=/32612026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=032622026/11/12 00:22:02 INFO Vacuumed table table=objects32632026/11/12 00:22:02 INFO Received uploads request method=POST path=/32642026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32652026/11/12 00:22:02 INFO Received push request method=POST path=/32662026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=032672026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032682026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32692026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32702026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32712026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32722026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32732026/11/12 00:22:02 INFO Vacuumed table table=closures32742026/11/12 00:22:02 INFO Received uploads request method=POST path=/32752026/11/12 00:22:02 INFO Vacuumed table table=closures32762026/11/12 00:22:02 INFO Vacuumed table table=objects32772026/11/12 00:22:02 INFO Aborted multipart uploads count=0 kept=032782026/11/12 00:22:02 INFO Vacuumed table table=objects32792026/11/12 00:22:02 WARN Force mode enabled - objects will be deleted immediately without grace period32802026/11/12 00:22:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032812026/11/12 00:22:02 INFO Vacuumed table table=pending_closures32822026/11/12 00:22:02 INFO Vacuumed table table=pending_objects32832026/11/12 00:22:02 INFO Vacuumed table table=multipart_uploads32842026/11/12 00:22:02 INFO Received push request method=POST path=/32852026/11/12 00:22:02 INFO Received uploads request method=POST path=/32862026/11/12 00:22:02 INFO Vacuumed table table=closures32872026/11/12 00:22:02 INFO Vacuumed table table=objects32882026/11/12 00:22:02 INFO Received complete push request method=POST path=/32892026/11/12 00:22:02 INFO Received complete push request method=POST path=/32902026/11/12 00:22:02 INFO Received complete upload request method=POST path=/32912026/11/12 00:22:02 INFO Completed push id=6 objects=5 duration=7.465959ms32922026/11/12 00:22:02 INFO Completed push id=4 objects=4 duration=7.556459ms32932026/11/12 00:22:02 INFO Completed upload id=21 objects=7 duration=2.252333ms32942026/11/12 00:22:03 INFO Received push request method=POST path=/32952026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=032962026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period32972026/11/12 00:22:03 INFO Received uploads request method=POST path=/32982026/11/12 00:22:03 INFO Received complete push request method=POST path=/32992026/11/12 00:22:03 INFO Completed push id=11 objects=6 duration=2.237084ms33002026/11/12 00:22:03 INFO Received complete push request method=POST path=/33012026/11/12 00:22:03 INFO Completed push id=18 objects=10 duration=2.42175ms33022026/11/12 00:22:03 INFO Received complete push request method=POST path=/33032026/11/12 00:22:03 INFO Received complete push request method=POST path=/33042026/11/12 00:22:03 INFO Completed push id=6 objects=6 duration=1.087417ms33052026/11/12 00:22:03 INFO Completed push id=6 objects=7 duration=1.983541ms33062026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=033072026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33082026/11/12 00:22:03 INFO Received uploads request method=POST path=/33092026/11/12 00:22:03 INFO Vacuumed table table=pending_objects33102026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads33112026/11/12 00:22:03 INFO Vacuumed table table=closures33122026/11/12 00:22:03 INFO Vacuumed table table=objects33132026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033142026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period33152026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033162026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33172026/11/12 00:22:03 INFO Vacuumed table table=pending_objects33182026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads33192026/11/12 00:22:03 INFO Received complete push request method=POST path=/33202026/11/12 00:22:03 INFO Vacuumed table table=closures33212026/11/12 00:22:03 INFO Vacuumed table table=objects33222026/11/12 00:22:03 INFO Completed push id=1 objects=11 duration=1.220666ms33232026/11/12 00:22:03 INFO Received uploads request method=POST path=/33242026/11/12 00:22:03 INFO Received push request method=POST path=/33252026/11/12 00:22:03 INFO Received complete push request method=POST path=/33262026/11/12 00:22:03 INFO Completed push id=14 objects=3 duration=1.809375ms33272026/11/12 00:22:03 INFO Received push request method=POST path=/33282026/11/12 00:22:03 INFO Received complete upload request method=POST path=/33292026/11/12 00:22:03 INFO Completed upload id=10 objects=6 duration=1.682791ms33302026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033312026/11/12 00:22:03 INFO Received complete push request method=POST path=/33322026/11/12 00:22:03 INFO Completed push id=6 objects=6 duration=1.326916ms33332026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033342026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33352026/11/12 00:22:03 INFO Vacuumed table table=pending_objects33362026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads33372026/11/12 00:22:03 INFO Vacuumed table table=closures33382026/11/12 00:22:03 INFO Received push request method=POST path=/33392026/11/12 00:22:03 INFO Vacuumed table table=objects33402026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033412026/11/12 00:22:03 INFO Received complete push request method=POST path=/33422026/11/12 00:22:03 INFO Completed push id=18 objects=6 duration=5.262292ms33432026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period33442026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033452026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033462026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33472026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period33482026/11/12 00:22:03 INFO Vacuumed table table=pending_objects33492026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads33502026/11/12 00:22:03 INFO Vacuumed table table=closures33512026/11/12 00:22:03 INFO Vacuumed table table=objects33522026/11/12 00:22:03 INFO Received complete upload request method=POST path=/33532026/11/12 00:22:03 INFO Completed upload id=16 objects=3 duration=1.182542ms33542026/11/12 00:22:03 INFO Received uploads request method=POST path=/33552026/11/12 00:22:03 INFO Received push request method=POST path=/33562026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033572026/11/12 00:22:03 INFO Received push request method=POST path=/33582026/11/12 00:22:03 INFO Received uploads request method=POST path=/33592026/11/12 00:22:03 INFO Received push request method=POST path=/33602026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033612026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33622026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033632026/11/12 00:22:03 INFO Vacuumed table table=pending_objects33642026/11/12 00:22:03 INFO Received complete push request method=POST path=/33652026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads33662026/11/12 00:22:03 INFO Completed push id=5 objects=6 duration=7.627417ms33672026/11/12 00:22:03 INFO Vacuumed table table=closures33682026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033692026/11/12 00:22:03 INFO Received push request method=POST path=/33702026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033712026/11/12 00:22:03 INFO Vacuumed table table=objects33722026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33732026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period33742026/11/12 00:22:03 INFO Vacuumed table table=pending_objects33752026/11/12 00:22:03 INFO Received uploads request method=POST path=/33762026/11/12 00:22:03 INFO Received push request method=POST path=/33772026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads33782026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033792026/11/12 00:22:03 INFO Vacuumed table table=closures33802026/11/12 00:22:03 INFO Received uploads request method=POST path=/33812026/11/12 00:22:03 INFO Received push request method=POST path=/33822026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=033832026/11/12 00:22:03 INFO Vacuumed table table=objects33842026/11/12 00:22:03 INFO Received push request method=POST path=/33852026/11/12 00:22:03 INFO Received push request method=POST path=/33862026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period33872026/11/12 00:22:03 INFO Received complete upload request method=POST path=/33882026/11/12 00:22:03 INFO Completed upload id=27 objects=4 duration=3.37875ms33892026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033902026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33912026/11/12 00:22:03 INFO Vacuumed table table=pending_closures33922026/11/12 00:22:03 INFO Received complete push request method=POST path=/33932026/11/12 00:22:03 INFO Received complete upload request method=POST path=/33942026/11/12 00:22:03 INFO Received uploads request method=POST path=/33952026/11/12 00:22:03 INFO Vacuumed table table=pending_objects33962026/11/12 00:22:03 INFO Completed push id=10 objects=15 duration=1.156709ms33972026/11/12 00:22:03 INFO Completed upload id=20 objects=3 duration=4.098208ms33982026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads33992026/11/12 00:22:03 INFO Vacuumed table table=closures34002026/11/12 00:22:03 INFO Vacuumed table table=pending_objects34012026/11/12 00:22:03 INFO Vacuumed table table=objects34022026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads34032026/11/12 00:22:03 INFO Received push request method=POST path=/34042026/11/12 00:22:03 INFO Received push request method=POST path=/34052026/11/12 00:22:03 INFO Received uploads request method=POST path=/34062026/11/12 00:22:03 INFO Received complete push request method=POST path=/34072026/11/12 00:22:03 INFO Completed push id=13 objects=11 duration=1.457417ms34082026/11/12 00:22:03 INFO Vacuumed table table=closures34092026/11/12 00:22:03 INFO Vacuumed table table=objects34102026/11/12 00:22:03 INFO Received push request method=POST path=/34112026/11/12 00:22:03 INFO Received uploads request method=POST path=/34122026/11/12 00:22:03 INFO Received push request method=POST path=/34132026/11/12 00:22:03 INFO Received uploads request method=POST path=/34142026/11/12 00:22:03 INFO Received uploads request method=POST path=/34152026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=034162026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period34172026/11/12 00:22:03 INFO Received push request method=POST path=/34182026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=034192026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034202026/11/12 00:22:03 INFO Vacuumed table table=pending_closures34212026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=034222026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period34232026/11/12 00:22:03 INFO Received complete upload request method=POST path=/34242026/11/12 00:22:03 INFO Vacuumed table table=pending_objects34252026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034262026/11/12 00:22:03 INFO Completed upload id=14 objects=16 duration=3.439209ms34272026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads34282026/11/12 00:22:03 INFO Vacuumed table table=pending_closures34292026/11/12 00:22:03 INFO Vacuumed table table=closures34302026/11/12 00:22:03 INFO Vacuumed table table=pending_objects34312026/11/12 00:22:03 INFO Received push request method=POST path=/34322026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads34332026/11/12 00:22:03 INFO Vacuumed table table=objects34342026/11/12 00:22:03 INFO Received complete upload request method=POST path=/34352026/11/12 00:22:03 INFO Vacuumed table table=closures34362026/11/12 00:22:03 INFO Completed upload id=24 objects=7 duration=1.704ms34372026/11/12 00:22:03 INFO Vacuumed table table=objects34382026/11/12 00:22:03 INFO Received complete push request method=POST path=/34392026/11/12 00:22:03 INFO Completed push id=7 objects=27 duration=1.262083ms34402026/11/12 00:22:03 INFO Received push request method=POST path=/34412026/11/12 00:22:03 INFO Received push request method=POST path=/34422026/11/12 00:22:03 INFO Received complete upload request method=POST path=/34432026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=034442026/11/12 00:22:03 INFO Completed upload id=6 objects=24 duration=5.623583ms34452026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=034462026/11/12 00:22:03 INFO Aborted multipart uploads count=0 kept=034472026/11/12 00:22:03 INFO Received complete push request method=POST path=/34482026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period34492026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034502026/11/12 00:22:03 INFO Received uploads request method=POST path=/34512026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034522026/11/12 00:22:03 WARN Force mode enabled - objects will be deleted immediately without grace period34532026/11/12 00:22:03 INFO Completed push id=5 objects=9 duration=15.537917ms34542026/11/12 00:22:03 INFO Vacuumed table table=pending_closures34552026/11/12 00:22:03 INFO Received push request method=POST path=/34562026/11/12 00:22:03 INFO Vacuumed table table=pending_closures34572026/11/12 00:22:03 INFO Received uploads request method=POST path=/34582026/11/12 00:22:03 INFO Received push request method=POST path=/34592026/11/12 00:22:03 INFO Vacuumed table table=pending_objects34602026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads34612026/11/12 00:22:03 INFO Received complete push request method=POST path=/34622026/11/12 00:22:03 INFO Vacuumed table table=closures34632026/11/12 00:22:03 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034642026/11/12 00:22:03 INFO Vacuumed table table=pending_objects34652026/11/12 00:22:03 INFO Completed push id=9 objects=4 duration=2.545083ms34662026/11/12 00:22:03 INFO Vacuumed table table=objects34672026/11/12 00:22:03 INFO Vacuumed table table=multipart_uploads34682026/11/12 00:22:03 INFO Received push request method=POST path=/34692026/11/12 00:22:03 INFO Vacuumed table table=closures34702026/11/12 00:22:03 INFO Received push request method=POST path=/34712026/11/12 00:22:03 INFO Vacuumed table table=pending_closures34722026/11/12 00:22:03 INFO Vacuumed table table=objects34732026/11/12 00:22:03 INFO Received push request method=POST path=/34742026/11/12 00:22:04 INFO Received uploads request method=POST path=/34752026/11/12 00:22:04 INFO Vacuumed table table=pending_objects34762026/11/12 00:22:04 INFO Received push request method=POST path=/34772026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads34782026/11/12 00:22:04 INFO Vacuumed table table=closures34792026/11/12 00:22:04 INFO Vacuumed table table=objects34802026/11/12 00:22:04 INFO Received complete push request method=POST path=/34812026/11/12 00:22:04 INFO Received complete upload request method=POST path=/34822026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=034832026/11/12 00:22:04 INFO Completed push id=26 objects=14 duration=3.844875ms34842026/11/12 00:22:04 INFO Received uploads request method=POST path=/34852026/11/12 00:22:04 INFO Completed upload id=11 objects=6 duration=3.091584ms34862026/11/12 00:22:04 INFO Received complete upload request method=POST path=/34872026/11/12 00:22:04 INFO Completed upload id=3 objects=9 duration=3.362833ms34882026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034892026/11/12 00:22:04 INFO Received complete push request method=POST path=/34902026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=034912026/11/12 00:22:04 INFO Received complete push request method=POST path=/34922026/11/12 00:22:04 INFO Completed push id=8 objects=6 duration=2.396458ms34932026/11/12 00:22:04 INFO Received uploads request method=POST path=/34942026/11/12 00:22:04 INFO Received push request method=POST path=/34952026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=034962026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period34972026/11/12 00:22:04 INFO Completed push id=16 objects=14 duration=3.909292ms34982026/11/12 00:22:04 INFO Vacuumed table table=pending_closures34992026/11/12 00:22:04 INFO Received push request method=POST path=/35002026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period35012026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035022026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035032026/11/12 00:22:04 INFO Vacuumed table table=pending_closures35042026/11/12 00:22:04 INFO Received uploads request method=POST path=/35052026/11/12 00:22:04 INFO Vacuumed table table=pending_closures35062026/11/12 00:22:04 INFO Received uploads request method=POST path=/35072026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35082026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35092026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35102026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35112026/11/12 00:22:04 INFO Vacuumed table table=closures35122026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35132026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035142026/11/12 00:22:04 INFO Vacuumed table table=objects35152026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35162026/11/12 00:22:04 INFO Received push request method=POST path=/35172026/11/12 00:22:04 INFO Vacuumed table table=closures35182026/11/12 00:22:04 INFO Received uploads request method=POST path=/35192026/11/12 00:22:04 INFO Vacuumed table table=objects35202026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period35212026/11/12 00:22:04 INFO Received uploads request method=POST path=/35222026/11/12 00:22:04 INFO Vacuumed table table=closures35232026/11/12 00:22:04 INFO Received push request method=POST path=/35242026/11/12 00:22:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."35252026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035262026/11/12 00:22:04 INFO Vacuumed table table=objects35272026/11/12 00:22:04 INFO Received push request method=POST path=/35282026/11/12 00:22:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."35292026/11/12 00:22:04 INFO Vacuumed table table=pending_closures35302026/11/12 00:22:04 INFO Received complete push request method=POST path=/35312026/11/12 00:22:04 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."35322026/11/12 00:22:04 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."35332026/11/12 00:22:04 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."35342026/11/12 00:22:04 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."35352026/11/12 00:22:04 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."35362026/11/12 00:22:04 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35372026/11/12 00:22:04 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."35382026/11/12 00:22:04 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."35392026/11/12 00:22:04 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."35402026/11/12 00:22:04 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."35412026/11/12 00:22:04 INFO Received complete upload request method=POST path=/35422026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35432026/11/12 00:22:04 INFO Completed upload id=9 objects=3 duration=4.035917ms35442026/11/12 00:22:04 INFO Completed push id=17 objects=11 duration=5.980291ms35452026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35462026/11/12 00:22:04 INFO Received complete push request method=POST path=/35472026/11/12 00:22:04 INFO Completed push id=9 objects=5 duration=1.254292ms35482026/11/12 00:22:04 INFO Received complete push request method=POST path=/35492026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035502026/11/12 00:22:04 INFO Vacuumed table table=closures35512026/11/12 00:22:04 INFO Vacuumed table table=objects35522026/11/12 00:22:04 INFO Completed push id=5 objects=6 duration=2.02325ms35532026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035542026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035552026/11/12 00:22:04 INFO Vacuumed table table=pending_closures35562026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period35572026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35582026/11/12 00:22:04 INFO Received push request method=POST path=/35592026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35602026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035612026/11/12 00:22:04 INFO Vacuumed table table=closures35622026/11/12 00:22:04 INFO Received uploads request method=POST path=/35632026/11/12 00:22:04 INFO Received push request method=POST path=/35642026/11/12 00:22:04 INFO Vacuumed table table=pending_closures35652026/11/12 00:22:04 INFO Vacuumed table table=objects35662026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35672026/11/12 00:22:04 INFO Received uploads request method=POST path=/35682026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035692026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035702026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35712026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035722026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035732026/11/12 00:22:04 INFO Received complete push request method=POST path=/35742026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period35752026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period35762026/11/12 00:22:04 INFO Vacuumed table table=closures35772026/11/12 00:22:04 INFO Received push request method=POST path=/35782026/11/12 00:22:04 INFO Vacuumed table table=objects35792026/11/12 00:22:04 INFO Received complete push request method=POST path=/35802026/11/12 00:22:04 INFO Completed push id=4 objects=6 duration=3.250416ms35812026/11/12 00:22:04 INFO Completed push id=23 objects=28 duration=3.777583ms35822026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035832026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035842026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period35852026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=035862026/11/12 00:22:04 INFO Vacuumed table table=pending_closures35872026/11/12 00:22:04 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."35882026/11/12 00:22:04 INFO Received push request method=POST path=/35892026/11/12 00:22:04 INFO Vacuumed table table=pending_closures35902026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35912026/11/12 00:22:04 INFO Vacuumed table table=pending_objects35922026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35932026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads35942026/11/12 00:22:04 INFO Vacuumed table table=closures35952026/11/12 00:22:04 INFO Vacuumed table table=closures35962026/11/12 00:22:04 INFO Vacuumed table table=objects35972026/11/12 00:22:04 INFO Vacuumed table table=objects35982026/11/12 00:22:04 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."35992026/11/12 00:22:04 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."36002026/11/12 00:22:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."36012026/11/12 00:22:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."36022026/11/12 00:22:04 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."36032026/11/12 00:22:04 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."36042026/11/12 00:22:04 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."36052026/11/12 00:22:04 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."36062026/11/12 00:22:04 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36072026/11/12 00:22:04 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."36082026/11/12 00:22:04 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."36092026/11/12 00:22:04 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."36102026/11/12 00:22:04 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."36112026/11/12 00:22:04 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."36122026/11/12 00:22:04 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."36132026/11/12 00:22:04 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."36142026/11/12 00:22:04 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."36152026/11/12 00:22:04 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."36162026/11/12 00:22:04 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36172026/11/12 00:22:04 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36182026/11/12 00:22:04 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."36192026/11/12 00:22:04 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36202026/11/12 00:22:04 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."36212026/11/12 00:22:04 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."36222026/11/12 00:22:04 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."36232026/11/12 00:22:04 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36242026/11/12 00:22:04 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36252026/11/12 00:22:04 INFO Received uploads request method=POST path=/36262026/11/12 00:22:04 INFO Received uploads request method=POST path=/36272026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=036282026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period36292026/11/12 00:22:04 INFO Received push request method=POST path=/36302026/11/12 00:22:04 INFO Received push request method=POST path=/36312026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036322026/11/12 00:22:04 INFO Received push request method=POST path=/36332026/11/12 00:22:04 INFO Received push request method=POST path=/36342026/11/12 00:22:04 INFO Vacuumed table table=pending_closures36352026/11/12 00:22:04 INFO Vacuumed table table=pending_objects36362026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads36372026/11/12 00:22:04 INFO Vacuumed table table=closures36382026/11/12 00:22:04 INFO Vacuumed table table=objects36392026/11/12 00:22:04 INFO Received uploads request method=POST path=/36402026/11/12 00:22:04 INFO Received push request method=POST path=/36412026/11/12 00:22:04 INFO Received complete upload request method=POST path=/36422026/11/12 00:22:04 INFO Completed upload id=26 objects=6 duration=983.209µs36432026/11/12 00:22:04 INFO Received complete upload request method=POST path=/36442026/11/12 00:22:04 INFO Completed upload id=27 objects=8 duration=993.084µs36452026/11/12 00:22:04 INFO Received complete push request method=POST path=/36462026/11/12 00:22:04 INFO Received push request method=POST path=/36472026/11/12 00:22:04 INFO Completed push id=3 objects=21 duration=11.365333ms36482026/11/12 00:22:04 INFO Received push request method=POST path=/36492026/11/12 00:22:04 INFO Received uploads request method=POST path=/36502026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=036512026/11/12 00:22:04 INFO Received push request method=POST path=/36522026/11/12 00:22:04 INFO Received uploads request method=POST path=/36532026/11/12 00:22:04 WARN Force mode enabled - objects will be deleted immediately without grace period36542026/11/12 00:22:04 INFO Received complete upload request method=POST path=/36552026/11/12 00:22:04 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036562026/11/12 00:22:04 INFO Completed upload id=14 objects=8 duration=10.664542ms36572026/11/12 00:22:04 INFO Aborted multipart uploads count=0 kept=036582026/11/12 00:22:04 INFO Vacuumed table table=pending_closures36592026/11/12 00:22:04 INFO Vacuumed table table=pending_objects36602026/11/12 00:22:04 INFO Vacuumed table table=multipart_uploads36612026/11/12 00:22:04 INFO Vacuumed table table=closures36622026/11/12 00:22:04 INFO Vacuumed table table=objects36632026/11/12 00:22:04 INFO Received uploads request method=POST path=/36642026/11/12 00:22:04 INFO Received uploads request method=POST path=/36652026/11/12 00:22:04 INFO Received uploads request method=POST path=/36662026/11/12 00:22:04 INFO Received uploads request method=POST path=/36672026/11/12 00:22:05 INFO Received complete push request method=POST path=/36682026/11/12 00:22:05 INFO Received complete upload request method=POST path=/36692026/11/12 00:22:05 INFO Completed push id=12 objects=3 duration=759.209µs36702026/11/12 00:22:05 INFO Completed upload id=29 objects=5 duration=2.331542ms36712026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=036722026/11/12 00:22:05 INFO Vacuumed table table=pending_closures36732026/11/12 00:22:05 INFO Vacuumed table table=pending_objects36742026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads36752026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=036762026/11/12 00:22:05 INFO Received complete upload request method=POST path=/36772026/11/12 00:22:05 INFO Completed upload id=25 objects=6 duration=2.393667ms36782026/11/12 00:22:05 INFO Vacuumed table table=closures36792026/11/12 00:22:05 INFO Received push request method=POST path=/36802026/11/12 00:22:05 INFO Received uploads request method=POST path=/36812026/11/12 00:22:05 INFO Vacuumed table table=objects36822026/11/12 00:22:05 INFO Received uploads request method=POST path=/36832026/11/12 00:22:05 INFO Received push request method=POST path=/36842026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036852026/11/12 00:22:05 INFO Vacuumed table table=pending_closures36862026/11/12 00:22:05 INFO Vacuumed table table=pending_objects36872026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads36882026/11/12 00:22:05 INFO Vacuumed table table=closures36892026/11/12 00:22:05 INFO Received push request method=POST path=/36902026/11/12 00:22:05 INFO Vacuumed table table=objects36912026/11/12 00:22:05 INFO Received push request method=POST path=/36922026/11/12 00:22:05 INFO Received uploads request method=POST path=/36932026/11/12 00:22:05 INFO Received uploads request method=POST path=/36942026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=036952026/11/12 00:22:05 INFO Received uploads request method=POST path=/36962026/11/12 00:22:05 WARN Force mode enabled - objects will be deleted immediately without grace period36972026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036982026/11/12 00:22:05 INFO Vacuumed table table=pending_closures36992026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37002026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37012026/11/12 00:22:05 INFO Vacuumed table table=closures37022026/11/12 00:22:05 INFO Vacuumed table table=objects37032026/11/12 00:22:05 INFO Received uploads request method=POST path=/37042026/11/12 00:22:05 INFO Received complete upload request method=POST path=/37052026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=037062026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=037072026/11/12 00:22:05 INFO Received push request method=POST path=/37082026/11/12 00:22:05 INFO Received complete upload request method=POST path=/37092026/11/12 00:22:05 INFO Completed upload id=23 objects=18 duration=3.587833ms37102026/11/12 00:22:05 INFO Received uploads request method=POST path=/37112026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037122026/11/12 00:22:05 INFO Received uploads request method=POST path=/37132026/11/12 00:22:05 INFO Vacuumed table table=pending_closures37142026/11/12 00:22:05 INFO Completed upload id=10 objects=2 duration=20.537ms37152026/11/12 00:22:05 WARN Force mode enabled - objects will be deleted immediately without grace period37162026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37172026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37182026/11/12 00:22:05 INFO Received push request method=POST path=/37192026/11/12 00:22:05 INFO Vacuumed table table=closures37202026/11/12 00:22:05 INFO Received push request method=POST path=/37212026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037222026/11/12 00:22:05 INFO Vacuumed table table=objects37232026/11/12 00:22:05 INFO Vacuumed table table=pending_closures37242026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37252026/11/12 00:22:05 INFO Received uploads request method=POST path=/37262026/11/12 00:22:05 INFO Received uploads request method=POST path=/37272026/11/12 00:22:05 INFO Received push request method=POST path=/37282026/11/12 00:22:05 INFO Received complete push request method=POST path=/37292026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37302026/11/12 00:22:05 INFO Vacuumed table table=closures37312026/11/12 00:22:05 INFO Completed push id=11 objects=11 duration=7.989458ms37322026/11/12 00:22:05 INFO Vacuumed table table=objects37332026/11/12 00:22:05 INFO Received push request method=POST path=/37342026/11/12 00:22:05 INFO Received complete upload request method=POST path=/37352026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=037362026/11/12 00:22:05 INFO Completed upload id=25 objects=16 duration=4.255333ms37372026/11/12 00:22:05 INFO Received complete push request method=POST path=/37382026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=037392026/11/12 00:22:05 INFO Completed push id=6 objects=26 duration=3.292625ms37402026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=037412026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=037422026/11/12 00:22:05 INFO Vacuumed table table=pending_closures37432026/11/12 00:22:05 WARN Force mode enabled - objects will be deleted immediately without grace period37442026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37452026/11/12 00:22:05 WARN Force mode enabled - objects will be deleted immediately without grace period37462026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37472026/11/12 00:22:05 INFO Vacuumed table table=closures37482026/11/12 00:22:05 INFO Received complete upload request method=POST path=/37492026/11/12 00:22:05 INFO Vacuumed table table=objects37502026/11/12 00:22:05 INFO Completed upload id=16 objects=23 duration=1.80825ms37512026/11/12 00:22:05 INFO Received push request method=POST path=/37522026/11/12 00:22:05 INFO Received complete push request method=POST path=/37532026/11/12 00:22:05 INFO Completed push id=15 objects=5 duration=1.066833ms37542026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037552026/11/12 00:22:05 INFO Vacuumed table table=pending_closures37562026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37572026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37582026/11/12 00:22:05 INFO Vacuumed table table=closures37592026/11/12 00:22:05 INFO Received complete push request method=POST path=/37602026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037612026/11/12 00:22:05 INFO Completed push id=11 objects=9 duration=1.07475ms37622026/11/12 00:22:05 INFO Vacuumed table table=pending_closures37632026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=037642026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37652026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37662026/11/12 00:22:05 INFO Vacuumed table table=objects37672026/11/12 00:22:05 WARN Force mode enabled - objects will be deleted immediately without grace period37682026/11/12 00:22:05 INFO Vacuumed table table=closures37692026/11/12 00:22:05 INFO Vacuumed table table=objects37702026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037712026/11/12 00:22:05 INFO Vacuumed table table=pending_closures37722026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37732026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37742026/11/12 00:22:05 INFO Vacuumed table table=closures37752026/11/12 00:22:05 INFO Vacuumed table table=objects37762026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=037772026/11/12 00:22:05 WARN Force mode enabled - objects will be deleted immediately without grace period37782026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037792026/11/12 00:22:05 INFO Vacuumed table table=pending_closures37802026/11/12 00:22:05 INFO Vacuumed table table=pending_objects37812026/11/12 00:22:05 INFO Received push request method=POST path=/37822026/11/12 00:22:05 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."37832026/11/12 00:22:05 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."37842026/11/12 00:22:05 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37852026/11/12 00:22:05 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."37862026/11/12 00:22:05 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."37872026/11/12 00:22:05 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."37882026/11/12 00:22:05 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."37892026/11/12 00:22:05 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."37902026/11/12 00:22:05 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."37912026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads37922026/11/12 00:22:05 INFO Received uploads request method=POST path=/37932026/11/12 00:22:05 INFO Vacuumed table table=closures37942026/11/12 00:22:05 INFO Received complete upload request method=POST path=/37952026/11/12 00:22:05 INFO Vacuumed table table=objects37962026/11/12 00:22:05 INFO Completed upload id=15 objects=6 duration=40.79325ms37972026/11/12 00:22:05 INFO Received complete upload request method=POST path=/37982026/11/12 00:22:05 INFO Completed upload id=22 objects=22 duration=3.4195ms37992026/11/12 00:22:05 INFO Received push request method=POST path=/38002026/11/12 00:22:05 INFO Received complete push request method=POST path=/38012026/11/12 00:22:05 INFO Completed push id=15 objects=24 duration=1.101542ms38022026/11/12 00:22:05 INFO Received uploads request method=POST path=/38032026/11/12 00:22:05 INFO Aborted multipart uploads count=0 kept=038042026/11/12 00:22:05 INFO Received uploads request method=POST path=/38052026/11/12 00:22:05 WARN Force mode enabled - objects will be deleted immediately without grace period38062026/11/12 00:22:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038072026/11/12 00:22:05 INFO Vacuumed table table=pending_closures38082026/11/12 00:22:05 INFO Vacuumed table table=pending_objects38092026/11/12 00:22:05 INFO Vacuumed table table=multipart_uploads38102026/11/12 00:22:05 INFO Vacuumed table table=closures38112026/11/12 00:22:05 INFO Vacuumed table table=objects38122026/11/12 00:22:05 INFO Received push request method=POST path=/38132026/11/12 00:22:06 INFO Received uploads request method=POST path=/38142026/11/12 00:22:06 INFO Received complete push request method=POST path=/38152026/11/12 00:22:06 INFO Completed push id=4 objects=6 duration=10.79025ms38162026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038172026/11/12 00:22:06 WARN Force mode enabled - objects will be deleted immediately without grace period38182026/11/12 00:22:06 INFO Received complete push request method=POST path=/38192026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38202026/11/12 00:22:06 INFO Received complete push request method=POST path=/38212026/11/12 00:22:06 INFO Received push request method=POST path=/38222026/11/12 00:22:06 INFO Completed push id=27 objects=25 duration=20.878333ms38232026/11/12 00:22:06 INFO Completed upload id=17 objects=25 duration=21.188791ms38242026/11/12 00:22:06 INFO Completed push id=17 objects=22 duration=20.5155ms38252026/11/12 00:22:06 INFO Received uploads request method=POST path=/38262026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38272026/11/12 00:22:06 INFO Completed upload id=19 objects=2 duration=21.330875ms38282026/11/12 00:22:06 INFO Received push request method=POST path=/38292026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38302026/11/12 00:22:06 INFO Completed upload id=31 objects=3 duration=10.50875ms38312026/11/12 00:22:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=038322026/11/12 00:22:06 INFO Vacuumed table table=pending_closures38332026/11/12 00:22:06 INFO Vacuumed table table=pending_objects38342026/11/12 00:22:06 INFO Vacuumed table table=multipart_uploads38352026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038362026/11/12 00:22:06 INFO Received push request method=POST path=/38372026/11/12 00:22:06 INFO Vacuumed table table=closures38382026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38392026/11/12 00:22:06 INFO Vacuumed table table=objects38402026/11/12 00:22:06 INFO Completed upload id=17 objects=7 duration=6.89575ms38412026/11/12 00:22:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038422026/11/12 00:22:06 INFO Vacuumed table table=pending_closures38432026/11/12 00:22:06 INFO Vacuumed table table=pending_objects38442026/11/12 00:22:06 INFO Vacuumed table table=multipart_uploads38452026/11/12 00:22:06 INFO Vacuumed table table=closures38462026/11/12 00:22:06 INFO Vacuumed table table=objects38472026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038482026/11/12 00:22:06 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038492026/11/12 00:22:06 INFO Vacuumed table table=pending_closures38502026/11/12 00:22:06 INFO Vacuumed table table=pending_objects38512026/11/12 00:22:06 INFO Vacuumed table table=multipart_uploads38522026/11/12 00:22:06 INFO Vacuumed table table=closures38532026/11/12 00:22:06 INFO Vacuumed table table=objects38542026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38552026/11/12 00:22:06 INFO Completed upload id=22 objects=2 duration=8.212959ms38562026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38572026/11/12 00:22:06 INFO Received push request method=POST path=/38582026/11/12 00:22:06 INFO Completed upload id=16 objects=17 duration=8.696667ms38592026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38602026/11/12 00:22:06 INFO Received complete push request method=POST path=/38612026/11/12 00:22:06 INFO Completed push id=26 objects=5 duration=3.435041ms38622026/11/12 00:22:06 INFO Completed upload id=24 objects=7 duration=3.879ms38632026/11/12 00:22:06 INFO Received push request method=POST path=/38642026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38652026/11/12 00:22:06 INFO Received push request method=POST path=/38662026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038672026/11/12 00:22:06 INFO Completed upload id=21 objects=2 duration=9.876542ms38682026/11/12 00:22:06 INFO Received uploads request method=POST path=/38692026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038702026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038712026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038722026/11/12 00:22:06 WARN Force mode enabled - objects will be deleted immediately without grace period38732026/11/12 00:22:06 WARN Force mode enabled - objects will be deleted immediately without grace period38742026/11/12 00:22:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."38752026/11/12 00:22:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38762026/11/12 00:22:06 WARN Force mode enabled - objects will be deleted immediately without grace period38772026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38782026/11/12 00:22:06 INFO Completed upload id=24 objects=14 duration=7.219833ms38792026/11/12 00:22:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038802026/11/12 00:22:06 INFO Vacuumed table table=pending_closures38812026/11/12 00:22:06 INFO Vacuumed table table=pending_objects38822026/11/12 00:22:06 INFO Vacuumed table table=multipart_uploads38832026/11/12 00:22:06 INFO Received complete push request method=POST path=/38842026/11/12 00:22:06 INFO Vacuumed table table=closures38852026/11/12 00:22:06 INFO Received complete push request method=POST path=/38862026/11/12 00:22:06 INFO Completed push id=26 objects=3 duration=1.164458ms38872026/11/12 00:22:06 INFO Vacuumed table table=objects38882026/11/12 00:22:06 INFO Completed push id=11 objects=8 duration=4.344834ms38892026/11/12 00:22:06 INFO Received complete upload request method=POST path=/38902026/11/12 00:22:06 INFO Completed upload id=9 objects=2 duration=1.214125ms38912026/11/12 00:22:06 INFO Received complete push request method=POST path=/38922026/11/12 00:22:06 INFO Completed push id=24 objects=5 duration=2.124166ms38932026/11/12 00:22:06 INFO Received push request method=POST path=/38942026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=038952026/11/12 00:22:06 WARN Force mode enabled - objects will be deleted immediately without grace period38962026/11/12 00:22:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038972026/11/12 00:22:06 INFO Vacuumed table table=pending_closures38982026/11/12 00:22:06 INFO Vacuumed table table=pending_objects38992026/11/12 00:22:06 INFO Vacuumed table table=multipart_uploads39002026/11/12 00:22:06 INFO Vacuumed table table=closures39012026/11/12 00:22:06 INFO Received complete push request method=POST path=/39022026/11/12 00:22:06 INFO Vacuumed table table=objects39032026/11/12 00:22:06 INFO Received complete push request method=POST path=/39042026/11/12 00:22:06 INFO Completed push id=8 objects=3 duration=10.687792ms39052026/11/12 00:22:06 INFO Received complete upload request method=POST path=/39062026/11/12 00:22:06 INFO Received uploads request method=POST path=/39072026/11/12 00:22:06 INFO Completed upload id=28 objects=8 duration=1.713167ms39082026/11/12 00:22:06 INFO Completed push id=8 objects=6 duration=1.95275ms39092026/11/12 00:22:06 INFO Aborted multipart uploads count=0 kept=039102026/11/12 00:22:06 INFO Received uploads request method=POST path=/39112026/11/12 00:22:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039122026/11/12 00:22:06 INFO Received uploads request method=POST path=/39132026/11/12 00:22:06 INFO Vacuumed table table=pending_closures39142026/11/12 00:22:06 INFO Received uploads request method=POST path=/39152026/11/12 00:22:06 INFO Vacuumed table table=pending_objects39162026/11/12 00:22:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39172026/11/12 00:22:06 INFO Vacuumed table table=multipart_uploads39182026/11/12 00:22:06 INFO Received complete upload request method=POST path=/39192026/11/12 00:22:06 INFO Completed upload id=29 objects=10 duration=263.959µs39202026/11/12 00:22:06 INFO Vacuumed table table=closures39212026/11/12 00:22:06 INFO Vacuumed table table=objects39222026/11/12 00:22:06 INFO Received complete push request method=POST path=/39232026/11/12 00:22:06 INFO Completed push id=19 objects=3 duration=1.459125ms39242026/11/12 00:22:06 INFO Received complete upload request method=POST path=/39252026/11/12 00:22:06 INFO Completed upload id=14 objects=6 duration=652.333µs39262026/11/12 00:22:06 INFO Received complete upload request method=POST path=/39272026/11/12 00:22:06 INFO Completed upload id=18 objects=2 duration=3.126875ms39282026/11/12 00:22:06 INFO Received push request method=POST path=/39292026/11/12 00:22:06 INFO Received complete upload request method=POST path=/39302026/11/12 00:22:06 INFO Received push request method=POST path=/39312026/11/12 00:22:06 INFO Received complete upload request method=POST path=/39322026/11/12 00:22:06 INFO Completed upload id=22 objects=5 duration=859.667µs39332026/11/12 00:22:06 INFO Completed upload id=30 objects=20 duration=434.417µs39342026/11/12 00:22:06 INFO Received push request method=POST path=/39352026/11/12 00:22:06 INFO Received push request method=POST path=/39362026/11/12 00:22:06 INFO Received complete push request method=POST path=/39372026/11/12 00:22:06 INFO Completed push id=15 objects=24 duration=1.382834ms39382026/11/12 00:22:06 INFO Received uploads request method=POST path=/39392026/11/12 00:22:06 INFO Received complete push request method=POST path=/39402026/11/12 00:22:06 INFO Completed push id=31 objects=11 duration=499.5µs39412026/11/12 00:22:07 INFO Received push request method=POST path=/39422026/11/12 00:22:07 INFO Received uploads request method=POST path=/39432026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39442026/11/12 00:22:07 INFO Completed upload id=13 objects=10 duration=658.917µs39452026/11/12 00:22:07 INFO Received complete push request method=POST path=/39462026/11/12 00:22:07 INFO Completed push id=13 objects=6 duration=2.134292ms39472026/11/12 00:22:07 INFO Received complete push request method=POST path=/39482026/11/12 00:22:07 INFO Completed push id=28 objects=3 duration=793.167µs39492026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39502026/11/12 00:22:07 INFO Completed upload id=32 objects=2 duration=519.667µs39512026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39522026/11/12 00:22:07 INFO Completed upload id=7 objects=3 duration=501.875µs39532026/11/12 00:22:07 INFO Received uploads request method=POST path=/39542026/11/12 00:22:07 INFO Received complete push request method=POST path=/39552026/11/12 00:22:07 INFO Received push request method=POST path=/39562026/11/12 00:22:07 INFO Received uploads request method=POST path=/39572026/11/12 00:22:07 INFO Completed push id=21 objects=26 duration=938.083µs39582026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=039592026/11/12 00:22:07 WARN Force mode enabled - objects will be deleted immediately without grace period39602026/11/12 00:22:07 INFO Received complete push request method=POST path=/39612026/11/12 00:22:07 INFO Completed push id=29 objects=15 duration=834.209µs39622026/11/12 00:22:07 INFO Received push request method=POST path=/39632026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39642026/11/12 00:22:07 INFO Completed upload id=7 objects=10 duration=3.222459ms39652026/11/12 00:22:07 INFO Received complete push request method=POST path=/39662026/11/12 00:22:07 INFO Completed push id=7 objects=24 duration=1.56375ms39672026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=039682026/11/12 00:22:07 WARN Force mode enabled - objects will be deleted immediately without grace period39692026/11/12 00:22:07 INFO Received complete push request method=POST path=/39702026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039712026/11/12 00:22:07 INFO Completed push id=33 objects=3 duration=456.833µs39722026/11/12 00:22:07 INFO Vacuumed table table=pending_closures39732026/11/12 00:22:07 INFO Vacuumed table table=pending_objects39742026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads39752026/11/12 00:22:07 INFO Vacuumed table table=closures39762026/11/12 00:22:07 INFO Received complete push request method=POST path=/39772026/11/12 00:22:07 INFO Vacuumed table table=objects39782026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39792026/11/12 00:22:07 INFO Completed push id=30 objects=6 duration=2.794667ms39802026/11/12 00:22:07 INFO Received uploads request method=POST path=/39812026/11/12 00:22:07 INFO Completed upload id=13 objects=7 duration=6.250875ms39822026/11/12 00:22:07 INFO Received uploads request method=POST path=/39832026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39842026/11/12 00:22:07 INFO Completed upload id=34 objects=3 duration=5.314833ms39852026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39862026/11/12 00:22:07 INFO Completed upload id=28 objects=17 duration=2.3515ms39872026/11/12 00:22:07 INFO Received complete push request method=POST path=/39882026/11/12 00:22:07 INFO Completed push id=18 objects=23 duration=1.739542ms39892026/11/12 00:22:07 INFO Received uploads request method=POST path=/39902026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39912026/11/12 00:22:07 INFO Completed upload id=25 objects=4 duration=6.070667ms39922026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=039932026/11/12 00:22:07 INFO Vacuumed table table=pending_closures39942026/11/12 00:22:07 INFO Vacuumed table table=pending_objects39952026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads39962026/11/12 00:22:07 INFO Vacuumed table table=closures39972026/11/12 00:22:07 INFO Vacuumed table table=objects39982026/11/12 00:22:07 INFO Received complete upload request method=POST path=/39992026/11/12 00:22:07 INFO Completed upload id=31 objects=3 duration=466.709µs40002026/11/12 00:22:07 INFO Received complete upload request method=POST path=/40012026/11/12 00:22:07 INFO Received uploads request method=POST path=/40022026/11/12 00:22:07 INFO Completed upload id=20 objects=4 duration=7.97875ms40032026/11/12 00:22:07 INFO Received uploads request method=POST path=/40042026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=040052026/11/12 00:22:07 INFO Received push request method=POST path=/40062026/11/12 00:22:07 INFO Received uploads request method=POST path=/40072026/11/12 00:22:07 INFO Received complete push request method=POST path=/40082026/11/12 00:22:07 WARN Force mode enabled - objects will be deleted immediately without grace period40092026/11/12 00:22:07 INFO Completed push id=29 objects=2 duration=5.023125ms40102026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040112026/11/12 00:22:07 INFO Received complete push request method=POST path=/40122026/11/12 00:22:07 INFO Completed push id=9 objects=24 duration=1.995458ms40132026/11/12 00:22:07 INFO Received complete upload request method=POST path=/40142026/11/12 00:22:07 INFO Vacuumed table table=pending_closures40152026/11/12 00:22:07 INFO Completed upload id=19 objects=25 duration=3.812417ms40162026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=040172026/11/12 00:22:07 INFO Vacuumed table table=pending_objects40182026/11/12 00:22:07 INFO Received complete upload request method=POST path=/40192026/11/12 00:22:07 INFO Received complete upload request method=POST path=/40202026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads40212026/11/12 00:22:07 INFO Completed upload id=23 objects=7 duration=2.838708ms40222026/11/12 00:22:07 INFO Completed upload id=6 objects=6 duration=5.283792ms40232026/11/12 00:22:07 WARN Force mode enabled - objects will be deleted immediately without grace period40242026/11/12 00:22:07 INFO Vacuumed table table=closures40252026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040262026/11/12 00:22:07 INFO Received complete push request method=POST path=/40272026/11/12 00:22:07 INFO Vacuumed table table=objects40282026/11/12 00:22:07 INFO Completed push id=31 objects=6 duration=514.333µs40292026/11/12 00:22:07 INFO Vacuumed table table=pending_closures40302026/11/12 00:22:07 INFO Vacuumed table table=pending_objects40312026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads40322026/11/12 00:22:07 INFO Vacuumed table table=closures40332026/11/12 00:22:07 INFO Vacuumed table table=objects40342026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=040352026/11/12 00:22:07 WARN Force mode enabled - objects will be deleted immediately without grace period40362026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040372026/11/12 00:22:07 INFO Vacuumed table table=pending_closures40382026/11/12 00:22:07 INFO Vacuumed table table=pending_objects40392026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads40402026/11/12 00:22:07 INFO Vacuumed table table=closures40412026/11/12 00:22:07 INFO Vacuumed table table=objects40422026/11/12 00:22:07 INFO Received complete push request method=POST path=/40432026/11/12 00:22:07 INFO Completed push id=11 objects=9 duration=5.142917ms40442026/11/12 00:22:07 INFO Received uploads request method=POST path=/40452026/11/12 00:22:07 INFO Received uploads request method=POST path=/40462026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=040472026/11/12 00:22:07 WARN Force mode enabled - objects will be deleted immediately without grace period40482026/11/12 00:22:07 INFO Received complete push request method=POST path=/40492026/11/12 00:22:07 INFO Received complete push request method=POST path=/40502026/11/12 00:22:07 INFO Completed push id=20 objects=8 duration=1.336875ms40512026/11/12 00:22:07 INFO Received complete push request method=POST path=/40522026/11/12 00:22:07 INFO Received push request method=POST path=/40532026/11/12 00:22:07 INFO Received complete upload request method=POST path=/40542026/11/12 00:22:07 INFO Received uploads request method=POST path=/40552026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040562026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=040572026/11/12 00:22:07 INFO Completed push id=32 objects=9 duration=1.395625ms40582026/11/12 00:22:07 INFO Completed push id=14 objects=3 duration=3.647416ms40592026/11/12 00:22:07 INFO Completed upload id=15 objects=3 duration=1.786334ms40602026/11/12 00:22:07 INFO Vacuumed table table=pending_closures40612026/11/12 00:22:07 WARN Force mode enabled - objects will be deleted immediately without grace period40622026/11/12 00:22:07 INFO Vacuumed table table=pending_objects40632026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads40642026/11/12 00:22:07 INFO Received complete push request method=POST path=/40652026/11/12 00:22:07 INFO Vacuumed table table=closures40662026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040672026/11/12 00:22:07 INFO Vacuumed table table=objects40682026/11/12 00:22:07 INFO Completed push id=17 objects=6 duration=5.135375ms40692026/11/12 00:22:07 INFO Received push request method=POST path=/40702026/11/12 00:22:07 INFO Vacuumed table table=pending_closures40712026/11/12 00:22:07 INFO Vacuumed table table=pending_objects40722026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads40732026/11/12 00:22:07 INFO Vacuumed table table=closures40742026/11/12 00:22:07 INFO Vacuumed table table=objects40752026/11/12 00:22:07 INFO Received complete push request method=POST path=/40762026/11/12 00:22:07 INFO Completed push id=13 objects=12 duration=4.106458ms40772026/11/12 00:22:07 INFO Received uploads request method=POST path=/40782026/11/12 00:22:07 INFO Received push request method=POST path=/40792026/11/12 00:22:07 INFO Received complete upload request method=POST path=/40802026/11/12 00:22:07 INFO Completed upload id=10 objects=10 duration=2.7265ms40812026/11/12 00:22:07 INFO Received uploads request method=POST path=/40822026/11/12 00:22:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=040832026/11/12 00:22:07 INFO Received push request method=POST path=/40842026/11/12 00:22:07 INFO Aborted multipart uploads count=0 kept=040852026/11/12 00:22:07 INFO Received push request method=POST path=/40862026/11/12 00:22:07 INFO Vacuumed table table=pending_closures40872026/11/12 00:22:07 INFO Received push request method=POST path=/40882026/11/12 00:22:07 INFO Vacuumed table table=pending_objects40892026/11/12 00:22:07 INFO Received complete push request method=POST path=/40902026/11/12 00:22:07 INFO Received uploads request method=POST path=/40912026/11/12 00:22:07 INFO Vacuumed table table=multipart_uploads40922026/11/12 00:22:07 INFO Completed push id=21 objects=9 duration=1.898334ms40932026/11/12 00:22:07 INFO Received push request method=POST path=/40942026/11/12 00:22:07 INFO Vacuumed table table=closures40952026/11/12 00:22:07 INFO Received complete push request method=POST path=/40962026/11/12 00:22:07 INFO Completed push id=13 objects=2 duration=1.940041ms40972026/11/12 00:22:07 INFO Vacuumed table table=objects40982026/11/12 00:22:07 INFO Received complete upload request method=POST path=/40992026/11/12 00:22:07 INFO Received uploads request method=POST path=/41002026/11/12 00:22:07 INFO Completed upload id=33 objects=9 duration=798.75µs41012026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=041022026/11/12 00:22:08 INFO Vacuumed table table=pending_closures41032026/11/12 00:22:08 INFO Vacuumed table table=pending_objects41042026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads41052026/11/12 00:22:08 INFO Vacuumed table table=closures41062026/11/12 00:22:08 INFO Vacuumed table table=objects41072026/11/12 00:22:08 INFO Received uploads request method=POST path=/41082026/11/12 00:22:08 INFO Received complete upload request method=POST path=/41092026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=041102026/11/12 00:22:08 INFO Completed upload id=27 objects=13 duration=2.405084ms41112026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period41122026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041132026/11/12 00:22:08 INFO Vacuumed table table=pending_closures41142026/11/12 00:22:08 INFO Received uploads request method=POST path=/41152026/11/12 00:22:08 INFO Received push request method=POST path=/41162026/11/12 00:22:08 INFO Vacuumed table table=pending_objects41172026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads41182026/11/12 00:22:08 INFO Vacuumed table table=closures41192026/11/12 00:22:08 INFO Vacuumed table table=objects41202026/11/12 00:22:08 INFO Received push request method=POST path=/41212026/11/12 00:22:08 INFO Received uploads request method=POST path=/41222026/11/12 00:22:08 INFO Received complete upload request method=POST path=/41232026/11/12 00:22:08 INFO Completed upload id=11 objects=7 duration=2.905458ms41242026/11/12 00:22:08 INFO Received complete push request method=POST path=/41252026/11/12 00:22:08 INFO Completed push id=27 objects=11 duration=593.709µs41262026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=041272026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period41282026/11/12 00:22:08 INFO Received complete push request method=POST path=/41292026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041302026/11/12 00:22:08 INFO Completed push id=4 objects=7 duration=2.425875ms41312026/11/12 00:22:08 INFO Vacuumed table table=pending_closures41322026/11/12 00:22:08 INFO Vacuumed table table=pending_objects41332026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads41342026/11/12 00:22:08 INFO Vacuumed table table=closures41352026/11/12 00:22:08 INFO Received push request method=POST path=/41362026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=041372026/11/12 00:22:08 INFO Vacuumed table table=objects41382026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period41392026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041402026/11/12 00:22:08 INFO Received push request method=POST path=/41412026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=041422026/11/12 00:22:08 INFO Received uploads request method=POST path=/41432026/11/12 00:22:08 INFO Vacuumed table table=pending_closures41442026/11/12 00:22:08 INFO Received complete upload request method=POST path=/41452026/11/12 00:22:08 INFO Vacuumed table table=pending_objects41462026/11/12 00:22:08 INFO Completed upload id=24 objects=18 duration=5.9405ms41472026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads41482026/11/12 00:22:08 INFO Vacuumed table table=closures41492026/11/12 00:22:08 INFO Vacuumed table table=objects41502026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041512026/11/12 00:22:08 INFO Received uploads request method=POST path=/41522026/11/12 00:22:08 INFO Vacuumed table table=pending_closures41532026/11/12 00:22:08 INFO Vacuumed table table=pending_objects41542026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads41552026/11/12 00:22:08 INFO Received uploads request method=POST path=/41562026/11/12 00:22:08 INFO Received push request method=POST path=/41572026/11/12 00:22:08 INFO Vacuumed table table=closures41582026/11/12 00:22:08 INFO Received complete upload request method=POST path=/41592026/11/12 00:22:08 INFO Completed upload id=29 objects=26 duration=2.144375ms41602026/11/12 00:22:08 INFO Vacuumed table table=objects41612026/11/12 00:22:08 INFO Received complete push request method=POST path=/41622026/11/12 00:22:08 INFO Completed push id=25 objects=2 duration=2.672ms41632026/11/12 00:22:08 INFO Received push request method=POST path=/41642026/11/12 00:22:08 INFO Received uploads request method=POST path=/41652026/11/12 00:22:08 INFO Received uploads request method=POST path=/41662026/11/12 00:22:08 INFO Received complete push request method=POST path=/41672026/11/12 00:22:08 INFO Received uploads request method=POST path=/41682026/11/12 00:22:08 INFO Completed push id=30 objects=8 duration=2.255833ms41692026/11/12 00:22:08 INFO Received uploads request method=POST path=/41702026/11/12 00:22:08 INFO Received complete push request method=POST path=/41712026/11/12 00:22:08 INFO Completed push id=27 objects=4 duration=679.042µs41722026/11/12 00:22:08 INFO Received push request method=POST path=/41732026/11/12 00:22:08 INFO Received complete push request method=POST path=/41742026/11/12 00:22:08 INFO Received push request method=POST path=/41752026/11/12 00:22:08 INFO Completed push id=14 objects=4 duration=1.995875ms41762026/11/12 00:22:08 INFO Received push request method=POST path=/41772026/11/12 00:22:08 INFO Received uploads request method=POST path=/41782026/11/12 00:22:08 INFO Received complete upload request method=POST path=/41792026/11/12 00:22:08 INFO Completed upload id=31 objects=5 duration=559.584µs41802026/11/12 00:22:08 INFO Received uploads request method=POST path=/41812026/11/12 00:22:08 INFO Received complete push request method=POST path=/41822026/11/12 00:22:08 INFO Completed push id=17 objects=22 duration=1.136458ms41832026/11/12 00:22:08 INFO Received push request method=POST path=/41842026/11/12 00:22:08 INFO Received push request method=POST path=/41852026/11/12 00:22:08 INFO Received complete push request method=POST path=/41862026/11/12 00:22:08 INFO Completed push id=32 objects=3 duration=780.959µs41872026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=041882026/11/12 00:22:08 INFO Received complete upload request method=POST path=/41892026/11/12 00:22:08 INFO Completed upload id=23 objects=15 duration=2.074041ms41902026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period41912026/11/12 00:22:08 INFO Received complete push request method=POST path=/41922026/11/12 00:22:08 INFO Completed push id=14 objects=3 duration=3.148458ms41932026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041942026/11/12 00:22:08 INFO Received push request method=POST path=/41952026/11/12 00:22:08 INFO Received push request method=POST path=/41962026/11/12 00:22:08 INFO Vacuumed table table=pending_closures41972026/11/12 00:22:08 INFO Vacuumed table table=pending_objects41982026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads41992026/11/12 00:22:08 INFO Vacuumed table table=closures42002026/11/12 00:22:08 INFO Vacuumed table table=objects42012026/11/12 00:22:08 INFO Received uploads request method=POST path=/42022026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=042032026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period42042026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042052026/11/12 00:22:08 INFO Vacuumed table table=pending_closures42062026/11/12 00:22:08 INFO Vacuumed table table=pending_objects42072026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads42082026/11/12 00:22:08 INFO Received uploads request method=POST path=/42092026/11/12 00:22:08 INFO Vacuumed table table=closures42102026/11/12 00:22:08 INFO Vacuumed table table=objects42112026/11/12 00:22:08 INFO Received complete push request method=POST path=/42122026/11/12 00:22:08 INFO Completed push id=28 objects=26 duration=2.746084ms42132026/11/12 00:22:08 INFO Received push request method=POST path=/42142026/11/12 00:22:08 INFO Received push request method=POST path=/42152026/11/12 00:22:08 INFO Received complete push request method=POST path=/42162026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=042172026/11/12 00:22:08 INFO Completed push id=19 objects=6 duration=1.729334ms42182026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period42192026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042202026/11/12 00:22:08 INFO Vacuumed table table=pending_closures42212026/11/12 00:22:08 INFO Vacuumed table table=pending_objects42222026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads42232026/11/12 00:22:08 INFO Vacuumed table table=closures42242026/11/12 00:22:08 INFO Vacuumed table table=objects42252026/11/12 00:22:08 INFO Received complete push request method=POST path=/42262026/11/12 00:22:08 INFO Completed push id=30 objects=3 duration=558.458µs42272026/11/12 00:22:08 INFO Received complete upload request method=POST path=/42282026/11/12 00:22:08 INFO Completed upload id=19 objects=6 duration=1.080458ms42292026/11/12 00:22:08 INFO Received uploads request method=POST path=/42302026/11/12 00:22:08 INFO Received uploads request method=POST path=/42312026/11/12 00:22:08 INFO Received push request method=POST path=/42322026/11/12 00:22:08 INFO Received complete push request method=POST path=/42332026/11/12 00:22:08 INFO Completed push id=32 objects=19 duration=1.27075ms42342026/11/12 00:22:08 INFO Received push request method=POST path=/42352026/11/12 00:22:08 INFO Received complete push request method=POST path=/42362026/11/12 00:22:08 INFO Completed push id=16 objects=4 duration=2.138458ms42372026/11/12 00:22:08 INFO Received uploads request method=POST path=/42382026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=042392026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=042402026/11/12 00:22:08 INFO Received complete push request method=POST path=/42412026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period42422026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period42432026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=042442026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=042452026/11/12 00:22:08 INFO Completed push id=7 objects=7 duration=2.32675ms42462026/11/12 00:22:08 INFO Received complete upload request method=POST path=/42472026/11/12 00:22:08 INFO Completed upload id=22 objects=7 duration=925.875µs42482026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042492026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042502026/11/12 00:22:08 INFO Vacuumed table table=pending_closures42512026/11/12 00:22:08 INFO Vacuumed table table=pending_closures42522026/11/12 00:22:08 INFO Vacuumed table table=pending_objects42532026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads42542026/11/12 00:22:08 INFO Vacuumed table table=pending_objects42552026/11/12 00:22:08 INFO Vacuumed table table=closures42562026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads42572026/11/12 00:22:08 INFO Vacuumed table table=objects42582026/11/12 00:22:08 INFO Vacuumed table table=closures42592026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period42602026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period42612026/11/12 00:22:08 INFO Vacuumed table table=objects42622026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042632026/11/12 00:22:08 INFO Received uploads request method=POST path=/42642026/11/12 00:22:08 INFO Vacuumed table table=pending_closures42652026/11/12 00:22:08 INFO Vacuumed table table=pending_objects42662026/11/12 00:22:08 INFO Received complete upload request method=POST path=/42672026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads42682026/11/12 00:22:08 INFO Completed upload id=29 objects=4 duration=1.880125ms42692026/11/12 00:22:08 INFO Vacuumed table table=closures42702026/11/12 00:22:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."42712026/11/12 00:22:08 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."42722026/11/12 00:22:08 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."42732026/11/12 00:22:08 INFO Received push request method=POST path=/42742026/11/12 00:22:08 INFO Received uploads request method=POST path=/42752026/11/12 00:22:08 INFO Vacuumed table table=objects42762026/11/12 00:22:08 INFO Received complete upload request method=POST path=/42772026/11/12 00:22:08 INFO Completed upload id=20 objects=5 duration=616.959µs42782026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=042792026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period42802026/11/12 00:22:08 INFO Received complete push request method=POST path=/42812026/11/12 00:22:08 INFO Completed push id=19 objects=14 duration=1.061625ms42822026/11/12 00:22:08 INFO Received uploads request method=POST path=/42832026/11/12 00:22:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."42842026/11/12 00:22:08 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."42852026/11/12 00:22:08 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."42862026/11/12 00:22:08 INFO Received push request method=POST path=/42872026/11/12 00:22:08 INFO Received uploads request method=POST path=/42882026/11/12 00:22:08 INFO Received uploads request method=POST path=/42892026/11/12 00:22:08 INFO Received complete upload request method=POST path=/42902026/11/12 00:22:08 INFO Completed upload id=13 objects=8 duration=1.697583ms42912026/11/12 00:22:08 INFO Received uploads request method=POST path=/42922026/11/12 00:22:08 INFO Received complete push request method=POST path=/42932026/11/12 00:22:08 INFO Completed push id=24 objects=9 duration=3.925ms42942026/11/12 00:22:08 INFO Received uploads request method=POST path=/42952026/11/12 00:22:08 INFO Received uploads request method=POST path=/42962026/11/12 00:22:08 INFO Received complete upload request method=POST path=/42972026/11/12 00:22:08 INFO Completed upload id=23 objects=16 duration=1.289542ms42982026/11/12 00:22:08 INFO Received uploads request method=POST path=/42992026/11/12 00:22:08 INFO Received push request method=POST path=/43002026/11/12 00:22:08 INFO Aborted multipart uploads count=0 kept=043012026/11/12 00:22:08 WARN Force mode enabled - objects will be deleted immediately without grace period43022026/11/12 00:22:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043032026/11/12 00:22:08 INFO Vacuumed table table=pending_closures43042026/11/12 00:22:08 INFO Vacuumed table table=pending_objects43052026/11/12 00:22:08 INFO Vacuumed table table=multipart_uploads43062026/11/12 00:22:08 INFO Vacuumed table table=closures43072026/11/12 00:22:08 INFO Received complete upload request method=POST path=/43082026/11/12 00:22:08 INFO Vacuumed table table=objects43092026/11/12 00:22:08 INFO Completed upload id=18 objects=11 duration=948.291µs43102026/11/12 00:22:08 INFO Received complete push request method=POST path=/43112026/11/12 00:22:08 INFO Received push request method=POST path=/43122026/11/12 00:22:08 INFO Completed push id=11 objects=6 duration=993.917µs43132026/11/12 00:22:08 INFO Received uploads request method=POST path=/43142026/11/12 00:22:08 INFO Received complete upload request method=POST path=/43152026/11/12 00:22:08 INFO Completed upload id=20 objects=6 duration=743.667µs43162026/11/12 00:22:08 INFO Received complete push request method=POST path=/43172026/11/12 00:22:08 INFO Completed push id=16 objects=16 duration=4.61375ms43182026/11/12 00:22:09 INFO Received uploads request method=POST path=/43192026/11/12 00:22:09 INFO Received push request method=POST path=/43202026/11/12 00:22:09 INFO Received push request method=POST path=/43212026/11/12 00:22:09 INFO Received push request method=POST path=/43222026/11/12 00:22:09 INFO Received uploads request method=POST path=/43232026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=043242026/11/12 00:22:09 WARN Force mode enabled - objects will be deleted immediately without grace period43252026/11/12 00:22:09 INFO Received push request method=POST path=/43262026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043272026/11/12 00:22:09 INFO Received push request method=POST path=/43282026/11/12 00:22:09 INFO Vacuumed table table=pending_closures43292026/11/12 00:22:09 INFO Vacuumed table table=pending_objects43302026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads43312026/11/12 00:22:09 INFO Received complete push request method=POST path=/43322026/11/12 00:22:09 INFO Vacuumed table table=closures43332026/11/12 00:22:09 INFO Vacuumed table table=objects43342026/11/12 00:22:09 INFO Completed push id=20 objects=13 duration=3.953917ms43352026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=043362026/11/12 00:22:09 INFO Received uploads request method=POST path=/43372026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=043382026/11/12 00:22:09 WARN Force mode enabled - objects will be deleted immediately without grace period43392026/11/12 00:22:09 INFO Received complete push request method=POST path=/43402026/11/12 00:22:09 INFO Completed push id=15 objects=7 duration=1.885458ms43412026/11/12 00:22:09 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."43422026/11/12 00:22:09 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."43432026/11/12 00:22:09 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."43442026/11/12 00:22:09 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."43452026/11/12 00:22:09 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43462026/11/12 00:22:09 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."43472026/11/12 00:22:09 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."43482026/11/12 00:22:09 INFO Received complete push request method=POST path=/43492026/11/12 00:22:09 INFO Completed push id=22 objects=4 duration=762.25µs43502026/11/12 00:22:09 INFO Received uploads request method=POST path=/43512026/11/12 00:22:09 INFO Received complete push request method=POST path=/43522026/11/12 00:22:09 INFO Completed push id=13 objects=15 duration=974.625µs43532026/11/12 00:22:09 INFO Received uploads request method=POST path=/43542026/11/12 00:22:09 INFO Received complete upload request method=POST path=/43552026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=043562026/11/12 00:22:09 INFO Completed upload id=21 objects=2 duration=2.268041ms43572026/11/12 00:22:09 INFO Received uploads request method=POST path=/43582026/11/12 00:22:09 WARN Force mode enabled - objects will be deleted immediately without grace period43592026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043602026/11/12 00:22:09 INFO Received uploads request method=POST path=/43612026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=043622026/11/12 00:22:09 INFO Vacuumed table table=pending_closures43632026/11/12 00:22:09 INFO Vacuumed table table=pending_objects43642026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads43652026/11/12 00:22:09 INFO Received uploads request method=POST path=/43662026/11/12 00:22:09 INFO Vacuumed table table=closures43672026/11/12 00:22:09 INFO Vacuumed table table=objects43682026/11/12 00:22:09 INFO Received complete push request method=POST path=/43692026/11/12 00:22:09 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43702026/11/12 00:22:09 INFO Completed push id=5 objects=28 duration=5.407ms43712026/11/12 00:22:09 INFO Received uploads request method=POST path=/43722026/11/12 00:22:09 INFO Received complete upload request method=POST path=/43732026/11/12 00:22:09 INFO Completed upload id=23 objects=7 duration=1.62625ms43742026/11/12 00:22:09 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."43752026/11/12 00:22:09 INFO Received uploads request method=POST path=/43762026/11/12 00:22:09 INFO Received uploads request method=POST path=/43772026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=043782026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043792026/11/12 00:22:09 INFO Vacuumed table table=pending_closures43802026/11/12 00:22:09 INFO Vacuumed table table=pending_objects43812026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads43822026/11/12 00:22:09 INFO Vacuumed table table=closures43832026/11/12 00:22:09 INFO Vacuumed table table=objects43842026/11/12 00:22:09 INFO Received complete push request method=POST path=/43852026/11/12 00:22:09 INFO Completed push id=10 objects=2 duration=991.542µs43862026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=043872026/11/12 00:22:09 WARN Force mode enabled - objects will be deleted immediately without grace period43882026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043892026/11/12 00:22:09 INFO Vacuumed table table=pending_closures43902026/11/12 00:22:09 INFO Vacuumed table table=pending_objects43912026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads43922026/11/12 00:22:09 INFO Vacuumed table table=closures43932026/11/12 00:22:09 INFO Vacuumed table table=objects43942026/11/12 00:22:09 INFO Received uploads request method=POST path=/43952026/11/12 00:22:09 INFO Received complete push request method=POST path=/43962026/11/12 00:22:09 INFO Completed push id=24 objects=5 duration=1.234708ms43972026/11/12 00:22:09 INFO Received uploads request method=POST path=/43982026/11/12 00:22:09 INFO Received push request method=POST path=/43992026/11/12 00:22:09 INFO Received push request method=POST path=/44002026/11/12 00:22:09 INFO Received complete upload request method=POST path=/44012026/11/12 00:22:09 INFO Received complete push request method=POST path=/44022026/11/12 00:22:09 INFO Completed upload id=25 objects=2 duration=914.791µs44032026/11/12 00:22:09 INFO Completed push id=21 objects=3 duration=803.958µs44042026/11/12 00:22:09 INFO Received complete upload request method=POST path=/44052026/11/12 00:22:09 INFO Received complete push request method=POST path=/44062026/11/12 00:22:09 INFO Completed upload id=17 objects=5 duration=622.916µs44072026/11/12 00:22:09 INFO Completed push id=20 objects=3 duration=936µs44082026/11/12 00:22:09 INFO Received push request method=POST path=/44092026/11/12 00:22:09 INFO Received complete push request method=POST path=/44102026/11/12 00:22:09 INFO Completed push id=15 objects=4 duration=11.926625ms44112026/11/12 00:22:09 INFO Received uploads request method=POST path=/44122026/11/12 00:22:09 INFO Received complete upload request method=POST path=/44132026/11/12 00:22:09 INFO Received uploads request method=POST path=/44142026/11/12 00:22:09 INFO Completed upload id=26 objects=20 duration=396.375µs44152026/11/12 00:22:09 INFO Received uploads request method=POST path=/44162026/11/12 00:22:09 INFO Received complete upload request method=POST path=/44172026/11/12 00:22:09 INFO Received uploads request method=POST path=/44182026/11/12 00:22:09 INFO Completed upload id=22 objects=2 duration=848.167µs44192026/11/12 00:22:09 INFO Received push request method=POST path=/44202026/11/12 00:22:09 INFO Received complete push request method=POST path=/44212026/11/12 00:22:09 INFO Completed push id=19 objects=3 duration=738.75µs44222026/11/12 00:22:09 INFO Received push request method=POST path=/44232026/11/12 00:22:09 INFO Received push request method=POST path=/44242026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=044252026/11/12 00:22:09 INFO Received complete push request method=POST path=/44262026/11/12 00:22:09 INFO Completed push id=7 objects=15 duration=8.567958ms44272026/11/12 00:22:09 INFO Received uploads request method=POST path=/44282026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044292026/11/12 00:22:09 INFO Vacuumed table table=pending_closures44302026/11/12 00:22:09 INFO Received complete push request method=POST path=/44312026/11/12 00:22:09 INFO Completed push id=28 objects=7 duration=1.256625ms44322026/11/12 00:22:09 INFO Vacuumed table table=pending_objects44332026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads44342026/11/12 00:22:09 INFO Vacuumed table table=closures44352026/11/12 00:22:09 INFO Vacuumed table table=objects44362026/11/12 00:22:09 INFO Received complete push request method=POST path=/44372026/11/12 00:22:09 INFO Received complete push request method=POST path=/44382026/11/12 00:22:09 INFO Completed push id=24 objects=24 duration=3.094875ms44392026/11/12 00:22:09 INFO Completed push id=18 objects=6 duration=2.73175ms44402026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=044412026/11/12 00:22:09 WARN Force mode enabled - objects will be deleted immediately without grace period44422026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044432026/11/12 00:22:09 INFO Vacuumed table table=pending_closures44442026/11/12 00:22:09 INFO Vacuumed table table=pending_objects44452026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads44462026/11/12 00:22:09 INFO Vacuumed table table=closures44472026/11/12 00:22:09 INFO Vacuumed table table=objects44482026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=044492026/11/12 00:22:09 INFO Received push request method=POST path=/44502026/11/12 00:22:09 INFO Aborted multipart uploads count=0 kept=044512026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044522026/11/12 00:22:09 WARN Force mode enabled - objects will be deleted immediately without grace period44532026/11/12 00:22:09 INFO Vacuumed table table=pending_closures44542026/11/12 00:22:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044552026/11/12 00:22:09 INFO Vacuumed table table=pending_objects44562026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads44572026/11/12 00:22:09 INFO Vacuumed table table=pending_closures44582026/11/12 00:22:09 INFO Vacuumed table table=closures44592026/11/12 00:22:09 INFO Vacuumed table table=pending_objects44602026/11/12 00:22:09 INFO Vacuumed table table=objects44612026/11/12 00:22:09 INFO Vacuumed table table=multipart_uploads44622026/11/12 00:22:09 INFO Vacuumed table table=closures44632026/11/12 00:22:09 INFO Vacuumed table table=objects44642026/11/12 00:22:10 INFO Received complete push request method=POST path=/44652026/11/12 00:22:10 INFO Completed push id=20 objects=4 duration=512.583µs44662026/11/12 00:22:10 INFO Received push request method=POST path=/44672026/11/12 00:22:10 INFO Received complete upload request method=POST path=/44682026/11/12 00:22:10 INFO Completed upload id=23 objects=9 duration=357.334µs44692026/11/12 00:22:10 INFO Received complete upload request method=POST path=/44702026/11/12 00:22:10 INFO Received uploads request method=POST path=/44712026/11/12 00:22:10 INFO Completed upload id=30 objects=6 duration=8.028625ms44722026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=044732026/11/12 00:22:10 INFO Received push request method=POST path=/44742026/11/12 00:22:10 INFO Received push request method=POST path=/44752026/11/12 00:22:10 INFO Received uploads request method=POST path=/44762026/11/12 00:22:10 INFO Received push request method=POST path=/44772026/11/12 00:22:10 WARN Force mode enabled - objects will be deleted immediately without grace period44782026/11/12 00:22:10 INFO Received complete push request method=POST path=/44792026/11/12 00:22:10 INFO Completed push id=31 objects=5 duration=1.314ms44802026/11/12 00:22:10 INFO Received complete push request method=POST path=/44812026/11/12 00:22:10 INFO Received push request method=POST path=/44822026/11/12 00:22:10 INFO Completed push id=14 objects=3 duration=827.375µs44832026/11/12 00:22:10 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."44842026/11/12 00:22:10 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."44852026/11/12 00:22:10 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."44862026/11/12 00:22:10 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."44872026/11/12 00:22:10 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."44882026/11/12 00:22:10 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."44892026/11/12 00:22:10 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."44902026/11/12 00:22:10 INFO Received push request method=POST path=/44912026/11/12 00:22:10 INFO Received push request method=POST path=/44922026/11/12 00:22:10 INFO Received complete upload request method=POST path=/44932026/11/12 00:22:10 INFO Completed upload id=22 objects=8 duration=922.625µs44942026/11/12 00:22:10 INFO Received push request method=POST path=/44952026/11/12 00:22:10 INFO Received push request method=POST path=/44962026/11/12 00:22:10 INFO Received uploads request method=POST path=/44972026/11/12 00:22:10 INFO Received push request method=POST path=/44982026/11/12 00:22:10 INFO Received push request method=POST path=/44992026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45002026/11/12 00:22:10 INFO Completed upload id=22 objects=6 duration=760.792µs45012026/11/12 00:22:10 INFO Received complete push request method=POST path=/45022026/11/12 00:22:10 INFO Completed push id=32 objects=26 duration=1.551875ms45032026/11/12 00:22:10 INFO Received complete push request method=POST path=/45042026/11/12 00:22:10 INFO Received uploads request method=POST path=/45052026/11/12 00:22:10 INFO Completed push id=23 objects=5 duration=1.075083ms45062026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45072026/11/12 00:22:10 INFO Completed upload id=21 objects=3 duration=954.459µs45082026/11/12 00:22:10 INFO Received complete push request method=POST path=/45092026/11/12 00:22:10 INFO Completed push id=30 objects=4 duration=1.388041ms45102026/11/12 00:22:10 INFO Received uploads request method=POST path=/45112026/11/12 00:22:10 INFO Received complete push request method=POST path=/45122026/11/12 00:22:10 INFO Completed push id=31 objects=14 duration=2.931292ms45132026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45142026/11/12 00:22:10 INFO Completed upload id=33 objects=2 duration=754.167µs45152026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=045162026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45172026/11/12 00:22:10 WARN Force mode enabled - objects will be deleted immediately without grace period45182026/11/12 00:22:10 INFO Completed upload id=20 objects=4 duration=1.747041ms45192026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45202026/11/12 00:22:10 INFO Completed upload id=24 objects=8 duration=1.756958ms45212026/11/12 00:22:10 INFO Received uploads request method=POST path=/45222026/11/12 00:22:10 INFO Received push request method=POST path=/45232026/11/12 00:22:10 INFO Received push request method=POST path=/45242026/11/12 00:22:10 INFO Received push request method=POST path=/45252026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45262026/11/12 00:22:10 INFO Completed upload id=34 objects=5 duration=631.625µs45272026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=045282026/11/12 00:22:10 INFO Vacuumed table table=pending_closures45292026/11/12 00:22:10 INFO Vacuumed table table=pending_objects45302026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads45312026/11/12 00:22:10 INFO Vacuumed table table=closures45322026/11/12 00:22:10 INFO Received uploads request method=POST path=/45332026/11/12 00:22:10 INFO Vacuumed table table=objects45342026/11/12 00:22:10 INFO Received uploads request method=POST path=/45352026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=045362026/11/12 00:22:10 INFO Received push request method=POST path=/45372026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045382026/11/12 00:22:10 INFO Vacuumed table table=pending_closures45392026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45402026/11/12 00:22:10 INFO Vacuumed table table=pending_objects45412026/11/12 00:22:10 INFO Completed upload id=28 objects=5 duration=601.334µs45422026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads45432026/11/12 00:22:10 INFO Vacuumed table table=closures45442026/11/12 00:22:10 INFO Received uploads request method=POST path=/45452026/11/12 00:22:10 INFO Vacuumed table table=objects45462026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45472026/11/12 00:22:10 INFO Completed upload id=35 objects=11 duration=781.208µs45482026/11/12 00:22:10 INFO Received push request method=POST path=/45492026/11/12 00:22:10 INFO Received push request method=POST path=/45502026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=045512026/11/12 00:22:10 INFO Received push request method=POST path=/45522026/11/12 00:22:10 WARN Force mode enabled - objects will be deleted immediately without grace period45532026/11/12 00:22:10 INFO Received complete push request method=POST path=/45542026/11/12 00:22:10 INFO Completed push id=16 objects=5 duration=619.834µs45552026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045562026/11/12 00:22:10 INFO Vacuumed table table=pending_closures45572026/11/12 00:22:10 INFO Vacuumed table table=pending_objects45582026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads45592026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45602026/11/12 00:22:10 INFO Received complete push request method=POST path=/45612026/11/12 00:22:10 INFO Completed push id=16 objects=26 duration=1.541666ms45622026/11/12 00:22:10 INFO Vacuumed table table=closures45632026/11/12 00:22:10 INFO Completed upload id=36 objects=3 duration=5.018375ms45642026/11/12 00:22:10 INFO Vacuumed table table=objects45652026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=045662026/11/12 00:22:10 INFO Received push request method=POST path=/45672026/11/12 00:22:10 WARN Force mode enabled - objects will be deleted immediately without grace period45682026/11/12 00:22:10 INFO Received push request method=POST path=/45692026/11/12 00:22:10 INFO Received complete push request method=POST path=/45702026/11/12 00:22:10 INFO Received uploads request method=POST path=/45712026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045722026/11/12 00:22:10 INFO Vacuumed table table=pending_closures45732026/11/12 00:22:10 INFO Completed push id=32 objects=7 duration=6.024583ms45742026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=045752026/11/12 00:22:10 INFO Vacuumed table table=pending_objects45762026/11/12 00:22:10 INFO Received complete upload request method=POST path=/45772026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=045782026/11/12 00:22:10 INFO Completed upload id=30 objects=5 duration=4.187125ms45792026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads45802026/11/12 00:22:10 WARN Force mode enabled - objects will be deleted immediately without grace period45812026/11/12 00:22:10 INFO Vacuumed table table=closures45822026/11/12 00:22:10 INFO Vacuumed table table=objects45832026/11/12 00:22:10 INFO Received uploads request method=POST path=/45842026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045852026/11/12 00:22:10 WARN Force mode enabled - objects will be deleted immediately without grace period45862026/11/12 00:22:10 INFO Received complete push request method=POST path=/45872026/11/12 00:22:10 INFO Completed push id=19 objects=14 duration=1.400209ms45882026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045892026/11/12 00:22:10 INFO Vacuumed table table=pending_closures45902026/11/12 00:22:10 INFO Vacuumed table table=pending_closures45912026/11/12 00:22:10 INFO Vacuumed table table=pending_objects45922026/11/12 00:22:10 INFO Vacuumed table table=pending_objects45932026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads45942026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads45952026/11/12 00:22:10 INFO Received complete push request method=POST path=/45962026/11/12 00:22:10 INFO Vacuumed table table=closures45972026/11/12 00:22:10 INFO Completed push id=26 objects=4 duration=4.292417ms45982026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=045992026/11/12 00:22:10 INFO Vacuumed table table=closures46002026/11/12 00:22:10 INFO Vacuumed table table=objects46012026/11/12 00:22:10 INFO Vacuumed table table=objects46022026/11/12 00:22:10 INFO Received push request method=POST path=/46032026/11/12 00:22:10 INFO Received push request method=POST path=/46042026/11/12 00:22:10 INFO Received uploads request method=POST path=/46052026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=046062026/11/12 00:22:10 INFO Vacuumed table table=pending_closures46072026/11/12 00:22:10 INFO Vacuumed table table=pending_objects46082026/11/12 00:22:10 INFO Received uploads request method=POST path=/46092026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads46102026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=046112026/11/12 00:22:10 INFO Vacuumed table table=closures46122026/11/12 00:22:10 INFO Vacuumed table table=objects46132026/11/12 00:22:10 WARN Force mode enabled - objects will be deleted immediately without grace period46142026/11/12 00:22:10 INFO Received complete push request method=POST path=/46152026/11/12 00:22:10 INFO Completed push id=31 objects=3 duration=5.576791ms46162026/11/12 00:22:10 INFO Received complete push request method=POST path=/46172026/11/12 00:22:10 INFO Completed push id=22 objects=3 duration=2.251ms46182026/11/12 00:22:10 INFO Received uploads request method=POST path=/46192026/11/12 00:22:10 INFO Received complete upload request method=POST path=/46202026/11/12 00:22:10 INFO Completed upload id=28 objects=13 duration=810.959µs46212026/11/12 00:22:10 INFO Aborted multipart uploads count=0 kept=046222026/11/12 00:22:10 INFO Received complete push request method=POST path=/46232026/11/12 00:22:10 INFO Received complete push request method=POST path=/46242026/11/12 00:22:10 INFO Completed push id=7 objects=7 duration=777µs46252026/11/12 00:22:10 INFO Completed push id=27 objects=6 duration=2.903958ms46262026/11/12 00:22:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046272026/11/12 00:22:10 INFO Vacuumed table table=pending_closures46282026/11/12 00:22:10 INFO Vacuumed table table=pending_objects46292026/11/12 00:22:10 INFO Vacuumed table table=multipart_uploads46302026/11/12 00:22:10 INFO Vacuumed table table=closures46312026/11/12 00:22:10 INFO Vacuumed table table=objects46322026/11/12 00:22:10 INFO Received uploads request method=POST path=/46332026/11/12 00:22:11 INFO Received uploads request method=POST path=/46342026/11/12 00:22:11 INFO Received complete upload request method=POST path=/46352026/11/12 00:22:11 INFO Completed upload id=23 objects=7 duration=1.553208ms46362026/11/12 00:22:11 INFO Received uploads request method=POST path=/46372026/11/12 00:22:11 INFO Received complete push request method=POST path=/46382026/11/12 00:22:11 INFO Completed push id=29 objects=3 duration=1.268625ms46392026/11/12 00:22:11 INFO Received uploads request method=POST path=/46402026/11/12 00:22:11 INFO Received complete upload request method=POST path=/46412026/11/12 00:22:11 INFO Completed upload id=26 objects=5 duration=594.583µs46422026/11/12 00:22:11 INFO Received complete push request method=POST path=/46432026/11/12 00:22:11 INFO Received complete upload request method=POST path=/46442026/11/12 00:22:11 INFO Received uploads request method=POST path=/46452026/11/12 00:22:11 INFO Completed push id=12 objects=6 duration=2.368334ms46462026/11/12 00:22:11 INFO Completed upload id=31 objects=2 duration=728.333µs46472026/11/12 00:22:11 INFO Received push request method=POST path=/46482026/11/12 00:22:11 INFO Received complete upload request method=POST path=/46492026/11/12 00:22:11 INFO Completed upload id=15 objects=2 duration=1.028042ms46502026/11/12 00:22:11 INFO Received uploads request method=POST path=/46512026/11/12 00:22:11 INFO Received complete push request method=POST path=/46522026/11/12 00:22:11 INFO Completed push id=32 objects=6 duration=2.972375ms46532026/11/12 00:22:11 INFO Received push request method=POST path=/46542026/11/12 00:22:11 INFO Received complete push request method=POST path=/46552026/11/12 00:22:11 INFO Completed push id=27 objects=3 duration=1.186667ms46562026/11/12 00:22:11 INFO Received push request method=POST path=/46572026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=046582026/11/12 00:22:11 WARN Force mode enabled - objects will be deleted immediately without grace period46592026/11/12 00:22:11 INFO Received complete push request method=POST path=/46602026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046612026/11/12 00:22:11 INFO Completed push id=24 objects=3 duration=1.605125ms46622026/11/12 00:22:11 INFO Received complete upload request method=POST path=/46632026/11/12 00:22:11 INFO Vacuumed table table=pending_closures46642026/11/12 00:22:11 INFO Completed upload id=33 objects=7 duration=3.078834ms46652026/11/12 00:22:11 INFO Received push request method=POST path=/46662026/11/12 00:22:11 INFO Vacuumed table table=pending_objects46672026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads46682026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=046692026/11/12 00:22:11 INFO Vacuumed table table=closures46702026/11/12 00:22:11 INFO Received complete push request method=POST path=/46712026/11/12 00:22:11 INFO Vacuumed table table=objects46722026/11/12 00:22:11 INFO Completed push id=33 objects=6 duration=3.98975ms46732026/11/12 00:22:11 INFO Received complete upload request method=POST path=/46742026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046752026/11/12 00:22:11 INFO Completed upload id=28 objects=5 duration=725.458µs46762026/11/12 00:22:11 INFO Vacuumed table table=pending_closures46772026/11/12 00:22:11 INFO Vacuumed table table=pending_objects46782026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads46792026/11/12 00:22:11 INFO Vacuumed table table=closures46802026/11/12 00:22:11 INFO Vacuumed table table=objects46812026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=046822026/11/12 00:22:11 INFO Vacuumed table table=pending_closures46832026/11/12 00:22:11 INFO Vacuumed table table=pending_objects46842026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads46852026/11/12 00:22:11 INFO Vacuumed table table=closures46862026/11/12 00:22:11 INFO Vacuumed table table=objects46872026/11/12 00:22:11 INFO Received complete upload request method=POST path=/46882026/11/12 00:22:11 INFO Received push request method=POST path=/46892026/11/12 00:22:11 INFO Completed upload id=34 objects=3 duration=2.468542ms46902026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=046912026/11/12 00:22:11 WARN Force mode enabled - objects will be deleted immediately without grace period46922026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046932026/11/12 00:22:11 INFO Vacuumed table table=pending_closures46942026/11/12 00:22:11 INFO Vacuumed table table=pending_objects46952026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads46962026/11/12 00:22:11 INFO Vacuumed table table=closures46972026/11/12 00:22:11 INFO Vacuumed table table=objects46982026/11/12 00:22:11 INFO Received complete push request method=POST path=/46992026/11/12 00:22:11 INFO Completed push id=29 objects=17 duration=1.022917ms47002026/11/12 00:22:11 INFO Received push request method=POST path=/47012026/11/12 00:22:11 INFO Received complete push request method=POST path=/47022026/11/12 00:22:11 INFO Completed push id=35 objects=11 duration=2.021083ms47032026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047042026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047052026/11/12 00:22:11 INFO Received uploads request method=POST path=/47062026/11/12 00:22:11 WARN Force mode enabled - objects will be deleted immediately without grace period47072026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047082026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47092026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047102026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47112026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47122026/11/12 00:22:11 INFO Vacuumed table table=closures47132026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47142026/11/12 00:22:11 INFO Vacuumed table table=objects47152026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47162026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47172026/11/12 00:22:11 INFO Vacuumed table table=closures47182026/11/12 00:22:11 INFO Vacuumed table table=objects47192026/11/12 00:22:11 INFO Received push request method=POST path=/47202026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047212026/11/12 00:22:11 WARN Force mode enabled - objects will be deleted immediately without grace period47222026/11/12 00:22:11 INFO Received uploads request method=POST path=/47232026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047242026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047252026/11/12 00:22:11 WARN Force mode enabled - objects will be deleted immediately without grace period47262026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47272026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047282026/11/12 00:22:11 INFO Received complete push request method=POST path=/47292026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47302026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47312026/11/12 00:22:11 INFO Completed push id=13 objects=22 duration=2.291ms47322026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47332026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47342026/11/12 00:22:11 INFO Received uploads request method=POST path=/47352026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047362026/11/12 00:22:11 INFO Vacuumed table table=closures47372026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47382026/11/12 00:22:11 INFO Vacuumed table table=closures47392026/11/12 00:22:11 INFO Vacuumed table table=objects47402026/11/12 00:22:11 INFO Received push request method=POST path=/47412026/11/12 00:22:11 INFO Vacuumed table table=objects47422026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047432026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47442026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47452026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47462026/11/12 00:22:11 INFO Vacuumed table table=closures47472026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047482026/11/12 00:22:11 INFO Vacuumed table table=objects47492026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047502026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47512026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47522026/11/12 00:22:11 INFO Received complete push request method=POST path=/47532026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47542026/11/12 00:22:11 INFO Vacuumed table table=closures47552026/11/12 00:22:11 INFO Completed push id=21 objects=6 duration=1.344417ms47562026/11/12 00:22:11 INFO Vacuumed table table=objects47572026/11/12 00:22:11 INFO Received uploads request method=POST path=/47582026/11/12 00:22:11 INFO Received push request method=POST path=/47592026/11/12 00:22:11 INFO Received complete upload request method=POST path=/47602026/11/12 00:22:11 INFO Received uploads request method=POST path=/47612026/11/12 00:22:11 INFO Completed upload id=22 objects=7 duration=734.167µs47622026/11/12 00:22:11 INFO Received complete upload request method=POST path=/47632026/11/12 00:22:11 INFO Completed upload id=26 objects=8 duration=961.625µs47642026/11/12 00:22:11 INFO Received complete push request method=POST path=/47652026/11/12 00:22:11 INFO Completed push id=16 objects=7 duration=1.359042ms47662026/11/12 00:22:11 INFO Received complete upload request method=POST path=/47672026/11/12 00:22:11 INFO Completed upload id=33 objects=5 duration=4.381541ms47682026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047692026/11/12 00:22:11 INFO Received push request method=POST path=/47702026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=047712026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47722026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47732026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47742026/11/12 00:22:11 INFO Received complete push request method=POST path=/47752026/11/12 00:22:11 INFO Vacuumed table table=closures47762026/11/12 00:22:11 INFO Vacuumed table table=objects47772026/11/12 00:22:11 INFO Completed push id=28 objects=14 duration=797.291µs47782026/11/12 00:22:11 INFO Received complete push request method=POST path=/47792026/11/12 00:22:11 INFO Completed push id=24 objects=4 duration=2.406958ms47802026/11/12 00:22:11 INFO Received push request method=POST path=/47812026/11/12 00:22:11 INFO Received complete push request method=POST path=/47822026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047832026/11/12 00:22:11 INFO Completed push id=25 objects=9 duration=902.916µs47842026/11/12 00:22:11 INFO Received uploads request method=POST path=/47852026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047862026/11/12 00:22:11 INFO Vacuumed table table=pending_closures47872026/11/12 00:22:11 INFO Vacuumed table table=pending_objects47882026/11/12 00:22:11 INFO Received uploads request method=POST path=/47892026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads47902026/11/12 00:22:11 INFO Vacuumed table table=closures47912026/11/12 00:22:11 INFO Vacuumed table table=objects47922026/11/12 00:22:11 INFO Received complete upload request method=POST path=/47932026/11/12 00:22:11 INFO Completed upload id=26 objects=3 duration=932.417µs47942026/11/12 00:22:11 INFO Received complete upload request method=POST path=/47952026/11/12 00:22:11 INFO Received complete push request method=POST path=/47962026/11/12 00:22:11 INFO Completed upload id=26 objects=2 duration=2.200834ms47972026/11/12 00:22:11 INFO Completed push id=23 objects=6 duration=1.077875ms47982026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=047992026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=048002026/11/12 00:22:11 WARN Force mode enabled - objects will be deleted immediately without grace period48012026/11/12 00:22:11 WARN Force mode enabled - objects will be deleted immediately without grace period48022026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048032026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048042026/11/12 00:22:11 INFO Vacuumed table table=pending_closures48052026/11/12 00:22:11 INFO Vacuumed table table=pending_closures48062026/11/12 00:22:11 INFO Vacuumed table table=pending_objects48072026/11/12 00:22:11 INFO Vacuumed table table=pending_objects48082026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads48092026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads48102026/11/12 00:22:11 INFO Received complete push request method=POST path=/48112026/11/12 00:22:11 INFO Vacuumed table table=closures48122026/11/12 00:22:11 INFO Vacuumed table table=closures48132026/11/12 00:22:11 INFO Completed push id=24 objects=6 duration=5.481208ms48142026/11/12 00:22:11 INFO Aborted multipart uploads count=0 kept=048152026/11/12 00:22:11 INFO Vacuumed table table=objects48162026/11/12 00:22:11 INFO Vacuumed table table=objects48172026/11/12 00:22:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048182026/11/12 00:22:11 INFO Vacuumed table table=pending_closures48192026/11/12 00:22:11 INFO Vacuumed table table=pending_objects48202026/11/12 00:22:11 INFO Vacuumed table table=multipart_uploads48212026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48222026/11/12 00:22:12 INFO Vacuumed table table=closures48232026/11/12 00:22:12 INFO Vacuumed table table=objects48242026/11/12 00:22:12 INFO Completed upload id=19 objects=2 duration=8.830125ms48252026/11/12 00:22:12 INFO Received uploads request method=POST path=/48262026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48272026/11/12 00:22:12 INFO Completed upload id=27 objects=7 duration=514.917µs48282026/11/12 00:22:12 INFO Received uploads request method=POST path=/48292026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48302026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48312026/11/12 00:22:12 INFO Completed upload id=28 objects=3 duration=2.492583ms48322026/11/12 00:22:12 INFO Completed upload id=21 objects=9 duration=3.851292ms48332026/11/12 00:22:12 INFO Received push request method=POST path=/48342026/11/12 00:22:12 INFO Received complete push request method=POST path=/48352026/11/12 00:22:12 INFO Completed push id=33 objects=7 duration=1.178291ms48362026/11/12 00:22:12 INFO Received complete push request method=POST path=/48372026/11/12 00:22:12 INFO Completed push id=25 objects=7 duration=892.5µs48382026/11/12 00:22:12 INFO Received complete push request method=POST path=/48392026/11/12 00:22:12 INFO Completed push id=25 objects=10 duration=1.122583ms48402026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48412026/11/12 00:22:12 INFO Completed upload id=21 objects=25 duration=871.292µs48422026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48432026/11/12 00:22:12 INFO Completed upload id=13 objects=14 duration=2.510791ms48442026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48452026/11/12 00:22:12 INFO Completed upload id=26 objects=3 duration=1.536375ms48462026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48472026/11/12 00:22:12 INFO Received complete push request method=POST path=/48482026/11/12 00:22:12 INFO Completed upload id=29 objects=5 duration=3.01ms48492026/11/12 00:22:12 INFO Completed push id=29 objects=23 duration=1.707209ms48502026/11/12 00:22:12 INFO Received uploads request method=POST path=/48512026/11/12 00:22:12 INFO Aborted multipart uploads count=0 kept=048522026/11/12 00:22:12 INFO Aborted multipart uploads count=0 kept=048532026/11/12 00:22:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048542026/11/12 00:22:12 INFO Vacuumed table table=pending_closures48552026/11/12 00:22:12 INFO Vacuumed table table=pending_objects48562026/11/12 00:22:12 INFO Received complete push request method=POST path=/48572026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48582026/11/12 00:22:12 INFO Vacuumed table table=multipart_uploads48592026/11/12 00:22:12 INFO Completed push id=30 objects=6 duration=954.625µs48602026/11/12 00:22:12 INFO Completed upload id=30 objects=2 duration=4.396125ms48612026/11/12 00:22:12 INFO Vacuumed table table=closures48622026/11/12 00:22:12 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."48632026/11/12 00:22:12 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."48642026/11/12 00:22:12 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48652026/11/12 00:22:12 INFO Vacuumed table table=objects48662026/11/12 00:22:12 INFO Aborted multipart uploads count=0 kept=048672026/11/12 00:22:12 WARN Force mode enabled - objects will be deleted immediately without grace period48682026/11/12 00:22:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048692026/11/12 00:22:12 INFO Vacuumed table table=pending_closures48702026/11/12 00:22:12 INFO Vacuumed table table=pending_objects48712026/11/12 00:22:12 INFO Vacuumed table table=multipart_uploads48722026/11/12 00:22:12 INFO Vacuumed table table=closures48732026/11/12 00:22:12 INFO Vacuumed table table=objects48742026/11/12 00:22:12 INFO Received push request method=POST path=/48752026/11/12 00:22:12 INFO Received complete push request method=POST path=/48762026/11/12 00:22:12 INFO Received complete push request method=POST path=/48772026/11/12 00:22:12 INFO Completed push id=33 objects=3 duration=5.912583ms48782026/11/12 00:22:12 INFO Completed push id=31 objects=20 duration=5.909792ms48792026/11/12 00:22:12 INFO Received complete push request method=POST path=/48802026/11/12 00:22:12 INFO Completed push id=31 objects=6 duration=2.347834ms48812026/11/12 00:22:12 INFO Received complete push request method=POST path=/48822026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48832026/11/12 00:22:12 INFO Completed upload id=26 objects=24 duration=2.287916ms48842026/11/12 00:22:12 INFO Completed push id=34 objects=6 duration=3.527459ms48852026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48862026/11/12 00:22:12 INFO Received uploads request method=POST path=/48872026/11/12 00:22:12 INFO Completed upload id=32 objects=5 duration=881.75µs48882026/11/12 00:22:12 INFO Received push request method=POST path=/48892026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48902026/11/12 00:22:12 INFO Completed upload id=12 objects=6 duration=3.536375ms48912026/11/12 00:22:12 INFO Received complete push request method=POST path=/48922026/11/12 00:22:12 INFO Completed push id=35 objects=8 duration=11.848334ms48932026/11/12 00:22:12 INFO Received complete upload request method=POST path=/48942026/11/12 00:22:12 INFO Completed upload id=33 objects=3 duration=733µs48952026/11/12 00:22:12 INFO Received complete push request method=POST path=/48962026/11/12 00:22:12 INFO Completed push id=28 objects=11 duration=1.45675ms48972026/11/12 00:22:12 INFO Aborted multipart uploads count=0 kept=048982026/11/12 00:22:12 WARN Force mode enabled - objects will be deleted immediately without grace period48992026/11/12 00:22:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049002026/11/12 00:22:12 INFO Received complete push request method=POST path=/49012026/11/12 00:22:12 INFO Vacuumed table table=pending_closures49022026/11/12 00:22:12 INFO Completed push id=17 objects=3 duration=1.095333ms49032026/11/12 00:22:12 INFO Vacuumed table table=pending_objects49042026/11/12 00:22:12 INFO Vacuumed table table=multipart_uploads49052026/11/12 00:22:12 INFO Vacuumed table table=closures49062026/11/12 00:22:12 INFO Vacuumed table table=objects49072026/11/12 00:22:12 INFO Received complete upload request method=POST path=/49082026/11/12 00:22:12 INFO Completed upload id=36 objects=9 duration=6.643667ms49092026/11/12 00:22:12 INFO Aborted multipart uploads count=0 kept=049102026/11/12 00:22:12 INFO Received complete push request method=POST path=/49112026/11/12 00:22:12 WARN Force mode enabled - objects will be deleted immediately without grace period49122026/11/12 00:22:12 INFO Completed push id=23 objects=5 duration=731.75µs49132026/11/12 00:22:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049142026/11/12 00:22:12 INFO Vacuumed table table=pending_closures49152026/11/12 00:22:12 INFO Received complete upload request method=POST path=/49162026/11/12 00:22:12 INFO Completed upload id=24 objects=3 duration=530.916µs49172026/11/12 00:22:12 INFO Vacuumed table table=pending_objects49182026/11/12 00:22:12 INFO Vacuumed table table=multipart_uploads49192026/11/12 00:22:12 INFO Vacuumed table table=closures49202026/11/12 00:22:12 INFO Vacuumed table table=objects49212026/11/12 00:22:12 INFO Received complete push request method=POST path=/49222026/11/12 00:22:12 INFO Completed push id=25 objects=4 duration=407.917µs49232026/11/12 00:22:12 INFO Received complete upload request method=POST path=/49242026/11/12 00:22:12 INFO Completed upload id=27 objects=26 duration=1.3265ms49252026/11/12 00:22:12 INFO Received complete push request method=POST path=/49262026/11/12 00:22:12 INFO Received complete push request method=POST path=/49272026/11/12 00:22:12 INFO Completed push id=28 objects=6 duration=543.167µs49282026/11/12 00:22:12 INFO Completed push id=9 objects=9 duration=4.176542ms49292026/11/12 00:22:12 INFO Received complete push request method=POST path=/49302026/11/12 00:22:12 INFO Completed push id=29 objects=24 duration=1.214916ms49312026/11/12 00:22:12 INFO Received complete push request method=POST path=/49322026/11/12 00:22:12 INFO Completed push id=29 objects=6 duration=335.125µs49332026/11/12 00:22:12 INFO Aborted multipart uploads count=0 kept=049342026/11/12 00:22:12 WARN Force mode enabled - objects will be deleted immediately without grace period49352026/11/12 00:22:12 INFO Received complete upload request method=POST path=/49362026/11/12 00:22:12 INFO Completed upload id=21 objects=8 duration=1.702458ms49372026/11/12 00:22:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049382026/11/12 00:22:12 INFO Vacuumed table table=pending_closures49392026/11/12 00:22:12 INFO Received complete push request method=POST path=/49402026/11/12 00:22:12 INFO Completed push id=22 objects=6 duration=1.227083ms49412026/11/12 00:22:12 INFO Vacuumed table table=pending_objects49422026/11/12 00:22:12 INFO Vacuumed table table=multipart_uploads49432026/11/12 00:22:13 INFO Vacuumed table table=closures49442026/11/12 00:22:13 INFO Vacuumed table table=objects49452026/11/12 00:22:13 INFO Received complete push request method=POST path=/49462026/11/12 00:22:13 INFO Completed push id=24 objects=8 duration=568.667µs49472026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49482026/11/12 00:22:13 INFO Completed upload id=25 objects=9 duration=536.167µs49492026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49502026/11/12 00:22:13 INFO Completed upload id=27 objects=3 duration=729µs49512026/11/12 00:22:13 INFO Received complete push request method=POST path=/49522026/11/12 00:22:13 INFO Completed push id=1 objects=9 duration=766.416µs49532026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49542026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49552026/11/12 00:22:13 INFO Completed upload id=30 objects=8 duration=861.459µs49562026/11/12 00:22:13 INFO Completed upload id=28 objects=6 duration=534.75µs49572026/11/12 00:22:13 INFO Received complete push request method=POST path=/49582026/11/12 00:22:13 INFO Completed push id=26 objects=23 duration=1.381959ms49592026/11/12 00:22:13 INFO Received complete push request method=POST path=/49602026/11/12 00:22:13 INFO Completed push id=30 objects=4 duration=481.542µs49612026/11/12 00:22:13 INFO Received complete push request method=POST path=/49622026/11/12 00:22:13 INFO Completed push id=31 objects=7 duration=415.208µs49632026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49642026/11/12 00:22:13 INFO Completed upload id=31 objects=6 duration=706.584µs49652026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49662026/11/12 00:22:13 INFO Completed upload id=32 objects=5 duration=699.417µs49672026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49682026/11/12 00:22:13 INFO Completed upload id=32 objects=4 duration=525.959µs49692026/11/12 00:22:13 INFO Received complete push request method=POST path=/49702026/11/12 00:22:13 INFO Completed push id=27 objects=15 duration=543.583µs49712026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49722026/11/12 00:22:13 INFO Completed upload id=33 objects=10 duration=734.625µs49732026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49742026/11/12 00:22:13 INFO Completed upload id=34 objects=23 duration=632.375µs49752026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49762026/11/12 00:22:13 INFO Received complete upload request method=POST path=/49772026/11/12 00:22:13 INFO Completed upload id=2 objects=7 duration=685.583µs49782026/11/12 00:22:13 INFO Completed upload id=34 objects=24 duration=1.252292ms49792026/11/12 00:22:13 INFO Aborted multipart uploads count=0 kept=049802026/11/12 00:22:13 WARN Force mode enabled - objects will be deleted immediately without grace period49812026/11/12 00:22:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049822026/11/12 00:22:13 INFO Vacuumed table table=pending_closures49832026/11/12 00:22:13 INFO Received complete push request method=POST path=/49842026/11/12 00:22:13 INFO Completed push id=35 objects=6 duration=971.75µs49852026/11/12 00:22:13 INFO Vacuumed table table=pending_objects49862026/11/12 00:22:13 INFO Vacuumed table table=multipart_uploads49872026/11/12 00:22:13 INFO Vacuumed table table=closures49882026/11/12 00:22:13 INFO Aborted multipart uploads count=0 kept=049892026/11/12 00:22:13 INFO Vacuumed table table=objects49902026/11/12 00:22:13 WARN Force mode enabled - objects will be deleted immediately without grace period49912026/11/12 00:22:13 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049922026/11/12 00:22:13 INFO Vacuumed table table=pending_closures49932026/11/12 00:22:13 INFO Received complete push request method=POST path=/49942026/11/12 00:22:13 INFO Vacuumed table table=pending_objects49952026/11/12 00:22:13 INFO Completed push id=28 objects=19 duration=6.757334ms49962026/11/12 00:22:13 INFO Vacuumed table table=multipart_uploads49972026/11/12 00:22:13 INFO Vacuumed table table=closures49982026/11/12 00:22:13 INFO Vacuumed table table=objects49992026/11/12 00:22:13 INFO Received complete push request method=POST path=/50002026/11/12 00:22:13 INFO Completed push id=29 objects=10 duration=647.666µs50012026/11/12 00:22:13 INFO Received complete upload request method=POST path=/50022026/11/12 00:22:13 INFO Completed upload id=8 objects=6 duration=540.792µs50032026/11/12 00:22:13 INFO Received complete upload request method=POST path=/50042026/11/12 00:22:13 INFO Completed upload id=30 objects=6 duration=644.583µs50052026/11/12 00:22:13 INFO Received complete upload request method=POST path=/50062026/11/12 00:22:13 INFO Completed upload id=31 objects=5 duration=523.459µs50072026/11/12 00:22:13 INFO Received complete upload request method=POST path=/50082026/11/12 00:22:13 INFO Completed upload id=34 objects=3 duration=412.292µs50092026/11/12 00:22:13 INFO Aborted multipart uploads count=0 kept=050102026/11/12 00:22:13 WARN Force mode enabled - objects will be deleted immediately without grace period50112026/11/12 00:22:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050122026/11/12 00:22:13 INFO Vacuumed table table=pending_closures50132026/11/12 00:22:13 INFO Vacuumed table table=pending_objects50142026/11/12 00:22:13 INFO Vacuumed table table=multipart_uploads50152026/11/12 00:22:13 INFO Vacuumed table table=closures50162026/11/12 00:22:13 INFO Vacuumed table table=objects50172026/11/12 00:22:13 INFO Received complete upload request method=POST path=/50182026/11/12 00:22:13 INFO Completed upload id=12 objects=9 duration=686.333µs50192026/11/12 00:22:14 INFO Received complete upload request method=POST path=/50202026/11/12 00:22:14 INFO Completed upload id=18 objects=22 duration=655.291µs50212026/11/12 00:22:14 INFO Received complete upload request method=POST path=/50222026/11/12 00:22:14 INFO Completed upload id=21 objects=8 duration=487.166µs50232026/11/12 00:22:14 INFO Received complete upload request method=POST path=/50242026/11/12 00:22:14 INFO Completed upload id=22 objects=7 duration=581.791µs50252026/11/12 00:22:14 INFO Received complete push request method=POST path=/50262026/11/12 00:22:14 INFO Completed push id=24 objects=5 duration=579.625µs50272026/11/12 00:22:14 INFO Received complete upload request method=POST path=/50282026/11/12 00:22:14 INFO Completed upload id=25 objects=8 duration=564.292µs50292026/11/12 00:22:14 INFO Received complete push request method=POST path=/50302026/11/12 00:22:14 INFO Completed push id=26 objects=4 duration=593.333µs50312026/11/12 00:22:14 INFO Received complete push request method=POST path=/50322026/11/12 00:22:14 INFO Completed push id=30 objects=2 duration=527.25µs50332026/11/12 00:22:14 INFO Received complete push request method=POST path=/50342026/11/12 00:22:14 INFO Completed push id=32 objects=7 duration=672.208µs50352026/11/12 00:22:14 INFO Received complete upload request method=POST path=/50362026/11/12 00:22:14 INFO Completed upload id=33 objects=6 duration=493.875µs50372026/11/12 00:22:14 INFO Received complete upload request method=POST path=/50382026/11/12 00:22:14 INFO Completed upload id=34 objects=3 duration=457.583µs50392026/11/12 00:22:14 INFO Aborted multipart uploads count=0 kept=050402026/11/12 00:22:14 WARN Force mode enabled - objects will be deleted immediately without grace period50412026/11/12 00:22:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050422026/11/12 00:22:14 INFO Vacuumed table table=pending_closures50432026/11/12 00:22:14 INFO Vacuumed table table=pending_objects50442026/11/12 00:22:14 INFO Vacuumed table table=multipart_uploads50452026/11/12 00:22:14 INFO Vacuumed table table=closures50462026/11/12 00:22:14 INFO Vacuumed table table=objects5047--- PASS: FuzzPushGC (0.00s)5048 --- PASS: FuzzPushGC/seed#5 (14.03s)5049 --- PASS: FuzzPushGC/seed#15 (14.05s)5050 --- PASS: FuzzPushGC/seed#0 (14.43s)5051 --- PASS: FuzzPushGC/seed#3 (14.76s)5052 --- PASS: FuzzPushGC/seed#7 (14.91s)5053 --- PASS: FuzzPushGC/seed#11 (17.21s)5054 --- PASS: FuzzPushGC/seed#14 (17.67s)5055 --- PASS: FuzzPushGC/seed#1 (17.77s)5056 --- PASS: FuzzPushGC/seed#8 (17.84s)5057 --- PASS: FuzzPushGC/seed#6 (18.93s)5058 --- PASS: FuzzPushGC/seed#4 (19.05s)5059 --- PASS: FuzzPushGC/seed#2 (19.27s)5060 --- PASS: FuzzPushGC/seed#9 (19.84s)5061 --- PASS: FuzzPushGC/seed#12 (19.88s)5062 --- PASS: FuzzPushGC/seed#13 (20.08s)5063 --- PASS: FuzzPushGC/seed#10 (20.66s)5064PASS5065{"timestamp":"2026-11-12T00:22:14.438883Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49503","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(5)"}5066{"timestamp":"2026-11-12T00:22:14.438893Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49272","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(3)"}5067{"timestamp":"2026-11-12T00:22:14.438897Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49576","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}5068{"timestamp":"2026-11-12T00:22:14.439231Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:49563","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(6)"}50692026-11-12 00:22:14.689 UTC [58585] LOG: received smart shutdown request50702026-11-12 00:22:14.690 UTC [58585] LOG: background worker "logical replication launcher" (PID 58595) exited with exit code 150712026-11-12 00:22:16.182 UTC [58590] LOG: checkpoint complete: wrote 3471 buffers (21.2%), wrote 4 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=33.333 s, sync=1.283 s, total=34.782 s; sync files=42365, longest=0.002 s, average=0.001 s; distance=533045 kB, estimate=533045 kB; lsn=0/278706A0, redo lsn=0/2201A6A050722026-11-12 00:22:16.186 UTC [58590] LOG: shutting down50732026-11-12 00:22:16.186 UTC [58590] LOG: checkpoint starting: shutdown immediate50742026-11-12 00:22:16.899 UTC [58590] LOG: checkpoint complete: wrote 8553 buffers (52.2%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.653 s, sync=0.050 s, total=0.714 s; sync files=2778, longest=0.001 s, average=0.001 s; distance=90456 kB, estimate=488786 kB; lsn=0/27870718, redo lsn=0/2787071850752026-11-12 00:22:16.908 UTC [58585] LOG: database system is shut down5076Running OIDC tests...5077=== RUN TestAudienceForIssuer5078=== PAUSE TestAudienceForIssuer5079=== RUN TestGlobMatch5080=== PAUSE TestGlobMatch5081=== RUN TestValidateToken_ValidToken5082=== PAUSE TestValidateToken_ValidToken5083=== RUN TestValidateToken_WrongAudience5084=== PAUSE TestValidateToken_WrongAudience5085=== RUN TestValidateToken_Expired5086=== PAUSE TestValidateToken_Expired5087=== RUN TestValidateToken_BoundClaimsMismatch5088=== PAUSE TestValidateToken_BoundClaimsMismatch5089=== RUN TestValidateToken_BoundSubjectMismatch5090=== PAUSE TestValidateToken_BoundSubjectMismatch5091=== RUN TestValidateToken_MultipleProviders5092=== PAUSE TestValidateToken_MultipleProviders5093=== RUN TestValidateToken_NoMatchingProvider5094=== PAUSE TestValidateToken_NoMatchingProvider5095=== RUN TestValidateToken_KubernetesServiceAccount5096=== PAUSE TestValidateToken_KubernetesServiceAccount5097=== RUN TestNewValidator_KubernetesRequiresCA5098=== PAUSE TestNewValidator_KubernetesRequiresCA5099=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5100=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5101=== RUN TestPins_ReservedForMatchingRule5102=== PAUSE TestPins_ReservedForMatchingRule5103=== RUN TestPins_TopLevelShorthand5104=== PAUSE TestPins_TopLevelShorthand5105=== RUN TestPins_ConfigValidation5106=== PAUSE TestPins_ConfigValidation5107=== RUN TestScopes_LegacyProviderDefaultsToWrite5108=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5109=== RUN TestScopes_Rules5110=== PAUSE TestScopes_Rules5111=== RUN TestScopes_ConfigValidation5112=== PAUSE TestScopes_ConfigValidation5113=== CONT TestAudienceForIssuer5114--- PASS: TestAudienceForIssuer (0.00s)5115=== CONT TestPins_ConfigValidation5116=== CONT TestPins_ReservedForMatchingRule5117=== CONT TestValidateToken_NoMatchingProvider5118=== CONT TestValidateToken_MultipleProviders5119=== CONT TestValidateToken_BoundSubjectMismatch5120=== CONT TestValidateToken_BoundClaimsMismatch5121=== CONT TestValidateToken_Expired5122=== CONT TestValidateToken_WrongAudience5123--- PASS: TestPins_ConfigValidation (0.01s)5124=== CONT TestValidateToken_ValidToken5125=== CONT TestGlobMatch5126=== RUN TestGlobMatch/foo_foo5127=== PAUSE TestGlobMatch/foo_foo5128=== RUN TestGlobMatch/foo_bar5129=== PAUSE TestGlobMatch/foo_bar5130=== RUN TestGlobMatch/*_5131=== PAUSE TestGlobMatch/*_5132=== RUN TestGlobMatch/*_anything5133=== PAUSE TestGlobMatch/*_anything5134=== RUN TestGlobMatch/foo*_foo5135=== PAUSE TestGlobMatch/foo*_foo5136=== RUN TestGlobMatch/foo*_foobar5137=== PAUSE TestGlobMatch/foo*_foobar5138=== RUN TestGlobMatch/foo*_bar5139=== PAUSE TestGlobMatch/foo*_bar5140=== RUN TestGlobMatch/*bar_bar5141=== PAUSE TestGlobMatch/*bar_bar5142=== RUN TestGlobMatch/*bar_foobar5143=== PAUSE TestGlobMatch/*bar_foobar5144=== RUN TestGlobMatch/*bar_foo5145=== PAUSE TestGlobMatch/*bar_foo5146=== RUN TestGlobMatch/foo*bar_foobar5147=== PAUSE TestGlobMatch/foo*bar_foobar5148=== RUN TestGlobMatch/foo*bar_foo123bar5149=== PAUSE TestGlobMatch/foo*bar_foo123bar5150=== RUN TestGlobMatch/foo*bar_foobarbaz5151=== PAUSE TestGlobMatch/foo*bar_foobarbaz5152=== RUN TestGlobMatch/*/*_foo/bar5153=== PAUSE TestGlobMatch/*/*_foo/bar5154=== RUN TestGlobMatch/*/*_foo5155=== PAUSE TestGlobMatch/*/*_foo5156=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5157=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5158=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05159=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05160=== RUN TestGlobMatch/refs/*/main_refs/heads/main5161=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5162=== RUN TestGlobMatch/fo?_foo5163=== PAUSE TestGlobMatch/fo?_foo5164=== RUN TestGlobMatch/fo?_fo5165=== PAUSE TestGlobMatch/fo?_fo5166=== RUN TestGlobMatch/fo?_fooo5167=== PAUSE TestGlobMatch/fo?_fooo5168=== RUN TestGlobMatch/?oo_foo5169=== PAUSE TestGlobMatch/?oo_foo5170=== RUN TestGlobMatch/?oo_boo5171=== PAUSE TestGlobMatch/?oo_boo5172=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5173=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5174=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5175=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5176=== CONT TestGlobMatch/foo_foo5177=== CONT TestScopes_Rules5178=== CONT TestGlobMatch/*/*_foo/bar5179=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5180=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5181=== CONT TestGlobMatch/?oo_boo5182=== CONT TestGlobMatch/?oo_foo5183=== CONT TestGlobMatch/fo?_fooo5184=== CONT TestGlobMatch/fo?_fo5185=== CONT TestGlobMatch/fo?_foo5186=== CONT TestGlobMatch/refs/*/main_refs/heads/main5187=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05188=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5189=== CONT TestGlobMatch/*/*_foo5190=== CONT TestGlobMatch/*bar_bar5191=== CONT TestGlobMatch/foo*bar_foobarbaz5192=== CONT TestGlobMatch/foo*bar_foo123bar5193=== CONT TestGlobMatch/foo*bar_foobar5194=== CONT TestGlobMatch/*bar_foo5195=== CONT TestGlobMatch/*bar_foobar5196=== CONT TestGlobMatch/foo*_foo5197=== CONT TestGlobMatch/foo*_bar5198=== CONT TestGlobMatch/foo*_foobar5199=== CONT TestGlobMatch/*_5200=== CONT TestGlobMatch/*_anything5201=== CONT TestGlobMatch/foo_bar5202--- PASS: TestGlobMatch (0.00s)5203 --- PASS: TestGlobMatch/foo_foo (0.00s)5204 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5205 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5206 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5207 --- PASS: TestGlobMatch/?oo_boo (0.00s)5208 --- PASS: TestGlobMatch/?oo_foo (0.00s)5209 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5210 --- PASS: TestGlobMatch/fo?_fo (0.00s)5211 --- PASS: TestGlobMatch/fo?_foo (0.00s)5212 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5213 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5214 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5215 --- PASS: TestGlobMatch/*/*_foo (0.00s)5216 --- PASS: TestGlobMatch/*bar_bar (0.00s)5217 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5218 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5219 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5220 --- PASS: TestGlobMatch/*bar_foo (0.00s)5221 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5222 --- PASS: TestGlobMatch/foo*_foo (0.00s)5223 --- PASS: TestGlobMatch/foo*_bar (0.00s)5224 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5225 --- PASS: TestGlobMatch/*_ (0.00s)5226 --- PASS: TestGlobMatch/*_anything (0.00s)5227 --- PASS: TestGlobMatch/foo_bar (0.00s)5228=== CONT TestValidateToken_KubernetesServiceAccount5229=== CONT TestScopes_ConfigValidation5230--- PASS: TestScopes_ConfigValidation (0.00s)5231=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5232=== CONT TestScopes_LegacyProviderDefaultsToWrite5233=== CONT TestNewValidator_KubernetesRequiresCA5234=== CONT TestPins_TopLevelShorthand52352026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50343/oidc5236--- PASS: TestValidateToken_WrongAudience (0.06s)52372026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50345/oidc52382026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50348/oidc52392026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50346/oidc5240--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.08s)52412026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50351/oidc5242--- PASS: TestValidateToken_BoundClaimsMismatch (0.10s)5243--- PASS: TestPins_ReservedForMatchingRule (0.10s)5244--- PASS: TestScopes_Rules (0.10s)52452026/11/12 00:22:19 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235246--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.14s)52472026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50357/oidc52482026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50356/oidc5249--- PASS: TestValidateToken_Expired (0.23s)5250--- PASS: TestValidateToken_BoundSubjectMismatch (0.23s)52512026/11/12 00:22:19 http: TLS handshake error from 127.0.0.1:50363: read tcp 127.0.0.1:50359->127.0.0.1:50363: use of closed network connection5252--- PASS: TestNewValidator_KubernetesRequiresCA (0.22s)52532026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50365/oidc52542026/11/12 00:22:19 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:503605255--- PASS: TestPins_TopLevelShorthand (0.21s)5256--- PASS: TestValidateToken_KubernetesServiceAccount (0.23s)52572026/11/12 00:22:19 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:50368/oidc5258--- PASS: TestValidateToken_ValidToken (0.24s)52592026/11/12 00:22:19 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:50358/oidc52602026/11/12 00:22:19 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:50370/oidc5261--- PASS: TestValidateToken_MultipleProviders (0.37s)52622026/11/12 00:22:20 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:50355/oidc5263--- PASS: TestValidateToken_NoMatchingProvider (0.47s)5264PASS5265Running hook tests...5266=== RUN TestSendPathsEmpty5267=== PAUSE TestSendPathsEmpty5268=== RUN TestQueueEnqueueAndFetch5269=== PAUSE TestQueueEnqueueAndFetch5270=== RUN TestQueueDeduplication5271=== PAUSE TestQueueDeduplication5272=== RUN TestQueueRemove5273=== PAUSE TestQueueRemove5274=== RUN TestQueueFetchBatchLimit5275=== PAUSE TestQueueFetchBatchLimit5276=== RUN TestQueueRetryMovesToBack5277=== PAUSE TestQueueRetryMovesToBack5278=== RUN TestQueueFetchRemoveLifecycle5279=== PAUSE TestQueueFetchRemoveLifecycle5280=== RUN TestQueueConcurrentWriters5281=== PAUSE TestQueueConcurrentWriters5282=== RUN TestQueueRemoveLargeClosure5283=== PAUSE TestQueueRemoveLargeClosure5284=== RUN TestServerClientIntegration5285--- PASS: TestServerClientIntegration (0.00s)5286=== RUN TestServerQueueError52872026/11/12 00:22:20 ERROR Failed to queue paths error="permission denied" count=15288--- PASS: TestServerQueueError (0.00s)5289=== RUN TestGetListenerSocketActivation5290 server_test.go:147: systemd socket activation does not exist on darwin5291--- SKIP: TestGetListenerSocketActivation (0.00s)5292=== RUN TestServerStalledClientDoesNotBlockShutdown5293=== PAUSE TestServerStalledClientDoesNotBlockShutdown5294=== RUN TestServerBacksOffOnAcceptErrors5295=== PAUSE TestServerBacksOffOnAcceptErrors5296=== RUN TestDrainIsolatesPoisonPath5297=== PAUSE TestDrainIsolatesPoisonPath5298=== RUN TestRunNotBlockedByPoisonHead5299=== PAUSE TestRunNotBlockedByPoisonHead5300=== RUN TestDrainGivesUpWhenServerDown5301=== PAUSE TestDrainGivesUpWhenServerDown5302=== RUN TestFailedPathPrunedByLaterClosure5303=== PAUSE TestFailedPathPrunedByLaterClosure5304=== RUN TestWorkerUploadsAndRemoves5305=== PAUSE TestWorkerUploadsAndRemoves5306=== RUN TestWorkerSkipsGCdPaths5307=== PAUSE TestWorkerSkipsGCdPaths5308=== RUN TestWorkerPrunesClosureDeps5309=== PAUSE TestWorkerPrunesClosureDeps5310=== RUN TestDrainTimeout5311=== PAUSE TestDrainTimeout5312=== RUN TestWorkerRemoveFailureIsNotProgress5313=== PAUSE TestWorkerRemoveFailureIsNotProgress5314=== RUN TestWorkerKeepsPathItCannotStat5315=== PAUSE TestWorkerKeepsPathItCannotStat5316=== CONT TestSendPathsEmpty5317--- PASS: TestSendPathsEmpty (0.00s)5318=== CONT TestWorkerKeepsPathItCannotStat5319=== CONT TestServerBacksOffOnAcceptErrors5320=== CONT TestQueueEnqueueAndFetch5321=== CONT TestRunNotBlockedByPoisonHead53222000/01/01 00:00:00 ERROR Accept failed error="too many open files"53232000/01/01 00:00:00 ERROR Accept failed error="too many open files"53242000/01/01 00:00:00 ERROR Accept failed error="too many open files"53252000/01/01 00:00:00 ERROR Accept failed error="too many open files"5326=== CONT TestServerStalledClientDoesNotBlockShutdown53272000/01/01 00:00:00 ERROR Accept failed error="too many open files"53282000/01/01 00:00:00 ERROR Accept failed error="too many open files"53292000/01/01 00:00:00 ERROR Accept failed error="too many open files"53302000/01/01 00:00:00 ERROR Accept failed error="too many open files"5331--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5332=== CONT TestQueueRemoveLargeClosure5333=== CONT TestQueueConcurrentWriters5334=== CONT TestQueueFetchRemoveLifecycle5335=== CONT TestWorkerUploadsAndRemoves5336=== CONT TestWorkerRemoveFailureIsNotProgress5337=== CONT TestDrainTimeout5338=== CONT TestWorkerPrunesClosureDeps5339=== CONT TestWorkerSkipsGCdPaths5340=== CONT TestQueueRemove5341=== CONT TestQueueFetchBatchLimit5342=== CONT TestQueueDeduplication5343=== CONT TestDrainGivesUpWhenServerDown5344=== CONT TestFailedPathPrunedByLaterClosure5345=== CONT TestDrainIsolatesPoisonPath5346=== CONT TestQueueRetryMovesToBack5347--- PASS: TestQueueFetchBatchLimit (0.01s)53482026/11/12 00:22:20 INFO Upload queue status pending=353492026/11/12 00:22:20 INFO Uploading batch count=153502026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=153512026/11/12 00:22:20 INFO Upload queue status pending=253522026/11/12 00:22:20 INFO Uploading batch count=25353--- PASS: TestQueueEnqueueAndFetch (0.01s)53542026/11/12 00:22:20 INFO Uploading batch count=253552026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=253562026/11/12 00:22:20 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestDrainGivesUpWhenServerDown3933146925/002/a53572026/11/12 00:22:20 INFO Uploading batch count=453582026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=453592026/11/12 00:22:20 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerRemoveFailureIsNotProgress2611236554/002/nonexistent53602026/11/12 00:22:20 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestDrainGivesUpWhenServerDown3933146925/002/b53612026/11/12 00:22:20 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153622000/01/01 00:00:00 INFO Uploading batch count=25363--- PASS: TestQueueFetchRemoveLifecycle (0.01s)53642000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253652026/11/12 00:22:20 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerRemoveFailureIsNotProgress2611236554/002/nonexistent53662026/11/12 00:22:20 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153672026/11/12 00:22:20 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerRemoveFailureIsNotProgress2611236554/002/nonexistent53682026/11/12 00:22:20 WARN Cannot stat store path, will retry later path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerKeepsPathItCannotStat3988315401/002/locked/aaa error="stat /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerKeepsPathItCannotStat3988315401/002/locked/aaa: permission denied"53692000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453702026/11/12 00:22:20 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153712026/11/12 00:22:20 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestDrainIsolatesPoisonPath1522605747/002/bbb53722026/11/12 00:22:20 WARN Cannot stat store path, will retry later path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerKeepsPathItCannotStat3988315401/002/locked/aaa error="stat /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerKeepsPathItCannotStat3988315401/002/locked/aaa: permission denied"53732026/11/12 00:22:20 INFO Upload queue status pending=253742026/11/12 00:22:20 INFO Uploading batch count=153752026/11/12 00:22:20 INFO Uploading batch count=153762026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=153772026/11/12 00:22:20 INFO Upload queue status pending=253782026/11/12 00:22:20 INFO Uploading batch count=253792026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=253802026/11/12 00:22:20 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestDrainGivesUpWhenServerDown3933146925/002/c53812026/11/12 00:22:20 WARN Cannot stat store path, will retry later path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerKeepsPathItCannotStat3988315401/002/locked/aaa error="stat /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerKeepsPathItCannotStat3988315401/002/locked/aaa: permission denied"53822026/11/12 00:22:20 ERROR Drain finished with paths left in queue remaining=153832026/11/12 00:22:20 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestWorkerSkipsGCdPaths314350588/002/nonexistent5384--- PASS: TestQueueRetryMovesToBack (0.01s)5385--- PASS: TestQueueRemove (0.01s)53862026/11/12 00:22:20 INFO Uploading batch count=153872026/11/12 00:22:20 INFO Uploading batch count=153882026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=153892026/11/12 00:22:20 INFO Uploading batch count=153902026/11/12 00:22:20 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestDrainGivesUpWhenServerDown3933146925/002/d53912026/11/12 00:22:20 ERROR Drain finished with paths left in queue remaining=153922026/11/12 00:22:20 INFO Uploading batch count=153932026/11/12 00:22:20 INFO Uploading batch count=253942026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=253952026/11/12 00:22:20 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestDrainGivesUpWhenServerDown3933146925/002/e5396--- PASS: TestQueueDeduplication (0.01s)53972026/11/12 00:22:20 INFO Uploading batch count=153982026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=15399--- PASS: TestDrainTimeout (0.01s)54002026/11/12 00:22:20 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestDrainGivesUpWhenServerDown3933146925/002/f54012026/11/12 00:22:20 ERROR Drain finished with paths left in queue remaining=1054022026/11/12 00:22:20 INFO Uploading batch count=154032026/11/12 00:22:20 ERROR Upload failed error="upload failed" count=15404--- PASS: TestWorkerRemoveFailureIsNotProgress (0.01s)5405--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)54062026/11/12 00:22:20 ERROR Drain finished with paths left in queue remaining=15407--- PASS: TestFailedPathPrunedByLaterClosure (0.01s)5408--- PASS: TestDrainGivesUpWhenServerDown (0.01s)5409--- PASS: TestDrainIsolatesPoisonPath (0.01s)5410--- PASS: TestWorkerUploadsAndRemoves (0.03s)5411--- PASS: TestWorkerSkipsGCdPaths (0.03s)5412--- PASS: TestWorkerPrunesClosureDeps (0.03s)5413--- PASS: TestQueueRemoveLargeClosure (0.07s)5414--- PASS: TestQueueConcurrentWriters (0.08s)54152026/11/12 00:22:20 ERROR Failed to decode request error="read unix /private/var/tb/1/scratch/d3a820deccb6fcfd/build/hook469044078/test.sock->: i/o timeout"54162026/11/12 00:22:20 ERROR Failed to write response error="write unix /private/var/tb/1/scratch/d3a820deccb6fcfd/build/hook469044078/test.sock->: i/o timeout"5417--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)54182026/11/12 00:22:21 INFO Uploading batch count=154192026/11/12 00:22:21 INFO Uploading batch count=154202026/11/12 00:22:21 INFO Uploading batch count=154212026/11/12 00:22:21 ERROR Upload failed error="upload failed" count=154222026/11/12 00:22:21 INFO Uploading batch count=154232026/11/12 00:22:21 ERROR Upload failed error="upload failed" count=154242026/11/12 00:22:21 INFO Uploading batch count=154252026/11/12 00:22:21 ERROR Upload failed error="upload failed" count=154262026/11/12 00:22:21 INFO Uploading batch count=154272026/11/12 00:22:21 ERROR Upload failed error="upload failed" count=154282026/11/12 00:22:21 ERROR Drain finished with paths left in queue remaining=15429--- PASS: TestRunNotBlockedByPoisonHead (1.02s)5430PASS5431Running signing tests...5432=== RUN TestExternalSigner5433=== PAUSE TestExternalSigner5434=== RUN TestExternalSignerWrongPublicKey5435=== PAUSE TestExternalSignerWrongPublicKey5436=== RUN TestExternalSignerErrors5437=== PAUSE TestExternalSignerErrors5438=== RUN TestExternalSignerRestartsAfterCrash5439=== PAUSE TestExternalSignerRestartsAfterCrash5440=== RUN TestExternalSignerProgramErrorKeepsProcess5441=== PAUSE TestExternalSignerProgramErrorKeepsProcess5442=== RUN TestExternalSignerClose5443=== PAUSE TestExternalSignerClose5444=== RUN TestExternalSignerRejectsInvalidPublicKey5445=== PAUSE TestExternalSignerRejectsInvalidPublicKey5446=== RUN TestExternalSignerCancellation5447=== PAUSE TestExternalSignerCancellation5448=== RUN TestGenerateFingerprint5449=== PAUSE TestGenerateFingerprint5450=== RUN TestParseSigningKey5451=== PAUSE TestParseSigningKey5452=== RUN TestSign5453=== PAUSE TestSign5454=== RUN TestSignBatch5455=== PAUSE TestSignBatch5456=== RUN TestNilKeyPublicKey5457=== PAUSE TestNilKeyPublicKey5458=== RUN TestSignErrors5459=== PAUSE TestSignErrors5460=== CONT TestExternalSigner5461=== CONT TestExternalSignerProgramErrorKeepsProcess5462=== CONT TestSignBatch5463=== CONT TestExternalSignerCancellation5464=== CONT TestSign5465=== CONT TestParseSigningKey5466=== RUN TestParseSigningKey/valid_32-byte_key5467=== PAUSE TestParseSigningKey/valid_32-byte_key5468=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5469=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5470=== RUN TestParseSigningKey/no_colon5471=== PAUSE TestParseSigningKey/no_colon5472=== RUN TestParseSigningKey/empty_name5473=== PAUSE TestParseSigningKey/empty_name5474=== CONT TestGenerateFingerprint5475=== RUN TestGenerateFingerprint/basic_with_references5476=== PAUSE TestGenerateFingerprint/basic_with_references5477=== RUN TestGenerateFingerprint/no_references5478=== PAUSE TestGenerateFingerprint/no_references5479=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5480=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5481=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5482=== CONT TestSignErrors5483=== CONT TestExternalSignerRestartsAfterCrash5484=== CONT TestNilKeyPublicKey5485--- PASS: TestNilKeyPublicKey (0.00s)5486=== CONT TestExternalSignerWrongPublicKey5487=== CONT TestExternalSignerRejectsInvalidPublicKey5488=== RUN TestExternalSignerRejectsInvalidPublicKey/#005489=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005490=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5491=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5492=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5493=== CONT TestExternalSignerClose5494=== RUN TestParseSigningKey/invalid_base645495=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5496=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5497=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5498=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5499=== PAUSE TestParseSigningKey/invalid_base645500=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5501=== RUN TestParseSigningKey/wrong_length5502=== RUN TestGenerateFingerprint/invalid_nar_hash_length5503=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5504=== RUN TestGenerateFingerprint/invalid_store_path_prefix5505=== PAUSE TestParseSigningKey/wrong_length5506=== CONT TestParseSigningKey/invalid_base645507=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5508=== CONT TestParseSigningKey/valid_32-byte_key5509=== CONT TestExternalSignerErrors5510=== RUN TestExternalSignerErrors/exit_on_request5511=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5512=== CONT TestParseSigningKey/empty_name5513=== CONT TestParseSigningKey/no_colon5514=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5515=== CONT TestExternalSignerRejectsInvalidPublicKey/#005516=== CONT TestParseSigningKey/wrong_length5517=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5518=== PAUSE TestExternalSignerErrors/exit_on_request5519=== RUN TestExternalSignerErrors/invalid_JSON5520=== PAUSE TestExternalSignerErrors/invalid_JSON5521=== RUN TestExternalSignerErrors/wrong_id5522=== RUN TestGenerateFingerprint/invalid_reference_prefix5523=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5524=== CONT TestGenerateFingerprint/basic_with_references5525=== CONT TestGenerateFingerprint/invalid_reference_prefix5526=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5527=== CONT TestGenerateFingerprint/invalid_store_path_prefix5528=== CONT TestGenerateFingerprint/invalid_nar_hash_length5529=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5530=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5531--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5532 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5533 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5534 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5535 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5536 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5537=== PAUSE TestExternalSignerErrors/wrong_id5538=== RUN TestExternalSignerErrors/missing_signatures5539=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5540=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5541=== CONT TestGenerateFingerprint/no_references5542--- PASS: TestGenerateFingerprint (0.00s)5543 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5544 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5545 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5546 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5547 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5548 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5549 --- PASS: TestGenerateFingerprint/no_references (0.00s)5550=== PAUSE TestExternalSignerErrors/missing_signatures5551=== RUN TestExternalSignerErrors/too_few_signatures5552=== PAUSE TestExternalSignerErrors/too_few_signatures5553=== RUN TestExternalSignerErrors/invalid_signature_base645554=== PAUSE TestExternalSignerErrors/invalid_signature_base645555=== RUN TestExternalSignerErrors/null5556=== PAUSE TestExternalSignerErrors/null5557=== RUN TestExternalSignerErrors/program_error5558=== PAUSE TestExternalSignerErrors/program_error5559=== RUN TestExternalSignerErrors/exit_at_startup5560=== PAUSE TestExternalSignerErrors/exit_at_startup5561=== RUN TestExternalSignerErrors/wrong_shape5562=== PAUSE TestExternalSignerErrors/wrong_shape5563=== RUN TestExternalSignerErrors/null_signatures5564=== PAUSE TestExternalSignerErrors/null_signatures5565=== RUN TestExternalSignerErrors/unrequested_key5566=== PAUSE TestExternalSignerErrors/unrequested_key5567=== RUN TestExternalSignerErrors/missing_signature_name5568=== PAUSE TestExternalSignerErrors/missing_signature_name5569=== RUN TestExternalSignerErrors/wrong_signatures_shape5570=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5571=== RUN TestExternalSignerErrors/wrong_signature_name5572=== PAUSE TestExternalSignerErrors/wrong_signature_name5573=== RUN TestExternalSignerErrors/wrong_signature_length5574=== PAUSE TestExternalSignerErrors/wrong_signature_length5575=== CONT TestExternalSignerErrors/exit_on_request5576=== CONT TestExternalSignerErrors/wrong_shape5577=== CONT TestExternalSignerErrors/too_few_signatures5578--- PASS: TestSign (0.00s)5579=== CONT TestExternalSignerErrors/wrong_id5580=== CONT TestExternalSignerErrors/wrong_signature_length5581=== RUN TestSignErrors/nil_key5582=== PAUSE TestSignErrors/nil_key5583=== RUN TestSignErrors/nil_narinfo5584=== PAUSE TestSignErrors/nil_narinfo5585=== RUN TestSignErrors/invalid_narinfo5586=== PAUSE TestSignErrors/invalid_narinfo5587=== RUN TestSignErrors/canceled5588=== PAUSE TestSignErrors/canceled5589=== CONT TestExternalSignerErrors/missing_signatures5590=== CONT TestSignErrors/nil_key5591=== CONT TestSignErrors/canceled5592=== CONT TestSignErrors/invalid_narinfo5593=== CONT TestSignErrors/nil_narinfo5594--- PASS: TestParseSigningKey (0.00s)5595 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5596 --- PASS: TestParseSigningKey/empty_name (0.00s)5597 --- PASS: TestParseSigningKey/no_colon (0.00s)5598 --- PASS: TestParseSigningKey/wrong_length (0.00s)5599 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5600 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5601=== CONT TestExternalSignerErrors/missing_signature_name5602--- PASS: TestSignErrors (0.00s)5603 --- PASS: TestSignErrors/nil_key (0.00s)5604 --- PASS: TestSignErrors/canceled (0.00s)5605 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5606 --- PASS: TestSignErrors/nil_narinfo (0.00s)5607=== CONT TestExternalSignerErrors/unrequested_key5608=== CONT TestExternalSignerErrors/null_signatures5609--- PASS: TestSignBatch (0.00s)5610=== CONT TestExternalSignerErrors/program_error5611=== CONT TestExternalSignerErrors/null5612=== CONT TestExternalSignerErrors/invalid_signature_base645613=== CONT TestExternalSignerErrors/exit_at_startup5614=== CONT TestExternalSignerErrors/invalid_JSON5615=== CONT TestExternalSignerErrors/wrong_signature_name5616=== CONT TestExternalSignerErrors/wrong_signatures_shape5617--- PASS: TestExternalSigner (0.13s)5618--- PASS: TestExternalSignerCancellation (0.41s)5619--- PASS: TestExternalSignerProgramErrorKeepsProcess (1.09s)5620--- PASS: TestExternalSignerWrongPublicKey (1.19s)5621--- PASS: TestExternalSignerClose (1.30s)5622--- PASS: TestExternalSignerRestartsAfterCrash (1.51s)5623/private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestExternalSignerRestartsAfterCrash1749600662/001/sign program: line 7: /private/var/tb/1/scratch/d3a820deccb6fcfd/build/TestExternalSignerRestartsAfterCrash1749600662/001/sign program.starts: No such file or directory5624--- PASS: TestExternalSignerErrors (0.00s)5625 --- PASS: TestExternalSignerErrors/wrong_shape (0.21s)5626 --- PASS: TestExternalSignerErrors/unrequested_key (0.29s)5627 --- PASS: TestExternalSignerErrors/wrong_id (0.47s)5628 --- PASS: TestExternalSignerErrors/null_signatures (0.57s)5629 --- PASS: TestExternalSignerErrors/too_few_signatures (0.67s)5630 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.75s)5631 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.84s)5632 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.92s)5633 --- PASS: TestExternalSignerErrors/exit_on_request (1.01s)5634 --- PASS: TestExternalSignerErrors/null (1.39s)5635 --- PASS: TestExternalSignerErrors/invalid_JSON (1.57s)5636 --- PASS: TestExternalSignerErrors/wrong_signature_length (1.66s)5637 --- PASS: TestExternalSignerErrors/program_error (1.75s)5638 --- PASS: TestExternalSignerErrors/missing_signature_name (1.83s)5639 --- PASS: TestExternalSignerErrors/exit_at_startup (1.92s)5640 --- PASS: TestExternalSignerErrors/missing_signatures (2.00s)5641PASS5642Running OpenBao signer tests...5643=== RUN TestSign5644=== PAUSE TestSign5645=== RUN TestSignErrors5646=== PAUSE TestSignErrors5647=== RUN TestServe5648=== PAUSE TestServe5649=== RUN TestServeRejectsGarbage5650=== PAUSE TestServeRejectsGarbage5651=== CONT TestSign5652=== CONT TestServe5653=== CONT TestServeRejectsGarbage5654=== CONT TestSignErrors5655=== RUN TestSignErrors/no_prefix5656--- PASS: TestServeRejectsGarbage (0.00s)5657=== PAUSE TestSignErrors/no_prefix5658=== RUN TestSignErrors/invalid_base645659=== PAUSE TestSignErrors/invalid_base645660=== RUN TestSignErrors/http_error5661=== PAUSE TestSignErrors/http_error5662=== RUN TestSignErrors/invalid_json5663=== PAUSE TestSignErrors/invalid_json5664=== CONT TestSignErrors/no_prefix5665=== CONT TestSignErrors/http_error5666--- PASS: TestServe (0.00s)5667=== CONT TestSignErrors/invalid_json5668=== CONT TestSignErrors/invalid_base645669--- PASS: TestSign (0.00s)5670--- PASS: TestSignErrors (0.00s)5671 --- PASS: TestSignErrors/no_prefix (0.00s)5672 --- PASS: TestSignErrors/invalid_base64 (0.00s)5673 --- PASS: TestSignErrors/http_error (0.00s)5674 --- PASS: TestSignErrors/invalid_json (0.00s)5675PASS