niks3-go-unit-tests
checks.aarch64-darwin.go-unit-tests
· build #496
· raw
1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.13s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushAnswersEveryLineWhenCanceled86=== PAUSE TestStreamPushAnswersEveryLineWhenCanceled87=== RUN TestStreamPushStopsWhenCanceled88=== PAUSE TestStreamPushStopsWhenCanceled89=== RUN TestSetClientTLS90=== PAUSE TestSetClientTLS91=== RUN TestSetClientTLSDoesNotMutateDefaultTransport92=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport93=== RUN TestSetClientTLSErrors94=== PAUSE TestSetClientTLSErrors95=== RUN TestStaticToken96=== PAUSE TestStaticToken97=== RUN TestFileTokenReadsAndCaches98=== PAUSE TestFileTokenReadsAndCaches99=== RUN TestFileTokenMissing100=== PAUSE TestFileTokenMissing101=== RUN TestFileTokenEmpty102=== PAUSE TestFileTokenEmpty103=== RUN TestScriptTokenNoExpiryRerunsEveryCall104=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall105=== RUN TestScriptTokenCachesUntilRefresh106=== PAUSE TestScriptTokenCachesUntilRefresh107=== RUN TestScriptTokenEmptyToken108=== PAUSE TestScriptTokenEmptyToken109=== RUN TestScriptTokenBadJSON110=== PAUSE TestScriptTokenBadJSON111=== RUN TestScriptTokenScriptFails112=== PAUSE TestScriptTokenScriptFails113=== RUN TestScriptTokenEmptyCommand114=== PAUSE TestScriptTokenEmptyCommand115=== CONT TestDoServerRequestAttachesToken116=== CONT TestRetryTimeoutOutlastsMaxRetries117=== CONT TestEncodeNixBase32WithRealHash118=== CONT TestSetClientTLS119=== CONT TestScriptTokenEmptyCommand120=== CONT TestStreamPushBatchesUnderLoad121=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false122=== CONT TestScriptTokenScriptFails123=== CONT TestScriptTokenBadJSON124=== CONT TestScriptTokenEmptyToken125=== CONT TestScriptTokenCachesUntilRefresh126=== CONT TestScriptTokenNoExpiryRerunsEveryCall127=== CONT TestFileTokenEmpty128=== CONT TestFileTokenMissing129=== CONT TestFileTokenReadsAndCaches130=== CONT TestStaticToken131=== CONT TestShellSplit132=== CONT TestSetClientTLSErrors133=== CONT TestSetClientTLSDoesNotMutateDefaultTransport134=== CONT TestStreamPushGivesUpOnDeadServer135=== CONT TestStreamPushStopsWhenCanceled136=== CONT TestStreamPushAnswersEveryLineWhenCanceled137=== CONT TestClientSignaturesByStorePath138=== CONT TestStreamPushReportsSkippedPaths139=== CONT TestStreamPushReportsSignatures140=== CONT TestStreamPushRequestLine141=== CONT TestStreamPushReportsEveryPath142--- PASS: TestEncodeNixBase32WithRealHash (0.00s)143--- PASS: TestScriptTokenEmptyCommand (0.00s)144--- PASS: TestStaticToken (0.00s)145=== CONT TestStreamPushIsolatesFailures146=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false147=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true148=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true1492026/11/12 00:20:13 ERROR Upload failed error="connection refused" count=201502026/11/12 00:20:13 ERROR Server seems unavailable, giving up on batch untried=17151=== CONT TestShellSplitErrors152=== CONT TestDoWithRetry_BodyReplayedViaGetBody153--- PASS: TestStreamPushReportsSkippedPaths (0.00s)154--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)155--- PASS: TestShellSplitErrors (0.00s)156=== CONT TestDoWithRetry_FinalResponseBodyReadable157--- PASS: TestClientSignaturesByStorePath (0.00s)158=== CONT TestResolveStorePath159=== CONT TestRateLimiterFeedback160=== RUN TestRateLimiterFeedback/429_enables_limiter161=== PAUSE TestRateLimiterFeedback/429_enables_limiter162=== RUN TestRateLimiterFeedback/503_enables_limiter163--- PASS: TestFileTokenMissing (0.00s)164--- PASS: TestShellSplit (0.00s)165=== CONT TestRegisterUploadedObject_OutlivesPush166=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer167--- PASS: TestFileTokenEmpty (0.00s)168=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess169=== PAUSE TestRateLimiterFeedback/503_enables_limiter170=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter1712026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:643481722026/11/12 00:20:13 WARN Rate limiter enabled after throttle name=server-test rate=5173=== RUN TestSetClientTLSErrors/missing_cert_file174=== PAUSE TestSetClientTLSErrors/missing_cert_file175=== RUN TestSetClientTLSErrors/missing_key_file1762000/01/01 00:00:00 ERROR Upload failed error="context canceled" count=2177--- PASS: TestStreamPushStopsWhenCanceled (0.00s)178--- PASS: TestStreamPushAnswersEveryLineWhenCanceled (0.00s)179--- PASS: TestDoServerRequestAttachesToken (0.01s)180=== PAUSE TestSetClientTLSErrors/missing_key_file1812026/11/12 00:20:13 ERROR Upload failed error="bad path" count=3182--- PASS: TestStreamPushReportsEveryPath (0.00s)183=== RUN TestSetClientTLSErrors/missing_ca_file184--- PASS: TestStreamPushIsolatesFailures (0.00s)1852026/11/12 00:20:13 ERROR Upload failed error=boom count=11862026/11/12 00:20:13 ERROR Upload failed error=boom count=1187=== PAUSE TestSetClientTLSErrors/missing_ca_file188=== RUN TestSetClientTLSErrors/invalid_ca_file189=== CONT TestEncodeNixBase32190=== CONT TestUploadMultipart_PartsInParallel191=== CONT TestDumpPathWriterError192--- PASS: TestStreamPushReportsSignatures (0.00s)193--- PASS: TestScriptTokenScriptFails (0.01s)194=== CONT TestUploadMultipart_ReturnsAfterDroppedBody195=== RUN TestEncodeNixBase32/test_string_hash1962026/11/12 00:20:13 WARN Rate limiter enabled after throttle name=server-test rate=51972026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:643541982026/11/12 00:20:13 WARN Rate limiter backed off name=server-test rate=5199=== CONT TestDumpPathSingleFile200=== CONT TestPartSizeForNAR201=== RUN TestPartSizeForNAR/zero_stays_at_minimum202=== CONT TestDumpPathMatchesNix203=== PAUSE TestSetClientTLSErrors/invalid_ca_file204=== PAUSE TestEncodeNixBase32/test_string_hash205--- PASS: TestFileTokenReadsAndCaches (0.01s)2062026/11/12 00:20:13 WARN Rate limiter enabled after throttle name=server-test rate=5207--- PASS: TestResolveStorePath (0.01s)2082026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:64352209=== CONT TestGetStorePathHash210=== RUN TestSetClientTLS/rejects_connection_without_client_cert211=== CONT TestPathInfoHashCompatibility212=== CONT TestConvertHashToNix32213=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun214=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum215=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter216=== RUN TestEncodeNixBase32/empty_input217=== CONT TestCaseHackSuffix218=== CONT TestFilterOversizedClosures219=== CONT TestRegisterUploadedObjectReusesConnections2202026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:64348221=== RUN TestGetStorePathHash/valid_store_path222=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert223=== PAUSE TestGetStorePathHash/valid_store_path224=== RUN TestGetStorePathHash/basename_without_hyphen_should_error225=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error226=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error227=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error228=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter229=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error230=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error231--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.00s)232--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)233--- PASS: TestRegisterUploadedObject_OutlivesPush (0.01s)234--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.01s)235=== RUN TestPartSizeForNAR/small_stays_at_minimum236=== PAUSE TestPartSizeForNAR/small_stays_at_minimum237=== RUN TestFilterOversizedClosures/no_limit_keeps_everything238--- PASS: TestScriptTokenEmptyToken (0.02s)239=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything240=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped241=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped242=== RUN TestFilterOversizedClosures/all_closures_skipped243=== PAUSE TestFilterOversizedClosures/all_closures_skipped244=== CONT TestPathInfoCACompatibility245=== RUN TestPathInfoCACompatibility/null_ca_field246=== PAUSE TestPathInfoCACompatibility/null_ca_field247=== RUN TestPathInfoCACompatibility/old_string_format_-_text248=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text249=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive250=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive251=== RUN TestPathInfoCACompatibility/new_structured_format_-_text252=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text253=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method254=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method255=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter256=== CONT TestParsePathInfoJSONMultiplePaths257=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths258=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths259=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths260=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths261=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry262--- PASS: TestScriptTokenBadJSON (0.03s)263=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true264=== CONT TestSetClientTLSErrors/missing_cert_file265=== CONT TestSetClientTLSErrors/invalid_ca_file266=== CONT TestSetClientTLSErrors/missing_ca_file267=== CONT TestSetClientTLSErrors/missing_key_file268=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum269=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum270=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts271=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts272=== RUN TestPartSizeForNAR/1_TiB273=== PAUSE TestPartSizeForNAR/1_TiB274=== RUN TestPartSizeForNAR/5_TiB_S3_max_object275=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object276=== RUN TestPartSizeForNAR/capped_at_5_GiB277=== PAUSE TestPartSizeForNAR/capped_at_5_GiB278=== CONT TestParsePathInfoJSON279=== RUN TestParsePathInfoJSON/Nix_format280=== PAUSE TestParsePathInfoJSON/Nix_format281=== RUN TestParsePathInfoJSON/Lix_format282=== PAUSE TestParsePathInfoJSON/Lix_format283=== RUN TestParsePathInfoJSON/empty_input284=== PAUSE TestParsePathInfoJSON/empty_input285=== RUN TestParsePathInfoJSON/whitespace_only286=== PAUSE TestParsePathInfoJSON/whitespace_only287=== RUN TestParsePathInfoJSON/invalid_JSON288=== PAUSE TestParsePathInfoJSON/invalid_JSON289=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false290=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)291=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)292=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon293=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon294=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI295=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI296=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512297=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512298=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error299=== CONT TestFilterOversizedClosures/no_limit_keeps_everything300=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error301=== CONT TestRateLimiterFeedback/429_enables_limiter302=== RUN TestConvertHashToNix32/SRI_format_to_Nix32303=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32304=== RUN TestConvertHashToNix32/already_Nix32_format305=== PAUSE TestConvertHashToNix32/already_Nix32_format306=== RUN TestConvertHashToNix32/invalid_format307=== PAUSE TestConvertHashToNix32/invalid_format308=== CONT TestGetStorePathHash/basename_without_hyphen_should_error309=== CONT TestFilterOversizedClosures/all_closures_skipped3102026/11/12 00:20:13 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50311=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3122026/11/12 00:20:13 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000313--- PASS: TestFilterOversizedClosures (0.00s)314 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)315 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)316 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)317=== CONT TestPathInfoCACompatibility/null_ca_field318=== PAUSE TestEncodeNixBase32/empty_input319=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths320=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method321=== CONT TestPartSizeForNAR/zero_stays_at_minimum322=== CONT TestPathInfoCACompatibility/new_structured_format_-_text323=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter324--- PASS: TestStreamPushBatchesUnderLoad (0.00s)325 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)326 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)327=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica328=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA329=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA330=== RUN TestSetClientTLS/preserves_debug_logging_transport331=== PAUSE TestSetClientTLS/preserves_debug_logging_transport332=== CONT TestRateLimiterFeedback/503_enables_limiter3332026/11/12 00:20:13 WARN Rate limiter backed off name=server-test rate=53342026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:643523352026/11/12 00:20:13 WARN Rate limiter enabled after throttle name=server-test rate=53362026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:64435337=== CONT TestGetStorePathHash/valid_store_path338=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)3392026/11/12 00:20:13 WARN Rate limiter enabled after throttle name=server-test rate=53402026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:644173412026/11/12 00:20:13 WARN Rate limiter backed off name=server-test rate=53422026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:64348343=== CONT TestPartSizeForNAR/5_TiB_S3_max_object344=== CONT TestPartSizeForNAR/1_TiB345=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts346=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum347=== CONT TestPartSizeForNAR/small_stays_at_minimum348=== CONT TestParsePathInfoJSON/invalid_JSON349=== CONT TestParsePathInfoJSON/whitespace_only350=== CONT TestParsePathInfoJSON/empty_input351=== CONT TestParsePathInfoJSON/Lix_format352=== CONT TestConvertHashToNix32/SRI_format_to_Nix32353=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512354=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI355=== CONT TestEncodeNixBase32/test_string_hash356=== CONT TestConvertHashToNix32/invalid_format357=== CONT TestParsePathInfoJSON/Nix_format358=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon359=== CONT TestSetClientTLS/preserves_debug_logging_transport360=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths361=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA362=== CONT TestPathInfoCACompatibility/old_string_format_-_text363=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive364=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter365=== CONT TestPartSizeForNAR/capped_at_5_GiB366--- PASS: TestStreamPushRequestLine (0.02s)367=== CONT TestConvertHashToNix32/already_Nix32_format368=== CONT TestEncodeNixBase32/empty_input369--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.03s)370--- PASS: TestSetClientTLSErrors (0.01s)371 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)372 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)373 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)374 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)375=== CONT TestSetClientTLS/rejects_connection_without_client_cert376--- PASS: TestParsePathInfoJSON (0.00s)377 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)378 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)379 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)380 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)381 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)382--- PASS: TestPathInfoHashCompatibility (0.00s)383 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)384 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)385 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)386 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)387--- PASS: TestConvertHashToNix32 (0.00s)388 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)389 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)390 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)391--- PASS: TestEncodeNixBase32 (0.02s)392 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)393 --- PASS: TestEncodeNixBase32/empty_input (0.00s)394--- PASS: TestPartSizeForNAR (0.01s)395 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)396 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)397 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)398 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)399 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)400 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)401 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)402--- PASS: TestGetStorePathHash (0.00s)403 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)404 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)405 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)406 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)407--- PASS: TestPathInfoCACompatibility (0.00s)408 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)409 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)410 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)411 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)412 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)413--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)414 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)415 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)4162026/11/12 00:20:13 WARN Rate limiter backed off name=server-test rate=54172026/11/12 00:20:13 WARN Rate limiter enabled after throttle name=s3-test rate=54182026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:644074192026/11/12 00:20:13 INFO Garbage collection started4202026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:643484212026/11/12 00:20:13 INFO Garbage collection started422--- PASS: TestRateLimiterFeedback (0.02s)423 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)424 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)425 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.01s)426 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)4272026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:64348428--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)4292026/11/12 00:20:13 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=04302026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:643484312026/11/12 00:20:13 WARN Garbage collection finished on another replica. Its statistics are not available here432--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)433--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)4342026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:643484352026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:643484362026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:643484372026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:643484382026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:643484392026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:643484402026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:643484412026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:643484422026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:643484432026/11/12 00:20:13 http: TLS handshake error from 127.0.0.1:64452: remote error: tls: bad certificate4442026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:643484452026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:64348446--- PASS: TestRegisterUploadedObjectReusesConnections (0.04s)447--- PASS: TestSetClientTLS (0.03s)448 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)449 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.01s)450 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.03s)4512026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:643484522026/11/12 00:20:13 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:64348453--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.07s)454--- PASS: TestScriptTokenCachesUntilRefresh (0.11s)4552026/11/12 00:20:13 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:64357/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:64357/api/uploads/complete4562026/11/12 00:20:13 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:64357/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:64357/api/uploads/complete4572026/11/12 00:20:13 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4582026/11/12 00:20:13 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"459--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.11s)4602026/11/12 00:20:13 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:64357/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:64357/api/uploads/complete4612026/11/12 00:20:13 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"462--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.11s)463--- PASS: TestDumpPathWriterError (0.10s)464--- PASS: TestDumpPathSingleFile (0.16s)465--- PASS: TestCaseHackSuffix (0.15s)466--- PASS: TestDumpPathMatchesNix (0.24s)467--- PASS: TestUploadMultipart_PartsInParallel (0.64s)468--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)469PASS470Running server tests...471The files belonging to this database system will be owned by user "_tribuchetbld3".472This user must also own the server process.473474The database cluster will be initialized with locale "C".475The default database encoding has accordingly been set to "SQL_ASCII".476The default text search configuration will be set to "english".477478Data page checksums are enabled.479480creating directory /private/var/tb/3/scratch/62601a543565e631/build/postgres2134869968/data ... ok481creating subdirectories ... ok482selecting dynamic shared memory implementation ... posix483selecting default "max_connections" ... 100484selecting default "shared_buffers" ... 128MB485selecting default time zone ... UTC486creating configuration files ... ok487running bootstrap script ... ok488performing post-bootstrap initialization ... ok489syncing data to disk ... ok490491initdb: warning: enabling "trust" authentication for local connections492initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.493494Success. You can now start the database server using:495496 pg_ctl -D /private/var/tb/3/scratch/62601a543565e631/build/postgres2134869968/data -l logfile start497498/private/var/tb/3/scratch/62601a543565e631/build/postgres2134869968:5432 - no response4992026-11-12 00:20:15.685 UTC [55856] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit5002026-11-12 00:20:15.686 UTC [55856] LOG: listening on Unix socket "/private/var/tb/3/scratch/62601a543565e631/build/postgres2134869968/.s.PGSQL.5432"5012026-11-12 00:20:15.697 UTC [55877] LOG: database system was shut down at 2026-11-12 00:20:15 UTC5022026-11-12 00:20:15.699 UTC [55856] LOG: database system is ready to accept connections503/private/var/tb/3/scratch/62601a543565e631/build/postgres2134869968:5432 - accepting connections504{"timestamp":"2026-11-12T00:20:16.044231Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"d8bda38a-dbca-48c7-bc18-12ff48bad942","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":13,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(11)"}505{"timestamp":"2026-11-12T00:20:16.154793Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"5724a409-16c0-4a0e-a2bd-5b40f556f3fa","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(10)"}506{"timestamp":"2026-11-12T00:20:16.25726Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"c4440318-c6cd-43a1-8be9-46a6293e277d","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(5)"}507=== RUN TestAbortSucceeded508=== PAUSE TestAbortSucceeded509=== RUN TestService_AuthMiddleware510=== PAUSE TestService_AuthMiddleware511=== RUN TestService_AuthMiddleware_MTLSProxyHeader512=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader513=== RUN TestService_AuthMiddleware_MTLSBoundSubjects514=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects515=== RUN TestService_ReadAuthMiddleware516=== PAUSE TestService_ReadAuthMiddleware517=== RUN TestService_AuthMiddleware_OIDC518=== PAUSE TestService_AuthMiddleware_OIDC519=== RUN TestService_RequireScope_OIDC520=== PAUSE TestService_RequireScope_OIDC521=== RUN TestService_ReadScope_PublicByDefault522=== PAUSE TestService_ReadScope_PublicByDefault523=== RUN TestCacheConfigHandler524=== PAUSE TestCacheConfigHandler525=== RUN TestCacheStatsHandler526=== PAUSE TestCacheStatsHandler527=== RUN TestClientCADerivations528=== PAUSE TestClientCADerivations529=== RUN TestClientErrorHandling530=== PAUSE TestClientErrorHandling531=== RUN TestClientIntegration532=== PAUSE TestClientIntegration533=== RUN TestClientMultipleUploads534=== PAUSE TestClientMultipleUploads535=== RUN TestClientWithDependencies536=== PAUSE TestClientWithDependencies537=== RUN TestClientSharedPathCommittedMidPush538=== PAUSE TestClientSharedPathCommittedMidPush539=== RUN TestPinProtectsFromGC540=== PAUSE TestPinProtectsFromGC541=== RUN TestClientPushesUseOnePush542=== PAUSE TestClientPushesUseOnePush543=== RUN TestClientFallsBackToClosures544=== PAUSE TestClientFallsBackToClosures545=== RUN TestDeleteClosuresRacingPins546=== PAUSE TestDeleteClosuresRacingPins547=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock548=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock549=== RUN TestResolveDBConnectionString550=== PAUSE TestResolveDBConnectionString551=== RUN TestConnectWaitsForAPeerMigration552=== PAUSE TestConnectWaitsForAPeerMigration553=== RUN TestLeadElectsOneAndHandsOver554=== PAUSE TestLeadElectsOneAndHandsOver555=== RUN TestLeadIncumbentWinsAfterRestart5562026/11/12 00:20:16 INFO lead: acquired remote=192.0.2.1:1234 generation=15572026/11/12 00:20:17 INFO lead: released remote=192.0.2.1:1234 generation=15582026/11/12 00:20:17 INFO lead: acquired remote=192.0.2.1:1234 generation=25592026/11/12 00:20:18 INFO lead: released remote=192.0.2.1:1234 generation=2560--- PASS: TestLeadIncumbentWinsAfterRestart (2.14s)561=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent562=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent563=== RUN TestLeadIncumbentAnnouncesAtOnce564=== PAUSE TestLeadIncumbentAnnouncesAtOnce565=== RUN TestLeadGenerationGrowsWithEveryGrant566=== PAUSE TestLeadGenerationGrowsWithEveryGrant567=== RUN TestLeadEndsOnShutdown568=== PAUSE TestLeadEndsOnShutdown569=== RUN TestLeadEndsWhenItsConnectionHangs5702026/11/12 00:20:18 INFO lead: acquired remote=192.0.2.1:1234 generation=15712026-11-12 00:20:20.111 UTC [56527] FATAL: terminating connection due to administrator command5722026/11/12 00:20:20 INFO lead: acquired remote=192.0.2.1:1234 generation=25732026/11/12 00:20:21 WARN lead: lock connection lost error="timeout: context deadline exceeded"5742026/11/12 00:20:21 INFO lead: released remote=192.0.2.1:1234 generation=15752026/11/12 00:20:21 INFO lead: released remote=192.0.2.1:1234 generation=2576--- PASS: TestLeadEndsWhenItsConnectionHangs (3.01s)577=== RUN TestGCAdvisoryLockBlocksConcurrentRun578=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun579=== RUN TestGCBugBareHashReferences580=== PAUSE TestGCBugBareHashReferences581=== RUN TestGCMetrics582=== PAUSE TestGCMetrics583=== RUN TestPushDedupSurvivesConcurrentGC584=== PAUSE TestPushDedupSurvivesConcurrentGC585=== RUN TestDeduplicatedObjectsRecordedAsPending586=== PAUSE TestDeduplicatedObjectsRecordedAsPending587=== RUN TestGCSweepSkipsPendingObjects588=== PAUSE TestGCSweepSkipsPendingObjects589=== RUN TestTombstonedObjectOfferedWithoutWaiting590=== PAUSE TestTombstonedObjectOfferedWithoutWaiting591=== RUN TestGCSweepDeliversEachKeyOnce592=== PAUSE TestGCSweepDeliversEachKeyOnce593=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection594=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection595=== RUN TestForceGCDuringPushOffersSweptObject596=== PAUSE TestForceGCDuringPushOffersSweptObject597=== RUN TestFailedS3DeleteKeepsTombstone598=== PAUSE TestFailedS3DeleteKeepsTombstone599=== RUN TestSweepSkipsRowsLockedByPush600=== PAUSE TestSweepSkipsRowsLockedByPush601=== RUN TestRegisterCompletedObjectNeedsPendingRow602=== PAUSE TestRegisterCompletedObjectNeedsPendingRow603=== RUN TestGetExistingObjectsLocksRows604=== PAUSE TestGetExistingObjectsLocksRows605=== RUN TestSweepRechecksPendingAfterLocking606=== PAUSE TestSweepRechecksPendingAfterLocking607=== RUN TestPushWaitsForSweepHoldingRow608=== PAUSE TestPushWaitsForSweepHoldingRow609=== RUN TestCommitRacingPendingCleanupKeepsObjects610=== PAUSE TestCommitRacingPendingCleanupKeepsObjects611=== RUN TestGCEndsOnShutdown612=== PAUSE TestGCEndsOnShutdown613=== RUN TestGCTaskStore_StartNew614=== PAUSE TestGCTaskStore_StartNew615=== RUN TestGCTaskStore_DeduplicateSameParams616=== PAUSE TestGCTaskStore_DeduplicateSameParams617=== RUN TestGCTaskStore_ConflictDifferentParams618=== PAUSE TestGCTaskStore_ConflictDifferentParams619=== RUN TestGCTaskStore_GetEmpty620=== PAUSE TestGCTaskStore_GetEmpty621=== RUN TestGCTaskStore_GetReturnsLatest622=== PAUSE TestGCTaskStore_GetReturnsLatest623=== RUN TestGCTaskStore_CompletedAllowsNewTask624=== PAUSE TestGCTaskStore_CompletedAllowsNewTask625=== RUN TestGCTaskStore_PhaseUpdates626=== PAUSE TestGCTaskStore_PhaseUpdates627=== RUN TestGCTaskStore_Fail628=== PAUSE TestGCTaskStore_Fail629=== RUN TestGracefulShutdownDrainsInflight630=== PAUSE TestGracefulShutdownDrainsInflight631=== RUN TestService_healthCheckHandler632=== PAUSE TestService_healthCheckHandler633=== RUN TestService_readinessHandler634=== PAUSE TestService_readinessHandler635=== RUN TestGenerateLandingPage636=== PAUSE TestGenerateLandingPage637=== RUN TestEnsureAbortUploadsRule638=== PAUSE TestEnsureAbortUploadsRule639=== RUN TestCacheConfigHandlerMaxNarSize640=== PAUSE TestCacheConfigHandlerMaxNarSize641=== RUN TestCreatePendingClosureRejectsOversizedNAR642=== PAUSE TestCreatePendingClosureRejectsOversizedNAR643=== RUN TestNARDeduplicationMetadataUploadBug644=== PAUSE TestNARDeduplicationMetadataUploadBug645=== RUN TestMetricsInventory646=== PAUSE TestMetricsInventory647=== RUN TestMetricsCommit648=== PAUSE TestMetricsCommit649=== RUN TestService_NativeMTLS650=== PAUSE TestService_NativeMTLS651=== RUN TestServerTLSConfig652=== PAUSE TestServerTLSConfig653=== RUN TestMultipartCleanup654=== PAUSE TestMultipartCleanup655=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded656=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded657=== RUN TestPendingCleanupUsesOneCutoff658=== PAUSE TestPendingCleanupUsesOneCutoff659=== RUN TestObjectStatsTrigger660=== PAUSE TestObjectStatsTrigger661=== RUN TestObjectStatsUpdatedOncePerCommit662=== PAUSE TestObjectStatsUpdatedOncePerCommit663=== RUN TestOrphanedObjectsGC664=== PAUSE TestOrphanedObjectsGC665=== RUN TestOrphanedObjectsGCStressTest666=== PAUSE TestOrphanedObjectsGCStressTest667=== RUN TestCreatePin_ReservedPins668=== PAUSE TestCreatePin_ReservedPins669=== RUN TestPresentReportsOnlyClosureRoots670=== PAUSE TestPresentReportsOnlyClosureRoots671=== RUN TestPresentRefreshesOnlyStaleClosures672=== PAUSE TestPresentRefreshesOnlyStaleClosures673=== RUN TestPresentNotReportedWhileGCDeletesClosure674=== PAUSE TestPresentNotReportedWhileGCDeletesClosure675=== RUN TestParseSingleRange676=== PAUSE TestParseSingleRange677=== RUN TestProxyHeadersOnlyTrustedOnSocket6782026/11/12 00:20:21 INFO Starting HTTP server address=127.0.0.1:644776792026/11/12 00:20:21 INFO Starting HTTP server address=proxy.sock6802026/11/12 00:20:21 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6812026/11/12 00:20:21 INFO Shutdown signal received, draining in-flight requests timeout=10s682--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.21s)683=== RUN TestIsValidCachePath684=== PAUSE TestIsValidCachePath685=== RUN TestReadProxyNarinfo686=== PAUSE TestReadProxyNarinfo687=== RUN TestReadProxyNarinfoAlreadyDecompressed688=== PAUSE TestReadProxyNarinfoAlreadyDecompressed689=== RUN TestReadProxyContentNegotiation690=== PAUSE TestReadProxyContentNegotiation691=== RUN TestReadProxyNarStreaming692=== PAUSE TestReadProxyNarStreaming693=== RUN TestReadProxy404694=== PAUSE TestReadProxy404695=== RUN TestReadProxyInvalidPath696=== PAUSE TestReadProxyInvalidPath697=== RUN TestReadProxyHead698=== PAUSE TestReadProxyHead699=== RUN TestReadProxyOutlastsServerWriteTimeout700=== PAUSE TestReadProxyOutlastsServerWriteTimeout701=== RUN TestReadProxyConditionalGet702=== PAUSE TestReadProxyConditionalGet703=== RUN TestReadProxyRootRedirectsToIndexHTML704=== PAUSE TestReadProxyRootRedirectsToIndexHTML705=== RUN TestReadProxyDisabled706=== PAUSE TestReadProxyDisabled707=== RUN TestReadRedirectNar708=== PAUSE TestReadRedirectNar709=== RUN TestReadRedirectKeepsNarinfoProxied710=== PAUSE TestReadRedirectKeepsNarinfoProxied711=== RUN TestReadProxyRangeRequest712=== PAUSE TestReadProxyRangeRequest713=== RUN TestReadRedirectUsesPublicS3URL714=== PAUSE TestReadRedirectUsesPublicS3URL715=== RUN TestPush_OverlappingRootsStoreOneRowPerKey716=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey717=== RUN TestPush_CompleteCommitsEveryRoot718=== PAUSE TestPush_CompleteCommitsEveryRoot719=== RUN TestPush_RepeatedRootCommitsOnce720=== PAUSE TestPush_RepeatedRootCommitsOnce721=== RUN TestPush_CompleteWaitsForCleanupOfThePush722=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush723=== RUN TestPush_RootsAreLockedInByteOrder724=== PAUSE TestPush_RootsAreLockedInByteOrder725=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit726=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit727=== RUN TestPush_RejectsBadRequests728=== PAUSE TestPush_RejectsBadRequests729=== RUN TestPush_SignsNarinfosOfItsPendingObjects730=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects731=== RUN TestSharedNarBothUploadsComplete732=== PAUSE TestSharedNarBothUploadsComplete733=== RUN TestCompleteMultipartUpload_ErrorButObjectExists734=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists735=== RUN TestCompletedNarNotReofferedAcrossClosures736=== PAUSE TestCompletedNarNotReofferedAcrossClosures737=== RUN TestPresignedUploadRegisteredBeforeCommit738=== PAUSE TestPresignedUploadRegisteredBeforeCommit739=== RUN TestService_Rustfstest740=== PAUSE TestService_Rustfstest741=== RUN TestParseSize742=== PAUSE TestParseSize743=== RUN TestSkippedUploadsHandler744=== PAUSE TestSkippedUploadsHandler745=== RUN TestSystemdListenerNotActivated746--- PASS: TestSystemdListenerNotActivated (0.00s)747=== RUN TestWatchdogBeatsWhenHealthy748--- PASS: TestWatchdogBeatsWhenHealthy (0.03s)749=== RUN TestWatchdogSkipsWhenUnhealthy7502026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7532026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7542026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7552026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7562026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7582026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7592026/11/12 00:20:21 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"760--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)761=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle762=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle763=== RUN TestProxyWriteTimeout764=== PAUSE TestProxyWriteTimeout765=== RUN TestPendingClosureWriteTimeout766=== PAUSE TestPendingClosureWriteTimeout767=== RUN TestIsValidUploadKey768=== PAUSE TestIsValidUploadKey769=== RUN TestUploadHandlersRejectInvalidKeys770=== PAUSE TestUploadHandlersRejectInvalidKeys771=== RUN TestUploadHandlersRejectOversizedBody772=== PAUSE TestUploadHandlersRejectOversizedBody773=== RUN TestService_cleanupPendingClosuresHandler774=== PAUSE TestService_cleanupPendingClosuresHandler775=== RUN TestService_createPendingClosureHandler776=== PAUSE TestService_createPendingClosureHandler777=== RUN TestService_verifyS3Integrity778=== PAUSE TestService_verifyS3Integrity779=== RUN TestService_verifyS3ReuploadsStaleNarinfo780=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo781=== RUN TestCompleteMultipartUnregistered782=== PAUSE TestCompleteMultipartUnregistered783=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT784=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT785=== CONT TestCompletedNarNotReofferedAcrossClosures786=== CONT TestPush_OverlappingRootsStoreOneRowPerKey787=== CONT TestAbortSucceeded788--- PASS: TestAbortSucceeded (0.00s)789=== CONT TestResolveDBConnectionString790=== CONT TestLeadIncumbentAnnouncesAtOnce791=== CONT TestGCTaskStore_Fail792--- PASS: TestGCTaskStore_Fail (0.00s)793=== CONT TestClientFallsBackToClosures794=== CONT TestGracefulShutdownDrainsInflight795=== CONT TestGCTaskStore_PhaseUpdates796--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)797=== CONT TestGCTaskStore_CompletedAllowsNewTask798--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)799=== CONT TestCompleteMultipartUpload_ErrorButObjectExists800=== CONT TestGCTaskStore_GetReturnsLatest801--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)802=== CONT TestSharedNarBothUploadsComplete803=== RUN TestResolveDBConnectionString/flag_wins804=== CONT TestPush_SignsNarinfosOfItsPendingObjects805=== PAUSE TestResolveDBConnectionString/flag_wins806=== CONT TestPush_RejectsBadRequests807=== CONT TestGCTaskStore_GetEmpty808--- PASS: TestGCTaskStore_GetEmpty (0.00s)809=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit810=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent811=== CONT TestPush_RootsAreLockedInByteOrder812=== CONT TestLeadElectsOneAndHandsOver813=== CONT TestPush_CompleteWaitsForCleanupOfThePush814=== CONT TestConnectWaitsForAPeerMigration815=== CONT TestPush_RepeatedRootCommitsOnce816=== CONT TestPush_CompleteCommitsEveryRoot817=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock818=== CONT TestDeleteClosuresRacingPins819=== CONT TestGCTaskStore_ConflictDifferentParams820--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)821=== CONT TestGCTaskStore_DeduplicateSameParams822--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)823=== CONT TestGCTaskStore_StartNew824--- PASS: TestGCTaskStore_StartNew (0.00s)825=== RUN TestResolveDBConnectionString/file_when_flag_empty826=== PAUSE TestResolveDBConnectionString/file_when_flag_empty827=== CONT TestClientPushesUseOnePush828=== CONT TestPinProtectsFromGC829=== CONT TestClientSharedPathCommittedMidPush830=== CONT TestGCEndsOnShutdown831=== RUN TestGCEndsOnShutdown/before_the_run832=== PAUSE TestGCEndsOnShutdown/before_the_run833=== RUN TestResolveDBConnectionString/missing_file_is_an_error834=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error835=== RUN TestResolveDBConnectionString/PGHOST_allows_empty836=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty837=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete838=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete839=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark840=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark841=== RUN TestResolveDBConnectionString/nothing_configured842=== CONT TestClientWithDependencies843=== PAUSE TestResolveDBConnectionString/nothing_configured844=== CONT TestCommitRacingPendingCleanupKeepsObjects8452026/11/12 00:20:21 INFO Starting HTTP server address=127.0.0.1:644798462026/11/12 00:20:21 INFO Shutdown signal received, draining in-flight requests timeout=10s847--- PASS: TestGracefulShutdownDrainsInflight (0.08s)848=== CONT TestPushWaitsForSweepHoldingRow8492026/11/12 00:20:23 INFO lead: acquired remote=192.0.2.1:1234 generation=18502026/11/12 00:20:23 INFO lead: released remote=192.0.2.1:1234 generation=1851--- PASS: TestLeadIncumbentAnnouncesAtOnce (1.52s)852=== CONT TestSweepRechecksPendingAfterLocking8532026/11/12 00:20:23 INFO Received push request method=POST path=/api/pushes8542026-11-12 00:20:24.027 UTC [56866] ERROR: could not obtain lock on row in relation "closures"8552026-11-12 00:20:24.027 UTC [56866] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8562026-11-12 00:20:24.040 UTC [56866] ERROR: could not obtain lock on row in relation "closures"8572026-11-12 00:20:24.040 UTC [56866] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8582026-11-12 00:20:24.059 UTC [56866] ERROR: could not obtain lock on row in relation "closures"8592026-11-12 00:20:24.059 UTC [56866] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8602026-11-12 00:20:24.065 UTC [56866] ERROR: could not obtain lock on row in relation "closures"8612026-11-12 00:20:24.065 UTC [56866] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8622026-11-12 00:20:24.084 UTC [56866] ERROR: could not obtain lock on row in relation "closures"8632026-11-12 00:20:24.084 UTC [56866] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT8642026/11/12 00:20:24 INFO Received uploads request method=POST path=/api/pending_closures865--- PASS: TestPush_RootsAreLockedInByteOrder (2.15s)866=== CONT TestGetExistingObjectsLocksRows8672026/11/12 00:20:24 INFO Received push request method=POST path=/api/pushes8682026/11/12 00:20:24 INFO Received complete multipart upload request method=POST path=/api/multipart/complete8692026/11/12 00:20:24 INFO Received uploads request method=POST path=/api/pending_closures8702026/11/12 00:20:24 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ODAyYzY4Y2EtMTk0Mi00YjZhLTg1ZGQtODUzY2NlNTdiOGNhLmZmYWM2M2RhLTFhYWItNDNmNy05YmZiLWZlYjc2ODQ3ZDg3ZngxNzk0NDQyODI0MTU3MjUyMDAw8712026/11/12 00:20:24 INFO Received uploads request method=POST path=/api/pending_closures872--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (2.57s)873=== CONT TestClientMultipleUploads8742026/11/12 00:20:24 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)8752026/11/12 00:20:24 INFO Uploading w3yj3fn9w5sl0v1p1yr5dsxnmyvrql8q-shared-dep (136B)8762026/11/12 00:20:24 INFO Uploading v5r6ksc2ssaa4gsndhxrl14ib30rzz23-a (256B)8772026/11/12 00:20:24 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ODAyYzY4Y2EtMTk0Mi00YjZhLTg1ZGQtODUzY2NlNTdiOGNhLmZmYWM2M2RhLTFhYWItNDNmNy05YmZiLWZlYjc2ODQ3ZDg3ZngxNzk0NDQyODI0MTU3MjUyMDAw parts=1878--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (2.61s)879=== CONT TestRegisterCompletedObjectNeedsPendingRow8802026/11/12 00:20:24 WARN Failed to register uploaded object key=46vscr67qr3j448b962yv9jggpm88kln.ls error="server returned 404: 404 page not found\n"8812026/11/12 00:20:24 WARN Failed to register uploaded object key=v5r6ksc2ssaa4gsndhxrl14ib30rzz23.ls error="server returned 404: 404 page not found\n"8822026/11/12 00:20:24 WARN Failed to register uploaded object key=w3yj3fn9w5sl0v1p1yr5dsxnmyvrql8q.ls error="server returned 404: 404 page not found\n"8832026/11/12 00:20:24 WARN Failed to register uploaded object key=nar/1knsazlc86zimz0nmrcqs9x48457kk14w16agwisw26i9n453pwx.nar.zst error="server returned 404: 404 page not found\n"8842026/11/12 00:20:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign8852026/11/12 00:20:24 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"8862026/11/12 00:20:24 INFO Signed narinfos id=1 count=28872026/11/12 00:20:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign8882026/11/12 00:20:24 INFO Signed narinfos id=2 count=28892026/11/12 00:20:24 INFO Uploading 4 narinfos8902026/11/12 00:20:24 WARN Failed to register uploaded object key=w3yj3fn9w5sl0v1p1yr5dsxnmyvrql8q.narinfo error="server returned 404: 404 page not found\n"8912026/11/12 00:20:24 WARN Failed to register uploaded object key=46vscr67qr3j448b962yv9jggpm88kln.narinfo error="server returned 404: 404 page not found\n"8922026/11/12 00:20:24 WARN Failed to register uploaded object key=v5r6ksc2ssaa4gsndhxrl14ib30rzz23.narinfo error="server returned 404: 404 page not found\n"8932026/11/12 00:20:24 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete8942026/11/12 00:20:24 WARN Failed to register uploaded object key=w3yj3fn9w5sl0v1p1yr5dsxnmyvrql8q.narinfo error="server returned 404: 404 page not found\n"8952026/11/12 00:20:24 INFO Received push request method=POST path=/api/pushes8962026/11/12 00:20:24 INFO Completed upload id=1 objects=6 duration=8.158125ms8972026/11/12 00:20:24 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete8982026/11/12 00:20:24 INFO Completed upload id=2 objects=6 duration=1.8835ms8992026/11/12 00:20:24 INFO Upload complete. (637ms)900=== NAME TestClientFallsBackToClosures901 client_pushes_test.go:112: Retrieved narinfo from S3:902 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientFallsBackToClosures1354173321/001/store/w3yj3fn9w5sl0v1p1yr5dsxnmyvrql8q-shared-dep903 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst904 Compression: zstd905 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82906 NarSize: 136907 References: 908 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n909 client_pushes_test.go:112: Retrieved narinfo from S3:910 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientFallsBackToClosures1354173321/001/store/v5r6ksc2ssaa4gsndhxrl14ib30rzz23-a911 URL: nar/1knsazlc86zimz0nmrcqs9x48457kk14w16agwisw26i9n453pwx.nar.zst912 Compression: zstd913 NarHash: sha256:1knsazlc86zimz0nmrcqs9x48457kk14w16agwisw26i9n453pwx914 NarSize: 256915 References: /private/var/tb/3/scratch/62601a543565e631/build/TestClientFallsBackToClosures1354173321/001/store/w3yj3fn9w5sl0v1p1yr5dsxnmyvrql8q-shared-dep916 CA: text:sha256:0mkwj7x0nx9iw19ajvf73jgjjdahzll4mw5gq65x9hvr41jvssy8917 client_pushes_test.go:112: Retrieved narinfo from S3:918 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientFallsBackToClosures1354173321/001/store/46vscr67qr3j448b962yv9jggpm88kln-b919 URL: nar/1knsazlc86zimz0nmrcqs9x48457kk14w16agwisw26i9n453pwx.nar.zst920 Compression: zstd921 NarHash: sha256:1knsazlc86zimz0nmrcqs9x48457kk14w16agwisw26i9n453pwx922 NarSize: 256923 References: /private/var/tb/3/scratch/62601a543565e631/build/TestClientFallsBackToClosures1354173321/001/store/w3yj3fn9w5sl0v1p1yr5dsxnmyvrql8q-shared-dep924 CA: text:sha256:0mkwj7x0nx9iw19ajvf73jgjjdahzll4mw5gq65x9hvr41jvssy8925--- PASS: TestClientFallsBackToClosures (2.80s)926=== CONT TestClientIntegration9272026/11/12 00:20:24 INFO Received complete push request method=POST path=/api/pushes/1/complete9282026/11/12 00:20:24 INFO Completed push id=1 objects=4 duration=27.901459ms929--- PASS: TestPush_RepeatedRootCommitsOnce (2.83s)930=== CONT TestClientErrorHandling931=== RUN TestClientErrorHandling/InvalidStorePath932=== PAUSE TestClientErrorHandling/InvalidStorePath933=== RUN TestClientErrorHandling/InvalidAuthToken934=== PAUSE TestClientErrorHandling/InvalidAuthToken935=== RUN TestClientErrorHandling/ServerNotAvailable936=== PAUSE TestClientErrorHandling/ServerNotAvailable937=== CONT TestSweepSkipsRowsLockedByPush9382026/11/12 00:20:24 INFO Received uploads request method=POST path=/api/pending_closures9392026/11/12 00:20:25 INFO lead: acquired remote=192.0.2.1:1234 generation=19402026/11/12 00:20:25 INFO Received push request method=POST path=/api/pushes9412026/11/12 00:20:25 INFO Received complete push request method=POST path=/api/pushes/1/complete9422026/11/12 00:20:25 INFO Completed push id=1 objects=4 duration=22.75825ms9432026/11/12 00:20:25 INFO Received push request method=POST path=/api/pushes9442026/11/12 00:20:25 INFO Aborted multipart uploads count=0 kept=09452026/11/12 00:20:25 WARN Force mode enabled - objects will be deleted immediately without grace period9462026/11/12 00:20:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=09472026/11/12 00:20:25 INFO Vacuumed table table=pending_closures9482026/11/12 00:20:25 INFO Vacuumed table table=pending_objects9492026/11/12 00:20:25 INFO Vacuumed table table=multipart_uploads9502026/11/12 00:20:25 INFO Vacuumed table table=closures9512026/11/12 00:20:25 INFO Vacuumed table table=objects9522026/11/12 00:20:25 INFO Received complete push request method=POST path=/api/pushes/2/complete9532026/11/12 00:20:25 INFO Completed push id=2 objects=4 duration=12.656916ms954--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (3.51s)955=== CONT TestClientCADerivations956=== RUN TestPush_RejectsBadRequests/bad_root957=== PAUSE TestPush_RejectsBadRequests/bad_root958=== RUN TestPush_RejectsBadRequests/root_not_in_objects959=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects960=== RUN TestPush_RejectsBadRequests/no_roots961=== PAUSE TestPush_RejectsBadRequests/no_roots962=== RUN TestPush_RejectsBadRequests/no_objects963=== PAUSE TestPush_RejectsBadRequests/no_objects964=== CONT TestFailedS3DeleteKeepsTombstone965=== NAME TestPinProtectsFromGC966 client_integration_test.go:779: Pinned store path: /private/var/tb/3/scratch/62601a543565e631/build/TestPinProtectsFromGC3531001164/001/store/d8jy9kd84r5iicmwgyxmpq3iqn9mwv98-pinned-file.txt967 client_integration_test.go:780: Unpinned store path: /private/var/tb/3/scratch/62601a543565e631/build/TestPinProtectsFromGC3531001164/001/store/lrrcsg8sqdbhxdkwqfkvj35c3wmqkgqb-unpinned-file.txt9682026/11/12 00:20:25 INFO Received push request method=POST path=/api/pushes9692026/11/12 00:20:25 INFO Received push request method=POST path=/api/pushes9702026/11/12 00:20:25 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)9712026/11/12 00:20:25 INFO Uploading d8jy9kd84r5iicmwgyxmpq3iqn9mwv98-pinned-file.txt (128B)9722026/11/12 00:20:25 WARN Failed to register uploaded object key=d8jy9kd84r5iicmwgyxmpq3iqn9mwv98.ls error="server returned 404: 404 page not found\n"9732026/11/12 00:20:25 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9742026/11/12 00:20:25 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"9752026/11/12 00:20:25 INFO Signed narinfos id=1 count=19762026/11/12 00:20:25 INFO Uploading 1 narinfos9772026/11/12 00:20:25 INFO Received uploads request method=POST path=/api/pending_closures9782026/11/12 00:20:25 INFO Received complete push request method=POST path=/api/pushes/1/complete9792026/11/12 00:20:25 WARN Failed to register uploaded object key=d8jy9kd84r5iicmwgyxmpq3iqn9mwv98.narinfo error="server returned 404: 404 page not found\n"9802026/11/12 00:20:25 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9812026/11/12 00:20:25 INFO Completed push id=1 objects=3 duration=13.94375ms9822026/11/12 00:20:25 INFO Upload complete. (269ms)9832026/11/12 00:20:26 INFO Received uploads request method=POST path=/api/pending_closures9842026/11/12 00:20:26 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ODAyYzY4Y2EtMTk0Mi00YjZhLTg1ZGQtODUzY2NlNTdiOGNhLmY3M2FjMWEwLTQwNWUtNGQ3MC1iNDVjLTViMzJmNzk5NGRlYngxNzk0NDQyODI0ODgxOTIwMDAw parts=129852026/11/12 00:20:26 INFO Received uploads request method=POST path=/api/pending_closures986--- PASS: TestCompletedNarNotReofferedAcrossClosures (4.14s)987=== CONT TestCacheStatsHandler9882026-11-12 00:20:26.129 UTC [57010] ERROR: Push does not exist: id=19892026-11-12 00:20:26.129 UTC [57010] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE9902026-11-12 00:20:26.129 UTC [57010] STATEMENT: SELECT commit_push($1)991--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (4.17s)992=== CONT TestForceGCDuringPushOffersSweptObject993=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows994=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows995=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check996=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check997=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection9982026/11/12 00:20:26 INFO Received push request method=POST path=/api/pushes9992026/11/12 00:20:26 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)10002026/11/12 00:20:26 INFO Uploading lrrcsg8sqdbhxdkwqfkvj35c3wmqkgqb-unpinned-file.txt (128B)10012026/11/12 00:20:26 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"10022026/11/12 00:20:26 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign10032026/11/12 00:20:26 WARN Failed to register uploaded object key=lrrcsg8sqdbhxdkwqfkvj35c3wmqkgqb.ls error="server returned 404: 404 page not found\n"10042026/11/12 00:20:26 INFO Signed narinfos id=2 count=110052026/11/12 00:20:26 INFO lead: acquired remote=192.0.2.1:1234 generation=110062026/11/12 00:20:26 INFO Uploading 1 narinfos10072026/11/12 00:20:26 INFO lead: released remote=192.0.2.1:1234 generation=110082026/11/12 00:20:26 INFO Received complete push request method=POST path=/api/pushes/2/complete10092026/11/12 00:20:26 WARN Failed to register uploaded object key=lrrcsg8sqdbhxdkwqfkvj35c3wmqkgqb.narinfo error="server returned 404: 404 page not found\n"10102026/11/12 00:20:26 INFO Completed push id=2 objects=3 duration=1.279708ms10112026/11/12 00:20:26 INFO Upload complete. (236ms)10122026/11/12 00:20:26 INFO lead: acquired remote=192.0.2.1:1234 generation=210132026/11/12 00:20:26 INFO Starting cleanup of old closures method=DELETE path=/api/closures10142026/11/12 00:20:26 INFO Garbage collection started10152026/11/12 00:20:26 INFO Aborted multipart uploads count=0 kept=010162026/11/12 00:20:26 WARN Force mode enabled - objects will be deleted immediately without grace period10172026/11/12 00:20:26 INFO Received push request method=POST path=/api/pushes10182026/11/12 00:20:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10192026/11/12 00:20:26 INFO Signed narinfos id=1 count=110202026/11/12 00:20:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10212026/11/12 00:20:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10222026/11/12 00:20:26 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"10232026/11/12 00:20:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10242026/11/12 00:20:26 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"10252026/11/12 00:20:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10262026/11/12 00:20:26 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=110272026/11/12 00:20:26 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10282026/11/12 00:20:26 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1029--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (4.64s)1030=== CONT TestGCSweepDeliversEachKeyOnce10312026/11/12 00:20:26 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=010322026/11/12 00:20:26 INFO Vacuumed table table=pending_closures10332026/11/12 00:20:26 INFO Vacuumed table table=pending_objects10342026/11/12 00:20:26 INFO Vacuumed table table=multipart_uploads10352026/11/12 00:20:26 INFO Vacuumed table table=closures10362026/11/12 00:20:26 INFO Vacuumed table table=objects10372026/11/12 00:20:27 INFO Received push request method=POST path=/api/pushes10382026/11/12 00:20:27 INFO lead: released remote=192.0.2.1:1234 generation=110392026/11/12 00:20:27 INFO Received complete push request method=POST path=/api/pushes/1/complete10402026/11/12 00:20:27 INFO lead: acquired remote=192.0.2.1:1234 generation=210412026/11/12 00:20:27 INFO Completed push id=1 objects=6 duration=33.699666ms1042--- PASS: TestPush_CompleteCommitsEveryRoot (5.44s)1043=== CONT TestTombstonedObjectOfferedWithoutWaiting10442026/11/12 00:20:27 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=210452026/11/12 00:20:27 INFO lead: released remote=192.0.2.1:1234 generation=21046--- PASS: TestLeadElectsOneAndHandsOver (5.46s)1047=== CONT TestGCSweepSkipsPendingObjects10482026/11/12 00:20:27 INFO lead: acquired remote=192.0.2.1:1234 generation=310492026/11/12 00:20:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10502026/11/12 00:20:27 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ODAyYzY4Y2EtMTk0Mi00YjZhLTg1ZGQtODUzY2NlNTdiOGNhLjg0MWNhNGJjLTYxOTktNDlmZC1hMjhhLTUxOGU5ZjhjMzc0OHgxNzk0NDQyODI1OTc3NzgwMDAw parts=121051--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (5.78s)1052=== CONT TestDeduplicatedObjectsRecordedAsPending10532026/11/12 00:20:28 INFO Aborted multipart uploads count=0 kept=010542026/11/12 00:20:28 WARN Force mode enabled - objects will be deleted immediately without grace period10552026/11/12 00:20:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010562026/11/12 00:20:28 INFO Vacuumed table table=pending_closures10572026/11/12 00:20:28 INFO Vacuumed table table=pending_objects10582026/11/12 00:20:28 INFO Vacuumed table table=multipart_uploads10592026/11/12 00:20:28 INFO Vacuumed table table=closures10602026/11/12 00:20:28 INFO Vacuumed table table=objects1061--- PASS: TestCommitRacingPendingCleanupKeepsObjects (6.35s)1062=== CONT TestPushDedupSurvivesConcurrentGC10632026/11/12 00:20:28 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=010642026/11/12 00:20:28 INFO Received create pin request method=POST path=/api/pins/myapp1065=== NAME TestClientWithDependencies1066 client_integration_test.go:661: Built derivation: /private/var/tb/3/scratch/62601a543565e631/build/TestClientWithDependencies1610996761/001/store/fz0nzqkb5i7xfm5n2krlf2q6lk45lmbs-test-script10672026/11/12 00:20:28 INFO Created/updated pin name=myapp store_path=/private/var/tb/3/scratch/62601a543565e631/build/TestPinProtectsFromGC3531001164/001/store/d8jy9kd84r5iicmwgyxmpq3iqn9mwv98-pinned-file.txt narinfo_key=d8jy9kd84r5iicmwgyxmpq3iqn9mwv98.narinfo10682026/11/12 00:20:28 INFO Starting cleanup of old closures method=DELETE path=/api/closures10692026/11/12 00:20:28 INFO Garbage collection started10702026/11/12 00:20:28 INFO Aborted multipart uploads count=0 kept=010712026/11/12 00:20:28 WARN Force mode enabled - objects will be deleted immediately without grace period10722026/11/12 00:20:28 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010732026/11/12 00:20:28 INFO Vacuumed table table=pending_closures10742026/11/12 00:20:28 INFO Vacuumed table table=pending_objects1075--- PASS: TestConnectWaitsForAPeerMigration (6.57s)1076=== CONT TestCacheConfigHandler1077=== RUN TestCacheConfigHandler/full_config,_no_issuer1078=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1079=== RUN TestCacheConfigHandler/no_cache_url_configured1080=== PAUSE TestCacheConfigHandler/no_cache_url_configured1081=== RUN TestCacheConfigHandler/no_signing_keys1082=== PAUSE TestCacheConfigHandler/no_signing_keys1083=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1084=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator10852026/11/12 00:20:28 INFO Vacuumed table table=multipart_uploads1086=== CONT TestGCMetrics1087=== NAME TestClientWithDependencies1088 client_integration_test.go:663: Found 1 dependencies (including self)10892026/11/12 00:20:28 INFO Vacuumed table table=closures10902026/11/12 00:20:28 INFO Vacuumed table table=objects10912026/11/12 00:20:28 INFO Received uploads request method=POST path=/api/pending_closures10922026/11/12 00:20:28 INFO Received push request method=POST path=/api/pushes10932026/11/12 00:20:28 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)10942026/11/12 00:20:28 INFO Uploading fz0nzqkb5i7xfm5n2krlf2q6lk45lmbs-test-script (136B)10952026/11/12 00:20:28 WARN Failed to register uploaded object key=fz0nzqkb5i7xfm5n2krlf2q6lk45lmbs.ls error="server returned 404: 404 page not found\n"10962026/11/12 00:20:28 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10972026/11/12 00:20:28 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"10982026/11/12 00:20:28 WARN Failed to register uploaded object key=log/jw6cwlwmyapq9a50frhwlpgb6di450a8-test-script.drv error="server returned 404: 404 page not found\n"10992026/11/12 00:20:28 INFO Signed narinfos id=1 count=111002026/11/12 00:20:28 INFO Uploading 1 narinfos11012026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes11022026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete11032026/11/12 00:20:29 WARN Failed to register uploaded object key=fz0nzqkb5i7xfm5n2krlf2q6lk45lmbs.narinfo error="server returned 404: 404 page not found\n"11042026/11/12 00:20:29 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)11052026/11/12 00:20:29 INFO Uploading ab5hxawgwxflqanj48h76ya5rb6fs37g-top (264B)11062026/11/12 00:20:29 INFO Uploading w9mbc9p91czldj471196v3m88i68kc3d-shared-dep (136B)11072026/11/12 00:20:29 INFO Completed push id=1 objects=4 duration=71.377042ms11082026/11/12 00:20:29 INFO Upload complete. (437ms)1109 client_integration_test.go:665: Skipping nix copy test - isolated store (/private/var/tb/3/scratch/62601a543565e631/build/TestClientWithDependencies1610996761/001/store) requires matching store prefix11102026/11/12 00:20:29 WARN Failed to register uploaded object key=ab5hxawgwxflqanj48h76ya5rb6fs37g.ls error="server returned 404: 404 page not found\n"11112026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11122026/11/12 00:20:29 WARN Failed to register uploaded object key=nar/0sz016z0wkydk15q9x8kpdmmciihasczpv84fwblsb16szjsf816.nar.zst error="server returned 404: 404 page not found\n"11132026/11/12 00:20:29 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11142026/11/12 00:20:29 WARN Failed to register uploaded object key=w9mbc9p91czldj471196v3m88i68kc3d.ls error="server returned 404: 404 page not found\n"11152026/11/12 00:20:29 INFO Signed narinfos id=1 count=211162026/11/12 00:20:29 INFO Uploading 2 narinfos1117--- PASS: TestPushWaitsForSweepHoldingRow (7.22s)1118=== CONT TestService_ReadScope_PublicByDefault1119--- PASS: TestClientWithDependencies (7.33s)1120=== CONT TestGCBugBareHashReferences11212026/11/12 00:20:29 WARN Failed to register uploaded object key=ab5hxawgwxflqanj48h76ya5rb6fs37g.narinfo error="server returned 404: 404 page not found\n"11222026/11/12 00:20:29 INFO Received complete push request method=POST path=/api/pushes/1/complete11232026/11/12 00:20:29 WARN Failed to register uploaded object key=w9mbc9p91czldj471196v3m88i68kc3d.narinfo error="server returned 404: 404 page not found\n"11242026/11/12 00:20:29 INFO lead: released remote=192.0.2.1:1234 generation=31125--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (7.37s)1126=== CONT TestService_RequireScope_OIDC11272026/11/12 00:20:29 INFO Completed push id=1 objects=6 duration=16.483667ms11282026/11/12 00:20:29 INFO Upload complete. (464ms)1129=== NAME TestClientSharedPathCommittedMidPush1130 client_integration_test.go:728: Retrieved narinfo from S3:1131 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientSharedPathCommittedMidPush564419509/001/store/w9mbc9p91czldj471196v3m88i68kc3d-shared-dep1132 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1133 Compression: zstd1134 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821135 NarSize: 1361136 References: 1137 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1138 client_integration_test.go:728: Retrieved narinfo from S3:1139 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientSharedPathCommittedMidPush564419509/001/store/ab5hxawgwxflqanj48h76ya5rb6fs37g-top1140 URL: nar/0sz016z0wkydk15q9x8kpdmmciihasczpv84fwblsb16szjsf816.nar.zst1141 Compression: zstd1142 NarHash: sha256:0sz016z0wkydk15q9x8kpdmmciihasczpv84fwblsb16szjsf8161143 NarSize: 2641144 References: /private/var/tb/3/scratch/62601a543565e631/build/TestClientSharedPathCommittedMidPush564419509/001/store/w9mbc9p91czldj471196v3m88i68kc3d-shared-dep1145 CA: text:sha256:0aa2si8rjhyqbi2w2h8azid6a5yi0396pjj60sy2mp9yrn9njpdl11462026/11/12 00:20:29 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:64554/oidc1147--- PASS: TestClientSharedPathCommittedMidPush (7.51s)1148=== CONT TestGCAdvisoryLockBlocksConcurrentRun11492026/11/12 00:20:29 INFO Aborted multipart uploads count=0 kept=011502026/11/12 00:20:29 WARN Force mode enabled - objects will be deleted immediately without grace period11512026/11/12 00:20:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011522026/11/12 00:20:29 INFO Vacuumed table table=pending_closures11532026/11/12 00:20:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11542026/11/12 00:20:29 INFO Received push request method=POST path=/api/pushes11552026/11/12 00:20:30 INFO Vacuumed table table=pending_objects11562026/11/12 00:20:30 INFO Vacuumed table table=multipart_uploads11572026/11/12 00:20:30 INFO Vacuumed table table=closures11582026/11/12 00:20:30 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)11592026/11/12 00:20:30 INFO Uploading r2r9b5fhn0vyvvfaihkfyyfj4pcvws23-shared-dep (136B)11602026/11/12 00:20:30 INFO Uploading iq0jg0fmm2wwq98qrfk1i4g3xkv9jn9b-a (256B)11612026/11/12 00:20:30 INFO Vacuumed table table=objects1162--- PASS: TestSweepRechecksPendingAfterLocking (6.66s)1163=== CONT TestService_AuthMiddleware_OIDC11642026/11/12 00:20:30 WARN Failed to register uploaded object key=iq0jg0fmm2wwq98qrfk1i4g3xkv9jn9b.ls error="server returned 404: 404 page not found\n"11652026/11/12 00:20:30 WARN Failed to register uploaded object key=f9calhjkgkzhl96i0x0r6qh5g6qcqm8i.ls error="server returned 404: 404 page not found\n"11662026/11/12 00:20:30 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ODAyYzY4Y2EtMTk0Mi00YjZhLTg1ZGQtODUzY2NlNTdiOGNhLmE3YmFkYzA0LWVmNWItNDBmOS1hM2YyLTA2YTlmY2Q3ZWMxN3gxNzk0NDQyODI2MDQ3ODc2MDAw parts=121167--- PASS: TestSharedNarBothUploadsComplete (8.21s)11682026/11/12 00:20:30 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"11692026/11/12 00:20:30 WARN Failed to register uploaded object key=nar/0h7zwf07vzpcff1f0hil8m97v6lj414nhhji499vvymis31p4pmi.nar.zst error="server returned 404: 404 page not found\n"1170=== CONT TestLeadEndsOnShutdown11712026/11/12 00:20:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign11722026/11/12 00:20:30 WARN Failed to register uploaded object key=r2r9b5fhn0vyvvfaihkfyyfj4pcvws23.ls error="server returned 404: 404 page not found\n"11732026/11/12 00:20:30 INFO Signed narinfos id=1 count=31174--- PASS: TestGetExistingObjectsLocksRows (6.07s)1175=== CONT TestLeadGenerationGrowsWithEveryGrant11762026/11/12 00:20:30 INFO Uploading 3 narinfos11772026/11/12 00:20:30 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:64565/oidc11782026/11/12 00:20:30 WARN Failed to register uploaded object key=r2r9b5fhn0vyvvfaihkfyyfj4pcvws23.narinfo error="server returned 404: 404 page not found\n"11792026/11/12 00:20:30 INFO Received complete push request method=POST path=/api/pushes/1/complete11802026/11/12 00:20:30 WARN Failed to register uploaded object key=iq0jg0fmm2wwq98qrfk1i4g3xkv9jn9b.narinfo error="server returned 404: 404 page not found\n"11812026/11/12 00:20:30 WARN Failed to register uploaded object key=f9calhjkgkzhl96i0x0r6qh5g6qcqm8i.narinfo error="server returned 404: 404 page not found\n"11822026/11/12 00:20:30 INFO Completed push id=1 objects=8 duration=58.678ms11832026/11/12 00:20:30 INFO Upload complete. (558ms)1184=== NAME TestClientPushesUseOnePush1185 client_pushes_test.go:97: Retrieved narinfo from S3:1186 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientPushesUseOnePush2718764164/001/store/r2r9b5fhn0vyvvfaihkfyyfj4pcvws23-shared-dep1187 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1188 Compression: zstd1189 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821190 NarSize: 1361191 References: 1192 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1193 client_pushes_test.go:97: Retrieved narinfo from S3:1194 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientPushesUseOnePush2718764164/001/store/iq0jg0fmm2wwq98qrfk1i4g3xkv9jn9b-a1195 URL: nar/0h7zwf07vzpcff1f0hil8m97v6lj414nhhji499vvymis31p4pmi.nar.zst1196 Compression: zstd1197 NarHash: sha256:0h7zwf07vzpcff1f0hil8m97v6lj414nhhji499vvymis31p4pmi1198 NarSize: 2561199 References: /private/var/tb/3/scratch/62601a543565e631/build/TestClientPushesUseOnePush2718764164/001/store/r2r9b5fhn0vyvvfaihkfyyfj4pcvws23-shared-dep1200 CA: text:sha256:1mnzgj9bmby4xm2jhv5q0igj6ir3amlgnhmj33928l0sv57flbai1201 client_pushes_test.go:97: Retrieved narinfo from S3:1202 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientPushesUseOnePush2718764164/001/store/f9calhjkgkzhl96i0x0r6qh5g6qcqm8i-b1203 URL: nar/0h7zwf07vzpcff1f0hil8m97v6lj414nhhji499vvymis31p4pmi.nar.zst1204 Compression: zstd1205 NarHash: sha256:0h7zwf07vzpcff1f0hil8m97v6lj414nhhji499vvymis31p4pmi1206 NarSize: 2561207 References: /private/var/tb/3/scratch/62601a543565e631/build/TestClientPushesUseOnePush2718764164/001/store/r2r9b5fhn0vyvvfaihkfyyfj4pcvws23-shared-dep1208 CA: text:sha256:1mnzgj9bmby4xm2jhv5q0igj6ir3amlgnhmj33928l0sv57flbai1209--- PASS: TestClientPushesUseOnePush (8.42s)1210=== CONT TestUploadHandlersRejectInvalidKeys1211=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1212=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1213=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1214=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal1215=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1216=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key1217=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1218=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key1219=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT1220=== NAME TestPinProtectsFromGC1221 client_integration_test.go:888: Pin successfully protected closure from garbage collection1222--- PASS: TestPinProtectsFromGC (8.57s)1223=== CONT TestCompleteMultipartUnregistered1224=== NAME TestClientMultipleUploads1225 client_integration_test.go:406: Created store path 0: /private/var/tb/3/scratch/62601a543565e631/build/TestClientMultipleUploads1953410901/001/store/49hy35qsc9pw6c5v5j74flgn7dsfaczs-test-file-0.txt1226--- PASS: TestRegisterCompletedObjectNeedsPendingRow (6.32s)1227=== CONT TestService_verifyS3ReuploadsStaleNarinfo1228=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1229=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1230=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1231=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed1232=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1233=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1234=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1235=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1236=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1237=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1238=== CONT TestService_verifyS3Integrity1239=== NAME TestClientMultipleUploads1240 client_integration_test.go:406: Created store path 1: /private/var/tb/3/scratch/62601a543565e631/build/TestClientMultipleUploads1953410901/001/store/m86wb5p5a869ddjl8wjqi6mw6x334yv0-test-file-1.txt1241 client_integration_test.go:406: Created store path 2: /private/var/tb/3/scratch/62601a543565e631/build/TestClientMultipleUploads1953410901/001/store/k0h6qmw8k3245v4zwdcqwih7kzyna5wl-test-file-2.txt1242--- PASS: TestDeleteClosuresRacingPins (9.27s)1243=== CONT TestService_createPendingClosureHandler12442026/11/12 00:20:31 INFO Received push request method=POST path=/api/pushes1245=== NAME TestClientIntegration1246 client_integration_test.go:292: Created store path: /private/var/tb/3/scratch/62601a543565e631/build/TestClientIntegration3060554255/002/store/vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27-test-file.txt12472026/11/12 00:20:31 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)12482026/11/12 00:20:31 INFO Uploading 49hy35qsc9pw6c5v5j74flgn7dsfaczs-test-file-0.txt (160B)12492026/11/12 00:20:31 INFO Uploading k0h6qmw8k3245v4zwdcqwih7kzyna5wl-test-file-2.txt (160B)12502026/11/12 00:20:31 INFO Uploading m86wb5p5a869ddjl8wjqi6mw6x334yv0-test-file-1.txt (160B)12512026/11/12 00:20:31 WARN Failed to register uploaded object key=49hy35qsc9pw6c5v5j74flgn7dsfaczs.ls error="server returned 404: 404 page not found\n"12522026/11/12 00:20:31 WARN Failed to register uploaded object key=k0h6qmw8k3245v4zwdcqwih7kzyna5wl.ls error="server returned 404: 404 page not found\n"12532026/11/12 00:20:31 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"12542026/11/12 00:20:31 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"12552026/11/12 00:20:31 WARN Failed to register uploaded object key=m86wb5p5a869ddjl8wjqi6mw6x334yv0.ls error="server returned 404: 404 page not found\n"12562026/11/12 00:20:31 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12572026/11/12 00:20:31 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"12582026/11/12 00:20:31 INFO Signed narinfos id=1 count=312592026/11/12 00:20:31 INFO Uploading 3 narinfos12602026/11/12 00:20:31 WARN Failed to register uploaded object key=m86wb5p5a869ddjl8wjqi6mw6x334yv0.narinfo error="server returned 404: 404 page not found\n"12612026/11/12 00:20:31 INFO Received complete push request method=POST path=/api/pushes/1/complete12622026/11/12 00:20:31 WARN Failed to register uploaded object key=k0h6qmw8k3245v4zwdcqwih7kzyna5wl.narinfo error="server returned 404: 404 page not found\n"12632026/11/12 00:20:31 WARN Failed to register uploaded object key=49hy35qsc9pw6c5v5j74flgn7dsfaczs.narinfo error="server returned 404: 404 page not found\n"12642026/11/12 00:20:31 INFO Received push request method=POST path=/api/pushes12652026/11/12 00:20:31 INFO Completed push id=1 objects=9 duration=19.001167ms12662026/11/12 00:20:31 INFO Upload complete. (483ms)1267=== NAME TestClientMultipleUploads1268 client_integration_test.go:417: Uploaded 3 paths in 678.471833ms12692026/11/12 00:20:31 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12702026/11/12 00:20:31 INFO Uploading vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27-test-file.txt (152B)1271--- PASS: TestClientMultipleUploads (7.43s)1272=== CONT TestService_cleanupPendingClosuresHandler12732026/11/12 00:20:31 WARN Failed to register uploaded object key=vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27.ls error="server returned 404: 404 page not found\n"12742026/11/12 00:20:31 INFO Aborted multipart uploads count=0 kept=012752026/11/12 00:20:31 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12762026/11/12 00:20:31 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"12772026/11/12 00:20:31 INFO Signed narinfos id=1 count=112782026/11/12 00:20:31 INFO Uploading 1 narinfos12792026/11/12 00:20:32 WARN Force mode enabled - objects will be deleted immediately without grace period12802026/11/12 00:20:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012812026/11/12 00:20:32 INFO Vacuumed table table=pending_closures12822026/11/12 00:20:32 INFO Vacuumed table table=pending_objects12832026/11/12 00:20:32 INFO Vacuumed table table=multipart_uploads12842026/11/12 00:20:32 INFO Vacuumed table table=closures12852026/11/12 00:20:32 INFO Received complete push request method=POST path=/api/pushes/1/complete12862026/11/12 00:20:32 WARN Failed to register uploaded object key=vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27.narinfo error="server returned 404: 404 page not found\n"12872026/11/12 00:20:32 INFO Vacuumed table table=objects12882026/11/12 00:20:32 INFO Completed push id=1 objects=3 duration=31.345458ms12892026/11/12 00:20:32 INFO Upload complete. (369ms)12902026/11/12 00:20:32 INFO Aborted multipart uploads count=0 kept=012912026/11/12 00:20:32 WARN Force mode enabled - objects will be deleted immediately without grace period12922026/11/12 00:20:32 INFO Received uploads request method=POST path=/api/pending_closures12932026/11/12 00:20:32 INFO All 1 paths already cached1294=== NAME TestClientIntegration1295 client_integration_test.go:318: Retrieved narinfo from S3:1296 StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientIntegration3060554255/002/store/vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27-test-file.txt1297 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1298 Compression: zstd1299 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11300 NarSize: 1521301 References: 1302 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11303 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1304 client_integration_test.go:319: Decompressed .ls content (64 bytes):1305 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}13062026/11/12 00:20:32 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=013072026/11/12 00:20:32 INFO Vacuumed table table=pending_closures13082026/11/12 00:20:32 INFO Vacuumed table table=pending_objects13092026/11/12 00:20:32 INFO Vacuumed table table=multipart_uploads13102026/11/12 00:20:32 INFO Vacuumed table table=closures13112026/11/12 00:20:32 INFO Vacuumed table table=objects1312--- PASS: TestSweepSkipsRowsLockedByPush (7.51s)1313=== CONT TestUploadHandlersRejectOversizedBody13142026/11/12 00:20:32 INFO Aborted multipart uploads count=0 kept=01315=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure1316=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure1317=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart1318=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart1319=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts1320=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts1321=== CONT TestService_AuthMiddleware_MTLSProxyHeader13222026/11/12 00:20:32 WARN Force mode enabled - objects will be deleted immediately without grace period13232026/11/12 00:20:32 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."13242026/11/12 00:20:32 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1325--- PASS: TestFailedS3DeleteKeepsTombstone (6.83s)1326=== CONT TestService_AuthMiddleware_MTLSBoundSubjects13272026/11/12 00:20:32 INFO Received push request method=POST path=/api/pushes13282026/11/12 00:20:32 INFO Object in database but missing from S3 key=vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27.ls13292026/11/12 00:20:32 WARN Found objects in DB but missing from S3, will re-upload count=113302026/11/12 00:20:32 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1331--- PASS: TestCacheStatsHandler (6.54s)1332=== CONT TestService_AuthMiddleware13332026/11/12 00:20:32 INFO Received complete push request method=POST path=/api/pushes/2/complete13342026/11/12 00:20:32 INFO Completed push id=2 objects=3 duration=3.91675ms13352026/11/12 00:20:32 INFO Upload complete. (171ms)13362026/11/12 00:20:32 WARN Failed to register uploaded object key=vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27.ls error="server returned 404: 404 page not found\n"1337=== NAME TestClientIntegration1338 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1339 client_integration_test.go:336: Decompressed .ls content (49 bytes):1340 {"version":1,"root":{"type":"regular","size":39}}13412026/11/12 00:20:32 INFO Received uploads request method=POST path=/api/pending_closures1342=== NAME TestClientCADerivations1343 client_ca_test.go:136: Built CA derivation: /private/var/tb/3/scratch/62601a543565e631/build/TestClientCADerivations1914400394/001/store/66rp4d28bfkli8hldjfjxdmfp8l7l5ji-ca-test1344 client_ca_test.go:139: Found 1 dependencies (including self)13452026/11/12 00:20:33 INFO Received push request method=POST path=/api/pushes13462026/11/12 00:20:33 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)1347--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (6.98s)1348=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle13492026/11/12 00:20:33 INFO Received complete push request method=POST path=/api/pushes/3/complete13502026/11/12 00:20:33 WARN Failed to register uploaded object key=vq1kjxk26bv1vn1qnxcpdhf7p6ifzl27.ls error="server returned 404: 404 page not found\n"13512026/11/12 00:20:33 INFO Completed push id=3 objects=3 duration=4.647334ms13522026/11/12 00:20:33 INFO Upload complete. (152ms)1353=== NAME TestClientIntegration1354 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1355 client_integration_test.go:353: Decompressed .ls content (49 bytes):1356 {"version":1,"root":{"type":"regular","size":39}}1357 client_integration_test.go:364: Testing garbage collection...13582026/11/12 00:20:33 INFO Starting cleanup of old closures method=DELETE path=/api/closures13592026/11/12 00:20:33 INFO Garbage collection started13602026/11/12 00:20:33 INFO Aborted multipart uploads count=0 kept=013612026/11/12 00:20:33 INFO Aborted multipart uploads count=0 kept=013622026/11/12 00:20:33 WARN Force mode enabled - objects will be deleted immediately without grace period13632026/11/12 00:20:33 WARN Force mode enabled - objects will be deleted immediately without grace period13642026/11/12 00:20:33 INFO Received push request method=POST path=/api/pushes13652026/11/12 00:20:33 INFO Received uploads request method=POST path=/api/pending_closures13662026/11/12 00:20:33 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13672026/11/12 00:20:33 INFO Uploading 66rp4d28bfkli8hldjfjxdmfp8l7l5ji-ca-test (144B)13682026/11/12 00:20:33 WARN Failed to register uploaded object key=66rp4d28bfkli8hldjfjxdmfp8l7l5ji.ls error="server returned 404: 404 page not found\n"13692026/11/12 00:20:33 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"13702026/11/12 00:20:33 WARN Failed to register uploaded object key=log/gcxfvy4rrw7mi5bv1q4j43m7dmcdakf0-ca-test.drv error="server returned 404: 404 page not found\n"13712026/11/12 00:20:33 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13722026/11/12 00:20:33 INFO Signed narinfos id=1 count=113732026/11/12 00:20:33 INFO Uploading 1 narinfos13742026/11/12 00:20:33 INFO Aborted multipart uploads count=0 kept=013752026/11/12 00:20:33 WARN Force mode enabled - objects will be deleted immediately without grace period13762026/11/12 00:20:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013772026/11/12 00:20:33 INFO Received complete push request method=POST path=/api/pushes/1/complete13782026/11/12 00:20:33 WARN Failed to register uploaded object key=66rp4d28bfkli8hldjfjxdmfp8l7l5ji.narinfo error="server returned 404: 404 page not found\n"13792026/11/12 00:20:33 INFO Completed push id=1 objects=4 duration=89.662917ms13802026/11/12 00:20:33 INFO Upload complete. (516ms)13812026/11/12 00:20:33 INFO Vacuumed table table=pending_closures1382=== NAME TestClientCADerivations1383 client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/3/scratch/62601a543565e631/build/TestClientCADerivations1914400394/001/store/66rp4d28bfkli8hldjfjxdmfp8l7l5ji-ca-test1384 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1385 Compression: zstd1386 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1387 NarSize: 1441388 References: 1389 Deriver: /private/var/tb/3/scratch/62601a543565e631/build/TestClientCADerivations1914400394/001/store/gcxfvy4rrw7mi5bv1q4j43m7dmcdakf0-ca-test.drv1390 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1391 client_ca_test.go:185: Checking for realisation files in S3...1392 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1393 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache13942026/11/12 00:20:33 INFO Vacuumed table table=pending_objects13952026/11/12 00:20:33 INFO Vacuumed table table=multipart_uploads13962026/11/12 00:20:33 INFO Vacuumed table table=closures13972026/11/12 00:20:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=013982026/11/12 00:20:33 INFO Vacuumed table table=objects13992026/11/12 00:20:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014002026/11/12 00:20:33 INFO Vacuumed table table=pending_closures14012026/11/12 00:20:33 INFO Vacuumed table table=pending_objects14022026/11/12 00:20:33 INFO Vacuumed table table=multipart_uploads14032026/11/12 00:20:33 INFO Vacuumed table table=pending_closures14042026/11/12 00:20:33 INFO Vacuumed table table=closures1405--- PASS: TestGCSweepSkipsPendingObjects (6.32s)1406=== CONT TestIsValidUploadKey1407=== RUN TestIsValidUploadKey/narinfo1408=== PAUSE TestIsValidUploadKey/narinfo1409=== RUN TestIsValidUploadKey/nar_zst1410=== PAUSE TestIsValidUploadKey/nar_zst1411=== RUN TestIsValidUploadKey/nar_xz1412=== PAUSE TestIsValidUploadKey/nar_xz1413=== RUN TestIsValidUploadKey/nar_plain1414=== PAUSE TestIsValidUploadKey/nar_plain1415=== RUN TestIsValidUploadKey/listing1416=== PAUSE TestIsValidUploadKey/listing1417=== RUN TestIsValidUploadKey/build_log1418=== PAUSE TestIsValidUploadKey/build_log1419=== RUN TestIsValidUploadKey/build_log_home-manager_file1420=== PAUSE TestIsValidUploadKey/build_log_home-manager_file1421=== RUN TestIsValidUploadKey/build_log_plus_in_name1422=== PAUSE TestIsValidUploadKey/build_log_plus_in_name1423=== RUN TestIsValidUploadKey/build_log_question_mark1424=== PAUSE TestIsValidUploadKey/build_log_question_mark1425=== RUN TestIsValidUploadKey/build_log_equals1426=== PAUSE TestIsValidUploadKey/build_log_equals1427=== RUN TestIsValidUploadKey/realisation1428=== PAUSE TestIsValidUploadKey/realisation1429=== RUN TestIsValidUploadKey/realisation_plus_in_output1430=== PAUSE TestIsValidUploadKey/realisation_plus_in_output1431=== RUN TestIsValidUploadKey/nix-cache-info1432=== PAUSE TestIsValidUploadKey/nix-cache-info1433=== RUN TestIsValidUploadKey/index.html1434=== PAUSE TestIsValidUploadKey/index.html1435=== RUN TestIsValidUploadKey/narinfo_key,_nar_type1436=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type1437=== RUN TestIsValidUploadKey/nar_key,_narinfo_type1438=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type1439=== RUN TestIsValidUploadKey/listing_key,_narinfo_type1440=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type1441=== RUN TestIsValidUploadKey/traversal1442=== PAUSE TestIsValidUploadKey/traversal1443=== RUN TestIsValidUploadKey/traversal_nar1444=== PAUSE TestIsValidUploadKey/traversal_nar1445=== RUN TestIsValidUploadKey/absolute1446=== PAUSE TestIsValidUploadKey/absolute1447=== RUN TestIsValidUploadKey/empty_key1448=== PAUSE TestIsValidUploadKey/empty_key1449=== RUN TestIsValidUploadKey/unknown_type1450=== PAUSE TestIsValidUploadKey/unknown_type1451=== CONT TestPendingClosureWriteTimeout1452=== RUN TestPendingClosureWriteTimeout/empty1453=== PAUSE TestPendingClosureWriteTimeout/empty1454=== RUN TestPendingClosureWriteTimeout/400_objects1455=== PAUSE TestPendingClosureWriteTimeout/400_objects1456=== RUN TestPendingClosureWriteTimeout/670k_objects1457=== PAUSE TestPendingClosureWriteTimeout/670k_objects1458=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1459=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1460=== CONT TestProxyWriteTimeout1461=== RUN TestProxyWriteTimeout/narinfo1462=== PAUSE TestProxyWriteTimeout/narinfo1463=== RUN TestProxyWriteTimeout/1_GiB_nar1464=== PAUSE TestProxyWriteTimeout/1_GiB_nar1465=== RUN TestProxyWriteTimeout/10_GiB_nar1466=== PAUSE TestProxyWriteTimeout/10_GiB_nar1467=== RUN TestProxyWriteTimeout/unknown_size1468=== PAUSE TestProxyWriteTimeout/unknown_size1469=== CONT TestParseSize1470--- PASS: TestParseSize (0.00s)1471=== CONT TestSkippedUploadsHandler14722026/11/12 00:20:33 INFO Client skipped oversized paths paths=3 nar_bytes=50000000001473--- PASS: TestSkippedUploadsHandler (0.00s)14742026/11/12 00:20:33 INFO Vacuumed table table=pending_objects1475=== CONT TestPresentReportsOnlyClosureRoots14762026/11/12 00:20:33 INFO Received uploads request method=POST path=/api/pending_closures14772026/11/12 00:20:33 INFO Vacuumed table table=objects1478--- PASS: TestGCSweepDeliversEachKeyOnce (7.15s)1479=== CONT TestReadRedirectUsesPublicS3URL14802026/11/12 00:20:33 INFO Vacuumed table table=multipart_uploads14812026/11/12 00:20:33 INFO Vacuumed table table=closures1482--- PASS: TestTombstonedObjectOfferedWithoutWaiting (6.39s)1483=== CONT TestReadProxyRangeRequest14842026/11/12 00:20:33 INFO Vacuumed table table=objects1485=== NAME TestClientCADerivations1486 client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket34?endpoint=http://localhost:64456®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/3/scratch/62601a543565e631/build/TestClientCADerivations1914400394/001/store'1487 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11488--- PASS: TestClientCADerivations (8.37s)1489=== CONT TestReadRedirectKeepsNarinfoProxied14902026/11/12 00:20:34 INFO Received uploads request method=POST path=/api/pending_closures1491--- PASS: TestDeduplicatedObjectsRecordedAsPending (6.29s)1492=== CONT TestReadRedirectNar14932026/11/12 00:20:34 INFO Aborted multipart uploads count=0 kept=014942026/11/12 00:20:34 WARN Force mode enabled - objects will be deleted immediately without grace period14952026/11/12 00:20:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014962026/11/12 00:20:34 INFO Vacuumed table table=pending_closures14972026/11/12 00:20:34 INFO Vacuumed table table=pending_objects14982026/11/12 00:20:34 INFO Vacuumed table table=multipart_uploads14992026/11/12 00:20:34 INFO Vacuumed table table=closures15002026/11/12 00:20:34 INFO Vacuumed table table=objects1501--- PASS: TestGCMetrics (5.70s)1502=== CONT TestReadProxyDisabled15032026/11/12 00:20:34 INFO Received uploads request method=POST path=/api/pending_closures1504--- PASS: TestService_ReadScope_PublicByDefault (5.28s)1505=== CONT TestReadProxyRootRedirectsToIndexHTML15062026/11/12 00:20:34 INFO Aborted multipart uploads count=0 kept=015072026/11/12 00:20:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015082026/11/12 00:20:34 INFO Vacuumed table table=pending_closures15092026/11/12 00:20:34 INFO Vacuumed table table=pending_objects15102026/11/12 00:20:34 INFO Vacuumed table table=multipart_uploads15112026/11/12 00:20:34 INFO Vacuumed table table=closures15122026/11/12 00:20:34 INFO Vacuumed table table=objects15132026/11/12 00:20:34 INFO Aborted multipart uploads count=0 kept=015142026/11/12 00:20:34 WARN Force mode enabled - objects will be deleted immediately without grace period15152026/11/12 00:20:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015162026/11/12 00:20:34 INFO Vacuumed table table=pending_closures15172026/11/12 00:20:34 INFO Vacuumed table table=pending_objects15182026/11/12 00:20:34 INFO Vacuumed table table=multipart_uploads15192026/11/12 00:20:34 INFO Vacuumed table table=closures15202026/11/12 00:20:34 INFO Vacuumed table table=objects1521--- PASS: TestPushDedupSurvivesConcurrentGC (6.46s)1522=== CONT TestReadProxyConditionalGet1523=== RUN TestService_RequireScope_OIDC/builder_may_write1524=== PAUSE TestService_RequireScope_OIDC/builder_may_write1525=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1526=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1527=== RUN TestService_RequireScope_OIDC/ops_may_admin1528=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1529=== RUN TestService_RequireScope_OIDC/ops_may_not_write1530=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1531=== RUN TestService_RequireScope_OIDC/reader_may_not_write1532=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1533=== RUN TestService_RequireScope_OIDC/static_token_may_admin1534=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1535=== RUN TestService_RequireScope_OIDC/static_token_may_write1536=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1537=== RUN TestService_RequireScope_OIDC/reader_may_read1538=== PAUSE TestService_RequireScope_OIDC/reader_may_read1539=== RUN TestService_RequireScope_OIDC/writer_implies_read1540=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1541=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1542=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1543=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1544=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1545=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1546=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1547=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1548=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1549=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1550=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1551=== CONT TestReadProxyOutlastsServerWriteTimeout1552--- PASS: TestGCBugBareHashReferences (5.72s)1553=== CONT TestService_ReadAuthMiddleware15542026/11/12 00:20:35 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld3 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1555--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (5.66s)1556=== CONT TestReadProxyHead15572026/11/12 00:20:35 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=01558=== NAME TestClientIntegration1559 client_integration_test.go:371: Objects in database after GC:1560 client_integration_test.go:371: Successfully deleted all objects with GC --force1561--- PASS: TestClientIntegration (10.52s)1562=== CONT TestReadProxyInvalidPath15632026/11/12 00:20:35 INFO lead: acquired remote=192.0.2.1:1234 generation=11564=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1565=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1566=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1567=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1568=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1569=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1570=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1571=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1572=== CONT TestService_Rustfstest15732026/11/12 00:20:35 INFO lead: acquired remote=192.0.2.1:1234 generation=115742026/11/12 00:20:35 INFO Received uploads request method=POST path=/api/pending_closures1575--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (5.40s)1576=== CONT TestReadProxy40415772026/11/12 00:20:35 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15782026/11/12 00:20:35 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1579--- PASS: TestCompleteMultipartUnregistered (5.42s)1580=== CONT TestReadProxyNarinfo15812026/11/12 00:20:36 INFO Received uploads request method=POST path=/api/pending_closures15822026/11/12 00:20:36 INFO Received uploads request method=POST path=/api/pending_closures15832026/11/12 00:20:36 INFO Received uploads request method=POST path=/api/pending_closures15842026/11/12 00:20:36 INFO Received uploads request method=POST path=/api/pending_closures15852026/11/12 00:20:36 INFO lead: released remote=192.0.2.1:1234 generation=115862026/11/12 00:20:36 INFO lead: acquired remote=192.0.2.1:1234 generation=215872026/11/12 00:20:36 INFO Received cleanup request method=DELETE path=/api/pending_closures15882026/11/12 00:20:36 INFO Aborted multipart uploads count=0 kept=015892026/11/12 00:20:36 INFO Received uploads request method=POST path=/api/pending_closures15902026/11/12 00:20:36 INFO lead: released remote=192.0.2.1:1234 generation=11591--- PASS: TestLeadEndsOnShutdown (6.51s)1592=== CONT TestIsValidCachePath1593=== RUN TestIsValidCachePath/narinfo1594=== PAUSE TestIsValidCachePath/narinfo1595=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1596=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1597=== RUN TestIsValidCachePath/nar_zst1598=== PAUSE TestIsValidCachePath/nar_zst1599=== RUN TestIsValidCachePath/nar_xz1600=== PAUSE TestIsValidCachePath/nar_xz1601=== RUN TestIsValidCachePath/nar_bz21602=== PAUSE TestIsValidCachePath/nar_bz21603=== RUN TestIsValidCachePath/nar_uncompressed1604=== PAUSE TestIsValidCachePath/nar_uncompressed1605=== RUN TestIsValidCachePath/ls1606=== PAUSE TestIsValidCachePath/ls1607=== RUN TestIsValidCachePath/log1608=== PAUSE TestIsValidCachePath/log1609=== RUN TestIsValidCachePath/realisation1610=== PAUSE TestIsValidCachePath/realisation1611=== RUN TestIsValidCachePath/nix-cache-info1612=== PAUSE TestIsValidCachePath/nix-cache-info1613=== RUN TestIsValidCachePath/index.html1614=== PAUSE TestIsValidCachePath/index.html1615=== RUN TestIsValidCachePath/traversal_parent1616=== PAUSE TestIsValidCachePath/traversal_parent1617=== RUN TestIsValidCachePath/traversal_in_middle1618=== PAUSE TestIsValidCachePath/traversal_in_middle1619=== RUN TestIsValidCachePath/invalid_char_e1620=== PAUSE TestIsValidCachePath/invalid_char_e1621=== RUN TestIsValidCachePath/invalid_char_u1622=== PAUSE TestIsValidCachePath/invalid_char_u1623=== RUN TestIsValidCachePath/random_path1624=== PAUSE TestIsValidCachePath/random_path1625=== RUN TestIsValidCachePath/empty1626=== PAUSE TestIsValidCachePath/empty1627=== RUN TestIsValidCachePath/leading_slash1628=== PAUSE TestIsValidCachePath/leading_slash1629=== RUN TestIsValidCachePath/wrong_extension1630=== PAUSE TestIsValidCachePath/wrong_extension1631=== RUN TestIsValidCachePath/short_hash1632=== PAUSE TestIsValidCachePath/short_hash1633=== CONT TestReadProxyNarStreaming16342026/11/12 00:20:36 INFO Received cleanup request method=DELETE path=/api/pending_closures16352026/11/12 00:20:36 INFO Aborted multipart uploads count=1 kept=016362026/11/12 00:20:36 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16372026-11-12 00:20:36.762 UTC [57445] ERROR: Closure does not exist: id=116382026-11-12 00:20:36.762 UTC [57445] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE16392026-11-12 00:20:36.762 UTC [57445] STATEMENT: -- name: CommitPendingClosure :one1640 SELECT commit_pending_closure($1::bigint)::bigint1641 1642--- PASS: TestService_cleanupPendingClosuresHandler (4.80s)1643=== CONT TestParseSingleRange1644=== RUN TestParseSingleRange/none1645=== PAUSE TestParseSingleRange/none1646=== RUN TestParseSingleRange/unknown_unit1647=== PAUSE TestParseSingleRange/unknown_unit1648=== RUN TestParseSingleRange/multi-range_ignored1649=== PAUSE TestParseSingleRange/multi-range_ignored1650=== RUN TestParseSingleRange/malformed_no_dash1651=== PAUSE TestParseSingleRange/malformed_no_dash1652=== RUN TestParseSingleRange/malformed_both_empty1653=== PAUSE TestParseSingleRange/malformed_both_empty1654=== RUN TestParseSingleRange/malformed_end_before_start1655=== PAUSE TestParseSingleRange/malformed_end_before_start1656=== RUN TestParseSingleRange/closed1657=== PAUSE TestParseSingleRange/closed1658=== RUN TestParseSingleRange/open-ended1659=== PAUSE TestParseSingleRange/open-ended1660=== RUN TestParseSingleRange/end_clamped_to_size1661=== PAUSE TestParseSingleRange/end_clamped_to_size1662=== RUN TestParseSingleRange/suffix1663=== PAUSE TestParseSingleRange/suffix1664=== RUN TestParseSingleRange/suffix_exceeds_size1665=== PAUSE TestParseSingleRange/suffix_exceeds_size1666=== RUN TestParseSingleRange/single_byte1667=== PAUSE TestParseSingleRange/single_byte1668=== RUN TestParseSingleRange/start_past_EOF1669=== PAUSE TestParseSingleRange/start_past_EOF1670=== RUN TestParseSingleRange/start_far_past_EOF1671=== PAUSE TestParseSingleRange/start_far_past_EOF1672=== CONT TestReadProxyContentNegotiation1673--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (4.55s)1674=== CONT TestPresentNotReportedWhileGCDeletesClosure1675=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1676=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1677=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1678=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1679=== CONT TestReadProxyNarinfoAlreadyDecompressed16802026/11/12 00:20:37 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"16812026/11/12 00:20:37 WARN mTLS auth: bound subjects configured but subject DN unavailable16822026/11/12 00:20:37 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1683--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (4.72s)1684=== CONT TestPresignedUploadRegisteredBeforeCommit16852026/11/12 00:20:37 INFO Received complete multipart upload request method=POST path=/api/multipart/complete16862026/11/12 00:20:37 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1687--- PASS: TestService_AuthMiddleware (4.62s)1688=== CONT TestPresentRefreshesOnlyStaleClosures16892026/11/12 00:20:37 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ODAyYzY4Y2EtMTk0Mi00YjZhLTg1ZGQtODUzY2NlNTdiOGNhLmE5ZTFjODhiLTU3YjEtNDE4Mi04NTYwLTNkZjI2ZTc3ZjRhN3gxNzk0NDQyODM2MTYzNDM1MDAw parts=1016902026/11/12 00:20:37 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16912026/11/12 00:20:37 INFO Completed upload id=1 objects=2 duration=7.792375ms16922026/11/12 00:20:37 INFO Received uploads request method=POST path=/api/pending_closures16932026/11/12 00:20:37 INFO Received uploads request method=POST path=/api/pending_closures16942026/11/12 00:20:37 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo16952026/11/12 00:20:37 WARN Found objects in DB but missing from S3, will re-upload count=11696--- PASS: TestService_verifyS3Integrity (6.41s)1697=== CONT TestObjectStatsTrigger16982026/11/12 00:20:37 INFO Received uploads request method=POST path=/api/pending_closures16992026/11/12 00:20:37 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17002026/11/12 00:20:37 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ODAyYzY4Y2EtMTk0Mi00YjZhLTg1ZGQtODUzY2NlNTdiOGNhLmM4NjExOTE3LThhNmEtNGRhOS05Y2ExLWI2MDZjNmUwOGEyM3gxNzk0NDQyODM2NDI1MDI3MDAw parts=1017012026/11/12 00:20:37 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17022026/11/12 00:20:37 INFO Completed upload id=1 objects=2 duration=11.532209ms17032026/11/12 00:20:37 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000017042026/11/12 00:20:37 INFO Received uploads request method=POST path=/api/pending_closures17052026/11/12 00:20:37 INFO lead: released remote=192.0.2.1:1234 generation=217062026/11/12 00:20:37 INFO Starting cleanup of old closures method=DELETE path=/api/closures17072026/11/12 00:20:37 INFO lead: acquired remote=192.0.2.1:1234 generation=317082026/11/12 00:20:37 INFO lead: released remote=192.0.2.1:1234 generation=31709--- PASS: TestLeadGenerationGrowsWithEveryGrant (7.35s)1710=== CONT TestCreatePin_ReservedPins17112026/11/12 00:20:37 INFO Aborted multipart uploads count=0 kept=017122026/11/12 00:20:37 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017132026/11/12 00:20:37 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:64646/oidc17142026/11/12 00:20:37 INFO Vacuumed table table=pending_closures17152026/11/12 00:20:37 INFO Vacuumed table table=pending_objects17162026/11/12 00:20:37 INFO Vacuumed table table=multipart_uploads1717--- PASS: TestPresentReportsOnlyClosureRoots (3.87s)1718=== CONT TestService_NativeMTLS17192026/11/12 00:20:37 INFO Vacuumed table table=closures17202026/11/12 00:20:37 INFO Vacuumed table table=objects17212026/11/12 00:20:37 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001722--- PASS: TestService_createPendingClosureHandler (6.40s)1723=== CONT TestPendingCleanupUsesOneCutoff17242026/11/12 00:20:37 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1725--- PASS: TestReadRedirectUsesPublicS3URL (4.20s)1726=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1727--- PASS: TestReadRedirectKeepsNarinfoProxied (4.40s)1728=== CONT TestOrphanedObjectsGCStressTest1729--- PASS: TestReadProxyRangeRequest (4.82s)1730=== CONT TestOrphanedObjectsGC1731--- PASS: TestReadRedirectNar (4.82s)1732=== CONT TestMultipartCleanup1733--- PASS: TestReadProxyDisabled (4.72s)1734=== CONT TestObjectStatsUpdatedOncePerCommit1735--- PASS: TestReadProxyRootRedirectsToIndexHTML (4.54s)1736=== CONT TestCacheConfigHandlerMaxNarSize1737--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1738=== CONT TestServerTLSConfig1739=== RUN TestServerTLSConfig/no_client_CA1740=== PAUSE TestServerTLSConfig/no_client_CA1741=== RUN TestServerTLSConfig/missing_CA_file1742=== PAUSE TestServerTLSConfig/missing_CA_file1743=== RUN TestServerTLSConfig/not_a_PEM_file1744=== PAUSE TestServerTLSConfig/not_a_PEM_file1745=== CONT TestNARDeduplicationMetadataUploadBug1746--- PASS: TestReadProxyConditionalGet (4.44s)1747=== CONT TestMetricsInventory1748--- PASS: TestService_ReadAuthMiddleware (4.45s)1749=== CONT TestMetricsCommit1750--- PASS: TestReadProxyHead (4.48s)1751=== CONT TestCreatePendingClosureRejectsOversizedNAR17522026/11/12 00:20:39 INFO Received uploads request method=POST path=/api/pending_closures1753--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1754=== CONT TestGenerateLandingPage1755--- PASS: TestGenerateLandingPage (0.00s)1756=== CONT TestEnsureAbortUploadsRule1757--- PASS: TestReadProxyInvalidPath (4.45s)1758=== CONT TestService_readinessHandler1759--- PASS: TestService_Rustfstest (4.48s)1760=== CONT TestService_healthCheckHandler17612026/11/12 00:20:40 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1762--- PASS: TestReadProxyNarinfo (4.33s)1763=== CONT TestGCEndsOnShutdown/before_the_run1764--- PASS: TestReadProxy404 (4.56s)1765=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1766--- PASS: TestReadProxyNarStreaming (3.73s)1767=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1768--- PASS: TestReadProxyContentNegotiation (3.90s)1769=== CONT TestResolveDBConnectionString/flag_wins1770=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1771=== CONT TestResolveDBConnectionString/nothing_configured1772=== CONT TestResolveDBConnectionString/missing_file_is_an_error1773=== CONT TestResolveDBConnectionString/file_when_flag_empty1774=== CONT TestClientErrorHandling/InvalidStorePath1775--- PASS: TestResolveDBConnectionString (0.01s)1776 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1777 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1778 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1779 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1780 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1781--- PASS: TestReadProxyNarinfoAlreadyDecompressed (3.86s)1782=== CONT TestClientErrorHandling/ServerNotAvailable17832026/11/12 00:20:40 INFO Received uploads request method=POST path=/api/pending_closures17842026/11/12 00:20:41 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present17852026/11/12 00:20:41 WARN Request failed, retrying attempt=2 backoff=208.915712ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present17862026/11/12 00:20:41 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst17872026/11/12 00:20:41 INFO Received uploads request method=POST path=/api/pending_closures1788--- PASS: TestPresignedUploadRegisteredBeforeCommit (4.06s)1789=== CONT TestClientErrorHandling/InvalidAuthToken1790--- PASS: TestPresentRefreshesOnlyStaleClosures (3.97s)1791=== CONT TestPush_RejectsBadRequests/bad_root17922026/11/12 00:20:41 INFO Received push request method=POST path=/api/pushes1793=== CONT TestPush_RejectsBadRequests/no_roots17942026/11/12 00:20:41 INFO Received push request method=POST path=/api/pushes1795=== CONT TestPush_RejectsBadRequests/no_objects17962026/11/12 00:20:41 INFO Received push request method=POST path=/api/pushes1797=== CONT TestPush_RejectsBadRequests/root_not_in_objects17982026/11/12 00:20:41 INFO Received push request method=POST path=/api/pushes1799=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1800--- PASS: TestPush_RejectsBadRequests (3.56s)1801 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1802 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1803 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1804 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)18052026/11/12 00:20:41 WARN Request failed, retrying attempt=3 backoff=396.806172ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1806--- PASS: TestObjectStatsTrigger (4.06s)1807=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check18082026/11/12 00:20:41 WARN mTLS auth: subject not in bound subjects subject="CN=reader"18092026/11/12 00:20:41 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1810--- PASS: TestService_NativeMTLS (3.82s)1811=== CONT TestCacheConfigHandler/full_config,_no_issuer1812=== CONT TestCacheConfigHandler/no_signing_keys1813=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1814=== CONT TestCacheConfigHandler/no_cache_url_configured1815--- PASS: TestCacheConfigHandler (0.01s)1816 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1817 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1818 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1819 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1820=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info18212026/11/12 00:20:41 INFO Received uploads request method=POST path=/1822=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key18232026/11/12 00:20:41 INFO Received complete multipart upload request method=POST path=/1824=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key18252026/11/12 00:20:41 INFO Received request for more parts method=POST path=/1826=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal18272026/11/12 00:20:41 INFO Received uploads request method=POST path=/1828--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1829 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1830 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1831 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1832 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1833=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed18342026/11/12 00:20:41 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux18352026/11/12 00:20:41 WARN Refused reserved pin name=worker-x86_64-linux18362026/11/12 00:20:41 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux18372026/11/12 00:20:41 INFO Received create pin request method=POST path=/api/pins/my-app18382026/11/12 00:20:41 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1839--- PASS: TestCreatePin_ReservedPins (4.00s)1840=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed18412026/11/12 00:20:41 INFO Received uploads request method=POST path=/api/pending_closures18422026/11/12 00:20:41 INFO Received uploads request method=POST path=/api/pending_closures18432026/11/12 00:20:41 WARN Request failed, retrying attempt=4 backoff=783.133431ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present18442026/11/12 00:20:41 INFO Received cleanup request method=DELETE path=/api/pending_closures18452026/11/12 00:20:41 INFO Received uploads request method=POST path=/api/pending_closures1846--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (3.90s)1847=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost18482026/11/12 00:20:42 INFO Received uploads request method=POST path=/api/pending_closures18492026/11/12 00:20:42 INFO Received cleanup request method=DELETE path=/api/pending_closures18502026/11/12 00:20:42 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket86/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""18512026/11/12 00:20:42 INFO Aborted multipart uploads count=0 kept=118522026/11/12 00:20:42 INFO Received cleanup request method=DELETE path=/api/pending_closures18532026/11/12 00:20:42 INFO Aborted multipart uploads count=1 kept=01854--- PASS: TestMultipartCleanup (3.66s)1855=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed18562026/11/12 00:20:42 WARN Request failed, retrying attempt=5 backoff=1.507034722s error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present1857--- PASS: TestObjectStatsUpdatedOncePerCommit (3.59s)1858=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed18592026/11/12 00:20:42 WARN Rate limiter enabled after throttle name=s3-test rate=518602026/11/12 00:20:42 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1861=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1862 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101863 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001864--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (9.61s)1865=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure18662026/11/12 00:20:42 INFO Received uploads request method=POST path=/1867=== NAME TestNARDeduplicationMetadataUploadBug1868 metadata_upload_test.go:48: First store path: /private/var/tb/3/scratch/62601a543565e631/build/TestNARDeduplicationMetadataUploadBug973166221/001/store/gbg6kbfqjbqrzvacs5ws9fys6mc0vhic-file1.txt1869--- PASS: TestMetricsInventory (3.67s)1870=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart18712026/11/12 00:20:42 INFO Received complete multipart upload request method=POST path=/1872=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts18732026/11/12 00:20:42 INFO Received request for more parts method=POST path=/1874=== CONT TestIsValidUploadKey/narinfo1875=== CONT TestPendingClosureWriteTimeout/empty1876=== CONT TestIsValidUploadKey/unknown_type1877=== CONT TestIsValidUploadKey/empty_key1878=== CONT TestIsValidUploadKey/absolute1879=== CONT TestIsValidUploadKey/traversal_nar1880=== CONT TestIsValidUploadKey/traversal1881=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1882=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1883=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1884=== CONT TestIsValidUploadKey/index.html1885=== CONT TestIsValidUploadKey/nix-cache-info1886=== CONT TestIsValidUploadKey/realisation_plus_in_output1887=== CONT TestIsValidUploadKey/realisation1888=== CONT TestIsValidUploadKey/build_log_equals1889=== CONT TestIsValidUploadKey/build_log_question_mark1890=== CONT TestIsValidUploadKey/build_log_plus_in_name1891=== CONT TestIsValidUploadKey/build_log_home-manager_file1892=== CONT TestIsValidUploadKey/build_log1893=== CONT TestIsValidUploadKey/listing1894=== CONT TestIsValidUploadKey/nar_plain1895=== CONT TestIsValidUploadKey/nar_xz1896=== CONT TestIsValidUploadKey/nar_zst1897--- PASS: TestIsValidUploadKey (0.00s)1898 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1899 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1900 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1901 --- PASS: TestIsValidUploadKey/absolute (0.00s)1902 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1903 --- PASS: TestIsValidUploadKey/traversal (0.00s)1904 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1905 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1906 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1907 --- PASS: TestIsValidUploadKey/index.html (0.00s)1908 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1909 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1910 --- PASS: TestIsValidUploadKey/realisation (0.00s)1911 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1912 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1913 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1914 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1915 --- PASS: TestIsValidUploadKey/build_log (0.00s)1916 --- PASS: TestIsValidUploadKey/listing (0.00s)1917 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1918 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1919 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1920=== CONT TestProxyWriteTimeout/narinfo1921=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline19222026/11/12 00:20:43 INFO Received push request method=POST path=/api/pushes19232026/11/12 00:20:43 INFO Received push request method=POST path=/api/pushes19242026/11/12 00:20:43 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19252026/11/12 00:20:43 INFO Uploading gbg6kbfqjbqrzvacs5ws9fys6mc0vhic-file1.txt (160B)19262026/11/12 00:20:43 INFO Received complete push request method=POST path=/api/pushes/1/complete19272026/11/12 00:20:43 INFO Completed push id=1 objects=4 duration=22.040625ms19282026/11/12 00:20:43 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete19292026/11/12 00:20:43 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"19302026/11/12 00:20:43 INFO Completed upload id=2 objects=2 duration=4.976ms1931--- PASS: TestMetricsCommit (3.77s)1932=== CONT TestPendingClosureWriteTimeout/670k_objects1933=== CONT TestPendingClosureWriteTimeout/400_objects1934=== CONT TestProxyWriteTimeout/unknown_size1935=== CONT TestProxyWriteTimeout/10_GiB_nar1936=== CONT TestProxyWriteTimeout/1_GiB_nar1937--- PASS: TestProxyWriteTimeout (0.00s)1938 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1939 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1940 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1941 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1942=== CONT TestService_RequireScope_OIDC/builder_may_write19432026/11/12 00:20:43 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19442026/11/12 00:20:43 WARN Failed to register uploaded object key=gbg6kbfqjbqrzvacs5ws9fys6mc0vhic.ls error="server returned 404: 404 page not found\n"1945=== CONT TestService_RequireScope_OIDC/reader_may_read1946=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1947=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read19482026/11/12 00:20:43 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1949=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1950=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1951=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1952=== CONT TestService_RequireScope_OIDC/writer_implies_read1953=== CONT TestService_RequireScope_OIDC/reader_may_not_write1954=== CONT TestService_RequireScope_OIDC/static_token_may_write1955=== CONT TestService_RequireScope_OIDC/static_token_may_admin1956=== CONT TestService_RequireScope_OIDC/ops_may_admin1957=== CONT TestService_RequireScope_OIDC/ops_may_not_write1958=== CONT TestService_RequireScope_OIDC/builder_may_not_admin19592026/11/12 00:20:43 INFO Signed narinfos id=1 count=11960--- PASS: TestService_RequireScope_OIDC (5.64s)1961 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1962 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1963 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1964 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1965 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1966 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1967 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1968 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1969 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1970 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1971 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1972 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1973 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1974 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1975=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token19762026/11/12 00:20:43 INFO Uploading 1 narinfos1977=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected19782026/11/12 00:20:43 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1979=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1980=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected19812026/11/12 00:20:43 WARN Authentication failed token_preview=eyJhbGciOi...Ly6J1UTNAQ token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1982=== CONT TestIsValidCachePath/narinfo1983=== CONT TestIsValidCachePath/index.html1984=== CONT TestIsValidCachePath/short_hash1985=== CONT TestIsValidCachePath/wrong_extension1986=== CONT TestIsValidCachePath/leading_slash1987=== CONT TestIsValidCachePath/empty1988=== CONT TestIsValidCachePath/random_path1989=== CONT TestIsValidCachePath/invalid_char_u1990=== CONT TestIsValidCachePath/invalid_char_e1991=== CONT TestIsValidCachePath/traversal_in_middle1992=== CONT TestIsValidCachePath/traversal_parent1993=== CONT TestIsValidCachePath/nar_uncompressed1994=== CONT TestIsValidCachePath/nix-cache-info1995=== CONT TestIsValidCachePath/realisation1996=== CONT TestIsValidCachePath/log1997=== CONT TestIsValidCachePath/ls1998=== CONT TestIsValidCachePath/nar_xz1999=== CONT TestIsValidCachePath/nar_bz22000=== CONT TestIsValidCachePath/nar_zst
loading 619 hidden lines…
26202026/11/12 00:21:03 WARN Force mode enabled - objects will be deleted immediately without grace period26212026/11/12 00:21:03 INFO Vacuumed table table=pending_closures26222026/11/12 00:21:03 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026232026/11/12 00:21:04 INFO Vacuumed table table=pending_closures26242026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=026252026/11/12 00:21:04 INFO Vacuumed table table=pending_objects26262026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads26272026/11/12 00:21:04 INFO Vacuumed table table=pending_objects26282026/11/12 00:21:04 INFO Vacuumed table table=closures26292026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads26302026/11/12 00:21:04 WARN Force mode enabled - objects will be deleted immediately without grace period26312026/11/12 00:21:04 INFO Vacuumed table table=objects26322026/11/12 00:21:04 INFO Vacuumed table table=closures26332026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026342026/11/12 00:21:04 INFO Received push request method=POST path=/26352026/11/12 00:21:04 INFO Vacuumed table table=objects26362026/11/12 00:21:04 INFO Received complete push request method=POST path=/26372026/11/12 00:21:04 INFO Vacuumed table table=pending_closures26382026/11/12 00:21:04 INFO Completed push id=6 objects=6 duration=5.100667ms26392026/11/12 00:21:04 INFO Vacuumed table table=pending_objects26402026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads26412026/11/12 00:21:04 INFO Vacuumed table table=closures26422026/11/12 00:21:04 INFO Vacuumed table table=objects26432026/11/12 00:21:04 INFO Received uploads request method=POST path=/26442026/11/12 00:21:04 INFO Received push request method=POST path=/26452026/11/12 00:21:04 INFO Received uploads request method=POST path=/26462026/11/12 00:21:04 INFO Received push request method=POST path=/26472026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=026482026/11/12 00:21:04 WARN Force mode enabled - objects will be deleted immediately without grace period26492026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026502026/11/12 00:21:04 INFO Vacuumed table table=pending_closures26512026/11/12 00:21:04 INFO Vacuumed table table=pending_objects26522026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads26532026/11/12 00:21:04 INFO Vacuumed table table=closures26542026/11/12 00:21:04 INFO Vacuumed table table=objects26552026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=026562026/11/12 00:21:04 WARN Force mode enabled - objects will be deleted immediately without grace period26572026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026582026/11/12 00:21:04 INFO Vacuumed table table=pending_closures26592026/11/12 00:21:04 INFO Vacuumed table table=pending_objects26602026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads26612026/11/12 00:21:04 INFO Vacuumed table table=closures26622026/11/12 00:21:04 INFO Vacuumed table table=objects26632026/11/12 00:21:04 INFO Received complete upload request method=POST path=/26642026/11/12 00:21:04 INFO Completed upload id=9 objects=7 duration=3.17975ms26652026/11/12 00:21:04 INFO Received complete upload request method=POST path=/26662026/11/12 00:21:04 INFO Received uploads request method=POST path=/26672026/11/12 00:21:04 INFO Received complete upload request method=POST path=/26682026/11/12 00:21:04 INFO Completed upload id=3 objects=24 duration=536.042µs26692026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=026702026/11/12 00:21:04 WARN Force mode enabled - objects will be deleted immediately without grace period26712026/11/12 00:21:04 INFO Completed upload id=1 objects=15 duration=13.606375ms26722026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026732026/11/12 00:21:04 INFO Vacuumed table table=pending_closures26742026/11/12 00:21:04 INFO Vacuumed table table=pending_objects26752026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=026762026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads26772026/11/12 00:21:04 INFO Received push request method=POST path=/26782026/11/12 00:21:04 INFO Received uploads request method=POST path=/26792026/11/12 00:21:04 INFO Received push request method=POST path=/26802026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026812026/11/12 00:21:04 INFO Vacuumed table table=closures26822026/11/12 00:21:04 INFO Received complete push request method=POST path=/26832026/11/12 00:21:04 INFO Received uploads request method=POST path=/26842026/11/12 00:21:04 INFO Vacuumed table table=pending_closures26852026/11/12 00:21:04 INFO Vacuumed table table=objects26862026/11/12 00:21:04 INFO Received complete upload request method=POST path=/26872026/11/12 00:21:04 INFO Completed push id=14 objects=3 duration=4.315583ms26882026/11/12 00:21:04 INFO Vacuumed table table=pending_objects26892026/11/12 00:21:04 INFO Completed upload id=2 objects=10 duration=1.72725ms26902026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads26912026/11/12 00:21:04 INFO Vacuumed table table=closures26922026/11/12 00:21:04 INFO Vacuumed table table=objects26932026/11/12 00:21:04 INFO Received push request method=POST path=/26942026/11/12 00:21:04 INFO Received uploads request method=POST path=/26952026/11/12 00:21:04 INFO Received complete push request method=POST path=/26962026/11/12 00:21:04 INFO Received push request method=POST path=/26972026/11/12 00:21:04 INFO Completed push id=1 objects=5 duration=6.970083ms26982026/11/12 00:21:04 INFO Received complete push request method=POST path=/26992026/11/12 00:21:04 INFO Received push request method=POST path=/27002026/11/12 00:21:04 INFO Completed push id=10 objects=15 duration=1.647875ms27012026/11/12 00:21:04 INFO Received push request method=POST path=/27022026/11/12 00:21:04 INFO Received complete push request method=POST path=/27032026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027042026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027052026/11/12 00:21:04 INFO Completed push id=5 objects=6 duration=1.3755ms27062026/11/12 00:21:04 INFO Received complete push request method=POST path=/27072026/11/12 00:21:04 INFO Received push request method=POST path=/27082026/11/12 00:21:04 INFO Completed push id=8 objects=6 duration=3.489916ms27092026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=027102026/11/12 00:21:04 INFO Vacuumed table table=pending_closures27112026/11/12 00:21:04 INFO Vacuumed table table=pending_objects27122026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027132026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads27142026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027152026/11/12 00:21:04 WARN Force mode enabled - objects will be deleted immediately without grace period27162026/11/12 00:21:04 INFO Vacuumed table table=closures27172026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027182026/11/12 00:21:04 INFO Vacuumed table table=pending_closures27192026/11/12 00:21:04 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27202026/11/12 00:21:04 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27212026/11/12 00:21:04 INFO Vacuumed table table=objects27222026/11/12 00:21:04 INFO Vacuumed table table=pending_objects27232026/11/12 00:21:04 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27242026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads27252026/11/12 00:21:04 INFO Vacuumed table table=closures27262026/11/12 00:21:04 INFO Vacuumed table table=objects27272026/11/12 00:21:04 INFO Received uploads request method=POST path=/27282026/11/12 00:21:04 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27292026/11/12 00:21:04 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27302026/11/12 00:21:04 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27312026/11/12 00:21:04 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27322026/11/12 00:21:04 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27332026/11/12 00:21:04 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27342026/11/12 00:21:04 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27352026/11/12 00:21:04 INFO Received uploads request method=POST path=/27362026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027372026/11/12 00:21:04 WARN Force mode enabled - objects will be deleted immediately without grace period27382026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027392026/11/12 00:21:04 INFO Vacuumed table table=pending_closures27402026/11/12 00:21:04 INFO Received push request method=POST path=/27412026/11/12 00:21:04 INFO Vacuumed table table=pending_objects27422026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads27432026/11/12 00:21:04 INFO Vacuumed table table=closures27442026/11/12 00:21:04 INFO Vacuumed table table=objects27452026/11/12 00:21:04 INFO Received push request method=POST path=/27462026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027472026/11/12 00:21:04 INFO Received push request method=POST path=/27482026/11/12 00:21:04 INFO Received uploads request method=POST path=/27492026/11/12 00:21:04 INFO Received push request method=POST path=/27502026/11/12 00:21:04 INFO Received uploads request method=POST path=/27512026/11/12 00:21:04 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27522026/11/12 00:21:04 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27532026/11/12 00:21:04 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27542026/11/12 00:21:04 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27552026/11/12 00:21:04 INFO Received uploads request method=POST path=/27562026/11/12 00:21:04 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27572026/11/12 00:21:04 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27582026/11/12 00:21:04 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27592026/11/12 00:21:04 INFO Received push request method=POST path=/27602026/11/12 00:21:04 INFO Received complete push request method=POST path=/27612026/11/12 00:21:04 INFO Completed push id=10 objects=2 duration=1.9335ms27622026/11/12 00:21:04 INFO Received push request method=POST path=/27632026/11/12 00:21:04 INFO Received push request method=POST path=/27642026/11/12 00:21:04 INFO Received push request method=POST path=/27652026/11/12 00:21:04 INFO Received uploads request method=POST path=/27662026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027672026/11/12 00:21:04 INFO Received complete upload request method=POST path=/27682026/11/12 00:21:04 WARN Force mode enabled - objects will be deleted immediately without grace period27692026/11/12 00:21:04 INFO Completed upload id=2 objects=7 duration=10.751ms27702026/11/12 00:21:04 INFO Received complete push request method=POST path=/27712026/11/12 00:21:04 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027722026/11/12 00:21:04 INFO Completed push id=3 objects=5 duration=1.708ms27732026/11/12 00:21:04 INFO Received push request method=POST path=/27742026/11/12 00:21:04 INFO Vacuumed table table=pending_closures27752026/11/12 00:21:04 INFO Received complete upload request method=POST path=/27762026/11/12 00:21:04 INFO Vacuumed table table=pending_objects27772026/11/12 00:21:04 INFO Completed upload id=12 objects=21 duration=1.113166ms27782026/11/12 00:21:04 INFO Vacuumed table table=multipart_uploads27792026/11/12 00:21:04 INFO Vacuumed table table=closures27802026/11/12 00:21:04 INFO Received complete upload request method=POST path=/27812026/11/12 00:21:04 INFO Vacuumed table table=objects27822026/11/12 00:21:04 INFO Completed upload id=5 objects=7 duration=2.448875ms27832026/11/12 00:21:04 INFO Received push request method=POST path=/27842026/11/12 00:21:04 INFO Received complete upload request method=POST path=/27852026/11/12 00:21:04 INFO Completed upload id=5 objects=5 duration=3.132292ms27862026/11/12 00:21:04 INFO Received push request method=POST path=/27872026/11/12 00:21:04 INFO Received push request method=POST path=/27882026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=027892026/11/12 00:21:04 INFO Received uploads request method=POST path=/27902026/11/12 00:21:04 INFO Received complete push request method=POST path=/27912026/11/12 00:21:04 INFO Completed push id=1 objects=2 duration=2.582459ms27922026/11/12 00:21:04 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27932026/11/12 00:21:04 INFO Received complete upload request method=POST path=/27942026/11/12 00:21:04 INFO Completed upload id=6 objects=9 duration=749.125µs27952026/11/12 00:21:04 INFO Received push request method=POST path=/27962026/11/12 00:21:04 INFO Received push request method=POST path=/27972026/11/12 00:21:04 INFO Received uploads request method=POST path=/27982026/11/12 00:21:04 INFO Received complete push request method=POST path=/27992026/11/12 00:21:04 INFO Completed push id=13 objects=23 duration=1.252875ms28002026/11/12 00:21:04 INFO Received complete push request method=POST path=/28012026/11/12 00:21:04 INFO Completed push id=7 objects=20 duration=1.751625ms28022026/11/12 00:21:04 INFO Received uploads request method=POST path=/28032026/11/12 00:21:04 INFO Received push request method=POST path=/28042026/11/12 00:21:04 INFO Aborted multipart uploads count=0 kept=028052026/11/12 00:21:05 INFO Received uploads request method=POST path=/28062026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028072026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028082026/11/12 00:21:05 INFO Vacuumed table table=pending_closures28092026/11/12 00:21:05 INFO Received complete push request method=POST path=/28102026/11/12 00:21:05 INFO Vacuumed table table=pending_objects28112026/11/12 00:21:05 INFO Completed push id=3 objects=5 duration=3.712291ms28122026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads28132026/11/12 00:21:05 INFO Vacuumed table table=closures28142026/11/12 00:21:05 INFO Vacuumed table table=objects28152026/11/12 00:21:05 INFO Received push request method=POST path=/28162026/11/12 00:21:05 INFO Received uploads request method=POST path=/28172026/11/12 00:21:05 INFO Received push request method=POST path=/28182026/11/12 00:21:05 INFO Received push request method=POST path=/28192026/11/12 00:21:05 INFO Received complete push request method=POST path=/28202026/11/12 00:21:05 INFO Completed push id=9 objects=24 duration=2.876ms28212026/11/12 00:21:05 INFO Received complete push request method=POST path=/28222026/11/12 00:21:05 INFO Completed push id=4 objects=4 duration=4.557833ms28232026/11/12 00:21:05 INFO Received uploads request method=POST path=/28242026/11/12 00:21:05 INFO Received uploads request method=POST path=/28252026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028262026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028272026/11/12 00:21:05 WARN Force mode enabled - objects will be deleted immediately without grace period28282026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028292026/11/12 00:21:05 INFO Received uploads request method=POST path=/28302026/11/12 00:21:05 INFO Vacuumed table table=pending_closures28312026/11/12 00:21:05 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."28322026/11/12 00:21:05 INFO Received uploads request method=POST path=/28332026/11/12 00:21:05 INFO Vacuumed table table=pending_objects28342026/11/12 00:21:05 INFO Received complete upload request method=POST path=/28352026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028362026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads28372026/11/12 00:21:05 INFO Received uploads request method=POST path=/28382026/11/12 00:21:05 INFO Completed upload id=3 objects=5 duration=1.375709ms28392026/11/12 00:21:05 INFO Vacuumed table table=closures28402026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028412026/11/12 00:21:05 INFO Received complete push request method=POST path=/28422026/11/12 00:21:05 INFO Vacuumed table table=objects28432026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028442026/11/12 00:21:05 INFO Vacuumed table table=pending_closures28452026/11/12 00:21:05 INFO Received complete upload request method=POST path=/28462026/11/12 00:21:05 INFO Completed upload id=10 objects=23 duration=1.158542ms28472026/11/12 00:21:05 WARN Force mode enabled - objects will be deleted immediately without grace period28482026/11/12 00:21:05 INFO Completed push id=1 objects=6 duration=18.132459ms28492026/11/12 00:21:05 INFO Vacuumed table table=pending_objects28502026/11/12 00:21:05 INFO Received uploads request method=POST path=/28512026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads28522026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028532026/11/12 00:21:05 INFO Received uploads request method=POST path=/28542026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028552026/11/12 00:21:05 INFO Received uploads request method=POST path=/28562026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028572026/11/12 00:21:05 INFO Vacuumed table table=closures28582026/11/12 00:21:05 INFO Vacuumed table table=objects28592026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028602026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028612026/11/12 00:21:05 INFO Received push request method=POST path=/28622026/11/12 00:21:05 INFO Received uploads request method=POST path=/28632026/11/12 00:21:05 INFO Received push request method=POST path=/28642026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028652026/11/12 00:21:05 INFO Vacuumed table table=pending_closures28662026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028672026/11/12 00:21:05 INFO Vacuumed table table=pending_closures28682026/11/12 00:21:05 INFO Vacuumed table table=pending_closures28692026/11/12 00:21:05 INFO Vacuumed table table=pending_closures28702026/11/12 00:21:05 INFO Vacuumed table table=pending_objects28712026/11/12 00:21:05 INFO Received complete upload request method=POST path=/28722026/11/12 00:21:05 INFO Vacuumed table table=pending_objects28732026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads28742026/11/12 00:21:05 INFO Completed upload id=10 objects=10 duration=1.08375ms28752026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads28762026/11/12 00:21:05 INFO Vacuumed table table=pending_objects28772026/11/12 00:21:05 INFO Vacuumed table table=pending_objects28782026/11/12 00:21:05 INFO Vacuumed table table=closures28792026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads28802026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads28812026/11/12 00:21:05 INFO Received complete upload request method=POST path=/28822026/11/12 00:21:05 INFO Vacuumed table table=objects28832026/11/12 00:21:05 INFO Vacuumed table table=closures28842026/11/12 00:21:05 INFO Vacuumed table table=closures28852026/11/12 00:21:05 INFO Vacuumed table table=closures28862026/11/12 00:21:05 INFO Completed upload id=14 objects=8 duration=2.177708ms28872026/11/12 00:21:05 INFO Vacuumed table table=objects28882026/11/12 00:21:05 INFO Vacuumed table table=objects28892026/11/12 00:21:05 INFO Vacuumed table table=objects28902026/11/12 00:21:05 INFO Received push request method=POST path=/28912026/11/12 00:21:05 INFO Received push request method=POST path=/28922026/11/12 00:21:05 INFO Received complete push request method=POST path=/28932026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=028942026/11/12 00:21:05 INFO Completed push id=5 objects=3 duration=7.325791ms28952026/11/12 00:21:05 WARN Force mode enabled - objects will be deleted immediately without grace period28962026/11/12 00:21:05 INFO Received complete upload request method=POST path=/28972026/11/12 00:21:05 INFO Completed upload id=11 objects=6 duration=1.397291ms28982026/11/12 00:21:05 INFO Received push request method=POST path=/28992026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=029002026/11/12 00:21:05 INFO Vacuumed table table=pending_closures29012026/11/12 00:21:05 INFO Vacuumed table table=pending_objects29022026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads29032026/11/12 00:21:05 INFO Received push request method=POST path=/29042026/11/12 00:21:05 INFO Vacuumed table table=closures29052026/11/12 00:21:05 INFO Received push request method=POST path=/29062026/11/12 00:21:05 INFO Vacuumed table table=objects29072026/11/12 00:21:05 INFO Received uploads request method=POST path=/29082026/11/12 00:21:05 INFO Received complete push request method=POST path=/29092026/11/12 00:21:05 INFO Completed push id=5 objects=3 duration=2.708459ms29102026/11/12 00:21:05 INFO Received push request method=POST path=/29112026/11/12 00:21:05 INFO Received uploads request method=POST path=/29122026/11/12 00:21:05 INFO Received push request method=POST path=/29132026/11/12 00:21:05 INFO Received push request method=POST path=/29142026/11/12 00:21:05 INFO Received uploads request method=POST path=/29152026/11/12 00:21:05 INFO Received complete push request method=POST path=/29162026/11/12 00:21:05 INFO Completed push id=2 objects=4 duration=1.660708ms29172026/11/12 00:21:05 INFO Received push request method=POST path=/29182026/11/12 00:21:05 INFO Received complete push request method=POST path=/29192026/11/12 00:21:05 INFO Received uploads request method=POST path=/29202026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=029212026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=029222026/11/12 00:21:05 INFO Completed push id=2 objects=10 duration=5.345417ms29232026/11/12 00:21:05 WARN Force mode enabled - objects will be deleted immediately without grace period29242026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=029252026/11/12 00:21:05 INFO Received complete upload request method=POST path=/29262026/11/12 00:21:05 INFO Received uploads request method=POST path=/29272026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=029282026/11/12 00:21:05 INFO Completed upload id=4 objects=7 duration=1.55775ms29292026/11/12 00:21:05 INFO Vacuumed table table=pending_closures29302026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029312026/11/12 00:21:05 INFO Vacuumed table table=pending_objects29322026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads29332026/11/12 00:21:05 INFO Vacuumed table table=pending_closures29342026/11/12 00:21:05 INFO Vacuumed table table=closures29352026/11/12 00:21:05 INFO Vacuumed table table=pending_objects29362026/11/12 00:21:05 INFO Received push request method=POST path=/29372026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads29382026/11/12 00:21:05 INFO Vacuumed table table=closures29392026/11/12 00:21:05 INFO Vacuumed table table=objects29402026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=029412026/11/12 00:21:05 INFO Received complete push request method=POST path=/29422026/11/12 00:21:05 INFO Completed push id=8 objects=6 duration=1.120375ms29432026/11/12 00:21:05 INFO Received complete push request method=POST path=/29442026/11/12 00:21:05 INFO Vacuumed table table=objects29452026/11/12 00:21:05 INFO Completed push id=12 objects=15 duration=3.83ms29462026/11/12 00:21:05 INFO Vacuumed table table=pending_closures29472026/11/12 00:21:05 INFO Received uploads request method=POST path=/29482026/11/12 00:21:05 INFO Vacuumed table table=pending_objects29492026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads29502026/11/12 00:21:05 INFO Received push request method=POST path=/29512026/11/12 00:21:05 INFO Vacuumed table table=closures29522026/11/12 00:21:05 INFO Vacuumed table table=objects29532026/11/12 00:21:05 INFO Received complete upload request method=POST path=/29542026/11/12 00:21:05 INFO Completed upload id=15 objects=4 duration=2.64375ms29552026/11/12 00:21:05 INFO Received push request method=POST path=/29562026/11/12 00:21:05 INFO Received push request method=POST path=/29572026/11/12 00:21:05 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29582026/11/12 00:21:05 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29592026/11/12 00:21:05 INFO Received push request method=POST path=/29602026/11/12 00:21:05 INFO Received push request method=POST path=/29612026/11/12 00:21:05 INFO Received push request method=POST path=/29622026/11/12 00:21:05 INFO Received uploads request method=POST path=/29632026/11/12 00:21:05 INFO Received uploads request method=POST path=/29642026/11/12 00:21:05 INFO Received push request method=POST path=/29652026/11/12 00:21:05 INFO Received complete push request method=POST path=/29662026/11/12 00:21:05 INFO Completed push id=8 objects=7 duration=6.259583ms29672026/11/12 00:21:05 INFO Received push request method=POST path=/29682026/11/12 00:21:05 INFO Received push request method=POST path=/29692026/11/12 00:21:05 INFO Received push request method=POST path=/29702026/11/12 00:21:05 INFO Received uploads request method=POST path=/29712026/11/12 00:21:05 INFO Received uploads request method=POST path=/29722026/11/12 00:21:05 INFO Received uploads request method=POST path=/29732026/11/12 00:21:05 INFO Received uploads request method=POST path=/29742026/11/12 00:21:05 INFO Received uploads request method=POST path=/29752026/11/12 00:21:05 INFO Aborted multipart uploads count=0 kept=029762026/11/12 00:21:05 INFO Received complete push request method=POST path=/29772026/11/12 00:21:05 INFO Received complete push request method=POST path=/29782026/11/12 00:21:05 WARN Force mode enabled - objects will be deleted immediately without grace period29792026/11/12 00:21:05 INFO Completed push id=14 objects=3 duration=765.208µs29802026/11/12 00:21:05 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029812026/11/12 00:21:05 INFO Completed push id=6 objects=26 duration=6.965708ms29822026/11/12 00:21:05 INFO Vacuumed table table=pending_closures29832026/11/12 00:21:05 INFO Vacuumed table table=pending_objects29842026/11/12 00:21:05 INFO Received uploads request method=POST path=/29852026/11/12 00:21:05 INFO Received push request method=POST path=/29862026/11/12 00:21:05 INFO Vacuumed table table=multipart_uploads29872026/11/12 00:21:05 INFO Vacuumed table table=closures29882026/11/12 00:21:05 INFO Received push request method=POST path=/29892026/11/12 00:21:05 INFO Vacuumed table table=objects29902026/11/12 00:21:05 INFO Received complete upload request method=POST path=/29912026/11/12 00:21:05 INFO Completed upload id=16 objects=3 duration=822.958µs29922026/11/12 00:21:05 INFO Received complete upload request method=POST path=/29932026/11/12 00:21:06 INFO Completed upload id=3 objects=8 duration=2.440958ms29942026/11/12 00:21:06 INFO Received push request method=POST path=/29952026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=029962026/11/12 00:21:06 INFO Received push request method=POST path=/29972026/11/12 00:21:06 INFO Received push request method=POST path=/29982026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period29992026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=030002026/11/12 00:21:06 INFO Received push request method=POST path=/30012026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030022026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030032026/11/12 00:21:06 INFO Received complete upload request method=POST path=/30042026/11/12 00:21:06 INFO Vacuumed table table=pending_closures30052026/11/12 00:21:06 INFO Received complete push request method=POST path=/30062026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=030072026/11/12 00:21:06 INFO Completed upload id=16 objects=17 duration=986.209µs30082026/11/12 00:21:06 INFO Vacuumed table table=pending_closures30092026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=030102026/11/12 00:21:06 INFO Received uploads request method=POST path=/30112026/11/12 00:21:06 INFO Received uploads request method=POST path=/30122026/11/12 00:21:06 INFO Vacuumed table table=pending_objects30132026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period30142026/11/12 00:21:06 INFO Completed push id=1 objects=7 duration=9.062042ms30152026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period30162026/11/12 00:21:06 INFO Received complete upload request method=POST path=/30172026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads30182026/11/12 00:21:06 INFO Vacuumed table table=pending_objects30192026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads30202026/11/12 00:21:06 INFO Vacuumed table table=closures30212026/11/12 00:21:06 INFO Completed upload id=9 objects=16 duration=8.203083ms30222026/11/12 00:21:06 INFO Received uploads request method=POST path=/30232026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030242026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030252026/11/12 00:21:06 INFO Vacuumed table table=objects30262026/11/12 00:21:06 INFO Vacuumed table table=pending_closures30272026/11/12 00:21:06 INFO Received push request method=POST path=/30282026/11/12 00:21:06 INFO Vacuumed table table=pending_closures30292026/11/12 00:21:06 INFO Vacuumed table table=pending_objects30302026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=030312026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads30322026/11/12 00:21:06 INFO Vacuumed table table=closures30332026/11/12 00:21:06 INFO Vacuumed table table=pending_objects30342026/11/12 00:21:06 INFO Received push request method=POST path=/30352026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period30362026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads30372026/11/12 00:21:06 INFO Vacuumed table table=closures30382026/11/12 00:21:06 INFO Vacuumed table table=closures30392026/11/12 00:21:06 INFO Vacuumed table table=objects30402026/11/12 00:21:06 INFO Vacuumed table table=objects30412026/11/12 00:21:06 INFO Vacuumed table table=objects30422026/11/12 00:21:06 INFO Received complete push request method=POST path=/30432026/11/12 00:21:06 INFO Completed push id=5 objects=9 duration=5.2065ms30442026/11/12 00:21:06 INFO Received complete upload request method=POST path=/30452026/11/12 00:21:06 INFO Completed upload id=10 objects=17 duration=4.558333ms30462026/11/12 00:21:06 INFO Received uploads request method=POST path=/30472026/11/12 00:21:06 INFO Received push request method=POST path=/30482026/11/12 00:21:06 INFO Received uploads request method=POST path=/30492026/11/12 00:21:06 INFO Received push request method=POST path=/30502026/11/12 00:21:06 INFO Received push request method=POST path=/30512026/11/12 00:21:06 INFO Received push request method=POST path=/30522026/11/12 00:21:06 INFO Received uploads request method=POST path=/30532026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=030542026/11/12 00:21:06 INFO Received complete push request method=POST path=/30552026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period30562026/11/12 00:21:06 INFO Received uploads request method=POST path=/30572026/11/12 00:21:06 INFO Completed push id=6 objects=7 duration=5.481ms30582026/11/12 00:21:06 INFO Received uploads request method=POST path=/30592026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030602026/11/12 00:21:06 INFO Vacuumed table table=pending_closures30612026/11/12 00:21:06 INFO Vacuumed table table=pending_objects30622026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads30632026/11/12 00:21:06 INFO Received uploads request method=POST path=/30642026/11/12 00:21:06 INFO Vacuumed table table=closures30652026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=030662026/11/12 00:21:06 INFO Received push request method=POST path=/30672026/11/12 00:21:06 INFO Received push request method=POST path=/30682026/11/12 00:21:06 INFO Vacuumed table table=objects30692026/11/12 00:21:06 INFO Received uploads request method=POST path=/30702026/11/12 00:21:06 INFO Received push request method=POST path=/30712026/11/12 00:21:06 INFO Received uploads request method=POST path=/30722026/11/12 00:21:06 INFO Received uploads request method=POST path=/30732026/11/12 00:21:06 INFO Received complete push request method=POST path=/30742026/11/12 00:21:06 INFO Completed push id=10 objects=3 duration=917.667µs30752026/11/12 00:21:06 INFO Received uploads request method=POST path=/30762026/11/12 00:21:06 INFO Received push request method=POST path=/30772026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=030782026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period30792026/11/12 00:21:06 INFO Received uploads request method=POST path=/30802026/11/12 00:21:06 INFO Received uploads request method=POST path=/30812026/11/12 00:21:06 INFO Received complete upload request method=POST path=/30822026/11/12 00:21:06 INFO Received push request method=POST path=/30832026/11/12 00:21:06 INFO Received complete upload request method=POST path=/30842026/11/12 00:21:06 INFO Received uploads request method=POST path=/30852026/11/12 00:21:06 INFO Completed upload id=6 objects=24 duration=2.349333ms30862026/11/12 00:21:06 INFO Completed upload id=18 objects=4 duration=2.758708ms30872026/11/12 00:21:06 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30882026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30892026/11/12 00:21:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30902026/11/12 00:21:06 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30912026/11/12 00:21:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30922026/11/12 00:21:06 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30932026/11/12 00:21:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30942026/11/12 00:21:06 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30952026/11/12 00:21:06 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30962026/11/12 00:21:06 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30972026/11/12 00:21:06 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30982026/11/12 00:21:06 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30992026/11/12 00:21:06 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."31002026/11/12 00:21:06 INFO Received uploads request method=POST path=/31012026/11/12 00:21:06 INFO Received push request method=POST path=/31022026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."31032026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."31042026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."31052026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."31062026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31072026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."31082026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31092026/11/12 00:21:06 INFO Received complete upload request method=POST path=/31102026/11/12 00:21:06 INFO Completed upload id=15 objects=6 duration=3.095417ms31112026/11/12 00:21:06 INFO Received complete push request method=POST path=/31122026/11/12 00:21:06 INFO Completed push id=14 objects=6 duration=871.5µs31132026/11/12 00:21:06 INFO Received uploads request method=POST path=/31142026/11/12 00:21:06 INFO Received push request method=POST path=/31152026/11/12 00:21:06 INFO Received push request method=POST path=/31162026/11/12 00:21:06 INFO Received push request method=POST path=/31172026/11/12 00:21:06 INFO Received uploads request method=POST path=/31182026/11/12 00:21:06 INFO Received push request method=POST path=/31192026/11/12 00:21:06 INFO Received uploads request method=POST path=/31202026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=031212026/11/12 00:21:06 INFO Received push request method=POST path=/31222026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period31232026/11/12 00:21:06 INFO Received push request method=POST path=/31242026/11/12 00:21:06 INFO Received push request method=POST path=/31252026/11/12 00:21:06 INFO Received push request method=POST path=/31262026/11/12 00:21:06 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31272026/11/12 00:21:06 INFO Received push request method=POST path=/31282026/11/12 00:21:06 INFO Received uploads request method=POST path=/31292026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=031302026/11/12 00:21:06 INFO Received push request method=POST path=/31312026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period31322026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031332026/11/12 00:21:06 INFO Vacuumed table table=pending_closures31342026/11/12 00:21:06 INFO Received complete push request method=POST path=/31352026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=031362026/11/12 00:21:06 INFO Vacuumed table table=pending_objects31372026/11/12 00:21:06 INFO Completed push id=17 objects=6 duration=824.125µs31382026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads31392026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=031402026/11/12 00:21:06 INFO Vacuumed table table=pending_closures31412026/11/12 00:21:06 INFO Vacuumed table table=closures31422026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period31432026/11/12 00:21:06 INFO Vacuumed table table=objects31442026/11/12 00:21:06 INFO Vacuumed table table=pending_objects31452026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads31462026/11/12 00:21:06 INFO Vacuumed table table=closures31472026/11/12 00:21:06 INFO Vacuumed table table=objects31482026/11/12 00:21:06 INFO Received uploads request method=POST path=/31492026/11/12 00:21:06 INFO Received complete push request method=POST path=/31502026/11/12 00:21:06 INFO Completed push id=3 objects=21 duration=5.538167ms31512026/11/12 00:21:06 INFO Received complete push request method=POST path=/31522026/11/12 00:21:06 INFO Completed push id=8 objects=6 duration=1.778041ms31532026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=031542026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=031552026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period31562026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031572026/11/12 00:21:06 INFO Vacuumed table table=pending_closures31582026/11/12 00:21:06 WARN Force mode enabled - objects will be deleted immediately without grace period31592026/11/12 00:21:06 INFO Vacuumed table table=pending_objects31602026/11/12 00:21:06 INFO Received push request method=POST path=/31612026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads31622026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031632026/11/12 00:21:06 INFO Vacuumed table table=closures31642026/11/12 00:21:06 INFO Vacuumed table table=objects31652026/11/12 00:21:06 INFO Vacuumed table table=pending_closures31662026/11/12 00:21:06 INFO Vacuumed table table=pending_objects31672026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads31682026/11/12 00:21:06 INFO Vacuumed table table=closures31692026/11/12 00:21:06 INFO Vacuumed table table=objects31702026/11/12 00:21:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=031712026/11/12 00:21:06 INFO Vacuumed table table=pending_closures31722026/11/12 00:21:06 INFO Received complete upload request method=POST path=/31732026/11/12 00:21:06 INFO Vacuumed table table=pending_objects31742026/11/12 00:21:06 INFO Completed upload id=10 objects=6 duration=4.215375ms31752026/11/12 00:21:06 INFO Received uploads request method=POST path=/31762026/11/12 00:21:06 INFO Vacuumed table table=multipart_uploads31772026/11/12 00:21:06 INFO Vacuumed table table=closures31782026/11/12 00:21:06 INFO Vacuumed table table=objects31792026/11/12 00:21:06 INFO Received uploads request method=POST path=/31802026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=031812026/11/12 00:21:06 INFO Aborted multipart uploads count=0 kept=031822026/11/12 00:21:07 WARN Force mode enabled - objects will be deleted immediately without grace period31832026/11/12 00:21:07 WARN Force mode enabled - objects will be deleted immediately without grace period31842026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031852026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=031862026/11/12 00:21:07 INFO Vacuumed table table=pending_closures31872026/11/12 00:21:07 INFO Vacuumed table table=pending_closures31882026/11/12 00:21:07 INFO Vacuumed table table=pending_objects31892026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads31902026/11/12 00:21:07 INFO Vacuumed table table=pending_objects31912026/11/12 00:21:07 INFO Vacuumed table table=closures31922026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads31932026/11/12 00:21:07 INFO Vacuumed table table=objects31942026/11/12 00:21:07 INFO Received push request method=POST path=/31952026/11/12 00:21:07 INFO Vacuumed table table=closures31962026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=031972026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=031982026/11/12 00:21:07 INFO Vacuumed table table=objects31992026/11/12 00:21:07 WARN Force mode enabled - objects will be deleted immediately without grace period32002026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=032012026/11/12 00:21:07 INFO Vacuumed table table=pending_closures32022026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032032026/11/12 00:21:07 INFO Vacuumed table table=pending_objects32042026/11/12 00:21:07 INFO Vacuumed table table=pending_closures32052026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads32062026/11/12 00:21:07 INFO Vacuumed table table=closures32072026/11/12 00:21:07 INFO Vacuumed table table=pending_objects32082026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads32092026/11/12 00:21:07 INFO Vacuumed table table=objects32102026/11/12 00:21:07 INFO Received complete push request method=POST path=/32112026/11/12 00:21:07 INFO Received push request method=POST path=/32122026/11/12 00:21:07 INFO Vacuumed table table=closures32132026/11/12 00:21:07 INFO Completed push id=16 objects=27 duration=970.333µs32142026/11/12 00:21:07 INFO Vacuumed table table=objects32152026/11/12 00:21:07 INFO Received push request method=POST path=/32162026/11/12 00:21:07 INFO Received complete upload request method=POST path=/32172026/11/12 00:21:07 INFO Completed upload id=10 objects=6 duration=2.845958ms32182026/11/12 00:21:07 INFO Received complete push request method=POST path=/32192026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=032202026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032212026/11/12 00:21:07 INFO Received complete upload request method=POST path=/32222026/11/12 00:21:07 INFO Received complete push request method=POST path=/32232026/11/12 00:21:07 INFO Completed upload id=9 objects=3 duration=2.314375ms32242026/11/12 00:21:07 INFO Completed push id=3 objects=22 duration=14.2805ms32252026/11/12 00:21:07 INFO Vacuumed table table=pending_closures32262026/11/12 00:21:07 INFO Completed push id=6 objects=5 duration=3.229667ms32272026/11/12 00:21:07 INFO Vacuumed table table=pending_objects32282026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads32292026/11/12 00:21:07 INFO Vacuumed table table=closures32302026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=032312026/11/12 00:21:07 INFO Vacuumed table table=objects32322026/11/12 00:21:07 INFO Received uploads request method=POST path=/32332026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=032342026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=032352026/11/12 00:21:07 INFO Received uploads request method=POST path=/32362026/11/12 00:21:07 INFO Received uploads request method=POST path=/32372026/11/12 00:21:07 INFO Received push request method=POST path=/32382026/11/12 00:21:07 WARN Force mode enabled - objects will be deleted immediately without grace period32392026/11/12 00:21:07 WARN Force mode enabled - objects will be deleted immediately without grace period32402026/11/12 00:21:07 INFO Received uploads request method=POST path=/32412026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032422026/11/12 00:21:07 INFO Received push request method=POST path=/32432026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032442026/11/12 00:21:07 INFO Vacuumed table table=pending_closures32452026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032462026/11/12 00:21:07 INFO Received uploads request method=POST path=/32472026/11/12 00:21:07 INFO Received push request method=POST path=/32482026/11/12 00:21:07 INFO Vacuumed table table=pending_closures32492026/11/12 00:21:07 INFO Received complete upload request method=POST path=/32502026/11/12 00:21:07 INFO Vacuumed table table=pending_objects32512026/11/12 00:21:07 INFO Vacuumed table table=pending_closures32522026/11/12 00:21:07 INFO Vacuumed table table=pending_objects32532026/11/12 00:21:07 INFO Received push request method=POST path=/32542026/11/12 00:21:07 INFO Received uploads request method=POST path=/32552026/11/12 00:21:07 INFO Vacuumed table table=pending_objects32562026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads32572026/11/12 00:21:07 INFO Completed upload id=19 objects=25 duration=10.453833ms32582026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads32592026/11/12 00:21:07 INFO Received push request method=POST path=/32602026/11/12 00:21:07 INFO Vacuumed table table=closures32612026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads32622026/11/12 00:21:07 INFO Vacuumed table table=closures32632026/11/12 00:21:07 INFO Vacuumed table table=closures32642026/11/12 00:21:07 INFO Received uploads request method=POST path=/32652026/11/12 00:21:07 INFO Vacuumed table table=objects32662026/11/12 00:21:07 INFO Vacuumed table table=objects32672026/11/12 00:21:07 INFO Vacuumed table table=objects32682026/11/12 00:21:07 INFO Received push request method=POST path=/32692026/11/12 00:21:07 INFO Received push request method=POST path=/32702026/11/12 00:21:07 INFO Received uploads request method=POST path=/32712026/11/12 00:21:07 INFO Received uploads request method=POST path=/32722026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=032732026/11/12 00:21:07 INFO Received push request method=POST path=/32742026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=032752026/11/12 00:21:07 WARN Force mode enabled - objects will be deleted immediately without grace period32762026/11/12 00:21:07 INFO Received uploads request method=POST path=/32772026/11/12 00:21:07 INFO Received push request method=POST path=/32782026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032792026/11/12 00:21:07 INFO Vacuumed table table=pending_closures32802026/11/12 00:21:07 INFO Vacuumed table table=pending_objects32812026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads32822026/11/12 00:21:07 INFO Vacuumed table table=closures32832026/11/12 00:21:07 INFO Received complete upload request method=POST path=/32842026/11/12 00:21:07 INFO Vacuumed table table=objects32852026/11/12 00:21:07 INFO Completed upload id=15 objects=3 duration=1.459333ms32862026/11/12 00:21:07 INFO Received push request method=POST path=/32872026/11/12 00:21:07 INFO Received complete push request method=POST path=/32882026/11/12 00:21:07 INFO Completed push id=8 objects=6 duration=3.610792ms32892026/11/12 00:21:07 INFO Received complete push request method=POST path=/32902026/11/12 00:21:07 INFO Completed push id=15 objects=5 duration=1.790709ms32912026/11/12 00:21:07 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32922026/11/12 00:21:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32932026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=032942026/11/12 00:21:07 WARN Force mode enabled - objects will be deleted immediately without grace period32952026/11/12 00:21:07 INFO Received complete push request method=POST path=/32962026/11/12 00:21:07 INFO Completed push id=12 objects=2 duration=1.366917ms32972026/11/12 00:21:07 INFO Received complete push request method=POST path=/32982026/11/12 00:21:07 INFO Completed push id=18 objects=10 duration=834.25µs32992026/11/12 00:21:07 INFO Received uploads request method=POST path=/33002026/11/12 00:21:07 INFO Received uploads request method=POST path=/33012026/11/12 00:21:07 INFO Received push request method=POST path=/33022026/11/12 00:21:07 INFO Received complete push request method=POST path=/33032026/11/12 00:21:07 INFO Completed push id=12 objects=7 duration=586.208µs33042026/11/12 00:21:07 INFO Received uploads request method=POST path=/33052026/11/12 00:21:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33062026/11/12 00:21:07 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."33072026/11/12 00:21:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33082026/11/12 00:21:07 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33092026/11/12 00:21:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33102026/11/12 00:21:07 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33112026/11/12 00:21:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."33122026/11/12 00:21:07 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."33132026/11/12 00:21:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33142026/11/12 00:21:07 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."33152026/11/12 00:21:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."33162026/11/12 00:21:07 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33172026/11/12 00:21:07 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."33182026/11/12 00:21:07 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."33192026/11/12 00:21:07 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33202026/11/12 00:21:07 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."33212026/11/12 00:21:07 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."33222026/11/12 00:21:07 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33232026/11/12 00:21:07 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33242026/11/12 00:21:07 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33252026/11/12 00:21:07 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33262026/11/12 00:21:07 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33272026/11/12 00:21:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33282026/11/12 00:21:07 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33292026/11/12 00:21:07 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33302026/11/12 00:21:07 INFO Received uploads request method=POST path=/33312026/11/12 00:21:07 INFO Received push request method=POST path=/33322026/11/12 00:21:07 INFO Received uploads request method=POST path=/33332026/11/12 00:21:07 INFO Received uploads request method=POST path=/33342026/11/12 00:21:07 INFO Received push request method=POST path=/33352026/11/12 00:21:07 INFO Received uploads request method=POST path=/33362026/11/12 00:21:07 INFO Received push request method=POST path=/33372026/11/12 00:21:07 INFO Received push request method=POST path=/33382026/11/12 00:21:07 INFO Aborted multipart uploads count=0 kept=033392026/11/12 00:21:07 INFO Received uploads request method=POST path=/33402026/11/12 00:21:07 INFO Received push request method=POST path=/33412026/11/12 00:21:07 INFO Received complete push request method=POST path=/33422026/11/12 00:21:07 INFO Completed push id=11 objects=11 duration=2.096209ms33432026/11/12 00:21:07 INFO Received complete push request method=POST path=/33442026/11/12 00:21:07 INFO Completed push id=18 objects=6 duration=971.166µs33452026/11/12 00:21:07 INFO Received uploads request method=POST path=/33462026/11/12 00:21:07 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033472026/11/12 00:21:07 INFO Vacuumed table table=pending_closures33482026/11/12 00:21:07 INFO Vacuumed table table=pending_objects33492026/11/12 00:21:07 INFO Vacuumed table table=multipart_uploads33502026/11/12 00:21:07 INFO Vacuumed table table=closures33512026/11/12 00:21:07 INFO Vacuumed table table=objects33522026/11/12 00:21:08 INFO Received uploads request method=POST path=/33532026/11/12 00:21:08 INFO Received uploads request method=POST path=/33542026/11/12 00:21:08 INFO Received uploads request method=POST path=/33552026/11/12 00:21:08 INFO Received complete push request method=POST path=/33562026/11/12 00:21:08 INFO Completed push id=12 objects=3 duration=3.129542ms33572026/11/12 00:21:08 INFO Received complete upload request method=POST path=/33582026/11/12 00:21:08 INFO Completed upload id=23 objects=9 duration=2.657292ms33592026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=033602026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=033612026/11/12 00:21:08 INFO Received push request method=POST path=/33622026/11/12 00:21:08 WARN Force mode enabled - objects will be deleted immediately without grace period33632026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033642026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033652026/11/12 00:21:08 INFO Vacuumed table table=pending_closures33662026/11/12 00:21:08 INFO Received push request method=POST path=/33672026/11/12 00:21:08 INFO Received uploads request method=POST path=/33682026/11/12 00:21:08 INFO Vacuumed table table=pending_objects33692026/11/12 00:21:08 INFO Received complete upload request method=POST path=/33702026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads33712026/11/12 00:21:08 INFO Completed upload id=11 objects=6 duration=5.466375ms33722026/11/12 00:21:08 INFO Vacuumed table table=closures33732026/11/12 00:21:08 INFO Vacuumed table table=objects33742026/11/12 00:21:08 INFO Received push request method=POST path=/33752026/11/12 00:21:08 INFO Vacuumed table table=pending_closures33762026/11/12 00:21:08 INFO Received complete upload request method=POST path=/33772026/11/12 00:21:08 INFO Completed upload id=17 objects=6 duration=2.765875ms33782026/11/12 00:21:08 INFO Vacuumed table table=pending_objects33792026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads33802026/11/12 00:21:08 INFO Vacuumed table table=closures33812026/11/12 00:21:08 INFO Vacuumed table table=objects33822026/11/12 00:21:08 INFO Received complete upload request method=POST path=/33832026/11/12 00:21:08 INFO Completed upload id=2 objects=2 duration=944.083µs33842026/11/12 00:21:08 INFO Received uploads request method=POST path=/33852026/11/12 00:21:08 INFO Received uploads request method=POST path=/33862026/11/12 00:21:08 INFO Received uploads request method=POST path=/33872026/11/12 00:21:08 INFO Received uploads request method=POST path=/33882026/11/12 00:21:08 INFO Received uploads request method=POST path=/33892026/11/12 00:21:08 INFO Received uploads request method=POST path=/33902026/11/12 00:21:08 INFO Received push request method=POST path=/33912026/11/12 00:21:08 INFO Received push request method=POST path=/33922026/11/12 00:21:08 INFO Received push request method=POST path=/33932026/11/12 00:21:08 INFO Received push request method=POST path=/33942026/11/12 00:21:08 INFO Received uploads request method=POST path=/33952026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=033962026/11/12 00:21:08 INFO Received push request method=POST path=/33972026/11/12 00:21:08 INFO Received complete push request method=POST path=/33982026/11/12 00:21:08 INFO Completed push id=17 objects=22 duration=1.487416ms33992026/11/12 00:21:08 INFO Received push request method=POST path=/34002026/11/12 00:21:08 INFO Received complete push request method=POST path=/34012026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034022026/11/12 00:21:08 INFO Completed push id=6 objects=5 duration=16.351666ms34032026/11/12 00:21:08 INFO Vacuumed table table=pending_closures34042026/11/12 00:21:08 INFO Received uploads request method=POST path=/34052026/11/12 00:21:08 INFO Vacuumed table table=pending_objects34062026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads34072026/11/12 00:21:08 INFO Vacuumed table table=closures34082026/11/12 00:21:08 INFO Received uploads request method=POST path=/34092026/11/12 00:21:08 INFO Vacuumed table table=objects34102026/11/12 00:21:08 INFO Received push request method=POST path=/34112026/11/12 00:21:08 INFO Received uploads request method=POST path=/34122026/11/12 00:21:08 INFO Received uploads request method=POST path=/34132026/11/12 00:21:08 INFO Received push request method=POST path=/34142026/11/12 00:21:08 INFO Received push request method=POST path=/34152026/11/12 00:21:08 INFO Received complete push request method=POST path=/34162026/11/12 00:21:08 INFO Completed push id=4 objects=6 duration=12.104709ms34172026/11/12 00:21:08 INFO Received push request method=POST path=/34182026/11/12 00:21:08 INFO Received push request method=POST path=/34192026/11/12 00:21:08 INFO Received push request method=POST path=/34202026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=034212026/11/12 00:21:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34222026/11/12 00:21:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."34232026/11/12 00:21:08 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."34242026/11/12 00:21:08 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."34252026/11/12 00:21:08 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."34262026/11/12 00:21:08 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."34272026/11/12 00:21:08 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."34282026/11/12 00:21:08 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34292026/11/12 00:21:08 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."34302026/11/12 00:21:08 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."34312026/11/12 00:21:08 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."34322026/11/12 00:21:08 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."34332026/11/12 00:21:08 INFO Received uploads request method=POST path=/34342026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=034352026/11/12 00:21:08 INFO Received push request method=POST path=/34362026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=034372026/11/12 00:21:08 WARN Force mode enabled - objects will be deleted immediately without grace period34382026/11/12 00:21:08 WARN Force mode enabled - objects will be deleted immediately without grace period34392026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034402026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034412026/11/12 00:21:08 INFO Vacuumed table table=pending_closures34422026/11/12 00:21:08 INFO Vacuumed table table=pending_closures34432026/11/12 00:21:08 INFO Vacuumed table table=pending_objects34442026/11/12 00:21:08 INFO Vacuumed table table=pending_objects34452026/11/12 00:21:08 INFO Received uploads request method=POST path=/34462026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads34472026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads34482026/11/12 00:21:08 INFO Received push request method=POST path=/34492026/11/12 00:21:08 INFO Vacuumed table table=closures34502026/11/12 00:21:08 INFO Vacuumed table table=closures34512026/11/12 00:21:08 INFO Vacuumed table table=objects34522026/11/12 00:21:08 INFO Vacuumed table table=objects34532026/11/12 00:21:08 INFO Received complete push request method=POST path=/34542026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=034552026/11/12 00:21:08 INFO Completed push id=9 objects=5 duration=1.209917ms34562026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034572026/11/12 00:21:08 INFO Vacuumed table table=pending_closures34582026/11/12 00:21:08 INFO Vacuumed table table=pending_objects34592026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads34602026/11/12 00:21:08 INFO Vacuumed table table=closures34612026/11/12 00:21:08 INFO Vacuumed table table=objects34622026/11/12 00:21:08 INFO Received uploads request method=POST path=/34632026/11/12 00:21:08 INFO Received uploads request method=POST path=/34642026/11/12 00:21:08 INFO Received push request method=POST path=/34652026/11/12 00:21:08 INFO Received push request method=POST path=/34662026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=034672026/11/12 00:21:08 WARN Force mode enabled - objects will be deleted immediately without grace period34682026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034692026/11/12 00:21:08 INFO Vacuumed table table=pending_closures34702026/11/12 00:21:08 INFO Vacuumed table table=pending_objects34712026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads34722026/11/12 00:21:08 INFO Vacuumed table table=closures34732026/11/12 00:21:08 INFO Vacuumed table table=objects34742026/11/12 00:21:08 INFO Received uploads request method=POST path=/34752026/11/12 00:21:08 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34762026/11/12 00:21:08 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34772026/11/12 00:21:08 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34782026/11/12 00:21:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."34792026/11/12 00:21:08 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."34802026/11/12 00:21:08 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."34812026/11/12 00:21:08 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34822026/11/12 00:21:08 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."34832026/11/12 00:21:08 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34842026/11/12 00:21:08 INFO Received push request method=POST path=/34852026/11/12 00:21:08 INFO Received push request method=POST path=/34862026/11/12 00:21:08 INFO Received complete push request method=POST path=/34872026/11/12 00:21:08 INFO Received complete upload request method=POST path=/34882026/11/12 00:21:08 INFO Completed upload id=16 objects=23 duration=3.440458ms34892026/11/12 00:21:08 INFO Completed push id=22 objects=3 duration=3.502417ms34902026/11/12 00:21:08 INFO Received push request method=POST path=/34912026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=034922026/11/12 00:21:08 WARN Force mode enabled - objects will be deleted immediately without grace period34932026/11/12 00:21:08 INFO Received uploads request method=POST path=/34942026/11/12 00:21:08 INFO Received complete upload request method=POST path=/34952026/11/12 00:21:08 INFO Completed upload id=9 objects=2 duration=821.875µs34962026/11/12 00:21:08 INFO Received push request method=POST path=/34972026/11/12 00:21:08 INFO Received push request method=POST path=/34982026/11/12 00:21:08 INFO Received push request method=POST path=/34992026/11/12 00:21:08 INFO Received uploads request method=POST path=/35002026/11/12 00:21:08 INFO Received complete push request method=POST path=/35012026/11/12 00:21:08 INFO Received push request method=POST path=/35022026/11/12 00:21:08 INFO Completed push id=16 objects=4 duration=5.265875ms35032026/11/12 00:21:08 INFO Received uploads request method=POST path=/35042026/11/12 00:21:08 INFO Received complete upload request method=POST path=/35052026/11/12 00:21:08 INFO Completed upload id=15 objects=6 duration=1.166792ms35062026/11/12 00:21:08 INFO Received push request method=POST path=/35072026/11/12 00:21:08 INFO Received push request method=POST path=/35082026/11/12 00:21:08 INFO Received push request method=POST path=/35092026/11/12 00:21:08 INFO Received complete push request method=POST path=/35102026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=035112026/11/12 00:21:08 INFO Completed push id=8 objects=3 duration=5.71725ms35122026/11/12 00:21:08 INFO Received push request method=POST path=/35132026/11/12 00:21:08 INFO Received uploads request method=POST path=/35142026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=035152026/11/12 00:21:08 WARN Force mode enabled - objects will be deleted immediately without grace period35162026/11/12 00:21:08 INFO Received uploads request method=POST path=/35172026/11/12 00:21:08 INFO Received uploads request method=POST path=/35182026/11/12 00:21:08 INFO Received uploads request method=POST path=/35192026/11/12 00:21:08 INFO Received push request method=POST path=/35202026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=035212026/11/12 00:21:08 INFO Received uploads request method=POST path=/35222026/11/12 00:21:08 INFO Received push request method=POST path=/35232026/11/12 00:21:08 INFO Vacuumed table table=pending_closures35242026/11/12 00:21:08 INFO Aborted multipart uploads count=0 kept=035252026/11/12 00:21:08 INFO Received push request method=POST path=/35262026/11/12 00:21:08 INFO Vacuumed table table=pending_objects35272026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads35282026/11/12 00:21:08 INFO Received complete push request method=POST path=/35292026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035302026/11/12 00:21:08 INFO Received complete push request method=POST path=/35312026/11/12 00:21:08 INFO Completed push id=19 objects=14 duration=1.202417ms35322026/11/12 00:21:08 INFO Completed push id=14 objects=3 duration=4.884291ms35332026/11/12 00:21:08 INFO Vacuumed table table=closures35342026/11/12 00:21:08 INFO Vacuumed table table=pending_closures35352026/11/12 00:21:08 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=035362026/11/12 00:21:08 INFO Received complete upload request method=POST path=/35372026/11/12 00:21:08 INFO Completed upload id=20 objects=5 duration=649.417µs35382026/11/12 00:21:08 INFO Vacuumed table table=objects35392026/11/12 00:21:08 INFO Vacuumed table table=pending_closures35402026/11/12 00:21:08 INFO Vacuumed table table=pending_objects35412026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads35422026/11/12 00:21:08 INFO Vacuumed table table=pending_objects35432026/11/12 00:21:08 INFO Vacuumed table table=multipart_uploads35442026/11/12 00:21:08 INFO Vacuumed table table=closures35452026/11/12 00:21:08 INFO Vacuumed table table=objects35462026/11/12 00:21:08 INFO Vacuumed table table=closures35472026/11/12 00:21:08 INFO Vacuumed table table=objects35482026/11/12 00:21:09 INFO Received complete push request method=POST path=/35492026/11/12 00:21:09 INFO Completed push id=26 objects=14 duration=5.904458ms35502026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=035512026/11/12 00:21:09 INFO Received push request method=POST path=/35522026/11/12 00:21:09 WARN Force mode enabled - objects will be deleted immediately without grace period35532026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035542026/11/12 00:21:09 INFO Vacuumed table table=pending_closures35552026/11/12 00:21:09 INFO Vacuumed table table=pending_objects35562026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads35572026/11/12 00:21:09 INFO Vacuumed table table=closures35582026/11/12 00:21:09 INFO Vacuumed table table=objects35592026/11/12 00:21:09 INFO Received push request method=POST path=/35602026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=035612026/11/12 00:21:09 WARN Force mode enabled - objects will be deleted immediately without grace period35622026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035632026/11/12 00:21:09 INFO Received uploads request method=POST path=/35642026/11/12 00:21:09 INFO Vacuumed table table=pending_closures35652026/11/12 00:21:09 INFO Received push request method=POST path=/35662026/11/12 00:21:09 INFO Vacuumed table table=pending_objects35672026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads35682026/11/12 00:21:09 INFO Vacuumed table table=closures35692026/11/12 00:21:09 INFO Received complete upload request method=POST path=/35702026/11/12 00:21:09 INFO Vacuumed table table=objects35712026/11/12 00:21:09 INFO Completed upload id=17 objects=7 duration=890.125µs35722026/11/12 00:21:09 INFO Received uploads request method=POST path=/35732026/11/12 00:21:09 INFO Received uploads request method=POST path=/35742026/11/12 00:21:09 INFO Received push request method=POST path=/35752026/11/12 00:21:09 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35762026/11/12 00:21:09 INFO Received uploads request method=POST path=/35772026/11/12 00:21:09 INFO Received complete push request method=POST path=/35782026/11/12 00:21:09 INFO Completed push id=16 objects=26 duration=887.291µs35792026/11/12 00:21:09 INFO Received uploads request method=POST path=/35802026/11/12 00:21:09 INFO Received uploads request method=POST path=/35812026/11/12 00:21:09 INFO Received uploads request method=POST path=/35822026/11/12 00:21:09 INFO Received complete push request method=POST path=/35832026/11/12 00:21:09 INFO Received complete upload request method=POST path=/35842026/11/12 00:21:09 INFO Completed upload id=13 objects=10 duration=612.75µs35852026/11/12 00:21:09 INFO Completed push id=21 objects=9 duration=878.875µs35862026/11/12 00:21:09 INFO Received complete upload request method=POST path=/35872026/11/12 00:21:09 INFO Completed upload id=10 objects=2 duration=1.715458ms35882026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=035892026/11/12 00:21:09 INFO Received uploads request method=POST path=/35902026/11/12 00:21:09 INFO Received push request method=POST path=/35912026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=035922026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035932026/11/12 00:21:09 INFO Vacuumed table table=pending_closures35942026/11/12 00:21:09 INFO Vacuumed table table=pending_objects35952026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads35962026/11/12 00:21:09 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35972026/11/12 00:21:09 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35982026/11/12 00:21:09 INFO Vacuumed table table=closures35992026/11/12 00:21:09 INFO Vacuumed table table=objects36002026/11/12 00:21:09 INFO Received uploads request method=POST path=/36012026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036022026/11/12 00:21:09 INFO Received push request method=POST path=/36032026/11/12 00:21:09 WARN Force mode enabled - objects will be deleted immediately without grace period36042026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036052026/11/12 00:21:09 INFO Vacuumed table table=pending_closures36062026/11/12 00:21:09 INFO Received complete push request method=POST path=/36072026/11/12 00:21:09 INFO Vacuumed table table=pending_objects36082026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036092026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads36102026/11/12 00:21:09 INFO Completed push id=19 objects=8 duration=5.860958ms36112026/11/12 00:21:09 WARN Force mode enabled - objects will be deleted immediately without grace period36122026/11/12 00:21:09 INFO Vacuumed table table=closures36132026/11/12 00:21:09 INFO Vacuumed table table=objects36142026/11/12 00:21:09 INFO Received complete upload request method=POST path=/36152026/11/12 00:21:09 INFO Completed upload id=20 objects=6 duration=659.292µs36162026/11/12 00:21:09 INFO Received complete upload request method=POST path=/36172026/11/12 00:21:09 INFO Completed upload id=15 objects=9 duration=581.041µs36182026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036192026/11/12 00:21:09 INFO Received push request method=POST path=/36202026/11/12 00:21:09 INFO Received push request method=POST path=/36212026/11/12 00:21:09 INFO Received uploads request method=POST path=/36222026/11/12 00:21:09 INFO Received uploads request method=POST path=/36232026/11/12 00:21:09 INFO Received push request method=POST path=/36242026/11/12 00:21:09 INFO Received complete push request method=POST path=/36252026/11/12 00:21:09 INFO Completed push id=27 objects=6 duration=1.955208ms36262026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036272026/11/12 00:21:09 INFO Received uploads request method=POST path=/36282026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036292026/11/12 00:21:09 INFO Vacuumed table table=pending_closures36302026/11/12 00:21:09 INFO Vacuumed table table=pending_objects36312026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads36322026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036332026/11/12 00:21:09 INFO Vacuumed table table=closures36342026/11/12 00:21:09 INFO Vacuumed table table=pending_closures36352026/11/12 00:21:09 INFO Vacuumed table table=objects36362026/11/12 00:21:09 INFO Vacuumed table table=pending_objects36372026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads36382026/11/12 00:21:09 INFO Vacuumed table table=closures36392026/11/12 00:21:09 INFO Vacuumed table table=objects36402026/11/12 00:21:09 INFO Received uploads request method=POST path=/36412026/11/12 00:21:09 INFO Received push request method=POST path=/36422026/11/12 00:21:09 INFO Received uploads request method=POST path=/36432026/11/12 00:21:09 INFO Received uploads request method=POST path=/36442026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036452026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036462026/11/12 00:21:09 INFO Received complete upload request method=POST path=/36472026/11/12 00:21:09 INFO Completed upload id=14 objects=6 duration=576µs36482026/11/12 00:21:09 INFO Vacuumed table table=pending_closures36492026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036502026/11/12 00:21:09 INFO Vacuumed table table=pending_objects36512026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads36522026/11/12 00:21:09 INFO Vacuumed table table=closures36532026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036542026/11/12 00:21:09 INFO Vacuumed table table=objects36552026/11/12 00:21:09 INFO Vacuumed table table=pending_closures36562026/11/12 00:21:09 INFO Vacuumed table table=pending_objects36572026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads36582026/11/12 00:21:09 INFO Vacuumed table table=closures36592026/11/12 00:21:09 INFO Vacuumed table table=objects36602026/11/12 00:21:09 INFO Received push request method=POST path=/36612026/11/12 00:21:09 INFO Received complete upload request method=POST path=/36622026/11/12 00:21:09 INFO Completed upload id=15 objects=2 duration=1.53775ms36632026/11/12 00:21:09 INFO Received push request method=POST path=/36642026/11/12 00:21:09 INFO Received complete push request method=POST path=/36652026/11/12 00:21:09 INFO Completed push id=21 objects=17 duration=3.637917ms36662026/11/12 00:21:09 INFO Received uploads request method=POST path=/36672026/11/12 00:21:09 INFO Received push request method=POST path=/36682026/11/12 00:21:09 INFO Received push request method=POST path=/36692026/11/12 00:21:09 INFO Received uploads request method=POST path=/36702026/11/12 00:21:09 INFO Received push request method=POST path=/36712026/11/12 00:21:09 INFO Received complete upload request method=POST path=/36722026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036732026/11/12 00:21:09 INFO Completed upload id=23 objects=18 duration=1.740417ms36742026/11/12 00:21:09 WARN Force mode enabled - objects will be deleted immediately without grace period36752026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036762026/11/12 00:21:09 INFO Vacuumed table table=pending_closures36772026/11/12 00:21:09 INFO Vacuumed table table=pending_objects36782026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads36792026/11/12 00:21:09 INFO Vacuumed table table=closures36802026/11/12 00:21:09 INFO Vacuumed table table=objects36812026/11/12 00:21:09 INFO Received complete upload request method=POST path=/36822026/11/12 00:21:09 INFO Completed upload id=21 objects=2 duration=659.75µs36832026/11/12 00:21:09 INFO Received push request method=POST path=/36842026/11/12 00:21:09 INFO Received complete push request method=POST path=/36852026/11/12 00:21:09 INFO Completed push id=19 objects=7 duration=1.232625ms36862026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=036872026/11/12 00:21:09 WARN Force mode enabled - objects will be deleted immediately without grace period36882026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036892026/11/12 00:21:09 INFO Vacuumed table table=pending_closures36902026/11/12 00:21:09 INFO Received push request method=POST path=/36912026/11/12 00:21:09 INFO Received complete push request method=POST path=/36922026/11/12 00:21:09 INFO Vacuumed table table=pending_objects36932026/11/12 00:21:09 INFO Completed push id=9 objects=4 duration=2.301083ms36942026/11/12 00:21:09 INFO Received uploads request method=POST path=/36952026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads36962026/11/12 00:21:09 INFO Received uploads request method=POST path=/36972026/11/12 00:21:09 INFO Received push request method=POST path=/36982026/11/12 00:21:09 INFO Vacuumed table table=closures36992026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=037002026/11/12 00:21:09 INFO Received push request method=POST path=/37012026/11/12 00:21:09 INFO Vacuumed table table=objects37022026/11/12 00:21:09 INFO Received push request method=POST path=/37032026/11/12 00:21:09 INFO Received complete upload request method=POST path=/37042026/11/12 00:21:09 INFO Received push request method=POST path=/37052026/11/12 00:21:09 INFO Completed upload id=7 objects=3 duration=2.600167ms37062026/11/12 00:21:09 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."37072026/11/12 00:21:09 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."37082026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=037092026/11/12 00:21:09 INFO Received uploads request method=POST path=/37102026/11/12 00:21:09 INFO Received push request method=POST path=/37112026/11/12 00:21:09 INFO Vacuumed table table=pending_closures37122026/11/12 00:21:09 INFO Received push request method=POST path=/37132026/11/12 00:21:09 INFO Vacuumed table table=pending_objects37142026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=037152026/11/12 00:21:09 INFO Vacuumed table table=multipart_uploads37162026/11/12 00:21:09 INFO Vacuumed table table=closures37172026/11/12 00:21:09 INFO Vacuumed table table=objects37182026/11/12 00:21:09 WARN Force mode enabled - objects will be deleted immediately without grace period37192026/11/12 00:21:09 INFO Received push request method=POST path=/37202026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=037212026/11/12 00:21:09 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037222026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=037232026/11/12 00:21:09 INFO Aborted multipart uploads count=0 kept=037242026/11/12 00:21:10 WARN Force mode enabled - objects will be deleted immediately without grace period37252026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37262026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037272026/11/12 00:21:10 WARN Force mode enabled - objects will be deleted immediately without grace period37282026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037292026/11/12 00:21:10 INFO Vacuumed table table=pending_objects37302026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads37312026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37322026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37332026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037342026/11/12 00:21:10 INFO Vacuumed table table=closures37352026/11/12 00:21:10 INFO Vacuumed table table=pending_objects37362026/11/12 00:21:10 INFO Vacuumed table table=pending_objects37372026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37382026/11/12 00:21:10 INFO Vacuumed table table=objects37392026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads37402026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads37412026/11/12 00:21:10 INFO Vacuumed table table=pending_objects37422026/11/12 00:21:10 INFO Vacuumed table table=closures37432026/11/12 00:21:10 INFO Vacuumed table table=closures37442026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads37452026/11/12 00:21:10 INFO Received push request method=POST path=/37462026/11/12 00:21:10 INFO Vacuumed table table=objects37472026/11/12 00:21:10 INFO Received uploads request method=POST path=/37482026/11/12 00:21:10 INFO Vacuumed table table=closures37492026/11/12 00:21:10 INFO Vacuumed table table=objects37502026/11/12 00:21:10 INFO Vacuumed table table=objects37512026/11/12 00:21:10 INFO Received complete push request method=POST path=/37522026/11/12 00:21:10 INFO Completed push id=11 objects=8 duration=4.679917ms37532026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=037542026/11/12 00:21:10 INFO Received complete upload request method=POST path=/37552026/11/12 00:21:10 WARN Force mode enabled - objects will be deleted immediately without grace period37562026/11/12 00:21:10 INFO Completed upload id=22 objects=22 duration=986.959µs37572026/11/12 00:21:10 INFO Received complete push request method=POST path=/37582026/11/12 00:21:10 INFO Completed push id=18 objects=23 duration=1.777167ms37592026/11/12 00:21:10 INFO Received uploads request method=POST path=/37602026/11/12 00:21:10 INFO Received push request method=POST path=/37612026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=037622026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37632026/11/12 00:21:10 INFO Vacuumed table table=pending_objects37642026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads37652026/11/12 00:21:10 INFO Received uploads request method=POST path=/37662026/11/12 00:21:10 INFO Received push request method=POST path=/37672026/11/12 00:21:10 INFO Vacuumed table table=closures37682026/11/12 00:21:10 INFO Vacuumed table table=objects37692026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=037702026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037712026/11/12 00:21:10 INFO Received uploads request method=POST path=/37722026/11/12 00:21:10 INFO Received uploads request method=POST path=/37732026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37742026/11/12 00:21:10 INFO Vacuumed table table=pending_objects37752026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads37762026/11/12 00:21:10 INFO Vacuumed table table=closures37772026/11/12 00:21:10 INFO Vacuumed table table=objects37782026/11/12 00:21:10 INFO Received uploads request method=POST path=/37792026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=037802026/11/12 00:21:10 INFO Received push request method=POST path=/37812026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037822026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37832026/11/12 00:21:10 INFO Vacuumed table table=pending_objects37842026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads37852026/11/12 00:21:10 INFO Vacuumed table table=closures37862026/11/12 00:21:10 INFO Vacuumed table table=objects37872026/11/12 00:21:10 INFO Received push request method=POST path=/37882026/11/12 00:21:10 INFO Received push request method=POST path=/37892026/11/12 00:21:10 INFO Received uploads request method=POST path=/37902026/11/12 00:21:10 INFO Received push request method=POST path=/37912026/11/12 00:21:10 INFO Received complete upload request method=POST path=/37922026/11/12 00:21:10 INFO Completed upload id=21 objects=7 duration=3.714709ms37932026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=037942026/11/12 00:21:10 INFO Received complete push request method=POST path=/37952026/11/12 00:21:10 INFO Received push request method=POST path=/37962026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037972026/11/12 00:21:10 INFO Completed push id=14 objects=4 duration=4.571125ms37982026/11/12 00:21:10 INFO Vacuumed table table=pending_closures37992026/11/12 00:21:10 INFO Vacuumed table table=pending_objects38002026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads38012026/11/12 00:21:10 INFO Vacuumed table table=closures38022026/11/12 00:21:10 INFO Vacuumed table table=objects38032026/11/12 00:21:10 INFO Received uploads request method=POST path=/38042026/11/12 00:21:10 INFO Received complete upload request method=POST path=/38052026/11/12 00:21:10 INFO Completed upload id=26 objects=8 duration=1.728292ms38062026/11/12 00:21:10 INFO Received complete push request method=POST path=/38072026/11/12 00:21:10 INFO Completed push id=26 objects=5 duration=1.100209ms38082026/11/12 00:21:10 INFO Received complete push request method=POST path=/38092026/11/12 00:21:10 INFO Received complete push request method=POST path=/38102026/11/12 00:21:10 INFO Received complete push request method=POST path=/38112026/11/12 00:21:10 INFO Completed push id=11 objects=9 duration=1.025ms38122026/11/12 00:21:10 INFO Received complete push request method=POST path=/38132026/11/12 00:21:10 INFO Completed push id=24 objects=24 duration=2.904792ms38142026/11/12 00:21:10 INFO Completed push id=27 objects=25 duration=6.037ms38152026/11/12 00:21:10 INFO Completed push id=13 objects=12 duration=2.428084ms38162026/11/12 00:21:10 INFO Received push request method=POST path=/38172026/11/12 00:21:10 INFO Received push request method=POST path=/38182026/11/12 00:21:10 INFO Received complete upload request method=POST path=/38192026/11/12 00:21:10 INFO Received push request method=POST path=/38202026/11/12 00:21:10 INFO Completed upload id=12 objects=26 duration=1.436083ms38212026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=038222026/11/12 00:21:10 INFO Received complete upload request method=POST path=/38232026/11/12 00:21:10 INFO Completed upload id=24 objects=14 duration=324.708µs38242026/11/12 00:21:10 INFO Received complete push request method=POST path=/38252026/11/12 00:21:10 INFO Completed push id=28 objects=14 duration=1.982041ms38262026/11/12 00:21:10 INFO Received complete upload request method=POST path=/38272026/11/12 00:21:10 INFO Completed upload id=13 objects=8 duration=1.057958ms38282026/11/12 00:21:10 INFO Received push request method=POST path=/38292026/11/12 00:21:10 INFO Received uploads request method=POST path=/38302026/11/12 00:21:10 INFO Received uploads request method=POST path=/38312026/11/12 00:21:10 INFO Received complete push request method=POST path=/38322026/11/12 00:21:10 INFO Received complete push request method=POST path=/38332026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=038342026/11/12 00:21:10 INFO Completed push id=13 objects=6 duration=5.564334ms38352026/11/12 00:21:10 INFO Completed push id=15 objects=24 duration=7.836792ms38362026/11/12 00:21:10 INFO Received push request method=POST path=/38372026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=038382026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=038392026/11/12 00:21:10 INFO Vacuumed table table=pending_closures38402026/11/12 00:21:10 INFO Received push request method=POST path=/38412026/11/12 00:21:10 INFO Received uploads request method=POST path=/38422026/11/12 00:21:10 INFO Received complete push request method=POST path=/38432026/11/12 00:21:10 WARN Force mode enabled - objects will be deleted immediately without grace period38442026/11/12 00:21:10 INFO Vacuumed table table=pending_objects38452026/11/12 00:21:10 INFO Completed push id=19 objects=3 duration=1.07675ms38462026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads38472026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038482026/11/12 00:21:10 INFO Vacuumed table table=closures38492026/11/12 00:21:10 INFO Received push request method=POST path=/38502026/11/12 00:21:10 INFO Vacuumed table table=pending_closures38512026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038522026/11/12 00:21:10 INFO Vacuumed table table=objects38532026/11/12 00:21:10 INFO Vacuumed table table=pending_closures38542026/11/12 00:21:10 INFO Vacuumed table table=pending_objects38552026/11/12 00:21:10 INFO Vacuumed table table=pending_objects38562026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads38572026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads38582026/11/12 00:21:10 INFO Received uploads request method=POST path=/38592026/11/12 00:21:10 INFO Vacuumed table table=closures38602026/11/12 00:21:10 INFO Vacuumed table table=closures38612026/11/12 00:21:10 INFO Vacuumed table table=objects38622026/11/12 00:21:10 INFO Vacuumed table table=objects38632026/11/12 00:21:10 INFO Received uploads request method=POST path=/38642026/11/12 00:21:10 INFO Received push request method=POST path=/38652026/11/12 00:21:10 INFO Received uploads request method=POST path=/38662026/11/12 00:21:10 INFO Received uploads request method=POST path=/38672026/11/12 00:21:10 INFO Received push request method=POST path=/38682026/11/12 00:21:10 INFO Received uploads request method=POST path=/38692026/11/12 00:21:10 INFO Received uploads request method=POST path=/38702026/11/12 00:21:10 INFO Received uploads request method=POST path=/38712026/11/12 00:21:10 INFO Received complete push request method=POST path=/38722026/11/12 00:21:10 INFO Completed push id=24 objects=5 duration=6.495167ms38732026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=038742026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038752026/11/12 00:21:10 INFO Vacuumed table table=pending_closures38762026/11/12 00:21:10 INFO Vacuumed table table=pending_objects38772026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads38782026/11/12 00:21:10 INFO Vacuumed table table=closures38792026/11/12 00:21:10 INFO Vacuumed table table=objects38802026/11/12 00:21:10 INFO Received push request method=POST path=/38812026/11/12 00:21:10 INFO Aborted multipart uploads count=0 kept=038822026/11/12 00:21:10 WARN Force mode enabled - objects will be deleted immediately without grace period38832026/11/12 00:21:10 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038842026/11/12 00:21:10 INFO Received uploads request method=POST path=/38852026/11/12 00:21:10 INFO Vacuumed table table=pending_closures38862026/11/12 00:21:10 INFO Vacuumed table table=pending_objects38872026/11/12 00:21:10 INFO Vacuumed table table=multipart_uploads38882026/11/12 00:21:10 INFO Vacuumed table table=closures38892026/11/12 00:21:10 INFO Vacuumed table table=objects38902026/11/12 00:21:11 INFO Received push request method=POST path=/38912026/11/12 00:21:11 INFO Received complete push request method=POST path=/38922026/11/12 00:21:11 INFO Completed push id=16 objects=8 duration=29.140625ms38932026/11/12 00:21:11 INFO Received uploads request method=POST path=/38942026/11/12 00:21:11 INFO Received uploads request method=POST path=/38952026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=038962026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=038972026/11/12 00:21:11 WARN Force mode enabled - objects will be deleted immediately without grace period38982026/11/12 00:21:11 WARN Force mode enabled - objects will be deleted immediately without grace period38992026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039002026/11/12 00:21:11 INFO Vacuumed table table=pending_closures39012026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039022026/11/12 00:21:11 INFO Vacuumed table table=pending_closures39032026/11/12 00:21:11 INFO Vacuumed table table=pending_objects39042026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads39052026/11/12 00:21:11 INFO Vacuumed table table=closures39062026/11/12 00:21:11 INFO Vacuumed table table=pending_objects39072026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads39082026/11/12 00:21:11 INFO Vacuumed table table=objects39092026/11/12 00:21:11 INFO Vacuumed table table=closures39102026/11/12 00:21:11 INFO Vacuumed table table=objects39112026/11/12 00:21:11 INFO Received uploads request method=POST path=/39122026/11/12 00:21:11 INFO Received complete upload request method=POST path=/39132026/11/12 00:21:11 INFO Completed upload id=27 objects=4 duration=1.247458ms39142026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=039152026/11/12 00:21:11 WARN Force mode enabled - objects will be deleted immediately without grace period39162026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039172026/11/12 00:21:11 INFO Vacuumed table table=pending_closures39182026/11/12 00:21:11 INFO Vacuumed table table=pending_objects39192026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads39202026/11/12 00:21:11 INFO Vacuumed table table=closures39212026/11/12 00:21:11 INFO Vacuumed table table=objects39222026/11/12 00:21:11 INFO Received uploads request method=POST path=/39232026/11/12 00:21:11 INFO Received uploads request method=POST path=/39242026/11/12 00:21:11 INFO Received uploads request method=POST path=/39252026/11/12 00:21:11 INFO Received push request method=POST path=/39262026/11/12 00:21:11 INFO Received complete push request method=POST path=/39272026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=039282026/11/12 00:21:11 INFO Received complete upload request method=POST path=/39292026/11/12 00:21:11 INFO Completed push id=13 objects=11 duration=12.345459ms39302026/11/12 00:21:11 INFO Completed upload id=14 objects=16 duration=13.400334ms39312026/11/12 00:21:11 WARN Force mode enabled - objects will be deleted immediately without grace period39322026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039332026/11/12 00:21:11 INFO Vacuumed table table=pending_closures39342026/11/12 00:21:11 INFO Received push request method=POST path=/39352026/11/12 00:21:11 INFO Received complete push request method=POST path=/39362026/11/12 00:21:11 INFO Received uploads request method=POST path=/39372026/11/12 00:21:11 INFO Vacuumed table table=pending_objects39382026/11/12 00:21:11 INFO Completed push id=9 objects=7 duration=4.051875ms39392026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads39402026/11/12 00:21:11 INFO Vacuumed table table=closures39412026/11/12 00:21:11 INFO Vacuumed table table=objects39422026/11/12 00:21:11 INFO Received uploads request method=POST path=/39432026/11/12 00:21:11 INFO Received complete push request method=POST path=/39442026/11/12 00:21:11 INFO Completed push id=11 objects=6 duration=2.574208ms39452026/11/12 00:21:11 INFO Received push request method=POST path=/39462026/11/12 00:21:11 INFO Received complete upload request method=POST path=/39472026/11/12 00:21:11 INFO Completed upload id=6 objects=6 duration=2.918541ms39482026/11/12 00:21:11 INFO Received complete upload request method=POST path=/39492026/11/12 00:21:11 INFO Completed upload id=21 objects=9 duration=1.854541ms39502026/11/12 00:21:11 INFO Received complete push request method=POST path=/39512026/11/12 00:21:11 INFO Received uploads request method=POST path=/39522026/11/12 00:21:11 INFO Completed push id=17 objects=22 duration=1.783167ms39532026/11/12 00:21:11 INFO Received uploads request method=POST path=/39542026/11/12 00:21:11 INFO Received complete upload request method=POST path=/39552026/11/12 00:21:11 INFO Received complete push request method=POST path=/39562026/11/12 00:21:11 INFO Completed upload id=25 objects=4 duration=1.181042ms39572026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=039582026/11/12 00:21:11 INFO Completed push id=31 objects=14 duration=1.460084ms39592026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=039602026/11/12 00:21:11 WARN Force mode enabled - objects will be deleted immediately without grace period39612026/11/12 00:21:11 WARN Force mode enabled - objects will be deleted immediately without grace period39622026/11/12 00:21:11 INFO Received complete push request method=POST path=/39632026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039642026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=039652026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039662026/11/12 00:21:11 INFO Vacuumed table table=pending_closures39672026/11/12 00:21:11 INFO Vacuumed table table=pending_closures39682026/11/12 00:21:11 WARN Force mode enabled - objects will be deleted immediately without grace period39692026/11/12 00:21:11 INFO Completed push id=25 objects=10 duration=4.325708ms39702026/11/12 00:21:11 INFO Vacuumed table table=pending_objects39712026/11/12 00:21:11 INFO Vacuumed table table=pending_objects39722026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039732026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads39742026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads39752026/11/12 00:21:11 INFO Vacuumed table table=pending_closures39762026/11/12 00:21:11 INFO Vacuumed table table=closures39772026/11/12 00:21:11 INFO Vacuumed table table=closures39782026/11/12 00:21:11 INFO Vacuumed table table=pending_objects39792026/11/12 00:21:11 INFO Received complete push request method=POST path=/39802026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads39812026/11/12 00:21:11 INFO Received complete upload request method=POST path=/39822026/11/12 00:21:11 INFO Vacuumed table table=objects39832026/11/12 00:21:11 INFO Completed push id=17 objects=6 duration=1.503667ms39842026/11/12 00:21:11 INFO Vacuumed table table=objects39852026/11/12 00:21:11 INFO Completed upload id=10 objects=7 duration=778.75µs39862026/11/12 00:21:11 INFO Vacuumed table table=closures39872026/11/12 00:21:11 INFO Received complete push request method=POST path=/39882026/11/12 00:21:11 INFO Received complete push request method=POST path=/39892026/11/12 00:21:11 INFO Completed push id=21 objects=26 duration=6.774458ms39902026/11/12 00:21:11 INFO Vacuumed table table=objects39912026/11/12 00:21:11 INFO Completed push id=19 objects=6 duration=10.349792ms39922026/11/12 00:21:11 INFO Received push request method=POST path=/39932026/11/12 00:21:11 INFO Received complete push request method=POST path=/39942026/11/12 00:21:11 INFO Received complete upload request method=POST path=/39952026/11/12 00:21:11 INFO Completed push id=5 objects=28 duration=1.468084ms39962026/11/12 00:21:11 INFO Completed upload id=29 objects=5 duration=3.124ms39972026/11/12 00:21:11 INFO Received complete push request method=POST path=/39982026/11/12 00:21:11 INFO Completed push id=18 objects=6 duration=2.672042ms39992026/11/12 00:21:11 INFO Received push request method=POST path=/40002026/11/12 00:21:11 INFO Received uploads request method=POST path=/40012026/11/12 00:21:11 INFO Received complete push request method=POST path=/40022026/11/12 00:21:11 INFO Completed push id=16 objects=16 duration=4.111333ms40032026/11/12 00:21:11 INFO Received push request method=POST path=/40042026/11/12 00:21:11 INFO Received uploads request method=POST path=/40052026/11/12 00:21:11 INFO Received complete push request method=POST path=/40062026/11/12 00:21:11 INFO Aborted multipart uploads count=0 kept=040072026/11/12 00:21:11 INFO Completed push id=17 objects=11 duration=9.163417ms40082026/11/12 00:21:11 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040092026/11/12 00:21:11 INFO Vacuumed table table=pending_closures40102026/11/12 00:21:11 INFO Vacuumed table table=pending_objects40112026/11/12 00:21:11 INFO Vacuumed table table=multipart_uploads40122026/11/12 00:21:11 INFO Vacuumed table table=closures40132026/11/12 00:21:11 INFO Vacuumed table table=objects40142026/11/12 00:21:11 INFO Received complete upload request method=POST path=/40152026/11/12 00:21:11 INFO Completed upload id=30 objects=2 duration=703.542µs40162026/11/12 00:21:11 INFO Received complete push request method=POST path=/40172026/11/12 00:21:11 INFO Completed push id=10 objects=2 duration=697.25µs40182026/11/12 00:21:12 INFO Received uploads request method=POST path=/40192026/11/12 00:21:12 INFO Received push request method=POST path=/40202026/11/12 00:21:12 INFO Received push request method=POST path=/40212026/11/12 00:21:12 INFO Received uploads request method=POST path=/40222026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40232026/11/12 00:21:12 INFO Completed upload id=28 objects=17 duration=1.15825ms40242026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40252026/11/12 00:21:12 INFO Received complete push request method=POST path=/40262026/11/12 00:21:12 INFO Completed upload id=23 objects=15 duration=2.349292ms40272026/11/12 00:21:12 INFO Received complete push request method=POST path=/40282026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40292026/11/12 00:21:12 INFO Completed push id=4 objects=7 duration=3.250333ms40302026/11/12 00:21:12 INFO Completed push id=31 objects=20 duration=1.669125ms40312026/11/12 00:21:12 INFO Completed upload id=17 objects=5 duration=3.4325ms40322026/11/12 00:21:12 INFO Received complete push request method=POST path=/40332026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40342026/11/12 00:21:12 INFO Completed upload id=20 objects=3 duration=4.260375ms40352026/11/12 00:21:12 INFO Completed push id=20 objects=8 duration=4.712042ms40362026/11/12 00:21:12 INFO Received push request method=POST path=/40372026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=040382026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=040392026/11/12 00:21:12 INFO Received uploads request method=POST path=/40402026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period40412026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40422026/11/12 00:21:12 INFO Completed upload id=32 objects=5 duration=2.9845ms40432026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40442026/11/12 00:21:12 INFO Received uploads request method=POST path=/40452026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=040462026/11/12 00:21:12 INFO Completed upload id=23 objects=7 duration=3.031417ms40472026/11/12 00:21:12 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."40482026/11/12 00:21:12 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."40492026/11/12 00:21:12 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."40502026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period40512026/11/12 00:21:12 INFO Received complete push request method=POST path=/40522026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040532026/11/12 00:21:12 INFO Received uploads request method=POST path=/40542026/11/12 00:21:12 INFO Received complete push request method=POST path=/40552026/11/12 00:21:12 INFO Completed push id=29 objects=2 duration=747.75µs40562026/11/12 00:21:12 INFO Completed push id=16 objects=14 duration=6.978833ms40572026/11/12 00:21:12 INFO Vacuumed table table=pending_closures40582026/11/12 00:21:12 INFO Vacuumed table table=pending_objects40592026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads40602026/11/12 00:21:12 INFO Vacuumed table table=closures40612026/11/12 00:21:12 INFO Vacuumed table table=objects40622026/11/12 00:21:12 INFO Received uploads request method=POST path=/40632026/11/12 00:21:12 INFO Received complete push request method=POST path=/40642026/11/12 00:21:12 INFO Completed push id=19 objects=3 duration=880.25µs40652026/11/12 00:21:12 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40662026/11/12 00:21:12 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40672026/11/12 00:21:12 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40682026/11/12 00:21:12 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40692026/11/12 00:21:12 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40702026/11/12 00:21:12 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40712026/11/12 00:21:12 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40722026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40732026/11/12 00:21:12 INFO Completed upload id=33 objects=3 duration=472µs40742026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40752026/11/12 00:21:12 INFO Completed upload id=18 objects=4 duration=1.025042ms40762026/11/12 00:21:12 INFO Received push request method=POST path=/40772026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40782026/11/12 00:21:12 INFO Received push request method=POST path=/40792026/11/12 00:21:12 INFO Received complete push request method=POST path=/40802026/11/12 00:21:12 INFO Completed upload id=26 objects=5 duration=885.583µs40812026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=040822026/11/12 00:21:12 INFO Completed push id=31 objects=6 duration=5.359291ms40832026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40842026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period40852026/11/12 00:21:12 INFO Received push request method=POST path=/40862026/11/12 00:21:12 INFO Completed upload id=26 objects=6 duration=1.034375ms40872026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=040882026/11/12 00:21:12 INFO Received uploads request method=POST path=/40892026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period40902026/11/12 00:21:12 INFO Received complete upload request method=POST path=/40912026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040922026/11/12 00:21:12 INFO Vacuumed table table=pending_closures40932026/11/12 00:21:12 INFO Completed upload id=19 objects=6 duration=8.987ms40942026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=040952026/11/12 00:21:12 INFO Vacuumed table table=pending_objects40962026/11/12 00:21:12 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."40972026/11/12 00:21:12 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."40982026/11/12 00:21:12 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."40992026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41002026/11/12 00:21:12 INFO Vacuumed table table=closures41012026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period41022026/11/12 00:21:12 INFO Vacuumed table table=objects41032026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041042026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41052026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41062026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41072026/11/12 00:21:12 INFO Vacuumed table table=closures41082026/11/12 00:21:12 INFO Vacuumed table table=objects41092026/11/12 00:21:12 INFO Received complete push request method=POST path=/41102026/11/12 00:21:12 INFO Received uploads request method=POST path=/41112026/11/12 00:21:12 INFO Received complete push request method=POST path=/41122026/11/12 00:21:12 INFO Completed push id=27 objects=3 duration=1.169084ms41132026/11/12 00:21:12 INFO Completed push id=32 objects=9 duration=509.167µs41142026/11/12 00:21:12 INFO Received uploads request method=POST path=/41152026/11/12 00:21:12 INFO Received uploads request method=POST path=/41162026/11/12 00:21:12 INFO Received complete push request method=POST path=/41172026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=041182026/11/12 00:21:12 INFO Completed push id=20 objects=4 duration=2.346625ms41192026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041202026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41212026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41222026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41232026/11/12 00:21:12 INFO Vacuumed table table=closures41242026/11/12 00:21:12 INFO Vacuumed table table=objects41252026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41262026/11/12 00:21:12 INFO Completed upload id=22 objects=7 duration=1.598125ms41272026/11/12 00:21:12 INFO Received complete push request method=POST path=/41282026/11/12 00:21:12 INFO Completed push id=7 objects=7 duration=1.017458ms41292026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41302026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41312026/11/12 00:21:12 INFO Received complete push request method=POST path=/41322026/11/12 00:21:12 INFO Completed upload id=24 objects=7 duration=3.861917ms41332026/11/12 00:21:12 INFO Completed upload id=29 objects=5 duration=4.029916ms41342026/11/12 00:21:12 INFO Completed push id=27 objects=11 duration=4.025042ms41352026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=041362026/11/12 00:21:12 INFO Received push request method=POST path=/41372026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period41382026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041392026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41402026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41412026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41422026/11/12 00:21:12 INFO Completed upload id=28 objects=5 duration=568.375µs41432026/11/12 00:21:12 INFO Received push request method=POST path=/41442026/11/12 00:21:12 INFO Completed upload id=33 objects=9 duration=599.042µs41452026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41462026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41472026/11/12 00:21:12 INFO Vacuumed table table=closures41482026/11/12 00:21:12 INFO Vacuumed table table=objects41492026/11/12 00:21:12 INFO Received complete push request method=POST path=/41502026/11/12 00:21:12 INFO Completed push id=12 objects=6 duration=1.465125ms41512026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41522026/11/12 00:21:12 INFO Completed upload id=7 objects=10 duration=2.044708ms41532026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41542026/11/12 00:21:12 INFO Received uploads request method=POST path=/41552026/11/12 00:21:12 INFO Completed upload id=29 objects=26 duration=2.70275ms41562026/11/12 00:21:12 INFO Received complete push request method=POST path=/41572026/11/12 00:21:12 INFO Completed push id=29 objects=17 duration=934.083µs41582026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41592026/11/12 00:21:12 INFO Completed upload id=22 objects=8 duration=4.10525ms41602026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=041612026/11/12 00:21:12 INFO Received uploads request method=POST path=/41622026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period41632026/11/12 00:21:12 INFO Received complete upload request method=POST path=/41642026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041652026/11/12 00:21:12 INFO Completed upload id=25 objects=6 duration=1.847667ms41662026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41672026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41682026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41692026/11/12 00:21:12 INFO Vacuumed table table=closures41702026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=041712026/11/12 00:21:12 INFO Vacuumed table table=objects41722026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period41732026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=041742026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041752026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period41762026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41772026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41782026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041792026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41802026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41812026/11/12 00:21:12 INFO Vacuumed table table=closures41822026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=041832026/11/12 00:21:12 INFO Aborted multipart uploads count=0 kept=041842026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41852026/11/12 00:21:12 INFO Vacuumed table table=objects41862026/11/12 00:21:12 WARN Force mode enabled - objects will be deleted immediately without grace period41872026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041882026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41892026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41902026/11/12 00:21:12 INFO Vacuumed table table=closures41912026/11/12 00:21:12 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041922026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41932026/11/12 00:21:12 INFO Vacuumed table table=objects41942026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41952026/11/12 00:21:12 INFO Vacuumed table table=pending_closures41962026/11/12 00:21:12 INFO Vacuumed table table=closures41972026/11/12 00:21:12 INFO Vacuumed table table=pending_objects41982026/11/12 00:21:12 INFO Vacuumed table table=multipart_uploads41992026/11/12 00:21:12 INFO Vacuumed table table=objects42002026/11/12 00:21:12 INFO Received complete push request method=POST path=/42012026/11/12 00:21:12 INFO Received complete push request method=POST path=/42022026/11/12 00:21:12 INFO Vacuumed table table=closures42032026/11/12 00:21:12 INFO Completed push id=23 objects=5 duration=4.701292ms42042026/11/12 00:21:12 INFO Vacuumed table table=objects42052026/11/12 00:21:12 INFO Completed push id=30 objects=8 duration=4.792625ms42062026/11/12 00:21:12 INFO Received complete push request method=POST path=/42072026/11/12 00:21:13 INFO Completed push id=24 objects=9 duration=8.283875ms42082026/11/12 00:21:13 INFO Received push request method=POST path=/42092026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42102026/11/12 00:21:13 INFO Completed upload id=31 objects=5 duration=2.213541ms42112026/11/12 00:21:13 INFO Received push request method=POST path=/42122026/11/12 00:21:13 INFO Received complete push request method=POST path=/42132026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42142026/11/12 00:21:13 INFO Completed push id=15 objects=7 duration=1.989375ms42152026/11/12 00:21:13 INFO Completed upload id=24 objects=8 duration=1.860292ms42162026/11/12 00:21:13 INFO Received complete push request method=POST path=/42172026/11/12 00:21:13 INFO Completed push id=7 objects=7 duration=1.25125ms42182026/11/12 00:21:13 INFO Received push request method=POST path=/42192026/11/12 00:21:13 INFO Received uploads request method=POST path=/42202026/11/12 00:21:13 INFO Received uploads request method=POST path=/42212026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42222026/11/12 00:21:13 INFO Received uploads request method=POST path=/42232026/11/12 00:21:13 INFO Completed upload id=28 objects=5 duration=352.791µs42242026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=042252026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period42262026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042272026/11/12 00:21:13 INFO Vacuumed table table=pending_closures42282026/11/12 00:21:13 INFO Vacuumed table table=pending_objects42292026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads42302026/11/12 00:21:13 INFO Received push request method=POST path=/42312026/11/12 00:21:13 INFO Vacuumed table table=closures42322026/11/12 00:21:13 INFO Vacuumed table table=objects42332026/11/12 00:21:13 INFO Received complete push request method=POST path=/42342026/11/12 00:21:13 INFO Completed push id=32 objects=3 duration=907.917µs42352026/11/12 00:21:13 INFO Received uploads request method=POST path=/42362026/11/12 00:21:13 INFO Received push request method=POST path=/42372026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=042382026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period42392026/11/12 00:21:13 INFO Received uploads request method=POST path=/42402026/11/12 00:21:13 INFO Received complete push request method=POST path=/42412026/11/12 00:21:13 INFO Completed push id=13 objects=22 duration=3.096208ms42422026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=042432026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period42442026/11/12 00:21:13 INFO Received complete push request method=POST path=/42452026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42462026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042472026/11/12 00:21:13 INFO Completed upload id=30 objects=5 duration=1.053084ms42482026/11/12 00:21:13 INFO Completed push id=23 objects=28 duration=4.822791ms42492026/11/12 00:21:13 INFO Vacuumed table table=pending_closures42502026/11/12 00:21:13 INFO Vacuumed table table=pending_objects42512026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads42522026/11/12 00:21:13 INFO Vacuumed table table=closures42532026/11/12 00:21:13 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."42542026/11/12 00:21:13 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."42552026/11/12 00:21:13 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."42562026/11/12 00:21:13 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."42572026/11/12 00:21:13 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42582026/11/12 00:21:13 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."42592026/11/12 00:21:13 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42602026/11/12 00:21:13 INFO Received uploads request method=POST path=/42612026/11/12 00:21:13 INFO Vacuumed table table=objects42622026/11/12 00:21:13 INFO Received uploads request method=POST path=/42632026/11/12 00:21:13 INFO Received complete push request method=POST path=/42642026/11/12 00:21:13 INFO Completed push id=21 objects=6 duration=1.335041ms42652026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42662026/11/12 00:21:13 INFO Completed upload id=23 objects=16 duration=3.376084ms42672026/11/12 00:21:13 INFO Received complete push request method=POST path=/42682026/11/12 00:21:13 INFO Completed push id=11 objects=9 duration=2.947583ms42692026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42702026/11/12 00:21:13 INFO Received push request method=POST path=/42712026/11/12 00:21:13 INFO Completed upload id=31 objects=3 duration=1.930042ms42722026/11/12 00:21:13 INFO Received complete push request method=POST path=/42732026/11/12 00:21:13 INFO Completed push id=31 objects=3 duration=3.545625ms42742026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42752026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42762026/11/12 00:21:13 INFO Completed upload id=18 objects=11 duration=2.368833ms42772026/11/12 00:21:13 INFO Completed upload id=22 objects=7 duration=536.25µs42782026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42792026/11/12 00:21:13 INFO Completed upload id=21 objects=3 duration=820.042µs42802026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=042812026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period42822026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042832026/11/12 00:21:13 INFO Vacuumed table table=pending_closures42842026/11/12 00:21:13 INFO Vacuumed table table=pending_objects42852026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42862026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads42872026/11/12 00:21:13 INFO Completed upload id=27 objects=8 duration=834.667µs42882026/11/12 00:21:13 INFO Vacuumed table table=closures42892026/11/12 00:21:13 INFO Vacuumed table table=objects42902026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42912026/11/12 00:21:13 INFO Completed upload id=22 objects=2 duration=1.952291ms42922026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42932026/11/12 00:21:13 INFO Completed upload id=11 objects=7 duration=2.735541ms42942026/11/12 00:21:13 INFO Received uploads request method=POST path=/42952026/11/12 00:21:13 INFO Received complete push request method=POST path=/42962026/11/12 00:21:13 INFO Completed push id=15 objects=24 duration=1.071ms42972026/11/12 00:21:13 INFO Received complete upload request method=POST path=/42982026/11/12 00:21:13 INFO Completed upload id=22 objects=2 duration=3.779833ms42992026/11/12 00:21:13 INFO Received complete push request method=POST path=/43002026/11/12 00:21:13 INFO Received complete push request method=POST path=/43012026/11/12 00:21:13 INFO Completed push id=1 objects=11 duration=6.055334ms43022026/11/12 00:21:13 INFO Completed push id=24 objects=4 duration=6.177167ms43032026/11/12 00:21:13 INFO Received uploads request method=POST path=/43042026/11/12 00:21:13 INFO Received push request method=POST path=/43052026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=043062026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=043072026/11/12 00:21:13 INFO Received push request method=POST path=/43082026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period43092026/11/12 00:21:13 INFO Received uploads request method=POST path=/43102026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43112026/11/12 00:21:13 INFO Completed upload id=19 objects=2 duration=1.675875ms43122026/11/12 00:21:13 INFO Received complete push request method=POST path=/43132026/11/12 00:21:13 INFO Completed push id=25 objects=9 duration=1.975625ms43142026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=043152026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=043162026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period43172026/11/12 00:21:13 INFO Vacuumed table table=pending_closures43182026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043192026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43202026/11/12 00:21:13 INFO Received complete push request method=POST path=/43212026/11/12 00:21:13 INFO Completed upload id=24 objects=7 duration=1.485917ms43222026/11/12 00:21:13 INFO Vacuumed table table=pending_closures43232026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=043242026/11/12 00:21:13 INFO Completed push id=32 objects=6 duration=1.217333ms43252026/11/12 00:21:13 INFO Vacuumed table table=pending_objects43262026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=043272026/11/12 00:21:13 INFO Vacuumed table table=pending_objects43282026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads43292026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043302026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads43312026/11/12 00:21:13 INFO Vacuumed table table=pending_closures43322026/11/12 00:21:13 INFO Vacuumed table table=closures43332026/11/12 00:21:13 INFO Vacuumed table table=pending_closures43342026/11/12 00:21:13 INFO Vacuumed table table=closures43352026/11/12 00:21:13 INFO Vacuumed table table=objects43362026/11/12 00:21:13 INFO Vacuumed table table=pending_objects43372026/11/12 00:21:13 INFO Vacuumed table table=objects43382026/11/12 00:21:13 INFO Vacuumed table table=pending_objects43392026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads43402026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads43412026/11/12 00:21:13 INFO Received uploads request method=POST path=/43422026/11/12 00:21:13 INFO Vacuumed table table=closures43432026/11/12 00:21:13 INFO Vacuumed table table=closures43442026/11/12 00:21:13 INFO Vacuumed table table=objects43452026/11/12 00:21:13 INFO Vacuumed table table=objects43462026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43472026/11/12 00:21:13 INFO Completed upload id=16 objects=3 duration=818.958µs43482026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43492026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=043502026/11/12 00:21:13 INFO Completed upload id=26 objects=3 duration=615µs43512026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period43522026/11/12 00:21:13 INFO Received complete push request method=POST path=/43532026/11/12 00:21:13 INFO Completed push id=21 objects=3 duration=1.363625ms43542026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43552026/11/12 00:21:13 INFO Completed upload id=33 objects=7 duration=688µs43562026/11/12 00:21:13 INFO Received uploads request method=POST path=/43572026/11/12 00:21:13 INFO Received complete push request method=POST path=/43582026/11/12 00:21:13 INFO Completed push id=26 objects=3 duration=633.25µs43592026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43602026/11/12 00:21:13 INFO Received push request method=POST path=/43612026/11/12 00:21:13 INFO Completed upload id=18 objects=2 duration=2.382208ms43622026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=043632026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43642026/11/12 00:21:13 INFO Completed upload id=34 objects=3 duration=610µs43652026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043662026/11/12 00:21:13 INFO Vacuumed table table=pending_closures43672026/11/12 00:21:13 INFO Vacuumed table table=pending_objects43682026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads43692026/11/12 00:21:13 INFO Vacuumed table table=closures43702026/11/12 00:21:13 INFO Vacuumed table table=objects43712026/11/12 00:21:13 INFO Received uploads request method=POST path=/43722026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43732026/11/12 00:21:13 INFO Completed upload id=27 objects=7 duration=835µs43742026/11/12 00:21:13 INFO Aborted multipart uploads count=0 kept=043752026/11/12 00:21:13 INFO Received push request method=POST path=/43762026/11/12 00:21:13 INFO Received complete upload request method=POST path=/43772026/11/12 00:21:13 WARN Force mode enabled - objects will be deleted immediately without grace period43782026/11/12 00:21:13 INFO Received push request method=POST path=/43792026/11/12 00:21:13 INFO Completed upload id=28 objects=8 duration=600.834µs43802026/11/12 00:21:13 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043812026/11/12 00:21:13 INFO Vacuumed table table=pending_closures43822026/11/12 00:21:13 INFO Vacuumed table table=pending_objects43832026/11/12 00:21:13 INFO Vacuumed table table=multipart_uploads43842026/11/12 00:21:13 INFO Vacuumed table table=closures43852026/11/12 00:21:13 INFO Vacuumed table table=objects43862026/11/12 00:21:13 INFO Received push request method=POST path=/43872026/11/12 00:21:14 INFO Received complete push request method=POST path=/43882026/11/12 00:21:14 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."43892026/11/12 00:21:14 INFO Completed push id=35 objects=11 duration=5.910291ms43902026/11/12 00:21:14 INFO Received complete upload request method=POST path=/43912026/11/12 00:21:14 INFO Received complete upload request method=POST path=/43922026/11/12 00:21:14 INFO Completed upload id=28 objects=3 duration=592.75µs43932026/11/12 00:21:14 INFO Completed upload id=25 objects=16 duration=3.447208ms43942026/11/12 00:21:14 INFO Received complete upload request method=POST path=/43952026/11/12 00:21:14 INFO Received push request method=POST path=/43962026/11/12 00:21:14 INFO Completed upload id=29 objects=10 duration=765.875µs43972026/11/12 00:21:14 INFO Received uploads request method=POST path=/43982026/11/12 00:21:14 INFO Received push request method=POST path=/43992026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=044002026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=044012026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=044022026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period44032026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period44042026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044052026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044062026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44072026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044082026/11/12 00:21:14 INFO Received complete push request method=POST path=/44092026/11/12 00:21:14 INFO Vacuumed table table=pending_closures44102026/11/12 00:21:14 INFO Received push request method=POST path=/44112026/11/12 00:21:14 INFO Completed upload id=30 objects=20 duration=469.25µs44122026/11/12 00:21:14 INFO Vacuumed table table=pending_closures44132026/11/12 00:21:14 INFO Completed push id=22 objects=3 duration=2.190542ms44142026/11/12 00:21:14 INFO Vacuumed table table=pending_objects44152026/11/12 00:21:14 INFO Vacuumed table table=pending_closures44162026/11/12 00:21:14 INFO Vacuumed table table=pending_objects44172026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads44182026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads44192026/11/12 00:21:14 INFO Received complete push request method=POST path=/44202026/11/12 00:21:14 INFO Vacuumed table table=pending_objects44212026/11/12 00:21:14 INFO Vacuumed table table=closures44222026/11/12 00:21:14 INFO Vacuumed table table=closures44232026/11/12 00:21:14 INFO Completed push id=29 objects=23 duration=1.506792ms44242026/11/12 00:21:14 INFO Vacuumed table table=objects44252026/11/12 00:21:14 INFO Vacuumed table table=objects44262026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads44272026/11/12 00:21:14 INFO Vacuumed table table=closures44282026/11/12 00:21:14 INFO Vacuumed table table=objects44292026/11/12 00:21:14 INFO Received uploads request method=POST path=/44302026/11/12 00:21:14 INFO Received complete push request method=POST path=/44312026/11/12 00:21:14 INFO Completed push id=31 objects=11 duration=684.375µs44322026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=044332026/11/12 00:21:14 INFO Received complete push request method=POST path=/44342026/11/12 00:21:14 INFO Completed push id=30 objects=6 duration=2.407875ms44352026/11/12 00:21:14 INFO Received push request method=POST path=/44362026/11/12 00:21:14 INFO Received complete push request method=POST path=/44372026/11/12 00:21:14 INFO Received push request method=POST path=/44382026/11/12 00:21:14 INFO Completed push id=15 objects=4 duration=3.032625ms44392026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044402026/11/12 00:21:14 INFO Vacuumed table table=pending_closures44412026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44422026/11/12 00:21:14 INFO Completed upload id=13 objects=7 duration=5.382125ms44432026/11/12 00:21:14 INFO Vacuumed table table=pending_objects44442026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads44452026/11/12 00:21:14 INFO Vacuumed table table=closures44462026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44472026/11/12 00:21:14 INFO Vacuumed table table=objects44482026/11/12 00:21:14 INFO Completed upload id=32 objects=2 duration=2.248625ms44492026/11/12 00:21:14 INFO Received push request method=POST path=/44502026/11/12 00:21:14 INFO Received push request method=POST path=/44512026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44522026/11/12 00:21:14 INFO Received push request method=POST path=/44532026/11/12 00:21:14 INFO Completed upload id=20 objects=4 duration=2.547791ms44542026/11/12 00:21:14 INFO Received push request method=POST path=/44552026/11/12 00:21:14 INFO Received uploads request method=POST path=/44562026/11/12 00:21:14 INFO Received complete push request method=POST path=/44572026/11/12 00:21:14 INFO Received push request method=POST path=/44582026/11/12 00:21:14 INFO Completed push id=33 objects=3 duration=703.834µs44592026/11/12 00:21:14 INFO Received complete push request method=POST path=/44602026/11/12 00:21:14 INFO Received complete push request method=POST path=/44612026/11/12 00:21:14 INFO Completed push id=33 objects=3 duration=672.208µs44622026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44632026/11/12 00:21:14 INFO Completed push id=24 objects=3 duration=1.53875ms44642026/11/12 00:21:14 INFO Completed upload id=23 objects=7 duration=1.128209ms44652026/11/12 00:21:14 INFO Received complete push request method=POST path=/44662026/11/12 00:21:14 INFO Completed push id=30 objects=4 duration=4.362417ms44672026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44682026/11/12 00:21:14 INFO Completed upload id=34 objects=3 duration=7.474042ms44692026/11/12 00:21:14 INFO Received push request method=POST path=/44702026/11/12 00:21:14 INFO Received push request method=POST path=/44712026/11/12 00:21:14 INFO Received complete push request method=POST path=/44722026/11/12 00:21:14 INFO Completed push id=34 objects=6 duration=1.654375ms44732026/11/12 00:21:14 INFO Received uploads request method=POST path=/44742026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=044752026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period44762026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044772026/11/12 00:21:14 INFO Vacuumed table table=pending_closures44782026/11/12 00:21:14 INFO Received complete push request method=POST path=/44792026/11/12 00:21:14 INFO Received uploads request method=POST path=/44802026/11/12 00:21:14 INFO Vacuumed table table=pending_objects44812026/11/12 00:21:14 INFO Completed push id=35 objects=8 duration=2.356541ms44822026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads44832026/11/12 00:21:14 INFO Vacuumed table table=closures44842026/11/12 00:21:14 INFO Vacuumed table table=objects44852026/11/12 00:21:14 INFO Received uploads request method=POST path=/44862026/11/12 00:21:14 INFO Received push request method=POST path=/44872026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=044882026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period44892026/11/12 00:21:14 INFO Received complete push request method=POST path=/44902026/11/12 00:21:14 INFO Completed push id=7 objects=15 duration=1.19425ms44912026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044922026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44932026/11/12 00:21:14 INFO Vacuumed table table=pending_closures44942026/11/12 00:21:14 INFO Completed upload id=22 objects=5 duration=3.82425ms44952026/11/12 00:21:14 INFO Received complete upload request method=POST path=/44962026/11/12 00:21:14 INFO Vacuumed table table=pending_objects44972026/11/12 00:21:14 INFO Completed upload id=36 objects=9 duration=4.680417ms44982026/11/12 00:21:14 INFO Received complete push request method=POST path=/44992026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads45002026/11/12 00:21:14 INFO Completed push id=4 objects=6 duration=6.807416ms45012026/11/12 00:21:14 INFO Vacuumed table table=closures45022026/11/12 00:21:14 INFO Received complete push request method=POST path=/45032026/11/12 00:21:14 INFO Received push request method=POST path=/45042026/11/12 00:21:14 INFO Vacuumed table table=objects45052026/11/12 00:21:14 INFO Completed push id=14 objects=3 duration=4.238042ms45062026/11/12 00:21:14 INFO Received complete push request method=POST path=/45072026/11/12 00:21:14 INFO Completed push id=28 objects=3 duration=1.765166ms45082026/11/12 00:21:14 INFO Received complete push request method=POST path=/45092026/11/12 00:21:14 INFO Completed push id=32 objects=7 duration=740.25µs45102026/11/12 00:21:14 INFO Received complete push request method=POST path=/45112026/11/12 00:21:14 INFO Received uploads request method=POST path=/45122026/11/12 00:21:14 INFO Completed push id=29 objects=15 duration=1.2565ms45132026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=045142026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period45152026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045162026/11/12 00:21:14 INFO Vacuumed table table=pending_closures45172026/11/12 00:21:14 INFO Vacuumed table table=pending_objects45182026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads45192026/11/12 00:21:14 INFO Vacuumed table table=closures45202026/11/12 00:21:14 INFO Vacuumed table table=objects45212026/11/12 00:21:14 INFO Received uploads request method=POST path=/45222026/11/12 00:21:14 INFO Received complete upload request method=POST path=/45232026/11/12 00:21:14 INFO Received push request method=POST path=/45242026/11/12 00:21:14 INFO Completed upload id=24 objects=18 duration=8.407375ms45252026/11/12 00:21:14 INFO Received push request method=POST path=/45262026/11/12 00:21:14 INFO Received push request method=POST path=/45272026/11/12 00:21:14 INFO Received complete push request method=POST path=/45282026/11/12 00:21:14 INFO Received complete push request method=POST path=/45292026/11/12 00:21:14 INFO Completed push id=30 objects=6 duration=2.773459ms45302026/11/12 00:21:14 INFO Completed push id=26 objects=4 duration=2.907041ms45312026/11/12 00:21:14 INFO Received complete upload request method=POST path=/45322026/11/12 00:21:14 INFO Received complete push request method=POST path=/45332026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=045342026/11/12 00:21:14 INFO Completed upload id=20 objects=4 duration=757.25µs45352026/11/12 00:21:14 INFO Completed push id=25 objects=2 duration=986.375µs45362026/11/12 00:21:14 INFO Received push request method=POST path=/45372026/11/12 00:21:14 INFO Received complete upload request method=POST path=/45382026/11/12 00:21:14 INFO Completed upload id=31 objects=3 duration=958.709µs45392026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=045402026/11/12 00:21:14 INFO Received complete upload request method=POST path=/45412026/11/12 00:21:14 INFO Completed upload id=28 objects=13 duration=1.194459ms45422026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period45432026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045442026/11/12 00:21:14 INFO Vacuumed table table=pending_closures45452026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=045462026/11/12 00:21:14 INFO Vacuumed table table=pending_objects45472026/11/12 00:21:14 INFO Vacuumed table table=pending_closures45482026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads45492026/11/12 00:21:14 INFO Vacuumed table table=closures45502026/11/12 00:21:14 INFO Vacuumed table table=pending_objects45512026/11/12 00:21:14 INFO Vacuumed table table=objects45522026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads45532026/11/12 00:21:14 INFO Received complete push request method=POST path=/45542026/11/12 00:21:14 INFO Vacuumed table table=closures45552026/11/12 00:21:14 INFO Completed push id=27 objects=4 duration=5.000958ms45562026/11/12 00:21:14 INFO Vacuumed table table=objects45572026/11/12 00:21:14 INFO Received complete push request method=POST path=/45582026/11/12 00:21:14 INFO Completed push id=29 objects=3 duration=6.660875ms45592026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=045602026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=045612026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period45622026/11/12 00:21:14 WARN Force mode enabled - objects will be deleted immediately without grace period45632026/11/12 00:21:14 INFO Received complete push request method=POST path=/45642026/11/12 00:21:14 INFO Completed push id=13 objects=15 duration=1.769ms45652026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045662026/11/12 00:21:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045672026/11/12 00:21:14 INFO Vacuumed table table=pending_closures45682026/11/12 00:21:14 INFO Vacuumed table table=pending_closures45692026/11/12 00:21:14 INFO Vacuumed table table=pending_objects45702026/11/12 00:21:14 INFO Vacuumed table table=pending_objects45712026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads45722026/11/12 00:21:14 INFO Vacuumed table table=multipart_uploads45732026/11/12 00:21:14 INFO Vacuumed table table=closures45742026/11/12 00:21:14 INFO Vacuumed table table=closures45752026/11/12 00:21:14 INFO Vacuumed table table=objects45762026/11/12 00:21:14 INFO Vacuumed table table=objects45772026/11/12 00:21:14 INFO Received complete upload request method=POST path=/45782026/11/12 00:21:14 INFO Completed upload id=31 objects=2 duration=712.792µs45792026/11/12 00:21:14 INFO Received uploads request method=POST path=/45802026/11/12 00:21:14 INFO Aborted multipart uploads count=0 kept=045812026/11/12 00:21:14 INFO Received uploads request method=POST path=/45822026/11/12 00:21:14 INFO Received uploads request method=POST path=/45832026/11/12 00:21:14 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45842026/11/12 00:21:14 INFO Received complete push request method=POST path=/45852026/11/12 00:21:14 INFO Received complete upload request method=POST path=/45862026/11/12 00:21:14 INFO Completed push id=33 objects=6 duration=758µs45872026/11/12 00:21:14 INFO Completed upload id=26 objects=2 duration=1.157958ms45882026/11/12 00:21:15 INFO Received complete push request method=POST path=/45892026/11/12 00:21:15 INFO Completed push id=16 objects=5 duration=2.403041ms45902026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=045912026/11/12 00:21:15 WARN Force mode enabled - objects will be deleted immediately without grace period45922026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045932026/11/12 00:21:15 INFO Vacuumed table table=pending_closures45942026/11/12 00:21:15 INFO Vacuumed table table=pending_objects45952026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads45962026/11/12 00:21:15 INFO Vacuumed table table=closures45972026/11/12 00:21:15 INFO Vacuumed table table=objects45982026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=045992026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046002026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46012026/11/12 00:21:15 INFO Received complete push request method=POST path=/46022026/11/12 00:21:15 INFO Completed push id=28 objects=26 duration=10.230292ms46032026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46042026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46052026/11/12 00:21:15 INFO Received uploads request method=POST path=/46062026/11/12 00:21:15 INFO Vacuumed table table=closures46072026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046082026/11/12 00:21:15 INFO Vacuumed table table=objects46092026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046102026/11/12 00:21:15 WARN Force mode enabled - objects will be deleted immediately without grace period46112026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046122026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046132026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046142026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46152026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46162026/11/12 00:21:15 WARN Force mode enabled - objects will be deleted immediately without grace period46172026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46182026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46192026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46202026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46212026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046222026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46232026/11/12 00:21:15 INFO Vacuumed table table=closures46242026/11/12 00:21:15 INFO Vacuumed table table=closures46252026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46262026/11/12 00:21:15 INFO Vacuumed table table=objects46272026/11/12 00:21:15 INFO Vacuumed table table=objects46282026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46292026/11/12 00:21:15 INFO Vacuumed table table=closures46302026/11/12 00:21:15 INFO Received complete push request method=POST path=/46312026/11/12 00:21:15 INFO Vacuumed table table=objects46322026/11/12 00:21:15 INFO Completed push id=30 objects=3 duration=4.065417ms46332026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046342026/11/12 00:21:15 INFO Received complete push request method=POST path=/46352026/11/12 00:21:15 INFO Completed push id=20 objects=3 duration=3.709416ms46362026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=046372026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46382026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46392026/11/12 00:21:15 INFO Received complete push request method=POST path=/46402026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46412026/11/12 00:21:15 INFO Completed push id=32 objects=19 duration=1.700208ms46422026/11/12 00:21:15 INFO Vacuumed table table=closures46432026/11/12 00:21:15 INFO Vacuumed table table=objects46442026/11/12 00:21:15 INFO Received complete upload request method=POST path=/46452026/11/12 00:21:15 INFO Completed upload id=33 objects=5 duration=5.713375ms46462026/11/12 00:21:15 INFO Received push request method=POST path=/46472026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046482026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046492026/11/12 00:21:15 WARN Force mode enabled - objects will be deleted immediately without grace period46502026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046512026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046522026/11/12 00:21:15 INFO Received complete upload request method=POST path=/46532026/11/12 00:21:15 INFO Completed upload id=17 objects=25 duration=4.508042ms46542026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46552026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46562026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46572026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46582026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46592026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46602026/11/12 00:21:15 INFO Vacuumed table table=closures46612026/11/12 00:21:15 INFO Vacuumed table table=objects46622026/11/12 00:21:15 INFO Vacuumed table table=closures46632026/11/12 00:21:15 INFO Vacuumed table table=objects46642026/11/12 00:21:15 INFO Received complete push request method=POST path=/46652026/11/12 00:21:15 INFO Received push request method=POST path=/46662026/11/12 00:21:15 INFO Completed push id=19 objects=14 duration=1.949916ms46672026/11/12 00:21:15 INFO Received uploads request method=POST path=/46682026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046692026/11/12 00:21:15 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."46702026/11/12 00:21:15 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."46712026/11/12 00:21:15 INFO Received push request method=POST path=/46722026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046732026/11/12 00:21:15 INFO Received complete upload request method=POST path=/46742026/11/12 00:21:15 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46752026/11/12 00:21:15 INFO Completed upload id=21 objects=2 duration=974.083µs46762026/11/12 00:21:15 INFO Received uploads request method=POST path=/46772026/11/12 00:21:15 WARN Force mode enabled - objects will be deleted immediately without grace period46782026/11/12 00:21:15 INFO Received push request method=POST path=/46792026/11/12 00:21:15 INFO Received push request method=POST path=/46802026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046812026/11/12 00:21:15 WARN Force mode enabled - objects will be deleted immediately without grace period46822026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046832026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46842026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46852026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46862026/11/12 00:21:15 INFO Vacuumed table table=closures46872026/11/12 00:21:15 INFO Received push request method=POST path=/46882026/11/12 00:21:15 INFO Vacuumed table table=objects46892026/11/12 00:21:15 INFO Received push request method=POST path=/46902026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=046912026/11/12 00:21:15 INFO Received push request method=POST path=/46922026/11/12 00:21:15 INFO Received complete upload request method=POST path=/46932026/11/12 00:21:15 INFO Completed upload id=22 objects=6 duration=2.103667ms46942026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046952026/11/12 00:21:15 INFO Vacuumed table table=pending_closures46962026/11/12 00:21:15 INFO Vacuumed table table=pending_objects46972026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads46982026/11/12 00:21:15 INFO Vacuumed table table=closures46992026/11/12 00:21:15 INFO Vacuumed table table=objects47002026/11/12 00:21:15 INFO Received complete upload request method=POST path=/47012026/11/12 00:21:15 INFO Received complete push request method=POST path=/47022026/11/12 00:21:15 INFO Completed upload id=21 objects=25 duration=1.300208ms47032026/11/12 00:21:15 INFO Received uploads request method=POST path=/47042026/11/12 00:21:15 INFO Completed push id=28 objects=11 duration=2.989292ms47052026/11/12 00:21:15 INFO Received uploads request method=POST path=/47062026/11/12 00:21:15 INFO Aborted multipart uploads count=0 kept=047072026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047082026/11/12 00:21:15 INFO Vacuumed table table=pending_closures47092026/11/12 00:21:15 INFO Vacuumed table table=pending_objects47102026/11/12 00:21:15 INFO Vacuumed table table=multipart_uploads47112026/11/12 00:21:15 INFO Vacuumed table table=closures47122026/11/12 00:21:15 INFO Vacuumed table table=objects47132026/11/12 00:21:15 INFO Received uploads request method=POST path=/47142026/11/12 00:21:15 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=047152026/11/12 00:21:15 INFO Vacuumed table table=pending_closures47162026/11/12 00:21:16 INFO Received push request method=POST path=/47172026/11/12 00:21:16 INFO Vacuumed table table=pending_objects47182026/11/12 00:21:16 INFO Vacuumed table table=multipart_uploads47192026/11/12 00:21:16 INFO Vacuumed table table=closures47202026/11/12 00:21:16 INFO Vacuumed table table=objects47212026/11/12 00:21:16 INFO Received push request method=POST path=/47222026/11/12 00:21:16 INFO Received complete push request method=POST path=/47232026/11/12 00:21:16 INFO Completed push id=7 objects=24 duration=18.531959ms47242026/11/12 00:21:16 INFO Received push request method=POST path=/47252026/11/12 00:21:16 INFO Received push request method=POST path=/47262026/11/12 00:21:16 INFO Received push request method=POST path=/47272026/11/12 00:21:16 INFO Received uploads request method=POST path=/47282026/11/12 00:21:16 INFO Received complete push request method=POST path=/47292026/11/12 00:21:16 INFO Received uploads request method=POST path=/47302026/11/12 00:21:16 INFO Completed push id=29 objects=24 duration=3.290042ms47312026/11/12 00:21:16 INFO Received uploads request method=POST path=/47322026/11/12 00:21:16 INFO Received uploads request method=POST path=/47332026/11/12 00:21:16 INFO Received uploads request method=POST path=/47342026/11/12 00:21:16 INFO Received complete upload request method=POST path=/47352026/11/12 00:21:16 INFO Completed upload id=12 objects=6 duration=1.008209ms47362026/11/12 00:21:16 INFO Aborted multipart uploads count=0 kept=047372026/11/12 00:21:16 INFO Received push request method=POST path=/47382026/11/12 00:21:16 WARN Force mode enabled - objects will be deleted immediately without grace period47392026/11/12 00:21:16 INFO Received uploads request method=POST path=/47402026/11/12 00:21:16 INFO Aborted multipart uploads count=0 kept=047412026/11/12 00:21:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047422026/11/12 00:21:16 INFO Vacuumed table table=pending_closures47432026/11/12 00:21:16 INFO Vacuumed table table=pending_objects47442026/11/12 00:21:16 INFO Vacuumed table table=multipart_uploads47452026/11/12 00:21:16 INFO Vacuumed table table=closures47462026/11/12 00:21:16 INFO Received uploads request method=POST path=/47472026/11/12 00:21:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=047482026/11/12 00:21:16 INFO Vacuumed table table=objects47492026/11/12 00:21:16 INFO Vacuumed table table=pending_closures47502026/11/12 00:21:16 INFO Received complete push request method=POST path=/47512026/11/12 00:21:16 INFO Vacuumed table table=pending_objects47522026/11/12 00:21:16 INFO Completed push id=17 objects=3 duration=2.17375ms47532026/11/12 00:21:16 INFO Vacuumed table table=multipart_uploads47542026/11/12 00:21:16 INFO Vacuumed table table=closures47552026/11/12 00:21:16 INFO Vacuumed table table=objects47562026/11/12 00:21:16 INFO Received uploads request method=POST path=/47572026/11/12 00:21:16 INFO Received push request method=POST path=/47582026/11/12 00:21:16 INFO Aborted multipart uploads count=0 kept=047592026/11/12 00:21:16 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047602026/11/12 00:21:16 INFO Vacuumed table table=pending_closures47612026/11/12 00:21:16 INFO Vacuumed table table=pending_objects47622026/11/12 00:21:16 INFO Vacuumed table table=multipart_uploads47632026/11/12 00:21:16 INFO Vacuumed table table=closures47642026/11/12 00:21:16 INFO Vacuumed table table=objects47652026/11/12 00:21:16 INFO Received uploads request method=POST path=/47662026/11/12 00:21:16 INFO Received uploads request method=POST path=/47672026/11/12 00:21:16 INFO Aborted multipart uploads count=0 kept=047682026/11/12 00:21:16 WARN Force mode enabled - objects will be deleted immediately without grace period47692026/11/12 00:21:16 INFO Received complete push request method=POST path=/47702026/11/12 00:21:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047712026/11/12 00:21:16 INFO Completed push id=23 objects=5 duration=752.875µs47722026/11/12 00:21:16 INFO Vacuumed table table=pending_closures47732026/11/12 00:21:16 INFO Vacuumed table table=pending_objects47742026/11/12 00:21:16 INFO Vacuumed table table=multipart_uploads47752026/11/12 00:21:16 INFO Vacuumed table table=closures47762026/11/12 00:21:16 INFO Vacuumed table table=objects47772026/11/12 00:21:16 INFO Received complete push request method=POST path=/47782026/11/12 00:21:16 INFO Received uploads request method=POST path=/47792026/11/12 00:21:16 INFO Completed push id=13 objects=2 duration=11.596166ms47802026/11/12 00:21:16 INFO Received complete upload request method=POST path=/47812026/11/12 00:21:16 INFO Completed upload id=24 objects=3 duration=2.437792ms47822026/11/12 00:21:16 INFO Received complete upload request method=POST path=/47832026/11/12 00:21:16 INFO Completed upload id=19 objects=2 duration=2.886584ms47842026/11/12 00:21:16 INFO Received push request method=POST path=/47852026/11/12 00:21:16 INFO Received complete upload request method=POST path=/47862026/11/12 00:21:16 INFO Completed upload id=27 objects=13 duration=10.433958ms47872026/11/12 00:21:16 INFO Received complete push request method=POST path=/47882026/11/12 00:21:16 INFO Received uploads request method=POST path=/47892026/11/12 00:21:16 INFO Completed push id=25 objects=4 duration=2.269958ms47902026/11/12 00:21:16 INFO Aborted multipart uploads count=0 kept=047912026/11/12 00:21:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047922026/11/12 00:21:16 INFO Vacuumed table table=pending_closures47932026/11/12 00:21:16 INFO Vacuumed table table=pending_objects47942026/11/12 00:21:16 INFO Vacuumed table table=multipart_uploads47952026/11/12 00:21:16 INFO Vacuumed table table=closures47962026/11/12 00:21:16 INFO Vacuumed table table=objects47972026/11/12 00:21:16 INFO Received complete push request method=POST path=/47982026/11/12 00:21:16 INFO Completed push id=33 objects=7 duration=2.086042ms47992026/11/12 00:21:16 INFO Received uploads request method=POST path=/48002026/11/12 00:21:16 INFO Received complete upload request method=POST path=/48012026/11/12 00:21:16 INFO Completed upload id=26 objects=3 duration=2.011792ms48022026/11/12 00:21:16 INFO Received push request method=POST path=/48032026/11/12 00:21:16 INFO Aborted multipart uploads count=0 kept=048042026/11/12 00:21:16 WARN Force mode enabled - objects will be deleted immediately without grace period48052026/11/12 00:21:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048062026/11/12 00:21:16 INFO Vacuumed table table=pending_closures48072026/11/12 00:21:16 INFO Vacuumed table table=pending_objects48082026/11/12 00:21:16 INFO Vacuumed table table=multipart_uploads48092026/11/12 00:21:16 INFO Vacuumed table table=closures48102026/11/12 00:21:16 INFO Received push request method=POST path=/48112026/11/12 00:21:16 INFO Vacuumed table table=objects48122026/11/12 00:21:16 INFO Received uploads request method=POST path=/48132026/11/12 00:21:16 INFO Received uploads request method=POST path=/48142026/11/12 00:21:16 INFO Received complete upload request method=POST path=/48152026/11/12 00:21:17 INFO Completed upload id=29 objects=4 duration=4.425833ms48162026/11/12 00:21:17 INFO Received uploads request method=POST path=/48172026/11/12 00:21:17 INFO Received uploads request method=POST path=/48182026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48192026/11/12 00:21:17 INFO Completed upload id=13 objects=14 duration=1.988833ms48202026/11/12 00:21:17 INFO Aborted multipart uploads count=0 kept=048212026/11/12 00:21:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048222026/11/12 00:21:17 INFO Vacuumed table table=pending_closures48232026/11/12 00:21:17 INFO Vacuumed table table=pending_objects48242026/11/12 00:21:17 INFO Received complete push request method=POST path=/48252026/11/12 00:21:17 INFO Vacuumed table table=multipart_uploads48262026/11/12 00:21:17 INFO Completed push id=26 objects=23 duration=2.19175ms48272026/11/12 00:21:17 INFO Received complete push request method=POST path=/48282026/11/12 00:21:17 INFO Vacuumed table table=closures48292026/11/12 00:21:17 INFO Completed push id=11 objects=6 duration=43.849875ms48302026/11/12 00:21:17 INFO Vacuumed table table=objects48312026/11/12 00:21:17 INFO Received complete push request method=POST path=/48322026/11/12 00:21:17 INFO Completed push id=16 objects=7 duration=1.390875ms48332026/11/12 00:21:17 INFO Received complete push request method=POST path=/48342026/11/12 00:21:17 INFO Received complete push request method=POST path=/48352026/11/12 00:21:17 INFO Completed push id=20 objects=13 duration=568.708µs48362026/11/12 00:21:17 INFO Completed push id=31 objects=6 duration=10.15625ms48372026/11/12 00:21:17 INFO Received complete push request method=POST path=/48382026/11/12 00:21:17 INFO Received uploads request method=POST path=/48392026/11/12 00:21:17 INFO Completed push id=22 objects=4 duration=997.458µs48402026/11/12 00:21:17 INFO Received complete push request method=POST path=/48412026/11/12 00:21:17 INFO Completed push id=23 objects=6 duration=600.958µs48422026/11/12 00:21:17 INFO Received complete push request method=POST path=/48432026/11/12 00:21:17 INFO Completed push id=9 objects=9 duration=1.8695ms48442026/11/12 00:21:17 INFO Received complete push request method=POST path=/48452026/11/12 00:21:17 INFO Completed push id=24 objects=6 duration=493.959µs48462026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48472026/11/12 00:21:17 INFO Completed upload id=23 objects=7 duration=479.417µs48482026/11/12 00:21:17 INFO Received complete push request method=POST path=/48492026/11/12 00:21:17 INFO Completed push id=27 objects=15 duration=700.334µs48502026/11/12 00:21:17 INFO Received complete push request method=POST path=/48512026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48522026/11/12 00:21:17 INFO Completed push id=24 objects=5 duration=2.136125ms48532026/11/12 00:21:17 INFO Completed upload id=21 objects=8 duration=9.143625ms48542026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48552026/11/12 00:21:17 INFO Received complete push request method=POST path=/48562026/11/12 00:21:17 INFO Completed upload id=25 objects=2 duration=3.042666ms48572026/11/12 00:21:17 INFO Completed push id=22 objects=6 duration=636.125µs48582026/11/12 00:21:17 INFO Received complete push request method=POST path=/48592026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48602026/11/12 00:21:17 INFO Completed push id=24 objects=8 duration=3.14175ms48612026/11/12 00:21:17 INFO Completed upload id=26 objects=20 duration=479.875µs48622026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48632026/11/12 00:21:17 INFO Completed upload id=25 objects=9 duration=460.375µs48642026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48652026/11/12 00:21:17 INFO Completed upload id=27 objects=3 duration=695.791µs48662026/11/12 00:21:17 INFO Received complete push request method=POST path=/48672026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48682026/11/12 00:21:17 INFO Completed push id=25 objects=7 duration=613µs48692026/11/12 00:21:17 INFO Completed upload id=28 objects=6 duration=3.806958ms48702026/11/12 00:21:17 INFO Received complete push request method=POST path=/48712026/11/12 00:21:17 INFO Completed push id=30 objects=4 duration=706.25µs48722026/11/12 00:21:17 INFO Received complete push request method=POST path=/48732026/11/12 00:21:17 INFO Received complete push request method=POST path=/48742026/11/12 00:21:17 INFO Completed push id=28 objects=7 duration=483.583µs48752026/11/12 00:21:17 INFO Completed push id=31 objects=7 duration=669.125µs48762026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48772026/11/12 00:21:17 INFO Completed upload id=32 objects=5 duration=605.125µs48782026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48792026/11/12 00:21:17 INFO Completed upload id=30 objects=6 duration=415µs48802026/11/12 00:21:17 INFO Received complete push request method=POST path=/48812026/11/12 00:21:17 INFO Completed push id=31 objects=5 duration=348.584µs48822026/11/12 00:21:17 INFO Received complete push request method=POST path=/48832026/11/12 00:21:17 INFO Completed push id=28 objects=19 duration=476.083µs48842026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48852026/11/12 00:21:17 INFO Completed upload id=26 objects=24 duration=498.375µs48862026/11/12 00:21:17 INFO Received complete push request method=POST path=/48872026/11/12 00:21:17 INFO Completed push id=32 objects=26 duration=592.25µs48882026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48892026/11/12 00:21:17 INFO Completed upload id=33 objects=2 duration=576.541µs48902026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48912026/11/12 00:21:17 INFO Completed upload id=34 objects=5 duration=472.042µs48922026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48932026/11/12 00:21:17 INFO Completed upload id=35 objects=11 duration=383.25µs48942026/11/12 00:21:17 INFO Received complete upload request method=POST path=/48952026/11/12 00:21:17 INFO Completed upload id=36 objects=3 duration=521.916µs48962026/11/12 00:21:17 INFO Received complete push request method=POST path=/48972026/11/12 00:21:17 INFO Aborted multipart uploads count=0 kept=048982026/11/12 00:21:17 INFO Completed push id=29 objects=10 duration=2.145833ms48992026/11/12 00:21:17 WARN Force mode enabled - objects will be deleted immediately without grace period49002026/11/12 00:21:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049012026/11/12 00:21:17 INFO Received complete upload request method=POST path=/49022026/11/12 00:21:17 INFO Completed upload id=34 objects=24 duration=393.041µs49032026/11/12 00:21:17 INFO Received complete push request method=POST path=/49042026/11/12 00:21:17 INFO Vacuumed table table=pending_closures49052026/11/12 00:21:17 INFO Completed push id=1 objects=9 duration=2.726334ms49062026/11/12 00:21:17 INFO Aborted multipart uploads count=0 kept=049072026/11/12 00:21:17 WARN Force mode enabled - objects will be deleted immediately without grace period49082026/11/12 00:21:17 INFO Vacuumed table table=pending_objects49092026/11/12 00:21:17 INFO Vacuumed table table=multipart_uploads49102026/11/12 00:21:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049112026/11/12 00:21:17 INFO Vacuumed table table=closures49122026/11/12 00:21:17 INFO Vacuumed table table=pending_closures49132026/11/12 00:21:17 INFO Vacuumed table table=objects49142026/11/12 00:21:17 INFO Vacuumed table table=pending_objects49152026/11/12 00:21:17 INFO Vacuumed table table=multipart_uploads49162026/11/12 00:21:17 INFO Vacuumed table table=closures49172026/11/12 00:21:17 INFO Vacuumed table table=objects49182026/11/12 00:21:17 INFO Received complete upload request method=POST path=/49192026/11/12 00:21:17 INFO Completed upload id=30 objects=6 duration=312.5µs49202026/11/12 00:21:17 INFO Received complete upload request method=POST path=/49212026/11/12 00:21:17 INFO Completed upload id=31 objects=5 duration=494.459µs49222026/11/12 00:21:17 INFO Received complete upload request method=POST path=/49232026/11/12 00:21:17 INFO Received complete upload request method=POST path=/49242026/11/12 00:21:17 INFO Completed upload id=34 objects=3 duration=505.5µs49252026/11/12 00:21:17 INFO Completed upload id=27 objects=26 duration=304.291µs49262026/11/12 00:21:17 INFO Received complete push request method=POST path=/49272026/11/12 00:21:17 INFO Completed push id=28 objects=6 duration=314.292µs49282026/11/12 00:21:17 INFO Aborted multipart uploads count=0 kept=049292026/11/12 00:21:17 WARN Force mode enabled - objects will be deleted immediately without grace period49302026/11/12 00:21:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049312026/11/12 00:21:17 INFO Vacuumed table table=pending_closures49322026/11/12 00:21:17 INFO Vacuumed table table=pending_objects49332026/11/12 00:21:17 INFO Vacuumed table table=multipart_uploads49342026/11/12 00:21:17 INFO Received complete push request method=POST path=/49352026/11/12 00:21:17 INFO Vacuumed table table=closures49362026/11/12 00:21:17 INFO Completed push id=29 objects=6 duration=1.169792ms49372026/11/12 00:21:17 INFO Vacuumed table table=objects49382026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49392026/11/12 00:21:18 INFO Completed upload id=30 objects=8 duration=359.458µs49402026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49412026/11/12 00:21:18 INFO Completed upload id=2 objects=7 duration=740.625µs49422026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49432026/11/12 00:21:18 INFO Completed upload id=31 objects=6 duration=370.333µs49442026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49452026/11/12 00:21:18 INFO Completed upload id=32 objects=4 duration=641.417µs49462026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49472026/11/12 00:21:18 INFO Completed upload id=33 objects=10 duration=570.667µs49482026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49492026/11/12 00:21:18 INFO Completed upload id=8 objects=6 duration=432.541µs49502026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49512026/11/12 00:21:18 INFO Completed upload id=34 objects=23 duration=373.792µs49522026/11/12 00:21:18 INFO Received complete push request method=POST path=/49532026/11/12 00:21:18 INFO Completed push id=35 objects=6 duration=360.583µs49542026/11/12 00:21:18 INFO Aborted multipart uploads count=0 kept=049552026/11/12 00:21:18 WARN Force mode enabled - objects will be deleted immediately without grace period49562026/11/12 00:21:18 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049572026/11/12 00:21:18 INFO Vacuumed table table=pending_closures49582026/11/12 00:21:18 INFO Vacuumed table table=pending_objects49592026/11/12 00:21:18 INFO Vacuumed table table=multipart_uploads49602026/11/12 00:21:18 INFO Vacuumed table table=closures49612026/11/12 00:21:18 INFO Vacuumed table table=objects49622026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49632026/11/12 00:21:18 INFO Completed upload id=12 objects=9 duration=450.291µs49642026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49652026/11/12 00:21:18 INFO Completed upload id=18 objects=22 duration=802.25µs49662026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49672026/11/12 00:21:18 INFO Completed upload id=21 objects=8 duration=679.666µs49682026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49692026/11/12 00:21:18 INFO Completed upload id=22 objects=7 duration=733.209µs49702026/11/12 00:21:18 INFO Received complete push request method=POST path=/49712026/11/12 00:21:18 INFO Completed push id=24 objects=5 duration=718.167µs49722026/11/12 00:21:18 INFO Received complete upload request method=POST path=/49732026/11/12 00:21:18 INFO Completed upload id=25 objects=8 duration=477.375µs49742026/11/12 00:21:18 INFO Received complete push request method=POST path=/49752026/11/12 00:21:18 INFO Completed push id=26 objects=4 duration=725.375µs49762026/11/12 00:21:18 INFO Received complete push request method=POST path=/49772026/11/12 00:21:18 INFO Completed push id=30 objects=2 duration=674.666µs49782026/11/12 00:21:18 INFO Received complete push request method=POST path=/49792026/11/12 00:21:19 INFO Completed push id=32 objects=7 duration=1.022375ms49802026/11/12 00:21:19 INFO Received complete upload request method=POST path=/49812026/11/12 00:21:19 INFO Completed upload id=33 objects=6 duration=557.75µs49822026/11/12 00:21:19 INFO Received complete upload request method=POST path=/49832026/11/12 00:21:19 INFO Completed upload id=34 objects=3 duration=638.666µs49842026/11/12 00:21:19 INFO Aborted multipart uploads count=0 kept=049852026/11/12 00:21:19 WARN Force mode enabled - objects will be deleted immediately without grace period49862026/11/12 00:21:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049872026/11/12 00:21:19 INFO Vacuumed table table=pending_closures49882026/11/12 00:21:19 INFO Vacuumed table table=pending_objects49892026/11/12 00:21:19 INFO Vacuumed table table=multipart_uploads49902026/11/12 00:21:19 INFO Vacuumed table table=closures49912026/11/12 00:21:19 INFO Vacuumed table table=objects4992--- PASS: FuzzPushGC (0.00s)4993 --- PASS: FuzzPushGC/seed#6 (13.59s)4994 --- PASS: FuzzPushGC/seed#1 (14.02s)4995 --- PASS: FuzzPushGC/seed#0 (14.04s)4996 --- PASS: FuzzPushGC/seed#3 (14.50s)4997 --- PASS: FuzzPushGC/seed#8 (15.19s)4998 --- PASS: FuzzPushGC/seed#5 (15.53s)4999 --- PASS: FuzzPushGC/seed#4 (15.78s)5000 --- PASS: FuzzPushGC/seed#15 (16.00s)5001 --- PASS: FuzzPushGC/seed#14 (16.25s)5002 --- PASS: FuzzPushGC/seed#7 (16.40s)5003 --- PASS: FuzzPushGC/seed#2 (17.46s)5004 --- PASS: FuzzPushGC/seed#11 (18.94s)5005 --- PASS: FuzzPushGC/seed#9 (18.95s)5006 --- PASS: FuzzPushGC/seed#13 (19.00s)5007 --- PASS: FuzzPushGC/seed#12 (19.34s)5008 --- PASS: FuzzPushGC/seed#10 (20.22s)5009PASS5010{"timestamp":"2026-11-12T00:21:19.196955Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:64598","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}5011{"timestamp":"2026-11-12T00:21:19.197105Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:64643","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(4)"}50122026-11-12 00:21:23.547 UTC [55856] LOG: received smart shutdown request50132026-11-12 00:21:23.553 UTC [55856] LOG: background worker "logical replication launcher" (PID 55884) exited with exit code 150142026-11-12 00:21:29.681 UTC [55875] LOG: checkpoint complete: wrote 3299 buffers (20.1%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=40.038 s, sync=4.516 s, total=45.966 s; sync files=42371, longest=0.014 s, average=0.001 s; distance=533519 kB, estimate=533519 kB; lsn=0/2784DB78, redo lsn=0/22090DC850152026-11-12 00:21:29.691 UTC [55875] LOG: shutting down50162026-11-12 00:21:29.697 UTC [55875] LOG: checkpoint starting: shutdown immediate50172026/11/12 00:21:33 INFO killed process group pgid=5585650182026/11/12 00:21:33 ERROR failed to wait for process error="signal: killed"50192026-11-12 00:21:33.727 UTC [55875] PANIC: could not fsync file "pg_xact/0000": No such file or directory5020Running OIDC tests...5021=== RUN TestAudienceForIssuer5022=== PAUSE TestAudienceForIssuer5023=== RUN TestGlobMatch5024=== PAUSE TestGlobMatch5025=== RUN TestValidateToken_ValidToken5026=== PAUSE TestValidateToken_ValidToken5027=== RUN TestValidateToken_WrongAudience5028=== PAUSE TestValidateToken_WrongAudience5029=== RUN TestValidateToken_Expired5030=== PAUSE TestValidateToken_Expired5031=== RUN TestValidateToken_BoundClaimsMismatch5032=== PAUSE TestValidateToken_BoundClaimsMismatch5033=== RUN TestValidateToken_BoundSubjectMismatch5034=== PAUSE TestValidateToken_BoundSubjectMismatch5035=== RUN TestValidateToken_MultipleProviders5036=== PAUSE TestValidateToken_MultipleProviders5037=== RUN TestValidateToken_NoMatchingProvider5038=== PAUSE TestValidateToken_NoMatchingProvider5039=== RUN TestValidateToken_KubernetesServiceAccount5040=== PAUSE TestValidateToken_KubernetesServiceAccount5041=== RUN TestNewValidator_KubernetesRequiresCA5042=== PAUSE TestNewValidator_KubernetesRequiresCA5043=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5044=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5045=== RUN TestPins_ReservedForMatchingRule5046=== PAUSE TestPins_ReservedForMatchingRule5047=== RUN TestPins_TopLevelShorthand5048=== PAUSE TestPins_TopLevelShorthand5049=== RUN TestPins_ConfigValidation5050=== PAUSE TestPins_ConfigValidation5051=== RUN TestScopes_LegacyProviderDefaultsToWrite5052=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5053=== RUN TestScopes_Rules5054=== PAUSE TestScopes_Rules5055=== RUN TestScopes_ConfigValidation5056=== PAUSE TestScopes_ConfigValidation5057=== CONT TestAudienceForIssuer5058--- PASS: TestAudienceForIssuer (0.00s)5059=== CONT TestValidateToken_ValidToken5060=== CONT TestValidateToken_NoMatchingProvider5061=== CONT TestValidateToken_WrongAudience5062=== CONT TestValidateToken_KubernetesServiceAccount5063=== CONT TestScopes_ConfigValidation5064=== CONT TestScopes_Rules5065=== CONT TestScopes_LegacyProviderDefaultsToWrite5066--- PASS: TestScopes_ConfigValidation (0.00s)5067=== CONT TestPins_ConfigValidation5068=== CONT TestPins_TopLevelShorthand5069--- PASS: TestPins_ConfigValidation (0.00s)5070=== CONT TestPins_ReservedForMatchingRule5071=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5072=== CONT TestNewValidator_KubernetesRequiresCA5073=== CONT TestValidateToken_MultipleProviders5074=== CONT TestGlobMatch5075=== RUN TestGlobMatch/foo_foo5076=== PAUSE TestGlobMatch/foo_foo5077=== RUN TestGlobMatch/foo_bar5078=== PAUSE TestGlobMatch/foo_bar5079=== RUN TestGlobMatch/*_5080=== PAUSE TestGlobMatch/*_5081=== RUN TestGlobMatch/*_anything5082=== PAUSE TestGlobMatch/*_anything5083=== RUN TestGlobMatch/foo*_foo5084=== PAUSE TestGlobMatch/foo*_foo5085=== RUN TestGlobMatch/foo*_foobar5086=== PAUSE TestGlobMatch/foo*_foobar5087=== RUN TestGlobMatch/foo*_bar5088=== PAUSE TestGlobMatch/foo*_bar5089=== RUN TestGlobMatch/*bar_bar5090=== PAUSE TestGlobMatch/*bar_bar5091=== RUN TestGlobMatch/*bar_foobar5092=== PAUSE TestGlobMatch/*bar_foobar5093=== RUN TestGlobMatch/*bar_foo5094=== PAUSE TestGlobMatch/*bar_foo5095=== RUN TestGlobMatch/foo*bar_foobar5096=== PAUSE TestGlobMatch/foo*bar_foobar5097=== RUN TestGlobMatch/foo*bar_foo123bar5098=== PAUSE TestGlobMatch/foo*bar_foo123bar5099=== RUN TestGlobMatch/foo*bar_foobarbaz5100=== PAUSE TestGlobMatch/foo*bar_foobarbaz5101=== RUN TestGlobMatch/*/*_foo/bar5102=== PAUSE TestGlobMatch/*/*_foo/bar5103=== RUN TestGlobMatch/*/*_foo5104=== PAUSE TestGlobMatch/*/*_foo5105=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5106=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5107=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05108=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05109=== RUN TestGlobMatch/refs/*/main_refs/heads/main5110=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5111=== RUN TestGlobMatch/fo?_foo5112=== PAUSE TestGlobMatch/fo?_foo5113=== RUN TestGlobMatch/fo?_fo5114=== PAUSE TestGlobMatch/fo?_fo5115=== RUN TestGlobMatch/fo?_fooo5116=== PAUSE TestGlobMatch/fo?_fooo5117=== RUN TestGlobMatch/?oo_foo5118=== PAUSE TestGlobMatch/?oo_foo5119=== RUN TestGlobMatch/?oo_boo5120=== PAUSE TestGlobMatch/?oo_boo5121=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5122=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5123=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5124=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5125=== CONT TestGlobMatch/foo_foo5126=== CONT TestValidateToken_BoundSubjectMismatch5127=== CONT TestGlobMatch/*/*_foo/bar5128=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5129=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5130=== CONT TestGlobMatch/?oo_boo5131=== CONT TestGlobMatch/?oo_foo5132=== CONT TestGlobMatch/fo?_fooo5133=== CONT TestGlobMatch/fo?_fo5134=== CONT TestGlobMatch/fo?_foo5135=== CONT TestGlobMatch/refs/*/main_refs/heads/main5136=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05137=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5138=== CONT TestGlobMatch/*/*_foo5139=== CONT TestGlobMatch/*bar_bar5140=== CONT TestGlobMatch/foo*bar_foobarbaz5141=== CONT TestGlobMatch/foo*bar_foo123bar5142=== CONT TestGlobMatch/foo*bar_foobar5143=== CONT TestGlobMatch/*bar_foo5144=== CONT TestGlobMatch/*bar_foobar5145=== CONT TestGlobMatch/foo*_foo5146=== CONT TestGlobMatch/foo*_bar5147=== CONT TestGlobMatch/foo*_foobar5148=== CONT TestGlobMatch/*_5149=== CONT TestGlobMatch/*_anything5150=== CONT TestGlobMatch/foo_bar5151--- PASS: TestGlobMatch (0.00s)5152 --- PASS: TestGlobMatch/foo_foo (0.00s)5153 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5154 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5155 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5156 --- PASS: TestGlobMatch/?oo_boo (0.00s)5157 --- PASS: TestGlobMatch/?oo_foo (0.00s)5158 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5159 --- PASS: TestGlobMatch/fo?_fo (0.00s)5160 --- PASS: TestGlobMatch/fo?_foo (0.00s)5161 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5162 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5163 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5164 --- PASS: TestGlobMatch/*/*_foo (0.00s)5165 --- PASS: TestGlobMatch/*bar_bar (0.00s)5166 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5167 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5168 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5169 --- PASS: TestGlobMatch/*bar_foo (0.00s)5170 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5171 --- PASS: TestGlobMatch/foo*_foo (0.00s)5172 --- PASS: TestGlobMatch/foo*_bar (0.00s)5173 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5174 --- PASS: TestGlobMatch/*_ (0.00s)5175 --- PASS: TestGlobMatch/*_anything (0.00s)5176 --- PASS: TestGlobMatch/foo_bar (0.00s)5177=== CONT TestValidateToken_Expired5178=== CONT TestValidateToken_BoundClaimsMismatch51792026/11/12 00:21:49 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235180--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.03s)51812026/11/12 00:21:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49599/oidc51822026/11/12 00:21:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49601/oidc5183--- PASS: TestPins_TopLevelShorthand (0.06s)5184--- PASS: TestValidateToken_Expired (0.03s)51852026/11/12 00:21:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49604/oidc51862026/11/12 00:21:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49606/oidc5187--- PASS: TestValidateToken_WrongAudience (0.10s)5188--- PASS: TestValidateToken_BoundSubjectMismatch (0.09s)51892026/11/12 00:21:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49608/oidc5190--- PASS: TestValidateToken_ValidToken (0.12s)51912026/11/12 00:21:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49613/oidc5192--- PASS: TestPins_ReservedForMatchingRule (0.13s)51932026/11/12 00:21:49 http: TLS handshake error from 127.0.0.1:49611: read tcp 127.0.0.1:49610->127.0.0.1:49611: use of closed network connection5194--- PASS: TestNewValidator_KubernetesRequiresCA (0.13s)51952026/11/12 00:21:49 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49616/oidc51962026/11/12 00:21:49 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:496155197--- PASS: TestValidateToken_BoundClaimsMismatch (0.13s)5198--- PASS: TestValidateToken_KubernetesServiceAccount (0.18s)51992026/11/12 00:21:50 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:49612/oidc52002026/11/12 00:21:50 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:49619/oidc5201--- PASS: TestValidateToken_MultipleProviders (0.17s)52022026/11/12 00:21:50 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49622/oidc5203--- PASS: TestScopes_Rules (0.25s)52042026/11/12 00:21:50 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:49603/oidc5205--- PASS: TestValidateToken_NoMatchingProvider (0.29s)52062026/11/12 00:21:50 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:49626/oidc5207--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.39s)5208PASS5209Running hook tests...5210=== RUN TestSendPathsEmpty5211=== PAUSE TestSendPathsEmpty5212=== RUN TestQueueEnqueueAndFetch5213=== PAUSE TestQueueEnqueueAndFetch5214=== RUN TestQueueDeduplication5215=== PAUSE TestQueueDeduplication5216=== RUN TestQueueRemove5217=== PAUSE TestQueueRemove5218=== RUN TestQueueFetchBatchLimit5219=== PAUSE TestQueueFetchBatchLimit5220=== RUN TestQueueRetryMovesToBack5221=== PAUSE TestQueueRetryMovesToBack5222=== RUN TestQueueFetchRemoveLifecycle5223=== PAUSE TestQueueFetchRemoveLifecycle5224=== RUN TestQueueConcurrentWriters5225=== PAUSE TestQueueConcurrentWriters5226=== RUN TestQueueRemoveLargeClosure5227=== PAUSE TestQueueRemoveLargeClosure5228=== RUN TestServerClientIntegration5229--- PASS: TestServerClientIntegration (0.00s)5230=== RUN TestServerQueueError52312026/11/12 00:21:50 ERROR Failed to queue paths error="permission denied" count=15232--- PASS: TestServerQueueError (0.00s)5233=== RUN TestGetListenerSocketActivation5234 server_test.go:147: systemd socket activation does not exist on darwin5235--- SKIP: TestGetListenerSocketActivation (0.00s)5236=== RUN TestServerStalledClientDoesNotBlockShutdown5237=== PAUSE TestServerStalledClientDoesNotBlockShutdown5238=== RUN TestServerBacksOffOnAcceptErrors5239=== PAUSE TestServerBacksOffOnAcceptErrors5240=== RUN TestDrainIsolatesPoisonPath5241=== PAUSE TestDrainIsolatesPoisonPath5242=== RUN TestRunNotBlockedByPoisonHead5243=== PAUSE TestRunNotBlockedByPoisonHead5244=== RUN TestDrainGivesUpWhenServerDown5245=== PAUSE TestDrainGivesUpWhenServerDown5246=== RUN TestFailedPathPrunedByLaterClosure5247=== PAUSE TestFailedPathPrunedByLaterClosure5248=== RUN TestWorkerUploadsAndRemoves5249=== PAUSE TestWorkerUploadsAndRemoves5250=== RUN TestWorkerSkipsGCdPaths5251=== PAUSE TestWorkerSkipsGCdPaths5252=== RUN TestWorkerPrunesClosureDeps5253=== PAUSE TestWorkerPrunesClosureDeps5254=== RUN TestDrainTimeout5255=== PAUSE TestDrainTimeout5256=== RUN TestWorkerRemoveFailureIsNotProgress5257=== PAUSE TestWorkerRemoveFailureIsNotProgress5258=== RUN TestWorkerKeepsPathItCannotStat5259=== PAUSE TestWorkerKeepsPathItCannotStat5260=== CONT TestSendPathsEmpty5261=== CONT TestQueueFetchBatchLimit5262=== CONT TestQueueConcurrentWriters5263=== CONT TestWorkerKeepsPathItCannotStat5264=== CONT TestWorkerRemoveFailureIsNotProgress5265=== CONT TestDrainTimeout5266=== CONT TestWorkerPrunesClosureDeps5267=== CONT TestWorkerSkipsGCdPaths5268=== CONT TestWorkerUploadsAndRemoves5269=== CONT TestFailedPathPrunedByLaterClosure5270--- PASS: TestSendPathsEmpty (0.00s)5271=== CONT TestRunNotBlockedByPoisonHead5272=== CONT TestDrainGivesUpWhenServerDown5273=== CONT TestQueueRemove5274=== CONT TestDrainIsolatesPoisonPath5275=== CONT TestQueueDeduplication5276=== CONT TestServerBacksOffOnAcceptErrors52772000/01/01 00:00:00 ERROR Accept failed error="too many open files"52782000/01/01 00:00:00 ERROR Accept failed error="too many open files"52792000/01/01 00:00:00 ERROR Accept failed error="too many open files"52802000/01/01 00:00:00 ERROR Accept failed error="too many open files"5281=== CONT TestQueueEnqueueAndFetch52822000/01/01 00:00:00 ERROR Accept failed error="too many open files"52832000/01/01 00:00:00 ERROR Accept failed error="too many open files"52842000/01/01 00:00:00 ERROR Accept failed error="too many open files"52852000/01/01 00:00:00 ERROR Accept failed error="too many open files"5286--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5287=== CONT TestServerStalledClientDoesNotBlockShutdown5288=== CONT TestQueueRemoveLargeClosure5289=== CONT TestQueueFetchRemoveLifecycle5290=== CONT TestQueueRetryMovesToBack52912026/11/12 00:21:50 WARN Cannot stat store path, will retry later path=/private/var/tb/3/scratch/62601a543565e631/build/TestWorkerKeepsPathItCannotStat2185299446/002/locked/aaa error="stat /private/var/tb/3/scratch/62601a543565e631/build/TestWorkerKeepsPathItCannotStat2185299446/002/locked/aaa: permission denied"5292--- PASS: TestQueueRetryMovesToBack (0.01s)52932026/11/12 00:21:50 INFO Uploading batch count=152942026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=152952026/11/12 00:21:50 WARN Cannot stat store path, will retry later path=/private/var/tb/3/scratch/62601a543565e631/build/TestWorkerKeepsPathItCannotStat2185299446/002/locked/aaa error="stat /private/var/tb/3/scratch/62601a543565e631/build/TestWorkerKeepsPathItCannotStat2185299446/002/locked/aaa: permission denied"52962026/11/12 00:21:50 INFO Upload queue status pending=252972026/11/12 00:21:50 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/3/scratch/62601a543565e631/build/TestWorkerRemoveFailureIsNotProgress1821657232/002/nonexistent52982026/11/12 00:21:50 INFO Uploading batch count=452992026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=453002026/11/12 00:21:50 WARN Cannot stat store path, will retry later path=/private/var/tb/3/scratch/62601a543565e631/build/TestWorkerKeepsPathItCannotStat2185299446/002/locked/aaa error="stat /private/var/tb/3/scratch/62601a543565e631/build/TestWorkerKeepsPathItCannotStat2185299446/002/locked/aaa: permission denied"53012026/11/12 00:21:50 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/3/scratch/62601a543565e631/build/TestDrainIsolatesPoisonPath2791728407/002/bbb53022026/11/12 00:21:50 INFO Uploading batch count=253032026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=253042026/11/12 00:21:50 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/3/scratch/62601a543565e631/build/TestDrainGivesUpWhenServerDown1170370850/002/a53052026/11/12 00:21:50 ERROR Drain finished with paths left in queue remaining=153062026/11/12 00:21:50 INFO Uploading batch count=253072026/11/12 00:21:50 INFO Uploading batch count=153082000/01/01 00:00:00 INFO Uploading batch count=25309--- PASS: TestQueueDeduplication (0.01s)53102026/11/12 00:21:50 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153112000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253122026/11/12 00:21:50 INFO Upload queue status pending=253132000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453142026/11/12 00:21:50 INFO Uploading batch count=15315--- PASS: TestQueueRemove (0.01s)5316--- PASS: TestQueueEnqueueAndFetch (0.01s)53172026/11/12 00:21:50 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/3/scratch/62601a543565e631/build/TestWorkerRemoveFailureIsNotProgress1821657232/002/nonexistent53182026/11/12 00:21:50 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/3/scratch/62601a543565e631/build/TestDrainGivesUpWhenServerDown1170370850/002/b53192026/11/12 00:21:50 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153202026/11/12 00:21:50 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/3/scratch/62601a543565e631/build/TestWorkerRemoveFailureIsNotProgress1821657232/002/nonexistent53212026/11/12 00:21:50 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153222026/11/12 00:21:50 ERROR Drain finished with paths left in queue remaining=15323--- PASS: TestQueueFetchRemoveLifecycle (0.01s)53242026/11/12 00:21:50 INFO Upload queue status pending=253252026/11/12 00:21:50 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/3/scratch/62601a543565e631/build/TestWorkerSkipsGCdPaths3071463559/002/nonexistent53262026/11/12 00:21:50 INFO Uploading batch count=153272026/11/12 00:21:50 INFO Uploading batch count=15328--- PASS: TestQueueFetchBatchLimit (0.02s)53292026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=153302026/11/12 00:21:50 INFO Uploading batch count=15331--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)53322026/11/12 00:21:50 INFO Upload queue status pending=353332026/11/12 00:21:50 INFO Uploading batch count=153342026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=153352026/11/12 00:21:50 INFO Uploading batch count=253362026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=253372026/11/12 00:21:50 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/3/scratch/62601a543565e631/build/TestDrainGivesUpWhenServerDown1170370850/002/c53382026/11/12 00:21:50 INFO Uploading batch count=153392026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=153402026/11/12 00:21:50 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/3/scratch/62601a543565e631/build/TestDrainGivesUpWhenServerDown1170370850/002/d53412026/11/12 00:21:50 INFO Uploading batch count=253422026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=253432026/11/12 00:21:50 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/3/scratch/62601a543565e631/build/TestDrainGivesUpWhenServerDown1170370850/002/e5344--- PASS: TestDrainTimeout (0.02s)53452026/11/12 00:21:50 INFO Uploading batch count=153462026/11/12 00:21:50 ERROR Upload failed error="upload failed" count=153472026/11/12 00:21:50 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/3/scratch/62601a543565e631/build/TestDrainGivesUpWhenServerDown1170370850/002/f53482026/11/12 00:21:50 ERROR Drain finished with paths left in queue remaining=153492026/11/12 00:21:50 ERROR Drain finished with paths left in queue remaining=105350--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)5351--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5352--- PASS: TestDrainIsolatesPoisonPath (0.02s)5353--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5354--- PASS: TestWorkerUploadsAndRemoves (0.03s)5355--- PASS: TestWorkerPrunesClosureDeps (0.04s)5356--- PASS: TestWorkerSkipsGCdPaths (0.04s)5357--- PASS: TestQueueRemoveLargeClosure (0.07s)5358--- PASS: TestQueueConcurrentWriters (0.08s)53592026/11/12 00:21:50 ERROR Failed to decode request error="read unix /private/var/tb/3/scratch/62601a543565e631/build/hook2079656704/test.sock->: i/o timeout"53602026/11/12 00:21:50 ERROR Failed to write response error="write unix /private/var/tb/3/scratch/62601a543565e631/build/hook2079656704/test.sock->: i/o timeout"5361--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)53622026/11/12 00:21:51 INFO Uploading batch count=153632026/11/12 00:21:51 INFO Uploading batch count=153642026/11/12 00:21:51 INFO Uploading batch count=153652026/11/12 00:21:51 ERROR Upload failed error="upload failed" count=153662026/11/12 00:21:51 INFO Uploading batch count=153672026/11/12 00:21:51 ERROR Upload failed error="upload failed" count=153682026/11/12 00:21:51 INFO Uploading batch count=153692026/11/12 00:21:51 ERROR Upload failed error="upload failed" count=153702026/11/12 00:21:51 INFO Uploading batch count=153712026/11/12 00:21:51 ERROR Upload failed error="upload failed" count=153722026/11/12 00:21:51 ERROR Drain finished with paths left in queue remaining=15373--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5374PASS5375Running signing tests...5376=== RUN TestExternalSigner5377=== PAUSE TestExternalSigner5378=== RUN TestExternalSignerWrongPublicKey5379=== PAUSE TestExternalSignerWrongPublicKey5380=== RUN TestExternalSignerErrors5381=== PAUSE TestExternalSignerErrors5382=== RUN TestExternalSignerRestartsAfterCrash5383=== PAUSE TestExternalSignerRestartsAfterCrash5384=== RUN TestExternalSignerProgramErrorKeepsProcess5385=== PAUSE TestExternalSignerProgramErrorKeepsProcess5386=== RUN TestExternalSignerClose5387=== PAUSE TestExternalSignerClose5388=== RUN TestExternalSignerRejectsInvalidPublicKey5389=== PAUSE TestExternalSignerRejectsInvalidPublicKey5390=== RUN TestExternalSignerCancellation5391=== PAUSE TestExternalSignerCancellation5392=== RUN TestGenerateFingerprint5393=== PAUSE TestGenerateFingerprint5394=== RUN TestParseSigningKey5395=== PAUSE TestParseSigningKey5396=== RUN TestSign5397=== PAUSE TestSign5398=== RUN TestSignBatch5399=== PAUSE TestSignBatch5400=== RUN TestNilKeyPublicKey5401=== PAUSE TestNilKeyPublicKey5402=== RUN TestSignErrors5403=== PAUSE TestSignErrors5404=== CONT TestExternalSigner5405=== CONT TestExternalSignerCancellation5406=== CONT TestSignBatch5407=== CONT TestSignErrors5408=== CONT TestExternalSignerProgramErrorKeepsProcess5409=== CONT TestExternalSignerRejectsInvalidPublicKey5410=== RUN TestExternalSignerRejectsInvalidPublicKey/#005411=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005412=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5413=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5414=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5415=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5416=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5417=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5418=== CONT TestSign5419=== CONT TestParseSigningKey5420=== RUN TestParseSigningKey/valid_32-byte_key5421=== PAUSE TestParseSigningKey/valid_32-byte_key5422=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5423=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5424=== RUN TestParseSigningKey/no_colon5425=== PAUSE TestParseSigningKey/no_colon5426=== RUN TestParseSigningKey/empty_name5427=== PAUSE TestParseSigningKey/empty_name5428=== RUN TestParseSigningKey/invalid_base645429=== PAUSE TestParseSigningKey/invalid_base645430=== RUN TestParseSigningKey/wrong_length5431=== CONT TestGenerateFingerprint5432=== PAUSE TestParseSigningKey/wrong_length5433=== CONT TestParseSigningKey/valid_32-byte_key5434=== CONT TestExternalSignerErrors5435=== RUN TestGenerateFingerprint/basic_with_references5436=== PAUSE TestGenerateFingerprint/basic_with_references5437=== RUN TestExternalSignerErrors/missing_signatures5438=== PAUSE TestExternalSignerErrors/missing_signatures5439=== RUN TestExternalSignerErrors/wrong_signature_name5440=== PAUSE TestExternalSignerErrors/wrong_signature_name5441=== CONT TestNilKeyPublicKey5442=== RUN TestExternalSignerErrors/wrong_signature_length5443=== PAUSE TestExternalSignerErrors/wrong_signature_length5444=== RUN TestExternalSignerErrors/exit_on_request5445=== PAUSE TestExternalSignerErrors/exit_on_request5446=== RUN TestExternalSignerErrors/wrong_signatures_shape5447=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5448=== RUN TestExternalSignerErrors/too_few_signatures5449=== PAUSE TestExternalSignerErrors/too_few_signatures5450=== RUN TestExternalSignerErrors/wrong_shape5451=== PAUSE TestExternalSignerErrors/wrong_shape5452=== RUN TestExternalSignerErrors/unrequested_key5453=== PAUSE TestExternalSignerErrors/unrequested_key5454=== RUN TestExternalSignerErrors/program_error5455=== PAUSE TestExternalSignerErrors/program_error5456=== RUN TestExternalSignerErrors/invalid_JSON5457=== PAUSE TestExternalSignerErrors/invalid_JSON5458=== RUN TestExternalSignerErrors/wrong_id5459=== PAUSE TestExternalSignerErrors/wrong_id5460=== RUN TestExternalSignerErrors/null_signatures5461--- PASS: TestNilKeyPublicKey (0.00s)5462=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5463=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5464=== CONT TestExternalSignerRejectsInvalidPublicKey/#005465=== CONT TestExternalSignerClose5466=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5467=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5468=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5469=== CONT TestParseSigningKey/wrong_length5470=== CONT TestParseSigningKey/invalid_base645471=== CONT TestParseSigningKey/empty_name5472=== CONT TestParseSigningKey/no_colon5473=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5474=== CONT TestExternalSignerRestartsAfterCrash5475=== RUN TestGenerateFingerprint/no_references5476=== PAUSE TestGenerateFingerprint/no_references5477=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5478=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5479=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5480=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5481=== RUN TestGenerateFingerprint/invalid_nar_hash_length5482=== CONT TestExternalSignerWrongPublicKey5483=== PAUSE TestExternalSignerErrors/null_signatures5484=== RUN TestExternalSignerErrors/missing_signature_name5485=== PAUSE TestExternalSignerErrors/missing_signature_name5486=== RUN TestExternalSignerErrors/invalid_signature_base645487=== PAUSE TestExternalSignerErrors/invalid_signature_base645488=== RUN TestExternalSignerErrors/exit_at_startup5489=== PAUSE TestExternalSignerErrors/exit_at_startup5490=== RUN TestExternalSignerErrors/null5491=== PAUSE TestExternalSignerErrors/null5492=== CONT TestExternalSignerErrors/missing_signatures5493=== CONT TestExternalSignerErrors/invalid_JSON5494=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5495=== CONT TestExternalSignerErrors/wrong_signature_name5496--- PASS: TestSign (0.00s)5497=== CONT TestExternalSignerErrors/invalid_signature_base645498--- PASS: TestParseSigningKey (0.00s)5499 --- PASS: TestParseSigningKey/wrong_length (0.00s)5500 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5501 --- PASS: TestParseSigningKey/empty_name (0.00s)5502 --- PASS: TestParseSigningKey/no_colon (0.00s)5503 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5504 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5505--- PASS: TestSignBatch (0.00s)5506--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5507 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5508 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5509 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5510 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5511 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5512=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5513=== RUN TestGenerateFingerprint/invalid_store_path_prefix5514=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5515=== RUN TestGenerateFingerprint/invalid_reference_prefix5516=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5517=== CONT TestGenerateFingerprint/basic_with_references5518=== RUN TestSignErrors/nil_key5519=== PAUSE TestSignErrors/nil_key5520=== RUN TestSignErrors/nil_narinfo5521=== PAUSE TestSignErrors/nil_narinfo5522=== RUN TestSignErrors/invalid_narinfo5523=== PAUSE TestSignErrors/invalid_narinfo5524=== RUN TestSignErrors/canceled5525=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5526=== PAUSE TestSignErrors/canceled5527=== CONT TestGenerateFingerprint/invalid_store_path_prefix5528=== CONT TestExternalSignerErrors/null_signatures5529=== CONT TestExternalSignerErrors/missing_signature_name5530=== CONT TestExternalSignerErrors/wrong_id5531=== CONT TestExternalSignerErrors/too_few_signatures5532=== CONT TestExternalSignerErrors/program_error5533=== CONT TestExternalSignerErrors/unrequested_key5534=== CONT TestExternalSignerErrors/wrong_shape5535=== CONT TestExternalSignerErrors/exit_on_request5536=== CONT TestExternalSignerErrors/wrong_signature_length5537=== CONT TestExternalSignerErrors/wrong_signatures_shape5538=== CONT TestExternalSignerErrors/exit_at_startup5539=== CONT TestExternalSignerErrors/null5540=== CONT TestGenerateFingerprint/invalid_reference_prefix5541=== CONT TestGenerateFingerprint/invalid_nar_hash_length5542=== CONT TestGenerateFingerprint/no_references5543=== CONT TestSignErrors/nil_key5544=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5545--- PASS: TestGenerateFingerprint (0.00s)5546 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5547 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5548 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5549 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5550 --- PASS: TestGenerateFingerprint/no_references (0.00s)5551 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5552 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5553=== CONT TestSignErrors/canceled5554=== CONT TestSignErrors/nil_narinfo5555=== CONT TestSignErrors/invalid_narinfo5556--- PASS: TestSignErrors (0.00s)5557 --- PASS: TestSignErrors/nil_key (0.00s)5558 --- PASS: TestSignErrors/canceled (0.00s)5559 --- PASS: TestSignErrors/nil_narinfo (0.00s)5560 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5561--- PASS: TestExternalSignerCancellation (0.23s)5562--- PASS: TestExternalSigner (0.43s)5563--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.60s)5564--- PASS: TestExternalSignerWrongPublicKey (0.79s)5565/private/var/tb/3/scratch/62601a543565e631/build/TestExternalSignerRestartsAfterCrash518099141/001/sign program: line 7: /private/var/tb/3/scratch/62601a543565e631/build/TestExternalSignerRestartsAfterCrash518099141/001/sign program.starts: No such file or directory5566--- PASS: TestExternalSignerRestartsAfterCrash (2.02s)5567--- PASS: TestExternalSignerClose (2.43s)5568--- PASS: TestExternalSignerErrors (0.00s)5569 --- PASS: TestExternalSignerErrors/invalid_JSON (0.98s)5570 --- PASS: TestExternalSignerErrors/null (1.17s)5571 --- PASS: TestExternalSignerErrors/missing_signatures (1.37s)5572 --- PASS: TestExternalSignerErrors/wrong_id (1.58s)5573 --- PASS: TestExternalSignerErrors/too_few_signatures (1.79s)5574 --- PASS: TestExternalSignerErrors/wrong_signature_name (2.23s)5575 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (2.64s)5576 --- PASS: TestExternalSignerErrors/exit_at_startup (2.81s)5577 --- PASS: TestExternalSignerErrors/program_error (3.03s)5578 --- PASS: TestExternalSignerErrors/null_signatures (3.22s)5579 --- PASS: TestExternalSignerErrors/unrequested_key (3.41s)5580 --- PASS: TestExternalSignerErrors/missing_signature_name (3.63s)5581 --- PASS: TestExternalSignerErrors/wrong_signature_length (3.86s)5582 --- PASS: TestExternalSignerErrors/exit_on_request (4.05s)5583 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (4.26s)5584 --- PASS: TestExternalSignerErrors/wrong_shape (4.48s)5585PASS5586Running OpenBao signer tests...5587=== RUN TestSign5588=== PAUSE TestSign5589=== RUN TestSignErrors5590=== PAUSE TestSignErrors5591=== RUN TestServe5592=== PAUSE TestServe5593=== RUN TestServeRejectsGarbage5594=== PAUSE TestServeRejectsGarbage5595=== CONT TestSign5596=== CONT TestSignErrors5597=== RUN TestSignErrors/http_error5598=== CONT TestServe5599=== CONT TestServeRejectsGarbage5600=== PAUSE TestSignErrors/http_error5601--- PASS: TestServe (0.00s)5602--- PASS: TestServeRejectsGarbage (0.00s)5603=== RUN TestSignErrors/invalid_json5604=== PAUSE TestSignErrors/invalid_json5605=== RUN TestSignErrors/no_prefix5606=== PAUSE TestSignErrors/no_prefix5607=== RUN TestSignErrors/invalid_base645608=== PAUSE TestSignErrors/invalid_base645609=== CONT TestSignErrors/http_error5610=== CONT TestSignErrors/no_prefix5611=== CONT TestSignErrors/invalid_base645612=== CONT TestSignErrors/invalid_json5613--- PASS: TestSign (0.03s)5614--- PASS: TestSignErrors (0.02s)5615 --- PASS: TestSignErrors/invalid_base64 (0.01s)5616 --- PASS: TestSignErrors/http_error (0.02s)5617 --- PASS: TestSignErrors/no_prefix (0.02s)5618 --- PASS: TestSignErrors/invalid_json (0.02s)5619PASS