niks3-go-unit-tests
checks.aarch64-linux.go-unit-tests
· build #494
· raw
1tribuchet: building on eliza2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (1.38s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestDoServerRequestAttachesToken114=== CONT TestScriptTokenEmptyCommand115--- PASS: TestScriptTokenEmptyCommand (0.00s)116=== CONT TestRegisterUploadedObject_OutlivesPush117=== CONT TestDoWithRetry_BodyReplayedViaGetBody118=== CONT TestClientSignaturesByStorePath119--- PASS: TestClientSignaturesByStorePath (0.00s)120=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer121=== CONT TestStreamPushReportsSkippedPaths122=== CONT TestStreamPushReportsSignatures123=== CONT TestStreamPushRequestLine124=== CONT TestStreamPushGivesUpOnDeadServer125--- PASS: TestStreamPushReportsSkippedPaths (0.00s)126=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess1272026/11/12 00:05:11 ERROR Upload failed error=boom count=1128=== CONT TestStreamPushIsolatesFailures129=== CONT TestStreamPushBatchesUnderLoad130=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false131=== CONT TestStreamPushReportsEveryPath132=== CONT TestShellSplitErrors133=== CONT TestShellSplit134--- PASS: TestShellSplit (0.00s)1352026/11/12 00:05:13 ERROR Upload failed error=boom count=1136--- PASS: TestStreamPushReportsSignatures (1.86s)1372026/11/12 00:05:11 WARN Rate limiter enabled after throttle name=server-test rate=51382026/11/12 00:05:13 ERROR Upload failed error="connection refused" count=20139=== CONT TestPathInfoCACompatibility140=== CONT TestParsePathInfoJSONMultiplePaths141=== CONT TestDoWithRetry_FinalResponseBodyReadable142=== CONT TestRateLimiterFeedback1432026/11/12 00:05:13 ERROR Server seems unavailable, giving up on batch untried=17144=== RUN TestPathInfoCACompatibility/null_ca_field1452026/11/12 00:05:13 ERROR Upload failed error="bad path" count=3146=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false147=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths148=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true149=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true150=== CONT TestConvertHashToNix32151=== RUN TestRateLimiterFeedback/429_enables_limiter152=== CONT TestScriptTokenBadJSON153=== CONT TestScriptTokenEmptyToken154=== CONT TestScriptTokenCachesUntilRefresh155=== CONT TestScriptTokenNoExpiryRerunsEveryCall156=== CONT TestFileTokenEmpty157=== CONT TestEncodeNixBase32158=== CONT TestResolveStorePath159=== CONT TestScriptTokenScriptFails160--- PASS: TestShellSplitErrors (0.74s)161=== CONT TestRetryTimeoutOutlastsMaxRetries162=== CONT TestFileTokenMissing163=== PAUSE TestPathInfoCACompatibility/null_ca_field164=== CONT TestParsePathInfoJSON165=== CONT TestPathInfoHashCompatibility166=== CONT TestGetStorePathHash167=== RUN TestConvertHashToNix32/SRI_format_to_Nix32168=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32169=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths170=== RUN TestConvertHashToNix32/already_Nix32_format171=== PAUSE TestConvertHashToNix32/already_Nix32_format172=== RUN TestConvertHashToNix32/invalid_format173=== PAUSE TestConvertHashToNix32/invalid_format174=== CONT TestUploadMultipart_PartsInParallel175=== CONT TestDumpPathWriterError176=== PAUSE TestRateLimiterFeedback/429_enables_limiter177=== RUN TestRateLimiterFeedback/503_enables_limiter178=== PAUSE TestRateLimiterFeedback/503_enables_limiter179=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter180=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter181=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter182=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter183=== RUN TestGetStorePathHash/valid_store_path184=== PAUSE TestGetStorePathHash/valid_store_path185=== RUN TestGetStorePathHash/basename_without_hyphen_should_error186=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error187=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error188=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error189=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error190=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error191=== CONT TestDumpPathMatchesNix192=== CONT TestUploadMultipart_ReturnsAfterDroppedBody1932026/11/12 00:05:13 WARN Rate limiter enabled after throttle name=server-test rate=5194=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun1952026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:436451962026/11/12 00:05:13 WARN Rate limiter backed off name=server-test rate=51972026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:409751982026/11/12 00:05:13 WARN Rate limiter enabled after throttle name=server-test rate=51992026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:40247200=== CONT TestFilterOversizedClosures2012026/11/12 00:05:13 WARN Rate limiter backed off name=server-test rate=5202=== RUN TestFilterOversizedClosures/no_limit_keeps_everything2032026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:40247204=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything205=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped206=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped207=== RUN TestFilterOversizedClosures/all_closures_skipped208=== PAUSE TestFilterOversizedClosures/all_closures_skipped209=== CONT TestCaseHackSuffix210=== CONT TestRegisterUploadedObjectReusesConnections211=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths212=== CONT TestPartSizeForNAR213=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica214--- PASS: TestStreamPushReportsEveryPath (0.74s)215--- PASS: TestStreamPushGivesUpOnDeadServer (1.86s)2162026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:40975217--- PASS: TestStreamPushIsolatesFailures (1.86s)218--- PASS: TestFileTokenEmpty (0.00s)219--- PASS: TestResolveStorePath (0.00s)220--- PASS: TestScriptTokenBadJSON (0.04s)221--- PASS: TestScriptTokenEmptyToken (0.04s)222--- PASS: TestFileTokenMissing (0.00s)223--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.04s)224--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (1.91s)225--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)226--- PASS: TestDoServerRequestAttachesToken (1.91s)227=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry228=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths229=== CONT TestStaticToken230--- PASS: TestRegisterUploadedObject_OutlivesPush (1.91s)231--- PASS: TestScriptTokenScriptFails (0.01s)232--- PASS: TestStaticToken (0.00s)233=== CONT TestSetClientTLSDoesNotMutateDefaultTransport234=== RUN TestEncodeNixBase32/test_string_hash235=== RUN TestPathInfoCACompatibility/old_string_format_-_text236=== RUN TestParsePathInfoJSON/Nix_format237=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)238=== CONT TestEncodeNixBase32WithRealHash239=== CONT TestDumpPathSingleFile240=== RUN TestPartSizeForNAR/zero_stays_at_minimum241=== CONT TestSetClientTLSErrors242=== CONT TestFileTokenReadsAndCaches2432026/11/12 00:05:13 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38595/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38595/api/uploads/complete2442026/11/12 00:05:13 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"245=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text246=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive247=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive248=== RUN TestPathInfoCACompatibility/new_structured_format_-_text249=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text250=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method251=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum252=== RUN TestPartSizeForNAR/small_stays_at_minimum253=== PAUSE TestPartSizeForNAR/small_stays_at_minimum254=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum255=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum256=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts257=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts2582026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:40975259=== RUN TestPartSizeForNAR/1_TiB260=== PAUSE TestPartSizeForNAR/1_TiB261=== PAUSE TestEncodeNixBase32/test_string_hash262=== RUN TestEncodeNixBase32/empty_input263=== PAUSE TestEncodeNixBase32/empty_input264=== PAUSE TestParsePathInfoJSON/Nix_format265=== RUN TestParsePathInfoJSON/Lix_format266=== PAUSE TestParsePathInfoJSON/Lix_format267=== RUN TestParsePathInfoJSON/empty_input268=== PAUSE TestParsePathInfoJSON/empty_input269=== RUN TestParsePathInfoJSON/whitespace_only270=== PAUSE TestParsePathInfoJSON/whitespace_only271=== RUN TestParsePathInfoJSON/invalid_JSON272=== PAUSE TestParsePathInfoJSON/invalid_JSON273=== RUN TestPartSizeForNAR/5_TiB_S3_max_object274=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object275=== RUN TestPartSizeForNAR/capped_at_5_GiB276=== PAUSE TestPartSizeForNAR/capped_at_5_GiB277=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false278--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.89s)279=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method280=== CONT TestSetClientTLS281=== CONT TestStreamPushStopsWhenCanceled282--- PASS: TestStreamPushStopsWhenCanceled (0.00s)283=== CONT TestConvertHashToNix32/invalid_format284=== CONT TestConvertHashToNix32/already_Nix32_format285=== CONT TestRateLimiterFeedback/429_enables_limiter286--- PASS: TestFileTokenReadsAndCaches (0.00s)287=== CONT TestGetStorePathHash/valid_store_path288--- PASS: TestEncodeNixBase32WithRealHash (0.00s)289=== CONT TestFilterOversizedClosures/all_closures_skipped2902026/11/12 00:05:13 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50291=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped292=== CONT TestFilterOversizedClosures/no_limit_keeps_everything2932026/11/12 00:05:13 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38595/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38595/api/uploads/complete294=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error2952026/11/12 00:05:13 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"296=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true2972026/11/12 00:05:13 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000298--- PASS: TestFilterOversizedClosures (0.00s)299 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)300 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)301 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)302=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter3032026/11/12 00:05:13 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:38595/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:38595/api/uploads/complete3042026/11/12 00:05:13 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"305--- PASS: TestScriptTokenCachesUntilRefresh (0.90s)306=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter307--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (2.76s)308=== CONT TestGetStorePathHash/basename_without_hyphen_should_error309=== CONT TestConvertHashToNix32/SRI_format_to_Nix32310=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)311=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error312=== CONT TestRateLimiterFeedback/503_enables_limiter313=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths3142026/11/12 00:05:13 WARN Rate limiter enabled after throttle name=server-test rate=5315--- PASS: TestConvertHashToNix32 (0.00s)316 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)317 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)318 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)3192026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:37053320=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon321=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon322=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI323=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI324=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512325=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512326=== CONT TestPartSizeForNAR/5_TiB_S3_max_object327=== CONT TestPartSizeForNAR/1_TiB328=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts329=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum330=== CONT TestPartSizeForNAR/small_stays_at_minimum331=== CONT TestParsePathInfoJSON/invalid_JSON332=== CONT TestParsePathInfoJSON/whitespace_only333=== CONT TestParsePathInfoJSON/empty_input334=== CONT TestParsePathInfoJSON/Lix_format335=== CONT TestPathInfoCACompatibility/null_ca_field336=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method337=== CONT TestPathInfoCACompatibility/new_structured_format_-_text338=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive339=== CONT TestPathInfoCACompatibility/old_string_format_-_text3402026/11/12 00:05:13 WARN Rate limiter backed off name=server-test rate=5341=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)342=== CONT TestEncodeNixBase32/test_string_hash343=== CONT TestEncodeNixBase32/empty_input344=== CONT TestParsePathInfoJSON/Nix_format3452026/11/12 00:05:13 WARN Rate limiter enabled after throttle name=server-test rate=53462026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:359153472026/11/12 00:05:13 WARN Rate limiter backed off name=server-test rate=5348=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512349--- PASS: TestStreamPushBatchesUnderLoad (1.86s)350 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)351 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)352--- PASS: TestGetStorePathHash (0.04s)353 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)354 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)355 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)356 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)357=== CONT TestPartSizeForNAR/capped_at_5_GiB358=== CONT TestPartSizeForNAR/zero_stays_at_minimum359=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon360--- PASS: TestPartSizeForNAR (0.85s)361 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)362 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)363 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)364 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)365 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)366 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)367 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)3682026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:40975369=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths3702026/11/12 00:05:13 INFO Garbage collection started371=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI372--- PASS: TestParsePathInfoJSON (0.86s)373 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)374 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)375 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)376 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)377 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)378--- PASS: TestPathInfoCACompatibility (0.90s)379 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)380 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)381 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)382 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)383 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)384--- PASS: TestEncodeNixBase32 (0.90s)385 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)386 --- PASS: TestEncodeNixBase32/empty_input (0.00s)387--- PASS: TestRateLimiterFeedback (0.04s)388 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)389 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)390 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)391 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)392--- PASS: TestParsePathInfoJSONMultiplePaths (0.05s)393 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)394 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)395--- PASS: TestPathInfoHashCompatibility (0.86s)396 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)397 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)398 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)399 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)4002026/11/12 00:05:13 INFO Garbage collection started401=== RUN TestSetClientTLSErrors/missing_cert_file402=== RUN TestSetClientTLS/rejects_connection_without_client_cert4032026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:409754042026/11/12 00:05:13 WARN Rate limiter enabled after throttle name=s3-test rate=54052026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:41647406--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.88s)4072026/11/12 00:05:13 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0408--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.87s)409=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert410=== PAUSE TestSetClientTLSErrors/missing_cert_file411=== RUN TestSetClientTLSErrors/missing_key_file412=== PAUSE TestSetClientTLSErrors/missing_key_file413=== RUN TestSetClientTLSErrors/missing_ca_file414=== PAUSE TestSetClientTLSErrors/missing_ca_file415=== RUN TestSetClientTLSErrors/invalid_ca_file4162026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:40975417=== PAUSE TestSetClientTLSErrors/invalid_ca_file418=== CONT TestSetClientTLSErrors/missing_cert_file419=== CONT TestSetClientTLSErrors/invalid_ca_file420=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA421=== CONT TestSetClientTLSErrors/missing_ca_file422--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.87s)423=== CONT TestSetClientTLSErrors/missing_key_file424=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA425=== RUN TestSetClientTLS/preserves_debug_logging_transport426=== PAUSE TestSetClientTLS/preserves_debug_logging_transport427=== CONT TestSetClientTLS/rejects_connection_without_client_cert428=== CONT TestSetClientTLS/preserves_debug_logging_transport4292026/11/12 00:05:13 WARN Garbage collection finished on another replica. Its statistics are not available here430--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.88s)4312026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:40975432=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA433--- PASS: TestSetClientTLSErrors (0.02s)434 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)435 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)436 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)437 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)4382026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:409754392026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:409754402026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:409754412026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:409754422026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:409754432026/11/12 00:05:13 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:409754442026/11/12 00:05:14 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:409754452026/11/12 00:05:14 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:40975446--- PASS: TestDumpPathWriterError (0.93s)4472026/11/12 00:05:14 http: TLS handshake error from 127.0.0.1:37354: remote error: tls: bad certificate4482026/11/12 00:05:14 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:40975449--- PASS: TestSetClientTLS (0.02s)450 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)451 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)452 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)4532026/11/12 00:05:14 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:40975454--- PASS: TestStreamPushRequestLine (2.80s)455--- PASS: TestRegisterUploadedObjectReusesConnections (0.89s)4562026/11/12 00:05:14 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:409754572026/11/12 00:05:14 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:40975458--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.90s)459--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (2.86s)460--- PASS: TestUploadMultipart_PartsInParallel (1.53s)461--- PASS: TestCaseHackSuffix (1.53s)462--- PASS: TestDumpPathSingleFile (0.68s)463--- PASS: TestDumpPathMatchesNix (1.55s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "nixbld".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /build/postgres2064663277/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /build/postgres2064663277/data -l logfile start492493/build/postgres2064663277:5432 - no response494/build/postgres2064663277:5432 - no response495/build/postgres2064663277:5432 - no response496/build/postgres2064663277:5432 - no response497/build/postgres2064663277:5432 - no response498/build/postgres2064663277:5432 - no response499/build/postgres2064663277:5432 - no response500/build/postgres2064663277:5432 - no response501/build/postgres2064663277:5432 - no response502/build/postgres2064663277:5432 - no response503/build/postgres2064663277:5432 - no response504/build/postgres2064663277:5432 - no response505/build/postgres2064663277:5432 - no response5062026-11-12 00:05:26.991 UTC [129] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit5072026-11-12 00:05:26.991 UTC [129] LOG: listening on Unix socket "/build/postgres2064663277/.s.PGSQL.5432"5082026-11-12 00:05:26.998 UTC [148] LOG: database system was shut down at 2026-11-12 00:05:25 UTC5092026-11-12 00:05:27.003 UTC [129] LOG: database system is ready to accept connections510/build/postgres2064663277:5432 - accepting connections511{"timestamp":"2026-11-12T00:05:41.278634792Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"652fc4e7-1e91-4453-b131-902da8af1102","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":15,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(199)"}512{"timestamp":"2026-11-12T00:05:41.3906134Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cd0371cc-a3f3-4c2a-b9ee-52c7faa02b13","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(203)"}513=== RUN TestAbortSucceeded514=== PAUSE TestAbortSucceeded515=== RUN TestService_AuthMiddleware516=== PAUSE TestService_AuthMiddleware517=== RUN TestService_AuthMiddleware_MTLSProxyHeader518=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader519=== RUN TestService_AuthMiddleware_MTLSBoundSubjects520=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects521=== RUN TestService_ReadAuthMiddleware522=== PAUSE TestService_ReadAuthMiddleware523=== RUN TestService_AuthMiddleware_OIDC524=== PAUSE TestService_AuthMiddleware_OIDC525=== RUN TestService_RequireScope_OIDC526=== PAUSE TestService_RequireScope_OIDC527=== RUN TestService_ReadScope_PublicByDefault528=== PAUSE TestService_ReadScope_PublicByDefault529=== RUN TestCacheConfigHandler530=== PAUSE TestCacheConfigHandler531=== RUN TestCacheStatsHandler532=== PAUSE TestCacheStatsHandler533=== RUN TestClientCADerivations534=== PAUSE TestClientCADerivations535=== RUN TestClientErrorHandling536=== PAUSE TestClientErrorHandling537=== RUN TestClientIntegration538=== PAUSE TestClientIntegration539=== RUN TestClientMultipleUploads540=== PAUSE TestClientMultipleUploads541=== RUN TestClientWithDependencies542=== PAUSE TestClientWithDependencies543=== RUN TestClientSharedPathCommittedMidPush544=== PAUSE TestClientSharedPathCommittedMidPush545=== RUN TestPinProtectsFromGC546=== PAUSE TestPinProtectsFromGC547=== RUN TestClientPushesUseOnePush548=== PAUSE TestClientPushesUseOnePush549=== RUN TestClientFallsBackToClosures550=== PAUSE TestClientFallsBackToClosures551=== RUN TestDeleteClosuresRacingPins552=== PAUSE TestDeleteClosuresRacingPins553=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock554=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock555=== RUN TestResolveDBConnectionString556=== PAUSE TestResolveDBConnectionString557=== RUN TestConnectWaitsForAPeerMigration558=== PAUSE TestConnectWaitsForAPeerMigration559=== RUN TestLeadElectsOneAndHandsOver560=== PAUSE TestLeadElectsOneAndHandsOver561=== RUN TestLeadIncumbentWinsAfterRestart5622026/11/12 00:05:44 INFO lead: acquired remote=192.0.2.1:1234 generation=15632026/11/12 00:05:45 INFO lead: released remote=192.0.2.1:1234 generation=15642026/11/12 00:05:45 INFO lead: acquired remote=192.0.2.1:1234 generation=25652026/11/12 00:05:46 INFO lead: released remote=192.0.2.1:1234 generation=2566--- PASS: TestLeadIncumbentWinsAfterRestart (5.19s)567=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent568=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent569=== RUN TestLeadIncumbentAnnouncesAtOnce570=== PAUSE TestLeadIncumbentAnnouncesAtOnce571=== RUN TestLeadGenerationGrowsWithEveryGrant572=== PAUSE TestLeadGenerationGrowsWithEveryGrant573=== RUN TestLeadEndsOnShutdown574=== PAUSE TestLeadEndsOnShutdown575=== RUN TestLeadEndsWhenItsConnectionHangs5762026/11/12 00:05:48 INFO lead: acquired remote=192.0.2.1:1234 generation=15772026-11-12 00:05:50.008 UTC [396] FATAL: terminating connection due to administrator command5782026/11/12 00:05:50 INFO lead: acquired remote=192.0.2.1:1234 generation=25792026/11/12 00:05:51 WARN lead: lock connection lost error="timeout: context deadline exceeded"5802026/11/12 00:05:51 INFO lead: released remote=192.0.2.1:1234 generation=15812026/11/12 00:05:51 INFO lead: released remote=192.0.2.1:1234 generation=2582--- PASS: TestLeadEndsWhenItsConnectionHangs (4.91s)583=== RUN TestGCAdvisoryLockBlocksConcurrentRun584=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun585=== RUN TestGCBugBareHashReferences586=== PAUSE TestGCBugBareHashReferences587=== RUN TestGCMetrics588=== PAUSE TestGCMetrics589=== RUN TestPushDedupSurvivesConcurrentGC590=== PAUSE TestPushDedupSurvivesConcurrentGC591=== RUN TestDeduplicatedObjectsRecordedAsPending592=== PAUSE TestDeduplicatedObjectsRecordedAsPending593=== RUN TestGCSweepSkipsPendingObjects594=== PAUSE TestGCSweepSkipsPendingObjects595=== RUN TestTombstonedObjectOfferedWithoutWaiting596=== PAUSE TestTombstonedObjectOfferedWithoutWaiting597=== RUN TestGCSweepDeliversEachKeyOnce598=== PAUSE TestGCSweepDeliversEachKeyOnce599=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection600=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection601=== RUN TestForceGCDuringPushOffersSweptObject602=== PAUSE TestForceGCDuringPushOffersSweptObject603=== RUN TestFailedS3DeleteKeepsTombstone604=== PAUSE TestFailedS3DeleteKeepsTombstone605=== RUN TestSweepSkipsRowsLockedByPush606=== PAUSE TestSweepSkipsRowsLockedByPush607=== RUN TestRegisterCompletedObjectNeedsPendingRow608=== PAUSE TestRegisterCompletedObjectNeedsPendingRow609=== RUN TestGetExistingObjectsLocksRows610=== PAUSE TestGetExistingObjectsLocksRows611=== RUN TestSweepRechecksPendingAfterLocking612=== PAUSE TestSweepRechecksPendingAfterLocking613=== RUN TestPushWaitsForSweepHoldingRow614=== PAUSE TestPushWaitsForSweepHoldingRow615=== RUN TestCommitRacingPendingCleanupKeepsObjects616=== PAUSE TestCommitRacingPendingCleanupKeepsObjects617=== RUN TestGCEndsOnShutdown618=== PAUSE TestGCEndsOnShutdown619=== RUN TestGCTaskStore_StartNew620=== PAUSE TestGCTaskStore_StartNew621=== RUN TestGCTaskStore_DeduplicateSameParams622=== PAUSE TestGCTaskStore_DeduplicateSameParams623=== RUN TestGCTaskStore_ConflictDifferentParams624=== PAUSE TestGCTaskStore_ConflictDifferentParams625=== RUN TestGCTaskStore_GetEmpty626=== PAUSE TestGCTaskStore_GetEmpty627=== RUN TestGCTaskStore_GetReturnsLatest628=== PAUSE TestGCTaskStore_GetReturnsLatest629=== RUN TestGCTaskStore_CompletedAllowsNewTask630=== PAUSE TestGCTaskStore_CompletedAllowsNewTask631=== RUN TestGCTaskStore_PhaseUpdates632=== PAUSE TestGCTaskStore_PhaseUpdates633=== RUN TestGCTaskStore_Fail634=== PAUSE TestGCTaskStore_Fail635=== RUN TestGracefulShutdownDrainsInflight636=== PAUSE TestGracefulShutdownDrainsInflight637=== RUN TestService_healthCheckHandler638=== PAUSE TestService_healthCheckHandler639=== RUN TestService_readinessHandler640=== PAUSE TestService_readinessHandler641=== RUN TestGenerateLandingPage642=== PAUSE TestGenerateLandingPage643=== RUN TestEnsureAbortUploadsRule644=== PAUSE TestEnsureAbortUploadsRule645=== RUN TestCacheConfigHandlerMaxNarSize646=== PAUSE TestCacheConfigHandlerMaxNarSize647=== RUN TestCreatePendingClosureRejectsOversizedNAR648=== PAUSE TestCreatePendingClosureRejectsOversizedNAR649=== RUN TestNARDeduplicationMetadataUploadBug650=== PAUSE TestNARDeduplicationMetadataUploadBug651=== RUN TestMetricsInventory652=== PAUSE TestMetricsInventory653=== RUN TestMetricsCommit654=== PAUSE TestMetricsCommit655=== RUN TestService_NativeMTLS656=== PAUSE TestService_NativeMTLS657=== RUN TestServerTLSConfig658=== PAUSE TestServerTLSConfig659=== RUN TestMultipartCleanup660=== PAUSE TestMultipartCleanup661=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded662=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded663=== RUN TestPendingCleanupUsesOneCutoff664=== PAUSE TestPendingCleanupUsesOneCutoff665=== RUN TestObjectStatsTrigger666=== PAUSE TestObjectStatsTrigger667=== RUN TestObjectStatsUpdatedOncePerCommit668=== PAUSE TestObjectStatsUpdatedOncePerCommit669=== RUN TestOrphanedObjectsGC670=== PAUSE TestOrphanedObjectsGC671=== RUN TestOrphanedObjectsGCStressTest672=== PAUSE TestOrphanedObjectsGCStressTest673=== RUN TestCreatePin_ReservedPins674=== PAUSE TestCreatePin_ReservedPins675=== RUN TestPresentReportsOnlyClosureRoots676=== PAUSE TestPresentReportsOnlyClosureRoots677=== RUN TestPresentRefreshesOnlyStaleClosures678=== PAUSE TestPresentRefreshesOnlyStaleClosures679=== RUN TestPresentNotReportedWhileGCDeletesClosure680=== PAUSE TestPresentNotReportedWhileGCDeletesClosure681=== RUN TestParseSingleRange682=== PAUSE TestParseSingleRange683=== RUN TestProxyHeadersOnlyTrustedOnSocket6842026/11/12 00:05:51 INFO Starting HTTP server address=proxy.sock6852026/11/12 00:05:51 INFO Starting HTTP server address=127.0.0.1:340976862026/11/12 00:05:51 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6872026/11/12 00:05:51 INFO Shutdown signal received, draining in-flight requests timeout=10s688--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.20s)689=== RUN TestIsValidCachePath690=== PAUSE TestIsValidCachePath691=== RUN TestReadProxyNarinfo692=== PAUSE TestReadProxyNarinfo693=== RUN TestReadProxyNarinfoAlreadyDecompressed694=== PAUSE TestReadProxyNarinfoAlreadyDecompressed695=== RUN TestReadProxyContentNegotiation696=== PAUSE TestReadProxyContentNegotiation697=== RUN TestReadProxyNarStreaming698=== PAUSE TestReadProxyNarStreaming699=== RUN TestReadProxy404700=== PAUSE TestReadProxy404701=== RUN TestReadProxyInvalidPath702=== PAUSE TestReadProxyInvalidPath703=== RUN TestReadProxyHead704=== PAUSE TestReadProxyHead705=== RUN TestReadProxyOutlastsServerWriteTimeout706=== PAUSE TestReadProxyOutlastsServerWriteTimeout707=== RUN TestReadProxyConditionalGet708=== PAUSE TestReadProxyConditionalGet709=== RUN TestReadProxyRootRedirectsToIndexHTML710=== PAUSE TestReadProxyRootRedirectsToIndexHTML711=== RUN TestReadProxyDisabled712=== PAUSE TestReadProxyDisabled713=== RUN TestReadRedirectNar714=== PAUSE TestReadRedirectNar715=== RUN TestReadRedirectKeepsNarinfoProxied716=== PAUSE TestReadRedirectKeepsNarinfoProxied717=== RUN TestReadProxyRangeRequest718=== PAUSE TestReadProxyRangeRequest719=== RUN TestReadRedirectUsesPublicS3URL720=== PAUSE TestReadRedirectUsesPublicS3URL721=== RUN TestPush_OverlappingRootsStoreOneRowPerKey722=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey723=== RUN TestPush_CompleteCommitsEveryRoot724=== PAUSE TestPush_CompleteCommitsEveryRoot725=== RUN TestPush_RepeatedRootCommitsOnce726=== PAUSE TestPush_RepeatedRootCommitsOnce727=== RUN TestPush_CompleteWaitsForCleanupOfThePush728=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush729=== RUN TestPush_RootsAreLockedInByteOrder730=== PAUSE TestPush_RootsAreLockedInByteOrder731=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit732=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit733=== RUN TestPush_RejectsBadRequests734=== PAUSE TestPush_RejectsBadRequests735=== RUN TestPush_SignsNarinfosOfItsPendingObjects736=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects737=== RUN TestSharedNarBothUploadsComplete738=== PAUSE TestSharedNarBothUploadsComplete739=== RUN TestCompleteMultipartUpload_ErrorButObjectExists740=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists741=== RUN TestCompletedNarNotReofferedAcrossClosures742=== PAUSE TestCompletedNarNotReofferedAcrossClosures743=== RUN TestPresignedUploadRegisteredBeforeCommit744=== PAUSE TestPresignedUploadRegisteredBeforeCommit745=== RUN TestService_Rustfstest746=== PAUSE TestService_Rustfstest747=== RUN TestParseSize748=== PAUSE TestParseSize749=== RUN TestSkippedUploadsHandler750=== PAUSE TestSkippedUploadsHandler751=== RUN TestSystemdListenerNotActivated752--- PASS: TestSystemdListenerNotActivated (0.00s)753=== RUN TestWatchdogBeatsWhenHealthy754--- PASS: TestWatchdogBeatsWhenHealthy (0.03s)755=== RUN TestWatchdogSkipsWhenUnhealthy7562026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7572026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7582026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7592026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7602026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7612026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7622026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7632026/11/12 00:05:51 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7642026/11/12 00:05:52 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7652026/11/12 00:05:52 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"766--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)767=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle768=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle769=== RUN TestProxyWriteTimeout770=== PAUSE TestProxyWriteTimeout771=== RUN TestPendingClosureWriteTimeout772=== PAUSE TestPendingClosureWriteTimeout773=== RUN TestIsValidUploadKey774=== PAUSE TestIsValidUploadKey775=== RUN TestUploadHandlersRejectInvalidKeys776=== PAUSE TestUploadHandlersRejectInvalidKeys777=== RUN TestUploadHandlersRejectOversizedBody778=== PAUSE TestUploadHandlersRejectOversizedBody779=== RUN TestService_cleanupPendingClosuresHandler780=== PAUSE TestService_cleanupPendingClosuresHandler781=== RUN TestService_createPendingClosureHandler782=== PAUSE TestService_createPendingClosureHandler783=== RUN TestService_verifyS3Integrity784=== PAUSE TestService_verifyS3Integrity785=== RUN TestService_verifyS3ReuploadsStaleNarinfo786=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo787=== RUN TestCompleteMultipartUnregistered788=== PAUSE TestCompleteMultipartUnregistered789=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT790=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT791=== CONT TestAbortSucceeded792--- PASS: TestAbortSucceeded (0.00s)793=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT794=== CONT TestCompleteMultipartUnregistered795=== CONT TestService_verifyS3ReuploadsStaleNarinfo796=== CONT TestService_verifyS3Integrity797=== CONT TestService_createPendingClosureHandler798=== CONT TestService_cleanupPendingClosuresHandler799=== CONT TestUploadHandlersRejectOversizedBody800=== CONT TestUploadHandlersRejectInvalidKeys801=== CONT TestIsValidUploadKey802=== CONT TestPendingClosureWriteTimeout803=== CONT TestProxyWriteTimeout804=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle805=== CONT TestSkippedUploadsHandler806=== CONT TestParseSize807=== CONT TestService_Rustfstest808=== CONT TestPresignedUploadRegisteredBeforeCommit809=== CONT TestCompletedNarNotReofferedAcrossClosures810=== CONT TestCompleteMultipartUpload_ErrorButObjectExists811=== CONT TestSharedNarBothUploadsComplete812=== CONT TestPush_SignsNarinfosOfItsPendingObjects813=== CONT TestPush_RejectsBadRequests814=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit815=== CONT TestPush_RootsAreLockedInByteOrder816=== CONT TestPush_CompleteWaitsForCleanupOfThePush817--- PASS: TestParseSize (0.11s)818=== CONT TestPush_RepeatedRootCommitsOnce819=== RUN TestIsValidUploadKey/narinfo820=== PAUSE TestIsValidUploadKey/narinfo821=== RUN TestPendingClosureWriteTimeout/empty822=== PAUSE TestPendingClosureWriteTimeout/empty823=== RUN TestPendingClosureWriteTimeout/400_objects824=== PAUSE TestPendingClosureWriteTimeout/400_objects825=== RUN TestPendingClosureWriteTimeout/670k_objects826=== PAUSE TestPendingClosureWriteTimeout/670k_objects827=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline828=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline829=== RUN TestProxyWriteTimeout/narinfo830=== PAUSE TestProxyWriteTimeout/narinfo831=== RUN TestProxyWriteTimeout/1_GiB_nar832=== PAUSE TestProxyWriteTimeout/1_GiB_nar833=== RUN TestProxyWriteTimeout/10_GiB_nar834=== PAUSE TestProxyWriteTimeout/10_GiB_nar835=== RUN TestProxyWriteTimeout/unknown_size836=== PAUSE TestProxyWriteTimeout/unknown_size837=== CONT TestPush_OverlappingRootsStoreOneRowPerKey838=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed839=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed840=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed841=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed842=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost843=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost844=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed845=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed846=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed847=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed848=== CONT TestReadRedirectUsesPublicS3URL849=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info850=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info851=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal852=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal853=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key854=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key855=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key856=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key857=== CONT TestReadProxyRangeRequest858=== RUN TestIsValidUploadKey/nar_zst859=== PAUSE TestIsValidUploadKey/nar_zst860=== RUN TestIsValidUploadKey/nar_xz861=== PAUSE TestIsValidUploadKey/nar_xz862=== RUN TestIsValidUploadKey/nar_plain863=== PAUSE TestIsValidUploadKey/nar_plain864=== RUN TestIsValidUploadKey/listing865=== PAUSE TestIsValidUploadKey/listing866=== RUN TestIsValidUploadKey/build_log867=== CONT TestReadRedirectKeepsNarinfoProxied868=== PAUSE TestIsValidUploadKey/build_log869=== RUN TestIsValidUploadKey/build_log_home-manager_file870=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure871=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure872=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart873=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart874=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts875=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts876=== CONT TestReadRedirectNar877=== PAUSE TestIsValidUploadKey/build_log_home-manager_file878=== RUN TestIsValidUploadKey/build_log_plus_in_name879=== PAUSE TestIsValidUploadKey/build_log_plus_in_name880=== RUN TestIsValidUploadKey/build_log_question_mark881=== PAUSE TestIsValidUploadKey/build_log_question_mark882=== RUN TestIsValidUploadKey/build_log_equals883=== PAUSE TestIsValidUploadKey/build_log_equals884=== RUN TestIsValidUploadKey/realisation885=== PAUSE TestIsValidUploadKey/realisation886=== RUN TestIsValidUploadKey/realisation_plus_in_output887=== PAUSE TestIsValidUploadKey/realisation_plus_in_output888=== RUN TestIsValidUploadKey/nix-cache-info889=== PAUSE TestIsValidUploadKey/nix-cache-info890=== RUN TestIsValidUploadKey/index.html891=== PAUSE TestIsValidUploadKey/index.html892=== RUN TestIsValidUploadKey/narinfo_key,_nar_type893=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type894=== RUN TestIsValidUploadKey/nar_key,_narinfo_type895=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type896=== RUN TestIsValidUploadKey/listing_key,_narinfo_type897=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type898=== RUN TestIsValidUploadKey/traversal899=== PAUSE TestIsValidUploadKey/traversal900=== RUN TestIsValidUploadKey/traversal_nar901=== PAUSE TestIsValidUploadKey/traversal_nar902=== RUN TestIsValidUploadKey/absolute903=== PAUSE TestIsValidUploadKey/absolute904=== RUN TestIsValidUploadKey/empty_key905=== PAUSE TestIsValidUploadKey/empty_key906=== RUN TestIsValidUploadKey/unknown_type907=== PAUSE TestIsValidUploadKey/unknown_type908=== CONT TestReadProxyDisabled9092026/11/12 00:05:53 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000910--- PASS: TestSkippedUploadsHandler (1.33s)911=== CONT TestReadProxyRootRedirectsToIndexHTML912--- PASS: TestReadProxyDisabled (2.86s)913=== CONT TestReadProxyConditionalGet9142026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures9152026/11/12 00:05:56 INFO Received push request method=POST path=/api/pushes9162026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures9172026/11/12 00:05:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9182026/11/12 00:05:56 INFO Signed narinfos id=1 count=19192026/11/12 00:05:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9202026/11/12 00:05:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9212026/11/12 00:05:56 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"9222026/11/12 00:05:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9232026/11/12 00:05:56 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"9242026/11/12 00:05:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9252026/11/12 00:05:56 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=19262026/11/12 00:05:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign9272026/11/12 00:05:56 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo928--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (4.24s)929=== CONT TestReadProxyOutlastsServerWriteTimeout930--- PASS: TestReadProxyRootRedirectsToIndexHTML (2.96s)931=== CONT TestReadProxyHead932--- PASS: TestReadRedirectUsesPublicS3URL (4.14s)933=== CONT TestReadProxyInvalidPath9342026/11/12 00:05:56 INFO Received push request method=POST path=/api/pushes9352026/11/12 00:05:56 INFO Received complete push request method=POST path=/api/pushes/1/complete9362026/11/12 00:05:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9372026/11/12 00:05:56 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst938--- PASS: TestCompleteMultipartUnregistered (4.36s)939=== CONT TestReadProxy4049402026/11/12 00:05:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9412026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures9422026/11/12 00:05:56 INFO Received push request method=POST path=/api/pushes943--- PASS: TestReadProxyRangeRequest (4.29s)944=== CONT TestReadProxyNarStreaming945--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (4.31s)946=== CONT TestReadProxyContentNegotiation9472026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures9482026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures9492026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures9502026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures951--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (4.54s)952=== CONT TestReadProxyNarinfoAlreadyDecompressed953=== RUN TestPush_RejectsBadRequests/root_not_in_objects954=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects955=== RUN TestPush_RejectsBadRequests/no_roots956=== PAUSE TestPush_RejectsBadRequests/no_roots957=== RUN TestPush_RejectsBadRequests/no_objects958=== PAUSE TestPush_RejectsBadRequests/no_objects959=== RUN TestPush_RejectsBadRequests/bad_root960=== PAUSE TestPush_RejectsBadRequests/bad_root961=== CONT TestReadProxyNarinfo962--- PASS: TestReadRedirectKeepsNarinfoProxied (4.46s)963=== CONT TestIsValidCachePath964=== RUN TestIsValidCachePath/narinfo965=== PAUSE TestIsValidCachePath/narinfo966=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars967=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars968=== RUN TestIsValidCachePath/nar_zst969=== PAUSE TestIsValidCachePath/nar_zst970=== RUN TestIsValidCachePath/nar_xz971=== PAUSE TestIsValidCachePath/nar_xz972=== RUN TestIsValidCachePath/nar_bz2973=== PAUSE TestIsValidCachePath/nar_bz2974=== RUN TestIsValidCachePath/nar_uncompressed975=== PAUSE TestIsValidCachePath/nar_uncompressed976=== RUN TestIsValidCachePath/ls977=== PAUSE TestIsValidCachePath/ls978=== RUN TestIsValidCachePath/log979=== PAUSE TestIsValidCachePath/log980=== RUN TestIsValidCachePath/realisation981=== PAUSE TestIsValidCachePath/realisation982=== RUN TestIsValidCachePath/nix-cache-info983=== PAUSE TestIsValidCachePath/nix-cache-info984=== RUN TestIsValidCachePath/index.html985=== PAUSE TestIsValidCachePath/index.html986=== RUN TestIsValidCachePath/traversal_parent987=== PAUSE TestIsValidCachePath/traversal_parent988=== RUN TestIsValidCachePath/traversal_in_middle989=== PAUSE TestIsValidCachePath/traversal_in_middle990=== RUN TestIsValidCachePath/invalid_char_e991=== PAUSE TestIsValidCachePath/invalid_char_e992=== RUN TestIsValidCachePath/invalid_char_u993=== PAUSE TestIsValidCachePath/invalid_char_u994=== RUN TestIsValidCachePath/random_path995=== PAUSE TestIsValidCachePath/random_path996=== RUN TestIsValidCachePath/empty997=== PAUSE TestIsValidCachePath/empty998=== RUN TestIsValidCachePath/leading_slash999=== PAUSE TestIsValidCachePath/leading_slash1000=== RUN TestIsValidCachePath/wrong_extension1001=== PAUSE TestIsValidCachePath/wrong_extension1002=== RUN TestIsValidCachePath/short_hash1003=== PAUSE TestIsValidCachePath/short_hash1004=== CONT TestParseSingleRange1005=== RUN TestParseSingleRange/none1006=== PAUSE TestParseSingleRange/none1007=== RUN TestParseSingleRange/unknown_unit1008=== PAUSE TestParseSingleRange/unknown_unit1009=== RUN TestParseSingleRange/multi-range_ignored1010=== PAUSE TestParseSingleRange/multi-range_ignored1011=== RUN TestParseSingleRange/malformed_no_dash1012=== PAUSE TestParseSingleRange/malformed_no_dash1013=== RUN TestParseSingleRange/malformed_both_empty1014=== PAUSE TestParseSingleRange/malformed_both_empty1015=== RUN TestParseSingleRange/malformed_end_before_start1016=== PAUSE TestParseSingleRange/malformed_end_before_start1017=== RUN TestParseSingleRange/closed1018=== PAUSE TestParseSingleRange/closed1019=== RUN TestParseSingleRange/open-ended1020=== PAUSE TestParseSingleRange/open-ended1021=== RUN TestParseSingleRange/end_clamped_to_size1022=== PAUSE TestParseSingleRange/end_clamped_to_size1023=== RUN TestParseSingleRange/suffix1024=== PAUSE TestParseSingleRange/suffix1025=== RUN TestParseSingleRange/suffix_exceeds_size1026=== PAUSE TestParseSingleRange/suffix_exceeds_size1027=== RUN TestParseSingleRange/single_byte1028=== PAUSE TestParseSingleRange/single_byte1029=== RUN TestParseSingleRange/start_past_EOF1030=== PAUSE TestParseSingleRange/start_past_EOF1031=== RUN TestParseSingleRange/start_far_past_EOF1032=== PAUSE TestParseSingleRange/start_far_past_EOF1033=== CONT TestPresentNotReportedWhileGCDeletesClosure1034=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1035=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1036=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1037=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1038=== CONT TestPresentRefreshesOnlyStaleClosures10392026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures10402026/11/12 00:05:56 INFO Received cleanup request method=DELETE path=/api/pending_closures10412026/11/12 00:05:56 INFO Aborted multipart uploads count=0 kept=010422026/11/12 00:05:56 INFO Received uploads request method=POST path=/api/pending_closures10432026/11/12 00:05:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10442026/11/12 00:05:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10452026/11/12 00:05:56 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10462026/11/12 00:05:57 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjU1MTc4OGE3LWM2MjctNGNkMi1iYjRkLWUyYWRiYjMzMzFjOHgxNzk0NDQxOTU2NTM0NzI3ODI2 parts=1010472026/11/12 00:05:57 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10482026/11/12 00:05:57 INFO Completed upload id=1 objects=2 duration=3.220971ms10492026/11/12 00:05:57 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010502026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures10512026/11/12 00:05:57 INFO Starting cleanup of old closures method=DELETE path=/api/closures10522026/11/12 00:05:57 INFO Completed push id=1 objects=4 duration=1.329679085s10532026/11/12 00:05:57 INFO Received push request method=POST path=/api/pushes10542026/11/12 00:05:57 INFO Aborted multipart uploads count=0 kept=010552026/11/12 00:05:57 INFO Received cleanup request method=DELETE path=/api/pending_closures10562026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures10572026/11/12 00:05:57 INFO Aborted multipart uploads count=1 kept=010582026/11/12 00:05:57 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10592026-11-12 00:05:57.741 UTC [460] ERROR: Closure does not exist: id=110602026-11-12 00:05:57.741 UTC [460] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE10612026-11-12 00:05:57.741 UTC [460] STATEMENT: -- name: CommitPendingClosure :one1062 SELECT commit_pending_closure($1::bigint)::bigint1063 1064=== CONT TestPresentReportsOnlyClosureRoots1065--- PASS: TestService_cleanupPendingClosuresHandler (5.70s)10662026/11/12 00:05:57 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjQxY2Q0M2Q5LWMzOTQtNDQ4My1iMjNhLTA5OTk1OWU2ZGQ5YXgxNzk0NDQxOTU2NDM0NTE3NDI5 parts=1010672026/11/12 00:05:57 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10682026/11/12 00:05:57 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLmRkZDFhMjk3LTYxYzctNGM5OC04NGVmLTQxMDdiMmVkNjI0MngxNzk0NDQxOTU2Mjg4NzI2Nzk5 parts=1210692026/11/12 00:05:57 INFO Completed upload id=1 objects=2 duration=2.964889ms10702026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures1071--- PASS: TestCompletedNarNotReofferedAcrossClosures (5.71s)1072=== CONT TestCreatePin_ReservedPins10732026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures10742026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures10752026/11/12 00:05:57 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10762026/11/12 00:05:57 WARN Found objects in DB but missing from S3, will re-upload count=11077--- PASS: TestService_verifyS3Integrity (5.74s)1078=== CONT TestOrphanedObjectsGCStressTest10792026/11/12 00:05:57 INFO Aborted multipart uploads count=0 kept=01080--- PASS: TestService_Rustfstest (5.75s)1081=== CONT TestOrphanedObjectsGC10822026/11/12 00:05:57 WARN Force mode enabled - objects will be deleted immediately without grace period10832026/11/12 00:05:57 INFO Received push request method=POST path=/api/pushes10842026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures10852026/11/12 00:05:57 INFO Received push request method=POST path=/api/pushes10862026/11/12 00:05:57 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10872026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures1088--- PASS: TestPresignedUploadRegisteredBeforeCommit (5.85s)1089=== CONT TestObjectStatsUpdatedOncePerCommit10902026/11/12 00:05:57 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35173/oidc1091--- PASS: TestReadRedirectNar (4.56s)1092=== CONT TestObjectStatsTrigger10932026/11/12 00:05:57 INFO Received push request method=POST path=/api/pushes10942026/11/12 00:05:57 INFO Received uploads request method=POST path=/api/pending_closures10952026/11/12 00:05:57 INFO Received complete push request method=POST path=/api/pushes/1/complete10962026-11-12 00:05:58.762 UTC [556] ERROR: could not obtain lock on row in relation "closures"10972026-11-12 00:05:58.762 UTC [556] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10982026-11-12 00:05:58.765 UTC [556] ERROR: could not obtain lock on row in relation "closures"10992026-11-12 00:05:58.765 UTC [556] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11002026-11-12 00:05:58.765 UTC [556] ERROR: could not obtain lock on row in relation "closures"11012026-11-12 00:05:58.765 UTC [556] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11022026-11-12 00:05:58.767 UTC [556] ERROR: could not obtain lock on row in relation "closures"11032026-11-12 00:05:58.767 UTC [556] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11042026-11-12 00:05:58.768 UTC [556] ERROR: could not obtain lock on row in relation "closures"11052026-11-12 00:05:58.768 UTC [556] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11062026-11-12 00:05:58.769 UTC [560] ERROR: Push does not exist: id=111072026-11-12 00:05:58.769 UTC [560] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE11082026-11-12 00:05:58.769 UTC [560] STATEMENT: SELECT commit_push($1)1109--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (6.71s)1110=== CONT TestPendingCleanupUsesOneCutoff11112026/11/12 00:05:58 INFO Completed push id=1 objects=4 duration=794.910644ms11122026/11/12 00:05:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01113--- PASS: TestPush_RepeatedRootCommitsOnce (6.62s)1114=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded1115--- PASS: TestPush_RootsAreLockedInByteOrder (6.71s)1116=== CONT TestMultipartCleanup1117--- PASS: TestReadProxyHead (2.45s)1118=== CONT TestServerTLSConfig1119=== RUN TestServerTLSConfig/no_client_CA1120=== PAUSE TestServerTLSConfig/no_client_CA1121=== RUN TestServerTLSConfig/missing_CA_file1122=== PAUSE TestServerTLSConfig/missing_CA_file1123=== RUN TestServerTLSConfig/not_a_PEM_file1124=== PAUSE TestServerTLSConfig/not_a_PEM_file1125=== CONT TestService_NativeMTLS11262026/11/12 00:05:58 INFO Vacuumed table table=pending_closures11272026/11/12 00:05:58 INFO Vacuumed table table=pending_objects11282026/11/12 00:05:58 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11292026/11/12 00:05:58 INFO Vacuumed table table=multipart_uploads11302026/11/12 00:05:58 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjlhYjg4ZjRkLWExYTUtNGNlNi04ZGQ1LTM0NTUyZTA4YmRkM3gxNzk0NDQxOTU3OTkwNTA1NDE111312026/11/12 00:05:58 INFO Vacuumed table table=closures11322026/11/12 00:05:58 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=011332026/11/12 00:05:58 INFO Vacuumed table table=objects11342026/11/12 00:05:58 INFO Vacuumed table table=pending_closures11352026/11/12 00:05:58 INFO Vacuumed table table=pending_objects11362026/11/12 00:05:58 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjlhYjg4ZjRkLWExYTUtNGNlNi04ZGQ1LTM0NTUyZTA4YmRkM3gxNzk0NDQxOTU3OTkwNTA1NDE1 parts=111372026/11/12 00:05:58 INFO Vacuumed table table=multipart_uploads1138--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (6.64s)1139=== CONT TestMetricsCommit11402026/11/12 00:05:58 INFO Vacuumed table table=closures11412026/11/12 00:05:58 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001142--- PASS: TestService_createPendingClosureHandler (6.78s)1143=== CONT TestMetricsInventory11442026/11/12 00:05:58 INFO Vacuumed table table=objects11452026/11/12 00:05:58 INFO Received complete push request method=POST path=/api/pushes/2/complete11462026/11/12 00:05:58 INFO Completed push id=2 objects=4 duration=5.400492ms1147--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (6.77s)1148=== CONT TestNARDeduplicationMetadataUploadBug1149--- PASS: TestReadProxyNarStreaming (2.36s)1150=== CONT TestCreatePendingClosureRejectsOversizedNAR11512026/11/12 00:05:58 INFO Received uploads request method=POST path=/api/pending_closures1152--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1153=== CONT TestCacheConfigHandlerMaxNarSize1154--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1155=== CONT TestEnsureAbortUploadsRule1156--- PASS: TestReadProxyContentNegotiation (2.35s)1157=== CONT TestGenerateLandingPage1158--- PASS: TestGenerateLandingPage (0.01s)1159=== CONT TestService_readinessHandler1160--- PASS: TestReadProxy404 (2.51s)1161=== CONT TestService_healthCheckHandler1162--- PASS: TestReadProxyConditionalGet (2.69s)1163=== CONT TestGracefulShutdownDrainsInflight11642026/11/12 00:05:58 INFO Starting HTTP server address=127.0.0.1:4500111652026/11/12 00:05:58 INFO Shutdown signal received, draining in-flight requests timeout=10s1166--- PASS: TestReadProxyNarinfoAlreadyDecompressed (2.36s)1167=== CONT TestGCTaskStore_Fail1168--- PASS: TestGCTaskStore_Fail (0.00s)1169=== CONT TestGCTaskStore_PhaseUpdates1170--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1171=== CONT TestGCTaskStore_CompletedAllowsNewTask1172--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1173=== CONT TestGCTaskStore_GetReturnsLatest1174--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1175=== CONT TestGCTaskStore_GetEmpty1176--- PASS: TestGCTaskStore_GetEmpty (0.00s)1177=== CONT TestGCTaskStore_ConflictDifferentParams1178--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1179=== CONT TestGCTaskStore_DeduplicateSameParams1180--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1181=== CONT TestGCTaskStore_StartNew1182--- PASS: TestGCTaskStore_StartNew (0.00s)1183=== CONT TestGCEndsOnShutdown1184=== RUN TestGCEndsOnShutdown/before_the_run1185=== PAUSE TestGCEndsOnShutdown/before_the_run1186=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1187=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1188=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1189=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1190=== CONT TestCommitRacingPendingCleanupKeepsObjects11912026/11/12 00:05:58 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1192--- PASS: TestReadProxyInvalidPath (2.63s)1193=== CONT TestPushWaitsForSweepHoldingRow1194--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1195=== CONT TestSweepRechecksPendingAfterLocking11962026/11/12 00:05:58 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjJhNDllMGU2LTNkN2QtNGY5YS04ZWYwLTQxNDYwYTU3ZmZlY3gxNzk0NDQxOTU3NzI2MjI5MDEz parts=121197--- PASS: TestPresentReportsOnlyClosureRoots (1.31s)1198=== CONT TestGetExistingObjectsLocksRows1199--- PASS: TestObjectStatsUpdatedOncePerCommit (1.16s)1200=== CONT TestRegisterCompletedObjectNeedsPendingRow1201--- PASS: TestPresentRefreshesOnlyStaleClosures (2.41s)1202=== CONT TestSweepSkipsRowsLockedByPush12032026/11/12 00:05:59 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1204--- PASS: TestReadProxyNarinfo (2.65s)1205=== CONT TestFailedS3DeleteKeepsTombstone12062026/11/12 00:05:59 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12072026/11/12 00:05:59 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjY2OWY5NzZkLTAwMTAtNDUyMy04YzRlLWVkNzA4MzkwMjU5NHgxNzk0NDQxOTU3NzM4OTY2OTE1 parts=121208--- PASS: TestSharedNarBothUploadsComplete (7.45s)1209=== CONT TestForceGCDuringPushOffersSweptObject1210=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1211=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1212=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1213=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1214=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection1215=== NAME TestOrphanedObjectsGC1216 orphaned_objects_gc_test.go:287: GC Test Summary:1217 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1218 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1219 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1220 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1221 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1222--- PASS: TestOrphanedObjectsGC (2.69s)1223=== CONT TestGCSweepDeliversEachKeyOnce12242026/11/12 00:06:00 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket43 days=21225--- PASS: TestEnsureAbortUploadsRule (1.66s)1226=== CONT TestTombstonedObjectOfferedWithoutWaiting1227--- PASS: TestGetExistingObjectsLocksRows (1.52s)1228=== CONT TestGCSweepSkipsPendingObjects1229--- PASS: TestService_healthCheckHandler (1.70s)1230=== CONT TestDeduplicatedObjectsRecordedAsPending12312026/11/12 00:06:00 INFO Aborted multipart uploads count=0 kept=012322026/11/12 00:06:00 WARN readiness check failed error="closed pool"1233--- PASS: TestService_readinessHandler (1.84s)1234=== CONT TestPushDedupSurvivesConcurrentGC12352026/11/12 00:06:00 INFO Received uploads request method=POST path=/api/pending_closures12362026/11/12 00:06:00 INFO Received uploads request method=POST path=/api/pending_closures12372026/11/12 00:06:00 INFO Aborted multipart uploads count=0 kept=012382026/11/12 00:06:00 INFO Aborted multipart uploads count=0 kept=012392026/11/12 00:06:00 INFO Received push request method=POST path=/api/pushes12402026/11/12 00:06:00 INFO Received complete push request method=POST path=/api/pushes/1/complete12412026/11/12 00:06:00 INFO Completed push id=1 objects=4 duration=10.989985ms12422026/11/12 00:06:00 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete1243--- PASS: TestRegisterCompletedObjectNeedsPendingRow (1.89s)1244=== CONT TestGCMetrics12452026/11/12 00:06:00 INFO Completed upload id=2 objects=2 duration=2.608625ms1246--- PASS: TestMetricsCommit (2.14s)1247=== CONT TestGCBugBareHashReferences1248--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (1.56s)1249=== CONT TestGCAdvisoryLockBlocksConcurrentRun12502026/11/12 00:06:01 WARN Force mode enabled - objects will be deleted immediately without grace period12512026/11/12 00:06:01 WARN Force mode enabled - objects will be deleted immediately without grace period12522026/11/12 00:06:01 WARN Force mode enabled - objects will be deleted immediately without grace period12532026/11/12 00:06:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012542026/11/12 00:06:01 INFO Vacuumed table table=pending_closures12552026/11/12 00:06:01 INFO Vacuumed table table=pending_objects12562026/11/12 00:06:01 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."12572026/11/12 00:06:01 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."12582026/11/12 00:06:01 INFO Vacuumed table table=multipart_uploads12592026/11/12 00:06:01 INFO Vacuumed table table=closures12602026/11/12 00:06:01 INFO Vacuumed table table=objects1261--- PASS: TestFailedS3DeleteKeepsTombstone (1.99s)1262=== CONT TestLeadEndsOnShutdown12632026/11/12 00:06:01 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012642026/11/12 00:06:01 INFO Aborted multipart uploads count=0 kept=012652026/11/12 00:06:01 INFO Vacuumed table table=pending_closures12662026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures12672026/11/12 00:06:01 INFO Vacuumed table table=pending_objects12682026/11/12 00:06:01 INFO Vacuumed table table=multipart_uploads12692026/11/12 00:06:01 INFO Vacuumed table table=closures12702026/11/12 00:06:01 INFO Vacuumed table table=objects1271=== NAME TestNARDeduplicationMetadataUploadBug1272 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug2784015754/001/store/mjdyvkca62jg5slb0v2byk6zad2ab607-file1.txt1273--- PASS: TestSweepRechecksPendingAfterLocking (2.28s)1274=== CONT TestLeadGenerationGrowsWithEveryGrant12752026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures12762026/11/12 00:06:01 INFO Received cleanup request method=DELETE path=/api/pending_closures1277--- PASS: TestMetricsInventory (2.49s)1278=== CONT TestLeadIncumbentAnnouncesAtOnce12792026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures12802026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures1281=== NAME TestOrphanedObjectsGCStressTest1282 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains12832026/11/12 00:06:01 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12842026/11/12 00:06:01 WARN Refused reserved pin name=worker-x86_64-linux12852026/11/12 00:06:01 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux12862026/11/12 00:06:01 INFO Received create pin request method=POST path=/api/pins/my-app12872026/11/12 00:06:01 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1288--- PASS: TestCreatePin_ReservedPins (3.62s)1289=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent1290--- PASS: TestObjectStatsTrigger (3.48s)1291=== CONT TestLeadElectsOneAndHandsOver12922026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures12932026/11/12 00:06:01 WARN mTLS auth: subject not in bound subjects subject="CN=reader"12942026/11/12 00:06:01 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1295--- PASS: TestService_NativeMTLS (2.65s)1296=== CONT TestConnectWaitsForAPeerMigration1297--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (2.67s)1298=== CONT TestResolveDBConnectionString1299=== RUN TestResolveDBConnectionString/flag_wins1300=== PAUSE TestResolveDBConnectionString/flag_wins1301=== RUN TestResolveDBConnectionString/file_when_flag_empty1302=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1303=== RUN TestResolveDBConnectionString/missing_file_is_an_error1304=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1305=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1306=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1307=== RUN TestResolveDBConnectionString/nothing_configured1308=== PAUSE TestResolveDBConnectionString/nothing_configured1309=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock13102026/11/12 00:06:01 INFO Received cleanup request method=DELETE path=/api/pending_closures13112026/11/12 00:06:01 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket58/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""13122026/11/12 00:06:01 INFO Aborted multipart uploads count=0 kept=113132026/11/12 00:06:01 INFO Received cleanup request method=DELETE path=/api/pending_closures13142026/11/12 00:06:01 INFO Aborted multipart uploads count=1 kept=013152026/11/12 00:06:01 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1316--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.42s)1317=== CONT TestDeleteClosuresRacingPins1318--- PASS: TestMultipartCleanup (2.71s)1319=== CONT TestClientFallsBackToClosures13202026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures1321--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.90s)1322=== CONT TestClientPushesUseOnePush13232026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures1324--- PASS: TestTombstonedObjectOfferedWithoutWaiting (1.01s)1325=== CONT TestPinProtectsFromGC13262026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures13272026/11/12 00:06:01 INFO Aborted multipart uploads count=0 kept=013282026/11/12 00:06:01 INFO Aborted multipart uploads count=0 kept=013292026/11/12 00:06:01 INFO Received uploads request method=POST path=/api/pending_closures13302026/11/12 00:06:01 INFO Aborted multipart uploads count=0 kept=013312026/11/12 00:06:01 INFO Aborted multipart uploads count=0 kept=01332--- PASS: TestPushWaitsForSweepHoldingRow (2.84s)1333=== CONT TestClientWithDependencies13342026/11/12 00:06:01 WARN Rate limiter enabled after throttle name=s3-test rate=513352026/11/12 00:06:01 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1336=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1337 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101338 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001339--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (9.79s)1340=== CONT TestClientMultipleUploads1341--- PASS: TestGCBugBareHashReferences (1.49s)1342=== CONT TestClientIntegration13432026/11/12 00:06:02 WARN Force mode enabled - objects will be deleted immediately without grace period13442026/11/12 00:06:02 WARN Force mode enabled - objects will be deleted immediately without grace period13452026/11/12 00:06:02 WARN Force mode enabled - objects will be deleted immediately without grace period1346--- PASS: TestCommitRacingPendingCleanupKeepsObjects (3.50s)1347=== CONT TestClientErrorHandling13482026/11/12 00:06:02 WARN Force mode enabled - objects will be deleted immediately without grace period1349=== RUN TestClientErrorHandling/InvalidStorePath1350=== PAUSE TestClientErrorHandling/InvalidStorePath1351=== RUN TestClientErrorHandling/InvalidAuthToken1352=== PAUSE TestClientErrorHandling/InvalidAuthToken1353=== RUN TestClientErrorHandling/ServerNotAvailable1354=== PAUSE TestClientErrorHandling/ServerNotAvailable1355=== CONT TestClientCADerivations13562026/11/12 00:06:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013572026/11/12 00:06:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013582026/11/12 00:06:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013592026/11/12 00:06:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013602026/11/12 00:06:02 INFO Vacuumed table table=pending_closures13612026/11/12 00:06:02 INFO Vacuumed table table=pending_closures13622026/11/12 00:06:02 INFO Vacuumed table table=pending_objects13632026/11/12 00:06:02 INFO Vacuumed table table=pending_objects13642026/11/12 00:06:02 INFO Vacuumed table table=multipart_uploads13652026/11/12 00:06:02 INFO Vacuumed table table=multipart_uploads13662026/11/12 00:06:02 INFO Vacuumed table table=closures13672026/11/12 00:06:02 INFO Vacuumed table table=closures13682026/11/12 00:06:02 INFO Vacuumed table table=objects1369=== NAME TestOrphanedObjectsGCStressTest1370 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion13712026/11/12 00:06:02 INFO Vacuumed table table=pending_closures13722026/11/12 00:06:02 INFO Vacuumed table table=pending_closures13732026/11/12 00:06:02 INFO Vacuumed table table=objects13742026/11/12 00:06:02 INFO Vacuumed table table=pending_objects13752026/11/12 00:06:02 INFO Vacuumed table table=multipart_uploads13762026/11/12 00:06:02 INFO Vacuumed table table=closures13772026/11/12 00:06:02 INFO Vacuumed table table=pending_objects13782026/11/12 00:06:02 INFO Vacuumed table table=multipart_uploads13792026/11/12 00:06:02 INFO Vacuumed table table=objects13802026/11/12 00:06:02 INFO Vacuumed table table=closures1381--- PASS: TestGCMetrics (1.53s)1382=== CONT TestCacheStatsHandler13832026/11/12 00:06:02 INFO Vacuumed table table=objects13842026/11/12 00:06:02 INFO Aborted multipart uploads count=0 kept=01385--- PASS: TestGCSweepSkipsPendingObjects (1.92s)1386=== CONT TestCacheConfigHandler1387=== RUN TestCacheConfigHandler/full_config,_no_issuer1388=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1389=== RUN TestCacheConfigHandler/no_cache_url_configured1390=== PAUSE TestCacheConfigHandler/no_cache_url_configured1391=== RUN TestCacheConfigHandler/no_signing_keys1392=== PAUSE TestCacheConfigHandler/no_signing_keys1393=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1394=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1395=== CONT TestService_ReadScope_PublicByDefault13962026/11/12 00:06:02 INFO Aborted multipart uploads count=0 kept=013972026/11/12 00:06:02 WARN Force mode enabled - objects will be deleted immediately without grace period13982026/11/12 00:06:02 WARN Force mode enabled - objects will be deleted immediately without grace period13992026/11/12 00:06:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=014002026/11/12 00:06:02 INFO Vacuumed table table=pending_closures14012026/11/12 00:06:02 INFO Vacuumed table table=pending_objects14022026/11/12 00:06:02 INFO Vacuumed table table=multipart_uploads14032026/11/12 00:06:02 INFO Vacuumed table table=closures14042026/11/12 00:06:02 INFO Vacuumed table table=objects1405--- PASS: TestPushDedupSurvivesConcurrentGC (1.80s)1406=== CONT TestService_RequireScope_OIDC14072026/11/12 00:06:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014082026/11/12 00:06:02 INFO Vacuumed table table=pending_closures14092026/11/12 00:06:02 INFO Vacuumed table table=pending_objects14102026/11/12 00:06:02 INFO Vacuumed table table=multipart_uploads14112026/11/12 00:06:02 INFO Vacuumed table table=closures14122026/11/12 00:06:02 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=014132026/11/12 00:06:02 INFO Vacuumed table table=pending_closures14142026/11/12 00:06:02 INFO Vacuumed table table=pending_objects14152026/11/12 00:06:02 INFO Vacuumed table table=multipart_uploads14162026/11/12 00:06:02 INFO Vacuumed table table=closures14172026/11/12 00:06:02 INFO Vacuumed table table=objects1418--- PASS: TestGCSweepDeliversEachKeyOnce (2.06s)1419=== CONT TestService_AuthMiddleware_OIDC14202026/11/12 00:06:02 INFO Vacuumed table table=objects1421--- PASS: TestSweepSkipsRowsLockedByPush (3.45s)1422=== CONT TestService_ReadAuthMiddleware14232026/11/12 00:06:02 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45117/oidc14242026/11/12 00:06:02 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:32833/oidc14252026/11/12 00:06:02 INFO lead: acquired remote=192.0.2.1:1234 generation=114262026/11/12 00:06:02 INFO lead: acquired remote=192.0.2.1:1234 generation=114272026/11/12 00:06:02 INFO lead: acquired remote=192.0.2.1:1234 generation=114282026/11/12 00:06:02 INFO lead: released remote=192.0.2.1:1234 generation=11429--- PASS: TestLeadIncumbentAnnouncesAtOnce (1.46s)1430=== CONT TestService_AuthMiddleware_MTLSBoundSubjects14312026/11/12 00:06:02 INFO lead: acquired remote=192.0.2.1:1234 generation=114322026/11/12 00:06:02 INFO lead: acquired remote=192.0.2.1:1234 generation=11433--- PASS: TestService_ReadScope_PublicByDefault (0.39s)1434=== CONT TestService_AuthMiddleware_MTLSProxyHeader1435--- PASS: TestCacheStatsHandler (0.44s)1436=== CONT TestService_AuthMiddleware1437--- PASS: TestReadProxyOutlastsServerWriteTimeout (6.84s)1438=== CONT TestPush_CompleteCommitsEveryRoot14392026/11/12 00:06:03 INFO Received push request method=POST path=/api/pushes1440=== NAME TestClientMultipleUploads1441 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1521147515/001/store/s771hhaggz5q9m16c09i1pz65yilg01x-test-file-0.txt14422026/11/12 00:06:03 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)14432026/11/12 00:06:03 INFO Uploading mjdyvkca62jg5slb0v2byk6zad2ab607-file1.txt (160B)14442026/11/12 00:06:03 WARN Failed to register uploaded object key=mjdyvkca62jg5slb0v2byk6zad2ab607.ls error="server returned 404: 404 page not found\n"14452026/11/12 00:06:03 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14462026/11/12 00:06:03 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"14472026/11/12 00:06:03 INFO Signed narinfos id=1 count=114482026/11/12 00:06:03 INFO Uploading 1 narinfos1449--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (2.16s)1450=== CONT TestClientSharedPathCommittedMidPush14512026/11/12 00:06:03 WARN Failed to register uploaded object key=mjdyvkca62jg5slb0v2byk6zad2ab607.narinfo error="server returned 404: 404 page not found\n"14522026/11/12 00:06:03 INFO Received complete push request method=POST path=/api/pushes/1/complete14532026/11/12 00:06:03 INFO Completed push id=1 objects=3 duration=7.573452ms14542026/11/12 00:06:03 INFO Upload complete. (1.116s)1455=== NAME TestNARDeduplicationMetadataUploadBug1456 metadata_upload_test.go:54: Retrieved narinfo from S3:1457 StorePath: /build/TestNARDeduplicationMetadataUploadBug2784015754/001/store/mjdyvkca62jg5slb0v2byk6zad2ab607-file1.txt1458 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1459 Compression: zstd1460 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1461 NarSize: 1601462 References: 1463 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1464 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1465 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1466 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1467--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.75s)1468=== CONT TestPendingClosureWriteTimeout/empty1469=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline1470=== NAME TestOrphanedObjectsGCStressTest1471 orphaned_objects_gc_test.go:503: Stress test completed successfully:1472 orphaned_objects_gc_test.go:504: - Active objects preserved: 201473 orphaned_objects_gc_test.go:505: - Objects deleted: 2101474 orphaned_objects_gc_test.go:506: - Total GC'd: 2101475--- PASS: TestOrphanedObjectsGCStressTest (5.86s)1476=== CONT TestPendingClosureWriteTimeout/670k_objects1477=== CONT TestPendingClosureWriteTimeout/400_objects1478=== CONT TestProxyWriteTimeout/narinfo1479=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1480--- PASS: TestService_ReadAuthMiddleware (1.11s)1481=== CONT TestProxyWriteTimeout/unknown_size1482=== CONT TestProxyWriteTimeout/10_GiB_nar1483=== CONT TestProxyWriteTimeout/1_GiB_nar1484=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info1485--- PASS: TestProxyWriteTimeout (0.16s)1486 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1487 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1488 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1489 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)14902026/11/12 00:06:03 INFO Received uploads request method=POST path=/1491=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed1492=== RUN TestService_RequireScope_OIDC/builder_may_write1493=== PAUSE TestService_RequireScope_OIDC/builder_may_write1494=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1495=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1496=== RUN TestService_RequireScope_OIDC/ops_may_admin1497=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1498=== RUN TestService_RequireScope_OIDC/ops_may_not_write1499=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1500=== RUN TestService_RequireScope_OIDC/reader_may_not_write1501=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1502=== RUN TestService_RequireScope_OIDC/static_token_may_admin1503=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1504=== RUN TestService_RequireScope_OIDC/static_token_may_write1505=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1506=== RUN TestService_RequireScope_OIDC/reader_may_read1507=== PAUSE TestService_RequireScope_OIDC/reader_may_read1508=== RUN TestService_RequireScope_OIDC/writer_implies_read1509=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1510=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1511=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1512=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1513=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1514=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1515=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1516=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1517=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1518=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1519=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1520=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed15212026/11/12 00:06:03 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"15222026/11/12 00:06:03 WARN mTLS auth: bound subjects configured but subject DN unavailable15232026/11/12 00:06:03 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1524--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.93s)1525=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1526=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1527=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1528=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1529=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1530=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1531=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1532=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1533=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1534=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed15352026/11/12 00:06:03 INFO Received push request method=POST path=/api/pushes15362026/11/12 00:06:03 INFO Received complete push request method=POST path=/api/pushes/1/complete15372026/11/12 00:06:03 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1538--- PASS: TestService_AuthMiddleware (0.89s)1539=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key15402026/11/12 00:06:03 INFO Received request for more parts method=POST path=/1541=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key15422026/11/12 00:06:03 INFO Received complete multipart upload request method=POST path=/1543=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal15442026/11/12 00:06:03 INFO Received uploads request method=POST path=/1545--- PASS: TestUploadHandlersRejectInvalidKeys (0.16s)1546 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1547 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1548 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1549 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1550=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure15512026/11/12 00:06:03 INFO Received uploads request method=POST path=/15522026/11/12 00:06:03 INFO lead: released remote=192.0.2.1:1234 generation=115532026/11/12 00:06:03 INFO lead: acquired remote=192.0.2.1:1234 generation=215542026/11/12 00:06:03 INFO lead: released remote=192.0.2.1:1234 generation=115552026/11/12 00:06:03 INFO lead: released remote=192.0.2.1:1234 generation=11556--- PASS: TestLeadEndsOnShutdown (2.67s)1557=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts15582026/11/12 00:06:03 INFO Received request for more parts method=POST path=/15592026/11/12 00:06:03 INFO lead: acquired remote=192.0.2.1:1234 generation=215602026/11/12 00:06:04 INFO Completed push id=1 objects=6 duration=496.355676ms1561--- PASS: TestPush_CompleteCommitsEveryRoot (1.16s)1562=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart15632026/11/12 00:06:04 INFO Received complete multipart upload request method=POST path=/1564=== NAME TestClientMultipleUploads1565 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1521147515/001/store/r8li0hy34x3f1i7ag033qrlcyhkl3sd9-test-file-1.txt1566=== NAME TestClientIntegration1567 client_integration_test.go:292: Created store path: /build/TestClientIntegration19819256/002/store/g7aqzqal6d4jg000hzx3z0vi04nfg4wb-test-file.txt1568=== NAME TestNARDeduplicationMetadataUploadBug1569 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug2784015754/001/store/2mbjdabmkmr82719jlgqhamlph26biib-file2.txt15702026/11/12 00:06:04 INFO lead: released remote=192.0.2.1:1234 generation=11571=== NAME TestPinProtectsFromGC1572 client_integration_test.go:791: Pinned store path: /build/TestPinProtectsFromGC3432879905/001/store/3h6q7zgkcpgq2zx3cbrgg3b2l1k132wm-pinned-file.txt1573 client_integration_test.go:792: Unpinned store path: /build/TestPinProtectsFromGC3432879905/001/store/d1fwamsf4smjc32hl6km16wb3ah6q5q7-unpinned-file.txt1574=== NAME TestClientWithDependencies1575 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies2492912984/001/store/rhlrqpphg53mam3clz52px9iyl7hc0d3-test-script1576=== CONT TestIsValidUploadKey/narinfo1577=== CONT TestIsValidUploadKey/realisation_plus_in_output1578=== CONT TestIsValidUploadKey/empty_key1579=== CONT TestIsValidUploadKey/absolute1580=== CONT TestIsValidUploadKey/traversal_nar1581=== CONT TestIsValidUploadKey/traversal15822026/11/12 00:06:04 INFO lead: acquired remote=192.0.2.1:1234 generation=21583=== CONT TestIsValidUploadKey/unknown_type1584=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1585=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1586=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1587=== CONT TestIsValidUploadKey/nix-cache-info1588=== CONT TestIsValidUploadKey/build_log_home-manager_file1589=== CONT TestIsValidUploadKey/realisation1590=== CONT TestIsValidUploadKey/build_log_equals1591=== CONT TestIsValidUploadKey/build_log_question_mark1592=== CONT TestIsValidUploadKey/build_log_plus_in_name1593=== CONT TestIsValidUploadKey/nar_plain1594=== CONT TestIsValidUploadKey/index.html1595=== CONT TestIsValidUploadKey/listing1596=== CONT TestIsValidUploadKey/nar_xz1597=== CONT TestIsValidUploadKey/build_log1598=== CONT TestPush_RejectsBadRequests/root_not_in_objects1599=== CONT TestIsValidUploadKey/nar_zst1600=== CONT TestPush_RejectsBadRequests/no_objects16012026/11/12 00:06:04 INFO Received push request method=POST path=/api/pushes16022026/11/12 00:06:04 INFO Received push request method=POST path=/api/pushes1603--- PASS: TestIsValidUploadKey (1.32s)1604 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1605 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1606 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1607 --- PASS: TestIsValidUploadKey/absolute (0.00s)1608 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1609 --- PASS: TestIsValidUploadKey/traversal (0.00s)1610 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1611 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1612 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1613 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1614 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1615 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1616 --- PASS: TestIsValidUploadKey/realisation (0.00s)1617 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1618 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1619 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1620 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1621 --- PASS: TestIsValidUploadKey/index.html (0.00s)1622 --- PASS: TestIsValidUploadKey/listing (0.00s)1623 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1624 --- PASS: TestIsValidUploadKey/build_log (0.00s)1625 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1626=== CONT TestPush_RejectsBadRequests/no_roots16272026/11/12 00:06:04 INFO Received push request method=POST path=/api/pushes1628=== CONT TestPush_RejectsBadRequests/bad_root16292026/11/12 00:06:04 INFO Received push request method=POST path=/api/pushes1630=== CONT TestIsValidCachePath/narinfo1631=== CONT TestIsValidCachePath/short_hash1632=== CONT TestIsValidCachePath/leading_slash1633--- PASS: TestPush_RejectsBadRequests (4.54s)1634 --- PASS: TestPush_RejectsBadRequests/no_objects (0.00s)1635 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.00s)1636 --- PASS: TestPush_RejectsBadRequests/no_roots (0.00s)1637 --- PASS: TestPush_RejectsBadRequests/bad_root (0.00s)1638=== CONT TestIsValidCachePath/empty1639=== CONT TestIsValidCachePath/wrong_extension1640=== CONT TestIsValidCachePath/invalid_char_u1641=== CONT TestIsValidCachePath/invalid_char_e1642=== CONT TestIsValidCachePath/traversal_in_middle1643=== CONT TestIsValidCachePath/traversal_parent1644=== CONT TestIsValidCachePath/random_path1645=== CONT TestIsValidCachePath/nix-cache-info1646=== CONT TestIsValidCachePath/realisation1647=== CONT TestIsValidCachePath/log1648=== CONT TestIsValidCachePath/ls1649=== CONT TestIsValidCachePath/nar_uncompressed1650=== CONT TestIsValidCachePath/nar_bz21651=== CONT TestIsValidCachePath/nar_xz1652=== CONT TestIsValidCachePath/nar_zst1653=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1654=== CONT TestIsValidCachePath/index.html1655--- PASS: TestIsValidCachePath (0.00s)1656 --- PASS: TestIsValidCachePath/narinfo (0.00s)1657 --- PASS: TestIsValidCachePath/short_hash (0.00s)1658 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1659 --- PASS: TestIsValidCachePath/empty (0.00s)1660 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1661 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1662 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1663 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1664 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1665 --- PASS: TestIsValidCachePath/random_path (0.00s)1666 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1667 --- PASS: TestIsValidCachePath/realisation (0.00s)1668 --- PASS: TestIsValidCachePath/log (0.00s)1669 --- PASS: TestIsValidCachePath/ls (0.00s)1670 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1671 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1672 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1673 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1674 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1675 --- PASS: TestIsValidCachePath/index.html (0.00s)1676=== CONT TestParseSingleRange/none1677=== CONT TestParseSingleRange/start_far_past_EOF1678=== CONT TestParseSingleRange/start_past_EOF1679=== CONT TestParseSingleRange/suffix_exceeds_size1680=== CONT TestParseSingleRange/suffix1681=== CONT TestParseSingleRange/end_clamped_to_size1682=== CONT TestParseSingleRange/open-ended1683=== CONT TestParseSingleRange/closed1684=== CONT TestParseSingleRange/malformed_end_before_start1685=== CONT TestParseSingleRange/malformed_both_empty1686=== CONT TestParseSingleRange/malformed_no_dash1687=== CONT TestParseSingleRange/multi-range_ignored1688=== CONT TestParseSingleRange/single_byte1689=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1690=== CONT TestParseSingleRange/unknown_unit1691=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1692--- PASS: TestParseSingleRange (0.00s)1693 --- PASS: TestParseSingleRange/none (0.00s)1694 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1695 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1696 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1697 --- PASS: TestParseSingleRange/suffix (0.00s)1698 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1699 --- PASS: TestParseSingleRange/open-ended (0.00s)1700 --- PASS: TestParseSingleRange/closed (0.00s)1701 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1702 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1703 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1704 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1705 --- PASS: TestParseSingleRange/single_byte (0.00s)1706 --- PASS: TestParseSingleRange/unknown_unit (0.00s)17072026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17082026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17092026/11/12 00:06:04 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=217102026/11/12 00:06:04 INFO lead: acquired remote=192.0.2.1:1234 generation=317112026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17122026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17132026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17142026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17152026/11/12 00:06:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17162026/11/12 00:06:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17172026/11/12 00:06:04 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjRhZTk1ODllLTk3N2EtNDRjMy05YzFjLTYwNWFhNDViYTI1ZHgxNzk0NDQxOTY0MzgzMDc5ODA1 parts=1017182026/11/12 00:06:04 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000200000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLmNiZjExMTA1LTc1N2EtNGYzNS1iNjhkLTcwOGNkNTc5YzkzYngxNzk0NDQxOTY0NDAzNTYyMjYw parts=1017192026/11/12 00:06:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17202026/11/12 00:06:04 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17212026/11/12 00:06:04 INFO Completed upload id=1 objects=3 duration=2.713925ms17222026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17232026/11/12 00:06:04 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17242026/11/12 00:06:04 INFO Completed upload id=1 objects=3 duration=4.563823ms17252026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures1726=== CONT TestServerTLSConfig/no_client_CA1727=== CONT TestServerTLSConfig/not_a_PEM_file1728=== CONT TestServerTLSConfig/missing_CA_file1729=== CONT TestGCEndsOnShutdown/before_the_run1730--- PASS: TestServerTLSConfig (0.00s)1731 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1732 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1733 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)17342026/11/12 00:06:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17352026/11/12 00:06:04 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjdlODVlZTg4LWMzOWItNDA2Yi1hYzM2LTRjNmY0NjM4NTI3Y3gxNzk0NDQxOTY0NDUwNjU1NTA5 parts=1017362026/11/12 00:06:04 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17372026/11/12 00:06:04 INFO Completed upload id=1 objects=3 duration=13.705511ms17382026/11/12 00:06:04 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjVjNTQ4MWMxLTZhZGQtNDY0YS1hNTljLTEwMzg3YTcwZjM4Y3gxNzk0NDQxOTY0NDc5ODU0NzI3 parts=1017392026/11/12 00:06:04 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17402026/11/12 00:06:04 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17412026/11/12 00:06:04 INFO lead: released remote=192.0.2.1:1234 generation=217422026/11/12 00:06:04 INFO Completed upload id=1 objects=3 duration=5.997597ms17432026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17442026/11/12 00:06:04 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjZkZGQxMzU5LTJiZWYtNDM5ZC1iOWFiLTRiMzY1NmZjNzIwYXgxNzk0NDQxOTY0NTMzNjc0MTYx parts=1017452026/11/12 00:06:04 INFO Received uploads request method=POST path=/api/pending_closures17462026/11/12 00:06:04 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17472026/11/12 00:06:04 INFO lead: acquired remote=192.0.2.1:1234 generation=317482026/11/12 00:06:04 INFO lead: released remote=192.0.2.1:1234 generation=31749--- PASS: TestLeadGenerationGrowsWithEveryGrant (3.71s)1750=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete17512026/11/12 00:06:04 INFO Completed upload id=1 objects=3 duration=5.901896ms17522026/11/12 00:06:04 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000100000000000000000000.nar.zst17532026/11/12 00:06:04 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0001.narinfo1754=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark17552026/11/12 00:06:05 INFO Received uploads request method=POST path=/api/pending_closures17562026/11/12 00:06:05 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000500000000000000000000.nar.zst17572026/11/12 00:06:05 WARN Found objects in DB but missing from S3, will re-upload count=117582026/11/12 00:06:05 INFO lead: released remote=192.0.2.1:1234 generation=21759--- PASS: TestLeadElectsOneAndHandsOver (3.61s)1760=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1761=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1762=== NAME TestClientWithDependencies1763 client_integration_test.go:663: Found 1 dependencies (including self)1764=== NAME TestClientMultipleUploads1765 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1521147515/001/store/xn3vkrwsg0hxslkjp7df27xkdidl1azv-test-file-2.txt1766--- PASS: TestPendingClosureWriteTimeout (0.14s)1767 --- PASS: TestPendingClosureWriteTimeout/empty (0.00s)1768 --- PASS: TestPendingClosureWriteTimeout/670k_objects (0.00s)1769 --- PASS: TestPendingClosureWriteTimeout/400_objects (0.00s)1770 --- PASS: TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline (1.61s)1771=== CONT TestResolveDBConnectionString/flag_wins1772=== CONT TestResolveDBConnectionString/nothing_configured1773=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1774=== CONT TestResolveDBConnectionString/missing_file_is_an_error1775=== CONT TestResolveDBConnectionString/file_when_flag_empty1776=== CONT TestClientErrorHandling/InvalidStorePath1777--- PASS: TestResolveDBConnectionString (0.00s)1778 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1779 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1780 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1781 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1782 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)17832026/11/12 00:06:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17842026/11/12 00:06:05 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100400000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLmFjNzk2ZmM1LWIzZjgtNGIxOC1hZjgyLTJiNWZhMWNlYzI4MXgxNzk0NDQxOTY0ODgyNzU1OTcy parts=1017852026/11/12 00:06:05 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17862026/11/12 00:06:05 INFO Completed upload id=2 objects=3 duration=1.347153ms1787--- PASS: TestUploadHandlersRejectOversizedBody (1.32s)1788 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.05s)1789 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.44s)1790 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (1.55s)1791=== CONT TestClientErrorHandling/ServerNotAvailable17922026/11/12 00:06:05 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1793=== CONT TestClientErrorHandling/InvalidAuthToken17942026/11/12 00:06:05 INFO Received uploads request method=POST path=/api/pending_closures17952026/11/12 00:06:05 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000400000000000000000000.nar.zst17962026/11/12 00:06:05 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0004.narinfo1797=== CONT TestCacheConfigHandler/full_config,_no_issuer1798=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator17992026/11/12 00:06:05 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f100300000000000000000000.nar.zst upload_id=ZDRjNjE0ODEtNzFhOC00ZDJjLTliODMtOGJhYjYzNmRlMDFkLjU1NjViYzAzLTliZDEtNGRhMy04MzJjLThhNzVkODIwYTA2MHgxNzk0NDQxOTY0OTUxMjgxNTY1 parts=101800=== CONT TestCacheConfigHandler/no_signing_keys18012026/11/12 00:06:05 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete1802=== CONT TestCacheConfigHandler/no_cache_url_configured1803--- PASS: TestCacheConfigHandler (0.00s)1804 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1805 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1806 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1807 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1808=== CONT TestService_RequireScope_OIDC/builder_may_write18092026/11/12 00:06:05 INFO Completed upload id=2 objects=3 duration=1.06181ms1810=== CONT TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read18112026/11/12 00:06:05 WARN Authentication failed token_preview=wrong token_length=5 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1812=== CONT TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1813=== CONT TestService_RequireScope_OIDC/reader_may_read_with_netrc1814=== CONT TestService_RequireScope_OIDC/anonymous_may_not_read1815=== CONT TestService_RequireScope_OIDC/writer_implies_read1816=== CONT TestService_RequireScope_OIDC/reader_may_read1817=== CONT TestService_RequireScope_OIDC/static_token_may_write1818=== CONT TestService_RequireScope_OIDC/static_token_may_admin1819=== CONT TestService_RequireScope_OIDC/reader_may_not_write1820=== CONT TestService_RequireScope_OIDC/ops_may_not_write1821=== CONT TestService_RequireScope_OIDC/ops_may_admin1822=== CONT TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1823=== CONT TestService_RequireScope_OIDC/builder_may_not_admin1824=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1825--- PASS: TestService_RequireScope_OIDC (1.19s)1826 --- PASS: TestService_RequireScope_OIDC/builder_may_write (0.00s)1827 --- PASS: TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read (0.00s)1828 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin_with_netrc (0.00s)1829 --- PASS: TestService_RequireScope_OIDC/reader_may_read_with_netrc (0.00s)1830 --- PASS: TestService_RequireScope_OIDC/anonymous_may_not_read (0.00s)1831 --- PASS: TestService_RequireScope_OIDC/writer_implies_read (0.00s)1832 --- PASS: TestService_RequireScope_OIDC/reader_may_read (0.00s)1833 --- PASS: TestService_RequireScope_OIDC/static_token_may_write (0.00s)1834 --- PASS: TestService_RequireScope_OIDC/static_token_may_admin (0.00s)1835 --- PASS: TestService_RequireScope_OIDC/reader_may_not_write (0.00s)1836 --- PASS: TestService_RequireScope_OIDC/ops_may_not_write (0.00s)1837 --- PASS: TestService_RequireScope_OIDC/ops_may_admin (0.00s)1838 --- PASS: TestService_RequireScope_OIDC/empty_netrc_password_may_not_read (0.00s)1839 --- PASS: TestService_RequireScope_OIDC/builder_may_not_admin (0.00s)1840=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected18412026/11/12 00:06:05 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1842=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1843=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected18442026/11/12 00:06:05 WARN Authentication failed token_preview=eyJhbGciOi...1yUFiB4-aA token_length=701 oidc_error="no rule matched: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1845--- PASS: TestService_AuthMiddleware_OIDC (1.19s)1846 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1847 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1848 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1849 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)18502026/11/12 00:06:05 INFO Received uploads request method=POST path=/api/pending_closures18512026/11/12 00:06:05 INFO Aborted multipart uploads count=0 kept=018522026/11/12 00:06:05 INFO Object in database but missing from S3 key=nar/dadb44fdadb44fdadb44fdadb44f000300000000000000000000.nar.zst18532026/11/12 00:06:05 WARN Narinfo references a NAR missing from S3, will re-upload narinfo=dadb44fdadb44fdadb44fdadb44f0003.narinfo18542026/11/12 00:06:05 INFO Received uploads request method=POST path=/api/pending_closures1855--- PASS: TestService_verifyS3ReuploadsStaleNarinfo (0.14s)1856 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed (1.15s)1857 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed (1.33s)1858 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed (1.37s)1859 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed (1.67s)1860 --- PASS: TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost (1.71s)18612026/11/12 00:06:05 INFO Aborted multipart uploads count=0 kept=018622026/11/12 00:06:05 INFO Received uploads request method=POST path=/api/pending_closures18632026/11/12 00:06:05 INFO Aborted multipart uploads count=0 kept=01864--- PASS: TestPresentNotReportedWhileGCDeletesClosure (0.00s)1865 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/stale (1.43s)1866 --- PASS: TestPresentNotReportedWhileGCDeletesClosure/fresh (1.47s)18672026/11/12 00:06:06 WARN Force mode enabled - objects will be deleted immediately without grace period18682026/11/12 00:06:06 WARN Force mode enabled - objects will be deleted immediately without grace period18692026/11/12 00:06:06 WARN Force mode enabled - objects will be deleted immediately without grace period18702026/11/12 00:06:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=018712026/11/12 00:06:06 INFO Received push request method=POST path=/api/pushes18722026/11/12 00:06:06 INFO Vacuumed table table=pending_closures18732026/11/12 00:06:06 INFO Vacuumed table table=pending_objects18742026/11/12 00:06:06 INFO Vacuumed table table=multipart_uploads18752026/11/12 00:06:06 INFO Vacuumed table table=closures18762026/11/12 00:06:06 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)18772026/11/12 00:06:06 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Post \"http://localhost:36127/bucket103/?delete=\": context canceled"18782026/11/12 00:06:06 INFO Vacuumed table table=objects18792026/11/12 00:06:06 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign18802026/11/12 00:06:06 WARN Failed to register uploaded object key=2mbjdabmkmr82719jlgqhamlph26biib.ls error="server returned 404: 404 page not found\n"18812026/11/12 00:06:06 INFO Signed narinfos id=2 count=118822026/11/12 00:06:06 INFO Uploading 1 narinfos18832026/11/12 00:06:06 INFO Received complete push request method=POST path=/api/pushes/2/complete18842026/11/12 00:06:06 WARN Failed to register uploaded object key=2mbjdabmkmr82719jlgqhamlph26biib.narinfo error="server returned 404: 404 page not found\n"18852026/11/12 00:06:06 INFO Completed push id=2 objects=3 duration=1.772377ms18862026/11/12 00:06:06 INFO Upload complete. (957ms)1887=== NAME TestClientCADerivations1888 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations3185336355/001/store/vd55bglfarv6h8a93vdq7l5f339mqzjz-ca-test18892026/11/12 00:06:06 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=1 objects-deleted-after-grace-period=1 objects-failed-to-delete=01890=== NAME TestNARDeduplicationMetadataUploadBug1891 metadata_upload_test.go:76: Retrieved narinfo from S3:1892 StorePath: /build/TestNARDeduplicationMetadataUploadBug2784015754/001/store/2mbjdabmkmr82719jlgqhamlph26biib-file2.txt1893 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1894 Compression: zstd1895 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1896 NarSize: 1601897 References: 1898 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf18992026/11/12 00:06:06 INFO Vacuumed table table=pending_closures19002026/11/12 00:06:06 INFO Vacuumed table table=pending_objects19012026/11/12 00:06:06 INFO Vacuumed table table=multipart_uploads19022026/11/12 00:06:06 INFO Vacuumed table table=closures1903 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1904 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1905 {"version":1,"root":{"type":"regular","size":44}}1906--- PASS: TestNARDeduplicationMetadataUploadBug (7.32s)19072026/11/12 00:06:06 INFO Vacuumed table table=objects1908--- PASS: TestForceGCDuringPushOffersSweptObject (0.00s)1909 --- PASS: TestForceGCDuringPushOffersSweptObject/after_presence_check (1.13s)1910 --- PASS: TestForceGCDuringPushOffersSweptObject/before_pending_rows (1.20s)19112026/11/12 00:06:06 INFO Aborted multipart uploads count=0 kept=019122026/11/12 00:06:06 INFO lead: released remote=192.0.2.1:1234 generation=31913--- PASS: TestLeadUnannouncedStandbyYieldsToIncumbent (4.93s)19142026/11/12 00:06:06 WARN Force mode enabled - objects will be deleted immediately without grace period1915=== NAME TestClientCADerivations1916 client_ca_test.go:139: Found 1 dependencies (including self)19172026/11/12 00:06:06 INFO Received push request method=POST path=/api/pushes19182026/11/12 00:06:06 INFO Received push request method=POST path=/api/pushes1919--- PASS: TestGCEndsOnShutdown (0.00s)1920 --- PASS: TestGCEndsOnShutdown/before_the_run (0.48s)1921 --- PASS: TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete (1.17s)1922 --- PASS: TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark (1.61s)19232026/11/12 00:06:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19242026/11/12 00:06:06 INFO Uploading rhlrqpphg53mam3clz52px9iyl7hc0d3-test-script (136B)19252026/11/12 00:06:06 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19262026/11/12 00:06:06 INFO Uploading g7aqzqal6d4jg000hzx3z0vi04nfg4wb-test-file.txt (152B)19272026/11/12 00:06:06 WARN Failed to register uploaded object key=rhlrqpphg53mam3clz52px9iyl7hc0d3.ls error="server returned 404: 404 page not found\n"19282026/11/12 00:06:06 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"19292026/11/12 00:06:06 WARN Failed to register uploaded object key=log/ks8701r3vk6kss1dggh13n58h9jyhr3w-test-script.drv error="server returned 404: 404 page not found\n"19302026/11/12 00:06:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19312026/11/12 00:06:06 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"19322026/11/12 00:06:06 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19332026/11/12 00:06:06 INFO Signed narinfos id=1 count=119342026/11/12 00:06:06 WARN Failed to register uploaded object key=g7aqzqal6d4jg000hzx3z0vi04nfg4wb.ls error="server returned 404: 404 page not found\n"19352026/11/12 00:06:06 INFO Uploading 1 narinfos19362026/11/12 00:06:06 INFO Signed narinfos id=1 count=119372026/11/12 00:06:06 INFO Uploading 1 narinfos19382026/11/12 00:06:06 INFO Received complete push request method=POST path=/api/pushes/1/complete19392026/11/12 00:06:06 INFO Received complete push request method=POST path=/api/pushes/1/complete19402026/11/12 00:06:06 WARN Failed to register uploaded object key=g7aqzqal6d4jg000hzx3z0vi04nfg4wb.narinfo error="server returned 404: 404 page not found\n"19412026/11/12 00:06:06 WARN Failed to register uploaded object key=rhlrqpphg53mam3clz52px9iyl7hc0d3.narinfo error="server returned 404: 404 page not found\n"19422026/11/12 00:06:06 INFO Completed push id=1 objects=3 duration=9.321689ms19432026/11/12 00:06:06 INFO Completed push id=1 objects=4 duration=11.840773ms19442026/11/12 00:06:06 INFO Upload complete. (462ms)19452026/11/12 00:06:06 INFO Upload complete. (494ms)19462026/11/12 00:06:06 INFO Aborted multipart uploads count=0 kept=01947--- PASS: TestConnectWaitsForAPeerMigration (5.57s)19482026/11/12 00:06:07 INFO Received push request method=POST path=/api/pushes19492026/11/12 00:06:07 INFO Received push request method=POST path=/api/pushes19502026/11/12 00:06:07 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19512026/11/12 00:06:07 INFO Received push request method=POST path=/api/pushes19522026/11/12 00:06:07 WARN Force mode enabled - objects will be deleted immediately without grace period19532026/11/12 00:06:07 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)19542026/11/12 00:06:07 INFO Uploading yxdlq27mfqjw1504l7rf9lwwg8mpa436-a (208B)19552026/11/12 00:06:07 INFO Uploading lxkkz9199ra5agq64idp06m5i191anmx-shared-dep (136B)19562026/11/12 00:06:07 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19572026/11/12 00:06:07 INFO Uploading 3h6q7zgkcpgq2zx3cbrgg3b2l1k132wm-pinned-file.txt (128B)19582026/11/12 00:06:07 WARN Authentication failed token_preview=invalid-token token_length=13 reason="static API token mismatch"19592026/11/12 00:06:07 INFO All 1 paths already cached19602026/11/12 00:06:07 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019612026/11/12 00:06:07 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)19622026/11/12 00:06:07 INFO Uploading r8li0hy34x3f1i7ag033qrlcyhkl3sd9-test-file-1.txt (160B)19632026/11/12 00:06:07 INFO Uploading s771hhaggz5q9m16c09i1pz65yilg01x-test-file-0.txt (160B)19642026/11/12 00:06:07 INFO Uploading xn3vkrwsg0hxslkjp7df27xkdidl1azv-test-file-2.txt (160B)1965=== NAME TestClientIntegration1966 client_integration_test.go:318: Retrieved narinfo from S3:1967 StorePath: /build/TestClientIntegration19819256/002/store/g7aqzqal6d4jg000hzx3z0vi04nfg4wb-test-file.txt1968 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1969 Compression: zstd1970 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11971 NarSize: 1521972 References: 1973 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk119742026/11/12 00:06:07 WARN Failed to register uploaded object key=lxkkz9199ra5agq64idp06m5i191anmx.ls error="server returned 404: 404 page not found\n"19752026/11/12 00:06:07 WARN Failed to register uploaded object key=3h6q7zgkcpgq2zx3cbrgg3b2l1k132wm.ls error="server returned 404: 404 page not found\n"19762026/11/12 00:06:07 WARN Failed to register uploaded object key=c0q9w7nc0wgd7x7pcif517b13a4cxi2h.ls error="server returned 404: 404 page not found\n"19772026/11/12 00:06:07 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19782026/11/12 00:06:07 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19792026/11/12 00:06:07 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"19802026/11/12 00:06:07 WARN Failed to register uploaded object key=nar/0ly7w5sq1rxj7599mfwl2gy54s8y6a34x5rbwwsqddnwy7kc183v.nar.zst error="server returned 404: 404 page not found\n"19812026/11/12 00:06:07 INFO Signed narinfos id=1 count=119822026/11/12 00:06:07 WARN Failed to register uploaded object key=yxdlq27mfqjw1504l7rf9lwwg8mpa436.ls error="server returned 404: 404 page not found\n"19832026/11/12 00:06:07 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"19842026/11/12 00:06:07 INFO Uploading 1 narinfos19852026/11/12 00:06:07 INFO Signed narinfos id=1 count=319862026/11/12 00:06:07 INFO Uploading 3 narinfos1987 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1988 client_integration_test.go:319: Decompressed .ls content (64 bytes):1989 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}19902026/11/12 00:06:07 INFO Vacuumed table table=pending_closures19912026/11/12 00:06:07 WARN Failed to register uploaded object key=xn3vkrwsg0hxslkjp7df27xkdidl1azv.ls error="server returned 404: 404 page not found\n"19922026/11/12 00:06:07 WARN Failed to register uploaded object key=r8li0hy34x3f1i7ag033qrlcyhkl3sd9.ls error="server returned 404: 404 page not found\n"19932026/11/12 00:06:07 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"19942026/11/12 00:06:07 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"19952026/11/12 00:06:07 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"19962026/11/12 00:06:07 WARN Failed to register uploaded object key=s771hhaggz5q9m16c09i1pz65yilg01x.ls error="server returned 404: 404 page not found\n"19972026/11/12 00:06:07 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19982026/11/12 00:06:07 INFO Vacuumed table table=pending_objects19992026/11/12 00:06:07 INFO Received complete push request method=POST path=/api/pushes/1/complete20002026/11/12 00:06:07 WARN Failed to register uploaded object key=3h6q7zgkcpgq2zx3cbrgg3b2l1k132wm.narinfo error="server returned 404: 404 page not found\n"
loading 637 hidden lines…
26382026/11/12 00:06:22 INFO Received complete push request method=POST path=/26392026/11/12 00:06:22 INFO Vacuumed table table=pending_closures26402026/11/12 00:06:22 INFO Vacuumed table table=pending_objects26412026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads26422026/11/12 00:06:22 INFO Completed push id=6 objects=7 duration=3.648954ms26432026/11/12 00:06:22 INFO Vacuumed table table=closures26442026/11/12 00:06:22 INFO Vacuumed table table=objects26452026/11/12 00:06:22 INFO Received uploads request method=POST path=/26462026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26472026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26482026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26492026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26502026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=026512026/11/12 00:06:22 INFO Completed upload id=2 objects=10 duration=2.014979ms26522026/11/12 00:06:22 INFO Completed upload id=11 objects=14 duration=1.853737ms26532026/11/12 00:06:22 INFO Completed upload id=4 objects=7 duration=1.826998ms26542026/11/12 00:06:22 INFO Received uploads request method=POST path=/26552026/11/12 00:06:22 INFO Completed upload id=1 objects=15 duration=4.21128ms26562026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26572026/11/12 00:06:22 INFO Received complete push request method=POST path=/26582026/11/12 00:06:22 INFO Completed upload id=5 objects=5 duration=1.736636ms26592026/11/12 00:06:22 INFO Received push request method=POST path=/26602026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=026612026/11/12 00:06:22 INFO Received push request method=POST path=/26622026/11/12 00:06:22 INFO Completed push id=3 objects=22 duration=7.724794ms26632026/11/12 00:06:22 INFO Received uploads request method=POST path=/26642026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26652026/11/12 00:06:22 INFO Received push request method=POST path=/26662026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026672026/11/12 00:06:22 INFO Vacuumed table table=pending_closures26682026/11/12 00:06:22 INFO Received uploads request method=POST path=/26692026/11/12 00:06:22 INFO Vacuumed table table=pending_objects26702026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads26712026/11/12 00:06:22 INFO Vacuumed table table=closures26722026/11/12 00:06:22 INFO Vacuumed table table=objects26732026/11/12 00:06:22 INFO Completed upload id=2 objects=26 duration=5.725515ms26742026/11/12 00:06:22 INFO Received uploads request method=POST path=/26752026/11/12 00:06:22 INFO Received uploads request method=POST path=/26762026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26772026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=026782026/11/12 00:06:22 INFO Received push request method=POST path=/26792026/11/12 00:06:22 INFO Completed upload id=5 objects=7 duration=2.478024ms26802026/11/12 00:06:22 INFO Vacuumed table table=pending_closures26812026/11/12 00:06:22 INFO Received uploads request method=POST path=/26822026/11/12 00:06:22 INFO Vacuumed table table=pending_objects26832026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads26842026/11/12 00:06:22 INFO Vacuumed table table=closures26852026/11/12 00:06:22 INFO Received push request method=POST path=/26862026/11/12 00:06:22 INFO Received uploads request method=POST path=/26872026/11/12 00:06:22 INFO Vacuumed table table=objects26882026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26892026/11/12 00:06:22 INFO Completed upload id=7 objects=4 duration=1.296772ms26902026/11/12 00:06:22 INFO Received uploads request method=POST path=/26912026/11/12 00:06:22 INFO Received push request method=POST path=/26922026/11/12 00:06:22 INFO Received complete upload request method=POST path=/26932026/11/12 00:06:22 INFO Completed upload id=6 objects=9 duration=1.143711ms26942026/11/12 00:06:22 INFO Received push request method=POST path=/26952026/11/12 00:06:22 INFO Received complete push request method=POST path=/26962026/11/12 00:06:22 INFO Completed push id=3 objects=5 duration=1.862698ms26972026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=026982026/11/12 00:06:22 INFO Received push request method=POST path=/26992026/11/12 00:06:22 INFO Received push request method=POST path=/27002026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027012026/11/12 00:06:22 INFO Received uploads request method=POST path=/27022026/11/12 00:06:22 INFO Received complete upload request method=POST path=/27032026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period27042026/11/12 00:06:22 INFO Completed upload id=3 objects=24 duration=1.649876ms27052026/11/12 00:06:22 INFO Received complete upload request method=POST path=/27062026/11/12 00:06:22 INFO Received push request method=POST path=/27072026/11/12 00:06:22 INFO Received uploads request method=POST path=/27082026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027092026/11/12 00:06:22 INFO Completed upload id=10 objects=23 duration=2.912787ms27102026/11/12 00:06:22 INFO Received push request method=POST path=/27112026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period27122026/11/12 00:06:22 INFO Received push request method=POST path=/27132026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027142026/11/12 00:06:22 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27152026/11/12 00:06:22 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27162026/11/12 00:06:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27172026/11/12 00:06:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27182026/11/12 00:06:22 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27192026/11/12 00:06:22 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27202026/11/12 00:06:22 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27212026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027222026/11/12 00:06:22 INFO Vacuumed table table=pending_closures27232026/11/12 00:06:22 INFO Received uploads request method=POST path=/27242026/11/12 00:06:22 INFO Vacuumed table table=pending_objects27252026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads27262026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period27272026/11/12 00:06:22 INFO Vacuumed table table=closures27282026/11/12 00:06:22 INFO Vacuumed table table=objects27292026/11/12 00:06:22 INFO Received uploads request method=POST path=/27302026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027312026/11/12 00:06:22 INFO Received complete push request method=POST path=/27322026/11/12 00:06:22 INFO Vacuumed table table=pending_closures27332026/11/12 00:06:22 INFO Received push request method=POST path=/27342026/11/12 00:06:22 INFO Vacuumed table table=pending_objects27352026/11/12 00:06:22 INFO Received uploads request method=POST path=/27362026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads27372026/11/12 00:06:22 INFO Vacuumed table table=closures27382026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027392026/11/12 00:06:22 INFO Completed push id=2 objects=25 duration=4.17244ms27402026/11/12 00:06:22 INFO Vacuumed table table=objects27412026/11/12 00:06:22 INFO Received complete push request method=POST path=/27422026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period27432026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027442026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027452026/11/12 00:06:22 INFO Completed push id=2 objects=7 duration=4.575604ms27462026/11/12 00:06:22 INFO Vacuumed table table=pending_closures27472026/11/12 00:06:22 INFO Received complete upload request method=POST path=/27482026/11/12 00:06:22 INFO Vacuumed table table=pending_objects27492026/11/12 00:06:22 INFO Completed upload id=3 objects=5 duration=1.623335ms27502026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads27512026/11/12 00:06:22 INFO Received push request method=POST path=/27522026/11/12 00:06:22 INFO Received uploads request method=POST path=/27532026/11/12 00:06:22 INFO Vacuumed table table=closures27542026/11/12 00:06:22 INFO Vacuumed table table=objects27552026/11/12 00:06:22 INFO Received uploads request method=POST path=/27562026/11/12 00:06:22 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."27572026/11/12 00:06:22 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."27582026/11/12 00:06:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."27592026/11/12 00:06:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."27602026/11/12 00:06:22 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."27612026/11/12 00:06:22 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."27622026/11/12 00:06:22 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."27632026/11/12 00:06:22 INFO Received complete upload request method=POST path=/27642026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027652026/11/12 00:06:22 INFO Completed upload id=2 objects=2 duration=2.0059ms27662026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027672026/11/12 00:06:22 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27682026/11/12 00:06:22 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."27692026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027702026/11/12 00:06:22 INFO Received uploads request method=POST path=/27712026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period27722026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027732026/11/12 00:06:22 INFO Received uploads request method=POST path=/27742026/11/12 00:06:22 INFO Received push request method=POST path=/27752026/11/12 00:06:22 INFO Received push request method=POST path=/27762026/11/12 00:06:22 INFO Received complete upload request method=POST path=/27772026/11/12 00:06:22 INFO Received complete upload request method=POST path=/27782026/11/12 00:06:22 INFO Completed upload id=11 objects=6 duration=1.350552ms27792026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027802026/11/12 00:06:22 INFO Completed upload id=9 objects=7 duration=2.471583ms27812026/11/12 00:06:22 INFO Received complete push request method=POST path=/27822026/11/12 00:06:22 INFO Vacuumed table table=pending_closures27832026/11/12 00:06:22 INFO Received complete push request method=POST path=/27842026/11/12 00:06:22 INFO Vacuumed table table=pending_objects27852026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads27862026/11/12 00:06:22 INFO Completed push id=4 objects=4 duration=2.218461ms27872026/11/12 00:06:22 INFO Vacuumed table table=closures27882026/11/12 00:06:22 INFO Received push request method=POST path=/27892026/11/12 00:06:22 INFO Completed push id=6 objects=5 duration=2.16852ms27902026/11/12 00:06:22 INFO Vacuumed table table=objects27912026/11/12 00:06:22 INFO Received complete push request method=POST path=/27922026/11/12 00:06:22 INFO Completed push id=14 objects=3 duration=3.052729ms27932026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=027942026/11/12 00:06:22 INFO Received uploads request method=POST path=/27952026/11/12 00:06:22 INFO Received push request method=POST path=/27962026/11/12 00:06:22 INFO Received complete push request method=POST path=/27972026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027982026/11/12 00:06:22 INFO Received complete upload request method=POST path=/27992026/11/12 00:06:22 INFO Received push request method=POST path=/28002026/11/12 00:06:22 INFO Received push request method=POST path=/28012026/11/12 00:06:22 INFO Completed push id=7 objects=20 duration=4.117619ms28022026/11/12 00:06:22 INFO Completed upload id=3 objects=8 duration=3.955798ms28032026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28042026/11/12 00:06:22 INFO Vacuumed table table=pending_objects28052026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads28062026/11/12 00:06:22 INFO Received uploads request method=POST path=/28072026/11/12 00:06:22 INFO Vacuumed table table=closures28082026/11/12 00:06:22 INFO Vacuumed table table=objects28092026/11/12 00:06:22 INFO Received push request method=POST path=/28102026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028112026/11/12 00:06:22 INFO Received push request method=POST path=/28122026/11/12 00:06:22 INFO Received push request method=POST path=/28132026/11/12 00:06:22 INFO Received uploads request method=POST path=/28142026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=028152026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28162026/11/12 00:06:22 INFO Received complete push request method=POST path=/28172026/11/12 00:06:22 INFO Vacuumed table table=pending_objects28182026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads28192026/11/12 00:06:22 INFO Completed push id=5 objects=6 duration=1.562935ms28202026/11/12 00:06:22 INFO Vacuumed table table=closures28212026/11/12 00:06:22 INFO Received complete push request method=POST path=/28222026/11/12 00:06:22 INFO Received push request method=POST path=/28232026/11/12 00:06:22 INFO Received uploads request method=POST path=/28242026/11/12 00:06:22 INFO Vacuumed table table=objects28252026/11/12 00:06:22 INFO Received complete push request method=POST path=/28262026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028272026/11/12 00:06:22 INFO Completed push id=12 objects=15 duration=2.275242ms28282026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028292026/11/12 00:06:22 INFO Completed push id=2 objects=10 duration=3.793896ms28302026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028312026/11/12 00:06:22 INFO Received push request method=POST path=/28322026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period28332026/11/12 00:06:22 INFO Received push request method=POST path=/28342026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028352026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28362026/11/12 00:06:22 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28372026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028382026/11/12 00:06:22 INFO Vacuumed table table=pending_objects28392026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads28402026/11/12 00:06:22 INFO Vacuumed table table=closures28412026/11/12 00:06:22 INFO Vacuumed table table=objects28422026/11/12 00:06:22 INFO Received push request method=POST path=/28432026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period28442026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028452026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28462026/11/12 00:06:22 INFO Received push request method=POST path=/28472026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028482026/11/12 00:06:22 INFO Vacuumed table table=pending_objects28492026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=028502026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads28512026/11/12 00:06:22 INFO Vacuumed table table=closures28522026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28532026/11/12 00:06:22 INFO Vacuumed table table=objects28542026/11/12 00:06:22 INFO Received uploads request method=POST path=/28552026/11/12 00:06:22 INFO Vacuumed table table=pending_objects28562026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads28572026/11/12 00:06:22 INFO Vacuumed table table=closures28582026/11/12 00:06:22 INFO Vacuumed table table=objects28592026/11/12 00:06:22 INFO Received push request method=POST path=/28602026/11/12 00:06:22 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28612026/11/12 00:06:22 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28622026/11/12 00:06:22 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28632026/11/12 00:06:22 INFO Received push request method=POST path=/28642026/11/12 00:06:22 INFO Received push request method=POST path=/28652026/11/12 00:06:22 INFO Received complete push request method=POST path=/28662026/11/12 00:06:22 INFO Completed push id=5 objects=3 duration=1.287173ms28672026/11/12 00:06:22 INFO Received complete push request method=POST path=/28682026/11/12 00:06:22 INFO Received complete upload request method=POST path=/28692026/11/12 00:06:22 INFO Completed push id=5 objects=3 duration=2.384923ms28702026/11/12 00:06:22 INFO Completed upload id=10 objects=6 duration=1.856318ms28712026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028722026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028732026/11/12 00:06:22 INFO Received push request method=POST path=/28742026/11/12 00:06:22 INFO Received uploads request method=POST path=/28752026/11/12 00:06:22 INFO Received complete push request method=POST path=/28762026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028772026/11/12 00:06:22 INFO Completed push id=8 objects=7 duration=1.928599ms28782026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028792026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period28802026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028812026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28822026/11/12 00:06:22 INFO Vacuumed table table=pending_objects28832026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads28842026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028852026/11/12 00:06:22 INFO Vacuumed table table=closures28862026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28872026/11/12 00:06:22 INFO Vacuumed table table=objects28882026/11/12 00:06:22 INFO Vacuumed table table=pending_objects28892026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads28902026/11/12 00:06:22 INFO Vacuumed table table=closures28912026/11/12 00:06:22 INFO Received uploads request method=POST path=/28922026/11/12 00:06:22 INFO Vacuumed table table=objects28932026/11/12 00:06:22 INFO Received uploads request method=POST path=/28942026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028952026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=028962026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=028972026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028982026/11/12 00:06:22 INFO Vacuumed table table=pending_closures28992026/11/12 00:06:22 INFO Vacuumed table table=pending_closures29002026/11/12 00:06:22 INFO Received complete push request method=POST path=/29012026/11/12 00:06:22 INFO Vacuumed table table=pending_objects29022026/11/12 00:06:22 INFO Vacuumed table table=pending_objects29032026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads29042026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads29052026/11/12 00:06:22 INFO Received complete push request method=POST path=/29062026/11/12 00:06:22 INFO Vacuumed table table=closures29072026/11/12 00:06:22 INFO Received push request method=POST path=/29082026/11/12 00:06:22 INFO Vacuumed table table=closures29092026/11/12 00:06:22 INFO Vacuumed table table=objects29102026/11/12 00:06:22 INFO Completed push id=4 objects=4 duration=2.865568ms29112026/11/12 00:06:22 INFO Completed push id=10 objects=2 duration=1.860098ms29122026/11/12 00:06:22 INFO Received push request method=POST path=/29132026/11/12 00:06:22 INFO Vacuumed table table=objects29142026/11/12 00:06:22 INFO Received uploads request method=POST path=/29152026/11/12 00:06:22 INFO Received complete upload request method=POST path=/29162026/11/12 00:06:22 INFO Completed upload id=9 objects=16 duration=1.430554ms29172026/11/12 00:06:22 INFO Received complete push request method=POST path=/29182026/11/12 00:06:22 INFO Received complete upload request method=POST path=/29192026/11/12 00:06:22 INFO Completed upload id=12 objects=21 duration=1.014789ms29202026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=029212026/11/12 00:06:22 INFO Completed push id=8 objects=6 duration=1.889978ms29222026/11/12 00:06:22 INFO Received push request method=POST path=/29232026/11/12 00:06:22 INFO Received complete push request method=POST path=/29242026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029252026/11/12 00:06:22 INFO Completed push id=6 objects=6 duration=1.452354ms29262026/11/12 00:06:22 INFO Received push request method=POST path=/29272026/11/12 00:06:22 INFO Vacuumed table table=pending_closures29282026/11/12 00:06:22 INFO Received uploads request method=POST path=/29292026/11/12 00:06:22 INFO Vacuumed table table=pending_objects29302026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads29312026/11/12 00:06:22 INFO Received push request method=POST path=/29322026/11/12 00:06:22 INFO Vacuumed table table=closures29332026/11/12 00:06:22 INFO Received push request method=POST path=/29342026/11/12 00:06:22 INFO Vacuumed table table=objects29352026/11/12 00:06:22 INFO Received push request method=POST path=/29362026/11/12 00:06:22 INFO Received push request method=POST path=/29372026/11/12 00:06:22 INFO Received uploads request method=POST path=/29382026/11/12 00:06:22 INFO Received push request method=POST path=/29392026/11/12 00:06:22 INFO Received push request method=POST path=/29402026/11/12 00:06:22 INFO Received uploads request method=POST path=/29412026/11/12 00:06:22 INFO Received push request method=POST path=/29422026/11/12 00:06:22 INFO Received complete push request method=POST path=/29432026/11/12 00:06:22 INFO Completed push id=13 objects=23 duration=1.834078ms29442026/11/12 00:06:22 INFO Received complete push request method=POST path=/29452026/11/12 00:06:22 INFO Received uploads request method=POST path=/29462026/11/12 00:06:22 INFO Received uploads request method=POST path=/29472026/11/12 00:06:22 INFO Received complete push request method=POST path=/29482026/11/12 00:06:22 INFO Received complete push request method=POST path=/29492026/11/12 00:06:22 INFO Completed push id=10 objects=3 duration=1.417293ms29502026/11/12 00:06:22 INFO Completed push id=6 objects=26 duration=4.347442ms29512026/11/12 00:06:22 INFO Completed push id=5 objects=9 duration=4.479083ms29522026/11/12 00:06:22 INFO Received uploads request method=POST path=/29532026/11/12 00:06:22 INFO Received complete upload request method=POST path=/29542026/11/12 00:06:22 INFO Completed upload id=11 objects=6 duration=2.186121ms29552026/11/12 00:06:22 INFO Received complete push request method=POST path=/29562026/11/12 00:06:22 INFO Completed push id=7 objects=27 duration=1.868058ms29572026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=029582026/11/12 00:06:22 INFO Received push request method=POST path=/29592026/11/12 00:06:22 INFO Received push request method=POST path=/29602026/11/12 00:06:22 INFO Received uploads request method=POST path=/29612026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period29622026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029632026/11/12 00:06:22 INFO Received uploads request method=POST path=/29642026/11/12 00:06:22 INFO Vacuumed table table=pending_closures29652026/11/12 00:06:22 INFO Received uploads request method=POST path=/29662026/11/12 00:06:22 INFO Vacuumed table table=pending_objects29672026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads29682026/11/12 00:06:22 INFO Vacuumed table table=closures29692026/11/12 00:06:22 INFO Vacuumed table table=objects29702026/11/12 00:06:22 INFO Received push request method=POST path=/29712026/11/12 00:06:22 INFO Received push request method=POST path=/29722026/11/12 00:06:22 INFO Received uploads request method=POST path=/29732026/11/12 00:06:22 INFO Received push request method=POST path=/29742026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=029752026/11/12 00:06:22 INFO Received push request method=POST path=/29762026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period29772026/11/12 00:06:22 INFO Received push request method=POST path=/29782026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029792026/11/12 00:06:22 INFO Received complete upload request method=POST path=/29802026/11/12 00:06:22 INFO Received uploads request method=POST path=/29812026/11/12 00:06:22 INFO Vacuumed table table=pending_closures29822026/11/12 00:06:22 INFO Completed upload id=3 objects=9 duration=1.06985ms29832026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=029842026/11/12 00:06:22 INFO Vacuumed table table=pending_objects29852026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads29862026/11/12 00:06:22 INFO Vacuumed table table=closures29872026/11/12 00:06:22 INFO Vacuumed table table=objects29882026/11/12 00:06:22 INFO Received complete upload request method=POST path=/29892026/11/12 00:06:22 INFO Completed upload id=15 objects=6 duration=849.088µs29902026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=029912026/11/12 00:06:22 INFO Received push request method=POST path=/29922026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=029932026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period29942026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=029952026/11/12 00:06:22 INFO Received complete upload request method=POST path=/29962026/11/12 00:06:22 INFO Received push request method=POST path=/29972026/11/12 00:06:22 INFO Vacuumed table table=pending_closures29982026/11/12 00:06:22 INFO Completed upload id=10 objects=17 duration=1.281772ms29992026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030002026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30012026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30022026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30032026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030042026/11/12 00:06:22 INFO Vacuumed table table=closures30052026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30062026/11/12 00:06:22 INFO Vacuumed table table=objects30072026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30082026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30092026/11/12 00:06:22 INFO Received uploads request method=POST path=/30102026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30112026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30122026/11/12 00:06:22 INFO Vacuumed table table=closures30132026/11/12 00:06:22 INFO Vacuumed table table=closures30142026/11/12 00:06:22 INFO Vacuumed table table=objects30152026/11/12 00:06:22 INFO Vacuumed table table=objects30162026/11/12 00:06:22 INFO Received uploads request method=POST path=/30172026/11/12 00:06:22 INFO Received complete push request method=POST path=/30182026/11/12 00:06:22 INFO Completed push id=9 objects=5 duration=2.441023ms30192026/11/12 00:06:22 INFO Received push request method=POST path=/30202026/11/12 00:06:22 INFO Received push request method=POST path=/30212026/11/12 00:06:22 INFO Received push request method=POST path=/30222026/11/12 00:06:22 INFO Received push request method=POST path=/30232026/11/12 00:06:22 INFO Received complete push request method=POST path=/30242026/11/12 00:06:22 INFO Completed push id=5 objects=6 duration=1.189151ms30252026/11/12 00:06:22 INFO Received uploads request method=POST path=/30262026/11/12 00:06:22 INFO Received push request method=POST path=/30272026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030282026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period30292026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030302026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30312026/11/12 00:06:22 INFO Received push request method=POST path=/30322026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30332026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30342026/11/12 00:06:22 INFO Vacuumed table table=closures30352026/11/12 00:06:22 INFO Vacuumed table table=objects30362026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030372026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030382026/11/12 00:06:22 INFO Received uploads request method=POST path=/30392026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period30402026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030412026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030422026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30432026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period30442026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30452026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30462026/11/12 00:06:22 INFO Vacuumed table table=closures30472026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030482026/11/12 00:06:22 INFO Received push request method=POST path=/30492026/11/12 00:06:22 INFO Vacuumed table table=objects30502026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period30512026/11/12 00:06:22 INFO Received uploads request method=POST path=/30522026/11/12 00:06:22 INFO Received push request method=POST path=/30532026/11/12 00:06:22 INFO Received uploads request method=POST path=/30542026/11/12 00:06:22 INFO Received uploads request method=POST path=/30552026/11/12 00:06:22 INFO Received push request method=POST path=/30562026/11/12 00:06:22 INFO Received complete push request method=POST path=/30572026/11/12 00:06:22 INFO Received complete push request method=POST path=/30582026/11/12 00:06:22 INFO Completed push id=14 objects=6 duration=1.543035ms30592026/11/12 00:06:22 INFO Completed push id=16 objects=27 duration=2.0035ms30602026/11/12 00:06:22 INFO Received uploads request method=POST path=/30612026/11/12 00:06:22 INFO Received push request method=POST path=/30622026/11/12 00:06:22 INFO Received uploads request method=POST path=/30632026/11/12 00:06:22 INFO Received push request method=POST path=/30642026/11/12 00:06:22 INFO Received uploads request method=POST path=/30652026/11/12 00:06:22 INFO Received push request method=POST path=/30662026/11/12 00:06:22 INFO Received push request method=POST path=/30672026/11/12 00:06:22 INFO Received complete upload request method=POST path=/30682026/11/12 00:06:22 INFO Completed upload id=15 objects=4 duration=1.959178ms30692026/11/12 00:06:22 INFO Received push request method=POST path=/30702026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=030712026/11/12 00:06:22 INFO Received push request method=POST path=/30722026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30732026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30742026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30752026/11/12 00:06:22 INFO Vacuumed table table=closures30762026/11/12 00:06:22 INFO Vacuumed table table=objects30772026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030782026/11/12 00:06:22 INFO Received push request method=POST path=/30792026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030802026/11/12 00:06:22 WARN Force mode enabled - objects will be deleted immediately without grace period30812026/11/12 00:06:22 INFO Aborted multipart uploads count=0 kept=030822026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030832026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030842026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30852026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30862026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30872026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30882026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30892026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30902026/11/12 00:06:22 INFO Vacuumed table table=closures30912026/11/12 00:06:22 INFO Vacuumed table table=closures30922026/11/12 00:06:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=030932026/11/12 00:06:22 INFO Vacuumed table table=objects30942026/11/12 00:06:22 INFO Vacuumed table table=objects30952026/11/12 00:06:22 INFO Vacuumed table table=pending_closures30962026/11/12 00:06:22 INFO Vacuumed table table=pending_objects30972026/11/12 00:06:22 INFO Vacuumed table table=multipart_uploads30982026/11/12 00:06:23 INFO Vacuumed table table=closures30992026/11/12 00:06:23 INFO Vacuumed table table=objects31002026/11/12 00:06:23 INFO Received uploads request method=POST path=/31012026/11/12 00:06:23 INFO Received uploads request method=POST path=/31022026/11/12 00:06:23 INFO Received complete push request method=POST path=/31032026/11/12 00:06:23 INFO Completed push id=18 objects=10 duration=1.845478ms31042026/11/12 00:06:23 INFO Received complete upload request method=POST path=/31052026/11/12 00:06:23 INFO Received complete push request method=POST path=/31062026/11/12 00:06:23 INFO Completed upload id=16 objects=3 duration=1.243891ms31072026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031082026/11/12 00:06:23 INFO Completed push id=9 objects=7 duration=3.07841ms31092026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period31102026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031112026/11/12 00:06:23 INFO Vacuumed table table=pending_closures31122026/11/12 00:06:23 INFO Vacuumed table table=pending_objects31132026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads31142026/11/12 00:06:23 INFO Vacuumed table table=closures31152026/11/12 00:06:23 INFO Vacuumed table table=objects31162026/11/12 00:06:23 INFO Received push request method=POST path=/31172026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031182026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031192026/11/12 00:06:23 INFO Received complete upload request method=POST path=/31202026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031212026/11/12 00:06:23 INFO Completed upload id=17 objects=6 duration=3.19479ms31222026/11/12 00:06:23 INFO Vacuumed table table=pending_closures31232026/11/12 00:06:23 INFO Vacuumed table table=pending_objects31242026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads31252026/11/12 00:06:23 INFO Received push request method=POST path=/31262026/11/12 00:06:23 INFO Vacuumed table table=closures31272026/11/12 00:06:23 INFO Received complete upload request method=POST path=/31282026/11/12 00:06:23 INFO Vacuumed table table=objects31292026/11/12 00:06:23 INFO Completed upload id=10 objects=7 duration=1.931478ms31302026/11/12 00:06:23 INFO Received uploads request method=POST path=/31312026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031322026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period31332026/11/12 00:06:23 INFO Received complete push request method=POST path=/31342026/11/12 00:06:23 INFO Received uploads request method=POST path=/31352026/11/12 00:06:23 INFO Received uploads request method=POST path=/31362026/11/12 00:06:23 INFO Completed push id=15 objects=24 duration=3.481013ms31372026/11/12 00:06:23 INFO Received uploads request method=POST path=/31382026/11/12 00:06:23 INFO Received push request method=POST path=/31392026/11/12 00:06:23 INFO Received uploads request method=POST path=/31402026/11/12 00:06:23 INFO Received complete push request method=POST path=/31412026/11/12 00:06:23 INFO Completed push id=12 objects=2 duration=2.836687ms31422026/11/12 00:06:23 INFO Received uploads request method=POST path=/31432026/11/12 00:06:23 INFO Received complete upload request method=POST path=/31442026/11/12 00:06:23 INFO Received uploads request method=POST path=/31452026/11/12 00:06:23 INFO Completed upload id=18 objects=4 duration=1.916058ms31462026/11/12 00:06:23 INFO Received uploads request method=POST path=/31472026/11/12 00:06:23 INFO Received complete push request method=POST path=/31482026/11/12 00:06:23 INFO Received complete upload request method=POST path=/31492026/11/12 00:06:23 INFO Completed upload id=18 objects=4 duration=1.160051ms31502026/11/12 00:06:23 INFO Completed push id=3 objects=21 duration=3.658315ms31512026/11/12 00:06:23 INFO Received push request method=POST path=/31522026/11/12 00:06:23 INFO Received uploads request method=POST path=/31532026/11/12 00:06:23 INFO Received complete push request method=POST path=/31542026/11/12 00:06:23 INFO Received uploads request method=POST path=/31552026/11/12 00:06:23 INFO Completed push id=17 objects=6 duration=1.893678ms31562026/11/12 00:06:23 INFO Received uploads request method=POST path=/31572026/11/12 00:06:23 INFO Received uploads request method=POST path=/31582026/11/12 00:06:23 INFO Received push request method=POST path=/31592026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031602026/11/12 00:06:23 INFO Received complete upload request method=POST path=/31612026/11/12 00:06:23 INFO Received push request method=POST path=/31622026/11/12 00:06:23 INFO Completed upload id=6 objects=6 duration=1.192212ms31632026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031642026/11/12 00:06:23 INFO Vacuumed table table=pending_closures31652026/11/12 00:06:23 INFO Received uploads request method=POST path=/31662026/11/12 00:06:23 INFO Vacuumed table table=pending_objects31672026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads31682026/11/12 00:06:23 INFO Vacuumed table table=closures31692026/11/12 00:06:23 INFO Received push request method=POST path=/31702026/11/12 00:06:23 INFO Vacuumed table table=objects31712026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031722026/11/12 00:06:23 INFO Received uploads request method=POST path=/31732026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period31742026/11/12 00:06:23 INFO Received uploads request method=POST path=/31752026/11/12 00:06:23 INFO Received push request method=POST path=/31762026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031772026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031782026/11/12 00:06:23 INFO Vacuumed table table=pending_closures31792026/11/12 00:06:23 INFO Vacuumed table table=pending_objects31802026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads31812026/11/12 00:06:23 INFO Vacuumed table table=closures31822026/11/12 00:06:23 INFO Vacuumed table table=objects31832026/11/12 00:06:23 INFO Received complete upload request method=POST path=/31842026/11/12 00:06:23 INFO Completed upload id=15 objects=9 duration=1.03775ms31852026/11/12 00:06:23 INFO Received uploads request method=POST path=/31862026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=031872026/11/12 00:06:23 INFO Received complete push request method=POST path=/31882026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031892026/11/12 00:06:23 INFO Completed push id=15 objects=5 duration=1.620315ms31902026/11/12 00:06:23 INFO Vacuumed table table=pending_closures31912026/11/12 00:06:23 INFO Vacuumed table table=pending_objects31922026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads31932026/11/12 00:06:23 INFO Vacuumed table table=closures31942026/11/12 00:06:23 INFO Vacuumed table table=objects31952026/11/12 00:06:23 INFO Received complete push request method=POST path=/31962026/11/12 00:06:23 INFO Completed push id=1 objects=11 duration=3.340432ms31972026/11/12 00:06:23 INFO Received complete push request method=POST path=/31982026/11/12 00:06:23 INFO Received uploads request method=POST path=/31992026/11/12 00:06:23 INFO Completed push id=21 objects=17 duration=2.168281ms32002026/11/12 00:06:23 INFO Received complete push request method=POST path=/32012026/11/12 00:06:23 INFO Completed push id=22 objects=3 duration=2.458763ms32022026/11/12 00:06:23 INFO Received complete push request method=POST path=/32032026/11/12 00:06:23 INFO Completed push id=19 objects=7 duration=1.576115ms32042026/11/12 00:06:23 INFO Received complete upload request method=POST path=/32052026/11/12 00:06:23 INFO Completed upload id=16 objects=3 duration=1.02799ms32062026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032072026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period32082026/11/12 00:06:23 INFO Received push request method=POST path=/32092026/11/12 00:06:23 INFO Received push request method=POST path=/32102026/11/12 00:06:23 INFO Received push request method=POST path=/32112026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32122026/11/12 00:06:23 INFO Received push request method=POST path=/32132026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032142026/11/12 00:06:23 INFO Received push request method=POST path=/32152026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period32162026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032172026/11/12 00:06:23 INFO Vacuumed table table=pending_closures32182026/11/12 00:06:23 INFO Vacuumed table table=pending_objects32192026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads32202026/11/12 00:06:23 INFO Vacuumed table table=closures32212026/11/12 00:06:23 INFO Received uploads request method=POST path=/32222026/11/12 00:06:23 INFO Vacuumed table table=objects32232026/11/12 00:06:23 INFO Received uploads request method=POST path=/32242026/11/12 00:06:23 INFO Received uploads request method=POST path=/32252026/11/12 00:06:23 INFO Received push request method=POST path=/32262026/11/12 00:06:23 INFO Received uploads request method=POST path=/32272026/11/12 00:06:23 INFO Received push request method=POST path=/32282026/11/12 00:06:23 INFO Received complete upload request method=POST path=/32292026/11/12 00:06:23 INFO Completed upload id=12 objects=26 duration=3.918697ms32302026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032312026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032322026/11/12 00:06:23 INFO Vacuumed table table=pending_closures32332026/11/12 00:06:23 INFO Vacuumed table table=pending_objects32342026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads32352026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032362026/11/12 00:06:23 INFO Vacuumed table table=closures32372026/11/12 00:06:23 INFO Vacuumed table table=objects32382026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032392026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period32402026/11/12 00:06:23 INFO Received complete push request method=POST path=/32412026/11/12 00:06:23 INFO Completed push id=4 objects=6 duration=1.551215ms32422026/11/12 00:06:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32432026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."32442026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032452026/11/12 00:06:23 INFO Vacuumed table table=pending_closures32462026/11/12 00:06:23 INFO Vacuumed table table=pending_objects32472026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads32482026/11/12 00:06:23 INFO Vacuumed table table=closures32492026/11/12 00:06:23 INFO Received push request method=POST path=/32502026/11/12 00:06:23 INFO Vacuumed table table=objects32512026/11/12 00:06:23 INFO Received uploads request method=POST path=/32522026/11/12 00:06:23 INFO Received uploads request method=POST path=/32532026/11/12 00:06:23 INFO Received uploads request method=POST path=/32542026/11/12 00:06:23 INFO Received complete push request method=POST path=/32552026/11/12 00:06:23 INFO Completed push id=16 objects=8 duration=1.756617ms32562026/11/12 00:06:23 INFO Received push request method=POST path=/32572026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032582026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period32592026/11/12 00:06:23 INFO Received uploads request method=POST path=/32602026/11/12 00:06:23 INFO Received push request method=POST path=/32612026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=032622026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032632026/11/12 00:06:23 INFO Vacuumed table table=pending_closures32642026/11/12 00:06:23 INFO Vacuumed table table=pending_objects32652026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads32662026/11/12 00:06:23 INFO Vacuumed table table=closures32672026/11/12 00:06:23 INFO Vacuumed table table=objects32682026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period32692026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032702026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032712026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period32722026/11/12 00:06:23 INFO Vacuumed table table=pending_closures32732026/11/12 00:06:23 INFO Vacuumed table table=pending_objects32742026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads32752026/11/12 00:06:23 INFO Received complete push request method=POST path=/32762026/11/12 00:06:23 INFO Vacuumed table table=closures32772026/11/12 00:06:23 INFO Vacuumed table table=objects32782026/11/12 00:06:23 INFO Received complete push request method=POST path=/32792026/11/12 00:06:23 INFO Completed push id=26 objects=14 duration=10.948204ms32802026/11/12 00:06:23 INFO Completed push id=14 objects=3 duration=6.198979ms32812026/11/12 00:06:23 INFO Received complete upload request method=POST path=/32822026/11/12 00:06:23 INFO Completed upload id=21 objects=7 duration=2.164521ms32832026/11/12 00:06:23 INFO Received push request method=POST path=/32842026/11/12 00:06:23 INFO Received complete push request method=POST path=/32852026/11/12 00:06:23 INFO Received complete push request method=POST path=/32862026/11/12 00:06:23 INFO Completed push id=11 objects=6 duration=12.54246ms32872026/11/12 00:06:23 INFO Received uploads request method=POST path=/32882026/11/12 00:06:23 INFO Completed push id=21 objects=9 duration=13.521909ms32892026/11/12 00:06:23 INFO Received push request method=POST path=/32902026/11/12 00:06:23 INFO Received complete push request method=POST path=/32912026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032922026/11/12 00:06:23 INFO Completed push id=18 objects=6 duration=1.346253ms32932026/11/12 00:06:23 INFO Vacuumed table table=pending_closures32942026/11/12 00:06:23 INFO Vacuumed table table=pending_objects32952026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads32962026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032972026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=032982026/11/12 00:06:23 INFO Vacuumed table table=closures32992026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33002026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033012026/11/12 00:06:23 INFO Vacuumed table table=objects33022026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=033032026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33042026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033052026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33062026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33072026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033082026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033092026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033102026/11/12 00:06:23 INFO Vacuumed table table=closures33112026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33122026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33132026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33142026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=033152026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33162026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33172026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33182026/11/12 00:06:23 INFO Vacuumed table table=objects33192026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33202026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33212026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33222026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33232026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33242026/11/12 00:06:23 INFO Vacuumed table table=closures33252026/11/12 00:06:23 INFO Vacuumed table table=closures33262026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33272026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33282026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33292026/11/12 00:06:23 INFO Vacuumed table table=objects33302026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33312026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33322026/11/12 00:06:23 INFO Vacuumed table table=closures33332026/11/12 00:06:23 INFO Vacuumed table table=objects33342026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33352026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33362026/11/12 00:06:23 INFO Vacuumed table table=closures33372026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33382026/11/12 00:06:23 INFO Vacuumed table table=objects33392026/11/12 00:06:23 INFO Vacuumed table table=closures33402026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33412026/11/12 00:06:23 INFO Vacuumed table table=closures33422026/11/12 00:06:23 INFO Vacuumed table table=objects33432026/11/12 00:06:23 INFO Received push request method=POST path=/33442026/11/12 00:06:23 INFO Vacuumed table table=objects33452026/11/12 00:06:23 INFO Vacuumed table table=closures33462026/11/12 00:06:23 INFO Vacuumed table table=objects33472026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=033482026/11/12 00:06:23 INFO Vacuumed table table=objects33492026/11/12 00:06:23 INFO Received uploads request method=POST path=/33502026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=033512026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33522026/11/12 00:06:23 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."33532026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."33542026/11/12 00:06:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."33552026/11/12 00:06:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33562026/11/12 00:06:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33572026/11/12 00:06:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33582026/11/12 00:06:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."33592026/11/12 00:06:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."33602026/11/12 00:06:23 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."33612026/11/12 00:06:23 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33622026/11/12 00:06:23 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33632026/11/12 00:06:23 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."33642026/11/12 00:06:23 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."33652026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."33662026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33672026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."33682026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33692026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."33702026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."33712026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."33722026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period33732026/11/12 00:06:23 INFO Received uploads request method=POST path=/33742026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033752026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33762026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33772026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33782026/11/12 00:06:23 INFO Vacuumed table table=closures33792026/11/12 00:06:23 INFO Received uploads request method=POST path=/33802026/11/12 00:06:23 INFO Vacuumed table table=objects33812026/11/12 00:06:23 INFO Received push request method=POST path=/33822026/11/12 00:06:23 INFO Received uploads request method=POST path=/33832026/11/12 00:06:23 INFO Received complete push request method=POST path=/33842026/11/12 00:06:23 INFO Completed push id=8 objects=6 duration=2.669946ms33852026/11/12 00:06:23 INFO Received push request method=POST path=/33862026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=033872026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period33882026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033892026/11/12 00:06:23 INFO Vacuumed table table=pending_closures33902026/11/12 00:06:23 INFO Received push request method=POST path=/33912026/11/12 00:06:23 INFO Vacuumed table table=pending_objects33922026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads33932026/11/12 00:06:23 INFO Received uploads request method=POST path=/33942026/11/12 00:06:23 INFO Vacuumed table table=closures33952026/11/12 00:06:23 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33962026/11/12 00:06:23 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33972026/11/12 00:06:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33982026/11/12 00:06:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33992026/11/12 00:06:23 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."34002026/11/12 00:06:23 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."34012026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."34022026/11/12 00:06:23 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."34032026/11/12 00:06:23 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."34042026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."34052026/11/12 00:06:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."34062026/11/12 00:06:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."34072026/11/12 00:06:23 INFO Vacuumed table table=objects34082026/11/12 00:06:23 INFO Received push request method=POST path=/34092026/11/12 00:06:23 INFO Received uploads request method=POST path=/34102026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=034112026/11/12 00:06:23 INFO Received uploads request method=POST path=/34122026/11/12 00:06:23 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34132026/11/12 00:06:23 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."34142026/11/12 00:06:23 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34152026/11/12 00:06:23 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."34162026/11/12 00:06:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."34172026/11/12 00:06:23 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."34182026/11/12 00:06:23 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."34192026/11/12 00:06:23 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."34202026/11/12 00:06:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."34212026/11/12 00:06:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."34222026/11/12 00:06:23 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."34232026/11/12 00:06:23 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."34242026/11/12 00:06:23 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."34252026/11/12 00:06:23 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."34262026/11/12 00:06:23 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."34272026/11/12 00:06:23 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."34282026/11/12 00:06:23 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."34292026/11/12 00:06:23 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."34302026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34312026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."34322026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."34332026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."34342026/11/12 00:06:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."34352026/11/12 00:06:23 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."34362026/11/12 00:06:23 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."34372026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period34382026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034392026/11/12 00:06:23 INFO Vacuumed table table=pending_closures34402026/11/12 00:06:23 INFO Vacuumed table table=pending_objects34412026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads34422026/11/12 00:06:23 INFO Vacuumed table table=closures34432026/11/12 00:06:23 INFO Received uploads request method=POST path=/34442026/11/12 00:06:23 INFO Vacuumed table table=objects34452026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=034462026/11/12 00:06:23 INFO Received complete upload request method=POST path=/34472026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period34482026/11/12 00:06:23 INFO Received push request method=POST path=/34492026/11/12 00:06:23 INFO Completed upload id=27 objects=4 duration=1.13765ms34502026/11/12 00:06:23 INFO Received push request method=POST path=/34512026/11/12 00:06:23 INFO Received push request method=POST path=/34522026/11/12 00:06:23 INFO Received push request method=POST path=/34532026/11/12 00:06:23 INFO Received push request method=POST path=/34542026/11/12 00:06:23 INFO Received uploads request method=POST path=/34552026/11/12 00:06:23 INFO Received complete upload request method=POST path=/34562026/11/12 00:06:23 INFO Received complete push request method=POST path=/34572026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=034582026/11/12 00:06:23 INFO Received push request method=POST path=/34592026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period34602026/11/12 00:06:23 INFO Completed upload id=19 objects=25 duration=3.075089ms34612026/11/12 00:06:23 INFO Received uploads request method=POST path=/34622026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034632026/11/12 00:06:23 INFO Completed push id=11 objects=11 duration=4.086399ms34642026/11/12 00:06:23 INFO Received complete push request method=POST path=/34652026/11/12 00:06:23 INFO Vacuumed table table=pending_closures34662026/11/12 00:06:23 INFO Vacuumed table table=pending_objects34672026/11/12 00:06:23 INFO Completed push id=13 objects=11 duration=1.728657ms34682026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=034692026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads34702026/11/12 00:06:23 INFO Vacuumed table table=closures34712026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=034722026/11/12 00:06:23 INFO Vacuumed table table=objects34732026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period34742026/11/12 00:06:23 INFO Vacuumed table table=pending_closures34752026/11/12 00:06:23 INFO Received uploads request method=POST path=/34762026/11/12 00:06:23 INFO Vacuumed table table=pending_objects34772026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034782026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads34792026/11/12 00:06:23 INFO Vacuumed table table=pending_closures34802026/11/12 00:06:23 INFO Vacuumed table table=closures34812026/11/12 00:06:23 INFO Vacuumed table table=pending_objects34822026/11/12 00:06:23 INFO Vacuumed table table=objects34832026/11/12 00:06:23 INFO Received push request method=POST path=/34842026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads34852026/11/12 00:06:23 INFO Vacuumed table table=closures34862026/11/12 00:06:23 INFO Received push request method=POST path=/34872026/11/12 00:06:23 INFO Received push request method=POST path=/34882026/11/12 00:06:23 INFO Received push request method=POST path=/34892026/11/12 00:06:23 INFO Vacuumed table table=objects34902026/11/12 00:06:23 INFO Received uploads request method=POST path=/34912026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=034922026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period34932026/11/12 00:06:23 INFO Received push request method=POST path=/34942026/11/12 00:06:23 INFO Received push request method=POST path=/34952026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034962026/11/12 00:06:23 INFO Received uploads request method=POST path=/34972026/11/12 00:06:23 INFO Vacuumed table table=pending_closures34982026/11/12 00:06:23 INFO Vacuumed table table=pending_objects34992026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads35002026/11/12 00:06:23 INFO Received push request method=POST path=/35012026/11/12 00:06:23 INFO Vacuumed table table=closures35022026/11/12 00:06:23 INFO Vacuumed table table=objects35032026/11/12 00:06:23 INFO Received push request method=POST path=/35042026/11/12 00:06:23 INFO Received push request method=POST path=/35052026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035062026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035072026/11/12 00:06:23 INFO Received complete upload request method=POST path=/35082026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period35092026/11/12 00:06:23 INFO Completed upload id=17 objects=25 duration=3.414732ms35102026/11/12 00:06:23 INFO Received push request method=POST path=/35112026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=035122026/11/12 00:06:23 INFO Vacuumed table table=pending_closures35132026/11/12 00:06:23 INFO Received complete push request method=POST path=/35142026/11/12 00:06:23 INFO Vacuumed table table=pending_objects35152026/11/12 00:06:23 INFO Received complete push request method=POST path=/35162026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads35172026/11/12 00:06:23 INFO Completed push id=9 objects=24 duration=1.932558ms35182026/11/12 00:06:23 INFO Received uploads request method=POST path=/35192026/11/12 00:06:23 INFO Vacuumed table table=closures35202026/11/12 00:06:23 INFO Completed push id=4 objects=6 duration=2.346063ms35212026/11/12 00:06:23 INFO Vacuumed table table=objects35222026/11/12 00:06:23 INFO Received push request method=POST path=/35232026/11/12 00:06:23 INFO Received complete upload request method=POST path=/35242026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035252026/11/12 00:06:23 INFO Completed upload id=6 objects=24 duration=3.366112ms35262026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035272026/11/12 00:06:23 INFO Received complete upload request method=POST path=/35282026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035292026/11/12 00:06:23 INFO Received push request method=POST path=/35302026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period35312026/11/12 00:06:23 INFO Vacuumed table table=pending_closures35322026/11/12 00:06:23 INFO Received push request method=POST path=/35332026/11/12 00:06:23 INFO Completed upload id=4 objects=14 duration=3.282711ms35342026/11/12 00:06:23 INFO Vacuumed table table=pending_objects35352026/11/12 00:06:23 INFO Received push request method=POST path=/35362026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads35372026/11/12 00:06:23 INFO Vacuumed table table=closures35382026/11/12 00:06:23 INFO Vacuumed table table=objects35392026/11/12 00:06:23 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."35402026/11/12 00:06:23 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."35412026/11/12 00:06:23 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."35422026/11/12 00:06:23 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."35432026/11/12 00:06:23 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."35442026/11/12 00:06:23 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."35452026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."35462026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."35472026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35482026/11/12 00:06:23 INFO Received uploads request method=POST path=/35492026/11/12 00:06:23 INFO Received push request method=POST path=/35502026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035512026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period35522026/11/12 00:06:23 INFO Received push request method=POST path=/35532026/11/12 00:06:23 INFO Received complete upload request method=POST path=/35542026/11/12 00:06:23 INFO Received uploads request method=POST path=/35552026/11/12 00:06:23 INFO Received push request method=POST path=/35562026/11/12 00:06:23 INFO Completed upload id=21 objects=2 duration=1.301192ms35572026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035582026/11/12 00:06:23 INFO Vacuumed table table=pending_closures35592026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035602026/11/12 00:06:23 INFO Received push request method=POST path=/35612026/11/12 00:06:23 INFO Vacuumed table table=pending_objects35622026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads35632026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period35642026/11/12 00:06:23 INFO Vacuumed table table=closures35652026/11/12 00:06:23 INFO Vacuumed table table=objects35662026/11/12 00:06:23 INFO Received uploads request method=POST path=/35672026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035682026/11/12 00:06:23 INFO Received complete upload request method=POST path=/35692026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period35702026/11/12 00:06:23 INFO Received uploads request method=POST path=/35712026/11/12 00:06:23 INFO Completed upload id=10 objects=2 duration=2.0623ms35722026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035732026/11/12 00:06:23 INFO Received uploads request method=POST path=/35742026/11/12 00:06:23 INFO Vacuumed table table=pending_closures35752026/11/12 00:06:23 INFO Received complete upload request method=POST path=/35762026/11/12 00:06:23 INFO Received uploads request method=POST path=/35772026/11/12 00:06:23 INFO Vacuumed table table=pending_objects35782026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads35792026/11/12 00:06:23 INFO Completed upload id=9 objects=2 duration=1.321633ms35802026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035812026/11/12 00:06:23 INFO Vacuumed table table=closures35822026/11/12 00:06:23 INFO Received complete push request method=POST path=/35832026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period35842026/11/12 00:06:23 INFO Received push request method=POST path=/35852026/11/12 00:06:23 INFO Vacuumed table table=objects35862026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=035872026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035882026/11/12 00:06:23 INFO Received complete push request method=POST path=/35892026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=035902026/11/12 00:06:23 INFO Completed push id=27 objects=25 duration=3.409373ms35912026/11/12 00:06:23 INFO Vacuumed table table=pending_closures35922026/11/12 00:06:23 INFO Vacuumed table table=pending_closures35932026/11/12 00:06:23 INFO Completed push id=8 objects=6 duration=1.923558ms35942026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period35952026/11/12 00:06:23 INFO Vacuumed table table=pending_objects35962026/11/12 00:06:23 INFO Vacuumed table table=pending_objects35972026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads35982026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035992026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads36002026/11/12 00:06:23 INFO Vacuumed table table=closures36012026/11/12 00:06:23 INFO Vacuumed table table=closures36022026/11/12 00:06:23 INFO Vacuumed table table=pending_closures36032026/11/12 00:06:23 INFO Vacuumed table table=objects36042026/11/12 00:06:23 INFO Vacuumed table table=objects36052026/11/12 00:06:23 INFO Vacuumed table table=pending_objects36062026/11/12 00:06:23 INFO Received uploads request method=POST path=/36072026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads36082026/11/12 00:06:23 INFO Vacuumed table table=closures36092026/11/12 00:06:23 INFO Received push request method=POST path=/36102026/11/12 00:06:23 INFO Vacuumed table table=objects36112026/11/12 00:06:23 INFO Received complete upload request method=POST path=/36122026/11/12 00:06:23 INFO Received complete push request method=POST path=/36132026/11/12 00:06:23 INFO Completed upload id=10 objects=10 duration=963.869µs36142026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=036152026/11/12 00:06:23 INFO Completed push id=8 objects=3 duration=1.564875ms36162026/11/12 00:06:23 INFO Received uploads request method=POST path=/36172026/11/12 00:06:23 INFO Received push request method=POST path=/36182026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=036192026/11/12 00:06:23 INFO Received push request method=POST path=/36202026/11/12 00:06:23 INFO Received uploads request method=POST path=/36212026/11/12 00:06:23 INFO Received complete upload request method=POST path=/36222026/11/12 00:06:23 INFO Received uploads request method=POST path=/36232026/11/12 00:06:23 INFO Received complete upload request method=POST path=/36242026/11/12 00:06:23 INFO Completed upload id=13 objects=8 duration=1.351253ms36252026/11/12 00:06:23 INFO Received push request method=POST path=/36262026/11/12 00:06:23 INFO Completed upload id=9 objects=3 duration=1.685796ms36272026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036282026/11/12 00:06:23 INFO Received push request method=POST path=/36292026/11/12 00:06:23 INFO Vacuumed table table=pending_closures36302026/11/12 00:06:23 INFO Received push request method=POST path=/36312026/11/12 00:06:23 INFO Vacuumed table table=pending_objects36322026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=036332026/11/12 00:06:23 INFO Received uploads request method=POST path=/36342026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads36352026/11/12 00:06:23 INFO Vacuumed table table=closures36362026/11/12 00:06:23 INFO Vacuumed table table=pending_closures36372026/11/12 00:06:23 INFO Vacuumed table table=objects36382026/11/12 00:06:23 INFO Vacuumed table table=pending_objects36392026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads36402026/11/12 00:06:23 INFO Vacuumed table table=closures36412026/11/12 00:06:23 INFO Received uploads request method=POST path=/36422026/11/12 00:06:23 INFO Vacuumed table table=objects36432026/11/12 00:06:23 INFO Received push request method=POST path=/36442026/11/12 00:06:23 INFO Received uploads request method=POST path=/36452026/11/12 00:06:23 INFO Received push request method=POST path=/36462026/11/12 00:06:23 INFO Received uploads request method=POST path=/36472026/11/12 00:06:23 INFO Received push request method=POST path=/36482026/11/12 00:06:23 INFO Received uploads request method=POST path=/36492026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=036502026/11/12 00:06:23 INFO Received complete push request method=POST path=/36512026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036522026/11/12 00:06:23 INFO Completed push id=24 objects=5 duration=2.249542ms36532026/11/12 00:06:23 INFO Received uploads request method=POST path=/36542026/11/12 00:06:23 INFO Vacuumed table table=pending_closures36552026/11/12 00:06:23 INFO Vacuumed table table=pending_objects36562026/11/12 00:06:23 INFO Received push request method=POST path=/36572026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads36582026/11/12 00:06:23 INFO Vacuumed table table=closures36592026/11/12 00:06:23 INFO Vacuumed table table=objects36602026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=036612026/11/12 00:06:23 INFO Received uploads request method=POST path=/36622026/11/12 00:06:23 INFO Received complete upload request method=POST path=/36632026/11/12 00:06:23 INFO Received push request method=POST path=/36642026/11/12 00:06:23 INFO Received push request method=POST path=/36652026/11/12 00:06:23 INFO Completed upload id=13 objects=10 duration=932.689µs36662026/11/12 00:06:23 INFO Received push request method=POST path=/36672026/11/12 00:06:23 INFO Received uploads request method=POST path=/36682026/11/12 00:06:23 INFO Received complete push request method=POST path=/36692026/11/12 00:06:23 INFO Completed push id=7 objects=24 duration=3.342972ms36702026/11/12 00:06:23 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36712026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36722026/11/12 00:06:23 INFO Received complete upload request method=POST path=/36732026/11/12 00:06:23 INFO Received complete upload request method=POST path=/36742026/11/12 00:06:23 INFO Received complete push request method=POST path=/36752026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=036762026/11/12 00:06:23 INFO Received push request method=POST path=/36772026/11/12 00:06:23 INFO Received push request method=POST path=/36782026/11/12 00:06:23 INFO Completed upload id=10 objects=6 duration=1.891858ms36792026/11/12 00:06:23 INFO Completed upload id=23 objects=18 duration=2.445383ms36802026/11/12 00:06:23 INFO Completed push id=5 objects=8 duration=2.755147ms36812026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period36822026/11/12 00:06:23 INFO Received complete upload request method=POST path=/36832026/11/12 00:06:23 INFO Completed upload id=23 objects=16 duration=2.08524ms36842026/11/12 00:06:23 INFO Received push request method=POST path=/36852026/11/12 00:06:23 INFO Received uploads request method=POST path=/36862026/11/12 00:06:23 INFO Received uploads request method=POST path=/36872026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=036882026/11/12 00:06:23 INFO Received uploads request method=POST path=/36892026/11/12 00:06:23 INFO Received push request method=POST path=/36902026/11/12 00:06:23 INFO Received complete push request method=POST path=/36912026/11/12 00:06:23 INFO Received uploads request method=POST path=/36922026/11/12 00:06:23 INFO Completed push id=11 objects=8 duration=2.951448ms36932026/11/12 00:06:23 INFO Received uploads request method=POST path=/36942026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=036952026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period36962026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036972026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036982026/11/12 00:06:23 INFO Vacuumed table table=pending_closures36992026/11/12 00:06:23 INFO Vacuumed table table=pending_closures37002026/11/12 00:06:23 INFO Vacuumed table table=pending_objects37012026/11/12 00:06:23 INFO Vacuumed table table=pending_objects37022026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads37032026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads37042026/11/12 00:06:23 INFO Received uploads request method=POST path=/37052026/11/12 00:06:23 INFO Vacuumed table table=closures37062026/11/12 00:06:23 INFO Vacuumed table table=closures37072026/11/12 00:06:23 INFO Vacuumed table table=objects37082026/11/12 00:06:23 INFO Vacuumed table table=objects37092026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=037102026/11/12 00:06:23 INFO Received uploads request method=POST path=/37112026/11/12 00:06:23 INFO Received uploads request method=POST path=/37122026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037132026/11/12 00:06:23 INFO Received push request method=POST path=/37142026/11/12 00:06:23 INFO Vacuumed table table=pending_closures37152026/11/12 00:06:23 INFO Received complete push request method=POST path=/37162026/11/12 00:06:23 INFO Vacuumed table table=pending_objects37172026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads37182026/11/12 00:06:23 INFO Vacuumed table table=closures37192026/11/12 00:06:23 INFO Completed push id=16 objects=4 duration=2.497104ms37202026/11/12 00:06:23 INFO Vacuumed table table=objects37212026/11/12 00:06:23 INFO Received uploads request method=POST path=/37222026/11/12 00:06:23 INFO Received push request method=POST path=/37232026/11/12 00:06:23 INFO Received push request method=POST path=/37242026/11/12 00:06:23 INFO Received complete push request method=POST path=/37252026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=037262026/11/12 00:06:23 INFO Received uploads request method=POST path=/37272026/11/12 00:06:23 INFO Completed push id=12 objects=3 duration=1.713517ms37282026/11/12 00:06:23 INFO Received complete push request method=POST path=/37292026/11/12 00:06:23 INFO Vacuumed table table=pending_closures37302026/11/12 00:06:23 INFO Received complete push request method=POST path=/37312026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=037322026/11/12 00:06:23 INFO Vacuumed table table=pending_objects37332026/11/12 00:06:23 INFO Completed push id=5 objects=28 duration=2.0748ms37342026/11/12 00:06:23 INFO Completed push id=21 objects=26 duration=1.491534ms37352026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads37362026/11/12 00:06:23 INFO Received uploads request method=POST path=/37372026/11/12 00:06:23 INFO Vacuumed table table=closures37382026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037392026/11/12 00:06:23 INFO Vacuumed table table=objects37402026/11/12 00:06:23 INFO Received push request method=POST path=/37412026/11/12 00:06:23 INFO Vacuumed table table=pending_closures37422026/11/12 00:06:23 INFO Received complete upload request method=POST path=/37432026/11/12 00:06:23 INFO Vacuumed table table=pending_objects37442026/11/12 00:06:23 INFO Received uploads request method=POST path=/37452026/11/12 00:06:23 INFO Completed upload id=22 objects=22 duration=853.709µs37462026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads37472026/11/12 00:06:23 INFO Vacuumed table table=closures37482026/11/12 00:06:23 INFO Vacuumed table table=objects37492026/11/12 00:06:23 INFO Received complete push request method=POST path=/37502026/11/12 00:06:23 INFO Received complete push request method=POST path=/37512026/11/12 00:06:23 INFO Completed push id=14 objects=3 duration=980.589µs37522026/11/12 00:06:23 INFO Completed push id=8 objects=6 duration=1.531514ms37532026/11/12 00:06:23 INFO Received uploads request method=POST path=/37542026/11/12 00:06:23 INFO Received complete push request method=POST path=/37552026/11/12 00:06:23 INFO Received complete push request method=POST path=/37562026/11/12 00:06:23 INFO Received push request method=POST path=/37572026/11/12 00:06:23 INFO Received uploads request method=POST path=/37582026/11/12 00:06:23 INFO Received complete push request method=POST path=/37592026/11/12 00:06:23 INFO Completed push id=13 objects=12 duration=3.12231ms37602026/11/12 00:06:23 INFO Received push request method=POST path=/37612026/11/12 00:06:23 INFO Completed push id=10 objects=2 duration=1.233732ms37622026/11/12 00:06:23 INFO Completed push id=13 objects=6 duration=2.934128ms37632026/11/12 00:06:23 INFO Received complete upload request method=POST path=/37642026/11/12 00:06:23 INFO Received push request method=POST path=/37652026/11/12 00:06:23 INFO Completed upload id=20 objects=5 duration=843.828µs37662026/11/12 00:06:23 INFO Received complete upload request method=POST path=/37672026/11/12 00:06:23 INFO Completed upload id=28 objects=17 duration=1.06839ms37682026/11/12 00:06:23 INFO Received push request method=POST path=/37692026/11/12 00:06:23 INFO Received uploads request method=POST path=/37702026/11/12 00:06:23 INFO Received complete push request method=POST path=/37712026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=037722026/11/12 00:06:23 INFO Completed push id=12 objects=7 duration=1.552855ms37732026/11/12 00:06:23 INFO Received push request method=POST path=/37742026/11/12 00:06:23 INFO Received uploads request method=POST path=/37752026/11/12 00:06:23 INFO Received push request method=POST path=/37762026/11/12 00:06:23 INFO Received uploads request method=POST path=/37772026/11/12 00:06:23 INFO Received uploads request method=POST path=/37782026/11/12 00:06:23 INFO Received uploads request method=POST path=/37792026/11/12 00:06:23 INFO Received complete upload request method=POST path=/37802026/11/12 00:06:23 INFO Received complete push request method=POST path=/37812026/11/12 00:06:23 INFO Received complete push request method=POST path=/37822026/11/12 00:06:23 INFO Completed upload id=17 objects=5 duration=1.05577ms37832026/11/12 00:06:23 INFO Completed push id=26 objects=5 duration=1.705457ms37842026/11/12 00:06:23 INFO Completed push id=29 objects=2 duration=1.05609ms37852026/11/12 00:06:23 INFO Received push request method=POST path=/37862026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037872026/11/12 00:06:23 INFO Vacuumed table table=pending_closures37882026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=037892026/11/12 00:06:23 INFO Vacuumed table table=pending_objects37902026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads37912026/11/12 00:06:23 INFO Received uploads request method=POST path=/37922026/11/12 00:06:23 INFO Vacuumed table table=closures37932026/11/12 00:06:23 INFO Vacuumed table table=objects37942026/11/12 00:06:23 INFO Received complete push request method=POST path=/37952026/11/12 00:06:23 INFO Completed push id=13 objects=2 duration=1.11443ms37962026/11/12 00:06:23 INFO Received complete push request method=POST path=/37972026/11/12 00:06:23 INFO Received push request method=POST path=/37982026/11/12 00:06:23 INFO Completed push id=14 objects=4 duration=2.842747ms37992026/11/12 00:06:23 INFO Received complete push request method=POST path=/38002026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38012026/11/12 00:06:23 INFO Completed upload id=24 objects=14 duration=958.49µs38022026/11/12 00:06:23 INFO Completed push id=31 objects=6 duration=1.305093ms38032026/11/12 00:06:23 INFO Received complete push request method=POST path=/38042026/11/12 00:06:23 INFO Received push request method=POST path=/38052026/11/12 00:06:23 INFO Completed push id=19 objects=3 duration=1.002689ms38062026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=038072026/11/12 00:06:23 INFO Received push request method=POST path=/38082026/11/12 00:06:23 INFO Vacuumed table table=pending_closures38092026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=038102026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38112026/11/12 00:06:23 INFO Vacuumed table table=pending_objects38122026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period38132026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads38142026/11/12 00:06:23 INFO Completed upload id=27 objects=13 duration=968.75µs38152026/11/12 00:06:23 INFO Vacuumed table table=closures38162026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038172026/11/12 00:06:23 INFO Vacuumed table table=objects38182026/11/12 00:06:23 INFO Received push request method=POST path=/38192026/11/12 00:06:23 INFO Vacuumed table table=pending_closures38202026/11/12 00:06:23 INFO Vacuumed table table=pending_objects38212026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads38222026/11/12 00:06:23 INFO Vacuumed table table=closures38232026/11/12 00:06:23 INFO Received push request method=POST path=/38242026/11/12 00:06:23 INFO Vacuumed table table=objects38252026/11/12 00:06:23 INFO Received complete push request method=POST path=/38262026/11/12 00:06:23 INFO Received complete push request method=POST path=/38272026/11/12 00:06:23 INFO Completed push id=19 objects=3 duration=917.729µs38282026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38292026/11/12 00:06:23 INFO Completed push id=32 objects=9 duration=1.335732ms38302026/11/12 00:06:23 INFO Completed upload id=16 objects=23 duration=1.130951ms38312026/11/12 00:06:23 INFO Received push request method=POST path=/38322026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38332026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=038342026/11/12 00:06:23 INFO Completed upload id=8 objects=3 duration=2.505864ms38352026/11/12 00:06:23 INFO Received complete push request method=POST path=/38362026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period38372026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=038382026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038392026/11/12 00:06:23 INFO Completed push id=17 objects=6 duration=1.869878ms38402026/11/12 00:06:23 INFO Vacuumed table table=pending_closures38412026/11/12 00:06:23 INFO Received push request method=POST path=/38422026/11/12 00:06:23 INFO Received complete push request method=POST path=/38432026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=038442026/11/12 00:06:23 INFO Vacuumed table table=pending_objects38452026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads38462026/11/12 00:06:23 INFO Received complete push request method=POST path=/38472026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038482026/11/12 00:06:23 INFO Completed push id=20 objects=4 duration=1.157151ms38492026/11/12 00:06:23 INFO Vacuumed table table=closures38502026/11/12 00:06:23 INFO Vacuumed table table=pending_closures38512026/11/12 00:06:23 INFO Vacuumed table table=objects38522026/11/12 00:06:23 INFO Vacuumed table table=pending_objects38532026/11/12 00:06:23 INFO Received uploads request method=POST path=/38542026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads38552026/11/12 00:06:23 INFO Completed push id=7 objects=15 duration=2.420063ms38562026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038572026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38582026/11/12 00:06:23 INFO Vacuumed table table=closures38592026/11/12 00:06:23 INFO Vacuumed table table=pending_closures38602026/11/12 00:06:23 INFO Completed upload id=15 objects=6 duration=834.828µs38612026/11/12 00:06:23 INFO Vacuumed table table=objects38622026/11/12 00:06:23 INFO Vacuumed table table=pending_objects38632026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads38642026/11/12 00:06:23 INFO Vacuumed table table=closures38652026/11/12 00:06:23 INFO Received uploads request method=POST path=/38662026/11/12 00:06:23 INFO Vacuumed table table=objects38672026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38682026/11/12 00:06:23 INFO Completed upload id=33 objects=9 duration=829.808µs38692026/11/12 00:06:23 INFO Received push request method=POST path=/38702026/11/12 00:06:23 INFO Received uploads request method=POST path=/38712026/11/12 00:06:23 INFO Received push request method=POST path=/38722026/11/12 00:06:23 INFO Received uploads request method=POST path=/38732026/11/12 00:06:23 INFO Received uploads request method=POST path=/38742026/11/12 00:06:23 INFO Received uploads request method=POST path=/38752026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=038762026/11/12 00:06:23 INFO Received complete push request method=POST path=/38772026/11/12 00:06:23 INFO Received complete push request method=POST path=/38782026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38792026/11/12 00:06:23 INFO Completed push id=4 objects=7 duration=1.565314ms38802026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period38812026/11/12 00:06:23 INFO Completed upload id=22 objects=8 duration=983.889µs38822026/11/12 00:06:23 INFO Completed push id=19 objects=6 duration=3.003029ms38832026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038842026/11/12 00:06:23 INFO Vacuumed table table=pending_closures38852026/11/12 00:06:23 INFO Received push request method=POST path=/38862026/11/12 00:06:23 INFO Vacuumed table table=pending_objects38872026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads38882026/11/12 00:06:23 INFO Vacuumed table table=closures38892026/11/12 00:06:23 INFO Vacuumed table table=objects38902026/11/12 00:06:23 INFO Received uploads request method=POST path=/38912026/11/12 00:06:23 INFO Received uploads request method=POST path=/38922026/11/12 00:06:23 INFO Received push request method=POST path=/38932026/11/12 00:06:23 INFO Received uploads request method=POST path=/38942026/11/12 00:06:23 INFO Received complete upload request method=POST path=/38952026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=038962026/11/12 00:06:23 INFO Completed upload id=17 objects=7 duration=1.494935ms38972026/11/12 00:06:23 INFO Received complete push request method=POST path=/38982026/11/12 00:06:23 INFO Completed push id=23 objects=5 duration=1.09003ms38992026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period39002026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039012026/11/12 00:06:23 INFO Received complete push request method=POST path=/39022026/11/12 00:06:23 INFO Vacuumed table table=pending_closures39032026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=039042026/11/12 00:06:23 INFO Received push request method=POST path=/39052026/11/12 00:06:23 INFO Completed push id=6 objects=6 duration=1.842298ms39062026/11/12 00:06:23 INFO Received complete push request method=POST path=/39072026/11/12 00:06:23 INFO Vacuumed table table=pending_objects39082026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads39092026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period39102026/11/12 00:06:23 INFO Vacuumed table table=closures39112026/11/12 00:06:23 INFO Received uploads request method=POST path=/39122026/11/12 00:06:23 INFO Vacuumed table table=objects39132026/11/12 00:06:23 INFO Completed push id=9 objects=4 duration=2.440083ms39142026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039152026/11/12 00:06:23 INFO Received uploads request method=POST path=/39162026/11/12 00:06:23 INFO Vacuumed table table=pending_closures39172026/11/12 00:06:23 INFO Received uploads request method=POST path=/39182026/11/12 00:06:23 INFO Vacuumed table table=pending_objects39192026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads39202026/11/12 00:06:23 INFO Received push request method=POST path=/39212026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=039222026/11/12 00:06:23 INFO Vacuumed table table=closures39232026/11/12 00:06:23 INFO Received uploads request method=POST path=/39242026/11/12 00:06:23 INFO Vacuumed table table=objects39252026/11/12 00:06:23 INFO Received complete upload request method=POST path=/39262026/11/12 00:06:23 INFO Completed upload id=24 objects=8 duration=782.148µs39272026/11/12 00:06:23 INFO Received push request method=POST path=/39282026/11/12 00:06:23 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."39292026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."39302026/11/12 00:06:23 INFO Received uploads request method=POST path=/39312026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=039322026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period39332026/11/12 00:06:23 INFO Received complete upload request method=POST path=/39342026/11/12 00:06:23 INFO Completed upload id=25 objects=4 duration=1.137791ms39352026/11/12 00:06:23 INFO Received uploads request method=POST path=/39362026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=039372026/11/12 00:06:23 INFO Received complete upload request method=POST path=/39382026/11/12 00:06:23 INFO Completed upload id=28 objects=5 duration=871.629µs39392026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=039402026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039412026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period39422026/11/12 00:06:23 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39432026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39442026/11/12 00:06:23 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39452026/11/12 00:06:23 INFO Vacuumed table table=pending_closures39462026/11/12 00:06:23 INFO Vacuumed table table=pending_objects39472026/11/12 00:06:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039482026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads39492026/11/12 00:06:23 INFO Vacuumed table table=closures39502026/11/12 00:06:23 INFO Vacuumed table table=pending_closures39512026/11/12 00:06:23 INFO Vacuumed table table=objects39522026/11/12 00:06:23 INFO Received push request method=POST path=/39532026/11/12 00:06:23 INFO Vacuumed table table=pending_objects39542026/11/12 00:06:23 INFO Vacuumed table table=multipart_uploads39552026/11/12 00:06:23 INFO Vacuumed table table=closures39562026/11/12 00:06:23 INFO Vacuumed table table=objects39572026/11/12 00:06:23 INFO Received push request method=POST path=/39582026/11/12 00:06:23 INFO Received uploads request method=POST path=/39592026/11/12 00:06:23 INFO Aborted multipart uploads count=0 kept=039602026/11/12 00:06:23 INFO Received complete push request method=POST path=/39612026/11/12 00:06:23 WARN Force mode enabled - objects will be deleted immediately without grace period39622026/11/12 00:06:23 INFO Completed push id=16 objects=16 duration=3.271551ms39632026/11/12 00:06:23 INFO Received complete upload request method=POST path=/39642026/11/12 00:06:23 INFO Completed upload id=30 objects=5 duration=893.529µs39652026/11/12 00:06:23 INFO Received uploads request method=POST path=/39662026/11/12 00:06:23 INFO Received push request method=POST path=/39672026/11/12 00:06:23 INFO Received complete push request method=POST path=/39682026/11/12 00:06:23 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."39692026/11/12 00:06:23 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."39702026/11/12 00:06:23 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."39712026/11/12 00:06:23 INFO Received complete upload request method=POST path=/39722026/11/12 00:06:23 INFO Completed upload id=14 objects=6 duration=829.068µs39732026/11/12 00:06:23 INFO Completed push id=10 objects=15 duration=1.716597ms39742026/11/12 00:06:24 INFO Received complete push request method=POST path=/39752026/11/12 00:06:24 INFO Completed push id=7 objects=7 duration=1.280532ms39762026/11/12 00:06:24 INFO Received complete upload request method=POST path=/39772026/11/12 00:06:24 INFO Completed upload id=29 objects=4 duration=1.546355ms39782026/11/12 00:06:24 INFO Received push request method=POST path=/39792026/11/12 00:06:24 INFO Received push request method=POST path=/39802026/11/12 00:06:24 INFO Received uploads request method=POST path=/39812026/11/12 00:06:24 INFO Received uploads request method=POST path=/39822026/11/12 00:06:24 INFO Received complete push request method=POST path=/39832026/11/12 00:06:24 INFO Received complete push request method=POST path=/39842026/11/12 00:06:24 INFO Completed push id=31 objects=3 duration=1.06109ms39852026/11/12 00:06:24 INFO Completed push id=20 objects=8 duration=1.053291ms39862026/11/12 00:06:24 INFO Received uploads request method=POST path=/39872026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=039882026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=039892026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period39902026/11/12 00:06:24 INFO Received complete push request method=POST path=/39912026/11/12 00:06:24 INFO Received push request method=POST path=/39922026/11/12 00:06:24 INFO Completed push id=19 objects=14 duration=1.817677ms39932026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039942026/11/12 00:06:24 INFO Received uploads request method=POST path=/39952026/11/12 00:06:24 INFO Vacuumed table table=pending_closures39962026/11/12 00:06:24 INFO Vacuumed table table=pending_objects39972026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads39982026/11/12 00:06:24 INFO Received uploads request method=POST path=/39992026/11/12 00:06:24 INFO Vacuumed table table=closures40002026/11/12 00:06:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40012026/11/12 00:06:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40022026/11/12 00:06:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40032026/11/12 00:06:24 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40042026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40052026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40062026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40072026/11/12 00:06:24 INFO Vacuumed table table=objects40082026/11/12 00:06:24 INFO Received uploads request method=POST path=/40092026/11/12 00:06:24 INFO Received complete upload request method=POST path=/40102026/11/12 00:06:24 INFO Completed upload id=7 objects=3 duration=1.148571ms40112026/11/12 00:06:24 INFO Received complete push request method=POST path=/40122026/11/12 00:06:24 INFO Completed push id=32 objects=6 duration=1.139431ms40132026/11/12 00:06:24 INFO Received complete push request method=POST path=/40142026/11/12 00:06:24 INFO Received complete upload request method=POST path=/40152026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=040162026/11/12 00:06:24 INFO Completed push id=15 objects=7 duration=1.829938ms40172026/11/12 00:06:24 INFO Completed upload id=18 objects=11 duration=1.684896ms40182026/11/12 00:06:24 INFO Received uploads request method=POST path=/40192026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period40202026/11/12 00:06:24 INFO Received uploads request method=POST path=/40212026/11/12 00:06:24 INFO Received push request method=POST path=/40222026/11/12 00:06:24 INFO Received complete upload request method=POST path=/40232026/11/12 00:06:24 INFO Completed upload id=33 objects=7 duration=825.348µs40242026/11/12 00:06:24 INFO Received complete push request method=POST path=/40252026/11/12 00:06:24 INFO Received uploads request method=POST path=/40262026/11/12 00:06:24 INFO Received complete push request method=POST path=/40272026/11/12 00:06:24 INFO Received uploads request method=POST path=/40282026/11/12 00:06:24 INFO Completed push id=11 objects=6 duration=1.930658ms40292026/11/12 00:06:24 INFO Completed push id=19 objects=8 duration=3.362612ms40302026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=040312026/11/12 00:06:24 INFO Received complete push request method=POST path=/40322026/11/12 00:06:24 INFO Received complete push request method=POST path=/40332026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period40342026/11/12 00:06:24 INFO Completed push id=18 objects=23 duration=1.668636ms40352026/11/12 00:06:24 INFO Completed push id=17 objects=22 duration=1.984099ms40362026/11/12 00:06:24 INFO Received uploads request method=POST path=/40372026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040382026/11/12 00:06:24 INFO Vacuumed table table=pending_closures40392026/11/12 00:06:24 INFO Vacuumed table table=pending_objects40402026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads40412026/11/12 00:06:24 INFO Vacuumed table table=closures40422026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=040432026/11/12 00:06:24 INFO Vacuumed table table=objects40442026/11/12 00:06:24 INFO Received complete upload request method=POST path=/40452026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period40462026/11/12 00:06:24 INFO Received uploads request method=POST path=/40472026/11/12 00:06:24 INFO Received uploads request method=POST path=/40482026/11/12 00:06:24 INFO Completed upload id=34 objects=3 duration=2.318322ms40492026/11/12 00:06:24 INFO Received push request method=POST path=/40502026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=040512026/11/12 00:06:24 INFO Received complete push request method=POST path=/40522026/11/12 00:06:24 INFO Received uploads request method=POST path=/40532026/11/12 00:06:24 INFO Vacuumed table table=pending_closures40542026/11/12 00:06:24 INFO Completed push id=20 objects=13 duration=1.231332ms40552026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=040562026/11/12 00:06:24 INFO Vacuumed table table=pending_objects40572026/11/12 00:06:24 INFO Received uploads request method=POST path=/40582026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads40592026/11/12 00:06:24 INFO Received push request method=POST path=/40602026/11/12 00:06:24 INFO Vacuumed table table=closures40612026/11/12 00:06:24 INFO Vacuumed table table=pending_closures40622026/11/12 00:06:24 INFO Vacuumed table table=objects40632026/11/12 00:06:24 INFO Received complete push request method=POST path=/40642026/11/12 00:06:24 INFO Completed push id=27 objects=11 duration=1.03779ms40652026/11/12 00:06:24 INFO Vacuumed table table=pending_objects40662026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads40672026/11/12 00:06:24 INFO Vacuumed table table=closures40682026/11/12 00:06:24 INFO Vacuumed table table=objects40692026/11/12 00:06:24 INFO Received complete push request method=POST path=/40702026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=040712026/11/12 00:06:24 INFO Completed push id=35 objects=11 duration=1.078831ms40722026/11/12 00:06:24 INFO Received complete push request method=POST path=/40732026/11/12 00:06:24 INFO Completed push id=22 objects=4 duration=1.470674ms40742026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040752026/11/12 00:06:24 INFO Vacuumed table table=pending_closures40762026/11/12 00:06:24 INFO Vacuumed table table=pending_objects40772026/11/12 00:06:24 INFO Received push request method=POST path=/40782026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads40792026/11/12 00:06:24 INFO Received push request method=POST path=/40802026/11/12 00:06:24 INFO Vacuumed table table=closures40812026/11/12 00:06:24 INFO Vacuumed table table=objects40822026/11/12 00:06:24 INFO Received complete upload request method=POST path=/40832026/11/12 00:06:24 INFO Received push request method=POST path=/40842026/11/12 00:06:24 INFO Completed upload id=29 objects=26 duration=854.468µs40852026/11/12 00:06:24 INFO Received complete push request method=POST path=/40862026/11/12 00:06:24 INFO Completed push id=11 objects=9 duration=1.450853ms40872026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=040882026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period40892026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=040902026/11/12 00:06:24 INFO Received complete push request method=POST path=/40912026/11/12 00:06:24 INFO Received uploads request method=POST path=/40922026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period40932026/11/12 00:06:24 INFO Completed push id=18 objects=6 duration=2.919588ms40942026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=040952026/11/12 00:06:24 INFO Received push request method=POST path=/40962026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period40972026/11/12 00:06:24 INFO Received uploads request method=POST path=/40982026/11/12 00:06:24 INFO Received complete upload request method=POST path=/40992026/11/12 00:06:24 INFO Received push request method=POST path=/41002026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041012026/11/12 00:06:24 INFO Received complete push request method=POST path=/41022026/11/12 00:06:24 INFO Completed upload id=23 objects=7 duration=1.212132ms41032026/11/12 00:06:24 INFO Vacuumed table table=pending_closures41042026/11/12 00:06:24 INFO Received push request method=POST path=/41052026/11/12 00:06:24 INFO Completed push id=30 objects=8 duration=1.698357ms41062026/11/12 00:06:24 INFO Vacuumed table table=pending_objects41072026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads41082026/11/12 00:06:24 INFO Vacuumed table table=closures41092026/11/12 00:06:24 INFO Vacuumed table table=objects41102026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41112026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."41122026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41132026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41142026/11/12 00:06:24 INFO Completed upload id=23 objects=9 duration=1.04209ms41152026/11/12 00:06:24 INFO Completed upload id=22 objects=2 duration=1.497194ms41162026/11/12 00:06:24 INFO Completed upload id=7 objects=10 duration=1.655036ms41172026/11/12 00:06:24 INFO Received uploads request method=POST path=/41182026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=041192026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41202026/11/12 00:06:24 INFO Received complete push request method=POST path=/41212026/11/12 00:06:24 INFO Received complete push request method=POST path=/41222026/11/12 00:06:24 INFO Completed upload id=31 objects=5 duration=793.608µs41232026/11/12 00:06:24 INFO Completed push id=24 objects=5 duration=1.04435ms41242026/11/12 00:06:24 INFO Received push request method=POST path=/41252026/11/12 00:06:24 INFO Received push request method=POST path=/41262026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041272026/11/12 00:06:24 INFO Completed push id=31 objects=14 duration=2.540684ms41282026/11/12 00:06:24 INFO Vacuumed table table=pending_closures41292026/11/12 00:06:24 INFO Vacuumed table table=pending_objects41302026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads41312026/11/12 00:06:24 INFO Vacuumed table table=closures41322026/11/12 00:06:24 INFO Received uploads request method=POST path=/41332026/11/12 00:06:24 INFO Vacuumed table table=objects41342026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=041352026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=041362026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period41372026/11/12 00:06:24 INFO Received complete push request method=POST path=/41382026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41392026/11/12 00:06:24 INFO Completed push id=32 objects=3 duration=1.141311ms41402026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=041412026/11/12 00:06:24 INFO Completed upload id=25 objects=2 duration=848.189µs41422026/11/12 00:06:24 INFO Received uploads request method=POST path=/41432026/11/12 00:06:24 INFO Vacuumed table table=pending_closures41442026/11/12 00:06:24 INFO Received uploads request method=POST path=/41452026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=041462026/11/12 00:06:24 INFO Vacuumed table table=pending_objects41472026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period41482026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads41492026/11/12 00:06:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."41502026/11/12 00:06:24 INFO Vacuumed table table=closures41512026/11/12 00:06:24 INFO Vacuumed table table=objects41522026/11/12 00:06:24 INFO Received complete push request method=POST path=/41532026/11/12 00:06:24 INFO Received push request method=POST path=/41542026/11/12 00:06:24 INFO Completed push id=15 objects=4 duration=1.05819ms41552026/11/12 00:06:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."41562026/11/12 00:06:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."41572026/11/12 00:06:24 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."41582026/11/12 00:06:24 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."41592026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."41602026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."41612026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."41622026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=041632026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41642026/11/12 00:06:24 INFO Received uploads request method=POST path=/41652026/11/12 00:06:24 INFO Completed upload id=26 objects=20 duration=759.767µs41662026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period41672026/11/12 00:06:24 INFO Received uploads request method=POST path=/41682026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041692026/11/12 00:06:24 INFO Received uploads request method=POST path=/41702026/11/12 00:06:24 INFO Vacuumed table table=pending_closures41712026/11/12 00:06:24 INFO Received push request method=POST path=/41722026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=041732026/11/12 00:06:24 INFO Vacuumed table table=pending_objects41742026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads41752026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period41762026/11/12 00:06:24 INFO Received uploads request method=POST path=/41772026/11/12 00:06:24 INFO Vacuumed table table=closures41782026/11/12 00:06:24 INFO Vacuumed table table=objects41792026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041802026/11/12 00:06:24 INFO Vacuumed table table=pending_closures41812026/11/12 00:06:24 INFO Vacuumed table table=pending_objects41822026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads41832026/11/12 00:06:24 INFO Vacuumed table table=closures41842026/11/12 00:06:24 INFO Vacuumed table table=objects41852026/11/12 00:06:24 INFO Received uploads request method=POST path=/41862026/11/12 00:06:24 INFO Received complete push request method=POST path=/41872026/11/12 00:06:24 INFO Completed push id=28 objects=7 duration=903.548µs41882026/11/12 00:06:24 INFO Received uploads request method=POST path=/41892026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41902026/11/12 00:06:24 INFO Completed upload id=14 objects=8 duration=1.793097ms41912026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41922026/11/12 00:06:24 INFO Completed upload id=21 objects=3 duration=907.409µs41932026/11/12 00:06:24 INFO Received push request method=POST path=/41942026/11/12 00:06:24 INFO Received uploads request method=POST path=/41952026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41962026/11/12 00:06:24 INFO Received push request method=POST path=/41972026/11/12 00:06:24 INFO Completed upload id=13 objects=14 duration=2.839527ms41982026/11/12 00:06:24 INFO Received complete upload request method=POST path=/41992026/11/12 00:06:24 INFO Completed upload id=11 objects=7 duration=1.101431ms42002026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=042012026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42022026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period42032026/11/12 00:06:24 INFO Completed upload id=30 objects=6 duration=796.668µs42042026/11/12 00:06:24 INFO Received uploads request method=POST path=/42052026/11/12 00:06:24 INFO Received push request method=POST path=/42062026/11/12 00:06:24 INFO Received complete push request method=POST path=/42072026/11/12 00:06:24 INFO Received push request method=POST path=/42082026/11/12 00:06:24 INFO Completed push id=7 objects=7 duration=1.913598ms42092026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=042102026/11/12 00:06:24 INFO Received complete push request method=POST path=/42112026/11/12 00:06:24 INFO Received uploads request method=POST path=/42122026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=042132026/11/12 00:06:24 INFO Completed push id=14 objects=3 duration=1.00011ms42142026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042152026/11/12 00:06:24 INFO Vacuumed table table=pending_closures42162026/11/12 00:06:24 INFO Vacuumed table table=pending_closures42172026/11/12 00:06:24 INFO Vacuumed table table=pending_objects42182026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads42192026/11/12 00:06:24 INFO Received complete push request method=POST path=/42202026/11/12 00:06:24 INFO Vacuumed table table=pending_objects42212026/11/12 00:06:24 INFO Vacuumed table table=closures42222026/11/12 00:06:24 INFO Completed push id=31 objects=5 duration=887.389µs42232026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads42242026/11/12 00:06:24 INFO Vacuumed table table=objects42252026/11/12 00:06:24 INFO Vacuumed table table=closures42262026/11/12 00:06:24 INFO Vacuumed table table=objects42272026/11/12 00:06:24 INFO Received push request method=POST path=/42282026/11/12 00:06:24 INFO Received complete push request method=POST path=/42292026/11/12 00:06:24 INFO Received push request method=POST path=/42302026/11/12 00:06:24 INFO Completed push id=12 objects=6 duration=1.07119ms42312026/11/12 00:06:24 INFO Received uploads request method=POST path=/42322026/11/12 00:06:24 INFO Received uploads request method=POST path=/42332026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=042342026/11/12 00:06:24 INFO Received complete push request method=POST path=/42352026/11/12 00:06:24 INFO Received complete push request method=POST path=/42362026/11/12 00:06:24 INFO Completed push id=16 objects=26 duration=1.268632ms42372026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42382026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042392026/11/12 00:06:24 INFO Completed push id=31 objects=6 duration=2.899127ms42402026/11/12 00:06:24 INFO Completed upload id=23 objects=15 duration=2.106021ms42412026/11/12 00:06:24 INFO Vacuumed table table=pending_closures42422026/11/12 00:06:24 INFO Vacuumed table table=pending_objects42432026/11/12 00:06:24 INFO Received complete push request method=POST path=/42442026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads42452026/11/12 00:06:24 INFO Vacuumed table table=closures42462026/11/12 00:06:24 INFO Completed push id=32 objects=26 duration=1.378114ms42472026/11/12 00:06:24 INFO Vacuumed table table=objects42482026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=042492026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42502026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period42512026/11/12 00:06:24 INFO Completed upload id=20 objects=4 duration=859.248µs42522026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042532026/11/12 00:06:24 INFO Vacuumed table table=pending_closures42542026/11/12 00:06:24 INFO Vacuumed table table=pending_objects42552026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads42562026/11/12 00:06:24 INFO Vacuumed table table=closures42572026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=042582026/11/12 00:06:24 INFO Vacuumed table table=objects42592026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period42602026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=042612026/11/12 00:06:24 INFO Received uploads request method=POST path=/42622026/11/12 00:06:24 INFO Received push request method=POST path=/42632026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period42642026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42652026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042662026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042672026/11/12 00:06:24 INFO Completed upload id=33 objects=2 duration=829.668µs42682026/11/12 00:06:24 INFO Vacuumed table table=pending_closures42692026/11/12 00:06:24 INFO Vacuumed table table=pending_closures42702026/11/12 00:06:24 INFO Vacuumed table table=pending_objects42712026/11/12 00:06:24 INFO Vacuumed table table=pending_objects42722026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads42732026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads42742026/11/12 00:06:24 INFO Vacuumed table table=closures42752026/11/12 00:06:24 INFO Vacuumed table table=closures42762026/11/12 00:06:24 INFO Received complete push request method=POST path=/42772026/11/12 00:06:24 INFO Received push request method=POST path=/42782026/11/12 00:06:24 INFO Vacuumed table table=objects42792026/11/12 00:06:24 INFO Vacuumed table table=objects42802026/11/12 00:06:24 INFO Completed push id=13 objects=22 duration=1.404273ms42812026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42822026/11/12 00:06:24 INFO Completed upload id=14 objects=16 duration=2.271042ms42832026/11/12 00:06:24 INFO Received push request method=POST path=/42842026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42852026/11/12 00:06:24 INFO Completed upload id=34 objects=5 duration=758.888µs42862026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42872026/11/12 00:06:24 INFO Received complete push request method=POST path=/42882026/11/12 00:06:24 INFO Completed upload id=19 objects=6 duration=1.329233ms42892026/11/12 00:06:24 INFO Completed push id=16 objects=5 duration=1.154111ms42902026/11/12 00:06:24 INFO Received complete push request method=POST path=/42912026/11/12 00:06:24 INFO Received complete push request method=POST path=/42922026/11/12 00:06:24 INFO Received complete push request method=POST path=/42932026/11/12 00:06:24 INFO Completed push id=22 objects=3 duration=2.559664ms42942026/11/12 00:06:24 INFO Completed push id=21 objects=6 duration=1.317873ms42952026/11/12 00:06:24 INFO Completed push id=27 objects=6 duration=2.777106ms42962026/11/12 00:06:24 INFO Received uploads request method=POST path=/42972026/11/12 00:06:24 INFO Received complete upload request method=POST path=/42982026/11/12 00:06:24 INFO Completed upload id=35 objects=11 duration=779.368µs42992026/11/12 00:06:24 INFO Received uploads request method=POST path=/43002026/11/12 00:06:24 INFO Received uploads request method=POST path=/43012026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43022026/11/12 00:06:24 INFO Completed upload id=22 objects=7 duration=1.09355ms43032026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43042026/11/12 00:06:24 INFO Received uploads request method=POST path=/43052026/11/12 00:06:24 INFO Completed upload id=22 objects=7 duration=950.929µs43062026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=043072026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43082026/11/12 00:06:24 INFO Completed upload id=36 objects=3 duration=795.927µs43092026/11/12 00:06:24 INFO Received complete push request method=POST path=/43102026/11/12 00:06:24 INFO Completed push id=17 objects=11 duration=1.678676ms43112026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=043122026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43132026/11/12 00:06:24 INFO Received complete push request method=POST path=/43142026/11/12 00:06:24 INFO Vacuumed table table=pending_closures43152026/11/12 00:06:24 INFO Completed upload id=15 objects=2 duration=879.128µs43162026/11/12 00:06:24 INFO Completed push id=24 objects=4 duration=984.21µs43172026/11/12 00:06:24 INFO Received complete push request method=POST path=/43182026/11/12 00:06:24 INFO Vacuumed table table=pending_objects43192026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads43202026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43212026/11/12 00:06:24 INFO Completed push id=1 objects=9 duration=1.225352ms43222026/11/12 00:06:24 INFO Received complete push request method=POST path=/43232026/11/12 00:06:24 INFO Vacuumed table table=closures43242026/11/12 00:06:24 INFO Vacuumed table table=objects43252026/11/12 00:06:24 INFO Completed push id=13 objects=15 duration=1.645616ms43262026/11/12 00:06:24 INFO Completed upload id=16 objects=17 duration=2.609325ms43272026/11/12 00:06:24 INFO Received push request method=POST path=/43282026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=043292026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period43302026/11/12 00:06:24 INFO Received uploads request method=POST path=/43312026/11/12 00:06:24 INFO Received push request method=POST path=/43322026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043332026/11/12 00:06:24 INFO Vacuumed table table=pending_closures43342026/11/12 00:06:24 INFO Vacuumed table table=pending_objects43352026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads43362026/11/12 00:06:24 INFO Vacuumed table table=closures43372026/11/12 00:06:24 INFO Vacuumed table table=objects43382026/11/12 00:06:24 INFO Received complete push request method=POST path=/43392026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=043402026/11/12 00:06:24 INFO Completed push id=25 objects=9 duration=990.37µs43412026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=043422026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=043432026/11/12 00:06:24 INFO Received complete push request method=POST path=/43442026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043452026/11/12 00:06:24 INFO Completed push id=24 objects=3 duration=945.229µs43462026/11/12 00:06:24 INFO Received push request method=POST path=/43472026/11/12 00:06:24 INFO Vacuumed table table=pending_closures43482026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period43492026/11/12 00:06:24 INFO Received push request method=POST path=/43502026/11/12 00:06:24 INFO Vacuumed table table=pending_objects43512026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads43522026/11/12 00:06:24 INFO Vacuumed table table=closures43532026/11/12 00:06:24 INFO Received complete push request method=POST path=/43542026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."43552026/11/12 00:06:24 INFO Vacuumed table table=objects43562026/11/12 00:06:24 INFO Completed push id=24 objects=9 duration=1.834198ms43572026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43582026/11/12 00:06:24 INFO Completed upload id=26 objects=3 duration=749.027µs43592026/11/12 00:06:24 INFO Received push request method=POST path=/43602026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=043612026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period43622026/11/12 00:06:24 INFO Received uploads request method=POST path=/43632026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043642026/11/12 00:06:24 INFO Vacuumed table table=pending_closures43652026/11/12 00:06:24 INFO Received push request method=POST path=/43662026/11/12 00:06:24 INFO Vacuumed table table=pending_objects43672026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads43682026/11/12 00:06:24 INFO Vacuumed table table=closures43692026/11/12 00:06:24 INFO Vacuumed table table=objects43702026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43712026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43722026/11/12 00:06:24 INFO Completed upload id=27 objects=7 duration=891.488µs43732026/11/12 00:06:24 INFO Received uploads request method=POST path=/43742026/11/12 00:06:24 INFO Completed upload id=2 objects=7 duration=1.05547ms43752026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43762026/11/12 00:06:24 INFO Received complete push request method=POST path=/43772026/11/12 00:06:24 INFO Completed upload id=26 objects=6 duration=899.248µs43782026/11/12 00:06:24 INFO Completed push id=20 objects=3 duration=1.462274ms43792026/11/12 00:06:24 INFO Received push request method=POST path=/43802026/11/12 00:06:24 INFO Received push request method=POST path=/43812026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=043822026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period43832026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043842026/11/12 00:06:24 INFO Vacuumed table table=pending_closures43852026/11/12 00:06:24 INFO Vacuumed table table=pending_objects43862026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads43872026/11/12 00:06:24 INFO Vacuumed table table=closures43882026/11/12 00:06:24 INFO Received push request method=POST path=/43892026/11/12 00:06:24 INFO Vacuumed table table=objects43902026/11/12 00:06:24 INFO Received complete upload request method=POST path=/43912026/11/12 00:06:24 INFO Completed upload id=28 objects=3 duration=1.08181ms43922026/11/12 00:06:24 INFO Received uploads request method=POST path=/43932026/11/12 00:06:24 INFO Received uploads request method=POST path=/43942026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=043952026/11/12 00:06:24 INFO Vacuumed table table=pending_closures43962026/11/12 00:06:24 INFO Received uploads request method=POST path=/43972026/11/12 00:06:24 INFO Vacuumed table table=pending_objects43982026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads43992026/11/12 00:06:24 INFO Vacuumed table table=closures44002026/11/12 00:06:24 INFO Vacuumed table table=objects44012026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=044022026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44032026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044042026/11/12 00:06:24 INFO Completed upload id=29 objects=5 duration=927.229µs44052026/11/12 00:06:24 INFO Vacuumed table table=pending_closures44062026/11/12 00:06:24 INFO Vacuumed table table=pending_objects44072026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads44082026/11/12 00:06:24 INFO Vacuumed table table=closures44092026/11/12 00:06:24 INFO Vacuumed table table=objects44102026/11/12 00:06:24 INFO Received uploads request method=POST path=/44112026/11/12 00:06:24 INFO Received complete push request method=POST path=/44122026/11/12 00:06:24 INFO Completed push id=29 objects=23 duration=3.585874ms44132026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44142026/11/12 00:06:24 INFO Received uploads request method=POST path=/44152026/11/12 00:06:24 INFO Completed upload id=8 objects=6 duration=953.129µs44162026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=044172026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=044182026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044192026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044202026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=044212026/11/12 00:06:24 INFO Vacuumed table table=pending_closures44222026/11/12 00:06:24 INFO Vacuumed table table=pending_closures44232026/11/12 00:06:24 INFO Vacuumed table table=pending_objects44242026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads44252026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period44262026/11/12 00:06:24 INFO Vacuumed table table=closures44272026/11/12 00:06:24 INFO Vacuumed table table=pending_objects44282026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=044292026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads44302026/11/12 00:06:24 INFO Vacuumed table table=objects44312026/11/12 00:06:24 INFO Vacuumed table table=closures44322026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044332026/11/12 00:06:24 INFO Vacuumed table table=objects44342026/11/12 00:06:24 INFO Vacuumed table table=pending_closures44352026/11/12 00:06:24 INFO Vacuumed table table=pending_objects44362026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads44372026/11/12 00:06:24 INFO Received complete push request method=POST path=/44382026/11/12 00:06:24 INFO Vacuumed table table=closures44392026/11/12 00:06:24 INFO Vacuumed table table=objects44402026/11/12 00:06:24 INFO Completed push id=30 objects=6 duration=1.501915ms44412026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44422026/11/12 00:06:24 INFO Received push request method=POST path=/44432026/11/12 00:06:24 INFO Completed upload id=26 objects=8 duration=1.00887ms44442026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=044452026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period44462026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044472026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=044482026/11/12 00:06:24 INFO Vacuumed table table=pending_closures44492026/11/12 00:06:24 INFO Vacuumed table table=pending_objects44502026/11/12 00:06:24 INFO Vacuumed table table=pending_closures44512026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads44522026/11/12 00:06:24 INFO Vacuumed table table=closures44532026/11/12 00:06:24 INFO Vacuumed table table=pending_objects44542026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads44552026/11/12 00:06:24 INFO Vacuumed table table=objects44562026/11/12 00:06:24 INFO Vacuumed table table=closures44572026/11/12 00:06:24 INFO Vacuumed table table=objects44582026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44592026/11/12 00:06:24 INFO Completed upload id=15 objects=3 duration=1.793477ms44602026/11/12 00:06:24 INFO Received push request method=POST path=/44612026/11/12 00:06:24 INFO Received complete push request method=POST path=/44622026/11/12 00:06:24 INFO Completed push id=28 objects=14 duration=1.089391ms44632026/11/12 00:06:24 INFO Received complete push request method=POST path=/44642026/11/12 00:06:24 INFO Completed push id=33 objects=3 duration=1.005429ms44652026/11/12 00:06:24 INFO Received push request method=POST path=/44662026/11/12 00:06:24 INFO Received push request method=POST path=/44672026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44682026/11/12 00:06:24 INFO Completed upload id=33 objects=5 duration=2.077579ms44692026/11/12 00:06:24 INFO Received push request method=POST path=/44702026/11/12 00:06:24 INFO Received push request method=POST path=/44712026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44722026/11/12 00:06:24 INFO Completed upload id=12 objects=9 duration=900.829µs44732026/11/12 00:06:24 INFO Received push request method=POST path=/44742026/11/12 00:06:24 INFO Received complete push request method=POST path=/44752026/11/12 00:06:24 INFO Completed push id=34 objects=6 duration=873.609µs44762026/11/12 00:06:24 INFO Received uploads request method=POST path=/44772026/11/12 00:06:24 INFO Received push request method=POST path=/44782026/11/12 00:06:24 INFO Received complete push request method=POST path=/44792026/11/12 00:06:24 INFO Completed push id=21 objects=3 duration=1.262692ms44802026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44812026/11/12 00:06:24 INFO Completed upload id=22 objects=6 duration=912.389µs44822026/11/12 00:06:24 INFO Received uploads request method=POST path=/44832026/11/12 00:06:24 INFO Received uploads request method=POST path=/44842026/11/12 00:06:24 INFO Received uploads request method=POST path=/44852026/11/12 00:06:24 INFO Received complete upload request method=POST path=/44862026/11/12 00:06:24 INFO Received complete push request method=POST path=/44872026/11/12 00:06:24 INFO Completed push id=35 objects=8 duration=3.09843ms44882026/11/12 00:06:24 INFO Completed upload id=26 objects=2 duration=3.378932ms44892026/11/12 00:06:24 INFO Received uploads request method=POST path=/44902026/11/12 00:06:24 INFO Received uploads request method=POST path=/44912026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=044922026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period44932026/11/12 00:06:24 INFO Received push request method=POST path=/44942026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044952026/11/12 00:06:24 INFO Vacuumed table table=pending_closures44962026/11/12 00:06:24 INFO Vacuumed table table=pending_objects44972026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads44982026/11/12 00:06:24 INFO Vacuumed table table=closures44992026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=045002026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45012026/11/12 00:06:24 INFO Vacuumed table table=objects45022026/11/12 00:06:24 INFO Completed upload id=21 objects=25 duration=1.412574ms45032026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45042026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045052026/11/12 00:06:24 INFO Completed upload id=36 objects=9 duration=696.106µs45062026/11/12 00:06:24 INFO Vacuumed table table=pending_closures45072026/11/12 00:06:24 INFO Vacuumed table table=pending_objects45082026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads45092026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45102026/11/12 00:06:24 INFO Vacuumed table table=closures45112026/11/12 00:06:24 INFO Completed upload id=31 objects=3 duration=2.843427ms45122026/11/12 00:06:24 INFO Vacuumed table table=objects45132026/11/12 00:06:24 INFO Received uploads request method=POST path=/45142026/11/12 00:06:24 INFO Received uploads request method=POST path=/45152026/11/12 00:06:24 INFO Received uploads request method=POST path=/45162026/11/12 00:06:24 INFO Received complete push request method=POST path=/45172026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=045182026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=045192026/11/12 00:06:24 INFO Completed push id=17 objects=22 duration=1.917498ms45202026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period45212026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45222026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045232026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045242026/11/12 00:06:24 INFO Completed upload id=22 objects=2 duration=925.349µs45252026/11/12 00:06:24 INFO Vacuumed table table=pending_closures45262026/11/12 00:06:24 INFO Vacuumed table table=pending_objects45272026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads45282026/11/12 00:06:24 INFO Vacuumed table table=closures45292026/11/12 00:06:24 INFO Vacuumed table table=pending_closures45302026/11/12 00:06:24 INFO Vacuumed table table=objects45312026/11/12 00:06:24 INFO Vacuumed table table=pending_objects45322026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads45332026/11/12 00:06:24 INFO Vacuumed table table=closures45342026/11/12 00:06:24 INFO Received push request method=POST path=/45352026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=045362026/11/12 00:06:24 INFO Vacuumed table table=objects45372026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45382026/11/12 00:06:24 INFO Completed upload id=21 objects=9 duration=3.978978ms45392026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045402026/11/12 00:06:24 INFO Received uploads request method=POST path=/45412026/11/12 00:06:24 INFO Vacuumed table table=pending_closures45422026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45432026/11/12 00:06:24 INFO Completed upload id=18 objects=22 duration=1.221031ms45442026/11/12 00:06:24 INFO Vacuumed table table=pending_objects45452026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=045462026/11/12 00:06:24 INFO Received push request method=POST path=/45472026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads45482026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=045492026/11/12 00:06:24 INFO Vacuumed table table=closures45502026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period45512026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045522026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period45532026/11/12 00:06:24 INFO Vacuumed table table=objects45542026/11/12 00:06:24 INFO Vacuumed table table=pending_closures45552026/11/12 00:06:24 INFO Vacuumed table table=pending_objects45562026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads45572026/11/12 00:06:24 INFO Received push request method=POST path=/45582026/11/12 00:06:24 INFO Vacuumed table table=closures45592026/11/12 00:06:24 INFO Vacuumed table table=objects45602026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045612026/11/12 00:06:24 INFO Vacuumed table table=pending_closures45622026/11/12 00:06:24 INFO Received complete push request method=POST path=/45632026/11/12 00:06:24 INFO Vacuumed table table=pending_objects45642026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads45652026/11/12 00:06:24 INFO Vacuumed table table=closures45662026/11/12 00:06:24 INFO Completed push id=25 objects=10 duration=1.562235ms45672026/11/12 00:06:24 INFO Vacuumed table table=objects45682026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45692026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45702026/11/12 00:06:24 INFO Received uploads request method=POST path=/45712026/11/12 00:06:24 INFO Received push request method=POST path=/45722026/11/12 00:06:24 INFO Completed upload id=21 objects=8 duration=8.075357ms45732026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45742026/11/12 00:06:24 INFO Completed upload id=19 objects=2 duration=11.777092ms45752026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45762026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45772026/11/12 00:06:24 INFO Completed upload id=12 objects=6 duration=14.420478ms45782026/11/12 00:06:24 INFO Received uploads request method=POST path=/45792026/11/12 00:06:24 INFO Completed upload id=18 objects=2 duration=15.407188ms45802026/11/12 00:06:24 INFO Completed upload id=29 objects=5 duration=11.46825ms45812026/11/12 00:06:24 INFO Received complete push request method=POST path=/45822026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=045832026/11/12 00:06:24 INFO Received push request method=POST path=/45842026/11/12 00:06:24 INFO Received uploads request method=POST path=/45852026/11/12 00:06:24 INFO Received push request method=POST path=/45862026/11/12 00:06:24 INFO Completed push id=17 objects=3 duration=13.050844ms45872026/11/12 00:06:24 INFO Received push request method=POST path=/45882026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45892026/11/12 00:06:24 INFO Received complete upload request method=POST path=/45902026/11/12 00:06:24 INFO Received complete push request method=POST path=/45912026/11/12 00:06:24 INFO Completed upload id=30 objects=2 duration=5.744374ms45922026/11/12 00:06:24 INFO Completed upload id=22 objects=7 duration=6.351821ms45932026/11/12 00:06:24 INFO Completed push id=33 objects=7 duration=3.709696ms45942026/11/12 00:06:24 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."45952026/11/12 00:06:24 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."45962026/11/12 00:06:24 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."45972026/11/12 00:06:24 INFO Received complete push request method=POST path=/45982026/11/12 00:06:24 INFO Completed push id=23 objects=5 duration=1.231171ms45992026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46002026/11/12 00:06:24 INFO Completed upload id=26 objects=3 duration=921.708µs46012026/11/12 00:06:24 INFO Received uploads request method=POST path=/46022026/11/12 00:06:24 INFO Received complete push request method=POST path=/46032026/11/12 00:06:24 INFO Received complete push request method=POST path=/46042026/11/12 00:06:24 INFO Received complete push request method=POST path=/46052026/11/12 00:06:24 INFO Completed push id=19 objects=14 duration=1.446374ms46062026/11/12 00:06:24 INFO Received complete push request method=POST path=/46072026/11/12 00:06:24 INFO Completed push id=24 objects=5 duration=898.369µs46082026/11/12 00:06:24 INFO Completed push id=31 objects=20 duration=1.01143ms46092026/11/12 00:06:24 INFO Completed push id=30 objects=4 duration=3.127069ms46102026/11/12 00:06:24 INFO Received push request method=POST path=/46112026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46122026/11/12 00:06:24 INFO Completed upload id=24 objects=3 duration=763.787µs46132026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46142026/11/12 00:06:24 INFO Completed upload id=13 objects=7 duration=999.59µs46152026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=046162026/11/12 00:06:24 INFO Received uploads request method=POST path=/46172026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period46182026/11/12 00:06:24 INFO Received uploads request method=POST path=/46192026/11/12 00:06:24 INFO Received uploads request method=POST path=/46202026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046212026/11/12 00:06:24 INFO Vacuumed table table=pending_closures46222026/11/12 00:06:24 INFO Received complete push request method=POST path=/46232026/11/12 00:06:24 INFO Vacuumed table table=pending_objects46242026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads46252026/11/12 00:06:24 INFO Completed push id=25 objects=4 duration=886.108µs46262026/11/12 00:06:24 INFO Vacuumed table table=closures46272026/11/12 00:06:24 INFO Vacuumed table table=objects46282026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46292026/11/12 00:06:24 INFO Completed upload id=20 objects=4 duration=943.769µs46302026/11/12 00:06:24 INFO Received push request method=POST path=/46312026/11/12 00:06:24 INFO Received uploads request method=POST path=/46322026/11/12 00:06:24 INFO Received push request method=POST path=/46332026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46342026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46352026/11/12 00:06:24 INFO Completed upload id=25 objects=8 duration=3.238251ms46362026/11/12 00:06:24 INFO Completed upload id=32 objects=5 duration=4.13958ms46372026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46382026/11/12 00:06:24 INFO Completed upload id=23 objects=7 duration=3.039869ms46392026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46402026/11/12 00:06:24 INFO Completed upload id=20 objects=6 duration=1.100831ms46412026/11/12 00:06:24 INFO Received push request method=POST path=/46422026/11/12 00:06:24 INFO Received complete push request method=POST path=/46432026/11/12 00:06:24 INFO Received push request method=POST path=/46442026/11/12 00:06:24 INFO Completed push id=28 objects=11 duration=2.14264ms46452026/11/12 00:06:24 INFO Received complete push request method=POST path=/46462026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46472026/11/12 00:06:24 INFO Completed push id=26 objects=4 duration=895.149µs46482026/11/12 00:06:24 INFO Completed upload id=33 objects=3 duration=729.827µs46492026/11/12 00:06:24 INFO Received complete push request method=POST path=/46502026/11/12 00:06:24 INFO Completed push id=26 objects=23 duration=1.124591ms46512026/11/12 00:06:24 INFO Received complete push request method=POST path=/46522026/11/12 00:06:24 INFO Received complete push request method=POST path=/46532026/11/12 00:06:24 INFO Completed push id=32 objects=7 duration=1.03409ms46542026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=046552026/11/12 00:06:24 INFO Received uploads request method=POST path=/46562026/11/12 00:06:24 INFO Completed push id=9 objects=9 duration=2.744366ms46572026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period46582026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046592026/11/12 00:06:24 INFO Received complete push request method=POST path=/46602026/11/12 00:06:24 INFO Completed push id=30 objects=2 duration=905.088µs46612026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46622026/11/12 00:06:24 INFO Vacuumed table table=pending_closures46632026/11/12 00:06:24 INFO Completed upload id=24 objects=18 duration=1.02597ms46642026/11/12 00:06:24 INFO Vacuumed table table=pending_objects46652026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads46662026/11/12 00:06:24 INFO Vacuumed table table=closures46672026/11/12 00:06:24 INFO Vacuumed table table=objects46682026/11/12 00:06:24 INFO Received complete push request method=POST path=/46692026/11/12 00:06:24 INFO Completed push id=26 objects=4 duration=1.12069ms46702026/11/12 00:06:24 INFO Received complete push request method=POST path=/46712026/11/12 00:06:24 INFO Received complete push request method=POST path=/46722026/11/12 00:06:24 INFO Completed push id=27 objects=15 duration=1.221491ms46732026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46742026/11/12 00:06:24 INFO Received complete push request method=POST path=/46752026/11/12 00:06:24 INFO Completed upload id=21 objects=8 duration=994.449µs46762026/11/12 00:06:24 INFO Received complete push request method=POST path=/46772026/11/12 00:06:24 INFO Completed push id=25 objects=2 duration=2.349863ms46782026/11/12 00:06:24 INFO Completed push id=29 objects=24 duration=1.391533ms46792026/11/12 00:06:24 INFO Completed push id=16 objects=7 duration=6.915206ms46802026/11/12 00:06:24 INFO Received complete push request method=POST path=/46812026/11/12 00:06:24 INFO Completed push id=32 objects=7 duration=1.280213ms46822026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46832026/11/12 00:06:24 INFO Completed upload id=28 objects=13 duration=924.149µs46842026/11/12 00:06:24 INFO Received complete push request method=POST path=/46852026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=046862026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period46872026/11/12 00:06:24 INFO Completed push id=27 objects=4 duration=852.969µs46882026/11/12 00:06:24 INFO Received complete push request method=POST path=/46892026/11/12 00:06:24 INFO Received complete upload request method=POST path=/46902026/11/12 00:06:24 INFO Completed upload id=21 objects=2 duration=968.13µs46912026/11/12 00:06:24 INFO Completed push id=22 objects=6 duration=2.319262ms46922026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=046932026/11/12 00:06:24 INFO Vacuumed table table=pending_closures46942026/11/12 00:06:24 INFO Vacuumed table table=pending_objects46952026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads46962026/11/12 00:06:24 INFO Vacuumed table table=closures46972026/11/12 00:06:24 INFO Vacuumed table table=objects46982026/11/12 00:06:24 INFO Received complete push request method=POST path=/46992026/11/12 00:06:24 INFO Completed push id=29 objects=3 duration=1.03527ms47002026/11/12 00:06:24 INFO Received complete push request method=POST path=/47012026/11/12 00:06:24 INFO Completed push id=23 objects=6 duration=1.185412ms47022026/11/12 00:06:24 INFO Received complete push request method=POST path=/47032026/11/12 00:06:24 INFO Received uploads request method=POST path=/47042026/11/12 00:06:24 INFO Completed push id=24 objects=8 duration=1.151471ms47052026/11/12 00:06:24 INFO Received complete push request method=POST path=/47062026/11/12 00:06:24 INFO Completed push id=28 objects=26 duration=1.415573ms47072026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47082026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47092026/11/12 00:06:24 INFO Completed upload id=33 objects=6 duration=881.649µs47102026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47112026/11/12 00:06:24 INFO Completed upload id=31 objects=2 duration=972.17µs47122026/11/12 00:06:24 INFO Completed upload id=25 objects=9 duration=934.829µs47132026/11/12 00:06:24 INFO Received complete push request method=POST path=/47142026/11/12 00:06:24 INFO Received complete push request method=POST path=/47152026/11/12 00:06:24 INFO Completed push id=24 objects=6 duration=1.00593ms47162026/11/12 00:06:24 INFO Completed push id=28 objects=19 duration=1.154111ms47172026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=047182026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period47192026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047202026/11/12 00:06:24 INFO Vacuumed table table=pending_closures47212026/11/12 00:06:24 INFO Vacuumed table table=pending_objects47222026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads47232026/11/12 00:06:24 INFO Vacuumed table table=closures47242026/11/12 00:06:24 INFO Vacuumed table table=objects47252026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47262026/11/12 00:06:24 INFO Completed upload id=27 objects=3 duration=801.468µs47272026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47282026/11/12 00:06:24 INFO Received complete push request method=POST path=/47292026/11/12 00:06:24 INFO Received complete push request method=POST path=/47302026/11/12 00:06:24 INFO Completed upload id=34 objects=3 duration=712.307µs47312026/11/12 00:06:24 INFO Completed push id=30 objects=3 duration=1.341073ms47322026/11/12 00:06:24 INFO Completed push id=33 objects=6 duration=981.129µs47332026/11/12 00:06:24 INFO Received complete push request method=POST path=/47342026/11/12 00:06:24 INFO Received complete push request method=POST path=/47352026/11/12 00:06:24 INFO Completed push id=29 objects=10 duration=1.308613ms47362026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47372026/11/12 00:06:24 INFO Completed push id=25 objects=7 duration=1.503215ms47382026/11/12 00:06:24 INFO Completed upload id=28 objects=6 duration=837.988µs47392026/11/12 00:06:24 INFO Received complete push request method=POST path=/47402026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=047412026/11/12 00:06:24 INFO Completed push id=32 objects=19 duration=1.188971ms47422026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=047432026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period47442026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period47452026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047462026/11/12 00:06:24 INFO Vacuumed table table=pending_closures47472026/11/12 00:06:24 INFO Vacuumed table table=pending_objects47482026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads47492026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047502026/11/12 00:06:24 INFO Vacuumed table table=closures47512026/11/12 00:06:24 INFO Received complete push request method=POST path=/47522026/11/12 00:06:24 INFO Vacuumed table table=objects47532026/11/12 00:06:24 INFO Completed push id=30 objects=4 duration=1.04391ms47542026/11/12 00:06:24 INFO Received uploads request method=POST path=/47552026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47562026/11/12 00:06:24 INFO Vacuumed table table=pending_closures47572026/11/12 00:06:24 INFO Vacuumed table table=pending_objects47582026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads47592026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=047602026/11/12 00:06:24 INFO Vacuumed table table=closures47612026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period47622026/11/12 00:06:24 INFO Completed upload id=30 objects=6 duration=7.568432ms47632026/11/12 00:06:24 INFO Vacuumed table table=objects47642026/11/12 00:06:24 INFO Received complete push request method=POST path=/47652026/11/12 00:06:24 INFO Completed push id=31 objects=7 duration=1.108871ms47662026/11/12 00:06:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047672026/11/12 00:06:24 INFO Vacuumed table table=pending_closures47682026/11/12 00:06:24 INFO Vacuumed table table=pending_objects47692026/11/12 00:06:24 INFO Vacuumed table table=multipart_uploads47702026/11/12 00:06:24 INFO Vacuumed table table=closures47712026/11/12 00:06:24 INFO Vacuumed table table=objects47722026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47732026/11/12 00:06:24 INFO Completed upload id=32 objects=5 duration=769.527µs47742026/11/12 00:06:24 INFO Received push request method=POST path=/47752026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47762026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47772026/11/12 00:06:24 INFO Completed upload id=31 objects=5 duration=11.411469ms47782026/11/12 00:06:24 INFO Completed upload id=26 objects=24 duration=3.362572ms47792026/11/12 00:06:24 INFO Received complete push request method=POST path=/47802026/11/12 00:06:24 INFO Completed push id=24 objects=24 duration=2.845007ms47812026/11/12 00:06:24 INFO Received complete upload request method=POST path=/47822026/11/12 00:06:24 INFO Completed upload id=34 objects=3 duration=690.747µs47832026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=047842026/11/12 00:06:24 WARN Force mode enabled - objects will be deleted immediately without grace period47852026/11/12 00:06:24 INFO Aborted multipart uploads count=0 kept=047862026/11/12 00:06:25 WARN Force mode enabled - objects will be deleted immediately without grace period47872026/11/12 00:06:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047882026/11/12 00:06:25 INFO Vacuumed table table=pending_closures47892026/11/12 00:06:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047902026/11/12 00:06:25 INFO Vacuumed table table=pending_objects47912026/11/12 00:06:25 INFO Vacuumed table table=pending_closures47922026/11/12 00:06:25 INFO Vacuumed table table=multipart_uploads47932026/11/12 00:06:25 INFO Vacuumed table table=closures47942026/11/12 00:06:25 INFO Vacuumed table table=objects47952026/11/12 00:06:25 INFO Vacuumed table table=pending_objects47962026/11/12 00:06:25 INFO Vacuumed table table=multipart_uploads47972026/11/12 00:06:25 INFO Received complete upload request method=POST path=/47982026/11/12 00:06:25 INFO Vacuumed table table=closures47992026/11/12 00:06:25 INFO Completed upload id=27 objects=26 duration=1.05369ms48002026/11/12 00:06:25 INFO Vacuumed table table=objects48012026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48022026/11/12 00:06:25 INFO Completed upload id=34 objects=24 duration=993.189µs48032026/11/12 00:06:25 INFO Received uploads request method=POST path=/48042026/11/12 00:06:25 INFO Received complete push request method=POST path=/48052026/11/12 00:06:25 INFO Completed push id=28 objects=6 duration=926.289µs48062026/11/12 00:06:25 INFO Aborted multipart uploads count=0 kept=048072026/11/12 00:06:25 WARN Force mode enabled - objects will be deleted immediately without grace period48082026/11/12 00:06:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048092026/11/12 00:06:25 INFO Vacuumed table table=pending_closures48102026/11/12 00:06:25 INFO Vacuumed table table=pending_objects48112026/11/12 00:06:25 INFO Vacuumed table table=multipart_uploads48122026/11/12 00:06:25 INFO Vacuumed table table=closures48132026/11/12 00:06:25 INFO Vacuumed table table=objects48142026/11/12 00:06:25 INFO Received uploads request method=POST path=/48152026/11/12 00:06:25 INFO Received complete push request method=POST path=/48162026/11/12 00:06:25 INFO Completed push id=29 objects=6 duration=947.569µs48172026/11/12 00:06:25 INFO Received push request method=POST path=/48182026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48192026/11/12 00:06:25 INFO Completed upload id=30 objects=8 duration=892.048µs48202026/11/12 00:06:25 INFO Received uploads request method=POST path=/48212026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48222026/11/12 00:06:25 INFO Completed upload id=31 objects=6 duration=909.508µs48232026/11/12 00:06:25 INFO Aborted multipart uploads count=0 kept=048242026/11/12 00:06:25 WARN Force mode enabled - objects will be deleted immediately without grace period48252026/11/12 00:06:25 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048262026/11/12 00:06:25 INFO Vacuumed table table=pending_closures48272026/11/12 00:06:25 INFO Vacuumed table table=pending_objects48282026/11/12 00:06:25 INFO Vacuumed table table=multipart_uploads48292026/11/12 00:06:25 INFO Vacuumed table table=closures48302026/11/12 00:06:25 INFO Vacuumed table table=objects48312026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48322026/11/12 00:06:25 INFO Completed upload id=32 objects=4 duration=938.769µs48332026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48342026/11/12 00:06:25 INFO Completed upload id=33 objects=10 duration=806.468µs48352026/11/12 00:06:25 INFO Received push request method=POST path=/48362026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48372026/11/12 00:06:25 INFO Completed upload id=34 objects=23 duration=1.017069ms48382026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48392026/11/12 00:06:25 INFO Completed upload id=23 objects=7 duration=2.788887ms48402026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48412026/11/12 00:06:25 INFO Completed upload id=26 objects=5 duration=818.968µs48422026/11/12 00:06:25 INFO Received complete push request method=POST path=/48432026/11/12 00:06:25 INFO Completed push id=35 objects=6 duration=996.45µs48442026/11/12 00:06:25 INFO Received complete push request method=POST path=/48452026/11/12 00:06:25 INFO Completed push id=27 objects=3 duration=1.482894ms48462026/11/12 00:06:25 INFO Aborted multipart uploads count=0 kept=048472026/11/12 00:06:25 WARN Force mode enabled - objects will be deleted immediately without grace period48482026/11/12 00:06:25 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=048492026/11/12 00:06:25 INFO Vacuumed table table=pending_closures48502026/11/12 00:06:25 INFO Received complete upload request method=POST path=/48512026/11/12 00:06:25 INFO Vacuumed table table=pending_objects48522026/11/12 00:06:25 INFO Vacuumed table table=multipart_uploads48532026/11/12 00:06:25 INFO Completed upload id=28 objects=5 duration=675.166µs48542026/11/12 00:06:25 INFO Vacuumed table table=closures48552026/11/12 00:06:25 INFO Vacuumed table table=objects48562026/11/12 00:06:25 INFO Received complete push request method=POST path=/48572026/11/12 00:06:25 INFO Completed push id=29 objects=17 duration=982.27µs48582026/11/12 00:06:25 INFO Aborted multipart uploads count=0 kept=048592026/11/12 00:06:25 WARN Force mode enabled - objects will be deleted immediately without grace period48602026/11/12 00:06:25 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048612026/11/12 00:06:25 INFO Vacuumed table table=pending_closures48622026/11/12 00:06:25 INFO Vacuumed table table=pending_objects48632026/11/12 00:06:25 INFO Vacuumed table table=multipart_uploads48642026/11/12 00:06:25 INFO Vacuumed table table=closures48652026/11/12 00:06:25 INFO Vacuumed table table=objects48662026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048672026/11/12 00:06:30 INFO Vacuumed table table=pending_closures48682026/11/12 00:06:30 INFO Vacuumed table table=pending_objects48692026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads48702026/11/12 00:06:30 INFO Vacuumed table table=closures48712026/11/12 00:06:30 INFO Vacuumed table table=objects48722026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048732026/11/12 00:06:30 INFO Vacuumed table table=pending_closures48742026/11/12 00:06:30 INFO Vacuumed table table=pending_objects48752026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads48762026/11/12 00:06:30 INFO Vacuumed table table=closures48772026/11/12 00:06:30 INFO Vacuumed table table=objects48782026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048792026/11/12 00:06:30 INFO Vacuumed table table=pending_closures48802026/11/12 00:06:30 INFO Vacuumed table table=pending_objects48812026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads48822026/11/12 00:06:30 INFO Vacuumed table table=closures48832026/11/12 00:06:30 INFO Vacuumed table table=objects48842026/11/12 00:06:30 INFO Received complete upload request method=POST path=/48852026/11/12 00:06:30 INFO Completed upload id=20 objects=3 duration=2.05724ms48862026/11/12 00:06:30 INFO Received uploads request method=POST path=/48872026/11/12 00:06:30 INFO Received push request method=POST path=/48882026/11/12 00:06:30 INFO Received complete push request method=POST path=/48892026/11/12 00:06:30 INFO Completed push id=16 objects=14 duration=3.001769ms48902026/11/12 00:06:30 INFO Received complete upload request method=POST path=/48912026/11/12 00:06:30 INFO Completed upload id=24 objects=7 duration=888.288µs48922026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=048932026/11/12 00:06:30 WARN Force mode enabled - objects will be deleted immediately without grace period48942026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048952026/11/12 00:06:30 INFO Vacuumed table table=pending_closures48962026/11/12 00:06:30 INFO Vacuumed table table=pending_objects48972026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads48982026/11/12 00:06:30 INFO Vacuumed table table=closures48992026/11/12 00:06:30 INFO Vacuumed table table=objects49002026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=049012026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049022026/11/12 00:06:30 INFO Vacuumed table table=pending_closures49032026/11/12 00:06:30 INFO Vacuumed table table=pending_objects49042026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads49052026/11/12 00:06:30 INFO Vacuumed table table=closures49062026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=049072026/11/12 00:06:30 INFO Vacuumed table table=objects49082026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049092026/11/12 00:06:30 INFO Vacuumed table table=pending_closures49102026/11/12 00:06:30 INFO Vacuumed table table=pending_objects49112026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads49122026/11/12 00:06:30 INFO Vacuumed table table=closures49132026/11/12 00:06:30 INFO Vacuumed table table=objects49142026/11/12 00:06:30 INFO Received uploads request method=POST path=/49152026/11/12 00:06:30 INFO Received push request method=POST path=/49162026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=049172026/11/12 00:06:30 INFO Received uploads request method=POST path=/49182026/11/12 00:06:30 WARN Force mode enabled - objects will be deleted immediately without grace period49192026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049202026/11/12 00:06:30 INFO Vacuumed table table=pending_closures49212026/11/12 00:06:30 INFO Vacuumed table table=pending_objects49222026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads49232026/11/12 00:06:30 INFO Vacuumed table table=closures49242026/11/12 00:06:30 INFO Vacuumed table table=objects49252026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49262026/11/12 00:06:30 INFO Completed upload id=25 objects=6 duration=1.527395ms49272026/11/12 00:06:30 INFO Received complete push request method=POST path=/49282026/11/12 00:06:30 INFO Completed push id=23 objects=28 duration=3.018349ms49292026/11/12 00:06:30 INFO Received push request method=POST path=/49302026/11/12 00:06:30 INFO Received uploads request method=POST path=/49312026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49322026/11/12 00:06:30 INFO Completed upload id=27 objects=8 duration=893.868µs49332026/11/12 00:06:30 INFO Received complete push request method=POST path=/49342026/11/12 00:06:30 INFO Received push request method=POST path=/49352026/11/12 00:06:30 INFO Completed push id=11 objects=9 duration=1.459014ms49362026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=049372026/11/12 00:06:30 INFO Received complete push request method=POST path=/49382026/11/12 00:06:30 INFO Completed push id=15 objects=24 duration=1.356453ms49392026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049402026/11/12 00:06:30 INFO Vacuumed table table=pending_closures49412026/11/12 00:06:30 INFO Vacuumed table table=pending_objects49422026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads49432026/11/12 00:06:30 INFO Vacuumed table table=closures49442026/11/12 00:06:30 INFO Vacuumed table table=objects49452026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49462026/11/12 00:06:30 INFO Completed upload id=19 objects=2 duration=1.00589ms49472026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=049482026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049492026/11/12 00:06:30 INFO Vacuumed table table=pending_closures49502026/11/12 00:06:30 INFO Vacuumed table table=pending_objects49512026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads49522026/11/12 00:06:30 INFO Vacuumed table table=closures49532026/11/12 00:06:30 INFO Vacuumed table table=objects49542026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49552026/11/12 00:06:30 INFO Completed upload id=24 objects=7 duration=958.369µs49562026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49572026/11/12 00:06:30 INFO Completed upload id=25 objects=16 duration=2.036319ms49582026/11/12 00:06:30 INFO Received complete push request method=POST path=/49592026/11/12 00:06:30 INFO Completed push id=26 objects=3 duration=1.02495ms49602026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49612026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=049622026/11/12 00:06:30 INFO Completed upload id=28 objects=8 duration=790.428µs49632026/11/12 00:06:30 WARN Force mode enabled - objects will be deleted immediately without grace period49642026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049652026/11/12 00:06:30 INFO Vacuumed table table=pending_closures49662026/11/12 00:06:30 INFO Vacuumed table table=pending_objects49672026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads49682026/11/12 00:06:30 INFO Vacuumed table table=closures49692026/11/12 00:06:30 INFO Vacuumed table table=objects49702026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49712026/11/12 00:06:30 INFO Completed upload id=29 objects=10 duration=790.988µs49722026/11/12 00:06:30 INFO Received push request method=POST path=/49732026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49742026/11/12 00:06:30 INFO Completed upload id=30 objects=20 duration=892.489µs49752026/11/12 00:06:30 INFO Received push request method=POST path=/49762026/11/12 00:06:30 INFO Received complete push request method=POST path=/49772026/11/12 00:06:30 INFO Completed push id=31 objects=11 duration=941.949µs49782026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49792026/11/12 00:06:30 INFO Completed upload id=32 objects=2 duration=746.947µs49802026/11/12 00:06:30 INFO Received complete push request method=POST path=/49812026/11/12 00:06:30 INFO Completed push id=33 objects=3 duration=928.809µs49822026/11/12 00:06:30 INFO Received uploads request method=POST path=/49832026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49842026/11/12 00:06:30 INFO Completed upload id=34 objects=3 duration=787.828µs49852026/11/12 00:06:30 INFO Received complete upload request method=POST path=/49862026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=049872026/11/12 00:06:30 WARN Force mode enabled - objects will be deleted immediately without grace period49882026/11/12 00:06:30 INFO Completed upload id=22 objects=5 duration=1.608315ms49892026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049902026/11/12 00:06:30 INFO Vacuumed table table=pending_closures49912026/11/12 00:06:30 INFO Vacuumed table table=pending_objects49922026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads49932026/11/12 00:06:30 INFO Vacuumed table table=closures49942026/11/12 00:06:30 INFO Vacuumed table table=objects49952026/11/12 00:06:30 INFO Received complete push request method=POST path=/49962026/11/12 00:06:30 INFO Completed push id=28 objects=3 duration=1.726217ms49972026/11/12 00:06:30 INFO Received complete push request method=POST path=/49982026/11/12 00:06:30 INFO Completed push id=29 objects=15 duration=1.171471ms49992026/11/12 00:06:30 INFO Received complete push request method=POST path=/50002026/11/12 00:06:30 INFO Completed push id=30 objects=6 duration=1.070231ms50012026/11/12 00:06:30 INFO Received complete upload request method=POST path=/50022026/11/12 00:06:30 INFO Completed upload id=31 objects=3 duration=914.209µs50032026/11/12 00:06:30 INFO Aborted multipart uploads count=0 kept=050042026/11/12 00:06:30 WARN Force mode enabled - objects will be deleted immediately without grace period50052026/11/12 00:06:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050062026/11/12 00:06:30 INFO Vacuumed table table=pending_closures50072026/11/12 00:06:30 INFO Vacuumed table table=pending_objects50082026/11/12 00:06:30 INFO Vacuumed table table=multipart_uploads50092026/11/12 00:06:30 INFO Vacuumed table table=closures50102026/11/12 00:06:30 INFO Vacuumed table table=objects5011--- PASS: FuzzPushGC (0.01s)5012 --- PASS: FuzzPushGC/seed#0 (3.28s)5013 --- PASS: FuzzPushGC/seed#3 (3.52s)5014 --- PASS: FuzzPushGC/seed#11 (3.67s)5015 --- PASS: FuzzPushGC/seed#4 (3.88s)5016 --- PASS: FuzzPushGC/seed#6 (4.11s)5017 --- PASS: FuzzPushGC/seed#2 (4.13s)5018 --- PASS: FuzzPushGC/seed#10 (4.26s)5019 --- PASS: FuzzPushGC/seed#14 (4.27s)5020 --- PASS: FuzzPushGC/seed#7 (4.28s)5021 --- PASS: FuzzPushGC/seed#13 (4.35s)5022 --- PASS: FuzzPushGC/seed#9 (4.38s)5023 --- PASS: FuzzPushGC/seed#12 (4.53s)5024 --- PASS: FuzzPushGC/seed#1 (4.57s)5025 --- PASS: FuzzPushGC/seed#8 (9.60s)5026 --- PASS: FuzzPushGC/seed#5 (9.96s)5027 --- PASS: FuzzPushGC/seed#15 (10.05s)5028PASS50292026-11-12 00:06:31.266 UTC [129] LOG: received smart shutdown request50302026-11-12 00:06:31.274 UTC [129] LOG: background worker "logical replication launcher" (PID 151) exited with exit code 150312026-11-12 00:06:36.038 UTC [146] LOG: checkpoint complete: wrote 3582 buffers (21.9%), wrote 6 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=10.643 s, sync=0.483 s, total=15.313 s; sync files=42374, longest=0.008 s, average=0.001 s; distance=536832 kB, estimate=536832 kB; lsn=0/26E22DB0, redo lsn=0/223A5EF050322026-11-12 00:06:36.044 UTC [146] LOG: shutting down50332026-11-12 00:06:36.046 UTC [146] LOG: checkpoint starting: shutdown immediate50342026-11-12 00:06:36.654 UTC [146] LOG: checkpoint complete: wrote 8577 buffers (52.3%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.187 s, sync=0.412 s, total=0.610 s; sync files=2549, longest=0.004 s, average=0.001 s; distance=76275 kB, estimate=490776 kB; lsn=0/26E22E28, redo lsn=0/26E22E2850352026-11-12 00:06:36.809 UTC [129] LOG: database system is shut down5036Running OIDC tests...5037=== RUN TestAudienceForIssuer5038=== PAUSE TestAudienceForIssuer5039=== RUN TestGlobMatch5040=== PAUSE TestGlobMatch5041=== RUN TestValidateToken_ValidToken5042=== PAUSE TestValidateToken_ValidToken5043=== RUN TestValidateToken_WrongAudience5044=== PAUSE TestValidateToken_WrongAudience5045=== RUN TestValidateToken_Expired5046=== PAUSE TestValidateToken_Expired5047=== RUN TestValidateToken_BoundClaimsMismatch5048=== PAUSE TestValidateToken_BoundClaimsMismatch5049=== RUN TestValidateToken_BoundSubjectMismatch5050=== PAUSE TestValidateToken_BoundSubjectMismatch5051=== RUN TestValidateToken_MultipleProviders5052=== PAUSE TestValidateToken_MultipleProviders5053=== RUN TestValidateToken_NoMatchingProvider5054=== PAUSE TestValidateToken_NoMatchingProvider5055=== RUN TestValidateToken_KubernetesServiceAccount5056=== PAUSE TestValidateToken_KubernetesServiceAccount5057=== RUN TestNewValidator_KubernetesRequiresCA5058=== PAUSE TestNewValidator_KubernetesRequiresCA5059=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5060=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5061=== RUN TestPins_ReservedForMatchingRule5062=== PAUSE TestPins_ReservedForMatchingRule5063=== RUN TestPins_TopLevelShorthand5064=== PAUSE TestPins_TopLevelShorthand5065=== RUN TestPins_ConfigValidation5066=== PAUSE TestPins_ConfigValidation5067=== RUN TestScopes_LegacyProviderDefaultsToWrite5068=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5069=== RUN TestScopes_Rules5070=== PAUSE TestScopes_Rules5071=== RUN TestScopes_ConfigValidation5072=== PAUSE TestScopes_ConfigValidation5073=== CONT TestAudienceForIssuer5074--- PASS: TestAudienceForIssuer (0.00s)5075=== CONT TestScopes_ConfigValidation5076--- PASS: TestScopes_ConfigValidation (0.00s)5077=== CONT TestScopes_Rules5078=== CONT TestScopes_LegacyProviderDefaultsToWrite5079=== CONT TestPins_ConfigValidation5080=== CONT TestPins_TopLevelShorthand5081--- PASS: TestPins_ConfigValidation (0.00s)5082=== CONT TestValidateToken_MultipleProviders5083=== CONT TestNewValidator_KubernetesRequiresCA5084=== CONT TestValidateToken_BoundSubjectMismatch5085=== CONT TestValidateToken_WrongAudience5086=== CONT TestValidateToken_BoundClaimsMismatch5087=== CONT TestValidateToken_ValidToken5088=== CONT TestPins_ReservedForMatchingRule5089=== CONT TestValidateToken_KubernetesServiceAccount5090=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5091=== CONT TestValidateToken_Expired5092=== CONT TestGlobMatch5093=== RUN TestGlobMatch/foo_foo5094=== PAUSE TestGlobMatch/foo_foo5095=== RUN TestGlobMatch/foo_bar5096=== PAUSE TestGlobMatch/foo_bar5097=== RUN TestGlobMatch/*_5098=== PAUSE TestGlobMatch/*_5099=== RUN TestGlobMatch/*_anything5100=== PAUSE TestGlobMatch/*_anything5101=== RUN TestGlobMatch/foo*_foo5102=== PAUSE TestGlobMatch/foo*_foo5103=== RUN TestGlobMatch/foo*_foobar5104=== PAUSE TestGlobMatch/foo*_foobar5105=== RUN TestGlobMatch/foo*_bar5106=== PAUSE TestGlobMatch/foo*_bar5107=== RUN TestGlobMatch/*bar_bar5108=== PAUSE TestGlobMatch/*bar_bar5109=== RUN TestGlobMatch/*bar_foobar5110=== PAUSE TestGlobMatch/*bar_foobar5111=== RUN TestGlobMatch/*bar_foo5112=== PAUSE TestGlobMatch/*bar_foo5113=== CONT TestValidateToken_NoMatchingProvider5114=== RUN TestGlobMatch/foo*bar_foobar5115=== PAUSE TestGlobMatch/foo*bar_foobar5116=== RUN TestGlobMatch/foo*bar_foo123bar5117=== PAUSE TestGlobMatch/foo*bar_foo123bar5118=== RUN TestGlobMatch/foo*bar_foobarbaz5119=== PAUSE TestGlobMatch/foo*bar_foobarbaz5120=== RUN TestGlobMatch/*/*_foo/bar5121=== PAUSE TestGlobMatch/*/*_foo/bar5122=== RUN TestGlobMatch/*/*_foo5123=== PAUSE TestGlobMatch/*/*_foo5124=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5125=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5126=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05127=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05128=== RUN TestGlobMatch/refs/*/main_refs/heads/main5129=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5130=== RUN TestGlobMatch/fo?_foo5131=== PAUSE TestGlobMatch/fo?_foo5132=== RUN TestGlobMatch/fo?_fo5133=== PAUSE TestGlobMatch/fo?_fo5134=== RUN TestGlobMatch/fo?_fooo5135=== PAUSE TestGlobMatch/fo?_fooo5136=== RUN TestGlobMatch/?oo_foo5137=== PAUSE TestGlobMatch/?oo_foo5138=== RUN TestGlobMatch/?oo_boo5139=== PAUSE TestGlobMatch/?oo_boo5140=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5141=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5142=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5143=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5144=== CONT TestGlobMatch/foo_foo5145=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5146=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5147=== CONT TestGlobMatch/?oo_boo5148=== CONT TestGlobMatch/?oo_foo5149=== CONT TestGlobMatch/fo?_fooo5150=== CONT TestGlobMatch/fo?_fo5151=== CONT TestGlobMatch/fo?_foo5152=== CONT TestGlobMatch/refs/*/main_refs/heads/main5153=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05154=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5155=== CONT TestGlobMatch/*/*_foo5156=== CONT TestGlobMatch/*/*_foo/bar5157=== CONT TestGlobMatch/foo*bar_foobarbaz5158=== CONT TestGlobMatch/foo*bar_foo123bar5159=== CONT TestGlobMatch/foo*bar_foobar5160=== CONT TestGlobMatch/*bar_foo5161=== CONT TestGlobMatch/*bar_foobar5162=== CONT TestGlobMatch/*bar_bar5163=== CONT TestGlobMatch/foo*_bar5164=== CONT TestGlobMatch/foo*_foobar5165=== CONT TestGlobMatch/foo*_foo5166=== CONT TestGlobMatch/*_anything5167=== CONT TestGlobMatch/*_5168=== CONT TestGlobMatch/foo_bar5169--- PASS: TestGlobMatch (0.00s)5170 --- PASS: TestGlobMatch/foo_foo (0.00s)5171 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5172 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5173 --- PASS: TestGlobMatch/?oo_boo (0.00s)5174 --- PASS: TestGlobMatch/?oo_foo (0.00s)5175 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5176 --- PASS: TestGlobMatch/fo?_fo (0.00s)5177 --- PASS: TestGlobMatch/fo?_foo (0.00s)5178 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5179 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5180 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5181 --- PASS: TestGlobMatch/*/*_foo (0.00s)5182 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5183 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5184 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5185 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5186 --- PASS: TestGlobMatch/*bar_foo (0.00s)5187 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5188 --- PASS: TestGlobMatch/*bar_bar (0.00s)5189 --- PASS: TestGlobMatch/foo*_bar (0.00s)5190 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5191 --- PASS: TestGlobMatch/foo*_foo (0.00s)5192 --- PASS: TestGlobMatch/*_anything (0.00s)5193 --- PASS: TestGlobMatch/*_ (0.00s)5194 --- PASS: TestGlobMatch/foo_bar (0.00s)51952026/11/12 00:06:41 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36405/oidc5196--- PASS: TestValidateToken_ValidToken (5.81s)51972026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35625/oidc51982026/11/12 00:06:45 http: TLS handshake error from 127.0.0.1:60378: remote error: tls: bad certificate5199--- PASS: TestNewValidator_KubernetesRequiresCA (5.82s)5200--- PASS: TestScopes_Rules (5.87s)52012026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45547/oidc52022026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45593/oidc5203--- PASS: TestValidateToken_WrongAudience (5.84s)5204--- PASS: TestScopes_LegacyProviderDefaultsToWrite (5.87s)52052026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40451/oidc5206--- PASS: TestValidateToken_Expired (5.85s)52072026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:45761/oidc5208--- PASS: TestValidateToken_BoundSubjectMismatch (5.92s)52092026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:43005/oidc52102026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42533/oidc5211--- PASS: TestPins_ReservedForMatchingRule (5.97s)5212--- PASS: TestValidateToken_BoundClaimsMismatch (5.98s)52132026/11/12 00:06:45 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:43855/oidc5214--- PASS: TestValidateToken_NoMatchingProvider (5.98s)52152026/11/12 00:06:45 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:462995216--- PASS: TestValidateToken_KubernetesServiceAccount (6.01s)52172026/11/12 00:06:45 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:46587/oidc52182026/11/12 00:06:45 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:42435/oidc5219--- PASS: TestValidateToken_MultipleProviders (6.10s)52202026/11/12 00:06:45 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:35131/oidc5221--- PASS: TestPins_TopLevelShorthand (6.18s)52222026/11/12 00:06:45 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC1235223--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (6.29s)5224PASS5225Running hook tests...5226=== RUN TestSendPathsEmpty5227=== PAUSE TestSendPathsEmpty5228=== RUN TestQueueEnqueueAndFetch5229=== PAUSE TestQueueEnqueueAndFetch5230=== RUN TestQueueDeduplication5231=== PAUSE TestQueueDeduplication5232=== RUN TestQueueRemove5233=== PAUSE TestQueueRemove5234=== RUN TestQueueFetchBatchLimit5235=== PAUSE TestQueueFetchBatchLimit5236=== RUN TestQueueRetryMovesToBack5237=== PAUSE TestQueueRetryMovesToBack5238=== RUN TestQueueFetchRemoveLifecycle5239=== PAUSE TestQueueFetchRemoveLifecycle5240=== RUN TestQueueConcurrentWriters5241=== PAUSE TestQueueConcurrentWriters5242=== RUN TestQueueRemoveLargeClosure5243=== PAUSE TestQueueRemoveLargeClosure5244=== RUN TestServerClientIntegration5245--- PASS: TestServerClientIntegration (0.01s)5246=== RUN TestServerQueueError52472026/11/12 00:06:47 ERROR Failed to queue paths error="permission denied" count=15248--- PASS: TestServerQueueError (0.00s)5249=== RUN TestGetListenerSocketActivation5250 server_test.go:220: === RUN TestGetListenerSocketActivation5251 --- PASS: TestGetListenerSocketActivation (0.00s)5252 PASS5253 5254--- PASS: TestGetListenerSocketActivation (1.02s)5255=== RUN TestServerStalledClientDoesNotBlockShutdown5256=== PAUSE TestServerStalledClientDoesNotBlockShutdown5257=== RUN TestServerBacksOffOnAcceptErrors5258=== PAUSE TestServerBacksOffOnAcceptErrors5259=== RUN TestDrainIsolatesPoisonPath5260=== PAUSE TestDrainIsolatesPoisonPath5261=== RUN TestRunNotBlockedByPoisonHead5262=== PAUSE TestRunNotBlockedByPoisonHead5263=== RUN TestDrainGivesUpWhenServerDown5264=== PAUSE TestDrainGivesUpWhenServerDown5265=== RUN TestFailedPathPrunedByLaterClosure5266=== PAUSE TestFailedPathPrunedByLaterClosure5267=== RUN TestWorkerUploadsAndRemoves5268=== PAUSE TestWorkerUploadsAndRemoves5269=== RUN TestWorkerSkipsGCdPaths5270=== PAUSE TestWorkerSkipsGCdPaths5271=== RUN TestWorkerPrunesClosureDeps5272=== PAUSE TestWorkerPrunesClosureDeps5273=== RUN TestDrainTimeout5274=== PAUSE TestDrainTimeout5275=== RUN TestWorkerRemoveFailureIsNotProgress5276=== PAUSE TestWorkerRemoveFailureIsNotProgress5277=== RUN TestWorkerKeepsPathItCannotStat5278=== PAUSE TestWorkerKeepsPathItCannotStat5279=== CONT TestSendPathsEmpty5280--- PASS: TestSendPathsEmpty (0.00s)5281=== CONT TestServerStalledClientDoesNotBlockShutdown5282=== CONT TestQueueRemoveLargeClosure5283=== CONT TestQueueConcurrentWriters5284=== CONT TestDrainTimeout5285=== CONT TestQueueFetchRemoveLifecycle5286=== CONT TestQueueRetryMovesToBack5287=== CONT TestQueueFetchBatchLimit5288=== CONT TestQueueRemove5289=== CONT TestQueueDeduplication5290=== CONT TestFailedPathPrunedByLaterClosure5291=== CONT TestWorkerPrunesClosureDeps5292=== CONT TestWorkerSkipsGCdPaths5293=== CONT TestWorkerUploadsAndRemoves5294=== CONT TestRunNotBlockedByPoisonHead5295=== CONT TestDrainGivesUpWhenServerDown5296=== CONT TestWorkerKeepsPathItCannotStat5297=== CONT TestQueueEnqueueAndFetch5298=== CONT TestDrainIsolatesPoisonPath5299=== CONT TestWorkerRemoveFailureIsNotProgress5300=== CONT TestServerBacksOffOnAcceptErrors53012000/01/01 00:00:00 ERROR Accept failed error="too many open files"53022000/01/01 00:00:00 ERROR Accept failed error="too many open files"53032000/01/01 00:00:00 ERROR Accept failed error="too many open files"53042000/01/01 00:00:00 ERROR Accept failed error="too many open files"53052000/01/01 00:00:00 ERROR Accept failed error="too many open files"53062000/01/01 00:00:00 ERROR Accept failed error="too many open files"53072000/01/01 00:00:00 ERROR Accept failed error="too many open files"53082000/01/01 00:00:00 ERROR Accept failed error="too many open files"5309--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)53102026/11/12 00:06:48 INFO Uploading batch count=153112026/11/12 00:06:48 ERROR Upload failed error="upload failed" count=153122026/11/12 00:06:48 INFO Uploading batch count=153132026/11/12 00:06:48 INFO Uploading batch count=15314--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)53152026/11/12 00:06:48 INFO Uploading batch count=253162026/11/12 00:06:48 ERROR Upload failed error="upload failed" count=253172026/11/12 00:06:48 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1674296167/002/a53182026/11/12 00:06:48 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1674296167/002/b53192026/11/12 00:06:48 INFO Uploading batch count=253202026/11/12 00:06:48 ERROR Upload failed error="upload failed" count=253212026/11/12 00:06:48 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1674296167/002/c53222026/11/12 00:06:48 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1674296167/002/d53232026/11/12 00:06:48 INFO Uploading batch count=253242026/11/12 00:06:48 ERROR Upload failed error="upload failed" count=253252026/11/12 00:06:48 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1674296167/002/e53262026/11/12 00:06:48 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1674296167/002/f53272026/11/12 00:06:48 ERROR Drain finished with paths left in queue remaining=105328--- PASS: TestDrainGivesUpWhenServerDown (0.02s)53292026/11/12 00:06:48 INFO Upload queue status pending=353302026/11/12 00:06:48 INFO Uploading batch count=153312026/11/12 00:06:48 ERROR Upload failed error="upload failed" count=153322026/11/12 00:06:48 INFO Upload queue status pending=253332026/11/12 00:06:48 INFO Uploading batch count=253342026/11/12 00:06:48 INFO Upload queue status pending=253352026/11/12 00:06:48 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths2019093702/002/nonexistent53362026/11/12 00:06:48 INFO Uploading batch count=153372026/11/12 00:06:48 INFO Uploading batch count=453382026/11/12 00:06:48 ERROR Upload failed error="upload failed" count=453392026/11/12 00:06:48 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath631425819/002/bbb53402026/11/12 00:06:48 INFO Uploading batch count=153412026/11/12 00:06:48 ERROR Upload failed error="upload failed" count=153422026/11/12 00:06:50 INFO Uploading batch count=153432026/11/12 00:06:50 ERROR Upload failed error="upload failed" count=153442026/11/12 00:06:50 INFO Uploading batch count=153452026/11/12 00:06:50 INFO Uploading batch count=153462026/11/12 00:06:50 ERROR Upload failed error="upload failed" count=153472026/11/12 00:06:50 ERROR Drain finished with paths left in queue remaining=153482026/11/12 00:06:50 ERROR Failed to decode request error="read unix /build/hook437416316/test.sock->@: i/o timeout"53492026/11/12 00:06:50 ERROR Failed to write response error="write unix /build/hook437416316/test.sock->@: i/o timeout"5350--- PASS: TestServerStalledClientDoesNotBlockShutdown (2.79s)53512026/11/12 00:06:50 INFO Uploading batch count=15352--- PASS: TestWorkerUploadsAndRemoves (2.79s)53532026/11/12 00:06:50 INFO Uploading batch count=153542026/11/12 00:06:50 ERROR Upload failed error="upload failed" count=15355--- PASS: TestQueueFetchBatchLimit (2.79s)53562026/11/12 00:06:50 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1827457747/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1827457747/002/locked/aaa: permission denied"5357--- PASS: TestQueueEnqueueAndFetch (2.79s)53582000/01/01 00:00:00 INFO Uploading batch count=253592000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253602000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=45361--- PASS: TestDrainIsolatesPoisonPath (2.79s)53622026/11/12 00:06:50 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1827457747/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1827457747/002/locked/aaa: permission denied"5363--- PASS: TestQueueRemove (2.79s)5364--- PASS: TestWorkerSkipsGCdPaths (2.79s)53652026/11/12 00:06:50 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat1827457747/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat1827457747/002/locked/aaa: permission denied"53662026/11/12 00:06:50 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress4118541191/002/nonexistent53672026/11/12 00:06:50 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153682026/11/12 00:06:50 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress4118541191/002/nonexistent5369--- PASS: TestQueueRetryMovesToBack (2.80s)53702026/11/12 00:06:50 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15371--- PASS: TestDrainTimeout (2.80s)53722026/11/12 00:06:50 ERROR Drain finished with paths left in queue remaining=153732026/11/12 00:06:50 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress4118541191/002/nonexistent53742026/11/12 00:06:50 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153752026/11/12 00:06:50 ERROR Drain finished with paths left in queue remaining=15376--- PASS: TestWorkerRemoveFailureIsNotProgress (2.79s)5377--- PASS: TestWorkerKeepsPathItCannotStat (2.80s)5378--- PASS: TestQueueFetchRemoveLifecycle (2.80s)5379--- PASS: TestQueueDeduplication (2.80s)53802026/11/12 00:06:51 INFO Uploading batch count=153812026/11/12 00:06:51 ERROR Upload failed error="upload failed" count=153822026/11/12 00:06:51 INFO Uploading batch count=153832026/11/12 00:06:51 ERROR Upload failed error="upload failed" count=153842026/11/12 00:06:51 INFO Uploading batch count=153852026/11/12 00:06:51 ERROR Upload failed error="upload failed" count=153862026/11/12 00:06:51 ERROR Drain finished with paths left in queue remaining=15387--- PASS: TestRunNotBlockedByPoisonHead (2.80s)5388--- PASS: TestQueueConcurrentWriters (2.91s)5389--- PASS: TestQueueRemoveLargeClosure (2.98s)53902026/11/12 00:06:52 INFO Upload queue status pending=253912026/11/12 00:06:52 INFO Uploading batch count=15392--- PASS: TestWorkerPrunesClosureDeps (4.71s)5393PASS5394Running signing tests...5395=== RUN TestExternalSigner5396=== PAUSE TestExternalSigner5397=== RUN TestExternalSignerWrongPublicKey5398=== PAUSE TestExternalSignerWrongPublicKey5399=== RUN TestExternalSignerErrors5400=== PAUSE TestExternalSignerErrors5401=== RUN TestExternalSignerRestartsAfterCrash5402=== PAUSE TestExternalSignerRestartsAfterCrash5403=== RUN TestExternalSignerProgramErrorKeepsProcess5404=== PAUSE TestExternalSignerProgramErrorKeepsProcess5405=== RUN TestExternalSignerClose5406=== PAUSE TestExternalSignerClose5407=== RUN TestExternalSignerRejectsInvalidPublicKey5408=== PAUSE TestExternalSignerRejectsInvalidPublicKey5409=== RUN TestExternalSignerCancellation5410=== PAUSE TestExternalSignerCancellation5411=== RUN TestGenerateFingerprint5412=== PAUSE TestGenerateFingerprint5413=== RUN TestParseSigningKey5414=== PAUSE TestParseSigningKey5415=== RUN TestSign5416=== PAUSE TestSign5417=== RUN TestSignBatch5418=== PAUSE TestSignBatch5419=== RUN TestNilKeyPublicKey5420=== PAUSE TestNilKeyPublicKey5421=== RUN TestSignErrors5422=== PAUSE TestSignErrors5423=== CONT TestExternalSigner5424=== CONT TestExternalSignerCancellation5425=== CONT TestSignBatch5426=== CONT TestNilKeyPublicKey5427--- PASS: TestNilKeyPublicKey (0.00s)5428=== CONT TestExternalSignerProgramErrorKeepsProcess5429=== CONT TestExternalSignerRejectsInvalidPublicKey5430=== RUN TestExternalSignerRejectsInvalidPublicKey/#005431=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005432=== CONT TestExternalSignerClose5433=== CONT TestExternalSignerErrors5434=== RUN TestExternalSignerErrors/exit_on_request5435=== CONT TestExternalSignerRestartsAfterCrash5436=== CONT TestParseSigningKey5437=== CONT TestSign5438=== CONT TestGenerateFingerprint5439=== CONT TestExternalSignerWrongPublicKey5440=== CONT TestSignErrors5441=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5442=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5443=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5444=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5445=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5446=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5447=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5448=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5449=== RUN TestSignErrors/nil_key5450=== RUN TestParseSigningKey/valid_32-byte_key5451=== RUN TestGenerateFingerprint/basic_with_references5452=== PAUSE TestGenerateFingerprint/basic_with_references5453=== PAUSE TestParseSigningKey/valid_32-byte_key5454=== CONT TestExternalSignerRejectsInvalidPublicKey/#005455=== RUN TestGenerateFingerprint/no_references5456=== PAUSE TestGenerateFingerprint/no_references5457=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5458=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5459=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5460=== PAUSE TestSignErrors/nil_key5461=== RUN TestSignErrors/nil_narinfo5462=== PAUSE TestSignErrors/nil_narinfo5463=== RUN TestSignErrors/invalid_narinfo5464=== PAUSE TestSignErrors/invalid_narinfo5465=== RUN TestSignErrors/canceled5466=== PAUSE TestSignErrors/canceled5467=== CONT TestSignErrors/nil_key5468=== PAUSE TestExternalSignerErrors/exit_on_request5469=== RUN TestExternalSignerErrors/wrong_shape5470=== PAUSE TestExternalSignerErrors/wrong_shape5471=== RUN TestExternalSignerErrors/wrong_id5472=== PAUSE TestExternalSignerErrors/wrong_id5473=== RUN TestExternalSignerErrors/missing_signatures5474=== PAUSE TestExternalSignerErrors/missing_signatures5475=== RUN TestExternalSignerErrors/missing_signature_name5476=== PAUSE TestExternalSignerErrors/missing_signature_name5477=== RUN TestExternalSignerErrors/wrong_signature_name5478=== PAUSE TestExternalSignerErrors/wrong_signature_name5479=== RUN TestExternalSignerErrors/wrong_signature_length5480=== PAUSE TestExternalSignerErrors/wrong_signature_length5481=== RUN TestExternalSignerErrors/program_error5482=== PAUSE TestExternalSignerErrors/program_error5483=== RUN TestExternalSignerErrors/exit_at_startup5484=== PAUSE TestExternalSignerErrors/exit_at_startup5485=== RUN TestExternalSignerErrors/null5486=== PAUSE TestExternalSignerErrors/null5487=== RUN TestExternalSignerErrors/invalid_signature_base645488=== PAUSE TestExternalSignerErrors/invalid_signature_base645489=== RUN TestExternalSignerErrors/null_signatures5490=== PAUSE TestExternalSignerErrors/null_signatures5491=== RUN TestExternalSignerErrors/wrong_signatures_shape5492=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5493=== RUN TestExternalSignerErrors/too_few_signatures5494=== PAUSE TestExternalSignerErrors/too_few_signatures5495=== RUN TestExternalSignerErrors/invalid_JSON5496=== CONT TestSignErrors/canceled5497=== PAUSE TestExternalSignerErrors/invalid_JSON5498=== RUN TestExternalSignerErrors/unrequested_key5499=== PAUSE TestExternalSignerErrors/unrequested_key5500=== CONT TestExternalSignerErrors/exit_on_request5501=== CONT TestSignErrors/nil_narinfo5502=== CONT TestSignErrors/invalid_narinfo5503=== CONT TestExternalSignerErrors/invalid_JSON5504=== CONT TestExternalSignerErrors/null5505=== CONT TestExternalSignerErrors/wrong_signatures_shape5506=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5507=== CONT TestExternalSignerErrors/null_signatures5508=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5509=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5510=== CONT TestExternalSignerErrors/wrong_shape5511=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5512=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5513=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5514--- PASS: TestSignBatch (0.00s)5515=== CONT TestExternalSignerErrors/unrequested_key5516=== RUN TestGenerateFingerprint/invalid_nar_hash_length5517=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5518=== CONT TestExternalSignerErrors/invalid_signature_base645519--- PASS: TestSign (0.00s)5520=== RUN TestParseSigningKey/no_colon5521=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5522=== RUN TestGenerateFingerprint/invalid_store_path_prefix5523=== CONT TestExternalSignerErrors/wrong_signature_name5524=== CONT TestExternalSignerErrors/exit_at_startup5525=== CONT TestExternalSignerErrors/program_error5526=== CONT TestExternalSignerErrors/wrong_signature_length5527=== CONT TestExternalSignerErrors/missing_signatures5528=== CONT TestExternalSignerErrors/missing_signature_name5529=== CONT TestExternalSignerErrors/wrong_id5530=== CONT TestExternalSignerErrors/too_few_signatures5531=== PAUSE TestParseSigningKey/no_colon5532--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5533 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5534 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5535 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5536 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5537 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5538=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5539--- PASS: TestSignErrors (0.00s)5540 --- PASS: TestSignErrors/nil_key (0.00s)5541 --- PASS: TestSignErrors/canceled (0.00s)5542 --- PASS: TestSignErrors/nil_narinfo (0.00s)5543 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5544--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.01s)5545--- PASS: TestExternalSignerClose (0.01s)5546=== RUN TestParseSigningKey/empty_name5547=== RUN TestGenerateFingerprint/invalid_reference_prefix5548=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5549=== CONT TestGenerateFingerprint/basic_with_references5550=== CONT TestGenerateFingerprint/invalid_store_path_prefix5551=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5552=== CONT TestGenerateFingerprint/invalid_reference_prefix5553=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5554=== CONT TestGenerateFingerprint/invalid_nar_hash_length5555=== CONT TestGenerateFingerprint/no_references5556--- PASS: TestExternalSignerWrongPublicKey (0.02s)5557=== PAUSE TestParseSigningKey/empty_name5558--- PASS: TestGenerateFingerprint (0.01s)5559 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5560 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5561 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5562 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5563 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5564 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5565 --- PASS: TestGenerateFingerprint/no_references (0.00s)5566=== RUN TestParseSigningKey/invalid_base645567=== PAUSE TestParseSigningKey/invalid_base645568=== RUN TestParseSigningKey/wrong_length5569=== PAUSE TestParseSigningKey/wrong_length5570=== CONT TestParseSigningKey/valid_32-byte_key5571=== CONT TestParseSigningKey/empty_name5572=== CONT TestParseSigningKey/no_colon5573=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5574=== CONT TestParseSigningKey/wrong_length5575=== CONT TestParseSigningKey/invalid_base645576--- PASS: TestExternalSigner (0.02s)5577--- PASS: TestParseSigningKey (0.02s)5578 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5579 --- PASS: TestParseSigningKey/empty_name (0.00s)5580 --- PASS: TestParseSigningKey/no_colon (0.00s)5581 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5582 --- PASS: TestParseSigningKey/wrong_length (0.00s)5583 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5584--- PASS: TestExternalSignerRestartsAfterCrash (0.02s)5585--- PASS: TestExternalSignerErrors (0.00s)5586 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5587 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5588 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5589 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5590 --- PASS: TestExternalSignerErrors/null (0.01s)5591 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5592 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.01s)5593 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5594 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5595 --- PASS: TestExternalSignerErrors/exit_at_startup (0.01s)5596 --- PASS: TestExternalSignerErrors/program_error (0.01s)5597 --- PASS: TestExternalSignerErrors/wrong_id (0.01s)5598 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5599 --- PASS: TestExternalSignerErrors/missing_signatures (0.01s)5600 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5601 --- PASS: TestExternalSignerErrors/too_few_signatures (0.01s)5602--- PASS: TestExternalSignerCancellation (0.02s)5603PASS5604Running OpenBao signer tests...5605=== RUN TestSign5606=== PAUSE TestSign5607=== RUN TestSignErrors5608=== PAUSE TestSignErrors5609=== RUN TestServe5610=== PAUSE TestServe5611=== RUN TestServeRejectsGarbage5612=== PAUSE TestServeRejectsGarbage5613=== CONT TestSign5614=== CONT TestSignErrors5615=== CONT TestServe5616--- PASS: TestServe (0.00s)5617=== RUN TestSignErrors/http_error5618=== PAUSE TestSignErrors/http_error5619=== RUN TestSignErrors/invalid_json5620=== PAUSE TestSignErrors/invalid_json5621=== RUN TestSignErrors/no_prefix5622=== PAUSE TestSignErrors/no_prefix5623=== RUN TestSignErrors/invalid_base645624=== PAUSE TestSignErrors/invalid_base645625=== CONT TestSignErrors/http_error5626=== CONT TestServeRejectsGarbage5627--- PASS: TestServeRejectsGarbage (0.00s)5628=== CONT TestSignErrors/invalid_base645629=== CONT TestSignErrors/no_prefix5630=== CONT TestSignErrors/invalid_json5631--- PASS: TestSign (1.57s)5632--- PASS: TestSignErrors (0.00s)5633 --- PASS: TestSignErrors/invalid_base64 (1.57s)5634 --- PASS: TestSignErrors/http_error (1.57s)5635 --- PASS: TestSignErrors/invalid_json (1.57s)5636 --- PASS: TestSignErrors/no_prefix (1.57s)5637PASS