niks3-go-unit-tests log

succeeded `niks3-go-unit-tests` `checks.aarch64-darwin.go-unit-tests` · build #494 · raw

1tribuchet: building on mac022Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.05s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestDoServerRequestAttachesToken114=== CONT TestRetryTimeoutOutlastsMaxRetries115=== CONT TestParsePathInfoJSONMultiplePaths116=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths117=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths118=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths119=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths120=== CONT TestRateLimiterFeedback121=== RUN TestRateLimiterFeedback/429_enables_limiter122=== PAUSE TestRateLimiterFeedback/429_enables_limiter123=== RUN TestRateLimiterFeedback/503_enables_limiter124=== PAUSE TestRateLimiterFeedback/503_enables_limiter125=== CONT TestPathInfoHashCompatibility126=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)127=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)128=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon129=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter130=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter131=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter132=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter133=== CONT TestDoWithRetry_BodyReplayedViaGetBody134=== CONT TestParsePathInfoJSON135=== CONT TestResolveStorePath136=== RUN TestParsePathInfoJSON/Nix_format137=== PAUSE TestParsePathInfoJSON/Nix_format138=== RUN TestParsePathInfoJSON/Lix_format139=== PAUSE TestParsePathInfoJSON/Lix_format140=== RUN TestParsePathInfoJSON/empty_input141=== PAUSE TestParsePathInfoJSON/empty_input142=== CONT TestRegisterUploadedObject_OutlivesPush143=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer144=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess145=== CONT TestSetClientTLS146=== CONT TestScriptTokenEmptyCommand147=== CONT TestScriptTokenScriptFails148=== CONT TestScriptTokenBadJSON149=== CONT TestScriptTokenEmptyToken150=== CONT TestScriptTokenCachesUntilRefresh151=== CONT TestScriptTokenNoExpiryRerunsEveryCall152=== CONT TestFileTokenEmpty153=== CONT TestFileTokenMissing154=== CONT TestFileTokenReadsAndCaches155=== CONT TestStaticToken156=== CONT TestSetClientTLSErrors157=== CONT TestSetClientTLSDoesNotMutateDefaultTransport1582026/11/12 00:02:19 WARN Rate limiter enabled after throttle name=server-test rate=5159=== CONT TestEncodeNixBase32WithRealHash160=== CONT TestStreamPushStopsWhenCanceled161=== CONT TestClientSignaturesByStorePath162=== CONT TestStreamPushReportsSkippedPaths163=== CONT TestPathInfoCACompatibility164=== RUN TestPathInfoCACompatibility/null_ca_field165=== PAUSE TestPathInfoCACompatibility/null_ca_field166=== RUN TestPathInfoCACompatibility/old_string_format_-_text167=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text168=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive169=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive170=== RUN TestPathInfoCACompatibility/new_structured_format_-_text171=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text172=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method173=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method174=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon175=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI176=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI177=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512178=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512179=== CONT TestStreamPushReportsSignatures180=== RUN TestParsePathInfoJSON/whitespace_only181=== PAUSE TestParsePathInfoJSON/whitespace_only182=== RUN TestParsePathInfoJSON/invalid_JSON183=== PAUSE TestParsePathInfoJSON/invalid_JSON184=== CONT TestStreamPushReportsEveryPath185=== CONT TestStreamPushRequestLine186=== CONT TestStreamPushGivesUpOnDeadServer187--- PASS: TestScriptTokenEmptyCommand (0.00s)188=== CONT TestStreamPushIsolatesFailures189--- PASS: TestEncodeNixBase32WithRealHash (0.00s)190--- PASS: TestStreamPushStopsWhenCanceled (0.00s)191--- PASS: TestClientSignaturesByStorePath (0.00s)192--- PASS: TestStreamPushReportsSkippedPaths (0.00s)1932026/11/12 00:02:19 ERROR Upload failed error="connection refused" count=201942026/11/12 00:02:19 ERROR Upload failed error="bad path" count=3195--- PASS: TestResolveStorePath (0.00s)196=== CONT TestStreamPushBatchesUnderLoad1972026/11/12 00:02:19 ERROR Server seems unavailable, giving up on batch untried=171982026/11/12 00:02:19 WARN Rate limiter enabled after throttle name=server-test rate=51992026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:62032200--- PASS: TestStreamPushIsolatesFailures (0.00s)201=== CONT TestEncodeNixBase32202=== RUN TestEncodeNixBase32/test_string_hash203=== PAUSE TestEncodeNixBase32/test_string_hash204=== RUN TestEncodeNixBase32/empty_input205=== PAUSE TestEncodeNixBase32/empty_input206=== CONT TestUploadMultipart_PartsInParallel207=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false208=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false209=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true210=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true211=== CONT TestDumpPathMatchesNix212--- PASS: TestStreamPushGivesUpOnDeadServer (0.00s)213=== CONT TestDumpPathWriterError2142026/11/12 00:02:19 WARN Rate limiter backed off name=server-test rate=52152026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:620322162026/11/12 00:02:19 ERROR Upload failed error=boom count=1217--- PASS: TestStaticToken (0.00s)218--- PASS: TestFileTokenMissing (0.00s)219--- PASS: TestDoServerRequestAttachesToken (0.00s)220=== CONT TestDumpPathSingleFile221--- PASS: TestStreamPushReportsSignatures (0.00s)222=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun223--- PASS: TestRegisterUploadedObject_OutlivesPush (0.00s)224=== CONT TestUploadMultipart_ReturnsAfterDroppedBody225=== CONT TestRegisterUploadedObjectReusesConnections2262026/11/12 00:02:19 ERROR Upload failed error=boom count=12272026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:62031228=== CONT TestShellSplit229=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica230=== CONT TestShellSplitErrors231=== CONT TestDoWithRetry_FinalResponseBodyReadable232=== CONT TestGetStorePathHash233=== RUN TestGetStorePathHash/valid_store_path234=== PAUSE TestGetStorePathHash/valid_store_path235=== RUN TestGetStorePathHash/basename_without_hyphen_should_error236=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error237=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error238=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error239=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error240=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error241=== CONT TestConvertHashToNix32242=== RUN TestConvertHashToNix32/SRI_format_to_Nix32243=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32244=== RUN TestConvertHashToNix32/already_Nix32_format245=== PAUSE TestConvertHashToNix32/already_Nix32_format246=== RUN TestConvertHashToNix32/invalid_format247=== PAUSE TestConvertHashToNix32/invalid_format248=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry249=== CONT TestPartSizeForNAR250=== RUN TestPartSizeForNAR/zero_stays_at_minimum251=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum252=== RUN TestPartSizeForNAR/small_stays_at_minimum253=== PAUSE TestPartSizeForNAR/small_stays_at_minimum254=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum255=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum256=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts257=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts258=== RUN TestPartSizeForNAR/1_TiB259=== PAUSE TestPartSizeForNAR/1_TiB260=== RUN TestPartSizeForNAR/5_TiB_S3_max_object261=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object262=== RUN TestPartSizeForNAR/capped_at_5_GiB263=== PAUSE TestPartSizeForNAR/capped_at_5_GiB264=== RUN TestSetClientTLS/rejects_connection_without_client_cert265=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert266=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA267=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA268=== RUN TestSetClientTLS/preserves_debug_logging_transport269=== PAUSE TestSetClientTLS/preserves_debug_logging_transport270=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths271=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter272=== RUN TestSetClientTLSErrors/missing_cert_file273=== PAUSE TestSetClientTLSErrors/missing_cert_file274=== RUN TestSetClientTLSErrors/missing_key_file275=== PAUSE TestSetClientTLSErrors/missing_key_file276=== RUN TestSetClientTLSErrors/missing_ca_file277=== CONT TestRateLimiterFeedback/429_enables_limiter278=== PAUSE TestSetClientTLSErrors/missing_ca_file279=== CONT TestCaseHackSuffix280--- PASS: TestStreamPushReportsEveryPath (0.00s)281=== CONT TestFilterOversizedClosures282=== RUN TestFilterOversizedClosures/no_limit_keeps_everything283=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything284=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped285=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped286=== RUN TestFilterOversizedClosures/all_closures_skipped287=== PAUSE TestFilterOversizedClosures/all_closures_skipped288=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths289=== CONT TestRateLimiterFeedback/503_enables_limiter290=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter291--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)292--- PASS: TestFileTokenEmpty (0.00s)293--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)294=== RUN TestSetClientTLSErrors/invalid_ca_file295=== PAUSE TestSetClientTLSErrors/invalid_ca_file296--- PASS: TestScriptTokenScriptFails (0.01s)297--- PASS: TestFileTokenReadsAndCaches (0.00s)298--- PASS: TestShellSplit (0.00s)299--- PASS: TestShellSplitErrors (0.00s)300=== CONT TestPathInfoCACompatibility/null_ca_field301--- PASS: TestParsePathInfoJSONMultiplePaths (0.00s)302 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)303 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)304=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)305=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method306=== CONT TestPathInfoCACompatibility/new_structured_format_-_text3072026/11/12 00:02:19 INFO Garbage collection started3082026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:62031309=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive310=== CONT TestPathInfoCACompatibility/old_string_format_-_text311--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.00s)312--- PASS: TestPathInfoCACompatibility (0.00s)313 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)314 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)315 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)316 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)317 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)318--- PASS: TestScriptTokenBadJSON (0.01s)319=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512320=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon3212026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:62031322--- PASS: TestScriptTokenEmptyToken (0.01s)323=== CONT TestParsePathInfoJSON/Nix_format324=== CONT TestParsePathInfoJSON/empty_input325=== CONT TestParsePathInfoJSON/Lix_format326=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI327=== CONT TestEncodeNixBase32/test_string_hash328=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true329=== CONT TestParsePathInfoJSON/invalid_JSON330=== CONT TestEncodeNixBase32/empty_input331=== CONT TestGetStorePathHash/valid_store_path332=== CONT TestConvertHashToNix32/SRI_format_to_Nix32333=== CONT TestPartSizeForNAR/zero_stays_at_minimum334=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error335=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error336=== CONT TestGetStorePathHash/basename_without_hyphen_should_error337=== CONT TestSetClientTLS/rejects_connection_without_client_cert338=== CONT TestPartSizeForNAR/capped_at_5_GiB339=== CONT TestPartSizeForNAR/5_TiB_S3_max_object340=== CONT TestPartSizeForNAR/1_TiB341=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts342=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum343=== CONT TestPartSizeForNAR/small_stays_at_minimum344=== CONT TestSetClientTLS/preserves_debug_logging_transport345--- PASS: TestPathInfoHashCompatibility (0.00s)346 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)347 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)348 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)349 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)350=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false351--- PASS: TestEncodeNixBase32 (0.00s)352 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)353 --- PASS: TestEncodeNixBase32/empty_input (0.00s)354--- PASS: TestGetStorePathHash (0.00s)355 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)356 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)357 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)358 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)359=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA360--- PASS: TestStreamPushBatchesUnderLoad (0.00s)361 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)362 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)363--- PASS: TestPartSizeForNAR (0.00s)364 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)365 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)366 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)367 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)368 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)369 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)370 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)3712026/11/12 00:02:19 INFO Garbage collection started3722026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:62031373=== CONT TestParsePathInfoJSON/whitespace_only374--- PASS: TestParsePathInfoJSON (0.00s)375 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)376 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)377 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)378 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)379 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)380=== CONT TestConvertHashToNix32/invalid_format381=== CONT TestConvertHashToNix32/already_Nix32_format382--- PASS: TestConvertHashToNix32 (0.00s)383 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)384 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)385 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)386=== CONT TestFilterOversizedClosures/no_limit_keeps_everything387=== CONT TestFilterOversizedClosures/all_closures_skipped3882026/11/12 00:02:19 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50389=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3902026/11/12 00:02:19 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000391--- PASS: TestFilterOversizedClosures (0.00s)392 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)393 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)394 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)395=== CONT TestSetClientTLSErrors/missing_cert_file396=== CONT TestSetClientTLSErrors/invalid_ca_file397=== CONT TestSetClientTLSErrors/missing_ca_file3982026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:620313992026/11/12 00:02:19 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=0400=== CONT TestSetClientTLSErrors/missing_key_file401--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.01s)402--- PASS: TestSetClientTLSErrors (0.01s)403 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)404 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)405 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)406 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)4072026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:620314082026/11/12 00:02:19 WARN Garbage collection finished on another replica. Its statistics are not available here4092026/11/12 00:02:19 WARN Rate limiter enabled after throttle name=s3-test rate=54102026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:621064112026/11/12 00:02:19 WARN Rate limiter enabled after throttle name=server-test rate=54122026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:620614132026/11/12 00:02:19 WARN Rate limiter enabled after throttle name=server-test rate=54142026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:620634152026/11/12 00:02:19 WARN Rate limiter backed off name=server-test rate=54162026/11/12 00:02:19 WARN Rate limiter enabled after throttle name=server-test rate=54172026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:620604182026/11/12 00:02:19 WARN Rate limiter backed off name=server-test rate=5419--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)420--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)4212026/11/12 00:02:19 WARN Rate limiter backed off name=server-test rate=5422--- PASS: TestRateLimiterFeedback (0.00s)423 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.01s)424 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)425 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.01s)426 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)4272026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:62031428--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.01s)4292026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:620314302026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:62031431--- PASS: TestStreamPushRequestLine (0.03s)4322026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:620314332026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:620314342026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:620314352026/11/12 00:02:19 http: TLS handshake error from 127.0.0.1:62109: remote error: tls: bad certificate436--- PASS: TestSetClientTLS (0.01s)437 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)438 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)439 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.02s)4402026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:62031441--- PASS: TestRegisterUploadedObjectReusesConnections (0.03s)4422026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:620314432026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:620314442026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:62031445--- PASS: TestScriptTokenCachesUntilRefresh (0.04s)4462026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:62031447--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.05s)4482026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:620314492026/11/12 00:02:19 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:62031450--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.06s)451--- PASS: TestDumpPathWriterError (0.06s)452--- PASS: TestDumpPathSingleFile (0.07s)453--- PASS: TestCaseHackSuffix (0.07s)454--- PASS: TestDumpPathMatchesNix (0.09s)4552026/11/12 00:02:19 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:62034/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:62034/api/uploads/complete4562026/11/12 00:02:19 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4572026/11/12 00:02:19 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:62034/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:62034/api/uploads/complete4582026/11/12 00:02:19 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4592026/11/12 00:02:19 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:62034/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:62034/api/uploads/complete4602026/11/12 00:02:19 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"461--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)462--- PASS: TestUploadMultipart_PartsInParallel (0.62s)463--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "_tribuchetbld8".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /private/var/tb/8/scratch/b3af67106d865ad2/build/postgres3482147964/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /private/var/tb/8/scratch/b3af67106d865ad2/build/postgres3482147964/data -l logfile start492493/private/var/tb/8/scratch/b3af67106d865ad2/build/postgres3482147964:5432 - no response4942026-11-12 00:02:20.974 UTC [5191] LOG: starting PostgreSQL 18.6 on aarch64-apple-darwin25.6.0, compiled by clang version 21.1.8, 64-bit4952026-11-12 00:02:20.974 UTC [5191] LOG: listening on Unix socket "/private/var/tb/8/scratch/b3af67106d865ad2/build/postgres3482147964/.s.PGSQL.5432"4962026-11-12 00:02:20.977 UTC [5198] LOG: database system was shut down at 2026-11-12 00:02:20 UTC4972026-11-12 00:02:20.978 UTC [5191] LOG: database system is ready to accept connections498/private/var/tb/8/scratch/b3af67106d865ad2/build/postgres3482147964:5432 - accepting connections499=== RUN TestAbortSucceeded500=== PAUSE TestAbortSucceeded501=== RUN TestService_AuthMiddleware502=== PAUSE TestService_AuthMiddleware503=== RUN TestService_AuthMiddleware_MTLSProxyHeader504=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader505=== RUN TestService_AuthMiddleware_MTLSBoundSubjects506=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects507=== RUN TestService_ReadAuthMiddleware508=== PAUSE TestService_ReadAuthMiddleware509=== RUN TestService_AuthMiddleware_OIDC510=== PAUSE TestService_AuthMiddleware_OIDC511=== RUN TestService_RequireScope_OIDC512=== PAUSE TestService_RequireScope_OIDC513=== RUN TestService_ReadScope_PublicByDefault514=== PAUSE TestService_ReadScope_PublicByDefault515=== RUN TestCacheConfigHandler516=== PAUSE TestCacheConfigHandler517=== RUN TestCacheStatsHandler518=== PAUSE TestCacheStatsHandler519=== RUN TestClientCADerivations520=== PAUSE TestClientCADerivations521=== RUN TestClientErrorHandling522=== PAUSE TestClientErrorHandling523=== RUN TestClientIntegration524=== PAUSE TestClientIntegration525=== RUN TestClientMultipleUploads526=== PAUSE TestClientMultipleUploads527=== RUN TestClientWithDependencies528=== PAUSE TestClientWithDependencies529=== RUN TestClientSharedPathCommittedMidPush530=== PAUSE TestClientSharedPathCommittedMidPush531=== RUN TestPinProtectsFromGC532=== PAUSE TestPinProtectsFromGC533=== RUN TestClientPushesUseOnePush534=== PAUSE TestClientPushesUseOnePush535=== RUN TestClientFallsBackToClosures536=== PAUSE TestClientFallsBackToClosures537=== RUN TestDeleteClosuresRacingPins538=== PAUSE TestDeleteClosuresRacingPins539=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock540=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock541=== RUN TestResolveDBConnectionString542=== PAUSE TestResolveDBConnectionString543=== RUN TestConnectWaitsForAPeerMigration544=== PAUSE TestConnectWaitsForAPeerMigration545=== RUN TestLeadElectsOneAndHandsOver546=== PAUSE TestLeadElectsOneAndHandsOver547=== RUN TestLeadIncumbentWinsAfterRestart5482026/11/12 00:02:21 INFO lead: acquired remote=192.0.2.1:1234 generation=15492026/11/12 00:02:22 INFO lead: released remote=192.0.2.1:1234 generation=15502026/11/12 00:02:22 INFO lead: acquired remote=192.0.2.1:1234 generation=25512026/11/12 00:02:23 INFO lead: released remote=192.0.2.1:1234 generation=2552--- PASS: TestLeadIncumbentWinsAfterRestart (2.01s)553=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent554=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent555=== RUN TestLeadIncumbentAnnouncesAtOnce556=== PAUSE TestLeadIncumbentAnnouncesAtOnce557=== RUN TestLeadGenerationGrowsWithEveryGrant558=== PAUSE TestLeadGenerationGrowsWithEveryGrant559=== RUN TestLeadEndsOnShutdown560=== PAUSE TestLeadEndsOnShutdown561=== RUN TestLeadEndsWhenItsConnectionHangs5622026/11/12 00:02:23 INFO lead: acquired remote=192.0.2.1:1234 generation=15632026-11-12 00:02:24.785 UTC [5249] FATAL: terminating connection due to administrator command5642026/11/12 00:02:24 INFO lead: acquired remote=192.0.2.1:1234 generation=25652026/11/12 00:02:25 WARN lead: lock connection lost error="timeout: context deadline exceeded"5662026/11/12 00:02:25 INFO lead: released remote=192.0.2.1:1234 generation=15672026/11/12 00:02:26 INFO lead: released remote=192.0.2.1:1234 generation=2568--- PASS: TestLeadEndsWhenItsConnectionHangs (2.97s)569=== RUN TestGCAdvisoryLockBlocksConcurrentRun570=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun571=== RUN TestGCBugBareHashReferences572=== PAUSE TestGCBugBareHashReferences573=== RUN TestGCMetrics574=== PAUSE TestGCMetrics575=== RUN TestPushDedupSurvivesConcurrentGC576=== PAUSE TestPushDedupSurvivesConcurrentGC577=== RUN TestDeduplicatedObjectsRecordedAsPending578=== PAUSE TestDeduplicatedObjectsRecordedAsPending579=== RUN TestGCSweepSkipsPendingObjects580=== PAUSE TestGCSweepSkipsPendingObjects581=== RUN TestTombstonedObjectOfferedWithoutWaiting582=== PAUSE TestTombstonedObjectOfferedWithoutWaiting583=== RUN TestGCSweepDeliversEachKeyOnce584=== PAUSE TestGCSweepDeliversEachKeyOnce585=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection586=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection587=== RUN TestForceGCDuringPushOffersSweptObject588=== PAUSE TestForceGCDuringPushOffersSweptObject589=== RUN TestFailedS3DeleteKeepsTombstone590=== PAUSE TestFailedS3DeleteKeepsTombstone591=== RUN TestSweepSkipsRowsLockedByPush592=== PAUSE TestSweepSkipsRowsLockedByPush593=== RUN TestRegisterCompletedObjectNeedsPendingRow594=== PAUSE TestRegisterCompletedObjectNeedsPendingRow595=== RUN TestGetExistingObjectsLocksRows596=== PAUSE TestGetExistingObjectsLocksRows597=== RUN TestSweepRechecksPendingAfterLocking598=== PAUSE TestSweepRechecksPendingAfterLocking599=== RUN TestPushWaitsForSweepHoldingRow600=== PAUSE TestPushWaitsForSweepHoldingRow601=== RUN TestCommitRacingPendingCleanupKeepsObjects602=== PAUSE TestCommitRacingPendingCleanupKeepsObjects603=== RUN TestGCEndsOnShutdown604=== PAUSE TestGCEndsOnShutdown605=== RUN TestGCTaskStore_StartNew606=== PAUSE TestGCTaskStore_StartNew607=== RUN TestGCTaskStore_DeduplicateSameParams608=== PAUSE TestGCTaskStore_DeduplicateSameParams609=== RUN TestGCTaskStore_ConflictDifferentParams610=== PAUSE TestGCTaskStore_ConflictDifferentParams611=== RUN TestGCTaskStore_GetEmpty612=== PAUSE TestGCTaskStore_GetEmpty613=== RUN TestGCTaskStore_GetReturnsLatest614=== PAUSE TestGCTaskStore_GetReturnsLatest615=== RUN TestGCTaskStore_CompletedAllowsNewTask616=== PAUSE TestGCTaskStore_CompletedAllowsNewTask617=== RUN TestGCTaskStore_PhaseUpdates618=== PAUSE TestGCTaskStore_PhaseUpdates619=== RUN TestGCTaskStore_Fail620=== PAUSE TestGCTaskStore_Fail621=== RUN TestGracefulShutdownDrainsInflight622=== PAUSE TestGracefulShutdownDrainsInflight623=== RUN TestService_healthCheckHandler624=== PAUSE TestService_healthCheckHandler625=== RUN TestService_readinessHandler626=== PAUSE TestService_readinessHandler627=== RUN TestGenerateLandingPage628=== PAUSE TestGenerateLandingPage629=== RUN TestEnsureAbortUploadsRule630=== PAUSE TestEnsureAbortUploadsRule631=== RUN TestCacheConfigHandlerMaxNarSize632=== PAUSE TestCacheConfigHandlerMaxNarSize633=== RUN TestCreatePendingClosureRejectsOversizedNAR634=== PAUSE TestCreatePendingClosureRejectsOversizedNAR635=== RUN TestNARDeduplicationMetadataUploadBug636=== PAUSE TestNARDeduplicationMetadataUploadBug637=== RUN TestMetricsInventory638=== PAUSE TestMetricsInventory639=== RUN TestMetricsCommit640=== PAUSE TestMetricsCommit641=== RUN TestService_NativeMTLS642=== PAUSE TestService_NativeMTLS643=== RUN TestServerTLSConfig644=== PAUSE TestServerTLSConfig645=== RUN TestMultipartCleanup646=== PAUSE TestMultipartCleanup647=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded648=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded649=== RUN TestPendingCleanupUsesOneCutoff650=== PAUSE TestPendingCleanupUsesOneCutoff651=== RUN TestObjectStatsTrigger652=== PAUSE TestObjectStatsTrigger653=== RUN TestObjectStatsUpdatedOncePerCommit654=== PAUSE TestObjectStatsUpdatedOncePerCommit655=== RUN TestOrphanedObjectsGC656=== PAUSE TestOrphanedObjectsGC657=== RUN TestOrphanedObjectsGCStressTest658=== PAUSE TestOrphanedObjectsGCStressTest659=== RUN TestCreatePin_ReservedPins660=== PAUSE TestCreatePin_ReservedPins661=== RUN TestPresentReportsOnlyClosureRoots662=== PAUSE TestPresentReportsOnlyClosureRoots663=== RUN TestPresentRefreshesOnlyStaleClosures664=== PAUSE TestPresentRefreshesOnlyStaleClosures665=== RUN TestPresentNotReportedWhileGCDeletesClosure666=== PAUSE TestPresentNotReportedWhileGCDeletesClosure667=== RUN TestParseSingleRange668=== PAUSE TestParseSingleRange669=== RUN TestProxyHeadersOnlyTrustedOnSocket6702026/11/12 00:02:26 INFO Starting HTTP server address=proxy.sock6712026/11/12 00:02:26 INFO Starting HTTP server address=127.0.0.1:621526722026/11/12 00:02:26 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6732026/11/12 00:02:26 INFO Shutdown signal received, draining in-flight requests timeout=10s674--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.17s)675=== RUN TestIsValidCachePath676=== PAUSE TestIsValidCachePath677=== RUN TestReadProxyNarinfo678=== PAUSE TestReadProxyNarinfo679=== RUN TestReadProxyNarinfoAlreadyDecompressed680=== PAUSE TestReadProxyNarinfoAlreadyDecompressed681=== RUN TestReadProxyContentNegotiation682=== PAUSE TestReadProxyContentNegotiation683=== RUN TestReadProxyNarStreaming684=== PAUSE TestReadProxyNarStreaming685=== RUN TestReadProxy404686=== PAUSE TestReadProxy404687=== RUN TestReadProxyInvalidPath688=== PAUSE TestReadProxyInvalidPath689=== RUN TestReadProxyHead690=== PAUSE TestReadProxyHead691=== RUN TestReadProxyOutlastsServerWriteTimeout692=== PAUSE TestReadProxyOutlastsServerWriteTimeout693=== RUN TestReadProxyConditionalGet694=== PAUSE TestReadProxyConditionalGet695=== RUN TestReadProxyRootRedirectsToIndexHTML696=== PAUSE TestReadProxyRootRedirectsToIndexHTML697=== RUN TestReadProxyDisabled698=== PAUSE TestReadProxyDisabled699=== RUN TestReadRedirectNar700=== PAUSE TestReadRedirectNar701=== RUN TestReadRedirectKeepsNarinfoProxied702=== PAUSE TestReadRedirectKeepsNarinfoProxied703=== RUN TestReadProxyRangeRequest704=== PAUSE TestReadProxyRangeRequest705=== RUN TestReadRedirectUsesPublicS3URL706=== PAUSE TestReadRedirectUsesPublicS3URL707=== RUN TestPush_OverlappingRootsStoreOneRowPerKey708=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey709=== RUN TestPush_CompleteCommitsEveryRoot710=== PAUSE TestPush_CompleteCommitsEveryRoot711=== RUN TestPush_RepeatedRootCommitsOnce712=== PAUSE TestPush_RepeatedRootCommitsOnce713=== RUN TestPush_CompleteWaitsForCleanupOfThePush714=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush715=== RUN TestPush_RootsAreLockedInByteOrder716=== PAUSE TestPush_RootsAreLockedInByteOrder717=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit718=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit719=== RUN TestPush_RejectsBadRequests720=== PAUSE TestPush_RejectsBadRequests721=== RUN TestPush_SignsNarinfosOfItsPendingObjects722=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects723=== RUN TestSharedNarBothUploadsComplete724=== PAUSE TestSharedNarBothUploadsComplete725=== RUN TestCompleteMultipartUpload_ErrorButObjectExists726=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists727=== RUN TestCompletedNarNotReofferedAcrossClosures728=== PAUSE TestCompletedNarNotReofferedAcrossClosures729=== RUN TestPresignedUploadRegisteredBeforeCommit730=== PAUSE TestPresignedUploadRegisteredBeforeCommit731=== RUN TestService_Rustfstest732=== PAUSE TestService_Rustfstest733=== RUN TestParseSize734=== PAUSE TestParseSize735=== RUN TestSkippedUploadsHandler736=== PAUSE TestSkippedUploadsHandler737=== RUN TestSystemdListenerNotActivated738--- PASS: TestSystemdListenerNotActivated (0.00s)739=== RUN TestWatchdogBeatsWhenHealthy740--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)741=== RUN TestWatchdogSkipsWhenUnhealthy7422026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7432026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7442026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7452026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7462026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7472026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7482026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7492026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/12 00:02:26 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"752--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)753=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle754=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle755=== RUN TestProxyWriteTimeout756=== PAUSE TestProxyWriteTimeout757=== RUN TestPendingClosureWriteTimeout758=== PAUSE TestPendingClosureWriteTimeout759=== RUN TestIsValidUploadKey760=== PAUSE TestIsValidUploadKey761=== RUN TestUploadHandlersRejectInvalidKeys762=== PAUSE TestUploadHandlersRejectInvalidKeys763=== RUN TestUploadHandlersRejectOversizedBody764=== PAUSE TestUploadHandlersRejectOversizedBody765=== RUN TestService_cleanupPendingClosuresHandler766=== PAUSE TestService_cleanupPendingClosuresHandler767=== RUN TestService_createPendingClosureHandler768=== PAUSE TestService_createPendingClosureHandler769=== RUN TestService_verifyS3Integrity770=== PAUSE TestService_verifyS3Integrity771=== RUN TestService_verifyS3ReuploadsStaleNarinfo772=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo773=== RUN TestCompleteMultipartUnregistered774=== PAUSE TestCompleteMultipartUnregistered775=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT776=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT777=== CONT TestIsValidUploadKey778=== CONT TestCompleteMultipartUnregistered779=== CONT TestGCTaskStore_Fail780=== CONT TestUploadHandlersRejectInvalidKeys781--- PASS: TestGCTaskStore_Fail (0.00s)782=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info783=== CONT TestGracefulShutdownDrainsInflight784=== CONT TestService_verifyS3ReuploadsStaleNarinfo785=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed786=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed787=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed788=== CONT TestAbortSucceeded789--- PASS: TestAbortSucceeded (0.00s)790=== CONT TestPendingClosureWriteTimeout791=== RUN TestPendingClosureWriteTimeout/empty792=== PAUSE TestPendingClosureWriteTimeout/empty793=== CONT TestService_cleanupPendingClosuresHandler794=== RUN TestPendingClosureWriteTimeout/400_objects795=== PAUSE TestPendingClosureWriteTimeout/400_objects796=== RUN TestPendingClosureWriteTimeout/670k_objects797=== PAUSE TestPendingClosureWriteTimeout/670k_objects798=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline799=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline800=== CONT TestPresentReportsOnlyClosureRoots8012026/11/12 00:02:26 INFO Starting HTTP server address=127.0.0.1:62154802=== CONT TestProxyWriteTimeout803=== RUN TestProxyWriteTimeout/narinfo804=== PAUSE TestProxyWriteTimeout/narinfo805=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle806=== RUN TestProxyWriteTimeout/1_GiB_nar807=== PAUSE TestProxyWriteTimeout/1_GiB_nar808=== RUN TestProxyWriteTimeout/10_GiB_nar809=== PAUSE TestProxyWriteTimeout/10_GiB_nar810=== RUN TestProxyWriteTimeout/unknown_size811=== PAUSE TestProxyWriteTimeout/unknown_size812=== CONT TestService_createPendingClosureHandler813=== CONT TestCreatePin_ReservedPins814=== CONT TestSkippedUploadsHandler815=== CONT TestUploadHandlersRejectOversizedBody816=== CONT TestParseSize817=== CONT TestService_Rustfstest818=== RUN TestIsValidUploadKey/narinfo819=== PAUSE TestIsValidUploadKey/narinfo820=== CONT TestService_verifyS3Integrity821=== CONT TestPresignedUploadRegisteredBeforeCommit822=== CONT TestCompletedNarNotReofferedAcrossClosures823=== CONT TestCompleteMultipartUpload_ErrorButObjectExists824=== CONT TestSharedNarBothUploadsComplete825=== CONT TestPush_SignsNarinfosOfItsPendingObjects826=== CONT TestParseSingleRange827=== CONT TestPresentNotReportedWhileGCDeletesClosure828=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info829=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT830=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed831=== CONT TestPresentRefreshesOnlyStaleClosures832--- PASS: TestParseSize (0.00s)833=== CONT TestOrphanedObjectsGCStressTest834=== RUN TestIsValidUploadKey/nar_zst835=== PAUSE TestIsValidUploadKey/nar_zst836=== RUN TestIsValidUploadKey/nar_xz837=== PAUSE TestIsValidUploadKey/nar_xz838=== RUN TestIsValidUploadKey/nar_plain839=== PAUSE TestIsValidUploadKey/nar_plain840=== RUN TestIsValidUploadKey/listing841=== PAUSE TestIsValidUploadKey/listing842=== RUN TestIsValidUploadKey/build_log843=== PAUSE TestIsValidUploadKey/build_log844=== RUN TestIsValidUploadKey/build_log_home-manager_file845=== PAUSE TestIsValidUploadKey/build_log_home-manager_file846=== RUN TestIsValidUploadKey/build_log_plus_in_name8472026/11/12 00:02:26 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000848=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal849=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal850=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key851=== PAUSE TestIsValidUploadKey/build_log_plus_in_name852=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale853=== RUN TestIsValidUploadKey/build_log_question_mark854=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost855=== RUN TestParseSingleRange/none8562026/11/12 00:02:26 INFO Shutdown signal received, draining in-flight requests timeout=10s857=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key858=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key859=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key860=== CONT TestOrphanedObjectsGC861=== PAUSE TestIsValidUploadKey/build_log_question_mark862=== RUN TestIsValidUploadKey/build_log_equals863=== PAUSE TestIsValidUploadKey/build_log_equals864=== RUN TestIsValidUploadKey/realisation865=== PAUSE TestIsValidUploadKey/realisation866=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale867=== PAUSE TestParseSingleRange/none868=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost869=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed870=== RUN TestIsValidUploadKey/realisation_plus_in_output871=== PAUSE TestIsValidUploadKey/realisation_plus_in_output872=== RUN TestIsValidUploadKey/nix-cache-info873=== PAUSE TestIsValidUploadKey/nix-cache-info874=== RUN TestIsValidUploadKey/index.html875=== PAUSE TestIsValidUploadKey/index.html876=== RUN TestIsValidUploadKey/narinfo_key,_nar_type877=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type878=== RUN TestIsValidUploadKey/nar_key,_narinfo_type879=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type880=== RUN TestIsValidUploadKey/listing_key,_narinfo_type881=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type882=== RUN TestIsValidUploadKey/traversal883=== PAUSE TestIsValidUploadKey/traversal884=== RUN TestIsValidUploadKey/traversal_nar885=== PAUSE TestIsValidUploadKey/traversal_nar886=== RUN TestIsValidUploadKey/absolute887=== PAUSE TestIsValidUploadKey/absolute888=== RUN TestIsValidUploadKey/empty_key889=== PAUSE TestIsValidUploadKey/empty_key890=== RUN TestIsValidUploadKey/unknown_type891=== PAUSE TestIsValidUploadKey/unknown_type892=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh893=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh894=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed895=== CONT TestObjectStatsUpdatedOncePerCommit896=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed897=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed898=== RUN TestParseSingleRange/unknown_unit899=== PAUSE TestParseSingleRange/unknown_unit900=== RUN TestParseSingleRange/multi-range_ignored901=== PAUSE TestParseSingleRange/multi-range_ignored902=== RUN TestParseSingleRange/malformed_no_dash903=== PAUSE TestParseSingleRange/malformed_no_dash904=== RUN TestParseSingleRange/malformed_both_empty905=== PAUSE TestParseSingleRange/malformed_both_empty906=== RUN TestParseSingleRange/malformed_end_before_start907=== PAUSE TestParseSingleRange/malformed_end_before_start908=== RUN TestParseSingleRange/closed909=== PAUSE TestParseSingleRange/closed910=== RUN TestParseSingleRange/open-ended911=== PAUSE TestParseSingleRange/open-ended912=== RUN TestParseSingleRange/end_clamped_to_size913=== PAUSE TestParseSingleRange/end_clamped_to_size914=== RUN TestParseSingleRange/suffix915=== PAUSE TestParseSingleRange/suffix916=== RUN TestParseSingleRange/suffix_exceeds_size917=== PAUSE TestParseSingleRange/suffix_exceeds_size918=== RUN TestParseSingleRange/single_byte919=== PAUSE TestParseSingleRange/single_byte920=== RUN TestParseSingleRange/start_past_EOF921=== PAUSE TestParseSingleRange/start_past_EOF922=== RUN TestParseSingleRange/start_far_past_EOF923=== CONT TestPush_RejectsBadRequests924=== PAUSE TestParseSingleRange/start_far_past_EOF925=== CONT TestIsValidCachePath926=== RUN TestIsValidCachePath/narinfo927=== PAUSE TestIsValidCachePath/narinfo928=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars929=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars930=== RUN TestIsValidCachePath/nar_zst931=== PAUSE TestIsValidCachePath/nar_zst932=== RUN TestIsValidCachePath/nar_xz933=== PAUSE TestIsValidCachePath/nar_xz934=== RUN TestIsValidCachePath/nar_bz2935=== PAUSE TestIsValidCachePath/nar_bz2936=== RUN TestIsValidCachePath/nar_uncompressed937=== PAUSE TestIsValidCachePath/nar_uncompressed938=== RUN TestIsValidCachePath/ls939=== PAUSE TestIsValidCachePath/ls940=== RUN TestIsValidCachePath/log941=== PAUSE TestIsValidCachePath/log942=== RUN TestIsValidCachePath/realisation943=== PAUSE TestIsValidCachePath/realisation944=== RUN TestIsValidCachePath/nix-cache-info945=== PAUSE TestIsValidCachePath/nix-cache-info946=== RUN TestIsValidCachePath/index.html947=== PAUSE TestIsValidCachePath/index.html948=== RUN TestIsValidCachePath/traversal_parent949=== PAUSE TestIsValidCachePath/traversal_parent950=== RUN TestIsValidCachePath/traversal_in_middle951=== PAUSE TestIsValidCachePath/traversal_in_middle952=== RUN TestIsValidCachePath/invalid_char_e953=== PAUSE TestIsValidCachePath/invalid_char_e954=== RUN TestIsValidCachePath/invalid_char_u955=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit956=== PAUSE TestIsValidCachePath/invalid_char_u957=== RUN TestIsValidCachePath/random_path958=== PAUSE TestIsValidCachePath/random_path959=== RUN TestIsValidCachePath/empty960=== PAUSE TestIsValidCachePath/empty961=== RUN TestIsValidCachePath/leading_slash962=== PAUSE TestIsValidCachePath/leading_slash963=== RUN TestIsValidCachePath/wrong_extension964=== PAUSE TestIsValidCachePath/wrong_extension965=== RUN TestIsValidCachePath/short_hash966=== PAUSE TestIsValidCachePath/short_hash967=== CONT TestPush_RootsAreLockedInByteOrder968--- PASS: TestSkippedUploadsHandler (0.04s)969=== CONT TestPush_CompleteWaitsForCleanupOfThePush970=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure971=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure972=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart973=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart974=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts975=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts976=== CONT TestGCTaskStore_PhaseUpdates977--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)978=== CONT TestPush_RepeatedRootCommitsOnce979--- PASS: TestGracefulShutdownDrainsInflight (0.12s)980=== CONT TestPush_CompleteCommitsEveryRoot9812026/11/12 00:02:26 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62157/oidc982--- PASS: TestService_Rustfstest (0.66s)983=== CONT TestObjectStatsTrigger9842026/11/12 00:02:27 INFO Received uploads request method=POST path=/api/pending_closures9852026/11/12 00:02:27 INFO Received cleanup request method=DELETE path=/api/pending_closures9862026/11/12 00:02:27 INFO Aborted multipart uploads count=0 kept=09872026/11/12 00:02:27 INFO Received uploads request method=POST path=/api/pending_closures9882026/11/12 00:02:27 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9892026/11/12 00:02:27 INFO Received cleanup request method=DELETE path=/api/pending_closures9902026/11/12 00:02:27 INFO Aborted multipart uploads count=1 kept=09912026/11/12 00:02:27 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9922026-11-12 00:02:27.973 UTC [5306] ERROR: Closure does not exist: id=19932026-11-12 00:02:27.973 UTC [5306] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE9942026-11-12 00:02:27.973 UTC [5306] STATEMENT: -- name: CommitPendingClosure :one995 SELECT commit_pending_closure($1::bigint)::bigint996 997--- PASS: TestService_cleanupPendingClosuresHandler (1.39s)998=== CONT TestGCTaskStore_CompletedAllowsNewTask999--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1000=== CONT TestPush_OverlappingRootsStoreOneRowPerKey1001--- PASS: TestPresentReportsOnlyClosureRoots (1.54s)1002=== CONT TestGCTaskStore_GetReturnsLatest1003--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1004=== CONT TestReadRedirectUsesPublicS3URL10052026/11/12 00:02:28 INFO Received uploads request method=POST path=/api/pending_closures10062026/11/12 00:02:28 INFO Received uploads request method=POST path=/api/pending_closures10072026/11/12 00:02:28 INFO Received uploads request method=POST path=/api/pending_closures10082026/11/12 00:02:28 INFO Received uploads request method=POST path=/api/pending_closures10092026/11/12 00:02:28 INFO Received uploads request method=POST path=/api/pending_closures10102026/11/12 00:02:29 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10112026/11/12 00:02:29 INFO Received uploads request method=POST path=/api/pending_closures1012--- PASS: TestPresignedUploadRegisteredBeforeCommit (2.49s)1013=== CONT TestReadProxyRangeRequest10142026/11/12 00:02:29 INFO Received uploads request method=POST path=/api/pending_closures10152026/11/12 00:02:29 INFO Received push request method=POST path=/api/pushes10162026/11/12 00:02:29 INFO Received complete push request method=POST path=/api/pushes/1/complete10172026/11/12 00:02:29 INFO Completed push id=1 objects=4 duration=22.03125ms10182026/11/12 00:02:29 INFO Received push request method=POST path=/api/pushes10192026/11/12 00:02:29 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10202026/11/12 00:02:29 INFO Aborted multipart uploads count=0 kept=010212026/11/12 00:02:29 WARN Force mode enabled - objects will be deleted immediately without grace period10222026/11/12 00:02:29 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=010232026/11/12 00:02:29 INFO Vacuumed table table=pending_closures10242026/11/12 00:02:29 INFO Vacuumed table table=pending_objects10252026/11/12 00:02:29 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=YWNjMGNlNjAtMThiNC00YWU1LTg2ODYtOTRjOGFkYzhmNGE4LjYxYTM0NDM0LTNjZmItNGZlMy1hZjExLTRjNWMwMTkyNzUyZngxNzk0NDQxNzQ4MzkzNjMwMDAw parts=1010262026/11/12 00:02:29 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10272026/11/12 00:02:29 INFO Vacuumed table table=multipart_uploads10282026/11/12 00:02:29 INFO Completed upload id=1 objects=2 duration=24.503917ms10292026/11/12 00:02:29 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010302026/11/12 00:02:29 INFO Received uploads request method=POST path=/api/pending_closures10312026/11/12 00:02:29 INFO Starting cleanup of old closures method=DELETE path=/api/closures10322026/11/12 00:02:29 INFO Vacuumed table table=closures10332026/11/12 00:02:30 INFO Aborted multipart uploads count=0 kept=010342026/11/12 00:02:30 INFO Vacuumed table table=objects10352026/11/12 00:02:30 INFO Received complete push request method=POST path=/api/pushes/2/complete10362026/11/12 00:02:30 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010372026/11/12 00:02:30 INFO Completed push id=2 objects=4 duration=7.488542ms1038--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (3.42s)1039=== CONT TestGCTaskStore_GetEmpty1040--- PASS: TestGCTaskStore_GetEmpty (0.00s)1041=== CONT TestReadRedirectKeepsNarinfoProxied10422026/11/12 00:02:30 INFO Vacuumed table table=pending_closures10432026/11/12 00:02:30 INFO Vacuumed table table=pending_objects1044--- PASS: TestObjectStatsUpdatedOncePerCommit (3.46s)1045=== CONT TestReadRedirectNar10462026/11/12 00:02:30 INFO Vacuumed table table=multipart_uploads10472026/11/12 00:02:30 INFO Vacuumed table table=closures10482026/11/12 00:02:30 INFO Vacuumed table table=objects10492026/11/12 00:02:30 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010502026/11/12 00:02:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete1051--- PASS: TestService_createPendingClosureHandler (3.61s)1052=== CONT TestPendingCleanupUsesOneCutoff1053--- PASS: TestPresentRefreshesOnlyStaleClosures (3.65s)1054=== CONT TestReadProxyDisabled10552026/11/12 00:02:30 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=YWNjMGNlNjAtMThiNC00YWU1LTg2ODYtOTRjOGFkYzhmNGE4LjdjMjlkNzRmLTI3OTUtNGYzNS1iM2FkLTg2NmUzODcwNDMxMngxNzk0NDQxNzQ4NzQ4OTMwMDAw parts=1010562026/11/12 00:02:30 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10572026/11/12 00:02:30 INFO Completed upload id=1 objects=2 duration=18.417458ms10582026/11/12 00:02:30 INFO Received uploads request method=POST path=/api/pending_closures10592026/11/12 00:02:30 INFO Received uploads request method=POST path=/api/pending_closures10602026/11/12 00:02:30 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10612026/11/12 00:02:30 WARN Found objects in DB but missing from S3, will re-upload count=11062--- PASS: TestService_verifyS3Integrity (3.75s)1063=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded10642026/11/12 00:02:30 INFO Received push request method=POST path=/api/pushes10652026/11/12 00:02:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10662026/11/12 00:02:30 INFO Signed narinfos id=1 count=110672026/11/12 00:02:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10682026/11/12 00:02:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10692026/11/12 00:02:30 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"10702026/11/12 00:02:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10712026/11/12 00:02:30 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"10722026/11/12 00:02:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10732026/11/12 00:02:30 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=110742026/11/12 00:02:30 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign10752026/11/12 00:02:30 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1076--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (3.90s)1077=== CONT TestReadProxyRootRedirectsToIndexHTML1078=== RUN TestPush_RejectsBadRequests/no_roots1079=== PAUSE TestPush_RejectsBadRequests/no_roots1080=== RUN TestPush_RejectsBadRequests/no_objects1081=== PAUSE TestPush_RejectsBadRequests/no_objects1082=== RUN TestPush_RejectsBadRequests/bad_root1083=== PAUSE TestPush_RejectsBadRequests/bad_root1084=== RUN TestPush_RejectsBadRequests/root_not_in_objects1085=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1086=== CONT TestMultipartCleanup10872026/11/12 00:02:30 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10882026/11/12 00:02:30 INFO Received push request method=POST path=/api/pushes10892026/11/12 00:02:30 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=YWNjMGNlNjAtMThiNC00YWU1LTg2ODYtOTRjOGFkYzhmNGE4Ljk4ZWQ4M2MzLTA2NzctNGQ5ZS1hNzZiLTg2ZjUyNjljYzkzMXgxNzk0NDQxNzQ5MzA1ODEyMDAw parts=1210902026/11/12 00:02:30 INFO Received uploads request method=POST path=/api/pending_closures1091--- PASS: TestCompletedNarNotReofferedAcrossClosures (4.34s)1092=== CONT TestGCTaskStore_ConflictDifferentParams1093--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1094=== CONT TestReadProxyConditionalGet10952026-11-12 00:02:31.003 UTC [5359] ERROR: could not obtain lock on row in relation "closures"10962026-11-12 00:02:31.003 UTC [5359] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10972026-11-12 00:02:31.004 UTC [5359] ERROR: could not obtain lock on row in relation "closures"10982026-11-12 00:02:31.004 UTC [5359] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT10992026-11-12 00:02:31.005 UTC [5359] ERROR: could not obtain lock on row in relation "closures"11002026-11-12 00:02:31.005 UTC [5359] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11012026-11-12 00:02:31.009 UTC [5359] ERROR: could not obtain lock on row in relation "closures"11022026-11-12 00:02:31.009 UTC [5359] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT11032026-11-12 00:02:31.009 UTC [5359] ERROR: could not obtain lock on row in relation "closures"11042026-11-12 00:02:31.009 UTC [5359] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1105--- PASS: TestPush_RootsAreLockedInByteOrder (4.38s)1106=== CONT TestGCTaskStore_DeduplicateSameParams1107--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1108=== CONT TestReadProxyOutlastsServerWriteTimeout11092026/11/12 00:02:31 INFO Received push request method=POST path=/api/pushes11102026/11/12 00:02:31 INFO Received complete push request method=POST path=/api/pushes/1/complete11112026/11/12 00:02:31 INFO Completed push id=1 objects=4 duration=28.8395ms1112--- PASS: TestPush_RepeatedRootCommitsOnce (4.54s)1113=== CONT TestGCTaskStore_StartNew1114--- PASS: TestGCTaskStore_StartNew (0.00s)1115=== CONT TestReadProxyHead11162026/11/12 00:02:31 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11172026/11/12 00:02:31 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst1118--- PASS: TestCompleteMultipartUnregistered (4.93s)1119=== CONT TestServerTLSConfig1120=== RUN TestServerTLSConfig/no_client_CA1121=== PAUSE TestServerTLSConfig/no_client_CA1122=== RUN TestServerTLSConfig/missing_CA_file1123=== PAUSE TestServerTLSConfig/missing_CA_file1124=== RUN TestServerTLSConfig/not_a_PEM_file1125=== PAUSE TestServerTLSConfig/not_a_PEM_file1126=== CONT TestGCEndsOnShutdown1127=== RUN TestGCEndsOnShutdown/before_the_run1128=== PAUSE TestGCEndsOnShutdown/before_the_run1129=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1130=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1131=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1132=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1133=== CONT TestReadProxyInvalidPath11342026/11/12 00:02:31 INFO Received push request method=POST path=/api/pushes11352026/11/12 00:02:32 INFO Received uploads request method=POST path=/api/pending_closures11362026/11/12 00:02:32 INFO Received uploads request method=POST path=/api/pending_closures11372026-11-12 00:02:32.263 UTC [5373] ERROR: Push does not exist: id=111382026-11-12 00:02:32.263 UTC [5373] CONTEXT: PL/pgSQL function commit_push(bigint) line 11 at RAISE11392026-11-12 00:02:32.263 UTC [5373] STATEMENT: SELECT commit_push($1)1140--- PASS: TestPush_CompleteWaitsForCleanupOfThePush (5.63s)1141=== CONT TestService_NativeMTLS11422026/11/12 00:02:32 INFO Received uploads request method=POST path=/api/pending_closures1143--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (5.92s)1144=== CONT TestReadProxy40411452026/11/12 00:02:32 INFO Received push request method=POST path=/api/pushes11462026/11/12 00:02:32 INFO Received complete push request method=POST path=/api/pushes/1/complete11472026/11/12 00:02:32 INFO Completed push id=1 objects=6 duration=56.874167ms1148--- PASS: TestPush_CompleteCommitsEveryRoot (6.21s)1149=== CONT TestMetricsCommit11502026/11/12 00:02:33 WARN Rate limiter enabled after throttle name=s3-test rate=511512026/11/12 00:02:33 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1152=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1153 throttle_test.go:213: Proxy stats: total=13, throttled=10, completeMultipart=101154 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001155--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (6.46s)1156=== CONT TestReadProxyNarStreaming11572026/11/12 00:02:33 INFO Received uploads request method=POST path=/api/pending_closures11582026/11/12 00:02:33 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11592026/11/12 00:02:33 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YWNjMGNlNjAtMThiNC00YWU1LTg2ODYtOTRjOGFkYzhmNGE4LmNhNDUzMTJkLTlhNzgtNDY1YS1iZDEzLWFiZDA5MWNkYzJlMngxNzk0NDQxNzUzMTgwNzMxMDAw11602026/11/12 00:02:33 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YWNjMGNlNjAtMThiNC00YWU1LTg2ODYtOTRjOGFkYzhmNGE4LmNhNDUzMTJkLTlhNzgtNDY1YS1iZDEzLWFiZDA5MWNkYzJlMngxNzk0NDQxNzUzMTgwNzMxMDAw parts=11161--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (7.02s)1162=== CONT TestMetricsInventory11632026/11/12 00:02:33 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11642026/11/12 00:02:33 WARN Refused reserved pin name=worker-x86_64-linux11652026/11/12 00:02:33 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux11662026/11/12 00:02:33 INFO Received create pin request method=POST path=/api/pins/my-app11672026/11/12 00:02:33 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux1168--- PASS: TestCreatePin_ReservedPins (7.34s)1169=== CONT TestReadProxyContentNegotiation1170--- PASS: TestObjectStatsTrigger (7.08s)1171=== CONT TestCommitRacingPendingCleanupKeepsObjects11722026/11/12 00:02:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11732026/11/12 00:02:34 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=YWNjMGNlNjAtMThiNC00YWU1LTg2ODYtOTRjOGFkYzhmNGE4LjNmYmZmNGM4LWMyZWQtNGQzOS04M2Q1LThhYTQ2NDM3YjhkMXgxNzk0NDQxNzUyMTgzNjg5MDAw parts=1211742026/11/12 00:02:34 INFO Received push request method=POST path=/api/pushes1175--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (6.86s)1176=== CONT TestNARDeduplicationMetadataUploadBug1177--- PASS: TestReadRedirectUsesPublicS3URL (7.07s)1178=== CONT TestReadProxyNarinfoAlreadyDecompressed1179--- PASS: TestReadProxyRangeRequest (6.67s)1180=== CONT TestPushWaitsForSweepHoldingRow1181=== NAME TestOrphanedObjectsGC1182 orphaned_objects_gc_test.go:287: GC Test Summary:1183 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1184 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1185 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1186 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1187 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects1188--- PASS: TestOrphanedObjectsGC (9.58s)1189=== CONT TestCreatePendingClosureRejectsOversizedNAR11902026/11/12 00:02:36 INFO Received uploads request method=POST path=/api/pending_closures1191--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1192=== CONT TestReadProxyNarinfo1193--- PASS: TestReadRedirectNar (6.30s)1194=== CONT TestSweepRechecksPendingAfterLocking1195--- PASS: TestReadRedirectKeepsNarinfoProxied (6.83s)1196=== CONT TestCacheConfigHandlerMaxNarSize1197--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1198=== CONT TestGetExistingObjectsLocksRows11992026/11/12 00:02:37 INFO Received uploads request method=POST path=/api/pending_closures12002026/11/12 00:02:37 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12012026/11/12 00:02:37 INFO Received uploads request method=POST path=/api/pending_closures12022026/11/12 00:02:37 INFO Received cleanup request method=DELETE path=/api/pending_closures12032026/11/12 00:02:37 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=YWNjMGNlNjAtMThiNC00YWU1LTg2ODYtOTRjOGFkYzhmNGE4LjMwYTBhZjZmLTM1NDctNDZmMS1iMmU5LTJmYjI0YWI4ODFjN3gxNzk0NDQxNzUyMjQxNTI0MDAw parts=121204--- PASS: TestSharedNarBothUploadsComplete (10.90s)1205=== CONT TestEnsureAbortUploadsRule1206--- PASS: TestReadProxyDisabled (7.38s)1207=== CONT TestGenerateLandingPage1208--- PASS: TestGenerateLandingPage (0.00s)1209=== CONT TestService_readinessHandler12102026/11/12 00:02:37 INFO Received uploads request method=POST path=/api/pending_closures1211--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (7.75s)1212=== CONT TestService_healthCheckHandler1213--- PASS: TestReadProxyRootRedirectsToIndexHTML (7.83s)1214=== CONT TestRegisterCompletedObjectNeedsPendingRow12152026/11/12 00:02:38 INFO Received uploads request method=POST path=/api/pending_closures12162026/11/12 00:02:38 INFO Received cleanup request method=DELETE path=/api/pending_closures12172026/11/12 00:02:38 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket38/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""12182026/11/12 00:02:38 INFO Aborted multipart uploads count=0 kept=112192026/11/12 00:02:38 INFO Received cleanup request method=DELETE path=/api/pending_closures12202026/11/12 00:02:38 INFO Aborted multipart uploads count=1 kept=01221--- PASS: TestMultipartCleanup (8.30s)1222=== CONT TestClientCADerivations1223--- PASS: TestReadProxyConditionalGet (8.09s)1224=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock1225--- PASS: TestReadProxyHead (8.56s)1226=== CONT TestDeleteClosuresRacingPins1227--- PASS: TestReadProxyInvalidPath (8.44s)1228=== CONT TestSweepSkipsRowsLockedByPush12292026/11/12 00:02:40 WARN mTLS auth: subject not in bound subjects subject="CN=reader"12302026/11/12 00:02:40 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1231--- PASS: TestService_NativeMTLS (8.04s)1232=== CONT TestClientFallsBackToClosures12332026/11/12 00:02:41 INFO Received push request method=POST path=/api/pushes12342026/11/12 00:02:41 INFO Received complete push request method=POST path=/api/pushes/1/complete12352026/11/12 00:02:41 INFO Completed push id=1 objects=4 duration=94.757958ms1236--- PASS: TestReadProxy404 (8.75s)1237=== CONT TestFailedS3DeleteKeepsTombstone12382026/11/12 00:02:41 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12392026/11/12 00:02:41 INFO Completed upload id=2 objects=2 duration=3.519375ms1240--- PASS: TestMetricsCommit (8.39s)1241=== CONT TestClientPushesUseOnePush1242--- PASS: TestReadProxyNarStreaming (8.41s)1243=== CONT TestForceGCDuringPushOffersSweptObject1244=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows1245=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows1246=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check1247=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check1248=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection1249--- PASS: TestMetricsInventory (8.15s)1250=== CONT TestGCSweepDeliversEachKeyOnce1251--- PASS: TestReadProxyContentNegotiation (8.30s)1252=== CONT TestCacheStatsHandler12532026/11/12 00:02:42 INFO Aborted multipart uploads count=0 kept=012542026/11/12 00:02:42 WARN Force mode enabled - objects will be deleted immediately without grace period12552026/11/12 00:02:42 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=012562026/11/12 00:02:42 INFO Vacuumed table table=pending_closures12572026/11/12 00:02:42 INFO Vacuumed table table=pending_objects12582026/11/12 00:02:42 INFO Vacuumed table table=multipart_uploads12592026/11/12 00:02:42 INFO Vacuumed table table=closures12602026/11/12 00:02:42 INFO Vacuumed table table=objects1261--- PASS: TestCommitRacingPendingCleanupKeepsObjects (8.40s)1262=== CONT TestTombstonedObjectOfferedWithoutWaiting1263=== NAME TestNARDeduplicationMetadataUploadBug1264 metadata_upload_test.go:48: First store path: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestNARDeduplicationMetadataUploadBug3679874575/001/store/cxdl8jy1q5jsq1ml966mxf4gin92x6ss-file1.txt1265--- PASS: TestReadProxyNarinfoAlreadyDecompressed (7.79s)1266=== CONT TestCacheConfigHandler1267=== RUN TestCacheConfigHandler/full_config,_no_issuer1268=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1269=== RUN TestCacheConfigHandler/no_cache_url_configured1270=== PAUSE TestCacheConfigHandler/no_cache_url_configured1271=== RUN TestCacheConfigHandler/no_signing_keys1272=== PAUSE TestCacheConfigHandler/no_signing_keys1273=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1274=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1275=== CONT TestGCSweepSkipsPendingObjects12762026/11/12 00:02:43 INFO Received push request method=POST path=/api/pushes12772026/11/12 00:02:43 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12782026/11/12 00:02:43 INFO Uploading cxdl8jy1q5jsq1ml966mxf4gin92x6ss-file1.txt (160B)12792026/11/12 00:02:43 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"12802026/11/12 00:02:43 WARN Failed to register uploaded object key=cxdl8jy1q5jsq1ml966mxf4gin92x6ss.ls error="server returned 404: 404 page not found\n"12812026/11/12 00:02:43 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12822026/11/12 00:02:43 INFO Signed narinfos id=1 count=112832026/11/12 00:02:43 INFO Uploading 1 narinfos12842026/11/12 00:02:43 INFO Received complete push request method=POST path=/api/pushes/1/complete12852026/11/12 00:02:43 WARN Failed to register uploaded object key=cxdl8jy1q5jsq1ml966mxf4gin92x6ss.narinfo error="server returned 404: 404 page not found\n"12862026/11/12 00:02:43 INFO Completed push id=1 objects=3 duration=49.982709ms12872026/11/12 00:02:43 INFO Upload complete. (262ms)1288=== NAME TestNARDeduplicationMetadataUploadBug1289 metadata_upload_test.go:54: Retrieved narinfo from S3:1290 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestNARDeduplicationMetadataUploadBug3679874575/001/store/cxdl8jy1q5jsq1ml966mxf4gin92x6ss-file1.txt1291 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1292 Compression: zstd1293 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1294 NarSize: 1601295 References: 1296 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1297 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1298 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1299 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}13002026/11/12 00:02:43 INFO Received uploads request method=POST path=/api/pending_closures1301 metadata_upload_test.go:64: Second store path (same content): /private/var/tb/8/scratch/b3af67106d865ad2/build/TestNARDeduplicationMetadataUploadBug3679874575/001/store/2myf4lv75qvzq9s73rfx9jhfmv6iadb2-file2.txt13022026/11/12 00:02:43 INFO Received push request method=POST path=/api/pushes13032026/11/12 00:02:43 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)13042026/11/12 00:02:43 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13052026/11/12 00:02:43 WARN Failed to register uploaded object key=2myf4lv75qvzq9s73rfx9jhfmv6iadb2.ls error="server returned 404: 404 page not found\n"13062026/11/12 00:02:43 INFO Aborted multipart uploads count=1 kept=013072026/11/12 00:02:43 INFO Signed narinfos id=2 count=113082026/11/12 00:02:43 INFO Uploading 1 narinfos13092026/11/12 00:02:43 INFO Received cleanup request method=DELETE path=/api/pending_closures13102026/11/12 00:02:43 INFO Aborted multipart uploads count=1 kept=01311--- PASS: TestPendingCleanupUsesOneCutoff (13.36s)1312=== CONT TestService_ReadScope_PublicByDefault13132026/11/12 00:02:43 INFO Received complete push request method=POST path=/api/pushes/2/complete13142026/11/12 00:02:43 WARN Failed to register uploaded object key=2myf4lv75qvzq9s73rfx9jhfmv6iadb2.narinfo error="server returned 404: 404 page not found\n"13152026/11/12 00:02:43 INFO Completed push id=2 objects=3 duration=1.03175ms13162026/11/12 00:02:43 INFO Upload complete. (177ms)1317=== NAME TestNARDeduplicationMetadataUploadBug1318 metadata_upload_test.go:76: Retrieved narinfo from S3:1319 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestNARDeduplicationMetadataUploadBug3679874575/001/store/2myf4lv75qvzq9s73rfx9jhfmv6iadb2-file2.txt1320 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1321 Compression: zstd1322 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1323 NarSize: 1601324 References: 1325 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1326 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1327 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1328 {"version":1,"root":{"type":"regular","size":44}}13292026/11/12 00:02:43 INFO Aborted multipart uploads count=0 kept=01330--- PASS: TestReadProxyOutlastsServerWriteTimeout (12.61s)1331=== CONT TestService_RequireScope_OIDC13322026/11/12 00:02:43 WARN Force mode enabled - objects will be deleted immediately without grace period13332026/11/12 00:02:43 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62280/oidc13342026/11/12 00:02:43 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01335--- PASS: TestNARDeduplicationMetadataUploadBug (8.85s)1336=== CONT TestDeduplicatedObjectsRecordedAsPending13372026/11/12 00:02:43 INFO Vacuumed table table=pending_closures13382026/11/12 00:02:43 INFO Vacuumed table table=pending_objects13392026/11/12 00:02:43 INFO Vacuumed table table=multipart_uploads13402026/11/12 00:02:43 INFO Vacuumed table table=closures13412026/11/12 00:02:43 INFO Vacuumed table table=objects1342--- PASS: TestSweepRechecksPendingAfterLocking (7.32s)1343=== CONT TestPushDedupSurvivesConcurrentGC1344--- PASS: TestPushWaitsForSweepHoldingRow (8.00s)1345=== CONT TestGCMetrics13462026/11/12 00:02:43 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1347--- PASS: TestReadProxyNarinfo (7.73s)1348=== CONT TestGCBugBareHashReferences1349--- PASS: TestGetExistingObjectsLocksRows (7.28s)1350=== CONT TestGCAdvisoryLockBlocksConcurrentRun13512026/11/12 00:02:44 WARN readiness check failed error="closed pool"1352--- PASS: TestService_readinessHandler (6.88s)1353=== CONT TestLeadEndsOnShutdown1354=== NAME TestOrphanedObjectsGCStressTest1355 orphaned_objects_gc_test.go:422: Created 10 active closures, 5 to-delete closures, 20 orphaned chains1356 orphaned_objects_gc_test.go:440: Marked 210 objects for deletion13572026/11/12 00:02:45 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket57 days=21358--- PASS: TestEnsureAbortUploadsRule (7.52s)1359=== CONT TestPinProtectsFromGC1360--- PASS: TestService_healthCheckHandler (7.11s)1361=== CONT TestLeadGenerationGrowsWithEveryGrant1362--- PASS: TestRegisterCompletedObjectNeedsPendingRow (7.26s)1363=== CONT TestClientSharedPathCommittedMidPush1364=== NAME TestClientCADerivations1365 client_ca_test.go:136: Built CA derivation: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientCADerivations606987216/001/store/v686mjkns76a1w6bsq2r0vsa84hy8wrb-ca-test1366 client_ca_test.go:139: Found 1 dependencies (including self)13672026/11/12 00:02:47 INFO Received push request method=POST path=/api/pushes13682026/11/12 00:02:47 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13692026/11/12 00:02:47 INFO Uploading v686mjkns76a1w6bsq2r0vsa84hy8wrb-ca-test (144B)13702026/11/12 00:02:47 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"13712026/11/12 00:02:47 WARN Failed to register uploaded object key=v686mjkns76a1w6bsq2r0vsa84hy8wrb.ls error="server returned 404: 404 page not found\n"13722026/11/12 00:02:47 WARN Failed to register uploaded object key=log/vgpvlldjp4y7bzifmn9q7g8x7yzf6lg4-ca-test.drv error="server returned 404: 404 page not found\n"13732026/11/12 00:02:47 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13742026/11/12 00:02:47 INFO Signed narinfos id=1 count=113752026/11/12 00:02:47 INFO Uploading 1 narinfos13762026/11/12 00:02:47 INFO Received complete push request method=POST path=/api/pushes/1/complete13772026/11/12 00:02:47 WARN Failed to register uploaded object key=v686mjkns76a1w6bsq2r0vsa84hy8wrb.narinfo error="server returned 404: 404 page not found\n"13782026/11/12 00:02:47 INFO Aborted multipart uploads count=0 kept=013792026/11/12 00:02:47 WARN Force mode enabled - objects will be deleted immediately without grace period13802026/11/12 00:02:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=013812026/11/12 00:02:47 INFO Vacuumed table table=pending_closures13822026/11/12 00:02:47 INFO Completed push id=1 objects=4 duration=60.420417ms13832026/11/12 00:02:47 INFO Vacuumed table table=pending_objects13842026/11/12 00:02:47 INFO Upload complete. (395ms)13852026/11/12 00:02:47 INFO Vacuumed table table=multipart_uploads13862026/11/12 00:02:47 INFO Vacuumed table table=closures1387 client_ca_test.go:180: Narinfo contains CA field: StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientCADerivations606987216/001/store/v686mjkns76a1w6bsq2r0vsa84hy8wrb-ca-test1388 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1389 Compression: zstd1390 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1391 NarSize: 1441392 References: 1393 Deriver: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientCADerivations606987216/001/store/vgpvlldjp4y7bzifmn9q7g8x7yzf6lg4-ca-test.drv1394 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1395 client_ca_test.go:185: Checking for realisation files in S3...1396 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1397 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache13982026/11/12 00:02:47 INFO Vacuumed table table=objects13992026/11/12 00:02:47 INFO Aborted multipart uploads count=0 kept=014002026/11/12 00:02:47 WARN Force mode enabled - objects will be deleted immediately without grace period1401 client_ca_test.go:258: nix copy output: error: binary cache 's3://bucket60?endpoint=http://localhost:62138&region=eu-west-1' is for Nix stores with prefix '/nix/store', not '/private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientCADerivations606987216/001/store'1402 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11403--- PASS: TestClientCADerivations (8.82s)1404=== CONT TestLeadIncumbentAnnouncesAtOnce14052026/11/12 00:02:47 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=014062026/11/12 00:02:47 INFO Vacuumed table table=pending_closures14072026/11/12 00:02:47 INFO Vacuumed table table=pending_objects14082026/11/12 00:02:47 INFO Vacuumed table table=multipart_uploads14092026/11/12 00:02:47 INFO Vacuumed table table=closures14102026/11/12 00:02:47 INFO Vacuumed table table=objects1411--- PASS: TestSweepSkipsRowsLockedByPush (7.90s)1412=== CONT TestClientWithDependencies14132026/11/12 00:02:48 INFO Received uploads request method=POST path=/api/pending_closures1414--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (9.04s)1415=== CONT TestService_AuthMiddleware_OIDC14162026/11/12 00:02:48 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62305/oidc14172026/11/12 00:02:48 INFO Received uploads request method=POST path=/api/pending_closures14182026/11/12 00:02:48 INFO Aborted multipart uploads count=0 kept=014192026/11/12 00:02:48 WARN Force mode enabled - objects will be deleted immediately without grace period14202026/11/12 00:02:48 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."14212026/11/12 00:02:48 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."1422--- PASS: TestFailedS3DeleteKeepsTombstone (7.05s)1423=== CONT TestClientMultipleUploads14242026/11/12 00:02:48 INFO Received uploads request method=POST path=/api/pending_closures14252026/11/12 00:02:48 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)14262026/11/12 00:02:48 INFO Uploading dlx8zp863mk55ikbilx1nq1i4hxgf2wx-b (256B)14272026/11/12 00:02:48 INFO Uploading axiafmv7bgqmvw2i657f9najc6gx14gv-shared-dep (136B)14282026/11/12 00:02:48 WARN Failed to register uploaded object key=dlx8zp863mk55ikbilx1nq1i4hxgf2wx.ls error="server returned 404: 404 page not found\n"14292026/11/12 00:02:48 WARN Failed to register uploaded object key=i5655wjjv0vkdxdk4qjacgcdqb48gqqi.ls error="server returned 404: 404 page not found\n"14302026/11/12 00:02:48 WARN Failed to register uploaded object key=axiafmv7bgqmvw2i657f9najc6gx14gv.ls error="server returned 404: 404 page not found\n"14312026/11/12 00:02:48 WARN Failed to register uploaded object key=nar/1xzaq48f8gdfnyiibznss3szwk4wlshqwvzwka7ni5ja22qpli3x.nar.zst error="server returned 404: 404 page not found\n"14322026/11/12 00:02:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign14332026/11/12 00:02:48 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14342026/11/12 00:02:48 INFO Signed narinfos id=1 count=214352026/11/12 00:02:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign14362026/11/12 00:02:48 INFO Signed narinfos id=2 count=214372026/11/12 00:02:48 INFO Uploading 4 narinfos14382026/11/12 00:02:48 WARN Failed to register uploaded object key=dlx8zp863mk55ikbilx1nq1i4hxgf2wx.narinfo error="server returned 404: 404 page not found\n"14392026/11/12 00:02:48 WARN Failed to register uploaded object key=axiafmv7bgqmvw2i657f9najc6gx14gv.narinfo error="server returned 404: 404 page not found\n"14402026/11/12 00:02:48 WARN Failed to register uploaded object key=i5655wjjv0vkdxdk4qjacgcdqb48gqqi.narinfo error="server returned 404: 404 page not found\n"14412026/11/12 00:02:48 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete14422026/11/12 00:02:48 WARN Failed to register uploaded object key=axiafmv7bgqmvw2i657f9najc6gx14gv.narinfo error="server returned 404: 404 page not found\n"14432026/11/12 00:02:48 INFO Completed upload id=1 objects=6 duration=71.821375ms14442026/11/12 00:02:48 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete14452026/11/12 00:02:48 INFO Completed upload id=2 objects=6 duration=522.041µs14462026/11/12 00:02:48 INFO Upload complete. (458ms)14472026/11/12 00:02:48 INFO Received uploads request method=POST path=/api/pending_closures1448=== NAME TestClientFallsBackToClosures1449 client_pushes_test.go:112: Retrieved narinfo from S3:1450 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientFallsBackToClosures3957052357/001/store/axiafmv7bgqmvw2i657f9najc6gx14gv-shared-dep1451 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1452 Compression: zstd1453 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821454 NarSize: 1361455 References: 1456 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1457 client_pushes_test.go:112: Retrieved narinfo from S3:1458 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientFallsBackToClosures3957052357/001/store/i5655wjjv0vkdxdk4qjacgcdqb48gqqi-a1459 URL: nar/1xzaq48f8gdfnyiibznss3szwk4wlshqwvzwka7ni5ja22qpli3x.nar.zst1460 Compression: zstd1461 NarHash: sha256:1xzaq48f8gdfnyiibznss3szwk4wlshqwvzwka7ni5ja22qpli3x1462 NarSize: 2561463 References: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientFallsBackToClosures3957052357/001/store/axiafmv7bgqmvw2i657f9najc6gx14gv-shared-dep1464 CA: text:sha256:09s5w6nz3gj9mm9vxs98r51z1gf80wkiqs0id6yx4pr34k5dbk451465 client_pushes_test.go:112: Retrieved narinfo from S3:1466 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientFallsBackToClosures3957052357/001/store/dlx8zp863mk55ikbilx1nq1i4hxgf2wx-b1467 URL: nar/1xzaq48f8gdfnyiibznss3szwk4wlshqwvzwka7ni5ja22qpli3x.nar.zst1468 Compression: zstd1469 NarHash: sha256:1xzaq48f8gdfnyiibznss3szwk4wlshqwvzwka7ni5ja22qpli3x1470 NarSize: 2561471 References: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientFallsBackToClosures3957052357/001/store/axiafmv7bgqmvw2i657f9najc6gx14gv-shared-dep1472 CA: text:sha256:09s5w6nz3gj9mm9vxs98r51z1gf80wkiqs0id6yx4pr34k5dbk451473--- PASS: TestClientFallsBackToClosures (8.48s)1474=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent1475--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (7.48s)1476=== CONT TestService_ReadAuthMiddleware14772026/11/12 00:02:48 INFO Received push request method=POST path=/api/pushes14782026/11/12 00:02:49 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)14792026/11/12 00:02:49 INFO Uploading cxrjzyxxag6hhkm64cj1j4jwql7g0lck-a (256B)14802026/11/12 00:02:49 INFO Uploading 407ij47i4dxkfj7m42rbmjdwn21jqrbw-shared-dep (136B)14812026/11/12 00:02:49 WARN Failed to register uploaded object key=cxrjzyxxag6hhkm64cj1j4jwql7g0lck.ls error="server returned 404: 404 page not found\n"14822026/11/12 00:02:49 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"14832026/11/12 00:02:49 WARN Failed to register uploaded object key=55rbljrxbq584444p2mim625bwf9h9gk.ls error="server returned 404: 404 page not found\n"14842026/11/12 00:02:49 WARN Failed to register uploaded object key=407ij47i4dxkfj7m42rbmjdwn21jqrbw.ls error="server returned 404: 404 page not found\n"14852026/11/12 00:02:49 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14862026/11/12 00:02:49 WARN Failed to register uploaded object key=nar/0a6bnxc5ivrip83nd33zaacw8yan9dva44h8q0j1ps9dyd5wfpx0.nar.zst error="server returned 404: 404 page not found\n"14872026/11/12 00:02:49 INFO Signed narinfos id=1 count=314882026/11/12 00:02:49 INFO Uploading 3 narinfos14892026/11/12 00:02:49 INFO Aborted multipart uploads count=0 kept=014902026/11/12 00:02:49 WARN Failed to register uploaded object key=cxrjzyxxag6hhkm64cj1j4jwql7g0lck.narinfo error="server returned 404: 404 page not found\n"14912026/11/12 00:02:49 WARN Force mode enabled - objects will be deleted immediately without grace period14922026/11/12 00:02:49 WARN Failed to register uploaded object key=55rbljrxbq584444p2mim625bwf9h9gk.narinfo error="server returned 404: 404 page not found\n"14932026/11/12 00:02:49 INFO Received complete push request method=POST path=/api/pushes/1/complete14942026/11/12 00:02:49 WARN Failed to register uploaded object key=407ij47i4dxkfj7m42rbmjdwn21jqrbw.narinfo error="server returned 404: 404 page not found\n"14952026/11/12 00:02:49 INFO Completed push id=1 objects=8 duration=54.639542ms14962026/11/12 00:02:49 INFO Upload complete. (364ms)1497=== NAME TestClientPushesUseOnePush1498 client_pushes_test.go:97: Retrieved narinfo from S3:1499 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientPushesUseOnePush3199783104/001/store/407ij47i4dxkfj7m42rbmjdwn21jqrbw-shared-dep1500 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1501 Compression: zstd1502 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821503 NarSize: 1361504 References: 1505 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1506 client_pushes_test.go:97: Retrieved narinfo from S3:1507 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientPushesUseOnePush3199783104/001/store/cxrjzyxxag6hhkm64cj1j4jwql7g0lck-a1508 URL: nar/0a6bnxc5ivrip83nd33zaacw8yan9dva44h8q0j1ps9dyd5wfpx0.nar.zst1509 Compression: zstd1510 NarHash: sha256:0a6bnxc5ivrip83nd33zaacw8yan9dva44h8q0j1ps9dyd5wfpx01511 NarSize: 2561512 References: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientPushesUseOnePush3199783104/001/store/407ij47i4dxkfj7m42rbmjdwn21jqrbw-shared-dep1513 CA: text:sha256:0r8r0z8zvg3b5vf054wrfkh25mqmqjbv7svzw947r4cc9ldk2pg01514 client_pushes_test.go:97: Retrieved narinfo from S3:1515 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientPushesUseOnePush3199783104/001/store/55rbljrxbq584444p2mim625bwf9h9gk-b1516 URL: nar/0a6bnxc5ivrip83nd33zaacw8yan9dva44h8q0j1ps9dyd5wfpx0.nar.zst1517 Compression: zstd1518 NarHash: sha256:0a6bnxc5ivrip83nd33zaacw8yan9dva44h8q0j1ps9dyd5wfpx01519 NarSize: 2561520 References: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientPushesUseOnePush3199783104/001/store/407ij47i4dxkfj7m42rbmjdwn21jqrbw-shared-dep1521 CA: text:sha256:0r8r0z8zvg3b5vf054wrfkh25mqmqjbv7svzw947r4cc9ldk2pg01522--- PASS: TestClientPushesUseOnePush (8.06s)1523=== CONT TestClientIntegration1524--- PASS: TestCacheStatsHandler (7.17s)1525=== CONT TestLeadElectsOneAndHandsOver15262026/11/12 00:02:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=015272026/11/12 00:02:49 INFO Received uploads request method=POST path=/api/pending_closures15282026/11/12 00:02:49 INFO Vacuumed table table=pending_closures15292026/11/12 00:02:49 INFO Vacuumed table table=pending_objects15302026/11/12 00:02:49 INFO Vacuumed table table=multipart_uploads15312026/11/12 00:02:49 INFO Vacuumed table table=closures1532--- PASS: TestTombstonedObjectOfferedWithoutWaiting (7.10s)1533=== CONT TestService_AuthMiddleware_MTLSBoundSubjects15342026/11/12 00:02:49 INFO Vacuumed table table=objects1535--- PASS: TestGCSweepDeliversEachKeyOnce (8.09s)1536=== CONT TestClientErrorHandling1537=== RUN TestClientErrorHandling/InvalidStorePath1538=== PAUSE TestClientErrorHandling/InvalidStorePath1539=== RUN TestClientErrorHandling/InvalidAuthToken1540=== PAUSE TestClientErrorHandling/InvalidAuthToken1541=== RUN TestClientErrorHandling/ServerNotAvailable1542=== PAUSE TestClientErrorHandling/ServerNotAvailable1543=== CONT TestConnectWaitsForAPeerMigration15442026/11/12 00:02:50 INFO Received uploads request method=POST path=/api/pending_closures15452026/11/12 00:02:50 INFO Aborted multipart uploads count=0 kept=015462026/11/12 00:02:50 WARN Force mode enabled - objects will be deleted immediately without grace period15472026/11/12 00:02:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015482026/11/12 00:02:50 INFO Vacuumed table table=pending_closures15492026/11/12 00:02:50 INFO Vacuumed table table=pending_objects15502026/11/12 00:02:50 INFO Vacuumed table table=multipart_uploads15512026/11/12 00:02:50 INFO Vacuumed table table=closures15522026/11/12 00:02:50 INFO Vacuumed table table=objects1553--- PASS: TestGCSweepSkipsPendingObjects (7.56s)1554=== CONT TestService_AuthMiddleware1555--- PASS: TestService_ReadScope_PublicByDefault (7.02s)1556=== CONT TestService_AuthMiddleware_MTLSProxyHeader15572026/11/12 00:02:51 INFO Received uploads request method=POST path=/api/pending_closures1558=== RUN TestService_RequireScope_OIDC/builder_may_write1559=== PAUSE TestService_RequireScope_OIDC/builder_may_write1560=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1561=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1562=== RUN TestService_RequireScope_OIDC/ops_may_admin1563=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1564=== RUN TestService_RequireScope_OIDC/ops_may_not_write1565=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1566=== RUN TestService_RequireScope_OIDC/reader_may_not_write1567=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1568=== RUN TestService_RequireScope_OIDC/static_token_may_admin1569=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1570=== RUN TestService_RequireScope_OIDC/static_token_may_write1571=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1572=== RUN TestService_RequireScope_OIDC/reader_may_read1573=== PAUSE TestService_RequireScope_OIDC/reader_may_read1574=== RUN TestService_RequireScope_OIDC/writer_implies_read1575=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1576=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1577=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1578=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1579=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1580=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1581=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1582=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1583=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1584=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1585=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1586=== CONT TestResolveDBConnectionString1587=== RUN TestResolveDBConnectionString/flag_wins1588=== PAUSE TestResolveDBConnectionString/flag_wins1589=== RUN TestResolveDBConnectionString/file_when_flag_empty1590=== PAUSE TestResolveDBConnectionString/file_when_flag_empty1591=== RUN TestResolveDBConnectionString/missing_file_is_an_error1592=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error1593=== RUN TestResolveDBConnectionString/PGHOST_allows_empty1594=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty1595=== RUN TestResolveDBConnectionString/nothing_configured1596=== PAUSE TestResolveDBConnectionString/nothing_configured1597=== CONT TestPendingClosureWriteTimeout/empty1598=== CONT TestPendingClosureWriteTimeout/670k_objects1599=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline16002026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=016012026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016022026/11/12 00:02:51 INFO Vacuumed table table=pending_closures16032026/11/12 00:02:51 INFO Vacuumed table table=pending_objects16042026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads16052026/11/12 00:02:51 INFO Vacuumed table table=closures16062026/11/12 00:02:51 INFO Vacuumed table table=objects16072026/11/12 00:02:51 INFO Aborted multipart uploads count=0 kept=016082026/11/12 00:02:51 WARN Force mode enabled - objects will be deleted immediately without grace period16092026/11/12 00:02:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016102026/11/12 00:02:51 INFO Vacuumed table table=pending_closures16112026/11/12 00:02:51 INFO Vacuumed table table=pending_objects16122026/11/12 00:02:51 INFO Received uploads request method=POST path=/api/pending_closures16132026/11/12 00:02:51 INFO Vacuumed table table=multipart_uploads16142026/11/12 00:02:51 INFO Vacuumed table table=closures16152026/11/12 00:02:51 INFO Vacuumed table table=objects1616--- PASS: TestPushDedupSurvivesConcurrentGC (8.06s)1617=== CONT TestPendingClosureWriteTimeout/400_objects1618=== CONT TestProxyWriteTimeout/narinfo1619=== CONT TestProxyWriteTimeout/unknown_size1620=== CONT TestProxyWriteTimeout/10_GiB_nar1621=== CONT TestProxyWriteTimeout/1_GiB_nar1622--- PASS: TestProxyWriteTimeout (0.00s)1623 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1624 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1625 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1626 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1627=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info16282026/11/12 00:02:51 INFO Received uploads request method=POST path=/1629=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key16302026/11/12 00:02:51 INFO Received request for more parts method=POST path=/1631=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key16322026/11/12 00:02:51 INFO Received complete multipart upload request method=POST path=/1633=== CONT TestIsValidUploadKey/narinfo1634=== CONT TestIsValidUploadKey/realisation_plus_in_output1635=== CONT TestIsValidUploadKey/empty_key1636=== CONT TestIsValidUploadKey/absolute1637=== CONT TestIsValidUploadKey/traversal_nar1638=== CONT TestIsValidUploadKey/traversal1639--- PASS: TestDeduplicatedObjectsRecordedAsPending (8.12s)1640=== CONT TestIsValidUploadKey/unknown_type1641=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1642=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1643=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1644=== CONT TestIsValidUploadKey/index.html1645=== CONT TestIsValidUploadKey/nix-cache-info1646=== CONT TestIsValidUploadKey/build_log1647=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1648=== CONT TestIsValidUploadKey/listing1649=== CONT TestIsValidUploadKey/nar_plain1650=== CONT TestIsValidUploadKey/nar_xz1651=== CONT TestIsValidUploadKey/nar_zst1652=== CONT TestIsValidUploadKey/realisation1653=== CONT TestIsValidUploadKey/build_log_equals1654=== CONT TestIsValidUploadKey/build_log_question_mark1655=== CONT TestIsValidUploadKey/build_log_plus_in_name1656=== CONT TestIsValidUploadKey/build_log_home-manager_file1657--- PASS: TestIsValidUploadKey (0.04s)1658 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1659 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1660 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1661 --- PASS: TestIsValidUploadKey/absolute (0.00s)1662 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1663 --- PASS: TestIsValidUploadKey/traversal (0.00s)1664 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1665 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1666 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1667 --- PASS: TestIsValidUploadKey/index.html (0.00s)1668 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1669 --- PASS: TestIsValidUploadKey/build_log (0.00s)1670 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1671 --- PASS: TestIsValidUploadKey/listing (0.00s)1672 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1673 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1674 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1675 --- PASS: TestIsValidUploadKey/realisation (0.00s)1676 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1677 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1678 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1679 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1680=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed16812026/11/12 00:02:52 INFO Aborted multipart uploads count=0 kept=016822026/11/12 00:02:52 WARN Force mode enabled - objects will be deleted immediately without grace period16832026/11/12 00:02:52 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=016842026/11/12 00:02:52 INFO Vacuumed table table=pending_closures16852026/11/12 00:02:52 INFO Vacuumed table table=pending_objects16862026/11/12 00:02:52 INFO Vacuumed table table=multipart_uploads16872026/11/12 00:02:52 INFO Vacuumed table table=closures16882026/11/12 00:02:52 INFO Vacuumed table table=objects1689--- PASS: TestGCMetrics (8.46s)1690=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1691--- PASS: TestGCBugBareHashReferences (8.89s)1692=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed16932026/11/12 00:02:52 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=_tribuchetbld8 database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1694--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (8.70s)1695=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1696--- PASS: TestDeleteClosuresRacingPins (13.20s)1697=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost16982026/11/12 00:02:53 INFO lead: acquired remote=192.0.2.1:1234 generation=116992026/11/12 00:02:53 INFO lead: acquired remote=192.0.2.1:1234 generation=11700=== NAME TestPinProtectsFromGC1701 client_integration_test.go:791: Pinned store path: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestPinProtectsFromGC3947715113/001/store/hrqplss195jb33bkkigjr1z72fdl3bhk-pinned-file.txt1702 client_integration_test.go:792: Unpinned store path: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestPinProtectsFromGC3947715113/001/store/vfv92y24q5hw9rkll67lga9xas5g7amw-unpinned-file.txt17032026/11/12 00:02:54 INFO Received push request method=POST path=/api/pushes17042026/11/12 00:02:54 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17052026/11/12 00:02:54 INFO Uploading hrqplss195jb33bkkigjr1z72fdl3bhk-pinned-file.txt (128B)17062026/11/12 00:02:54 WARN Failed to register uploaded object key=hrqplss195jb33bkkigjr1z72fdl3bhk.ls error="server returned 404: 404 page not found\n"17072026/11/12 00:02:54 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17082026/11/12 00:02:54 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"17092026/11/12 00:02:54 INFO Signed narinfos id=1 count=117102026/11/12 00:02:54 INFO Uploading 1 narinfos17112026/11/12 00:02:54 INFO lead: released remote=192.0.2.1:1234 generation=11712--- PASS: TestLeadEndsOnShutdown (9.72s)1713=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed17142026/11/12 00:02:54 INFO Received complete push request method=POST path=/api/pushes/1/complete17152026/11/12 00:02:54 WARN Failed to register uploaded object key=hrqplss195jb33bkkigjr1z72fdl3bhk.narinfo error="server returned 404: 404 page not found\n"17162026/11/12 00:02:54 INFO Completed push id=1 objects=3 duration=10.137917ms17172026/11/12 00:02:54 INFO Upload complete. (272ms)17182026/11/12 00:02:54 INFO Received push request method=POST path=/api/pushes17192026/11/12 00:02:54 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17202026/11/12 00:02:54 INFO Uploading vfv92y24q5hw9rkll67lga9xas5g7amw-unpinned-file.txt (128B)17212026/11/12 00:02:54 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"17222026/11/12 00:02:54 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign17232026/11/12 00:02:54 WARN Failed to register uploaded object key=vfv92y24q5hw9rkll67lga9xas5g7amw.ls error="server returned 404: 404 page not found\n"17242026/11/12 00:02:54 INFO Signed narinfos id=2 count=117252026/11/12 00:02:54 INFO Uploading 1 narinfos17262026/11/12 00:02:54 INFO lead: acquired remote=192.0.2.1:1234 generation=117272026/11/12 00:02:54 INFO lead: released remote=192.0.2.1:1234 generation=11728--- PASS: TestLeadIncumbentAnnouncesAtOnce (6.76s)1729=== CONT TestParseSingleRange/none1730=== CONT TestParseSingleRange/end_clamped_to_size1731=== CONT TestParseSingleRange/single_byte1732=== CONT TestParseSingleRange/start_far_past_EOF1733=== CONT TestParseSingleRange/start_past_EOF17342026/11/12 00:02:54 INFO Received complete push request method=POST path=/api/pushes/2/complete1735=== CONT TestParseSingleRange/malformed_both_empty17362026/11/12 00:02:54 WARN Failed to register uploaded object key=vfv92y24q5hw9rkll67lga9xas5g7amw.narinfo error="server returned 404: 404 page not found\n"17372026/11/12 00:02:54 INFO Completed push id=2 objects=3 duration=812.958µs1738=== CONT TestParseSingleRange/open-ended17392026/11/12 00:02:54 INFO Upload complete. (208ms)1740=== CONT TestParseSingleRange/closed1741=== CONT TestParseSingleRange/malformed_end_before_start1742=== CONT TestParseSingleRange/suffix_exceeds_size1743=== CONT TestParseSingleRange/suffix1744=== CONT TestParseSingleRange/multi-range_ignored1745=== CONT TestParseSingleRange/malformed_no_dash1746=== CONT TestParseSingleRange/unknown_unit1747--- PASS: TestParseSingleRange (0.01s)1748 --- PASS: TestParseSingleRange/none (0.00s)1749 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1750 --- PASS: TestParseSingleRange/single_byte (0.00s)1751 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1752 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1753 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1754 --- PASS: TestParseSingleRange/open-ended (0.00s)1755 --- PASS: TestParseSingleRange/closed (0.00s)1756 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1757 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1758 --- PASS: TestParseSingleRange/suffix (0.00s)1759 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1760 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1761 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1762=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal17632026/11/12 00:02:54 INFO Received uploads request method=POST path=/1764--- PASS: TestUploadHandlersRejectInvalidKeys (0.04s)1765 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1766 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1767 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1768 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.01s)1769=== CONT TestIsValidCachePath/narinfo1770=== CONT TestIsValidCachePath/index.html1771=== CONT TestIsValidCachePath/short_hash1772=== CONT TestIsValidCachePath/wrong_extension1773=== CONT TestIsValidCachePath/leading_slash1774=== CONT TestIsValidCachePath/empty1775=== CONT TestIsValidCachePath/random_path1776=== CONT TestIsValidCachePath/invalid_char_u1777=== CONT TestIsValidCachePath/invalid_char_e1778=== CONT TestIsValidCachePath/traversal_in_middle1779=== CONT TestIsValidCachePath/traversal_parent1780=== CONT TestIsValidCachePath/nar_uncompressed1781=== CONT TestIsValidCachePath/nix-cache-info1782=== CONT TestIsValidCachePath/realisation1783=== CONT TestIsValidCachePath/log1784=== CONT TestIsValidCachePath/ls1785=== CONT TestIsValidCachePath/nar_xz1786=== CONT TestIsValidCachePath/nar_bz21787=== CONT TestIsValidCachePath/nar_zst1788=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1789--- PASS: TestIsValidCachePath (0.00s)1790 --- PASS: TestIsValidCachePath/narinfo (0.00s)1791 --- PASS: TestIsValidCachePath/index.html (0.00s)1792 --- PASS: TestIsValidCachePath/short_hash (0.00s)1793 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1794 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1795 --- PASS: TestIsValidCachePath/empty (0.00s)1796 --- PASS: TestIsValidCachePath/random_path (0.00s)1797 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1798 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1799 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1800 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1801 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1802 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1803 --- PASS: TestIsValidCachePath/realisation (0.00s)1804 --- PASS: TestIsValidCachePath/log (0.00s)1805 --- PASS: TestIsValidCachePath/ls (0.00s)1806 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1807 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1808 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1809 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1810=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure18112026/11/12 00:02:54 INFO Received uploads request method=POST path=/18122026/11/12 00:02:54 INFO Starting cleanup of old closures method=DELETE path=/api/closures18132026/11/12 00:02:54 INFO Garbage collection started18142026/11/12 00:02:54 INFO Aborted multipart uploads count=0 kept=018152026/11/12 00:02:54 WARN Force mode enabled - objects will be deleted immediately without grace period18162026/11/12 00:02:54 INFO Received push request method=POST path=/api/pushes18172026/11/12 00:02:54 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)18182026/11/12 00:02:54 INFO Uploading r14092ggh2ks2i6jxr32xpcdair5d2b7-top (264B)18192026/11/12 00:02:54 INFO Uploading pqz01z5ar8lk1sf74kqm4cfl25wh874p-shared-dep (136B)18202026/11/12 00:02:54 WARN Failed to register uploaded object key=r14092ggh2ks2i6jxr32xpcdair5d2b7.ls error="server returned 404: 404 page not found\n"18212026/11/12 00:02:54 WARN Failed to register uploaded object key=pqz01z5ar8lk1sf74kqm4cfl25wh874p.ls error="server returned 404: 404 page not found\n"18222026/11/12 00:02:54 WARN Failed to register uploaded object key=nar/0v6a0635v2v5pi1qd28zsmfg6xlk2y33b953apv2sj7rsd9731dl.nar.zst error="server returned 404: 404 page not found\n"18232026/11/12 00:02:54 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign18242026/11/12 00:02:54 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"18252026/11/12 00:02:54 INFO Signed narinfos id=1 count=218262026/11/12 00:02:54 INFO Uploading 2 narinfos1827=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts18282026/11/12 00:02:54 INFO Received request for more parts method=POST path=/1829=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart18302026/11/12 00:02:54 INFO Received complete multipart upload request method=POST path=/18312026/11/12 00:02:54 WARN Failed to register uploaded object key=pqz01z5ar8lk1sf74kqm4cfl25wh874p.narinfo error="server returned 404: 404 page not found\n"1832--- PASS: TestUploadHandlersRejectOversizedBody (0.06s)1833 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (0.28s)1834 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.02s)1835 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.02s)1836=== CONT TestPush_RejectsBadRequests/no_roots18372026/11/12 00:02:54 INFO Received push request method=POST path=/api/pushes1838=== CONT TestPush_RejectsBadRequests/root_not_in_objects18392026/11/12 00:02:54 INFO Received complete push request method=POST path=/api/pushes/1/complete18402026/11/12 00:02:54 WARN Failed to register uploaded object key=r14092ggh2ks2i6jxr32xpcdair5d2b7.narinfo error="server returned 404: 404 page not found\n"18412026/11/12 00:02:54 INFO Received push request method=POST path=/api/pushes1842=== CONT TestPush_RejectsBadRequests/bad_root18432026/11/12 00:02:54 INFO Received push request method=POST path=/api/pushes1844=== CONT TestPush_RejectsBadRequests/no_objects18452026/11/12 00:02:54 INFO Received push request method=POST path=/api/pushes1846=== CONT TestServerTLSConfig/no_client_CA1847=== CONT TestServerTLSConfig/not_a_PEM_file1848--- PASS: TestPush_RejectsBadRequests (4.01s)1849 --- PASS: TestPush_RejectsBadRequests/no_roots (0.01s)1850 --- PASS: TestPush_RejectsBadRequests/root_not_in_objects (0.02s)1851 --- PASS: TestPush_RejectsBadRequests/bad_root (0.01s)1852 --- PASS: TestPush_RejectsBadRequests/no_objects (0.01s)18532026/11/12 00:02:54 INFO Completed push id=1 objects=6 duration=46.119875ms18542026/11/12 00:02:54 INFO Upload complete. (349ms)1855=== NAME TestClientSharedPathCommittedMidPush1856 client_integration_test.go:740: Retrieved narinfo from S3:1857 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientSharedPathCommittedMidPush4064522950/001/store/pqz01z5ar8lk1sf74kqm4cfl25wh874p-shared-dep1858 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1859 Compression: zstd1860 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821861 NarSize: 13618622026/11/12 00:02:55 INFO lead: released remote=192.0.2.1:1234 generation=11863 References: 18642026/11/12 00:02:55 INFO lead: acquired remote=192.0.2.1:1234 generation=21865=== CONT TestServerTLSConfig/missing_CA_file1866=== NAME TestClientSharedPathCommittedMidPush1867 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1868--- PASS: TestServerTLSConfig (0.01s)1869 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1870 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.04s)1871 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1872=== CONT TestGCEndsOnShutdown/before_the_run1873=== NAME TestClientSharedPathCommittedMidPush1874 client_integration_test.go:740: Retrieved narinfo from S3:1875 StorePath: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientSharedPathCommittedMidPush4064522950/001/store/r14092ggh2ks2i6jxr32xpcdair5d2b7-top1876 URL: nar/0v6a0635v2v5pi1qd28zsmfg6xlk2y33b953apv2sj7rsd9731dl.nar.zst1877 Compression: zstd1878 NarHash: sha256:0v6a0635v2v5pi1qd28zsmfg6xlk2y33b953apv2sj7rsd9731dl1879 NarSize: 2641880 References: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientSharedPathCommittedMidPush4064522950/001/store/pqz01z5ar8lk1sf74kqm4cfl25wh874p-shared-dep1881 CA: text:sha256:0him7wj5bizmrb98mb98w7g3dxjx1winfapa0qr71crjv62fybk618822026/11/12 00:02:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=01883--- PASS: TestClientSharedPathCommittedMidPush (9.55s)1884=== CONT TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark18852026/11/12 00:02:55 INFO Vacuumed table table=pending_closures18862026/11/12 00:02:55 INFO Vacuumed table table=pending_objects1887=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1888=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1889=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1890=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1891=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1892=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1893=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1894=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1895=== CONT TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete18962026/11/12 00:02:55 INFO Vacuumed table table=multipart_uploads18972026/11/12 00:02:55 INFO Vacuumed table table=closures18982026/11/12 00:02:55 INFO Vacuumed table table=objects1899=== NAME TestClientWithDependencies1900 client_integration_test.go:661: Built derivation: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientWithDependencies3574379375/001/store/df6khzh69s6d9cxc7s856gqy6zlmbvsr-test-script1901 client_integration_test.go:663: Found 1 dependencies (including self)19022026/11/12 00:02:55 INFO Received push request method=POST path=/api/pushes19032026/11/12 00:02:55 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)19042026/11/12 00:02:55 INFO Uploading df6khzh69s6d9cxc7s856gqy6zlmbvsr-test-script (136B)19052026/11/12 00:02:55 WARN Failed to register uploaded object key=df6khzh69s6d9cxc7s856gqy6zlmbvsr.ls error="server returned 404: 404 page not found\n"19062026/11/12 00:02:55 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"19072026/11/12 00:02:55 WARN Failed to register uploaded object key=log/ryfnhw4j7mh4pm19iy289iq7hg5rwq5w-test-script.drv error="server returned 404: 404 page not found\n"19082026/11/12 00:02:55 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19092026/11/12 00:02:55 INFO Signed narinfos id=1 count=119102026/11/12 00:02:55 INFO Uploading 1 narinfos19112026/11/12 00:02:55 INFO Received complete push request method=POST path=/api/pushes/1/complete19122026/11/12 00:02:55 WARN Failed to register uploaded object key=df6khzh69s6d9cxc7s856gqy6zlmbvsr.narinfo error="server returned 404: 404 page not found\n"19132026/11/12 00:02:55 INFO Completed push id=1 objects=4 duration=31.068334ms19142026/11/12 00:02:55 INFO Upload complete. (244ms)19152026/11/12 00:02:55 INFO Aborted multipart uploads count=0 kept=019162026/11/12 00:02:55 WARN Force mode enabled - objects will be deleted immediately without grace period19172026/11/12 00:02:55 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=01918=== NAME TestClientMultipleUploads1919 client_integration_test.go:406: Created store path 0: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientMultipleUploads1348759028/001/store/6dfihzlsx51anm5hxwb6pzl2pm54s0j1-test-file-0.txt19202026/11/12 00:02:55 INFO lead: acquired remote=192.0.2.1:1234 generation=119212026/11/12 00:02:55 INFO Vacuumed table table=pending_closures19222026/11/12 00:02:55 INFO Vacuumed table table=pending_objects19232026/11/12 00:02:55 INFO Vacuumed table table=multipart_uploads19242026/11/12 00:02:55 INFO Vacuumed table table=closures1925 client_integration_test.go:406: Created store path 1: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientMultipleUploads1348759028/001/store/xjkwqc8rf6yvqqmr801zzjaxcm8qlsby-test-file-1.txt19262026/11/12 00:02:55 INFO Vacuumed table table=objects1927=== NAME TestClientWithDependencies1928 client_integration_test.go:677: Skipping nix copy test - isolated store (/private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientWithDependencies3574379375/001/store) requires matching store prefix1929=== NAME TestClientMultipleUploads1930 client_integration_test.go:406: Created store path 2: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientMultipleUploads1348759028/001/store/ak2fcnvs578v6p2kkf56rpb3x888rhb7-test-file-2.txt1931--- PASS: TestClientWithDependencies (8.07s)1932=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows1933--- PASS: TestService_ReadAuthMiddleware (7.05s)1934=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check19352026/11/12 00:02:56 INFO Received push request method=POST path=/api/pushes19362026/11/12 00:02:56 INFO lead: released remote=192.0.2.1:1234 generation=219372026/11/12 00:02:56 INFO lead: acquired remote=192.0.2.1:1234 generation=319382026/11/12 00:02:56 INFO lead: released remote=192.0.2.1:1234 generation=31939--- PASS: TestLeadGenerationGrowsWithEveryGrant (10.93s)1940=== CONT TestCacheConfigHandler/full_config,_no_issuer1941=== CONT TestCacheConfigHandler/no_signing_keys1942=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1943=== CONT TestCacheConfigHandler/no_cache_url_configured1944--- PASS: TestCacheConfigHandler (0.00s)1945 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1946 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1947 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1948 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)1949=== CONT TestClientErrorHandling/InvalidStorePath19502026/11/12 00:02:56 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)19512026/11/12 00:02:56 INFO Uploading 6dfihzlsx51anm5hxwb6pzl2pm54s0j1-test-file-0.txt (160B)19522026/11/12 00:02:56 INFO Uploading ak2fcnvs578v6p2kkf56rpb3x888rhb7-test-file-2.txt (160B)19532026/11/12 00:02:56 INFO Uploading xjkwqc8rf6yvqqmr801zzjaxcm8qlsby-test-file-1.txt (160B)19542026/11/12 00:02:56 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"19552026/11/12 00:02:56 WARN Failed to register uploaded object key=xjkwqc8rf6yvqqmr801zzjaxcm8qlsby.ls error="server returned 404: 404 page not found\n"19562026/11/12 00:02:56 WARN Failed to register uploaded object key=ak2fcnvs578v6p2kkf56rpb3x888rhb7.ls error="server returned 404: 404 page not found\n"19572026/11/12 00:02:56 WARN Failed to register uploaded object key=6dfihzlsx51anm5hxwb6pzl2pm54s0j1.ls error="server returned 404: 404 page not found\n"19582026/11/12 00:02:56 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"19592026/11/12 00:02:56 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign19602026/11/12 00:02:56 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"19612026/11/12 00:02:56 INFO Signed narinfos id=1 count=319622026/11/12 00:02:56 INFO Uploading 3 narinfos19632026/11/12 00:02:56 WARN Failed to register uploaded object key=ak2fcnvs578v6p2kkf56rpb3x888rhb7.narinfo error="server returned 404: 404 page not found\n"19642026/11/12 00:02:56 WARN Failed to register uploaded object key=6dfihzlsx51anm5hxwb6pzl2pm54s0j1.narinfo error="server returned 404: 404 page not found\n"19652026/11/12 00:02:56 INFO Received complete push request method=POST path=/api/pushes/1/complete19662026/11/12 00:02:56 WARN Failed to register uploaded object key=xjkwqc8rf6yvqqmr801zzjaxcm8qlsby.narinfo error="server returned 404: 404 page not found\n"19672026/11/12 00:02:56 INFO Completed push id=1 objects=9 duration=23.931709ms19682026/11/12 00:02:56 INFO lead: acquired remote=192.0.2.1:1234 generation=119692026/11/12 00:02:56 INFO Upload complete. (351ms)1970=== NAME TestClientMultipleUploads1971 client_integration_test.go:417: Uploaded 3 paths in 385.262167ms1972--- PASS: TestClientMultipleUploads (8.07s)1973=== CONT TestClientErrorHandling/ServerNotAvailable19742026/11/12 00:02:56 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=3 objects_failed=019752026/11/12 00:02:56 INFO Received create pin request method=POST path=/api/pins/myapp19762026/11/12 00:02:56 INFO Created/updated pin name=myapp store_path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestPinProtectsFromGC3947715113/001/store/hrqplss195jb33bkkigjr1z72fdl3bhk-pinned-file.txt narinfo_key=hrqplss195jb33bkkigjr1z72fdl3bhk.narinfo19772026/11/12 00:02:56 INFO Starting cleanup of old closures method=DELETE path=/api/closures19782026/11/12 00:02:56 INFO Garbage collection started19792026/11/12 00:02:56 INFO Aborted multipart uploads count=0 kept=019802026/11/12 00:02:56 WARN Force mode enabled - objects will be deleted immediately without grace period19812026/11/12 00:02:56 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=019822026/11/12 00:02:56 INFO Vacuumed table table=pending_closures19832026/11/12 00:02:56 INFO Vacuumed table table=pending_objects19842026/11/12 00:02:56 INFO Vacuumed table table=multipart_uploads19852026/11/12 00:02:56 INFO Vacuumed table table=closures19862026/11/12 00:02:56 INFO Vacuumed table table=objects19872026/11/12 00:02:56 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19882026/11/12 00:02:56 WARN Request failed, retrying attempt=2 backoff=209.035751ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present19892026/11/12 00:02:56 INFO lead: released remote=192.0.2.1:1234 generation=119902026/11/12 00:02:56 INFO lead: acquired remote=192.0.2.1:1234 generation=219912026/11/12 00:02:56 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"19922026/11/12 00:02:56 WARN mTLS auth: bound subjects configured but subject DN unavailable19932026/11/12 00:02:56 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1994--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (7.08s)1995=== CONT TestClientErrorHandling/InvalidAuthToken19962026/11/12 00:02:56 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=219972026/11/12 00:02:56 INFO lead: acquired remote=192.0.2.1:1234 generation=31998=== NAME TestClientIntegration1999 client_integration_test.go:292: Created store path: /private/var/tb/8/scratch/b3af67106d865ad2/build/TestClientIntegration189420881/002/store/66qmymmv9yf9bd8jinzcb3ckw2as6yvm-test-file.txt20002026/11/12 00:02:57 WARN Request failed, retrying attempt=3 backoff=434.964719ms error="Post \"http://localhost:19999/api/objects/present\": dial tcp 127.0.0.1:19999: connect: connection refused" url=http://localhost:19999/api/objects/present

loading 658 hidden lines…

26592026/11/12 00:03:14 INFO Received push request method=POST path=/26602026/11/12 00:03:14 INFO Vacuumed table table=closures26612026/11/12 00:03:14 INFO Completed upload id=9 objects=7 duration=972.125µs26622026/11/12 00:03:14 INFO Received uploads request method=POST path=/26632026/11/12 00:03:14 INFO Vacuumed table table=objects26642026/11/12 00:03:14 INFO Received uploads request method=POST path=/26652026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=026662026/11/12 00:03:14 INFO Received uploads request method=POST path=/26672026/11/12 00:03:14 INFO Received push request method=POST path=/26682026/11/12 00:03:14 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."26692026/11/12 00:03:14 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."26702026/11/12 00:03:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."26712026/11/12 00:03:14 INFO Received uploads request method=POST path=/26722026/11/12 00:03:14 INFO Received push request method=POST path=/26732026/11/12 00:03:14 INFO Received uploads request method=POST path=/26742026/11/12 00:03:14 INFO Received uploads request method=POST path=/26752026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=026762026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=026772026/11/12 00:03:14 INFO Vacuumed table table=pending_closures26782026/11/12 00:03:14 INFO Received complete push request method=POST path=/26792026/11/12 00:03:14 INFO Vacuumed table table=pending_objects26802026/11/12 00:03:14 INFO Completed push id=10 objects=2 duration=554.458µs26812026/11/12 00:03:14 INFO Vacuumed table table=multipart_uploads26822026/11/12 00:03:14 INFO Vacuumed table table=closures26832026/11/12 00:03:14 INFO Vacuumed table table=objects26842026/11/12 00:03:14 INFO Received push request method=POST path=/26852026/11/12 00:03:14 INFO Received push request method=POST path=/26862026/11/12 00:03:14 INFO Received uploads request method=POST path=/26872026/11/12 00:03:14 INFO Received uploads request method=POST path=/26882026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=026892026/11/12 00:03:14 WARN Force mode enabled - objects will be deleted immediately without grace period26902026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026912026/11/12 00:03:14 INFO Vacuumed table table=pending_closures26922026/11/12 00:03:14 INFO Received complete upload request method=POST path=/26932026/11/12 00:03:14 INFO Completed upload id=12 objects=21 duration=566.708µs26942026/11/12 00:03:14 INFO Vacuumed table table=pending_objects26952026/11/12 00:03:14 INFO Vacuumed table table=multipart_uploads26962026/11/12 00:03:14 INFO Received push request method=POST path=/26972026/11/12 00:03:14 INFO Vacuumed table table=closures26982026/11/12 00:03:14 INFO Received complete upload request method=POST path=/26992026/11/12 00:03:14 INFO Vacuumed table table=objects27002026/11/12 00:03:14 INFO Completed upload id=5 objects=7 duration=1.32525ms27012026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=027022026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027032026/11/12 00:03:14 INFO Vacuumed table table=pending_closures27042026/11/12 00:03:14 INFO Vacuumed table table=pending_objects27052026/11/12 00:03:14 INFO Vacuumed table table=multipart_uploads27062026/11/12 00:03:14 INFO Vacuumed table table=closures27072026/11/12 00:03:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."27082026/11/12 00:03:14 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."27092026/11/12 00:03:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."27102026/11/12 00:03:14 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."27112026/11/12 00:03:14 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."27122026/11/12 00:03:14 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."27132026/11/12 00:03:14 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."27142026/11/12 00:03:14 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."27152026/11/12 00:03:14 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."27162026/11/12 00:03:14 INFO Vacuumed table table=objects27172026/11/12 00:03:14 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."27182026/11/12 00:03:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."27192026/11/12 00:03:14 INFO Received complete upload request method=POST path=/27202026/11/12 00:03:14 INFO Received complete push request method=POST path=/27212026/11/12 00:03:14 INFO Received push request method=POST path=/27222026/11/12 00:03:14 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."27232026/11/12 00:03:14 INFO Completed upload id=6 objects=9 duration=560.833µs27242026/11/12 00:03:14 INFO Completed push id=13 objects=23 duration=660.792µs27252026/11/12 00:03:14 INFO Received uploads request method=POST path=/27262026/11/12 00:03:14 INFO Received uploads request method=POST path=/27272026/11/12 00:03:14 INFO Received push request method=POST path=/27282026/11/12 00:03:14 INFO Received uploads request method=POST path=/27292026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=027302026/11/12 00:03:14 INFO Received push request method=POST path=/27312026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=027322026/11/12 00:03:14 INFO Received complete upload request method=POST path=/27332026/11/12 00:03:14 INFO Received push request method=POST path=/27342026/11/12 00:03:14 INFO Completed upload id=3 objects=5 duration=393.5µs27352026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=027362026/11/12 00:03:14 INFO Vacuumed table table=pending_closures27372026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=027382026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=027392026/11/12 00:03:14 INFO Vacuumed table table=pending_closures27402026/11/12 00:03:14 INFO Vacuumed table table=pending_objects27412026/11/12 00:03:14 INFO Vacuumed table table=pending_objects27422026/11/12 00:03:14 INFO Vacuumed table table=multipart_uploads27432026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027442026/11/12 00:03:14 INFO Vacuumed table table=multipart_uploads27452026/11/12 00:03:14 INFO Vacuumed table table=closures27462026/11/12 00:03:14 INFO Vacuumed table table=objects27472026/11/12 00:03:14 INFO Vacuumed table table=pending_closures27482026/11/12 00:03:14 INFO Received push request method=POST path=/27492026/11/12 00:03:14 INFO Received push request method=POST path=/27502026/11/12 00:03:14 INFO Vacuumed table table=closures27512026/11/12 00:03:14 INFO Vacuumed table table=pending_objects27522026/11/12 00:03:14 INFO Vacuumed table table=multipart_uploads27532026/11/12 00:03:14 INFO Vacuumed table table=objects27542026/11/12 00:03:14 INFO Vacuumed table table=closures27552026/11/12 00:03:14 INFO Received push request method=POST path=/27562026/11/12 00:03:14 INFO Vacuumed table table=objects27572026/11/12 00:03:14 INFO Received complete upload request method=POST path=/27582026/11/12 00:03:14 INFO Received complete push request method=POST path=/27592026/11/12 00:03:14 INFO Received uploads request method=POST path=/27602026/11/12 00:03:14 INFO Received uploads request method=POST path=/27612026/11/12 00:03:14 INFO Completed push id=15 objects=7 duration=845.75µs27622026/11/12 00:03:14 INFO Completed upload id=10 objects=2 duration=12.004375ms27632026/11/12 00:03:14 INFO Received uploads request method=POST path=/27642026/11/12 00:03:14 INFO Received push request method=POST path=/27652026/11/12 00:03:14 INFO Received push request method=POST path=/27662026/11/12 00:03:14 INFO Received uploads request method=POST path=/27672026/11/12 00:03:14 INFO Received complete upload request method=POST path=/27682026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=027692026/11/12 00:03:14 INFO Completed upload id=15 objects=4 duration=783.875µs27702026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=027712026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=027722026/11/12 00:03:14 INFO Received uploads request method=POST path=/27732026/11/12 00:03:14 INFO Vacuumed table table=pending_closures27742026/11/12 00:03:14 INFO Received push request method=POST path=/27752026/11/12 00:03:14 INFO Vacuumed table table=pending_objects27762026/11/12 00:03:14 INFO Received uploads request method=POST path=/27772026/11/12 00:03:14 INFO Vacuumed table table=multipart_uploads27782026/11/12 00:03:14 INFO Received push request method=POST path=/27792026/11/12 00:03:14 WARN Force mode enabled - objects will be deleted immediately without grace period27802026/11/12 00:03:14 INFO Received complete upload request method=POST path=/27812026/11/12 00:03:14 INFO Received complete upload request method=POST path=/27822026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=5 objects-failed-to-delete=027832026/11/12 00:03:14 INFO Vacuumed table table=closures27842026/11/12 00:03:14 INFO Completed upload id=22 objects=2 duration=5.764459ms27852026/11/12 00:03:14 INFO Completed upload id=16 objects=3 duration=5.797292ms27862026/11/12 00:03:14 INFO Vacuumed table table=objects27872026/11/12 00:03:14 INFO Aborted multipart uploads count=0 kept=027882026/11/12 00:03:14 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=027892026/11/12 00:03:15 INFO Vacuumed table table=pending_closures27902026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=027912026/11/12 00:03:15 INFO Vacuumed table table=pending_objects27922026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads27932026/11/12 00:03:15 INFO Vacuumed table table=pending_closures27942026/11/12 00:03:15 INFO Vacuumed table table=closures27952026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027962026/11/12 00:03:15 INFO Received push request method=POST path=/27972026/11/12 00:03:15 INFO Vacuumed table table=objects27982026/11/12 00:03:15 INFO Received complete push request method=POST path=/27992026/11/12 00:03:15 INFO Received uploads request method=POST path=/28002026/11/12 00:03:15 INFO Completed push id=2 objects=10 duration=5.189375ms28012026/11/12 00:03:15 INFO Vacuumed table table=pending_closures28022026/11/12 00:03:15 INFO Received complete push request method=POST path=/28032026/11/12 00:03:15 INFO Vacuumed table table=pending_objects28042026/11/12 00:03:15 INFO Completed push id=7 objects=20 duration=1.464125ms28052026/11/12 00:03:15 INFO Received push request method=POST path=/28062026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads28072026/11/12 00:03:15 INFO Received push request method=POST path=/28082026/11/12 00:03:15 INFO Received push request method=POST path=/28092026/11/12 00:03:15 INFO Vacuumed table table=closures28102026/11/12 00:03:15 INFO Vacuumed table table=pending_objects28112026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=028122026/11/12 00:03:15 INFO Received push request method=POST path=/28132026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads28142026/11/12 00:03:15 INFO Received complete push request method=POST path=/28152026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028162026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=028172026/11/12 00:03:15 INFO Vacuumed table table=closures28182026/11/12 00:03:15 INFO Completed push id=8 objects=7 duration=377.041µs28192026/11/12 00:03:15 INFO Vacuumed table table=objects28202026/11/12 00:03:15 INFO Vacuumed table table=pending_closures28212026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028222026/11/12 00:03:15 INFO Vacuumed table table=objects28232026/11/12 00:03:15 INFO Received uploads request method=POST path=/28242026/11/12 00:03:15 INFO Vacuumed table table=pending_closures28252026/11/12 00:03:15 INFO Received complete push request method=POST path=/28262026/11/12 00:03:15 INFO Received uploads request method=POST path=/28272026/11/12 00:03:15 INFO Vacuumed table table=pending_objects28282026/11/12 00:03:15 INFO Completed push id=9 objects=4 duration=6.543916ms28292026/11/12 00:03:15 INFO Vacuumed table table=pending_objects28302026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads28312026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads28322026/11/12 00:03:15 INFO Received complete upload request method=POST path=/28332026/11/12 00:03:15 INFO Vacuumed table table=closures28342026/11/12 00:03:15 INFO Received complete upload request method=POST path=/28352026/11/12 00:03:15 INFO Vacuumed table table=closures28362026/11/12 00:03:15 INFO Received complete push request method=POST path=/28372026/11/12 00:03:15 INFO Completed upload id=18 objects=4 duration=759.875µs28382026/11/12 00:03:15 INFO Vacuumed table table=objects28392026/11/12 00:03:15 INFO Vacuumed table table=objects28402026/11/12 00:03:15 INFO Received complete push request method=POST path=/28412026/11/12 00:03:15 INFO Completed push id=17 objects=6 duration=647.542µs28422026/11/12 00:03:15 INFO Completed push id=3 objects=22 duration=21.602291ms28432026/11/12 00:03:15 INFO Completed upload id=1 objects=9 duration=21.678708ms28442026/11/12 00:03:15 INFO Received push request method=POST path=/28452026/11/12 00:03:15 INFO Received push request method=POST path=/28462026/11/12 00:03:15 INFO Received push request method=POST path=/28472026/11/12 00:03:15 INFO Received push request method=POST path=/28482026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=028492026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028502026/11/12 00:03:15 INFO Received push request method=POST path=/28512026/11/12 00:03:15 INFO Received uploads request method=POST path=/28522026/11/12 00:03:15 INFO Received complete upload request method=POST path=/28532026/11/12 00:03:15 INFO Vacuumed table table=pending_closures28542026/11/12 00:03:15 INFO Vacuumed table table=pending_objects28552026/11/12 00:03:15 INFO Received push request method=POST path=/28562026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads28572026/11/12 00:03:15 INFO Vacuumed table table=closures28582026/11/12 00:03:15 INFO Received complete upload request method=POST path=/28592026/11/12 00:03:15 INFO Vacuumed table table=objects28602026/11/12 00:03:15 INFO Received uploads request method=POST path=/28612026/11/12 00:03:15 INFO Completed upload id=9 objects=16 duration=435.375µs28622026/11/12 00:03:15 INFO Completed upload id=1 objects=15 duration=26.997583ms28632026/11/12 00:03:15 INFO Received complete push request method=POST path=/28642026/11/12 00:03:15 INFO Received uploads request method=POST path=/28652026/11/12 00:03:15 INFO Received push request method=POST path=/28662026/11/12 00:03:15 INFO Received uploads request method=POST path=/28672026/11/12 00:03:15 INFO Completed push id=5 objects=3 duration=2.447583ms28682026/11/12 00:03:15 INFO Received push request method=POST path=/28692026/11/12 00:03:15 INFO Received push request method=POST path=/28702026/11/12 00:03:15 INFO Received push request method=POST path=/28712026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=028722026/11/12 00:03:15 INFO Received complete push request method=POST path=/28732026/11/12 00:03:15 WARN Force mode enabled - objects will be deleted immediately without grace period28742026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=028752026/11/12 00:03:15 INFO Received push request method=POST path=/28762026/11/12 00:03:15 INFO Received push request method=POST path=/28772026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=028782026/11/12 00:03:15 INFO Completed push id=10 objects=3 duration=423.209µs28792026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028802026/11/12 00:03:15 INFO Received complete push request method=POST path=/28812026/11/12 00:03:15 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28822026/11/12 00:03:15 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28832026/11/12 00:03:15 INFO Received complete push request method=POST path=/28842026/11/12 00:03:15 WARN Force mode enabled - objects will be deleted immediately without grace period28852026/11/12 00:03:15 INFO Received push request method=POST path=/28862026/11/12 00:03:15 INFO Received uploads request method=POST path=/28872026/11/12 00:03:15 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28882026/11/12 00:03:15 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28892026/11/12 00:03:15 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28902026/11/12 00:03:15 INFO Completed push id=8 objects=6 duration=460.334µs28912026/11/12 00:03:15 INFO Vacuumed table table=pending_closures28922026/11/12 00:03:15 INFO Completed push id=11 objects=8 duration=911.708µs28932026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028942026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=028952026/11/12 00:03:15 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28962026/11/12 00:03:15 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28972026/11/12 00:03:15 INFO Vacuumed table table=pending_objects28982026/11/12 00:03:15 INFO Vacuumed table table=pending_closures28992026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029002026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=029012026/11/12 00:03:15 INFO Received push request method=POST path=/29022026/11/12 00:03:15 INFO Received push request method=POST path=/29032026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads29042026/11/12 00:03:15 INFO Vacuumed table table=pending_closures29052026/11/12 00:03:15 INFO Vacuumed table table=pending_objects29062026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029072026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=029082026/11/12 00:03:15 INFO Vacuumed table table=closures29092026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=029102026/11/12 00:03:15 INFO Received uploads request method=POST path=/29112026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads29122026/11/12 00:03:15 INFO Vacuumed table table=pending_closures29132026/11/12 00:03:15 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29142026/11/12 00:03:15 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29152026/11/12 00:03:15 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29162026/11/12 00:03:15 INFO Vacuumed table table=closures29172026/11/12 00:03:15 INFO Vacuumed table table=objects29182026/11/12 00:03:15 INFO Vacuumed table table=pending_objects29192026/11/12 00:03:15 WARN Force mode enabled - objects will be deleted immediately without grace period29202026/11/12 00:03:15 INFO Received push request method=POST path=/29212026/11/12 00:03:15 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29222026/11/12 00:03:15 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29232026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads29242026/11/12 00:03:15 INFO Vacuumed table table=pending_objects29252026/11/12 00:03:15 INFO Vacuumed table table=objects29262026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029272026/11/12 00:03:15 INFO Received push request method=POST path=/29282026/11/12 00:03:15 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29292026/11/12 00:03:15 INFO Received uploads request method=POST path=/29302026/11/12 00:03:15 INFO Vacuumed table table=closures29312026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads29322026/11/12 00:03:15 INFO Vacuumed table table=pending_closures29332026/11/12 00:03:15 INFO Received push request method=POST path=/29342026/11/12 00:03:15 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29352026/11/12 00:03:15 INFO Vacuumed table table=closures29362026/11/12 00:03:15 INFO Vacuumed table table=pending_objects29372026/11/12 00:03:15 INFO Vacuumed table table=objects29382026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads29392026/11/12 00:03:15 INFO Vacuumed table table=objects29402026/11/12 00:03:15 INFO Vacuumed table table=closures29412026/11/12 00:03:15 INFO Received complete upload request method=POST path=/29422026/11/12 00:03:15 INFO Received push request method=POST path=/29432026/11/12 00:03:15 INFO Completed upload id=2 objects=2 duration=540.958µs29442026/11/12 00:03:15 INFO Vacuumed table table=objects29452026/11/12 00:03:15 INFO Received push request method=POST path=/29462026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=029472026/11/12 00:03:15 INFO Received uploads request method=POST path=/29482026/11/12 00:03:15 INFO Received uploads request method=POST path=/29492026/11/12 00:03:15 INFO Received complete push request method=POST path=/29502026/11/12 00:03:15 INFO Received push request method=POST path=/29512026/11/12 00:03:15 INFO Received uploads request method=POST path=/29522026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029532026/11/12 00:03:15 INFO Received push request method=POST path=/29542026/11/12 00:03:15 INFO Received uploads request method=POST path=/29552026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=029562026/11/12 00:03:15 INFO Received push request method=POST path=/29572026/11/12 00:03:15 INFO Vacuumed table table=pending_closures29582026/11/12 00:03:15 INFO Completed push id=1 objects=5 duration=17.728167ms29592026/11/12 00:03:15 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29602026/11/12 00:03:15 INFO Received uploads request method=POST path=/29612026/11/12 00:03:15 INFO Received complete push request method=POST path=/29622026/11/12 00:03:15 INFO Received push request method=POST path=/29632026/11/12 00:03:15 INFO Received push request method=POST path=/29642026/11/12 00:03:15 INFO Vacuumed table table=pending_objects29652026/11/12 00:03:15 INFO Received uploads request method=POST path=/29662026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads29672026/11/12 00:03:15 INFO Completed push id=26 objects=14 duration=1.1165ms29682026/11/12 00:03:15 INFO Received uploads request method=POST path=/29692026/11/12 00:03:15 INFO Received complete push request method=POST path=/29702026/11/12 00:03:15 INFO Vacuumed table table=closures29712026/11/12 00:03:15 INFO Received uploads request method=POST path=/29722026/11/12 00:03:15 INFO Vacuumed table table=objects29732026/11/12 00:03:15 INFO Completed push id=6 objects=5 duration=599.542µs29742026/11/12 00:03:15 INFO Received push request method=POST path=/29752026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=029762026/11/12 00:03:15 INFO Received push request method=POST path=/29772026/11/12 00:03:15 INFO Received push request method=POST path=/29782026/11/12 00:03:15 INFO Received uploads request method=POST path=/29792026/11/12 00:03:15 INFO Received complete upload request method=POST path=/29802026/11/12 00:03:15 INFO Received push request method=POST path=/29812026/11/12 00:03:15 INFO Completed upload id=10 objects=17 duration=313.542µs29822026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=029832026/11/12 00:03:15 INFO Received push request method=POST path=/29842026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029852026/11/12 00:03:15 INFO Vacuumed table table=pending_closures29862026/11/12 00:03:15 INFO Received uploads request method=POST path=/29872026/11/12 00:03:15 INFO Vacuumed table table=pending_objects29882026/11/12 00:03:15 INFO Received push request method=POST path=/29892026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads29902026/11/12 00:03:15 INFO Vacuumed table table=closures29912026/11/12 00:03:15 INFO Received complete upload request method=POST path=/29922026/11/12 00:03:15 INFO Vacuumed table table=objects29932026/11/12 00:03:15 INFO Received complete push request method=POST path=/29942026/11/12 00:03:15 INFO Received complete upload request method=POST path=/29952026/11/12 00:03:15 INFO Completed upload id=33 objects=5 duration=1.030167ms29962026/11/12 00:03:15 INFO Received push request method=POST path=/29972026/11/12 00:03:15 INFO Completed push id=14 objects=6 duration=465.667µs29982026/11/12 00:03:15 INFO Completed upload id=17 objects=6 duration=782.375µs29992026/11/12 00:03:15 INFO Received push request method=POST path=/30002026/11/12 00:03:15 INFO Received push request method=POST path=/30012026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030022026/11/12 00:03:15 INFO Received push request method=POST path=/30032026/11/12 00:03:15 INFO Received push request method=POST path=/30042026/11/12 00:03:15 INFO Received push request method=POST path=/30052026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030062026/11/12 00:03:15 INFO Received push request method=POST path=/30072026/11/12 00:03:15 INFO Received push request method=POST path=/30082026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030092026/11/12 00:03:15 INFO Vacuumed table table=pending_closures30102026/11/12 00:03:15 INFO Received uploads request method=POST path=/30112026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=030122026/11/12 00:03:15 INFO Vacuumed table table=pending_objects30132026/11/12 00:03:15 INFO Received push request method=POST path=/30142026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads30152026/11/12 00:03:15 INFO Vacuumed table table=pending_closures30162026/11/12 00:03:15 INFO Received push request method=POST path=/30172026/11/12 00:03:15 INFO Received uploads request method=POST path=/30182026/11/12 00:03:15 INFO Vacuumed table table=pending_objects30192026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads30202026/11/12 00:03:15 INFO Vacuumed table table=closures30212026/11/12 00:03:15 INFO Received complete upload request method=POST path=/30222026/11/12 00:03:15 INFO Completed upload id=15 objects=9 duration=393.5µs30232026/11/12 00:03:15 INFO Vacuumed table table=objects30242026/11/12 00:03:15 INFO Vacuumed table table=closures30252026/11/12 00:03:15 INFO Received complete push request method=POST path=/30262026/11/12 00:03:15 INFO Received push request method=POST path=/30272026/11/12 00:03:15 INFO Completed push id=13 objects=6 duration=1.034084ms30282026/11/12 00:03:15 INFO Received uploads request method=POST path=/30292026/11/12 00:03:15 INFO Received push request method=POST path=/30302026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030312026/11/12 00:03:15 INFO Received uploads request method=POST path=/30322026/11/12 00:03:15 INFO Received push request method=POST path=/30332026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030342026/11/12 00:03:15 INFO Received complete push request method=POST path=/30352026/11/12 00:03:15 INFO Received complete push request method=POST path=/30362026/11/12 00:03:15 INFO Vacuumed table table=objects30372026/11/12 00:03:15 INFO Vacuumed table table=pending_closures30382026/11/12 00:03:15 INFO Completed push id=12 objects=2 duration=826.208µs30392026/11/12 00:03:15 INFO Vacuumed table table=pending_objects30402026/11/12 00:03:15 INFO Completed push id=27 objects=25 duration=5.937666ms30412026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads30422026/11/12 00:03:15 INFO Received uploads request method=POST path=/30432026/11/12 00:03:15 INFO Received complete push request method=POST path=/30442026/11/12 00:03:15 INFO Vacuumed table table=closures30452026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030462026/11/12 00:03:15 INFO Completed push id=21 objects=17 duration=443.792µs30472026/11/12 00:03:15 INFO Vacuumed table table=objects30482026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030492026/11/12 00:03:15 INFO Vacuumed table table=pending_closures30502026/11/12 00:03:15 INFO Received uploads request method=POST path=/30512026/11/12 00:03:15 INFO Received complete push request method=POST path=/30522026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030532026/11/12 00:03:15 INFO Vacuumed table table=pending_objects30542026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030552026/11/12 00:03:15 INFO Received uploads request method=POST path=/30562026/11/12 00:03:15 INFO Completed push id=19 objects=7 duration=683.084µs30572026/11/12 00:03:15 WARN Force mode enabled - objects will be deleted immediately without grace period30582026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads30592026/11/12 00:03:15 WARN Force mode enabled - objects will be deleted immediately without grace period30602026/11/12 00:03:15 INFO Received complete push request method=POST path=/30612026/11/12 00:03:15 INFO Vacuumed table table=closures30622026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030632026/11/12 00:03:15 INFO Received push request method=POST path=/30642026/11/12 00:03:15 INFO Vacuumed table table=pending_closures30652026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030662026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030672026/11/12 00:03:15 INFO Vacuumed table table=objects30682026/11/12 00:03:15 INFO Vacuumed table table=pending_objects30692026/11/12 00:03:15 INFO Vacuumed table table=pending_closures30702026/11/12 00:03:15 INFO Received complete upload request method=POST path=/30712026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads30722026/11/12 00:03:15 INFO Completed push id=3 objects=5 duration=17.884167ms30732026/11/12 00:03:15 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030742026/11/12 00:03:15 INFO Received complete push request method=POST path=/30752026/11/12 00:03:15 INFO Completed upload id=5 objects=5 duration=431.125µs30762026/11/12 00:03:15 INFO Vacuumed table table=closures30772026/11/12 00:03:15 INFO Vacuumed table table=pending_objects30782026/11/12 00:03:15 INFO Vacuumed table table=pending_closures30792026/11/12 00:03:15 INFO Completed push id=24 objects=5 duration=834.792µs30802026/11/12 00:03:15 INFO Vacuumed table table=objects30812026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads30822026/11/12 00:03:15 INFO Aborted multipart uploads count=0 kept=030832026/11/12 00:03:15 INFO Received push request method=POST path=/30842026/11/12 00:03:15 INFO Received uploads request method=POST path=/30852026/11/12 00:03:15 INFO Vacuumed table table=closures30862026/11/12 00:03:15 INFO Vacuumed table table=pending_objects30872026/11/12 00:03:15 INFO Received uploads request method=POST path=/30882026/11/12 00:03:15 WARN Force mode enabled - objects will be deleted immediately without grace period30892026/11/12 00:03:15 INFO Vacuumed table table=multipart_uploads30902026/11/12 00:03:16 INFO Received uploads request method=POST path=/30912026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030922026/11/12 00:03:16 INFO Vacuumed table table=objects30932026/11/12 00:03:16 INFO Vacuumed table table=closures30942026/11/12 00:03:16 INFO Received push request method=POST path=/30952026/11/12 00:03:16 INFO Vacuumed table table=pending_closures30962026/11/12 00:03:16 INFO Received push request method=POST path=/30972026/11/12 00:03:16 INFO Vacuumed table table=objects30982026/11/12 00:03:16 INFO Received uploads request method=POST path=/30992026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31002026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031012026/11/12 00:03:16 INFO Received uploads request method=POST path=/31022026/11/12 00:03:16 INFO Received push request method=POST path=/31032026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31042026/11/12 00:03:16 INFO Vacuumed table table=closures31052026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031062026/11/12 00:03:16 INFO Received uploads request method=POST path=/31072026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31082026/11/12 00:03:16 INFO Vacuumed table table=objects31092026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031102026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031112026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31122026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31132026/11/12 00:03:16 INFO Received uploads request method=POST path=/31142026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031152026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031162026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=031172026/11/12 00:03:16 INFO Vacuumed table table=closures31182026/11/12 00:03:16 INFO Vacuumed table table=objects31192026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31202026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31212026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31222026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31232026/11/12 00:03:16 INFO Received push request method=POST path=/31242026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31252026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31262026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031272026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31282026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31292026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031302026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31312026/11/12 00:03:16 INFO Vacuumed table table=closures31322026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31332026/11/12 00:03:16 INFO Vacuumed table table=objects31342026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31352026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31362026/11/12 00:03:16 INFO Vacuumed table table=closures31372026/11/12 00:03:16 INFO Vacuumed table table=closures31382026/11/12 00:03:16 INFO Received complete push request method=POST path=/31392026/11/12 00:03:16 INFO Vacuumed table table=objects31402026/11/12 00:03:16 INFO Completed push id=22 objects=3 duration=6.195416ms31412026/11/12 00:03:16 INFO Vacuumed table table=objects31422026/11/12 00:03:16 INFO Vacuumed table table=closures31432026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031442026/11/12 00:03:16 WARN Force mode enabled - objects will be deleted immediately without grace period31452026/11/12 00:03:16 INFO Received push request method=POST path=/31462026/11/12 00:03:16 INFO Received complete upload request method=POST path=/31472026/11/12 00:03:16 INFO Vacuumed table table=objects31482026/11/12 00:03:16 INFO Received uploads request method=POST path=/31492026/11/12 00:03:16 INFO Completed upload id=21 objects=25 duration=378.25µs31502026/11/12 00:03:16 INFO Received uploads request method=POST path=/31512026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031522026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031532026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31542026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31552026/11/12 00:03:16 INFO Received complete upload request method=POST path=/31562026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31572026/11/12 00:03:16 INFO Received complete push request method=POST path=/31582026/11/12 00:03:16 INFO Vacuumed table table=closures31592026/11/12 00:03:16 INFO Completed upload id=12 objects=26 duration=877.125µs31602026/11/12 00:03:16 INFO Completed push id=21 objects=26 duration=367.834µs31612026/11/12 00:03:16 INFO Vacuumed table table=objects31622026/11/12 00:03:16 INFO Received uploads request method=POST path=/31632026/11/12 00:03:16 INFO Received complete push request method=POST path=/31642026/11/12 00:03:16 INFO Completed push id=16 objects=8 duration=498µs31652026/11/12 00:03:16 INFO Received push request method=POST path=/31662026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031672026/11/12 00:03:16 WARN Force mode enabled - objects will be deleted immediately without grace period31682026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031692026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31702026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31712026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31722026/11/12 00:03:16 INFO Vacuumed table table=closures31732026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=031742026/11/12 00:03:16 INFO Vacuumed table table=objects31752026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031762026/11/12 00:03:16 INFO Received complete push request method=POST path=/31772026/11/12 00:03:16 INFO Completed push id=17 objects=6 duration=661.417µs31782026/11/12 00:03:16 INFO Vacuumed table table=pending_closures31792026/11/12 00:03:16 INFO Received complete upload request method=POST path=/31802026/11/12 00:03:16 INFO Completed upload id=28 objects=17 duration=389.709µs31812026/11/12 00:03:16 INFO Received complete push request method=POST path=/31822026/11/12 00:03:16 INFO Vacuumed table table=pending_objects31832026/11/12 00:03:16 INFO Completed push id=29 objects=2 duration=6.034667ms31842026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads31852026/11/12 00:03:16 INFO Received complete push request method=POST path=/31862026/11/12 00:03:16 INFO Vacuumed table table=closures31872026/11/12 00:03:16 INFO Completed push id=31 objects=6 duration=6.215291ms31882026/11/12 00:03:16 INFO Received complete push request method=POST path=/31892026/11/12 00:03:16 INFO Vacuumed table table=objects31902026/11/12 00:03:16 INFO Completed push id=9 objects=7 duration=1.817541ms31912026/11/12 00:03:16 INFO Received complete push request method=POST path=/31922026/11/12 00:03:16 INFO Received complete push request method=POST path=/31932026/11/12 00:03:16 INFO Received complete push request method=POST path=/31942026/11/12 00:03:16 INFO Completed push id=32 objects=9 duration=397.041µs31952026/11/12 00:03:16 INFO Completed push id=11 objects=6 duration=1.259416ms31962026/11/12 00:03:16 INFO Completed push id=2 objects=25 duration=1.11875ms31972026/11/12 00:03:16 INFO Received push request method=POST path=/31982026/11/12 00:03:16 INFO Received uploads request method=POST path=/31992026/11/12 00:03:16 INFO Received push request method=POST path=/32002026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032012026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032022026/11/12 00:03:16 INFO Vacuumed table table=pending_closures32032026/11/12 00:03:16 INFO Received complete push request method=POST path=/32042026/11/12 00:03:16 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32052026/11/12 00:03:16 INFO Received uploads request method=POST path=/32062026/11/12 00:03:16 INFO Received uploads request method=POST path=/32072026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32082026/11/12 00:03:16 INFO Vacuumed table table=pending_objects32092026/11/12 00:03:16 INFO Completed upload id=33 objects=9 duration=304.667µs32102026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032112026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads32122026/11/12 00:03:16 INFO Completed push id=1 objects=7 duration=45.142792ms32132026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032142026/11/12 00:03:16 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."32152026/11/12 00:03:16 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."32162026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32172026/11/12 00:03:16 INFO Received uploads request method=POST path=/32182026/11/12 00:03:16 INFO Received complete push request method=POST path=/32192026/11/12 00:03:16 WARN Force mode enabled - objects will be deleted immediately without grace period32202026/11/12 00:03:16 INFO Completed upload id=10 objects=7 duration=502.083µs32212026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032222026/11/12 00:03:16 WARN Force mode enabled - objects will be deleted immediately without grace period32232026/11/12 00:03:16 INFO Vacuumed table table=closures32242026/11/12 00:03:16 INFO Completed push id=18 objects=6 duration=6.434625ms32252026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032262026/11/12 00:03:16 INFO Received uploads request method=POST path=/32272026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=032282026/11/12 00:03:16 INFO Vacuumed table table=pending_closures32292026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032302026/11/12 00:03:16 INFO Vacuumed table table=pending_objects32312026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads32322026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32332026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032342026/11/12 00:03:16 INFO Received uploads request method=POST path=/32352026/11/12 00:03:16 INFO Vacuumed table table=pending_closures32362026/11/12 00:03:16 INFO Vacuumed table table=closures32372026/11/12 00:03:16 INFO Completed upload id=12 objects=6 duration=284.792µs32382026/11/12 00:03:16 INFO Received complete push request method=POST path=/32392026/11/12 00:03:16 INFO Vacuumed table table=objects32402026/11/12 00:03:16 INFO Vacuumed table table=pending_closures32412026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32422026/11/12 00:03:16 INFO Vacuumed table table=pending_objects32432026/11/12 00:03:16 INFO Vacuumed table table=objects32442026/11/12 00:03:16 INFO Completed push id=4 objects=7 duration=625.583µs32452026/11/12 00:03:16 INFO Vacuumed table table=pending_objects32462026/11/12 00:03:16 INFO Received complete push request method=POST path=/32472026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads32482026/11/12 00:03:16 INFO Completed push id=17 objects=3 duration=555.958µs32492026/11/12 00:03:16 INFO Completed upload id=18 objects=4 duration=645.583µs32502026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads32512026/11/12 00:03:16 INFO Vacuumed table table=closures32522026/11/12 00:03:16 INFO Received uploads request method=POST path=/32532026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32542026/11/12 00:03:16 INFO Vacuumed table table=closures32552026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32562026/11/12 00:03:16 INFO Received push request method=POST path=/32572026/11/12 00:03:16 INFO Vacuumed table table=objects32582026/11/12 00:03:16 INFO Completed upload id=4 objects=7 duration=6.623667ms32592026/11/12 00:03:16 INFO Vacuumed table table=objects32602026/11/12 00:03:16 INFO Completed upload id=21 objects=7 duration=6.673458ms32612026/11/12 00:03:16 INFO Received uploads request method=POST path=/32622026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32632026/11/12 00:03:16 INFO Received push request method=POST path=/32642026/11/12 00:03:16 INFO Received push request method=POST path=/32652026/11/12 00:03:16 INFO Received push request method=POST path=/32662026/11/12 00:03:16 INFO Completed upload id=20 objects=3 duration=742µs32672026/11/12 00:03:16 INFO Received complete push request method=POST path=/32682026/11/12 00:03:16 INFO Received uploads request method=POST path=/32692026/11/12 00:03:16 INFO Completed push id=17 objects=22 duration=827.542µs32702026/11/12 00:03:16 INFO Received push request method=POST path=/32712026/11/12 00:03:16 INFO Received complete push request method=POST path=/32722026/11/12 00:03:16 INFO Completed push id=23 objects=5 duration=463.542µs32732026/11/12 00:03:16 INFO Received push request method=POST path=/32742026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032752026/11/12 00:03:16 INFO Received uploads request method=POST path=/32762026/11/12 00:03:16 INFO Received uploads request method=POST path=/32772026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32782026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032792026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32802026/11/12 00:03:16 INFO Received uploads request method=POST path=/32812026/11/12 00:03:16 INFO Completed upload id=24 objects=7 duration=333.167µs32822026/11/12 00:03:16 INFO Completed upload id=24 objects=3 duration=415.458µs32832026/11/12 00:03:16 INFO Received uploads request method=POST path=/32842026/11/12 00:03:16 INFO Received complete upload request method=POST path=/32852026/11/12 00:03:16 INFO Vacuumed table table=pending_closures32862026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032872026/11/12 00:03:16 INFO Received push request method=POST path=/32882026/11/12 00:03:16 INFO Received complete push request method=POST path=/32892026/11/12 00:03:16 INFO Completed upload id=27 objects=4 duration=390.625µs32902026/11/12 00:03:16 INFO Completed push id=25 objects=4 duration=5.768459ms32912026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032922026/11/12 00:03:16 INFO Vacuumed table table=pending_closures32932026/11/12 00:03:16 INFO Received complete push request method=POST path=/32942026/11/12 00:03:16 INFO Aborted multipart uploads count=0 kept=032952026/11/12 00:03:16 INFO Vacuumed table table=pending_objects32962026/11/12 00:03:16 INFO Completed push id=13 objects=11 duration=5.603292ms32972026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads32982026/11/12 00:03:16 INFO Vacuumed table table=pending_objects32992026/11/12 00:03:16 WARN Force mode enabled - objects will be deleted immediately without grace period33002026/11/12 00:03:16 INFO Vacuumed table table=closures33012026/11/12 00:03:16 INFO Vacuumed table table=multipart_uploads33022026/11/12 00:03:16 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033032026/11/12 00:03:16 INFO Received uploads request method=POST path=/33042026/11/12 00:03:16 INFO Vacuumed table table=objects33052026/11/12 00:03:16 INFO Vacuumed table table=closures33062026/11/12 00:03:16 INFO Received uploads request method=POST path=/33072026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033082026/11/12 00:03:17 INFO Vacuumed table table=objects33092026/11/12 00:03:17 INFO Received uploads request method=POST path=/33102026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033112026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period33122026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033132026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33142026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033152026/11/12 00:03:17 INFO Vacuumed table table=pending_objects33162026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33172026/11/12 00:03:17 INFO Received complete push request method=POST path=/33182026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period33192026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033202026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads33212026/11/12 00:03:17 INFO Completed push id=4 objects=4 duration=6.475291ms33222026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033232026/11/12 00:03:17 INFO Vacuumed table table=closures33242026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33252026/11/12 00:03:17 INFO Vacuumed table table=objects33262026/11/12 00:03:17 INFO Vacuumed table table=pending_objects33272026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads33282026/11/12 00:03:17 INFO Received uploads request method=POST path=/33292026/11/12 00:03:17 INFO Vacuumed table table=pending_objects33302026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33312026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads33322026/11/12 00:03:17 INFO Received complete upload request method=POST path=/33332026/11/12 00:03:17 INFO Vacuumed table table=closures33342026/11/12 00:03:17 INFO Vacuumed table table=objects33352026/11/12 00:03:17 INFO Vacuumed table table=closures33362026/11/12 00:03:17 INFO Vacuumed table table=pending_objects33372026/11/12 00:03:17 INFO Received complete upload request method=POST path=/33382026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads33392026/11/12 00:03:17 INFO Completed upload id=3 objects=6 duration=45.961958ms33402026/11/12 00:03:17 INFO Vacuumed table table=closures33412026/11/12 00:03:17 INFO Completed upload id=2 objects=10 duration=532.125µs33422026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033432026/11/12 00:03:17 INFO Received push request method=POST path=/33442026/11/12 00:03:17 INFO Vacuumed table table=objects33452026/11/12 00:03:17 INFO Received complete push request method=POST path=/33462026/11/12 00:03:17 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.narinfo error="Access Denied."33472026/11/12 00:03:17 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."33482026/11/12 00:03:17 INFO Completed push id=16 objects=14 duration=1.03275ms33492026/11/12 00:03:17 INFO Vacuumed table table=objects33502026/11/12 00:03:17 INFO Received uploads request method=POST path=/33512026/11/12 00:03:17 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."33522026/11/12 00:03:17 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."33532026/11/12 00:03:17 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."33542026/11/12 00:03:17 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."33552026/11/12 00:03:17 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33562026/11/12 00:03:17 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."33572026/11/12 00:03:17 ERROR failed to remove object object=nar/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst error="Access Denied."33582026/11/12 00:03:17 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."33592026/11/12 00:03:17 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."33602026/11/12 00:03:17 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."33612026/11/12 00:03:17 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."33622026/11/12 00:03:17 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33632026/11/12 00:03:17 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."33642026/11/12 00:03:17 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."33652026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033662026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033672026/11/12 00:03:17 INFO Received uploads request method=POST path=/33682026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=033692026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period33702026/11/12 00:03:17 INFO Received uploads request method=POST path=/33712026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33722026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033732026/11/12 00:03:17 INFO Received uploads request method=POST path=/33742026/11/12 00:03:17 INFO Vacuumed table table=pending_objects33752026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33762026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033772026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads33782026/11/12 00:03:17 INFO Received complete push request method=POST path=/33792026/11/12 00:03:17 INFO Vacuumed table table=pending_objects33802026/11/12 00:03:17 INFO Completed push id=6 objects=6 duration=634.875µs33812026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads33822026/11/12 00:03:17 INFO Received complete push request method=POST path=/33832026/11/12 00:03:17 INFO Vacuumed table table=closures33842026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period33852026/11/12 00:03:17 INFO Completed push id=23 objects=28 duration=2.202958ms33862026/11/12 00:03:17 INFO Received push request method=POST path=/33872026/11/12 00:03:17 INFO Vacuumed table table=objects33882026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=033892026/11/12 00:03:17 INFO Received complete upload request method=POST path=/33902026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=033912026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33922026/11/12 00:03:17 INFO Completed upload id=27 objects=8 duration=534.625µs33932026/11/12 00:03:17 INFO Vacuumed table table=closures33942026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033952026/11/12 00:03:17 INFO Vacuumed table table=pending_closures33962026/11/12 00:03:17 INFO Received complete push request method=POST path=/33972026/11/12 00:03:17 INFO Received push request method=POST path=/33982026/11/12 00:03:17 INFO Vacuumed table table=pending_objects33992026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34002026/11/12 00:03:17 INFO Completed push id=5 objects=9 duration=5.005208ms34012026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34022026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=034032026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34042026/11/12 00:03:17 INFO Vacuumed table table=closures34052026/11/12 00:03:17 INFO Vacuumed table table=closures34062026/11/12 00:03:17 INFO Vacuumed table table=objects34072026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034082026/11/12 00:03:17 INFO Received push request method=POST path=/34092026/11/12 00:03:17 INFO Vacuumed table table=objects34102026/11/12 00:03:17 INFO Vacuumed table table=objects34112026/11/12 00:03:17 INFO Vacuumed table table=pending_closures34122026/11/12 00:03:17 INFO Received complete upload request method=POST path=/34132026/11/12 00:03:17 INFO Received push request method=POST path=/34142026/11/12 00:03:17 INFO Completed upload id=10 objects=6 duration=1.340959ms34152026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34162026/11/12 00:03:17 INFO Received push request method=POST path=/34172026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34182026/11/12 00:03:17 INFO Received uploads request method=POST path=/34192026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=034202026/11/12 00:03:17 INFO Vacuumed table table=closures34212026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period34222026/11/12 00:03:17 INFO Vacuumed table table=objects34232026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034242026/11/12 00:03:17 INFO Received uploads request method=POST path=/34252026/11/12 00:03:17 INFO Received uploads request method=POST path=/34262026/11/12 00:03:17 INFO Vacuumed table table=pending_closures34272026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=034282026/11/12 00:03:17 INFO Received complete upload request method=POST path=/34292026/11/12 00:03:17 INFO Received push request method=POST path=/34302026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34312026/11/12 00:03:17 INFO Completed upload id=25 objects=6 duration=6.251583ms34322026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34332026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034342026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=034352026/11/12 00:03:17 INFO Vacuumed table table=pending_closures34362026/11/12 00:03:17 INFO Vacuumed table table=closures34372026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period34382026/11/12 00:03:17 INFO Received push request method=POST path=/34392026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34402026/11/12 00:03:17 INFO Vacuumed table table=objects34412026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034422026/11/12 00:03:17 INFO Received uploads request method=POST path=/34432026/11/12 00:03:17 INFO Received uploads request method=POST path=/34442026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34452026/11/12 00:03:17 INFO Vacuumed table table=pending_closures34462026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=034472026/11/12 00:03:17 INFO Received complete push request method=POST path=/34482026/11/12 00:03:17 INFO Vacuumed table table=closures34492026/11/12 00:03:17 INFO Completed push id=11 objects=9 duration=392.791µs34502026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34512026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034522026/11/12 00:03:17 INFO Vacuumed table table=objects34532026/11/12 00:03:17 INFO Vacuumed table table=pending_closures34542026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34552026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34562026/11/12 00:03:17 INFO Received complete upload request method=POST path=/34572026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34582026/11/12 00:03:17 INFO Completed upload id=25 objects=16 duration=1.198334ms34592026/11/12 00:03:17 INFO Received push request method=POST path=/34602026/11/12 00:03:17 INFO Vacuumed table table=closures34612026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=034622026/11/12 00:03:17 INFO Received push request method=POST path=/34632026/11/12 00:03:17 INFO Vacuumed table table=objects34642026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period34652026/11/12 00:03:17 INFO Received complete push request method=POST path=/34662026/11/12 00:03:17 INFO Vacuumed table table=closures34672026/11/12 00:03:17 INFO Received uploads request method=POST path=/34682026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034692026/11/12 00:03:17 INFO Completed push id=3 objects=5 duration=457.333µs34702026/11/12 00:03:17 INFO Vacuumed table table=objects34712026/11/12 00:03:17 INFO Received push request method=POST path=/34722026/11/12 00:03:17 INFO Vacuumed table table=pending_closures34732026/11/12 00:03:17 INFO Received complete push request method=POST path=/34742026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34752026/11/12 00:03:17 INFO Received push request method=POST path=/34762026/11/12 00:03:17 INFO Completed push id=7 objects=27 duration=563.25µs34772026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34782026/11/12 00:03:17 INFO Received uploads request method=POST path=/34792026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=034802026/11/12 00:03:17 INFO Vacuumed table table=closures34812026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period34822026/11/12 00:03:17 INFO Vacuumed table table=objects34832026/11/12 00:03:17 INFO Received complete push request method=POST path=/34842026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034852026/11/12 00:03:17 INFO Completed push id=15 objects=24 duration=507.542µs34862026/11/12 00:03:17 INFO Received push request method=POST path=/34872026/11/12 00:03:17 INFO Vacuumed table table=pending_closures34882026/11/12 00:03:17 INFO Received complete upload request method=POST path=/34892026/11/12 00:03:17 INFO Received complete push request method=POST path=/34902026/11/12 00:03:17 INFO Received complete upload request method=POST path=/34912026/11/12 00:03:17 INFO Received complete push request method=POST path=/34922026/11/12 00:03:17 INFO Received push request method=POST path=/34932026/11/12 00:03:17 INFO Vacuumed table table=pending_objects34942026/11/12 00:03:17 INFO Completed upload id=19 objects=2 duration=328.875µs34952026/11/12 00:03:17 INFO Vacuumed table table=multipart_uploads34962026/11/12 00:03:17 INFO Completed push id=26 objects=23 duration=666.792µs34972026/11/12 00:03:17 INFO Completed push id=7 objects=7 duration=616.542µs34982026/11/12 00:03:17 INFO Completed upload id=6 objects=24 duration=1.358708ms34992026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35002026/11/12 00:03:17 INFO Vacuumed table table=closures35012026/11/12 00:03:17 INFO Completed upload id=24 objects=7 duration=576.25µs35022026/11/12 00:03:17 INFO Received uploads request method=POST path=/35032026/11/12 00:03:17 INFO Received push request method=POST path=/35042026/11/12 00:03:17 INFO Received uploads request method=POST path=/35052026/11/12 00:03:17 INFO Vacuumed table table=objects35062026/11/12 00:03:17 INFO Received complete push request method=POST path=/35072026/11/12 00:03:17 INFO Completed push id=26 objects=3 duration=301.333µs35082026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35092026/11/12 00:03:17 INFO Completed upload id=22 objects=5 duration=697.625µs35102026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35112026/11/12 00:03:17 INFO Completed upload id=28 objects=8 duration=230.334µs35122026/11/12 00:03:17 INFO Received complete push request method=POST path=/35132026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35142026/11/12 00:03:17 INFO Completed push id=28 objects=3 duration=540.958µs35152026/11/12 00:03:17 INFO Completed upload id=29 objects=10 duration=230.042µs35162026/11/12 00:03:17 INFO Received complete push request method=POST path=/35172026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35182026/11/12 00:03:17 INFO Completed push id=29 objects=15 duration=404.125µs35192026/11/12 00:03:17 INFO Completed upload id=30 objects=20 duration=248.458µs35202026/11/12 00:03:17 INFO Received complete push request method=POST path=/35212026/11/12 00:03:17 INFO Completed push id=30 objects=6 duration=428.208µs35222026/11/12 00:03:17 INFO Received complete push request method=POST path=/35232026/11/12 00:03:17 INFO Received complete push request method=POST path=/35242026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35252026/11/12 00:03:17 INFO Completed push id=4 objects=4 duration=618.333µs35262026/11/12 00:03:17 INFO Completed push id=31 objects=11 duration=491.5µs35272026/11/12 00:03:17 INFO Completed upload id=31 objects=3 duration=259.917µs35282026/11/12 00:03:17 INFO Received complete push request method=POST path=/35292026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35302026/11/12 00:03:17 INFO Completed push id=8 objects=6 duration=6.411875ms35312026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=035322026/11/12 00:03:17 INFO Completed upload id=32 objects=2 duration=228.291µs35332026/11/12 00:03:17 INFO Received push request method=POST path=/35342026/11/12 00:03:17 WARN Force mode enabled - objects will be deleted immediately without grace period35352026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035362026/11/12 00:03:17 INFO Received complete push request method=POST path=/35372026/11/12 00:03:17 INFO Completed push id=33 objects=3 duration=292.292µs35382026/11/12 00:03:17 INFO Received complete push request method=POST path=/35392026/11/12 00:03:17 INFO Completed push id=5 objects=3 duration=553.458µs35402026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35412026/11/12 00:03:17 INFO Received complete upload request method=POST path=/35422026/11/12 00:03:17 INFO Aborted multipart uploads count=0 kept=035432026/11/12 00:03:17 INFO Vacuumed table table=pending_closures35442026/11/12 00:03:17 INFO Completed upload id=34 objects=3 duration=408.916µs35452026/11/12 00:03:17 INFO Completed upload id=3 objects=9 duration=375.667µs35462026/11/12 00:03:17 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=035472026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=035482026/11/12 00:03:18 INFO Vacuumed table table=pending_objects35492026/11/12 00:03:18 WARN Force mode enabled - objects will be deleted immediately without grace period35502026/11/12 00:03:18 INFO Received complete push request method=POST path=/35512026/11/12 00:03:18 INFO Vacuumed table table=multipart_uploads35522026/11/12 00:03:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035532026/11/12 00:03:18 INFO Completed push id=11 objects=11 duration=1.13575ms35542026/11/12 00:03:18 INFO Vacuumed table table=closures35552026/11/12 00:03:18 INFO Received complete upload request method=POST path=/35562026/11/12 00:03:18 INFO Received complete upload request method=POST path=/35572026/11/12 00:03:18 INFO Vacuumed table table=objects35582026/11/12 00:03:18 INFO Completed upload id=9 objects=3 duration=1.136708ms35592026/11/12 00:03:18 INFO Completed upload id=11 objects=6 duration=1.155833ms35602026/11/12 00:03:18 INFO Vacuumed table table=pending_closures35612026/11/12 00:03:18 INFO Received push request method=POST path=/35622026/11/12 00:03:18 INFO Received uploads request method=POST path=/35632026/11/12 00:03:18 INFO Vacuumed table table=pending_closures35642026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=035652026/11/12 00:03:18 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."35662026/11/12 00:03:18 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35672026/11/12 00:03:18 INFO Received uploads request method=POST path=/35682026/11/12 00:03:18 INFO Received uploads request method=POST path=/35692026/11/12 00:03:18 INFO Received push request method=POST path=/35702026/11/12 00:03:18 INFO Received complete upload request method=POST path=/35712026/11/12 00:03:18 INFO Vacuumed table table=pending_objects35722026/11/12 00:03:18 INFO Received uploads request method=POST path=/35732026/11/12 00:03:18 INFO Completed upload id=7 objects=10 duration=456.25µs35742026/11/12 00:03:18 INFO Received complete push request method=POST path=/35752026/11/12 00:03:18 INFO Vacuumed table table=multipart_uploads35762026/11/12 00:03:18 INFO Received complete push request method=POST path=/35772026/11/12 00:03:18 INFO Completed push id=12 objects=3 duration=591µs35782026/11/12 00:03:18 INFO Received push request method=POST path=/35792026/11/12 00:03:18 INFO Completed push id=9 objects=5 duration=660.917µs35802026/11/12 00:03:18 INFO Vacuumed table table=closures35812026/11/12 00:03:18 INFO Received complete upload request method=POST path=/35822026/11/12 00:03:18 INFO Vacuumed table table=pending_objects35832026/11/12 00:03:18 INFO Vacuumed table table=objects35842026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=035852026/11/12 00:03:18 INFO Received uploads request method=POST path=/35862026/11/12 00:03:18 INFO Completed upload id=18 objects=11 duration=403.25µs35872026/11/12 00:03:18 INFO Vacuumed table table=multipart_uploads35882026/11/12 00:03:18 INFO Received complete push request method=POST path=/35892026/11/12 00:03:18 WARN Force mode enabled - objects will be deleted immediately without grace period35902026/11/12 00:03:18 INFO Received uploads request method=POST path=/35912026/11/12 00:03:18 INFO Received uploads request method=POST path=/35922026/11/12 00:03:18 INFO Received complete push request method=POST path=/35932026/11/12 00:03:18 INFO Completed push id=5 objects=6 duration=489.667µs35942026/11/12 00:03:18 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035952026/11/12 00:03:18 INFO Vacuumed table table=pending_closures35962026/11/12 00:03:18 INFO Completed push id=27 objects=15 duration=8.130292ms35972026/11/12 00:03:18 INFO Vacuumed table table=pending_objects35982026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=035992026/11/12 00:03:18 INFO Received push request method=POST path=/36002026/11/12 00:03:18 INFO Received push request method=POST path=/36012026/11/12 00:03:18 INFO Vacuumed table table=multipart_uploads36022026/11/12 00:03:18 INFO Vacuumed table table=closures36032026/11/12 00:03:18 WARN Force mode enabled - objects will be deleted immediately without grace period36042026/11/12 00:03:18 INFO Vacuumed table table=closures36052026/11/12 00:03:18 INFO Vacuumed table table=objects36062026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=036072026/11/12 00:03:18 WARN Force mode enabled - objects will be deleted immediately without grace period36082026/11/12 00:03:18 INFO Received push request method=POST path=/36092026/11/12 00:03:18 INFO Vacuumed table table=objects36102026/11/12 00:03:18 INFO Received complete push request method=POST path=/36112026/11/12 00:03:18 INFO Received push request method=POST path=/36122026/11/12 00:03:18 INFO Completed push id=1 objects=11 duration=7.623125ms36132026/11/12 00:03:18 INFO Received uploads request method=POST path=/36142026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36152026/11/12 00:03:18 INFO Received complete push request method=POST path=/36162026/11/12 00:03:18 INFO Completed upload id=16 objects=23 duration=351µs36172026/11/12 00:03:18 INFO Received push request method=POST path=/36182026/11/12 00:03:18 INFO Completed push id=2 objects=7 duration=1.323917ms36192026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36202026/11/12 00:03:18 INFO Completed upload id=15 objects=6 duration=514.875µs36212026/11/12 00:03:18 INFO Received uploads request method=POST path=/36222026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=036232026/11/12 00:03:18 INFO Received uploads request method=POST path=/36242026/11/12 00:03:18 INFO Received uploads request method=POST path=/36252026/11/12 00:03:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036262026/11/12 00:03:18 INFO Received complete push request method=POST path=/36272026/11/12 00:03:18 INFO Received uploads request method=POST path=/36282026/11/12 00:03:18 INFO Vacuumed table table=pending_closures36292026/11/12 00:03:18 INFO Completed push id=3 objects=8 duration=25.42975ms36302026/11/12 00:03:18 INFO Vacuumed table table=pending_objects36312026/11/12 00:03:18 INFO Received push request method=POST path=/36322026/11/12 00:03:18 INFO Vacuumed table table=multipart_uploads36332026/11/12 00:03:18 INFO Received complete push request method=POST path=/36342026/11/12 00:03:18 INFO Completed push id=4 objects=7 duration=439.333µs36352026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36362026/11/12 00:03:18 INFO Vacuumed table table=closures36372026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36382026/11/12 00:03:18 INFO Received uploads request method=POST path=/36392026/11/12 00:03:18 INFO Completed upload id=17 objects=7 duration=320.375µs36402026/11/12 00:03:18 INFO Completed upload id=16 objects=3 duration=423.834µs36412026/11/12 00:03:18 INFO Received uploads request method=POST path=/36422026/11/12 00:03:18 INFO Vacuumed table table=objects36432026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=036442026/11/12 00:03:18 INFO Received push request method=POST path=/36452026/11/12 00:03:18 INFO Received push request method=POST path=/36462026/11/12 00:03:18 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."36472026/11/12 00:03:18 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."36482026/11/12 00:03:18 INFO Received uploads request method=POST path=/36492026/11/12 00:03:18 INFO Received push request method=POST path=/36502026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=036512026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=036522026/11/12 00:03:18 WARN Force mode enabled - objects will be deleted immediately without grace period36532026/11/12 00:03:18 WARN Force mode enabled - objects will be deleted immediately without grace period36542026/11/12 00:03:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036552026/11/12 00:03:18 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."36562026/11/12 00:03:18 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."36572026/11/12 00:03:18 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."36582026/11/12 00:03:18 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."36592026/11/12 00:03:18 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."36602026/11/12 00:03:18 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."36612026/11/12 00:03:18 INFO Vacuumed table table=pending_closures36622026/11/12 00:03:18 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."36632026/11/12 00:03:18 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."36642026/11/12 00:03:18 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."36652026/11/12 00:03:18 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."36662026/11/12 00:03:18 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."36672026/11/12 00:03:18 INFO Received push request method=POST path=/36682026/11/12 00:03:18 INFO Received complete push request method=POST path=/36692026/11/12 00:03:18 INFO Completed push id=8 objects=2 duration=442.5µs36702026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36712026/11/12 00:03:18 INFO Vacuumed table table=pending_objects36722026/11/12 00:03:18 INFO Received uploads request method=POST path=/36732026/11/12 00:03:18 INFO Completed upload id=14 objects=6 duration=277.291µs36742026/11/12 00:03:18 INFO Vacuumed table table=multipart_uploads36752026/11/12 00:03:18 INFO Received complete push request method=POST path=/36762026/11/12 00:03:18 INFO Vacuumed table table=closures36772026/11/12 00:03:18 INFO Completed push id=4 objects=6 duration=620.167µs36782026/11/12 00:03:18 INFO Received push request method=POST path=/36792026/11/12 00:03:18 INFO Vacuumed table table=objects36802026/11/12 00:03:18 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."36812026/11/12 00:03:18 INFO Received uploads request method=POST path=/36822026/11/12 00:03:18 INFO Received uploads request method=POST path=/36832026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36842026/11/12 00:03:18 INFO Completed upload id=7 objects=3 duration=373.875µs36852026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36862026/11/12 00:03:18 INFO Received complete push request method=POST path=/36872026/11/12 00:03:18 INFO Completed upload id=10 objects=6 duration=936.208µs36882026/11/12 00:03:18 INFO Completed push id=15 objects=4 duration=359.042µs36892026/11/12 00:03:18 INFO Received push request method=POST path=/36902026/11/12 00:03:18 INFO Received push request method=POST path=/36912026/11/12 00:03:18 INFO Received uploads request method=POST path=/36922026/11/12 00:03:18 INFO Received complete push request method=POST path=/36932026/11/12 00:03:18 INFO Received complete push request method=POST path=/36942026/11/12 00:03:18 INFO Received complete upload request method=POST path=/36952026/11/12 00:03:18 INFO Completed push id=28 objects=19 duration=424.084µs36962026/11/12 00:03:18 INFO Completed upload id=9 objects=2 duration=469.333µs36972026/11/12 00:03:18 INFO Completed push id=8 objects=6 duration=694.708µs36982026/11/12 00:03:18 INFO Received complete push request method=POST path=/36992026/11/12 00:03:18 INFO Completed push id=8 objects=3 duration=538.334µs37002026/11/12 00:03:18 INFO Received push request method=POST path=/37012026/11/12 00:03:18 INFO Received uploads request method=POST path=/37022026/11/12 00:03:18 INFO Received complete upload request method=POST path=/37032026/11/12 00:03:18 INFO Completed upload id=13 objects=10 duration=298.792µs37042026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=037052026/11/12 00:03:18 WARN Force mode enabled - objects will be deleted immediately without grace period37062026/11/12 00:03:18 INFO Received complete push request method=POST path=/37072026/11/12 00:03:18 INFO Received complete push request method=POST path=/37082026/11/12 00:03:18 INFO Completed push id=18 objects=23 duration=829.542µs37092026/11/12 00:03:18 INFO Completed push id=12 objects=7 duration=5.920833ms37102026/11/12 00:03:18 INFO Received complete upload request method=POST path=/37112026/11/12 00:03:18 INFO Aborted multipart uploads count=0 kept=037122026/11/12 00:03:18 INFO Received uploads request method=POST path=/37132026/11/12 00:03:18 INFO Completed upload id=11 objects=7 duration=388.958µs37142026/11/12 00:03:18 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=037152026/11/12 00:03:18 INFO Received uploads request method=POST path=/37162026/11/12 00:03:18 INFO Received uploads request method=POST path=/37172026/11/12 00:03:19 INFO Vacuumed table table=pending_closures37182026/11/12 00:03:19 INFO Received complete push request method=POST path=/37192026/11/12 00:03:19 INFO Completed push id=29 objects=10 duration=430.541µs37202026/11/12 00:03:19 INFO Vacuumed table table=pending_objects37212026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads37222026/11/12 00:03:19 INFO Vacuumed table table=closures37232026/11/12 00:03:19 INFO Vacuumed table table=objects37242026/11/12 00:03:19 INFO Received complete push request method=POST path=/37252026/11/12 00:03:19 INFO Completed push id=11 objects=9 duration=867.458µs37262026/11/12 00:03:19 INFO Received push request method=POST path=/37272026/11/12 00:03:19 INFO Received uploads request method=POST path=/37282026/11/12 00:03:19 INFO Received push request method=POST path=/37292026/11/12 00:03:19 INFO Received complete push request method=POST path=/37302026/11/12 00:03:19 INFO Received uploads request method=POST path=/37312026/11/12 00:03:19 INFO Received uploads request method=POST path=/37322026/11/12 00:03:19 INFO Received uploads request method=POST path=/37332026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=037342026/11/12 00:03:19 WARN Force mode enabled - objects will be deleted immediately without grace period37352026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037362026/11/12 00:03:19 INFO Completed push id=1 objects=10 duration=29.858666ms37372026/11/12 00:03:19 INFO Received uploads request method=POST path=/37382026/11/12 00:03:19 INFO Received complete push request method=POST path=/37392026/11/12 00:03:19 INFO Vacuumed table table=pending_closures37402026/11/12 00:03:19 INFO Received uploads request method=POST path=/37412026/11/12 00:03:19 INFO Completed push id=14 objects=3 duration=669.334µs37422026/11/12 00:03:19 INFO Vacuumed table table=pending_objects37432026/11/12 00:03:19 INFO Received push request method=POST path=/37442026/11/12 00:03:19 INFO Received push request method=POST path=/37452026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads37462026/11/12 00:03:19 INFO Received complete upload request method=POST path=/37472026/11/12 00:03:19 INFO Completed upload id=20 objects=4 duration=283.625µs37482026/11/12 00:03:19 INFO Vacuumed table table=closures37492026/11/12 00:03:19 INFO Received complete upload request method=POST path=/37502026/11/12 00:03:19 INFO Received push request method=POST path=/37512026/11/12 00:03:19 INFO Completed upload id=30 objects=6 duration=506.667µs37522026/11/12 00:03:19 INFO Vacuumed table table=objects37532026/11/12 00:03:19 INFO Received uploads request method=POST path=/37542026/11/12 00:03:19 INFO Received push request method=POST path=/37552026/11/12 00:03:19 INFO Received uploads request method=POST path=/37562026/11/12 00:03:19 INFO Received complete upload request method=POST path=/37572026/11/12 00:03:19 INFO Received complete upload request method=POST path=/37582026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=037592026/11/12 00:03:19 INFO Received complete push request method=POST path=/37602026/11/12 00:03:19 INFO Completed upload id=31 objects=5 duration=443.541µs37612026/11/12 00:03:19 INFO Received push request method=POST path=/37622026/11/12 00:03:19 INFO Completed upload id=3 objects=5 duration=434.417µs37632026/11/12 00:03:19 INFO Completed push id=4 objects=6 duration=705µs37642026/11/12 00:03:19 WARN Force mode enabled - objects will be deleted immediately without grace period37652026/11/12 00:03:19 INFO Received complete upload request method=POST path=/37662026/11/12 00:03:19 INFO Received complete upload request method=POST path=/37672026/11/12 00:03:19 INFO Received push request method=POST path=/37682026/11/12 00:03:19 INFO Completed upload id=34 objects=3 duration=304.292µs37692026/11/12 00:03:19 INFO Completed upload id=23 objects=15 duration=1.095208ms37702026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037712026/11/12 00:03:19 INFO Received uploads request method=POST path=/37722026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=037732026/11/12 00:03:19 INFO Received push request method=POST path=/37742026/11/12 00:03:19 WARN Force mode enabled - objects will be deleted immediately without grace period37752026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=037762026/11/12 00:03:19 INFO Vacuumed table table=pending_closures37772026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037782026/11/12 00:03:19 WARN Force mode enabled - objects will be deleted immediately without grace period37792026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037802026/11/12 00:03:19 INFO Vacuumed table table=pending_objects37812026/11/12 00:03:19 INFO Vacuumed table table=pending_closures37822026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads37832026/11/12 00:03:19 INFO Vacuumed table table=closures37842026/11/12 00:03:19 INFO Vacuumed table table=pending_closures37852026/11/12 00:03:19 INFO Vacuumed table table=pending_objects37862026/11/12 00:03:19 INFO Vacuumed table table=objects37872026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads37882026/11/12 00:03:19 INFO Vacuumed table table=closures37892026/11/12 00:03:19 INFO Vacuumed table table=objects37902026/11/12 00:03:19 INFO Vacuumed table table=pending_objects37912026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads37922026/11/12 00:03:19 INFO Received uploads request method=POST path=/37932026/11/12 00:03:19 INFO Vacuumed table table=closures37942026/11/12 00:03:19 INFO Vacuumed table table=objects37952026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=037962026/11/12 00:03:19 WARN Force mode enabled - objects will be deleted immediately without grace period37972026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037982026/11/12 00:03:19 INFO Vacuumed table table=pending_closures37992026/11/12 00:03:19 INFO Received complete upload request method=POST path=/38002026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38012026/11/12 00:03:19 INFO Completed upload id=19 objects=6 duration=494.875µs38022026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38032026/11/12 00:03:19 INFO Vacuumed table table=closures38042026/11/12 00:03:19 INFO Vacuumed table table=objects38052026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038062026/11/12 00:03:19 WARN Force mode enabled - objects will be deleted immediately without grace period38072026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038082026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38092026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38102026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38112026/11/12 00:03:19 INFO Vacuumed table table=closures38122026/11/12 00:03:19 INFO Received complete push request method=POST path=/38132026/11/12 00:03:19 INFO Vacuumed table table=objects38142026/11/12 00:03:19 INFO Completed push id=16 objects=5 duration=605.541µs38152026/11/12 00:03:19 INFO Received uploads request method=POST path=/38162026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038172026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=038182026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38192026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38202026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38212026/11/12 00:03:19 INFO Vacuumed table table=closures38222026/11/12 00:03:19 INFO Vacuumed table table=objects38232026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038242026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038252026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38262026/11/12 00:03:19 INFO Received complete push request method=POST path=/38272026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38282026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38292026/11/12 00:03:19 INFO Completed push id=4 objects=3 duration=10.570167ms38302026/11/12 00:03:19 INFO Vacuumed table table=closures38312026/11/12 00:03:19 INFO Vacuumed table table=objects38322026/11/12 00:03:19 INFO Received push request method=POST path=/38332026/11/12 00:03:19 INFO Received uploads request method=POST path=/38342026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=038352026/11/12 00:03:19 INFO Received push request method=POST path=/38362026/11/12 00:03:19 INFO Received complete upload request method=POST path=/38372026/11/12 00:03:19 INFO Received uploads request method=POST path=/38382026/11/12 00:03:19 INFO Received push request method=POST path=/38392026/11/12 00:03:19 INFO Completed upload id=22 objects=7 duration=352.667µs38402026/11/12 00:03:19 INFO Received push request method=POST path=/38412026/11/12 00:03:19 INFO Received uploads request method=POST path=/38422026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038432026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38442026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038452026/11/12 00:03:19 INFO Received complete push request method=POST path=/38462026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38472026/11/12 00:03:19 INFO Received complete push request method=POST path=/38482026/11/12 00:03:19 INFO Completed push id=24 objects=9 duration=581.792µs38492026/11/12 00:03:19 INFO Completed push id=19 objects=8 duration=1.405375ms38502026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38512026/11/12 00:03:19 INFO Received uploads request method=POST path=/38522026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38532026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38542026/11/12 00:03:19 INFO Received push request method=POST path=/38552026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038562026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38572026/11/12 00:03:19 INFO Vacuumed table table=closures38582026/11/12 00:03:19 INFO Received push request method=POST path=/38592026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038602026/11/12 00:03:19 INFO Vacuumed table table=closures38612026/11/12 00:03:19 INFO Vacuumed table table=objects38622026/11/12 00:03:19 INFO Vacuumed table table=objects38632026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38642026/11/12 00:03:19 INFO Received uploads request method=POST path=/38652026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38662026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038672026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38682026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038692026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038702026/11/12 00:03:19 INFO Received complete push request method=POST path=/38712026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38722026/11/12 00:03:19 INFO Completed push id=21 objects=3 duration=388.084µs38732026/11/12 00:03:19 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038742026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38752026/11/12 00:03:19 INFO Vacuumed table table=closures38762026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38772026/11/12 00:03:19 INFO Vacuumed table table=pending_closures38782026/11/12 00:03:19 INFO Vacuumed table table=objects38792026/11/12 00:03:19 INFO Received uploads request method=POST path=/38802026/11/12 00:03:19 INFO Vacuumed table table=closures38812026/11/12 00:03:19 INFO Vacuumed table table=pending_objects38822026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038832026/11/12 00:03:19 INFO Vacuumed table table=objects38842026/11/12 00:03:19 INFO Aborted multipart uploads count=0 kept=038852026/11/12 00:03:19 INFO Vacuumed table table=multipart_uploads38862026/11/12 00:03:20 INFO Vacuumed table table=closures38872026/11/12 00:03:20 INFO Received complete upload request method=POST path=/38882026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period38892026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038902026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038912026/11/12 00:03:20 INFO Received push request method=POST path=/38922026/11/12 00:03:20 INFO Vacuumed table table=objects38932026/11/12 00:03:20 INFO Vacuumed table table=pending_closures38942026/11/12 00:03:20 INFO Received complete upload request method=POST path=/38952026/11/12 00:03:20 INFO Vacuumed table table=pending_closures38962026/11/12 00:03:20 INFO Received push request method=POST path=/38972026/11/12 00:03:20 INFO Completed upload id=2 objects=26 duration=14.14925ms38982026/11/12 00:03:20 INFO Received push request method=POST path=/38992026/11/12 00:03:20 INFO Completed upload id=17 objects=25 duration=990.458µs39002026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39012026/11/12 00:03:20 INFO Received uploads request method=POST path=/39022026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39032026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39042026/11/12 00:03:20 INFO Received push request method=POST path=/39052026/11/12 00:03:20 INFO Received push request method=POST path=/39062026/11/12 00:03:20 INFO Received push request method=POST path=/39072026/11/12 00:03:20 INFO Received complete push request method=POST path=/39082026/11/12 00:03:20 INFO Vacuumed table table=closures39092026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39102026/11/12 00:03:20 INFO Completed push id=8 objects=6 duration=1.071334ms39112026/11/12 00:03:20 INFO Vacuumed table table=objects39122026/11/12 00:03:20 INFO Vacuumed table table=closures39132026/11/12 00:03:20 INFO Received complete upload request method=POST path=/39142026/11/12 00:03:20 INFO Completed upload id=21 objects=2 duration=1.098333ms39152026/11/12 00:03:20 INFO Vacuumed table table=objects39162026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=039172026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=039182026/11/12 00:03:20 INFO Received push request method=POST path=/39192026/11/12 00:03:20 INFO Received push request method=POST path=/39202026/11/12 00:03:20 INFO Received push request method=POST path=/39212026/11/12 00:03:20 INFO Vacuumed table table=pending_closures39222026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=039232026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039242026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39252026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period39262026/11/12 00:03:20 INFO Received uploads request method=POST path=/39272026/11/12 00:03:20 INFO Received push request method=POST path=/39282026/11/12 00:03:20 INFO Received push request method=POST path=/39292026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39302026/11/12 00:03:20 INFO Vacuumed table table=pending_closures39312026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039322026/11/12 00:03:20 INFO Received uploads request method=POST path=/39332026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=039342026/11/12 00:03:20 INFO Received complete upload request method=POST path=/39352026/11/12 00:03:20 INFO Vacuumed table table=closures39362026/11/12 00:03:20 INFO Received push request method=POST path=/39372026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39382026/11/12 00:03:20 INFO Vacuumed table table=pending_closures39392026/11/12 00:03:20 INFO Vacuumed table table=objects39402026/11/12 00:03:20 INFO Completed upload id=4 objects=14 duration=1.159959ms39412026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=039422026/11/12 00:03:20 INFO Received uploads request method=POST path=/39432026/11/12 00:03:20 INFO Received complete push request method=POST path=/39442026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39452026/11/12 00:03:20 INFO Vacuumed table table=pending_closures39462026/11/12 00:03:20 INFO Received push request method=POST path=/39472026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39482026/11/12 00:03:20 INFO Vacuumed table table=closures39492026/11/12 00:03:20 INFO Received complete upload request method=POST path=/39502026/11/12 00:03:20 INFO Completed push id=30 objects=4 duration=845.375µs39512026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=039522026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39532026/11/12 00:03:20 INFO Vacuumed table table=objects39542026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39552026/11/12 00:03:20 INFO Received push request method=POST path=/39562026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period39572026/11/12 00:03:20 INFO Completed upload id=10 objects=23 duration=978.833µs39582026/11/12 00:03:20 INFO Received push request method=POST path=/39592026/11/12 00:03:20 INFO Received uploads request method=POST path=/39602026/11/12 00:03:20 INFO Vacuumed table table=closures39612026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39622026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039632026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=039642026/11/12 00:03:20 INFO Vacuumed table table=objects39652026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period39662026/11/12 00:03:20 INFO Received uploads request method=POST path=/39672026/11/12 00:03:20 INFO Received uploads request method=POST path=/39682026/11/12 00:03:20 INFO Received push request method=POST path=/39692026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039702026/11/12 00:03:20 INFO Received push request method=POST path=/39712026/11/12 00:03:20 INFO Vacuumed table table=pending_closures39722026/11/12 00:03:20 INFO Vacuumed table table=pending_closures39732026/11/12 00:03:20 INFO Received complete push request method=POST path=/39742026/11/12 00:03:20 INFO Received complete push request method=POST path=/39752026/11/12 00:03:20 INFO Vacuumed table table=closures39762026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39772026/11/12 00:03:20 INFO Completed push id=15 objects=24 duration=876.167µs39782026/11/12 00:03:20 INFO Completed push id=32 objects=7 duration=326.583µs39792026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39802026/11/12 00:03:20 INFO Vacuumed table table=objects39812026/11/12 00:03:20 INFO Received uploads request method=POST path=/39822026/11/12 00:03:20 INFO Vacuumed table table=closures39832026/11/12 00:03:20 INFO Vacuumed table table=pending_objects39842026/11/12 00:03:20 INFO Received complete push request method=POST path=/39852026/11/12 00:03:20 INFO Received push request method=POST path=/39862026/11/12 00:03:20 INFO Received complete push request method=POST path=/39872026/11/12 00:03:20 INFO Received uploads request method=POST path=/39882026/11/12 00:03:20 INFO Vacuumed table table=objects39892026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads39902026/11/12 00:03:20 INFO Completed push id=26 objects=4 duration=497.875µs39912026/11/12 00:03:20 INFO Received push request method=POST path=/39922026/11/12 00:03:20 INFO Completed push id=14 objects=4 duration=766.5µs39932026/11/12 00:03:20 INFO Received uploads request method=POST path=/39942026/11/12 00:03:20 INFO Received complete upload request method=POST path=/39952026/11/12 00:03:20 INFO Vacuumed table table=closures39962026/11/12 00:03:20 INFO Received push request method=POST path=/39972026/11/12 00:03:20 INFO Completed upload id=19 objects=2 duration=1.0225ms39982026/11/12 00:03:20 INFO Received complete upload request method=POST path=/39992026/11/12 00:03:20 INFO Received push request method=POST path=/40002026/11/12 00:03:20 INFO Received complete upload request method=POST path=/40012026/11/12 00:03:20 INFO Vacuumed table table=objects40022026/11/12 00:03:20 INFO Completed upload id=28 objects=13 duration=584.083µs40032026/11/12 00:03:20 INFO Received uploads request method=POST path=/40042026/11/12 00:03:20 INFO Received uploads request method=POST path=/40052026/11/12 00:03:20 INFO Completed upload id=11 objects=6 duration=1.0635ms40062026/11/12 00:03:20 INFO Received complete push request method=POST path=/40072026/11/12 00:03:20 INFO Received complete push request method=POST path=/40082026/11/12 00:03:20 INFO Completed push id=33 objects=7 duration=1.208292ms40092026/11/12 00:03:20 INFO Received push request method=POST path=/40102026/11/12 00:03:20 INFO Received push request method=POST path=/40112026/11/12 00:03:20 INFO Completed push id=29 objects=3 duration=877.042µs40122026/11/12 00:03:20 INFO Received complete upload request method=POST path=/40132026/11/12 00:03:20 INFO Received push request method=POST path=/40142026/11/12 00:03:20 INFO Completed upload id=26 objects=3 duration=467.375µs40152026/11/12 00:03:20 INFO Received complete upload request method=POST path=/40162026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040172026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040182026/11/12 00:03:20 INFO Received push request method=POST path=/40192026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040202026/11/12 00:03:20 INFO Completed upload id=31 objects=2 duration=234.208µs40212026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period40222026/11/12 00:03:20 INFO Received complete upload request method=POST path=/40232026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period40242026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040252026/11/12 00:03:20 INFO Received push request method=POST path=/40262026/11/12 00:03:20 INFO Received complete push request method=POST path=/40272026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040282026/11/12 00:03:20 INFO Completed upload id=6 objects=6 duration=277.834µs40292026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040302026/11/12 00:03:20 INFO Completed push id=33 objects=6 duration=261.25µs40312026/11/12 00:03:20 INFO Received complete push request method=POST path=/40322026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40332026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40342026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040352026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40362026/11/12 00:03:20 INFO Completed push id=8 objects=13 duration=805.833µs40372026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40382026/11/12 00:03:20 INFO Received uploads request method=POST path=/40392026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040402026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period40412026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40422026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040432026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads40442026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads40452026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period40462026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040472026/11/12 00:03:20 INFO Received uploads request method=POST path=/40482026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040492026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40502026/11/12 00:03:20 INFO Vacuumed table table=closures40512026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads40522026/11/12 00:03:20 INFO Vacuumed table table=closures40532026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40542026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040552026/11/12 00:03:20 INFO Vacuumed table table=objects40562026/11/12 00:03:20 INFO Vacuumed table table=closures40572026/11/12 00:03:20 INFO Received push request method=POST path=/40582026/11/12 00:03:20 INFO Vacuumed table table=objects40592026/11/12 00:03:20 INFO Vacuumed table table=objects40602026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40612026/11/12 00:03:20 INFO Received push request method=POST path=/40622026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads40632026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40642026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40652026/11/12 00:03:20 INFO Vacuumed table table=closures40662026/11/12 00:03:20 INFO Received push request method=POST path=/40672026/11/12 00:03:20 INFO Received uploads request method=POST path=/40682026/11/12 00:03:20 INFO Vacuumed table table=objects40692026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40702026/11/12 00:03:20 INFO Received complete push request method=POST path=/40712026/11/12 00:03:20 INFO Received uploads request method=POST path=/40722026/11/12 00:03:20 INFO Received push request method=POST path=/40732026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads40742026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40752026/11/12 00:03:20 INFO Completed push id=12 objects=15 duration=489.958µs40762026/11/12 00:03:20 INFO Received push request method=POST path=/40772026/11/12 00:03:20 INFO Received uploads request method=POST path=/40782026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads40792026/11/12 00:03:20 INFO Vacuumed table table=closures40802026/11/12 00:03:20 INFO Received push request method=POST path=/40812026/11/12 00:03:20 INFO Vacuumed table table=objects40822026/11/12 00:03:20 INFO Received push request method=POST path=/40832026/11/12 00:03:20 INFO Vacuumed table table=closures40842026/11/12 00:03:20 INFO Received complete push request method=POST path=/40852026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040862026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=040872026/11/12 00:03:20 INFO Completed push id=5 objects=8 duration=1.574916ms40882026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period40892026/11/12 00:03:20 INFO Received uploads request method=POST path=/40902026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period40912026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040922026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040932026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40942026/11/12 00:03:20 INFO Vacuumed table table=pending_closures40952026/11/12 00:03:20 INFO Received push request method=POST path=/40962026/11/12 00:03:20 INFO Received uploads request method=POST path=/40972026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40982026/11/12 00:03:20 INFO Vacuumed table table=pending_objects40992026/11/12 00:03:20 INFO Vacuumed table table=objects41002026/11/12 00:03:20 INFO Received uploads request method=POST path=/41012026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads41022026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads41032026/11/12 00:03:20 INFO Received complete push request method=POST path=/41042026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=041052026/11/12 00:03:20 INFO Vacuumed table table=closures41062026/11/12 00:03:20 INFO Received complete upload request method=POST path=/41072026/11/12 00:03:20 INFO Received uploads request method=POST path=/41082026/11/12 00:03:20 INFO Vacuumed table table=closures41092026/11/12 00:03:20 INFO Completed upload id=15 objects=6 duration=223.666µs41102026/11/12 00:03:20 INFO Vacuumed table table=objects41112026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041122026/11/12 00:03:20 INFO Completed push id=7 objects=24 duration=1.336875ms41132026/11/12 00:03:20 INFO Vacuumed table table=objects41142026/11/12 00:03:20 INFO Received complete push request method=POST path=/41152026/11/12 00:03:20 INFO Vacuumed table table=pending_closures41162026/11/12 00:03:20 INFO Received uploads request method=POST path=/41172026/11/12 00:03:20 INFO Completed push id=9 objects=24 duration=528.917µs41182026/11/12 00:03:20 INFO Received push request method=POST path=/41192026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=041202026/11/12 00:03:20 INFO Vacuumed table table=pending_objects41212026/11/12 00:03:20 INFO Received push request method=POST path=/41222026/11/12 00:03:20 INFO Received uploads request method=POST path=/41232026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period41242026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads41252026/11/12 00:03:20 INFO Received uploads request method=POST path=/41262026/11/12 00:03:20 INFO Received uploads request method=POST path=/41272026/11/12 00:03:20 INFO Vacuumed table table=closures41282026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041292026/11/12 00:03:20 INFO Received uploads request method=POST path=/41302026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=041312026/11/12 00:03:20 INFO Vacuumed table table=objects41322026/11/12 00:03:20 INFO Vacuumed table table=pending_closures41332026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period41342026/11/12 00:03:20 INFO Received uploads request method=POST path=/41352026/11/12 00:03:20 INFO Received complete upload request method=POST path=/41362026/11/12 00:03:20 INFO Vacuumed table table=pending_objects41372026/11/12 00:03:20 INFO Received uploads request method=POST path=/41382026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041392026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads41402026/11/12 00:03:20 INFO Completed upload id=10 objects=10 duration=280.75µs41412026/11/12 00:03:20 INFO Received push request method=POST path=/41422026/11/12 00:03:20 INFO Vacuumed table table=closures41432026/11/12 00:03:20 INFO Vacuumed table table=pending_closures41442026/11/12 00:03:20 INFO Vacuumed table table=objects41452026/11/12 00:03:20 INFO Received uploads request method=POST path=/41462026/11/12 00:03:20 INFO Received uploads request method=POST path=/41472026/11/12 00:03:20 INFO Received uploads request method=POST path=/41482026/11/12 00:03:20 INFO Vacuumed table table=pending_objects41492026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads41502026/11/12 00:03:20 INFO Received push request method=POST path=/41512026/11/12 00:03:20 INFO Received push request method=POST path=/41522026/11/12 00:03:20 INFO Received complete push request method=POST path=/41532026/11/12 00:03:20 INFO Received push request method=POST path=/41542026/11/12 00:03:20 INFO Vacuumed table table=closures41552026/11/12 00:03:20 INFO Completed push id=13 objects=2 duration=470.834µs41562026/11/12 00:03:20 INFO Vacuumed table table=objects41572026/11/12 00:03:20 INFO Received push request method=POST path=/41582026/11/12 00:03:20 INFO Received complete upload request method=POST path=/41592026/11/12 00:03:20 INFO Received complete upload request method=POST path=/41602026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=041612026/11/12 00:03:20 INFO Completed upload id=27 objects=13 duration=521.542µs41622026/11/12 00:03:20 INFO Completed upload id=8 objects=3 duration=6.276875ms41632026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period41642026/11/12 00:03:20 INFO Received uploads request method=POST path=/41652026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041662026/11/12 00:03:20 INFO Received uploads request method=POST path=/41672026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=041682026/11/12 00:03:20 INFO Vacuumed table table=pending_closures41692026/11/12 00:03:20 INFO Received complete upload request method=POST path=/41702026/11/12 00:03:20 INFO Vacuumed table table=pending_objects41712026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041722026/11/12 00:03:20 INFO Received complete push request method=POST path=/41732026/11/12 00:03:20 INFO Completed upload id=10 objects=4 duration=757.875µs41742026/11/12 00:03:20 INFO Received complete push request method=POST path=/41752026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads41762026/11/12 00:03:20 INFO Vacuumed table table=pending_closures41772026/11/12 00:03:20 INFO Completed push id=14 objects=3 duration=248.625µs41782026/11/12 00:03:20 INFO Completed push id=6 objects=6 duration=476.875µs41792026/11/12 00:03:20 INFO Received push request method=POST path=/41802026/11/12 00:03:20 INFO Vacuumed table table=closures41812026/11/12 00:03:20 INFO Vacuumed table table=pending_objects41822026/11/12 00:03:20 INFO Received push request method=POST path=/41832026/11/12 00:03:20 INFO Received push request method=POST path=/41842026/11/12 00:03:20 INFO Received push request method=POST path=/41852026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads41862026/11/12 00:03:20 INFO Vacuumed table table=objects41872026/11/12 00:03:20 INFO Received push request method=POST path=/41882026/11/12 00:03:20 INFO Vacuumed table table=closures41892026/11/12 00:03:20 INFO Received push request method=POST path=/41902026/11/12 00:03:20 INFO Received complete upload request method=POST path=/41912026/11/12 00:03:20 INFO Received uploads request method=POST path=/41922026/11/12 00:03:20 INFO Vacuumed table table=objects41932026/11/12 00:03:20 INFO Received push request method=POST path=/41942026/11/12 00:03:20 INFO Completed upload id=14 objects=16 duration=731.833µs41952026/11/12 00:03:20 INFO Received complete push request method=POST path=/41962026/11/12 00:03:20 INFO Received uploads request method=POST path=/41972026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=041982026/11/12 00:03:20 INFO Completed push id=13 objects=15 duration=467.875µs41992026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period42002026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=042012026/11/12 00:03:20 INFO Received complete push request method=POST path=/42022026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042032026/11/12 00:03:20 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42042026/11/12 00:03:20 INFO Received push request method=POST path=/42052026/11/12 00:03:20 INFO Completed push id=16 objects=16 duration=1.320416ms42062026/11/12 00:03:20 INFO Received push request method=POST path=/42072026/11/12 00:03:20 INFO Vacuumed table table=pending_closures42082026/11/12 00:03:20 INFO Received uploads request method=POST path=/42092026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=042102026/11/12 00:03:20 INFO Vacuumed table table=pending_objects42112026/11/12 00:03:20 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."42122026/11/12 00:03:20 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."42132026/11/12 00:03:20 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."42142026/11/12 00:03:20 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."42152026/11/12 00:03:20 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42162026/11/12 00:03:20 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."42172026/11/12 00:03:20 INFO Received uploads request method=POST path=/42182026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads42192026/11/12 00:03:20 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42202026/11/12 00:03:20 INFO Vacuumed table table=closures42212026/11/12 00:03:20 INFO Received complete push request method=POST path=/42222026/11/12 00:03:20 INFO Vacuumed table table=objects42232026/11/12 00:03:20 INFO Received uploads request method=POST path=/42242026/11/12 00:03:20 INFO Received complete upload request method=POST path=/42252026/11/12 00:03:20 INFO Completed upload id=29 objects=4 duration=439.708µs42262026/11/12 00:03:20 INFO Received complete push request method=POST path=/42272026/11/12 00:03:20 INFO Completed push id=3 objects=21 duration=6.8955ms42282026/11/12 00:03:20 INFO Received uploads request method=POST path=/42292026/11/12 00:03:20 INFO Completed push id=16 objects=27 duration=979.166µs42302026/11/12 00:03:20 INFO Received uploads request method=POST path=/42312026/11/12 00:03:20 INFO Received push request method=POST path=/42322026/11/12 00:03:20 INFO Received push request method=POST path=/42332026/11/12 00:03:20 INFO Received uploads request method=POST path=/42342026/11/12 00:03:20 INFO Received uploads request method=POST path=/42352026/11/12 00:03:20 INFO Received uploads request method=POST path=/42362026/11/12 00:03:20 INFO Received uploads request method=POST path=/42372026/11/12 00:03:20 INFO Received uploads request method=POST path=/42382026/11/12 00:03:20 INFO Received push request method=POST path=/42392026/11/12 00:03:20 INFO Received complete push request method=POST path=/42402026/11/12 00:03:20 INFO Completed push id=10 objects=15 duration=1.495292ms42412026/11/12 00:03:20 INFO Received push request method=POST path=/42422026/11/12 00:03:20 INFO Received uploads request method=POST path=/42432026/11/12 00:03:20 INFO Received uploads request method=POST path=/42442026/11/12 00:03:20 INFO Received complete upload request method=POST path=/42452026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=042462026/11/12 00:03:20 INFO Received uploads request method=POST path=/42472026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=042482026/11/12 00:03:20 INFO Completed upload id=11 objects=14 duration=392.458µs42492026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period42502026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period42512026/11/12 00:03:20 INFO Aborted multipart uploads count=0 kept=042522026/11/12 00:03:20 INFO Received push request method=POST path=/42532026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042542026/11/12 00:03:20 INFO Received complete push request method=POST path=/42552026/11/12 00:03:20 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."42562026/11/12 00:03:20 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."42572026/11/12 00:03:20 INFO Vacuumed table table=pending_closures42582026/11/12 00:03:20 WARN Force mode enabled - objects will be deleted immediately without grace period42592026/11/12 00:03:20 INFO Received push request method=POST path=/42602026/11/12 00:03:20 INFO Completed push id=15 objects=5 duration=418.208µs42612026/11/12 00:03:20 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."42622026/11/12 00:03:20 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."42632026/11/12 00:03:20 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42642026/11/12 00:03:20 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."42652026/11/12 00:03:20 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42662026/11/12 00:03:20 INFO Vacuumed table table=pending_objects42672026/11/12 00:03:20 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=042682026/11/12 00:03:20 INFO Received push request method=POST path=/42692026/11/12 00:03:20 INFO Received uploads request method=POST path=/42702026/11/12 00:03:20 INFO Vacuumed table table=multipart_uploads42712026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=042722026/11/12 00:03:21 INFO Vacuumed table table=closures42732026/11/12 00:03:21 INFO Vacuumed table table=objects42742026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period42752026/11/12 00:03:21 INFO Received push request method=POST path=/42762026/11/12 00:03:21 INFO Received complete upload request method=POST path=/42772026/11/12 00:03:21 INFO Completed upload id=21 objects=3 duration=264.583µs42782026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042792026/11/12 00:03:21 INFO Received complete upload request method=POST path=/42802026/11/12 00:03:21 INFO Vacuumed table table=pending_closures42812026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=042822026/11/12 00:03:21 INFO Vacuumed table table=pending_closures42832026/11/12 00:03:21 INFO Received uploads request method=POST path=/42842026/11/12 00:03:21 INFO Completed upload id=23 objects=16 duration=726.167µs42852026/11/12 00:03:21 INFO Received uploads request method=POST path=/42862026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period42872026/11/12 00:03:21 INFO Vacuumed table table=pending_objects42882026/11/12 00:03:21 INFO Received complete push request method=POST path=/42892026/11/12 00:03:21 INFO Completed push id=11 objects=6 duration=756.209µs42902026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads42912026/11/12 00:03:21 INFO Vacuumed table table=pending_objects42922026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=042932026/11/12 00:03:21 INFO Received uploads request method=POST path=/42942026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=042952026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=042962026/11/12 00:03:21 INFO Vacuumed table table=closures42972026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads42982026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period42992026/11/12 00:03:21 INFO Received uploads request method=POST path=/43002026/11/12 00:03:21 INFO Vacuumed table table=closures43012026/11/12 00:03:21 INFO Received complete push request method=POST path=/43022026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043032026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period43042026/11/12 00:03:21 INFO Vacuumed table table=objects43052026/11/12 00:03:21 INFO Received complete push request method=POST path=/43062026/11/12 00:03:21 INFO Vacuumed table table=objects43072026/11/12 00:03:21 INFO Completed push id=9 objects=9 duration=990.958µs43082026/11/12 00:03:21 INFO Received uploads request method=POST path=/43092026/11/12 00:03:21 INFO Vacuumed table table=pending_closures43102026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043112026/11/12 00:03:21 INFO Completed push id=17 objects=11 duration=606.084µs43122026/11/12 00:03:21 INFO Vacuumed table table=pending_closures43132026/11/12 00:03:21 INFO Vacuumed table table=pending_objects43142026/11/12 00:03:21 INFO Vacuumed table table=pending_closures43152026/11/12 00:03:21 INFO Received complete push request method=POST path=/43162026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads43172026/11/12 00:03:21 INFO Received push request method=POST path=/43182026/11/12 00:03:21 INFO Received push request method=POST path=/43192026/11/12 00:03:21 INFO Received uploads request method=POST path=/43202026/11/12 00:03:21 INFO Completed push id=20 objects=3 duration=850.125µs43212026/11/12 00:03:21 INFO Vacuumed table table=closures43222026/11/12 00:03:21 INFO Vacuumed table table=pending_objects43232026/11/12 00:03:21 INFO Vacuumed table table=pending_objects43242026/11/12 00:03:21 INFO Received complete push request method=POST path=/43252026/11/12 00:03:21 INFO Received complete push request method=POST path=/43262026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads43272026/11/12 00:03:21 INFO Vacuumed table table=objects43282026/11/12 00:03:21 INFO Completed push id=14 objects=3 duration=3.600083ms43292026/11/12 00:03:21 INFO Completed push id=18 objects=6 duration=3.609667ms43302026/11/12 00:03:21 INFO Received complete push request method=POST path=/43312026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads43322026/11/12 00:03:21 INFO Completed push id=18 objects=10 duration=590.75µs43332026/11/12 00:03:21 INFO Vacuumed table table=closures43342026/11/12 00:03:21 INFO Received complete push request method=POST path=/43352026/11/12 00:03:21 INFO Vacuumed table table=closures43362026/11/12 00:03:21 INFO Completed push id=20 objects=13 duration=414.042µs43372026/11/12 00:03:21 INFO Vacuumed table table=objects43382026/11/12 00:03:21 INFO Vacuumed table table=objects43392026/11/12 00:03:21 INFO Received complete upload request method=POST path=/43402026/11/12 00:03:21 INFO Completed upload id=21 objects=8 duration=627.416µs43412026/11/12 00:03:21 INFO Received push request method=POST path=/43422026/11/12 00:03:21 INFO Received uploads request method=POST path=/43432026/11/12 00:03:21 INFO Received complete upload request method=POST path=/43442026/11/12 00:03:21 INFO Completed upload id=23 objects=9 duration=423.25µs43452026/11/12 00:03:21 INFO Received complete push request method=POST path=/43462026/11/12 00:03:21 INFO Completed push id=22 objects=4 duration=503.833µs43472026/11/12 00:03:21 INFO Received complete push request method=POST path=/43482026/11/12 00:03:21 INFO Completed push id=5 objects=6 duration=862.375µs43492026/11/12 00:03:21 INFO Received complete push request method=POST path=/43502026/11/12 00:03:21 INFO Completed push id=22 objects=6 duration=1.069458ms43512026/11/12 00:03:21 INFO Received uploads request method=POST path=/43522026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=043532026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period43542026/11/12 00:03:21 INFO Received push request method=POST path=/43552026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043562026/11/12 00:03:21 INFO Vacuumed table table=pending_closures43572026/11/12 00:03:21 INFO Received uploads request method=POST path=/43582026/11/12 00:03:21 INFO Vacuumed table table=pending_objects43592026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads43602026/11/12 00:03:21 INFO Vacuumed table table=closures43612026/11/12 00:03:21 INFO Vacuumed table table=objects43622026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=043632026/11/12 00:03:21 INFO Received complete push request method=POST path=/43642026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=043652026/11/12 00:03:21 INFO Received push request method=POST path=/43662026/11/12 00:03:21 INFO Received complete upload request method=POST path=/43672026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=043682026/11/12 00:03:21 INFO Received push request method=POST path=/43692026/11/12 00:03:21 INFO Completed upload id=26 objects=6 duration=743.708µs43702026/11/12 00:03:21 INFO Completed push id=24 objects=8 duration=1.037625ms43712026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period43722026/11/12 00:03:21 INFO Received push request method=POST path=/43732026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period43742026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period43752026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043762026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043772026/11/12 00:03:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."43782026/11/12 00:03:21 INFO Received complete upload request method=POST path=/43792026/11/12 00:03:21 INFO Received uploads request method=POST path=/43802026/11/12 00:03:21 INFO Received push request method=POST path=/43812026/11/12 00:03:21 INFO Received uploads request method=POST path=/43822026/11/12 00:03:21 INFO Completed upload id=25 objects=9 duration=346.875µs43832026/11/12 00:03:21 INFO Vacuumed table table=pending_closures43842026/11/12 00:03:21 INFO Vacuumed table table=pending_objects43852026/11/12 00:03:21 INFO Received complete upload request method=POST path=/43862026/11/12 00:03:21 INFO Received complete upload request method=POST path=/43872026/11/12 00:03:21 INFO Received complete push request method=POST path=/43882026/11/12 00:03:21 INFO Received complete upload request method=POST path=/43892026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads43902026/11/12 00:03:21 INFO Completed upload id=22 objects=6 duration=313µs43912026/11/12 00:03:21 INFO Vacuumed table table=closures43922026/11/12 00:03:21 INFO Completed push id=22 objects=3 duration=3.788167ms43932026/11/12 00:03:21 INFO Completed upload id=27 objects=3 duration=3.861542ms43942026/11/12 00:03:21 INFO Completed upload id=29 objects=5 duration=3.798833ms43952026/11/12 00:03:21 INFO Vacuumed table table=objects43962026/11/12 00:03:21 INFO Received uploads request method=POST path=/43972026/11/12 00:03:21 INFO Received push request method=POST path=/43982026/11/12 00:03:21 INFO Vacuumed table table=pending_closures43992026/11/12 00:03:21 INFO Received uploads request method=POST path=/44002026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44012026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44022026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=044032026/11/12 00:03:21 INFO Completed upload id=28 objects=6 duration=980µs44042026/11/12 00:03:21 INFO Received uploads request method=POST path=/44052026/11/12 00:03:21 INFO Completed upload id=23 objects=7 duration=640.833µs44062026/11/12 00:03:21 INFO Received complete push request method=POST path=/44072026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period44082026/11/12 00:03:21 INFO Received complete push request method=POST path=/44092026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044102026/11/12 00:03:21 INFO Completed push id=16 objects=26 duration=743.666µs44112026/11/12 00:03:21 INFO Completed push id=14 objects=3 duration=1.038458ms44122026/11/12 00:03:21 INFO Received push request method=POST path=/44132026/11/12 00:03:21 INFO Received push request method=POST path=/44142026/11/12 00:03:21 INFO Received complete push request method=POST path=/44152026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44162026/11/12 00:03:21 INFO Completed push id=30 objects=4 duration=912.833µs44172026/11/12 00:03:21 INFO Vacuumed table table=pending_objects44182026/11/12 00:03:21 INFO Vacuumed table table=pending_closures44192026/11/12 00:03:21 INFO Received complete push request method=POST path=/44202026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads44212026/11/12 00:03:21 INFO Completed upload id=14 objects=8 duration=1.789083ms44222026/11/12 00:03:21 INFO Completed push id=24 objects=5 duration=1.617625ms44232026/11/12 00:03:21 INFO Vacuumed table table=pending_objects44242026/11/12 00:03:21 INFO Received complete push request method=POST path=/44252026/11/12 00:03:21 INFO Received push request method=POST path=/44262026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=044272026/11/12 00:03:21 INFO Completed push id=24 objects=3 duration=577.917µs44282026/11/12 00:03:21 INFO Vacuumed table table=closures44292026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads44302026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period44312026/11/12 00:03:21 INFO Vacuumed table table=closures44322026/11/12 00:03:21 INFO Received complete push request method=POST path=/44332026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44342026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044352026/11/12 00:03:21 INFO Received push request method=POST path=/44362026/11/12 00:03:21 INFO Received uploads request method=POST path=/44372026/11/12 00:03:21 INFO Vacuumed table table=objects44382026/11/12 00:03:21 INFO Completed push id=31 objects=7 duration=561.375µs44392026/11/12 00:03:21 INFO Completed upload id=25 objects=2 duration=299.541µs44402026/11/12 00:03:21 INFO Vacuumed table table=pending_closures44412026/11/12 00:03:21 INFO Received uploads request method=POST path=/44422026/11/12 00:03:21 INFO Received uploads request method=POST path=/44432026/11/12 00:03:21 INFO Vacuumed table table=objects44442026/11/12 00:03:21 INFO Received push request method=POST path=/44452026/11/12 00:03:21 INFO Vacuumed table table=pending_objects44462026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=044472026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44482026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44492026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads44502026/11/12 00:03:21 INFO Completed upload id=26 objects=20 duration=365.333µs44512026/11/12 00:03:21 INFO Completed upload id=32 objects=5 duration=378.417µs44522026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period44532026/11/12 00:03:21 INFO Received uploads request method=POST path=/44542026/11/12 00:03:21 INFO Vacuumed table table=closures44552026/11/12 00:03:21 INFO Received complete push request method=POST path=/44562026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044572026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=044582026/11/12 00:03:21 INFO Vacuumed table table=objects44592026/11/12 00:03:21 INFO Completed push id=21 objects=9 duration=607.292µs44602026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period44612026/11/12 00:03:21 INFO Vacuumed table table=pending_closures44622026/11/12 00:03:21 INFO Vacuumed table table=pending_objects44632026/11/12 00:03:21 INFO Received uploads request method=POST path=/44642026/11/12 00:03:21 INFO Received uploads request method=POST path=/44652026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=044662026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads44672026/11/12 00:03:21 INFO Received complete push request method=POST path=/44682026/11/12 00:03:21 INFO Received push request method=POST path=/44692026/11/12 00:03:21 INFO Received uploads request method=POST path=/44702026/11/12 00:03:21 INFO Vacuumed table table=closures44712026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44722026/11/12 00:03:21 INFO Received uploads request method=POST path=/44732026/11/12 00:03:21 INFO Vacuumed table table=pending_closures44742026/11/12 00:03:21 INFO Completed push id=27 objects=6 duration=989.791µs44752026/11/12 00:03:21 INFO Completed upload id=31 objects=3 duration=569.333µs44762026/11/12 00:03:21 INFO Vacuumed table table=objects44772026/11/12 00:03:21 INFO Vacuumed table table=pending_objects44782026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44792026/11/12 00:03:21 INFO Received push request method=POST path=/44802026/11/12 00:03:21 INFO Received push request method=POST path=/44812026/11/12 00:03:21 INFO Received complete upload request method=POST path=/44822026/11/12 00:03:21 INFO Received push request method=POST path=/44832026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads44842026/11/12 00:03:21 INFO Completed upload id=22 objects=2 duration=340.291µs44852026/11/12 00:03:21 INFO Completed upload id=15 objects=2 duration=334.167µs44862026/11/12 00:03:21 INFO Vacuumed table table=closures44872026/11/12 00:03:21 INFO Vacuumed table table=objects44882026/11/12 00:03:21 INFO Received push request method=POST path=/44892026/11/12 00:03:21 INFO Received push request method=POST path=/44902026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=044912026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period44922026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=044932026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=044942026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044952026/11/12 00:03:21 INFO Received complete push request method=POST path=/44962026/11/12 00:03:21 INFO Received uploads request method=POST path=/44972026/11/12 00:03:21 INFO Vacuumed table table=pending_closures44982026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044992026/11/12 00:03:21 INFO Received push request method=POST path=/45002026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=045012026/11/12 00:03:21 INFO Completed push id=28 objects=7 duration=503.125µs45022026/11/12 00:03:21 INFO Vacuumed table table=pending_objects45032026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads45042026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45052026/11/12 00:03:21 INFO Vacuumed table table=closures45062026/11/12 00:03:21 INFO Completed upload id=13 objects=8 duration=6.264417ms45072026/11/12 00:03:21 INFO Vacuumed table table=objects45082026/11/12 00:03:21 INFO Received uploads request method=POST path=/45092026/11/12 00:03:21 INFO Vacuumed table table=pending_closures45102026/11/12 00:03:21 INFO Vacuumed table table=pending_closures45112026/11/12 00:03:21 INFO Received uploads request method=POST path=/45122026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45132026/11/12 00:03:21 INFO Received uploads request method=POST path=/45142026/11/12 00:03:21 INFO Received uploads request method=POST path=/45152026/11/12 00:03:21 INFO Completed upload id=18 objects=2 duration=700.666µs45162026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45172026/11/12 00:03:21 INFO Vacuumed table table=pending_objects45182026/11/12 00:03:21 INFO Completed upload id=26 objects=8 duration=5.664625ms45192026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads45202026/11/12 00:03:21 INFO Received push request method=POST path=/45212026/11/12 00:03:21 INFO Received push request method=POST path=/45222026/11/12 00:03:21 INFO Vacuumed table table=closures45232026/11/12 00:03:21 INFO Received complete push request method=POST path=/45242026/11/12 00:03:21 INFO Vacuumed table table=pending_objects45252026/11/12 00:03:21 INFO Completed push id=28 objects=14 duration=5.610875ms45262026/11/12 00:03:21 INFO Vacuumed table table=objects45272026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads45282026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45292026/11/12 00:03:21 INFO Vacuumed table table=closures45302026/11/12 00:03:21 INFO Received push request method=POST path=/45312026/11/12 00:03:21 INFO Completed upload id=30 objects=6 duration=251.292µs45322026/11/12 00:03:21 INFO Received uploads request method=POST path=/45332026/11/12 00:03:21 INFO Received uploads request method=POST path=/45342026/11/12 00:03:21 INFO Vacuumed table table=objects45352026/11/12 00:03:21 INFO Received complete push request method=POST path=/45362026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=045372026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45382026/11/12 00:03:21 INFO Completed push id=31 objects=5 duration=312.5µs45392026/11/12 00:03:21 INFO Completed upload id=26 objects=2 duration=5.98575ms45402026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period45412026/11/12 00:03:21 INFO Received uploads request method=POST path=/45422026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045432026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45442026/11/12 00:03:21 INFO Vacuumed table table=pending_closures45452026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=045462026/11/12 00:03:21 INFO Completed upload id=16 objects=17 duration=737.792µs45472026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045482026/11/12 00:03:21 INFO Vacuumed table table=pending_objects45492026/11/12 00:03:21 INFO Received uploads request method=POST path=/45502026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=045512026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads45522026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period45532026/11/12 00:03:21 INFO Vacuumed table table=closures45542026/11/12 00:03:21 INFO Vacuumed table table=pending_closures45552026/11/12 00:03:21 INFO Vacuumed table table=objects45562026/11/12 00:03:21 INFO Received uploads request method=POST path=/45572026/11/12 00:03:21 INFO Received uploads request method=POST path=/45582026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45592026/11/12 00:03:21 INFO Completed upload id=23 objects=18 duration=899.458µs45602026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45612026/11/12 00:03:21 INFO Vacuumed table table=pending_objects45622026/11/12 00:03:21 INFO Received uploads request method=POST path=/45632026/11/12 00:03:21 INFO Completed upload id=21 objects=9 duration=657.25µs45642026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads45652026/11/12 00:03:21 INFO Received complete push request method=POST path=/45662026/11/12 00:03:21 INFO Vacuumed table table=closures45672026/11/12 00:03:21 INFO Received push request method=POST path=/45682026/11/12 00:03:21 INFO Completed push id=25 objects=10 duration=6.465291ms45692026/11/12 00:03:21 INFO Vacuumed table table=objects45702026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45712026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45722026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45732026/11/12 00:03:21 INFO Completed upload id=13 objects=7 duration=360.416µs45742026/11/12 00:03:21 INFO Completed upload id=22 objects=22 duration=830.583µs45752026/11/12 00:03:21 INFO Completed upload id=29 objects=5 duration=6.000291ms45762026/11/12 00:03:21 INFO Received push request method=POST path=/45772026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45782026/11/12 00:03:21 INFO Received push request method=POST path=/45792026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45802026/11/12 00:03:21 INFO Completed upload id=30 objects=2 duration=241.083µs45812026/11/12 00:03:21 INFO Completed upload id=20 objects=4 duration=478.958µs45822026/11/12 00:03:21 INFO Received complete push request method=POST path=/45832026/11/12 00:03:21 INFO Received complete push request method=POST path=/45842026/11/12 00:03:21 INFO Completed push id=32 objects=26 duration=1.615417ms45852026/11/12 00:03:21 INFO Received complete push request method=POST path=/45862026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45872026/11/12 00:03:21 INFO Completed upload id=23 objects=7 duration=404.166µs45882026/11/12 00:03:21 INFO Completed push id=26 objects=5 duration=6.505208ms45892026/11/12 00:03:21 INFO Completed push id=31 objects=20 duration=6.404625ms45902026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=045912026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45922026/11/12 00:03:21 INFO Completed upload id=33 objects=2 duration=409.834µs45932026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45942026/11/12 00:03:21 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."45952026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45962026/11/12 00:03:21 INFO Completed upload id=24 objects=14 duration=442.125µs45972026/11/12 00:03:21 INFO Received complete upload request method=POST path=/45982026/11/12 00:03:21 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."45992026/11/12 00:03:21 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."46002026/11/12 00:03:21 INFO Completed upload id=32 objects=5 duration=228.792µs46012026/11/12 00:03:21 INFO Completed upload id=34 objects=5 duration=268.084µs46022026/11/12 00:03:21 INFO Received push request method=POST path=/46032026/11/12 00:03:21 INFO Received complete upload request method=POST path=/46042026/11/12 00:03:21 INFO Received complete push request method=POST path=/46052026/11/12 00:03:21 INFO Received complete upload request method=POST path=/46062026/11/12 00:03:21 INFO Received complete upload request method=POST path=/46072026/11/12 00:03:21 INFO Received push request method=POST path=/46082026/11/12 00:03:21 INFO Completed upload id=33 objects=3 duration=174.541µs46092026/11/12 00:03:21 INFO Completed upload id=34 objects=24 duration=325.584µs46102026/11/12 00:03:21 INFO Completed push id=19 objects=3 duration=371.375µs46112026/11/12 00:03:21 INFO Completed upload id=35 objects=11 duration=151µs46122026/11/12 00:03:21 INFO Received complete push request method=POST path=/46132026/11/12 00:03:21 INFO Completed push id=28 objects=11 duration=662.709µs46142026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=046152026/11/12 00:03:21 INFO Received complete upload request method=POST path=/46162026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=046172026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period46182026/11/12 00:03:21 INFO Completed upload id=36 objects=3 duration=180.5µs46192026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period46202026/11/12 00:03:21 INFO Received uploads request method=POST path=/46212026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046222026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046232026/11/12 00:03:21 INFO Aborted multipart uploads count=0 kept=046242026/11/12 00:03:21 INFO Received uploads request method=POST path=/46252026/11/12 00:03:21 WARN Force mode enabled - objects will be deleted immediately without grace period46262026/11/12 00:03:21 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046272026/11/12 00:03:21 INFO Vacuumed table table=pending_closures46282026/11/12 00:03:21 INFO Received complete upload request method=POST path=/46292026/11/12 00:03:21 INFO Completed upload id=25 objects=4 duration=274.167µs46302026/11/12 00:03:21 INFO Vacuumed table table=pending_closures46312026/11/12 00:03:21 INFO Received push request method=POST path=/46322026/11/12 00:03:21 INFO Vacuumed table table=pending_closures46332026/11/12 00:03:21 INFO Vacuumed table table=pending_objects46342026/11/12 00:03:21 INFO Vacuumed table table=pending_objects46352026/11/12 00:03:21 INFO Received complete push request method=POST path=/46362026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads46372026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads46382026/11/12 00:03:21 INFO Received complete push request method=POST path=/46392026/11/12 00:03:21 INFO Completed push id=7 objects=15 duration=926.417µs46402026/11/12 00:03:21 INFO Vacuumed table table=closures46412026/11/12 00:03:21 INFO Vacuumed table table=pending_objects46422026/11/12 00:03:21 INFO Vacuumed table table=closures46432026/11/12 00:03:21 INFO Completed push id=20 objects=8 duration=5.584ms46442026/11/12 00:03:21 INFO Received complete push request method=POST path=/46452026/11/12 00:03:21 INFO Vacuumed table table=objects46462026/11/12 00:03:21 INFO Vacuumed table table=multipart_uploads46472026/11/12 00:03:21 INFO Vacuumed table table=objects46482026/11/12 00:03:21 INFO Completed push id=5 objects=28 duration=578µs46492026/11/12 00:03:21 INFO Received uploads request method=POST path=/46502026/11/12 00:03:21 INFO Vacuumed table table=closures46512026/11/12 00:03:21 INFO Received push request method=POST path=/46522026/11/12 00:03:21 INFO Vacuumed table table=objects46532026/11/12 00:03:21 INFO Received push request method=POST path=/46542026/11/12 00:03:21 INFO Received push request method=POST path=/46552026/11/12 00:03:21 INFO Received complete push request method=POST path=/46562026/11/12 00:03:21 INFO Completed push id=10 objects=2 duration=420.25µs46572026/11/12 00:03:21 INFO Received complete push request method=POST path=/46582026/11/12 00:03:21 INFO Completed push id=27 objects=11 duration=318.25µs46592026/11/12 00:03:21 INFO Received complete upload request method=POST path=/46602026/11/12 00:03:21 INFO Completed upload id=29 objects=26 duration=287.792µs46612026/11/12 00:03:21 INFO Received complete push request method=POST path=/46622026/11/12 00:03:21 INFO Completed push id=30 objects=8 duration=194.416µs46632026/11/12 00:03:21 INFO Received complete upload request method=POST path=/46642026/11/12 00:03:21 INFO Completed upload id=31 objects=5 duration=136.875µs46652026/11/12 00:03:21 INFO Received complete push request method=POST path=/46662026/11/12 00:03:22 INFO Completed push id=32 objects=3 duration=172.583µs46672026/11/12 00:03:22 INFO Received complete push request method=POST path=/46682026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=046692026/11/12 00:03:22 INFO Completed push id=16 objects=7 duration=435.084µs46702026/11/12 00:03:22 INFO Received complete upload request method=POST path=/46712026/11/12 00:03:22 INFO Completed upload id=17 objects=5 duration=242.459µs46722026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period46732026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046742026/11/12 00:03:22 INFO Vacuumed table table=pending_closures46752026/11/12 00:03:22 INFO Vacuumed table table=pending_objects46762026/11/12 00:03:22 INFO Vacuumed table table=multipart_uploads46772026/11/12 00:03:22 INFO Received complete push request method=POST path=/46782026/11/12 00:03:22 INFO Vacuumed table table=closures46792026/11/12 00:03:22 INFO Completed push id=29 objects=24 duration=391.917µs46802026/11/12 00:03:22 INFO Vacuumed table table=objects46812026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=046822026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period46832026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=046842026/11/12 00:03:22 INFO Received complete push request method=POST path=/46852026/11/12 00:03:22 INFO Received complete upload request method=POST path=/46862026/11/12 00:03:22 INFO Completed push id=19 objects=3 duration=269.917µs46872026/11/12 00:03:22 INFO Completed upload id=24 objects=18 duration=5.932542ms46882026/11/12 00:03:22 INFO Vacuumed table table=pending_closures46892026/11/12 00:03:22 INFO Received complete push request method=POST path=/46902026/11/12 00:03:22 INFO Completed push id=25 objects=2 duration=443.083µs46912026/11/12 00:03:22 INFO Vacuumed table table=pending_objects46922026/11/12 00:03:22 INFO Vacuumed table table=multipart_uploads46932026/11/12 00:03:22 INFO Received complete push request method=POST path=/46942026/11/12 00:03:22 INFO Vacuumed table table=closures46952026/11/12 00:03:22 INFO Completed push id=27 objects=4 duration=179.75µs46962026/11/12 00:03:22 INFO Vacuumed table table=objects46972026/11/12 00:03:22 INFO Received complete push request method=POST path=/46982026/11/12 00:03:22 INFO Completed push id=23 objects=6 duration=457.833µs46992026/11/12 00:03:22 INFO Received complete push request method=POST path=/47002026/11/12 00:03:22 INFO Completed push id=24 objects=6 duration=262.041µs47012026/11/12 00:03:22 INFO Received complete push request method=POST path=/47022026/11/12 00:03:22 INFO Completed push id=20 objects=4 duration=666.583µs47032026/11/12 00:03:22 INFO Received complete push request method=POST path=/47042026/11/12 00:03:22 INFO Completed push id=28 objects=26 duration=629.458µs47052026/11/12 00:03:22 INFO Received complete push request method=POST path=/47062026/11/12 00:03:22 INFO Completed push id=30 objects=3 duration=327.167µs47072026/11/12 00:03:22 INFO Received complete push request method=POST path=/47082026/11/12 00:03:22 INFO Received complete push request method=POST path=/47092026/11/12 00:03:22 INFO Completed push id=32 objects=19 duration=365.708µs47102026/11/12 00:03:22 INFO Completed push id=25 objects=7 duration=372.542µs47112026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=047122026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period47132026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047142026/11/12 00:03:22 INFO Vacuumed table table=pending_closures47152026/11/12 00:03:22 INFO Vacuumed table table=pending_objects47162026/11/12 00:03:22 INFO Vacuumed table table=multipart_uploads47172026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47182026/11/12 00:03:22 INFO Vacuumed table table=closures47192026/11/12 00:03:22 INFO Completed upload id=22 objects=8 duration=574.542µs47202026/11/12 00:03:22 INFO Vacuumed table table=objects47212026/11/12 00:03:22 INFO Received complete push request method=POST path=/47222026/11/12 00:03:22 INFO Completed push id=23 objects=5 duration=573.459µs47232026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47242026/11/12 00:03:22 INFO Completed upload id=24 objects=8 duration=375.125µs47252026/11/12 00:03:22 INFO Received complete push request method=POST path=/47262026/11/12 00:03:22 INFO Completed push id=19 objects=14 duration=660.625µs47272026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=047282026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47292026/11/12 00:03:22 INFO Vacuumed table table=pending_closures47302026/11/12 00:03:22 INFO Completed upload id=28 objects=5 duration=227.5µs47312026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=047322026/11/12 00:03:22 INFO Vacuumed table table=pending_objects47332026/11/12 00:03:22 INFO Vacuumed table table=multipart_uploads47342026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period47352026/11/12 00:03:22 INFO Vacuumed table table=closures47362026/11/12 00:03:22 INFO Vacuumed table table=objects47372026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=047382026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period47392026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047402026/11/12 00:03:22 INFO Vacuumed table table=pending_closures47412026/11/12 00:03:22 INFO Vacuumed table table=pending_objects47422026/11/12 00:03:22 INFO Vacuumed table table=multipart_uploads47432026/11/12 00:03:22 INFO Vacuumed table table=closures47442026/11/12 00:03:22 INFO Vacuumed table table=objects47452026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47462026/11/12 00:03:22 INFO Received complete push request method=POST path=/47472026/11/12 00:03:22 INFO Completed upload id=15 objects=3 duration=863.834µs47482026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47492026/11/12 00:03:22 INFO Completed push id=6 objects=26 duration=2.031958ms47502026/11/12 00:03:22 INFO Completed upload id=30 objects=5 duration=600.666µs47512026/11/12 00:03:22 INFO Received push request method=POST path=/47522026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=047532026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period47542026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047552026/11/12 00:03:22 INFO Vacuumed table table=pending_closures47562026/11/12 00:03:22 INFO Vacuumed table table=pending_objects47572026/11/12 00:03:22 INFO Vacuumed table table=multipart_uploads47582026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47592026/11/12 00:03:22 INFO Vacuumed table table=closures47602026/11/12 00:03:22 INFO Completed upload id=26 objects=24 duration=5.453833ms47612026/11/12 00:03:22 INFO Vacuumed table table=objects47622026/11/12 00:03:22 INFO Received uploads request method=POST path=/47632026/11/12 00:03:22 INFO Received complete push request method=POST path=/47642026/11/12 00:03:22 INFO Completed push id=31 objects=3 duration=479.25µs47652026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=047662026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period47672026/11/12 00:03:22 INFO Received complete push request method=POST path=/47682026/11/12 00:03:22 INFO Completed push id=32 objects=6 duration=683.875µs47692026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47702026/11/12 00:03:22 INFO Completed upload id=33 objects=7 duration=273.875µs47712026/11/12 00:03:22 INFO Received complete upload request method=POST path=/47722026/11/12 00:03:22 INFO Completed upload id=34 objects=3 duration=276.5µs47732026/11/12 00:03:22 INFO Received complete push request method=POST path=/47742026/11/12 00:03:22 INFO Completed push id=35 objects=11 duration=355.041µs47752026/11/12 00:03:22 INFO Aborted multipart uploads count=0 kept=047762026/11/12 00:03:22 WARN Force mode enabled - objects will be deleted immediately without grace period47772026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047782026/11/12 00:03:22 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=047792026/11/12 00:03:22 INFO Vacuumed table table=pending_closures47802026/11/12 00:03:22 INFO Vacuumed table table=pending_closures47812026/11/12 00:03:22 INFO Vacuumed table table=pending_objects47822026/11/12 00:03:22 INFO Vacuumed table table=multipart_uploads47832026/11/12 00:03:23 INFO Received complete upload request method=POST path=/47842026/11/12 00:03:23 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."47852026/11/12 00:03:23 INFO Vacuumed table table=pending_objects47862026/11/12 00:03:23 INFO Vacuumed table table=closures47872026/11/12 00:03:23 INFO Completed upload id=27 objects=26 duration=418.375µs47882026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads47892026/11/12 00:03:23 INFO Vacuumed table table=objects47902026/11/12 00:03:23 INFO Vacuumed table table=closures47912026/11/12 00:03:23 INFO Vacuumed table table=objects47922026/11/12 00:03:23 INFO Received complete push request method=POST path=/47932026/11/12 00:03:23 INFO Received complete push request method=POST path=/47942026/11/12 00:03:23 INFO Completed push id=28 objects=6 duration=989.833µs47952026/11/12 00:03:23 INFO Completed push id=17 objects=22 duration=1.311292ms47962026/11/12 00:03:23 INFO Received push request method=POST path=/47972026/11/12 00:03:23 INFO Received uploads request method=POST path=/47982026/11/12 00:03:23 INFO Received uploads request method=POST path=/47992026/11/12 00:03:23 INFO Received complete push request method=POST path=/48002026/11/12 00:03:23 INFO Completed push id=29 objects=6 duration=305.25µs48012026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=048022026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048032026/11/12 00:03:23 INFO Vacuumed table table=pending_closures48042026/11/12 00:03:23 INFO Vacuumed table table=pending_objects48052026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads48062026/11/12 00:03:23 INFO Received push request method=POST path=/48072026/11/12 00:03:23 INFO Vacuumed table table=closures48082026/11/12 00:03:23 INFO Vacuumed table table=objects48092026/11/12 00:03:23 INFO Received complete push request method=POST path=/48102026/11/12 00:03:23 INFO Completed push id=19 objects=14 duration=436.333µs48112026/11/12 00:03:23 INFO Received uploads request method=POST path=/48122026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48132026/11/12 00:03:23 INFO Completed upload id=20 objects=6 duration=602.083µs48142026/11/12 00:03:23 INFO Received uploads request method=POST path=/48152026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48162026/11/12 00:03:23 INFO Completed upload id=21 objects=2 duration=242.333µs48172026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48182026/11/12 00:03:23 INFO Completed upload id=30 objects=8 duration=262µs48192026/11/12 00:03:23 INFO Received uploads request method=POST path=/48202026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=048212026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period48222026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048232026/11/12 00:03:23 INFO Vacuumed table table=pending_closures48242026/11/12 00:03:23 INFO Vacuumed table table=pending_objects48252026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads48262026/11/12 00:03:23 INFO Vacuumed table table=closures48272026/11/12 00:03:23 INFO Vacuumed table table=objects48282026/11/12 00:03:23 INFO Received uploads request method=POST path=/48292026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48302026/11/12 00:03:23 INFO Received push request method=POST path=/48312026/11/12 00:03:23 INFO Completed upload id=31 objects=6 duration=213.541µs48322026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48332026/11/12 00:03:23 INFO Received complete push request method=POST path=/48342026/11/12 00:03:23 INFO Completed upload id=32 objects=4 duration=166.417µs48352026/11/12 00:03:23 INFO Completed push id=24 objects=24 duration=762.834µs48362026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48372026/11/12 00:03:23 INFO Completed upload id=19 objects=25 duration=883.084µs48382026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=048392026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period48402026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=048412026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048422026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period48432026/11/12 00:03:23 INFO Vacuumed table table=pending_closures48442026/11/12 00:03:23 INFO Vacuumed table table=pending_objects48452026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads48462026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=048472026/11/12 00:03:23 INFO Vacuumed table table=closures48482026/11/12 00:03:23 INFO Vacuumed table table=pending_closures48492026/11/12 00:03:23 INFO Vacuumed table table=objects48502026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48512026/11/12 00:03:23 INFO Vacuumed table table=pending_objects48522026/11/12 00:03:23 INFO Completed upload id=33 objects=10 duration=5.716958ms48532026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads48542026/11/12 00:03:23 INFO Vacuumed table table=closures48552026/11/12 00:03:23 INFO Vacuumed table table=objects48562026/11/12 00:03:23 INFO Received uploads request method=POST path=/48572026/11/12 00:03:23 INFO Received uploads request method=POST path=/48582026/11/12 00:03:23 INFO Received push request method=POST path=/48592026/11/12 00:03:23 INFO Received uploads request method=POST path=/48602026/11/12 00:03:23 INFO Received push request method=POST path=/48612026/11/12 00:03:23 INFO Received uploads request method=POST path=/48622026/11/12 00:03:23 INFO Received uploads request method=POST path=/48632026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=048642026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period48652026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048662026/11/12 00:03:23 INFO Received push request method=POST path=/48672026/11/12 00:03:23 INFO Vacuumed table table=pending_closures48682026/11/12 00:03:23 INFO Vacuumed table table=pending_objects48692026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads48702026/11/12 00:03:23 INFO Vacuumed table table=closures48712026/11/12 00:03:23 INFO Received push request method=POST path=/48722026/11/12 00:03:23 INFO Vacuumed table table=objects48732026/11/12 00:03:23 INFO Received push request method=POST path=/48742026/11/12 00:03:23 INFO Received uploads request method=POST path=/48752026/11/12 00:03:23 INFO Received complete push request method=POST path=/48762026/11/12 00:03:23 INFO Completed push id=16 objects=4 duration=661.625µs48772026/11/12 00:03:23 INFO Received uploads request method=POST path=/48782026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48792026/11/12 00:03:23 INFO Completed upload id=20 objects=5 duration=197.458µs48802026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48812026/11/12 00:03:23 INFO Completed upload id=34 objects=23 duration=271.625µs48822026/11/12 00:03:23 INFO Received uploads request method=POST path=/48832026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48842026/11/12 00:03:23 INFO Received push request method=POST path=/48852026/11/12 00:03:23 INFO Completed upload id=23 objects=7 duration=773.25µs48862026/11/12 00:03:23 INFO Received push request method=POST path=/48872026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48882026/11/12 00:03:23 INFO Completed upload id=26 objects=5 duration=191.417µs48892026/11/12 00:03:23 INFO Received push request method=POST path=/48902026/11/12 00:03:23 INFO Received complete push request method=POST path=/48912026/11/12 00:03:23 INFO Completed push id=27 objects=3 duration=442.875µs48922026/11/12 00:03:23 INFO Received complete push request method=POST path=/48932026/11/12 00:03:23 INFO Received complete upload request method=POST path=/48942026/11/12 00:03:23 INFO Completed upload id=28 objects=5 duration=259.708µs48952026/11/12 00:03:23 INFO Completed push id=35 objects=6 duration=330.541µs48962026/11/12 00:03:23 INFO Received complete push request method=POST path=/48972026/11/12 00:03:23 INFO Completed push id=29 objects=17 duration=224.916µs48982026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=048992026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period49002026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=049012026/11/12 00:03:23 INFO Received complete upload request method=POST path=/49022026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period49032026/11/12 00:03:23 INFO Completed upload id=13 objects=14 duration=1.238292ms49042026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049052026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049062026/11/12 00:03:23 INFO Vacuumed table table=pending_closures49072026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=049082026/11/12 00:03:23 INFO Vacuumed table table=pending_objects49092026/11/12 00:03:23 INFO Vacuumed table table=pending_closures49102026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period49112026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads49122026/11/12 00:03:23 INFO Vacuumed table table=closures49132026/11/12 00:03:23 INFO Vacuumed table table=pending_objects49142026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049152026/11/12 00:03:23 INFO Vacuumed table table=objects49162026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads49172026/11/12 00:03:23 INFO Vacuumed table table=closures49182026/11/12 00:03:23 INFO Vacuumed table table=pending_closures49192026/11/12 00:03:23 INFO Vacuumed table table=objects49202026/11/12 00:03:23 INFO Vacuumed table table=pending_objects49212026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads49222026/11/12 00:03:23 INFO Vacuumed table table=closures49232026/11/12 00:03:23 INFO Vacuumed table table=objects49242026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=049252026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049262026/11/12 00:03:23 INFO Vacuumed table table=pending_closures49272026/11/12 00:03:23 INFO Vacuumed table table=pending_objects49282026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads49292026/11/12 00:03:23 INFO Vacuumed table table=closures49302026/11/12 00:03:23 INFO Vacuumed table table=objects49312026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=049322026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049332026/11/12 00:03:23 INFO Vacuumed table table=pending_closures49342026/11/12 00:03:23 INFO Vacuumed table table=pending_objects49352026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads49362026/11/12 00:03:23 INFO Vacuumed table table=closures49372026/11/12 00:03:23 INFO Vacuumed table table=objects49382026/11/12 00:03:23 INFO Received push request method=POST path=/49392026/11/12 00:03:23 INFO Received complete push request method=POST path=/49402026/11/12 00:03:23 INFO Completed push id=31 objects=6 duration=997.791µs49412026/11/12 00:03:23 INFO Received push request method=POST path=/49422026/11/12 00:03:23 INFO Received uploads request method=POST path=/49432026/11/12 00:03:23 INFO Aborted multipart uploads count=0 kept=049442026/11/12 00:03:23 WARN Force mode enabled - objects will be deleted immediately without grace period49452026/11/12 00:03:23 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049462026/11/12 00:03:23 INFO Vacuumed table table=pending_closures49472026/11/12 00:03:23 INFO Vacuumed table table=pending_objects49482026/11/12 00:03:23 INFO Vacuumed table table=multipart_uploads49492026/11/12 00:03:23 INFO Vacuumed table table=closures49502026/11/12 00:03:23 INFO Vacuumed table table=objects49512026/11/12 00:03:23 INFO Received push request method=POST path=/49522026/11/12 00:03:23 INFO Received push request method=POST path=/49532026/11/12 00:03:23 INFO Received complete push request method=POST path=/49542026/11/12 00:03:23 INFO Completed push id=31 objects=14 duration=593.625µs49552026/11/12 00:03:23 INFO Received uploads request method=POST path=/49562026/11/12 00:03:23 INFO Received complete push request method=POST path=/49572026/11/12 00:03:23 INFO Completed push id=7 objects=7 duration=760.084µs49582026/11/12 00:03:23 INFO Received complete push request method=POST path=/49592026/11/12 00:03:23 INFO Completed push id=12 objects=6 duration=437.25µs49602026/11/12 00:03:23 INFO Received complete push request method=POST path=/49612026/11/12 00:03:23 INFO Completed push id=1 objects=9 duration=361.083µs49622026/11/12 00:03:23 INFO Received complete push request method=POST path=/49632026/11/12 00:03:23 INFO Completed push id=13 objects=22 duration=1.450833ms49642026/11/12 00:03:23 INFO Received complete push request method=POST path=/49652026/11/12 00:03:23 INFO Completed push id=21 objects=6 duration=1.019583ms49662026/11/12 00:03:23 INFO Received complete upload request method=POST path=/49672026/11/12 00:03:23 INFO Completed upload id=22 objects=7 duration=526.042µs49682026/11/12 00:03:23 INFO Received complete push request method=POST path=/49692026/11/12 00:03:23 INFO Completed push id=24 objects=4 duration=332.792µs49702026/11/12 00:03:23 INFO Received complete push request method=POST path=/49712026/11/12 00:03:23 INFO Completed push id=25 objects=9 duration=275.292µs49722026/11/12 00:03:23 INFO Received complete upload request method=POST path=/49732026/11/12 00:03:23 INFO Completed upload id=2 objects=7 duration=387.709µs49742026/11/12 00:03:23 INFO Received complete upload request method=POST path=/49752026/11/12 00:03:23 INFO Completed upload id=26 objects=3 duration=180.958µs49762026/11/12 00:03:24 INFO Received complete upload request method=POST path=/49772026/11/12 00:03:24 INFO Completed upload id=27 objects=7 duration=248.75µs49782026/11/12 00:03:24 INFO Received complete upload request method=POST path=/49792026/11/12 00:03:24 INFO Completed upload id=28 objects=3 duration=200.375µs49802026/11/12 00:03:24 INFO Received complete push request method=POST path=/49812026/11/12 00:03:24 INFO Completed push id=29 objects=23 duration=324.5µs49822026/11/12 00:03:24 INFO Received complete push request method=POST path=/49832026/11/12 00:03:24 INFO Completed push id=30 objects=6 duration=245.791µs49842026/11/12 00:03:24 INFO Received complete upload request method=POST path=/49852026/11/12 00:03:24 INFO Completed upload id=8 objects=6 duration=291.083µs49862026/11/12 00:03:24 INFO Received complete push request method=POST path=/49872026/11/12 00:03:24 INFO Completed push id=33 objects=3 duration=256.958µs49882026/11/12 00:03:24 INFO Received complete push request method=POST path=/49892026/11/12 00:03:24 INFO Completed push id=34 objects=6 duration=380.958µs49902026/11/12 00:03:24 INFO Received complete push request method=POST path=/49912026/11/12 00:03:24 INFO Completed push id=35 objects=8 duration=337.833µs49922026/11/12 00:03:24 INFO Received complete upload request method=POST path=/49932026/11/12 00:03:24 INFO Completed upload id=36 objects=9 duration=164.25µs49942026/11/12 00:03:24 INFO Aborted multipart uploads count=0 kept=049952026/11/12 00:03:24 WARN Force mode enabled - objects will be deleted immediately without grace period49962026/11/12 00:03:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049972026/11/12 00:03:24 INFO Vacuumed table table=pending_closures49982026/11/12 00:03:24 INFO Vacuumed table table=pending_objects49992026/11/12 00:03:24 INFO Vacuumed table table=multipart_uploads50002026/11/12 00:03:24 INFO Vacuumed table table=closures50012026/11/12 00:03:24 INFO Vacuumed table table=objects50022026/11/12 00:03:24 INFO Received complete upload request method=POST path=/50032026/11/12 00:03:24 INFO Completed upload id=12 objects=9 duration=334.667µs50042026/11/12 00:03:24 INFO Received complete upload request method=POST path=/50052026/11/12 00:03:24 INFO Completed upload id=18 objects=22 duration=1.410375ms50062026/11/12 00:03:24 INFO Received complete upload request method=POST path=/50072026/11/12 00:03:24 INFO Completed upload id=21 objects=8 duration=661.667µs50082026/11/12 00:03:24 INFO Received complete upload request method=POST path=/50092026/11/12 00:03:24 INFO Completed upload id=22 objects=7 duration=803.958µs50102026/11/12 00:03:24 INFO Received complete push request method=POST path=/50112026/11/12 00:03:24 INFO Completed push id=24 objects=5 duration=403.875µs50122026/11/12 00:03:24 INFO Received complete upload request method=POST path=/50132026/11/12 00:03:24 INFO Completed upload id=25 objects=8 duration=322.75µs50142026/11/12 00:03:24 INFO Received complete push request method=POST path=/50152026/11/12 00:03:24 INFO Completed push id=26 objects=4 duration=332.291µs50162026/11/12 00:03:24 INFO Received complete push request method=POST path=/50172026/11/12 00:03:24 INFO Completed push id=30 objects=2 duration=239.334µs50182026/11/12 00:03:24 INFO Received complete push request method=POST path=/50192026/11/12 00:03:24 INFO Completed push id=32 objects=7 duration=245.25µs50202026/11/12 00:03:24 INFO Received complete upload request method=POST path=/50212026/11/12 00:03:24 INFO Completed upload id=33 objects=6 duration=191.125µs50222026/11/12 00:03:24 INFO Received complete upload request method=POST path=/50232026/11/12 00:03:24 INFO Completed upload id=34 objects=3 duration=165.625µs50242026/11/12 00:03:24 INFO Aborted multipart uploads count=0 kept=050252026/11/12 00:03:24 WARN Force mode enabled - objects will be deleted immediately without grace period50262026/11/12 00:03:24 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=050272026/11/12 00:03:24 INFO Vacuumed table table=pending_closures50282026/11/12 00:03:24 INFO Vacuumed table table=pending_objects50292026/11/12 00:03:24 INFO Vacuumed table table=multipart_uploads50302026/11/12 00:03:24 INFO Vacuumed table table=closures50312026/11/12 00:03:24 INFO Vacuumed table table=objects5032--- PASS: FuzzPushGC (0.00s)5033 --- PASS: FuzzPushGC/seed#0 (7.47s)5034 --- PASS: FuzzPushGC/seed#15 (8.67s)5035 --- PASS: FuzzPushGC/seed#5 (8.77s)5036 --- PASS: FuzzPushGC/seed#13 (10.12s)5037 --- PASS: FuzzPushGC/seed#14 (11.11s)5038 --- PASS: FuzzPushGC/seed#6 (12.53s)5039 --- PASS: FuzzPushGC/seed#9 (12.54s)5040 --- PASS: FuzzPushGC/seed#11 (12.55s)5041 --- PASS: FuzzPushGC/seed#3 (12.72s)5042 --- PASS: FuzzPushGC/seed#2 (12.79s)5043 --- PASS: FuzzPushGC/seed#7 (13.19s)5044 --- PASS: FuzzPushGC/seed#8 (13.63s)5045 --- PASS: FuzzPushGC/seed#12 (14.09s)5046 --- PASS: FuzzPushGC/seed#1 (14.09s)5047 --- PASS: FuzzPushGC/seed#4 (14.75s)5048 --- PASS: FuzzPushGC/seed#10 (15.28s)5049PASS5050{"timestamp":"2026-11-12T00:03:24.661601Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:62403","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(8)"}5051{"timestamp":"2026-11-12T00:03:24.662164Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:62176","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(5)"}5052{"timestamp":"2026-11-12T00:03:24.662288Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:62465","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(8)"}50532026-11-12 00:03:24.813 UTC [5191] LOG: received smart shutdown request50542026-11-12 00:03:24.814 UTC [5191] LOG: background worker "logical replication launcher" (PID 5201) exited with exit code 150552026-11-12 00:03:25.739 UTC [5196] LOG: checkpoint complete: wrote 3592 buffers (21.9%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=27.411 s, sync=0.811 s, total=28.284 s; sync files=42374, longest=0.001 s, average=0.001 s; distance=533348 kB, estimate=533348 kB; lsn=0/279856E0, redo lsn=0/220662D050562026-11-12 00:03:25.740 UTC [5196] LOG: shutting down50572026-11-12 00:03:25.740 UTC [5196] LOG: checkpoint starting: shutdown immediate50582026-11-12 00:03:26.247 UTC [5196] LOG: checkpoint complete: wrote 8796 buffers (53.7%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 5 recycled; write=0.455 s, sync=0.048 s, total=0.507 s; sync files=2906, longest=0.001 s, average=0.001 s; distance=91261 kB, estimate=489140 kB; lsn=0/27985758, redo lsn=0/2798575850592026-11-12 00:03:26.253 UTC [5191] LOG: database system is shut down5060Running OIDC tests...5061=== RUN TestAudienceForIssuer5062=== PAUSE TestAudienceForIssuer5063=== RUN TestGlobMatch5064=== PAUSE TestGlobMatch5065=== RUN TestValidateToken_ValidToken5066=== PAUSE TestValidateToken_ValidToken5067=== RUN TestValidateToken_WrongAudience5068=== PAUSE TestValidateToken_WrongAudience5069=== RUN TestValidateToken_Expired5070=== PAUSE TestValidateToken_Expired5071=== RUN TestValidateToken_BoundClaimsMismatch5072=== PAUSE TestValidateToken_BoundClaimsMismatch5073=== RUN TestValidateToken_BoundSubjectMismatch5074=== PAUSE TestValidateToken_BoundSubjectMismatch5075=== RUN TestValidateToken_MultipleProviders5076=== PAUSE TestValidateToken_MultipleProviders5077=== RUN TestValidateToken_NoMatchingProvider5078=== PAUSE TestValidateToken_NoMatchingProvider5079=== RUN TestValidateToken_KubernetesServiceAccount5080=== PAUSE TestValidateToken_KubernetesServiceAccount5081=== RUN TestNewValidator_KubernetesRequiresCA5082=== PAUSE TestNewValidator_KubernetesRequiresCA5083=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5084=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5085=== RUN TestPins_ReservedForMatchingRule5086=== PAUSE TestPins_ReservedForMatchingRule5087=== RUN TestPins_TopLevelShorthand5088=== PAUSE TestPins_TopLevelShorthand5089=== RUN TestPins_ConfigValidation5090=== PAUSE TestPins_ConfigValidation5091=== RUN TestScopes_LegacyProviderDefaultsToWrite5092=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5093=== RUN TestScopes_Rules5094=== PAUSE TestScopes_Rules5095=== RUN TestScopes_ConfigValidation5096=== PAUSE TestScopes_ConfigValidation5097=== CONT TestAudienceForIssuer5098--- PASS: TestAudienceForIssuer (0.00s)5099=== CONT TestValidateToken_Expired5100=== CONT TestScopes_ConfigValidation5101=== CONT TestValidateToken_BoundClaimsMismatch5102=== CONT TestValidateToken_KubernetesServiceAccount5103=== CONT TestScopes_Rules5104=== CONT TestScopes_LegacyProviderDefaultsToWrite5105=== CONT TestPins_ConfigValidation5106=== CONT TestPins_TopLevelShorthand5107--- PASS: TestScopes_ConfigValidation (0.00s)5108=== CONT TestPins_ReservedForMatchingRule5109--- PASS: TestPins_ConfigValidation (0.01s)5110=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5111=== CONT TestNewValidator_KubernetesRequiresCA5112=== CONT TestValidateToken_MultipleProviders5113=== CONT TestValidateToken_NoMatchingProvider5114=== CONT TestValidateToken_ValidToken5115=== CONT TestValidateToken_WrongAudience5116=== CONT TestGlobMatch5117=== RUN TestGlobMatch/foo_foo5118=== PAUSE TestGlobMatch/foo_foo5119=== RUN TestGlobMatch/foo_bar5120=== PAUSE TestGlobMatch/foo_bar5121=== RUN TestGlobMatch/*_5122=== PAUSE TestGlobMatch/*_5123=== RUN TestGlobMatch/*_anything5124=== PAUSE TestGlobMatch/*_anything5125=== RUN TestGlobMatch/foo*_foo5126=== PAUSE TestGlobMatch/foo*_foo5127=== RUN TestGlobMatch/foo*_foobar5128=== PAUSE TestGlobMatch/foo*_foobar5129=== RUN TestGlobMatch/foo*_bar5130=== PAUSE TestGlobMatch/foo*_bar5131=== RUN TestGlobMatch/*bar_bar5132=== PAUSE TestGlobMatch/*bar_bar5133=== RUN TestGlobMatch/*bar_foobar5134=== PAUSE TestGlobMatch/*bar_foobar5135=== RUN TestGlobMatch/*bar_foo5136=== PAUSE TestGlobMatch/*bar_foo5137=== RUN TestGlobMatch/foo*bar_foobar5138=== PAUSE TestGlobMatch/foo*bar_foobar5139=== RUN TestGlobMatch/foo*bar_foo123bar5140=== PAUSE TestGlobMatch/foo*bar_foo123bar5141=== RUN TestGlobMatch/foo*bar_foobarbaz5142=== PAUSE TestGlobMatch/foo*bar_foobarbaz5143=== RUN TestGlobMatch/*/*_foo/bar5144=== PAUSE TestGlobMatch/*/*_foo/bar5145=== RUN TestGlobMatch/*/*_foo5146=== PAUSE TestGlobMatch/*/*_foo5147=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5148=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5149=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05150=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05151=== RUN TestGlobMatch/refs/*/main_refs/heads/main5152=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5153=== RUN TestGlobMatch/fo?_foo5154=== PAUSE TestGlobMatch/fo?_foo5155=== RUN TestGlobMatch/fo?_fo5156=== PAUSE TestGlobMatch/fo?_fo5157=== RUN TestGlobMatch/fo?_fooo5158=== PAUSE TestGlobMatch/fo?_fooo5159=== RUN TestGlobMatch/?oo_foo5160=== PAUSE TestGlobMatch/?oo_foo5161=== RUN TestGlobMatch/?oo_boo5162=== PAUSE TestGlobMatch/?oo_boo5163=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5164=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5165=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5166=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5167=== CONT TestGlobMatch/foo_foo5168=== CONT TestValidateToken_BoundSubjectMismatch5169=== CONT TestGlobMatch/*/*_foo/bar5170=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5171=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5172=== CONT TestGlobMatch/?oo_boo5173=== CONT TestGlobMatch/?oo_foo5174=== CONT TestGlobMatch/fo?_fooo5175=== CONT TestGlobMatch/fo?_fo5176=== CONT TestGlobMatch/fo?_foo5177=== CONT TestGlobMatch/refs/*/main_refs/heads/main5178=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05179=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5180=== CONT TestGlobMatch/*/*_foo5181=== CONT TestGlobMatch/*bar_bar5182=== CONT TestGlobMatch/foo*bar_foobarbaz5183=== CONT TestGlobMatch/foo*bar_foo123bar5184=== CONT TestGlobMatch/foo*bar_foobar5185=== CONT TestGlobMatch/*bar_foo5186=== CONT TestGlobMatch/*bar_foobar5187=== CONT TestGlobMatch/foo*_foo5188=== CONT TestGlobMatch/foo*_bar5189=== CONT TestGlobMatch/foo*_foobar5190=== CONT TestGlobMatch/*_5191=== CONT TestGlobMatch/*_anything5192=== CONT TestGlobMatch/foo_bar5193--- PASS: TestGlobMatch (0.00s)5194 --- PASS: TestGlobMatch/foo_foo (0.00s)5195 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5196 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5197 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5198 --- PASS: TestGlobMatch/?oo_boo (0.00s)5199 --- PASS: TestGlobMatch/?oo_foo (0.00s)5200 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5201 --- PASS: TestGlobMatch/fo?_fo (0.00s)5202 --- PASS: TestGlobMatch/fo?_foo (0.00s)5203 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5204 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5205 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)5206 --- PASS: TestGlobMatch/*/*_foo (0.00s)5207 --- PASS: TestGlobMatch/*bar_bar (0.00s)5208 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5209 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5210 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5211 --- PASS: TestGlobMatch/*bar_foo (0.00s)5212 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5213 --- PASS: TestGlobMatch/foo*_foo (0.00s)5214 --- PASS: TestGlobMatch/foo*_bar (0.00s)5215 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5216 --- PASS: TestGlobMatch/*_ (0.00s)5217 --- PASS: TestGlobMatch/*_anything (0.00s)5218 --- PASS: TestGlobMatch/foo_bar (0.00s)52192026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62899/oidc5220--- PASS: TestPins_TopLevelShorthand (0.04s)52212026/11/12 00:03:28 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12352222026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62901/oidc52232026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62904/oidc52242026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62906/oidc5225--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.07s)5226--- PASS: TestValidateToken_ValidToken (0.06s)5227--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.06s)52282026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62909/oidc5229--- PASS: TestPins_ReservedForMatchingRule (0.06s)52302026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62911/oidc5231--- PASS: TestValidateToken_WrongAudience (0.06s)5232--- PASS: TestValidateToken_Expired (0.07s)52332026/11/12 00:03:28 http: TLS handshake error from 127.0.0.1:62914: remote error: tls: bad certificate5234--- PASS: TestNewValidator_KubernetesRequiresCA (0.08s)52352026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62916/oidc52362026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62918/oidc5237--- PASS: TestValidateToken_BoundClaimsMismatch (0.11s)5238--- PASS: TestScopes_Rules (0.11s)52392026/11/12 00:03:28 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:629205240--- PASS: TestValidateToken_KubernetesServiceAccount (0.12s)52412026/11/12 00:03:28 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:62922/oidc52422026/11/12 00:03:28 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:62915/oidc5243--- PASS: TestValidateToken_MultipleProviders (0.14s)52442026/11/12 00:03:28 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:62926/oidc5245--- PASS: TestValidateToken_BoundSubjectMismatch (0.16s)52462026/11/12 00:03:28 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:62925/oidc5247--- PASS: TestValidateToken_NoMatchingProvider (0.17s)5248PASS5249Running hook tests...5250=== RUN TestSendPathsEmpty5251=== PAUSE TestSendPathsEmpty5252=== RUN TestQueueEnqueueAndFetch5253=== PAUSE TestQueueEnqueueAndFetch5254=== RUN TestQueueDeduplication5255=== PAUSE TestQueueDeduplication5256=== RUN TestQueueRemove5257=== PAUSE TestQueueRemove5258=== RUN TestQueueFetchBatchLimit5259=== PAUSE TestQueueFetchBatchLimit5260=== RUN TestQueueRetryMovesToBack5261=== PAUSE TestQueueRetryMovesToBack5262=== RUN TestQueueFetchRemoveLifecycle5263=== PAUSE TestQueueFetchRemoveLifecycle5264=== RUN TestQueueConcurrentWriters5265=== PAUSE TestQueueConcurrentWriters5266=== RUN TestQueueRemoveLargeClosure5267=== PAUSE TestQueueRemoveLargeClosure5268=== RUN TestServerClientIntegration5269--- PASS: TestServerClientIntegration (0.00s)5270=== RUN TestServerQueueError52712026/11/12 00:03:28 ERROR Failed to queue paths error="permission denied" count=15272--- PASS: TestServerQueueError (0.00s)5273=== RUN TestGetListenerSocketActivation5274 server_test.go:147: systemd socket activation does not exist on darwin5275--- SKIP: TestGetListenerSocketActivation (0.00s)5276=== RUN TestServerStalledClientDoesNotBlockShutdown5277=== PAUSE TestServerStalledClientDoesNotBlockShutdown5278=== RUN TestServerBacksOffOnAcceptErrors5279=== PAUSE TestServerBacksOffOnAcceptErrors5280=== RUN TestDrainIsolatesPoisonPath5281=== PAUSE TestDrainIsolatesPoisonPath5282=== RUN TestRunNotBlockedByPoisonHead5283=== PAUSE TestRunNotBlockedByPoisonHead5284=== RUN TestDrainGivesUpWhenServerDown5285=== PAUSE TestDrainGivesUpWhenServerDown5286=== RUN TestFailedPathPrunedByLaterClosure5287=== PAUSE TestFailedPathPrunedByLaterClosure5288=== RUN TestWorkerUploadsAndRemoves5289=== PAUSE TestWorkerUploadsAndRemoves5290=== RUN TestWorkerSkipsGCdPaths5291=== PAUSE TestWorkerSkipsGCdPaths5292=== RUN TestWorkerPrunesClosureDeps5293=== PAUSE TestWorkerPrunesClosureDeps5294=== RUN TestDrainTimeout5295=== PAUSE TestDrainTimeout5296=== RUN TestWorkerRemoveFailureIsNotProgress5297=== PAUSE TestWorkerRemoveFailureIsNotProgress5298=== RUN TestWorkerKeepsPathItCannotStat5299=== PAUSE TestWorkerKeepsPathItCannotStat5300=== CONT TestWorkerKeepsPathItCannotStat5301=== CONT TestSendPathsEmpty5302--- PASS: TestSendPathsEmpty (0.00s)5303=== CONT TestWorkerRemoveFailureIsNotProgress5304=== CONT TestServerBacksOffOnAcceptErrors5305=== CONT TestQueueRetryMovesToBack53062000/01/01 00:00:00 ERROR Accept failed error="too many open files"53072000/01/01 00:00:00 ERROR Accept failed error="too many open files"53082000/01/01 00:00:00 ERROR Accept failed error="too many open files"53092000/01/01 00:00:00 ERROR Accept failed error="too many open files"53102000/01/01 00:00:00 ERROR Accept failed error="too many open files"53112000/01/01 00:00:00 ERROR Accept failed error="too many open files"53122000/01/01 00:00:00 ERROR Accept failed error="too many open files"53132000/01/01 00:00:00 ERROR Accept failed error="too many open files"5314=== CONT TestDrainTimeout5315=== CONT TestQueueRemove5316=== CONT TestQueueFetchBatchLimit5317=== CONT TestQueueConcurrentWriters5318=== CONT TestQueueFetchRemoveLifecycle5319=== CONT TestQueueEnqueueAndFetch5320=== CONT TestFailedPathPrunedByLaterClosure5321--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)5322=== CONT TestWorkerPrunesClosureDeps5323=== CONT TestWorkerSkipsGCdPaths5324=== CONT TestWorkerUploadsAndRemoves5325=== CONT TestRunNotBlockedByPoisonHead5326=== CONT TestDrainGivesUpWhenServerDown5327=== CONT TestQueueDeduplication5328=== CONT TestQueueRemoveLargeClosure5329=== CONT TestServerStalledClientDoesNotBlockShutdown5330=== CONT TestDrainIsolatesPoisonPath53312026/11/12 00:03:28 INFO Upload queue status pending=253322026/11/12 00:03:28 WARN Cannot stat store path, will retry later path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerKeepsPathItCannotStat1964195782/002/locked/aaa error="stat /private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerKeepsPathItCannotStat1964195782/002/locked/aaa: permission denied"53332026/11/12 00:03:28 INFO Uploading batch count=153342026/11/12 00:03:28 INFO Upload queue status pending=253352026/11/12 00:03:28 INFO Upload queue status pending=253362026/11/12 00:03:28 WARN Cannot stat store path, will retry later path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerKeepsPathItCannotStat1964195782/002/locked/aaa error="stat /private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerKeepsPathItCannotStat1964195782/002/locked/aaa: permission denied"53372026/11/12 00:03:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerSkipsGCdPaths3137037573/002/nonexistent53382026/11/12 00:03:28 INFO Uploading batch count=253392026/11/12 00:03:28 INFO Uploading batch count=153402000/01/01 00:00:00 INFO Uploading batch count=253412000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253422026/11/12 00:03:28 INFO Upload queue status pending=353432026/11/12 00:03:28 INFO Uploading batch count=153442026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=15345--- PASS: TestQueueEnqueueAndFetch (0.01s)53462026/11/12 00:03:28 INFO Uploading batch count=453472026/11/12 00:03:28 WARN Cannot stat store path, will retry later path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerKeepsPathItCannotStat1964195782/002/locked/aaa error="stat /private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerKeepsPathItCannotStat1964195782/002/locked/aaa: permission denied"53482026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=453492000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453502026/11/12 00:03:28 INFO Uploading batch count=153512026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=153522026/11/12 00:03:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerRemoveFailureIsNotProgress3325976520/002/nonexistent53532026/11/12 00:03:28 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15354--- PASS: TestQueueFetchRemoveLifecycle (0.01s)53552026/11/12 00:03:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerRemoveFailureIsNotProgress3325976520/002/nonexistent53562026/11/12 00:03:28 INFO Uploading batch count=25357--- PASS: TestQueueDeduplication (0.01s)53582026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=253592026/11/12 00:03:28 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestDrainGivesUpWhenServerDown2319674382/002/a53602026/11/12 00:03:28 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153612026/11/12 00:03:28 ERROR Drain finished with paths left in queue remaining=153622026/11/12 00:03:28 WARN Store path no longer exists (garbage collected?), removing from queue path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestWorkerRemoveFailureIsNotProgress3325976520/002/nonexistent53632026/11/12 00:03:28 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestDrainIsolatesPoisonPath1939788198/002/bbb53642026/11/12 00:03:28 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153652026/11/12 00:03:28 INFO Uploading batch count=153662026/11/12 00:03:28 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestDrainGivesUpWhenServerDown2319674382/002/b5367--- PASS: TestQueueRetryMovesToBack (0.02s)53682026/11/12 00:03:28 ERROR Drain finished with paths left in queue remaining=15369--- PASS: TestQueueFetchBatchLimit (0.02s)53702026/11/12 00:03:28 INFO Uploading batch count=253712026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=253722026/11/12 00:03:28 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestDrainGivesUpWhenServerDown2319674382/002/c53732026/11/12 00:03:28 INFO Uploading batch count=153742026/11/12 00:03:28 INFO Uploading batch count=153752026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=153762026/11/12 00:03:28 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestDrainGivesUpWhenServerDown2319674382/002/d53772026/11/12 00:03:28 INFO Uploading batch count=153782026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=153792026/11/12 00:03:28 INFO Uploading batch count=153802026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=153812026/11/12 00:03:28 INFO Uploading batch count=253822026/11/12 00:03:28 ERROR Upload failed error="upload failed" count=253832026/11/12 00:03:28 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestDrainGivesUpWhenServerDown2319674382/002/e5384--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)5385--- PASS: TestDrainTimeout (0.02s)53862026/11/12 00:03:28 ERROR Drain finished with paths left in queue remaining=15387--- PASS: TestQueueRemove (0.02s)53882026/11/12 00:03:28 ERROR Upload failed, will retry later error="upload failed" path=/private/var/tb/8/scratch/b3af67106d865ad2/build/TestDrainGivesUpWhenServerDown2319674382/002/f5389--- PASS: TestWorkerRemoveFailureIsNotProgress (0.02s)53902026/11/12 00:03:28 ERROR Drain finished with paths left in queue remaining=105391--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)5392--- PASS: TestDrainIsolatesPoisonPath (0.02s)5393--- PASS: TestDrainGivesUpWhenServerDown (0.02s)5394--- PASS: TestWorkerPrunesClosureDeps (0.03s)5395--- PASS: TestWorkerUploadsAndRemoves (0.03s)5396--- PASS: TestWorkerSkipsGCdPaths (0.03s)5397--- PASS: TestQueueRemoveLargeClosure (0.05s)5398--- PASS: TestQueueConcurrentWriters (0.06s)53992026/11/12 00:03:28 ERROR Failed to decode request error="read unix /private/var/tb/8/scratch/b3af67106d865ad2/build/hook1279259884/test.sock->: i/o timeout"54002026/11/12 00:03:28 ERROR Failed to write response error="write unix /private/var/tb/8/scratch/b3af67106d865ad2/build/hook1279259884/test.sock->: i/o timeout"5401--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)54022026/11/12 00:03:29 INFO Uploading batch count=154032026/11/12 00:03:29 INFO Uploading batch count=154042026/11/12 00:03:29 INFO Uploading batch count=154052026/11/12 00:03:29 ERROR Upload failed error="upload failed" count=154062026/11/12 00:03:29 INFO Uploading batch count=154072026/11/12 00:03:29 ERROR Upload failed error="upload failed" count=154082026/11/12 00:03:29 INFO Uploading batch count=154092026/11/12 00:03:29 ERROR Upload failed error="upload failed" count=154102026/11/12 00:03:29 INFO Uploading batch count=154112026/11/12 00:03:29 ERROR Upload failed error="upload failed" count=154122026/11/12 00:03:29 ERROR Drain finished with paths left in queue remaining=15413--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5414PASS5415Running signing tests...5416=== RUN TestExternalSigner5417=== PAUSE TestExternalSigner5418=== RUN TestExternalSignerWrongPublicKey5419=== PAUSE TestExternalSignerWrongPublicKey5420=== RUN TestExternalSignerErrors5421=== PAUSE TestExternalSignerErrors5422=== RUN TestExternalSignerRestartsAfterCrash5423=== PAUSE TestExternalSignerRestartsAfterCrash5424=== RUN TestExternalSignerProgramErrorKeepsProcess5425=== PAUSE TestExternalSignerProgramErrorKeepsProcess5426=== RUN TestExternalSignerClose5427=== PAUSE TestExternalSignerClose5428=== RUN TestExternalSignerRejectsInvalidPublicKey5429=== PAUSE TestExternalSignerRejectsInvalidPublicKey5430=== RUN TestExternalSignerCancellation5431=== PAUSE TestExternalSignerCancellation5432=== RUN TestGenerateFingerprint5433=== PAUSE TestGenerateFingerprint5434=== RUN TestParseSigningKey5435=== PAUSE TestParseSigningKey5436=== RUN TestSign5437=== PAUSE TestSign5438=== RUN TestSignBatch5439=== PAUSE TestSignBatch5440=== RUN TestNilKeyPublicKey5441=== PAUSE TestNilKeyPublicKey5442=== RUN TestSignErrors5443=== PAUSE TestSignErrors5444=== CONT TestExternalSigner5445=== CONT TestSignBatch5446=== CONT TestExternalSignerCancellation5447=== CONT TestExternalSignerProgramErrorKeepsProcess5448=== CONT TestParseSigningKey5449=== RUN TestParseSigningKey/valid_32-byte_key5450=== CONT TestSignErrors5451=== CONT TestNilKeyPublicKey5452--- PASS: TestNilKeyPublicKey (0.00s)5453=== CONT TestGenerateFingerprint5454=== RUN TestGenerateFingerprint/basic_with_references5455=== CONT TestExternalSignerErrors5456=== PAUSE TestGenerateFingerprint/basic_with_references5457=== RUN TestGenerateFingerprint/no_references5458=== PAUSE TestGenerateFingerprint/no_references5459=== RUN TestExternalSignerErrors/exit_at_startup5460=== CONT TestExternalSignerWrongPublicKey5461=== CONT TestExternalSignerRejectsInvalidPublicKey5462=== RUN TestExternalSignerRejectsInvalidPublicKey/#005463=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005464=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5465=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5466=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5467=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5468=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5469=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5470=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5471=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5472=== CONT TestExternalSignerRejectsInvalidPublicKey/#005473=== CONT TestExternalSignerClose5474=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5475=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5476=== PAUSE TestParseSigningKey/valid_32-byte_key5477=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5478=== CONT TestExternalSignerRestartsAfterCrash5479=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5480=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5481=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5482=== PAUSE TestExternalSignerErrors/exit_at_startup5483=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5484=== RUN TestExternalSignerErrors/too_few_signatures5485=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5486--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5487 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5488 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5489 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5490 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5491 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5492=== CONT TestSign5493=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5494=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5495=== PAUSE TestExternalSignerErrors/too_few_signatures5496=== RUN TestParseSigningKey/no_colon5497=== PAUSE TestParseSigningKey/no_colon5498=== RUN TestGenerateFingerprint/invalid_nar_hash_length5499=== RUN TestExternalSignerErrors/unrequested_key5500=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5501=== PAUSE TestExternalSignerErrors/unrequested_key5502=== RUN TestParseSigningKey/empty_name5503=== PAUSE TestParseSigningKey/empty_name5504=== RUN TestGenerateFingerprint/invalid_store_path_prefix5505=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5506=== RUN TestExternalSignerErrors/wrong_signature_name5507=== RUN TestGenerateFingerprint/invalid_reference_prefix5508=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5509=== RUN TestParseSigningKey/invalid_base645510=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5511=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5512=== CONT TestGenerateFingerprint/invalid_store_path_prefix5513=== CONT TestGenerateFingerprint/basic_with_references5514=== CONT TestGenerateFingerprint/invalid_reference_prefix5515=== CONT TestGenerateFingerprint/invalid_nar_hash_length5516=== CONT TestGenerateFingerprint/no_references5517--- PASS: TestGenerateFingerprint (0.00s)5518 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5519 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5520 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5521 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5522 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5523 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5524 --- PASS: TestGenerateFingerprint/no_references (0.00s)5525=== PAUSE TestExternalSignerErrors/wrong_signature_name5526=== PAUSE TestParseSigningKey/invalid_base645527=== RUN TestParseSigningKey/wrong_length5528=== PAUSE TestParseSigningKey/wrong_length5529=== CONT TestParseSigningKey/valid_32-byte_key5530=== RUN TestExternalSignerErrors/wrong_signature_length5531=== PAUSE TestExternalSignerErrors/wrong_signature_length5532=== CONT TestParseSigningKey/empty_name5533=== RUN TestExternalSignerErrors/exit_on_request5534=== PAUSE TestExternalSignerErrors/exit_on_request5535=== RUN TestExternalSignerErrors/invalid_JSON5536=== PAUSE TestExternalSignerErrors/invalid_JSON5537=== CONT TestParseSigningKey/no_colon5538=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5539=== CONT TestParseSigningKey/wrong_length5540=== CONT TestParseSigningKey/invalid_base645541=== RUN TestExternalSignerErrors/null5542=== PAUSE TestExternalSignerErrors/null5543=== RUN TestExternalSignerErrors/wrong_signatures_shape5544=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5545=== RUN TestExternalSignerErrors/wrong_shape5546=== PAUSE TestExternalSignerErrors/wrong_shape5547=== RUN TestExternalSignerErrors/wrong_id5548=== PAUSE TestExternalSignerErrors/wrong_id5549=== RUN TestExternalSignerErrors/null_signatures5550=== PAUSE TestExternalSignerErrors/null_signatures5551=== RUN TestExternalSignerErrors/invalid_signature_base645552=== PAUSE TestExternalSignerErrors/invalid_signature_base645553=== RUN TestExternalSignerErrors/program_error5554=== PAUSE TestExternalSignerErrors/program_error5555=== RUN TestExternalSignerErrors/missing_signatures5556=== PAUSE TestExternalSignerErrors/missing_signatures5557=== RUN TestExternalSignerErrors/missing_signature_name5558=== PAUSE TestExternalSignerErrors/missing_signature_name5559=== CONT TestExternalSignerErrors/exit_at_startup5560=== CONT TestExternalSignerErrors/wrong_signatures_shape5561=== CONT TestExternalSignerErrors/wrong_signature_length5562=== CONT TestExternalSignerErrors/wrong_signature_name5563--- PASS: TestSignBatch (0.00s)5564=== CONT TestExternalSignerErrors/exit_on_request5565=== CONT TestExternalSignerErrors/invalid_JSON5566=== CONT TestExternalSignerErrors/missing_signature_name5567=== CONT TestExternalSignerErrors/unrequested_key5568=== CONT TestExternalSignerErrors/program_error5569=== CONT TestExternalSignerErrors/too_few_signatures5570=== CONT TestExternalSignerErrors/wrong_id5571--- PASS: TestSign (0.00s)5572=== CONT TestExternalSignerErrors/invalid_signature_base645573--- PASS: TestParseSigningKey (0.00s)5574 --- PASS: TestParseSigningKey/empty_name (0.00s)5575 --- PASS: TestParseSigningKey/no_colon (0.00s)5576 --- PASS: TestParseSigningKey/wrong_length (0.00s)5577 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5578 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5579 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5580=== CONT TestExternalSignerErrors/null_signatures5581=== CONT TestExternalSignerErrors/wrong_shape5582=== CONT TestExternalSignerErrors/missing_signatures5583=== CONT TestExternalSignerErrors/null5584=== RUN TestSignErrors/nil_key5585=== PAUSE TestSignErrors/nil_key5586=== RUN TestSignErrors/nil_narinfo5587=== PAUSE TestSignErrors/nil_narinfo5588=== RUN TestSignErrors/invalid_narinfo5589=== PAUSE TestSignErrors/invalid_narinfo5590=== RUN TestSignErrors/canceled5591=== PAUSE TestSignErrors/canceled5592=== CONT TestSignErrors/nil_key5593=== CONT TestSignErrors/canceled5594=== CONT TestSignErrors/invalid_narinfo5595=== CONT TestSignErrors/nil_narinfo5596--- PASS: TestSignErrors (0.01s)5597 --- PASS: TestSignErrors/nil_key (0.00s)5598 --- PASS: TestSignErrors/canceled (0.00s)5599 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5600 --- PASS: TestSignErrors/nil_narinfo (0.00s)5601--- PASS: TestExternalSignerCancellation (0.24s)5602--- PASS: TestExternalSignerWrongPublicKey (0.41s)5603--- PASS: TestExternalSignerRestartsAfterCrash (0.61s)5604--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.78s)5605--- PASS: TestExternalSignerClose (0.97s)5606--- PASS: TestExternalSigner (1.18s)5607--- PASS: TestExternalSignerErrors (0.00s)5608 --- PASS: TestExternalSignerErrors/wrong_shape (1.35s)5609 --- PASS: TestExternalSignerErrors/exit_at_startup (1.54s)5610 --- PASS: TestExternalSignerErrors/program_error (1.73s)5611 --- PASS: TestExternalSignerErrors/missing_signature_name (1.92s)5612 --- PASS: TestExternalSignerErrors/wrong_signature_length (2.11s)5613 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (2.29s)5614 --- PASS: TestExternalSignerErrors/too_few_signatures (2.48s)5615 --- PASS: TestExternalSignerErrors/unrequested_key (2.67s)5616 --- PASS: TestExternalSignerErrors/exit_on_request (2.88s)5617 --- PASS: TestExternalSignerErrors/null (3.06s)5618 --- PASS: TestExternalSignerErrors/wrong_signature_name (3.25s)5619 --- PASS: TestExternalSignerErrors/null_signatures (3.45s)5620 --- PASS: TestExternalSignerErrors/invalid_JSON (3.64s)5621 --- PASS: TestExternalSignerErrors/missing_signatures (3.83s)5622 --- PASS: TestExternalSignerErrors/wrong_id (4.02s)5623 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (4.21s)5624PASS5625Running OpenBao signer tests...5626=== RUN TestSign5627=== PAUSE TestSign5628=== RUN TestSignErrors5629=== PAUSE TestSignErrors5630=== RUN TestServe5631=== PAUSE TestServe5632=== RUN TestServeRejectsGarbage5633=== PAUSE TestServeRejectsGarbage5634=== CONT TestSign5635=== CONT TestServe5636=== CONT TestServeRejectsGarbage5637=== CONT TestSignErrors5638=== RUN TestSignErrors/http_error5639=== PAUSE TestSignErrors/http_error5640--- PASS: TestServeRejectsGarbage (0.00s)5641=== RUN TestSignErrors/invalid_json5642=== PAUSE TestSignErrors/invalid_json5643=== RUN TestSignErrors/no_prefix5644=== PAUSE TestSignErrors/no_prefix5645=== RUN TestSignErrors/invalid_base645646=== PAUSE TestSignErrors/invalid_base645647=== CONT TestSignErrors/http_error5648=== CONT TestSignErrors/no_prefix5649=== CONT TestSignErrors/invalid_json5650--- PASS: TestServe (0.00s)5651=== CONT TestSignErrors/invalid_base645652--- PASS: TestSign (0.00s)5653--- PASS: TestSignErrors (0.00s)5654 --- PASS: TestSignErrors/invalid_json (0.00s)5655 --- PASS: TestSignErrors/no_prefix (0.00s)5656 --- PASS: TestSignErrors/invalid_base64 (0.00s)5657 --- PASS: TestSignErrors/http_error (0.00s)5658PASS

succeeded niks3-go-unit-tests checks.aarch64-darwin.go-unit-tests · build #494 · raw

succeeded `niks3-go-unit-tests` `checks.aarch64-darwin.go-unit-tests` · build #494 · raw