niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #495
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestUploadBuildLog_FileBodyReplayedOnRetry6=== PAUSE TestUploadBuildLog_FileBodyReplayedOnRetry7=== RUN TestRegisterUploadedObjectReusesConnections8=== PAUSE TestRegisterUploadedObjectReusesConnections9=== RUN TestRunGarbageCollection_FinishedOnAnotherReplica10=== PAUSE TestRunGarbageCollection_FinishedOnAnotherReplica11=== RUN TestRunGarbageCollection_NotFoundAfterLocalRun12=== PAUSE TestRunGarbageCollection_NotFoundAfterLocalRun13=== RUN TestCaseHackSuffix14=== PAUSE TestCaseHackSuffix15=== RUN TestFilterOversizedClosures16=== PAUSE TestFilterOversizedClosures17=== RUN TestUploadMultipart_PartsInParallel18=== PAUSE TestUploadMultipart_PartsInParallel19=== RUN TestUploadMultipart_ReturnsAfterDroppedBody20=== PAUSE TestUploadMultipart_ReturnsAfterDroppedBody21=== RUN TestPartSizeForNAR22=== PAUSE TestPartSizeForNAR23=== RUN TestDumpPathCaseHackMatchesNix24--- PASS: TestDumpPathCaseHackMatchesNix (0.03s)25=== RUN TestDumpPathCaseHackCollision26--- PASS: TestDumpPathCaseHackCollision (0.00s)27=== RUN TestDumpPathMatchesNix28=== PAUSE TestDumpPathMatchesNix29=== RUN TestDumpPathSingleFile30=== PAUSE TestDumpPathSingleFile31=== RUN TestDumpPathWriterError32=== PAUSE TestDumpPathWriterError33=== RUN TestEncodeNixBase3234=== PAUSE TestEncodeNixBase3235=== RUN TestEncodeNixBase32WithRealHash36=== PAUSE TestEncodeNixBase32WithRealHash37=== RUN TestConvertHashToNix3238=== PAUSE TestConvertHashToNix3239=== RUN TestGetStorePathHash40=== PAUSE TestGetStorePathHash41=== RUN TestPathInfoHashCompatibility42=== PAUSE TestPathInfoHashCompatibility43=== RUN TestParsePathInfoJSON44=== PAUSE TestParsePathInfoJSON45=== RUN TestParsePathInfoJSONMultiplePaths46=== PAUSE TestParsePathInfoJSONMultiplePaths47=== RUN TestPathInfoCACompatibility48=== PAUSE TestPathInfoCACompatibility49=== RUN TestRateLimiterFeedback50=== PAUSE TestRateLimiterFeedback51=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess52=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess53=== RUN TestRegisterUploadedObject_BoundedAgainstSilentServer54=== PAUSE TestRegisterUploadedObject_BoundedAgainstSilentServer55=== RUN TestRegisterUploadedObject_OutlivesPush56=== PAUSE TestRegisterUploadedObject_OutlivesPush57=== RUN TestResolveStorePath58=== PAUSE TestResolveStorePath59=== RUN TestDoWithRetry_BodyReplayedViaGetBody60=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody61=== RUN TestRetryTimeoutOutlastsMaxRetries62=== PAUSE TestRetryTimeoutOutlastsMaxRetries63=== RUN TestDoWithRetry_FinalResponseBodyReadable64=== PAUSE TestDoWithRetry_FinalResponseBodyReadable65=== RUN TestShellSplit66=== PAUSE TestShellSplit67=== RUN TestShellSplitErrors68=== PAUSE TestShellSplitErrors69=== RUN TestStreamPushReportsEveryPath70=== PAUSE TestStreamPushReportsEveryPath71=== RUN TestStreamPushBatchesUnderLoad72=== PAUSE TestStreamPushBatchesUnderLoad73=== RUN TestStreamPushIsolatesFailures74=== PAUSE TestStreamPushIsolatesFailures75=== RUN TestStreamPushGivesUpOnDeadServer76=== PAUSE TestStreamPushGivesUpOnDeadServer77=== RUN TestStreamPushRequestLine78=== PAUSE TestStreamPushRequestLine79=== RUN TestStreamPushReportsSignatures80=== PAUSE TestStreamPushReportsSignatures81=== RUN TestStreamPushReportsSkippedPaths82=== PAUSE TestStreamPushReportsSkippedPaths83=== RUN TestClientSignaturesByStorePath84=== PAUSE TestClientSignaturesByStorePath85=== RUN TestStreamPushStopsWhenCanceled86=== PAUSE TestStreamPushStopsWhenCanceled87=== RUN TestSetClientTLS88=== PAUSE TestSetClientTLS89=== RUN TestSetClientTLSDoesNotMutateDefaultTransport90=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport91=== RUN TestSetClientTLSErrors92=== PAUSE TestSetClientTLSErrors93=== RUN TestStaticToken94=== PAUSE TestStaticToken95=== RUN TestFileTokenReadsAndCaches96=== PAUSE TestFileTokenReadsAndCaches97=== RUN TestFileTokenMissing98=== PAUSE TestFileTokenMissing99=== RUN TestFileTokenEmpty100=== PAUSE TestFileTokenEmpty101=== RUN TestScriptTokenNoExpiryRerunsEveryCall102=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall103=== RUN TestScriptTokenCachesUntilRefresh104=== PAUSE TestScriptTokenCachesUntilRefresh105=== RUN TestScriptTokenEmptyToken106=== PAUSE TestScriptTokenEmptyToken107=== RUN TestScriptTokenBadJSON108=== PAUSE TestScriptTokenBadJSON109=== RUN TestScriptTokenScriptFails110=== PAUSE TestScriptTokenScriptFails111=== RUN TestScriptTokenEmptyCommand112=== PAUSE TestScriptTokenEmptyCommand113=== CONT TestDoServerRequestAttachesToken114=== CONT TestRegisterUploadedObject_OutlivesPush115=== CONT TestScriptTokenCachesUntilRefresh116=== CONT TestScriptTokenBadJSON117=== CONT TestScriptTokenScriptFails118=== CONT TestStreamPushReportsSignatures119=== CONT TestScriptTokenEmptyToken120=== CONT TestScriptTokenNoExpiryRerunsEveryCall121=== CONT TestFileTokenEmpty122=== CONT TestDumpPathWriterError123=== CONT TestFileTokenMissing124=== CONT TestRegisterUploadedObject_BoundedAgainstSilentServer125=== CONT TestFileTokenReadsAndCaches126=== CONT TestPathInfoHashCompatibility127=== CONT TestGetStorePathHash128=== CONT TestStaticToken129=== CONT TestConvertHashToNix32130=== CONT TestSetClientTLSErrors131=== CONT TestEncodeNixBase32WithRealHash132=== CONT TestSetClientTLSDoesNotMutateDefaultTransport133=== CONT TestScriptTokenEmptyCommand134=== CONT TestSetClientTLS135=== CONT TestEncodeNixBase32136=== CONT TestStreamPushReportsSkippedPaths137--- PASS: TestStaticToken (0.00s)138=== CONT TestClientSignaturesByStorePath139--- PASS: TestFileTokenEmpty (0.00s)140--- PASS: TestClientSignaturesByStorePath (0.00s)141=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess142--- PASS: TestFileTokenMissing (0.00s)1432026/11/12 00:09:25 WARN Rate limiter enabled after throttle name=server-test rate=5144--- PASS: TestScriptTokenBadJSON (0.01s)145=== RUN TestConvertHashToNix32/SRI_format_to_Nix32146--- PASS: TestScriptTokenScriptFails (0.00s)147=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32148=== RUN TestConvertHashToNix32/already_Nix32_format149=== PAUSE TestConvertHashToNix32/already_Nix32_format150--- PASS: TestEncodeNixBase32WithRealHash (0.00s)151=== CONT TestStreamPushReportsEveryPath152--- PASS: TestScriptTokenEmptyCommand (0.00s)153=== CONT TestStreamPushRequestLine154=== RUN TestConvertHashToNix32/invalid_format155=== PAUSE TestConvertHashToNix32/invalid_format156=== RUN TestEncodeNixBase32/test_string_hash157=== CONT TestStreamPushBatchesUnderLoad158=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=false159=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=false160=== RUN TestStreamPushBatchesUnderLoad/oneAtATime=true161=== PAUSE TestStreamPushBatchesUnderLoad/oneAtATime=true162=== PAUSE TestEncodeNixBase32/test_string_hash163=== RUN TestEncodeNixBase32/empty_input164=== PAUSE TestEncodeNixBase32/empty_input165=== CONT TestRetryTimeoutOutlastsMaxRetries166=== CONT TestStreamPushStopsWhenCanceled167=== CONT TestStreamPushGivesUpOnDeadServer168--- PASS: TestStreamPushReportsSkippedPaths (0.00s)169=== CONT TestFilterOversizedClosures170=== RUN TestGetStorePathHash/valid_store_path171=== CONT TestPathInfoCACompatibility172=== CONT TestRateLimiterFeedback173=== CONT TestDoWithRetry_BodyReplayedViaGetBody174--- PASS: TestStreamPushReportsEveryPath (0.00s)175--- PASS: TestStreamPushStopsWhenCanceled (0.00s)1762026/11/12 00:09:25 ERROR Upload failed error=boom count=1177=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)1782026/11/12 00:09:25 ERROR Upload failed error="connection refused" count=20179=== PAUSE TestGetStorePathHash/valid_store_path1802026/11/12 00:09:25 ERROR Server seems unavailable, giving up on batch untried=17181--- PASS: TestStreamPushReportsSignatures (0.01s)182=== CONT TestDumpPathSingleFile183=== RUN TestRateLimiterFeedback/429_enables_limiter184=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)185=== PAUSE TestRateLimiterFeedback/429_enables_limiter186=== RUN TestPathInfoCACompatibility/null_ca_field187=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon188=== RUN TestGetStorePathHash/basename_without_hyphen_should_error189=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon190=== RUN TestRateLimiterFeedback/503_enables_limiter191=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error192=== PAUSE TestRateLimiterFeedback/503_enables_limiter193=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter194=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error195=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error196=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter197--- PASS: TestFileTokenReadsAndCaches (0.01s)198=== CONT TestDumpPathMatchesNix199=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error200=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error201=== CONT TestShellSplitErrors202=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter203=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter2042026/11/12 00:09:25 WARN Rate limiter enabled after throttle name=server-test rate=5205--- PASS: TestShellSplitErrors (0.00s)206=== RUN TestFilterOversizedClosures/no_limit_keeps_everything207=== CONT TestUploadMultipart_ReturnsAfterDroppedBody208=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI209--- PASS: TestScriptTokenEmptyToken (0.01s)210=== CONT TestUploadMultipart_PartsInParallel2112026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:46331212=== CONT TestParsePathInfoJSONMultiplePaths2132026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:35717214=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths215--- PASS: TestDoServerRequestAttachesToken (0.01s)2162026/11/12 00:09:25 ERROR Upload failed error=boom count=1217=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything218=== CONT TestStreamPushIsolatesFailures219=== CONT TestPartSizeForNAR220=== PAUSE TestPathInfoCACompatibility/null_ca_field221=== CONT TestResolveStorePath2222026/11/12 00:09:25 ERROR Upload failed error="bad path" count=3223=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI224=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths225=== RUN TestPathInfoCACompatibility/old_string_format_-_text226=== RUN TestSetClientTLSErrors/missing_cert_file227--- PASS: TestStreamPushGivesUpOnDeadServer (0.01s)228=== CONT TestShellSplit229=== CONT TestCaseHackSuffix230=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped231=== RUN TestPartSizeForNAR/zero_stays_at_minimum232=== CONT TestDoWithRetry_FinalResponseBodyReadable233--- PASS: TestRegisterUploadedObject_OutlivesPush (0.01s)234=== RUN TestSetClientTLS/rejects_connection_without_client_cert235=== CONT TestRegisterUploadedObjectReusesConnections236=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths237=== CONT TestUploadBuildLog_FileBodyReplayedOnRetry238=== CONT TestRunGarbageCollection_FinishedOnAnotherReplica239--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)240--- PASS: TestStreamPushIsolatesFailures (0.00s)241--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)242--- PASS: TestResolveStorePath (0.00s)243=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths244=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped245=== PAUSE TestSetClientTLSErrors/missing_cert_file246=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum247=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert248--- PASS: TestShellSplit (0.00s)249=== CONT TestParsePathInfoJSON250=== CONT TestRunGarbageCollection_NotFoundAfterLocalRun251=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512252=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512253=== CONT TestConvertHashToNix32/invalid_format254=== CONT TestConvertHashToNix32/already_Nix32_format255=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=false2562026/11/12 00:09:25 WARN Rate limiter enabled after throttle name=server-test rate=52572026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:43289258=== CONT TestStreamPushBatchesUnderLoad/oneAtATime=true2592026/11/12 00:09:25 WARN Rate limiter backed off name=server-test rate=52602026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=2 backoff=0s status=503 url=http://127.0.0.1:35717261--- PASS: TestUploadMultipart_ReturnsAfterDroppedBody (0.02s)262--- PASS: TestScriptTokenCachesUntilRefresh (0.04s)263=== RUN TestFilterOversizedClosures/all_closures_skipped264=== PAUSE TestFilterOversizedClosures/all_closures_skipped265=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text2662026/11/12 00:09:25 INFO Garbage collection started267=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA268=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA269=== RUN TestSetClientTLS/preserves_debug_logging_transport270=== PAUSE TestSetClientTLS/preserves_debug_logging_transport271=== CONT TestEncodeNixBase32/test_string_hash2722026/11/12 00:09:25 INFO Garbage collection started273=== RUN TestParsePathInfoJSON/Nix_format274--- PASS: TestStreamPushBatchesUnderLoad (0.00s)275 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=true (0.00s)276 --- PASS: TestStreamPushBatchesUnderLoad/oneAtATime=false (0.00s)277=== CONT TestConvertHashToNix32/SRI_format_to_Nix32278--- PASS: TestConvertHashToNix32 (0.00s)279 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)280 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)281 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)282=== CONT TestGetStorePathHash/basename_without_hyphen_should_error283=== CONT TestRateLimiterFeedback/429_enables_limiter284--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.03s)2852026/11/12 00:09:25 WARN Rate limiter backed off name=server-test rate=5286=== PAUSE TestParsePathInfoJSON/Nix_format287=== RUN TestParsePathInfoJSON/Lix_format288=== PAUSE TestParsePathInfoJSON/Lix_format289=== RUN TestParsePathInfoJSON/empty_input290=== PAUSE TestParsePathInfoJSON/empty_input291=== RUN TestParsePathInfoJSON/whitespace_only292=== PAUSE TestParsePathInfoJSON/whitespace_only293=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter2942026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:46331295=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive296=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive297=== RUN TestParsePathInfoJSON/invalid_JSON2982026/11/12 00:09:25 WARN Rate limiter enabled after throttle name=server-test rate=5299=== PAUSE TestParsePathInfoJSON/invalid_JSON3002026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=429 url=http://127.0.0.1:46585301=== RUN TestPathInfoCACompatibility/new_structured_format_-_text302=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text303=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method304=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method3052026/11/12 00:09:25 WARN Rate limiter backed off name=server-test rate=5306=== RUN TestPartSizeForNAR/small_stays_at_minimum307=== PAUSE TestPartSizeForNAR/small_stays_at_minimum308=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum309=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum310=== CONT TestEncodeNixBase32/empty_input311=== CONT TestGetStorePathHash/valid_store_path312=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts313=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths314=== CONT TestFilterOversizedClosures/no_limit_keeps_everything315=== CONT TestSetClientTLS/rejects_connection_without_client_cert316=== CONT TestFilterOversizedClosures/all_closures_skipped3172026/11/12 00:09:25 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper nar_size=100 max_nar_size=50318=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped3192026/11/12 00:09:25 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000320=== CONT TestSetClientTLS/preserves_debug_logging_transport3212026/11/12 00:09:25 INFO Garbage collection progress phase=running_on_another_replica failed_uploads_deleted=0 old_closures_deleted=0 objects_marked=0 objects_deleted=0 objects_failed=03222026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=1 backoff=1ms status=502 url=http://127.0.0.1:46331323=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths3242026/11/12 00:09:25 WARN Rate limiter enabled after throttle name=s3-test rate=5325=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI3262026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:34053327=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts328=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error329--- PASS: TestDoWithRetry_FinalResponseBodyReadable (0.01s)330=== CONT TestRateLimiterFeedback/503_enables_limiter331=== RUN TestSetClientTLSErrors/missing_key_file332=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter333=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)334=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error335=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA336=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512337=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon338=== CONT TestParsePathInfoJSON/empty_input339=== CONT TestParsePathInfoJSON/Lix_format340=== RUN TestPartSizeForNAR/1_TiB3412026/11/12 00:09:25 WARN Rate limiter enabled after throttle name=server-test rate=5342=== PAUSE TestPartSizeForNAR/1_TiB3432026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=1 backoff=0s status=503 url=http://127.0.0.1:35309344=== RUN TestPartSizeForNAR/5_TiB_S3_max_object3452026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=2 backoff=1ms status=502 url=http://127.0.0.1:46331346=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object3472026/11/12 00:09:25 WARN Garbage collection finished on another replica. Its statistics are not available here348=== RUN TestPartSizeForNAR/capped_at_5_GiB3492026/11/12 00:09:25 WARN Rate limiter backed off name=server-test rate=5350=== PAUSE TestPartSizeForNAR/capped_at_5_GiB351=== CONT TestParsePathInfoJSON/Nix_format352=== CONT TestPathInfoCACompatibility/old_string_format_-_text353=== CONT TestPartSizeForNAR/capped_at_5_GiB354--- PASS: TestEncodeNixBase32 (0.00s)355 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)356 --- PASS: TestEncodeNixBase32/empty_input (0.00s)357=== PAUSE TestSetClientTLSErrors/missing_key_file358=== CONT TestParsePathInfoJSON/invalid_JSON359=== CONT TestParsePathInfoJSON/whitespace_only360=== CONT TestPathInfoCACompatibility/null_ca_field361=== CONT TestPathInfoCACompatibility/new_structured_format_-_text362=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum363=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive364=== CONT TestPartSizeForNAR/zero_stays_at_minimum365=== CONT TestPartSizeForNAR/5_TiB_S3_max_object3662026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=3 backoff=1ms status=502 url=http://127.0.0.1:46331367=== CONT TestPartSizeForNAR/1_TiB368=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts369--- PASS: TestRunGarbageCollection_NotFoundAfterLocalRun (0.00s)370=== CONT TestPartSizeForNAR/small_stays_at_minimum371=== RUN TestSetClientTLSErrors/missing_ca_file372=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method373--- PASS: TestUploadBuildLog_FileBodyReplayedOnRetry (0.00s)374--- PASS: TestRunGarbageCollection_FinishedOnAnotherReplica (0.01s)375--- PASS: TestFilterOversizedClosures (0.03s)376 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)377 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)378 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)379=== PAUSE TestSetClientTLSErrors/missing_ca_file380--- PASS: TestRateLimiterFeedback (0.00s)381 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.00s)382 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)383 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.00s)384 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.00s)385=== RUN TestSetClientTLSErrors/invalid_ca_file386--- PASS: TestGetStorePathHash (0.01s)387 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)388 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)389 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)390 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)391--- PASS: TestParsePathInfoJSONMultiplePaths (0.02s)392 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)393 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)394--- PASS: TestPathInfoHashCompatibility (0.03s)395 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)396 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)397 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)398 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)399=== PAUSE TestSetClientTLSErrors/invalid_ca_file4002026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=4 backoff=1ms status=502 url=http://127.0.0.1:46331401=== CONT TestSetClientTLSErrors/missing_cert_file402=== CONT TestSetClientTLSErrors/invalid_ca_file403--- PASS: TestPartSizeForNAR (0.03s)404 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)405 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)406 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)407 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)408 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)409 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)410 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)411--- PASS: TestParsePathInfoJSON (0.00s)412 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)413 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)414 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)415 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)416 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)417--- PASS: TestPathInfoCACompatibility (0.03s)418 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)419 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)420 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)421 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)422 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)423=== CONT TestSetClientTLSErrors/missing_ca_file424=== CONT TestSetClientTLSErrors/missing_key_file4252026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=5 backoff=1ms status=502 url=http://127.0.0.1:46331426--- PASS: TestSetClientTLSErrors (0.03s)427 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)428 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)429 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)430 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)431--- PASS: TestDumpPathSingleFile (0.03s)4322026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=6 backoff=1ms status=502 url=http://127.0.0.1:463314332026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=7 backoff=1ms status=502 url=http://127.0.0.1:463314342026/11/12 00:09:25 http: TLS handshake error from 127.0.0.1:37826: remote error: tls: bad certificate4352026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=8 backoff=1ms status=502 url=http://127.0.0.1:46331436--- PASS: TestSetClientTLS (0.03s)437 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)438 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)439 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)4402026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=9 backoff=1ms status=502 url=http://127.0.0.1:46331441--- PASS: TestStreamPushRequestLine (0.04s)4422026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=10 backoff=1ms status=502 url=http://127.0.0.1:46331443--- PASS: TestCaseHackSuffix (0.04s)4442026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=11 backoff=1ms status=502 url=http://127.0.0.1:463314452026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=12 backoff=1ms status=502 url=http://127.0.0.1:463314462026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=13 backoff=1ms status=502 url=http://127.0.0.1:463314472026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=14 backoff=1ms status=502 url=http://127.0.0.1:463314482026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=15 backoff=1ms status=502 url=http://127.0.0.1:463314492026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=16 backoff=1ms status=502 url=http://127.0.0.1:46331450--- PASS: TestRegisterUploadedObjectReusesConnections (0.05s)4512026/11/12 00:09:25 WARN Request returned retryable status, retrying attempt=17 backoff=1ms status=502 url=http://127.0.0.1:46331452--- PASS: TestRetryTimeoutOutlastsMaxRetries (0.07s)453--- PASS: TestDumpPathWriterError (0.08s)4542026/11/12 00:09:25 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:34107/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:34107/api/uploads/complete4552026/11/12 00:09:25 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4562026/11/12 00:09:25 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:34107/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:34107/api/uploads/complete4572026/11/12 00:09:25 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"4582026/11/12 00:09:25 WARN Request failed, retrying attempt=1 backoff=100ms error="Post \"http://127.0.0.1:34107/api/uploads/complete\": context deadline exceeded" url=http://127.0.0.1:34107/api/uploads/complete4592026/11/12 00:09:25 WARN Failed to register uploaded object key=abc.narinfo error="sending request: context canceled during retry: context deadline exceeded"460--- PASS: TestRegisterUploadedObject_BoundedAgainstSilentServer (0.10s)461--- PASS: TestDumpPathMatchesNix (0.13s)462--- PASS: TestUploadMultipart_PartsInParallel (0.68s)463--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)464PASS465Running server tests...466The files belonging to this database system will be owned by user "nixbld".467This user must also own the server process.468469The database cluster will be initialized with locale "C".470The default database encoding has accordingly been set to "SQL_ASCII".471The default text search configuration will be set to "english".472473Data page checksums are enabled.474475creating directory /build/postgres3321657469/data ... ok476creating subdirectories ... ok477selecting dynamic shared memory implementation ... posix478selecting default "max_connections" ... 100479selecting default "shared_buffers" ... 128MB480selecting default time zone ... UTC481creating configuration files ... ok482running bootstrap script ... ok483performing post-bootstrap initialization ... ok484syncing data to disk ... ok485486initdb: warning: enabling "trust" authentication for local connections487initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.488489Success. You can now start the database server using:490491 pg_ctl -D /build/postgres3321657469/data -l logfile start492493/build/postgres3321657469:5432 - no response4942026-11-12 00:09:27.292 UTC [129] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit4952026-11-12 00:09:27.293 UTC [129] LOG: listening on Unix socket "/build/postgres3321657469/.s.PGSQL.5432"4962026-11-12 00:09:27.301 UTC [136] LOG: database system was shut down at 2026-11-12 00:09:26 UTC4972026-11-12 00:09:27.304 UTC [129] LOG: database system is ready to accept connections498/build/postgres3321657469:5432 - accepting connections499{"timestamp":"2026-11-12T00:09:27.597822419Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"fb37c688-9f5c-49aa-8185-158b569c46bd","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"GET","uri":"/health/ready","status_code":503,"suppressed_errors":0,"duration_ms":0,"result":"server_error","target":"rustfs::server::http","filename":"rustfs/src/server/layer.rs","line_number":502,"threadName":"rustfs-worker","threadId":"ThreadId(393)"}500=== RUN TestAbortSucceeded501=== PAUSE TestAbortSucceeded502=== RUN TestService_AuthMiddleware503=== PAUSE TestService_AuthMiddleware504=== RUN TestService_AuthMiddleware_MTLSProxyHeader505=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader506=== RUN TestService_AuthMiddleware_MTLSBoundSubjects507=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects508=== RUN TestService_ReadAuthMiddleware509=== PAUSE TestService_ReadAuthMiddleware510=== RUN TestService_AuthMiddleware_OIDC511=== PAUSE TestService_AuthMiddleware_OIDC512=== RUN TestService_RequireScope_OIDC513=== PAUSE TestService_RequireScope_OIDC514=== RUN TestService_ReadScope_PublicByDefault515=== PAUSE TestService_ReadScope_PublicByDefault516=== RUN TestCacheConfigHandler517=== PAUSE TestCacheConfigHandler518=== RUN TestCacheStatsHandler519=== PAUSE TestCacheStatsHandler520=== RUN TestClientCADerivations521=== PAUSE TestClientCADerivations522=== RUN TestClientErrorHandling523=== PAUSE TestClientErrorHandling524=== RUN TestClientIntegration525=== PAUSE TestClientIntegration526=== RUN TestClientMultipleUploads527=== PAUSE TestClientMultipleUploads528=== RUN TestClientWithDependencies529=== PAUSE TestClientWithDependencies530=== RUN TestClientSharedPathCommittedMidPush531=== PAUSE TestClientSharedPathCommittedMidPush532=== RUN TestPinProtectsFromGC533=== PAUSE TestPinProtectsFromGC534=== RUN TestClientPushesUseOnePush535=== PAUSE TestClientPushesUseOnePush536=== RUN TestClientFallsBackToClosures537=== PAUSE TestClientFallsBackToClosures538=== RUN TestDeleteClosuresRacingPins539=== PAUSE TestDeleteClosuresRacingPins540=== RUN TestConcurrentCommitsSharingObjectsDoNotDeadlock541=== PAUSE TestConcurrentCommitsSharingObjectsDoNotDeadlock542=== RUN TestResolveDBConnectionString543=== PAUSE TestResolveDBConnectionString544=== RUN TestConnectWaitsForAPeerMigration545=== PAUSE TestConnectWaitsForAPeerMigration546=== RUN TestLeadElectsOneAndHandsOver547=== PAUSE TestLeadElectsOneAndHandsOver548=== RUN TestLeadIncumbentWinsAfterRestart5492026/11/12 00:09:27 INFO lead: acquired remote=192.0.2.1:1234 generation=15502026/11/12 00:09:28 INFO lead: released remote=192.0.2.1:1234 generation=15512026/11/12 00:09:28 INFO lead: acquired remote=192.0.2.1:1234 generation=25522026/11/12 00:09:29 INFO lead: released remote=192.0.2.1:1234 generation=2553--- PASS: TestLeadIncumbentWinsAfterRestart (1.89s)554=== RUN TestLeadUnannouncedStandbyYieldsToIncumbent555=== PAUSE TestLeadUnannouncedStandbyYieldsToIncumbent556=== RUN TestLeadIncumbentAnnouncesAtOnce557=== PAUSE TestLeadIncumbentAnnouncesAtOnce558=== RUN TestLeadGenerationGrowsWithEveryGrant559=== PAUSE TestLeadGenerationGrowsWithEveryGrant560=== RUN TestLeadEndsOnShutdown561=== PAUSE TestLeadEndsOnShutdown562=== RUN TestLeadEndsWhenItsConnectionHangs5632026/11/12 00:09:29 INFO lead: acquired remote=192.0.2.1:1234 generation=15642026-11-12 00:09:31.116 UTC [575] FATAL: terminating connection due to administrator command5652026/11/12 00:09:31 INFO lead: acquired remote=192.0.2.1:1234 generation=25662026/11/12 00:09:32 WARN lead: lock connection lost error="timeout: context deadline exceeded"5672026/11/12 00:09:32 INFO lead: released remote=192.0.2.1:1234 generation=15682026/11/12 00:09:32 INFO lead: released remote=192.0.2.1:1234 generation=2569--- PASS: TestLeadEndsWhenItsConnectionHangs (3.12s)570=== RUN TestGCAdvisoryLockBlocksConcurrentRun571=== PAUSE TestGCAdvisoryLockBlocksConcurrentRun572=== RUN TestGCBugBareHashReferences573=== PAUSE TestGCBugBareHashReferences574=== RUN TestGCMetrics575=== PAUSE TestGCMetrics576=== RUN TestPushDedupSurvivesConcurrentGC577=== PAUSE TestPushDedupSurvivesConcurrentGC578=== RUN TestDeduplicatedObjectsRecordedAsPending579=== PAUSE TestDeduplicatedObjectsRecordedAsPending580=== RUN TestGCSweepSkipsPendingObjects581=== PAUSE TestGCSweepSkipsPendingObjects582=== RUN TestTombstonedObjectOfferedWithoutWaiting583=== PAUSE TestTombstonedObjectOfferedWithoutWaiting584=== RUN TestGCSweepDeliversEachKeyOnce585=== PAUSE TestGCSweepDeliversEachKeyOnce586=== RUN TestCreatePendingClosureVerifyS3FailureReleasesConnection587=== PAUSE TestCreatePendingClosureVerifyS3FailureReleasesConnection588=== RUN TestForceGCDuringPushOffersSweptObject589=== PAUSE TestForceGCDuringPushOffersSweptObject590=== RUN TestFailedS3DeleteKeepsTombstone591=== PAUSE TestFailedS3DeleteKeepsTombstone592=== RUN TestSweepSkipsRowsLockedByPush593=== PAUSE TestSweepSkipsRowsLockedByPush594=== RUN TestRegisterCompletedObjectNeedsPendingRow595=== PAUSE TestRegisterCompletedObjectNeedsPendingRow596=== RUN TestGetExistingObjectsLocksRows597=== PAUSE TestGetExistingObjectsLocksRows598=== RUN TestSweepRechecksPendingAfterLocking599=== PAUSE TestSweepRechecksPendingAfterLocking600=== RUN TestPushWaitsForSweepHoldingRow601=== PAUSE TestPushWaitsForSweepHoldingRow602=== RUN TestCommitRacingPendingCleanupKeepsObjects603=== PAUSE TestCommitRacingPendingCleanupKeepsObjects604=== RUN TestGCEndsOnShutdown605=== PAUSE TestGCEndsOnShutdown606=== RUN TestGCTaskStore_StartNew607=== PAUSE TestGCTaskStore_StartNew608=== RUN TestGCTaskStore_DeduplicateSameParams609=== PAUSE TestGCTaskStore_DeduplicateSameParams610=== RUN TestGCTaskStore_ConflictDifferentParams611=== PAUSE TestGCTaskStore_ConflictDifferentParams612=== RUN TestGCTaskStore_GetEmpty613=== PAUSE TestGCTaskStore_GetEmpty614=== RUN TestGCTaskStore_GetReturnsLatest615=== PAUSE TestGCTaskStore_GetReturnsLatest616=== RUN TestGCTaskStore_CompletedAllowsNewTask617=== PAUSE TestGCTaskStore_CompletedAllowsNewTask618=== RUN TestGCTaskStore_PhaseUpdates619=== PAUSE TestGCTaskStore_PhaseUpdates620=== RUN TestGCTaskStore_Fail621=== PAUSE TestGCTaskStore_Fail622=== RUN TestGracefulShutdownDrainsInflight623=== PAUSE TestGracefulShutdownDrainsInflight624=== RUN TestService_healthCheckHandler625=== PAUSE TestService_healthCheckHandler626=== RUN TestService_readinessHandler627=== PAUSE TestService_readinessHandler628=== RUN TestGenerateLandingPage629=== PAUSE TestGenerateLandingPage630=== RUN TestEnsureAbortUploadsRule631=== PAUSE TestEnsureAbortUploadsRule632=== RUN TestCacheConfigHandlerMaxNarSize633=== PAUSE TestCacheConfigHandlerMaxNarSize634=== RUN TestCreatePendingClosureRejectsOversizedNAR635=== PAUSE TestCreatePendingClosureRejectsOversizedNAR636=== RUN TestNARDeduplicationMetadataUploadBug637=== PAUSE TestNARDeduplicationMetadataUploadBug638=== RUN TestMetricsInventory639=== PAUSE TestMetricsInventory640=== RUN TestMetricsCommit641=== PAUSE TestMetricsCommit642=== RUN TestService_NativeMTLS643=== PAUSE TestService_NativeMTLS644=== RUN TestServerTLSConfig645=== PAUSE TestServerTLSConfig646=== RUN TestMultipartCleanup647=== PAUSE TestMultipartCleanup648=== RUN TestMultipartUploadAbortedWhenCancelledBeforeRecorded649=== PAUSE TestMultipartUploadAbortedWhenCancelledBeforeRecorded650=== RUN TestPendingCleanupUsesOneCutoff651=== PAUSE TestPendingCleanupUsesOneCutoff652=== RUN TestObjectStatsTrigger653=== PAUSE TestObjectStatsTrigger654=== RUN TestObjectStatsUpdatedOncePerCommit655=== PAUSE TestObjectStatsUpdatedOncePerCommit656=== RUN TestOrphanedObjectsGC657=== PAUSE TestOrphanedObjectsGC658=== RUN TestOrphanedObjectsGCStressTest659=== PAUSE TestOrphanedObjectsGCStressTest660=== RUN TestCreatePin_ReservedPins661=== PAUSE TestCreatePin_ReservedPins662=== RUN TestPresentReportsOnlyClosureRoots663=== PAUSE TestPresentReportsOnlyClosureRoots664=== RUN TestPresentRefreshesOnlyStaleClosures665=== PAUSE TestPresentRefreshesOnlyStaleClosures666=== RUN TestPresentNotReportedWhileGCDeletesClosure667=== PAUSE TestPresentNotReportedWhileGCDeletesClosure668=== RUN TestParseSingleRange669=== PAUSE TestParseSingleRange670=== RUN TestProxyHeadersOnlyTrustedOnSocket6712026/11/12 00:09:32 INFO Starting HTTP server address=127.0.0.1:374456722026/11/12 00:09:32 INFO Starting HTTP server address=proxy.sock6732026/11/12 00:09:32 WARN mTLS auth: subject not in bound subjects subject="CN=someone"6742026/11/12 00:09:32 INFO Shutdown signal received, draining in-flight requests timeout=10s675--- PASS: TestProxyHeadersOnlyTrustedOnSocket (0.11s)676=== RUN TestIsValidCachePath677=== PAUSE TestIsValidCachePath678=== RUN TestReadProxyNarinfo679=== PAUSE TestReadProxyNarinfo680=== RUN TestReadProxyNarinfoAlreadyDecompressed681=== PAUSE TestReadProxyNarinfoAlreadyDecompressed682=== RUN TestReadProxyContentNegotiation683=== PAUSE TestReadProxyContentNegotiation684=== RUN TestReadProxyNarStreaming685=== PAUSE TestReadProxyNarStreaming686=== RUN TestReadProxy404687=== PAUSE TestReadProxy404688=== RUN TestReadProxyInvalidPath689=== PAUSE TestReadProxyInvalidPath690=== RUN TestReadProxyHead691=== PAUSE TestReadProxyHead692=== RUN TestReadProxyOutlastsServerWriteTimeout693=== PAUSE TestReadProxyOutlastsServerWriteTimeout694=== RUN TestReadProxyConditionalGet695=== PAUSE TestReadProxyConditionalGet696=== RUN TestReadProxyRootRedirectsToIndexHTML697=== PAUSE TestReadProxyRootRedirectsToIndexHTML698=== RUN TestReadProxyDisabled699=== PAUSE TestReadProxyDisabled700=== RUN TestReadRedirectNar701=== PAUSE TestReadRedirectNar702=== RUN TestReadRedirectKeepsNarinfoProxied703=== PAUSE TestReadRedirectKeepsNarinfoProxied704=== RUN TestReadProxyRangeRequest705=== PAUSE TestReadProxyRangeRequest706=== RUN TestReadRedirectUsesPublicS3URL707=== PAUSE TestReadRedirectUsesPublicS3URL708=== RUN TestPush_OverlappingRootsStoreOneRowPerKey709=== PAUSE TestPush_OverlappingRootsStoreOneRowPerKey710=== RUN TestPush_CompleteCommitsEveryRoot711=== PAUSE TestPush_CompleteCommitsEveryRoot712=== RUN TestPush_RepeatedRootCommitsOnce713=== PAUSE TestPush_RepeatedRootCommitsOnce714=== RUN TestPush_CompleteWaitsForCleanupOfThePush715=== PAUSE TestPush_CompleteWaitsForCleanupOfThePush716=== RUN TestPush_RootsAreLockedInByteOrder717=== PAUSE TestPush_RootsAreLockedInByteOrder718=== RUN TestPush_SkippedKeySurvivesGCBeforeCommit719=== PAUSE TestPush_SkippedKeySurvivesGCBeforeCommit720=== RUN TestPush_RejectsBadRequests721=== PAUSE TestPush_RejectsBadRequests722=== RUN TestPush_SignsNarinfosOfItsPendingObjects723=== PAUSE TestPush_SignsNarinfosOfItsPendingObjects724=== RUN TestSharedNarBothUploadsComplete725=== PAUSE TestSharedNarBothUploadsComplete726=== RUN TestCompleteMultipartUpload_ErrorButObjectExists727=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists728=== RUN TestCompletedNarNotReofferedAcrossClosures729=== PAUSE TestCompletedNarNotReofferedAcrossClosures730=== RUN TestPresignedUploadRegisteredBeforeCommit731=== PAUSE TestPresignedUploadRegisteredBeforeCommit732=== RUN TestService_Rustfstest733=== PAUSE TestService_Rustfstest734=== RUN TestParseSize735=== PAUSE TestParseSize736=== RUN TestSkippedUploadsHandler737=== PAUSE TestSkippedUploadsHandler738=== RUN TestSystemdListenerNotActivated739--- PASS: TestSystemdListenerNotActivated (0.00s)740=== RUN TestWatchdogBeatsWhenHealthy741--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)742=== RUN TestWatchdogSkipsWhenUnhealthy7432026/11/12 00:09:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7442026/11/12 00:09:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7452026/11/12 00:09:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7462026/11/12 00:09:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7472026/11/12 00:09:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7482026/11/12 00:09:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7492026/11/12 00:09:32 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7502026/11/12 00:09:33 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7512026/11/12 00:09:33 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"7522026/11/12 00:09:33 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"753--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)754=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle755=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle756=== RUN TestProxyWriteTimeout757=== PAUSE TestProxyWriteTimeout758=== RUN TestPendingClosureWriteTimeout759=== PAUSE TestPendingClosureWriteTimeout760=== RUN TestIsValidUploadKey761=== PAUSE TestIsValidUploadKey762=== RUN TestUploadHandlersRejectInvalidKeys763=== PAUSE TestUploadHandlersRejectInvalidKeys764=== RUN TestUploadHandlersRejectOversizedBody765=== PAUSE TestUploadHandlersRejectOversizedBody766=== RUN TestService_cleanupPendingClosuresHandler767=== PAUSE TestService_cleanupPendingClosuresHandler768=== RUN TestService_createPendingClosureHandler769=== PAUSE TestService_createPendingClosureHandler770=== RUN TestService_verifyS3Integrity771=== PAUSE TestService_verifyS3Integrity772=== RUN TestService_verifyS3ReuploadsStaleNarinfo773=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo774=== RUN TestCompleteMultipartUnregistered775=== PAUSE TestCompleteMultipartUnregistered776=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT777=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT778=== CONT TestAbortSucceeded779=== CONT TestSharedNarBothUploadsComplete780--- PASS: TestAbortSucceeded (0.00s)781=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT782=== CONT TestCompleteMultipartUnregistered783=== CONT TestService_verifyS3ReuploadsStaleNarinfo784=== CONT TestService_verifyS3Integrity785=== CONT TestService_createPendingClosureHandler786=== CONT TestService_cleanupPendingClosuresHandler787=== CONT TestUploadHandlersRejectOversizedBody788=== CONT TestUploadHandlersRejectInvalidKeys789=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info790=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info791=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal792=== CONT TestIsValidUploadKey793=== CONT TestPendingClosureWriteTimeout794=== CONT TestProxyWriteTimeout795=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle796=== RUN TestProxyWriteTimeout/narinfo797=== PAUSE TestProxyWriteTimeout/narinfo798=== CONT TestParseSize799=== CONT TestService_Rustfstest800=== CONT TestPresignedUploadRegisteredBeforeCommit801=== CONT TestCompletedNarNotReofferedAcrossClosures802=== CONT TestCompleteMultipartUpload_ErrorButObjectExists803=== CONT TestReadProxyInvalidPath804=== CONT TestGetExistingObjectsLocksRows805=== CONT TestRegisterCompletedObjectNeedsPendingRow806=== CONT TestSweepSkipsRowsLockedByPush807=== CONT TestFailedS3DeleteKeepsTombstone808=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed809=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal810=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key811=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key812=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key813=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key814=== RUN TestIsValidUploadKey/narinfo815=== CONT TestSkippedUploadsHandler816=== RUN TestPendingClosureWriteTimeout/empty817=== RUN TestProxyWriteTimeout/1_GiB_nar818--- PASS: TestParseSize (0.00s)819=== CONT TestForceGCDuringPushOffersSweptObject820=== RUN TestForceGCDuringPushOffersSweptObject/before_pending_rows821=== PAUSE TestForceGCDuringPushOffersSweptObject/before_pending_rows822=== RUN TestForceGCDuringPushOffersSweptObject/after_presence_check823=== PAUSE TestForceGCDuringPushOffersSweptObject/after_presence_check8242026/11/12 00:09:33 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000825=== CONT TestCreatePendingClosureVerifyS3FailureReleasesConnection826=== PAUSE TestProxyWriteTimeout/1_GiB_nar827=== CONT TestGCSweepDeliversEachKeyOnce828=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed829=== RUN TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed830=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed831=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost832=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost833=== RUN TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed834=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed835=== RUN TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed836=== PAUSE TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed837=== PAUSE TestIsValidUploadKey/narinfo838=== RUN TestIsValidUploadKey/nar_zst839=== PAUSE TestIsValidUploadKey/nar_zst840=== RUN TestIsValidUploadKey/nar_xz841=== PAUSE TestIsValidUploadKey/nar_xz842=== RUN TestIsValidUploadKey/nar_plain843=== PAUSE TestIsValidUploadKey/nar_plain844=== RUN TestIsValidUploadKey/listing845=== PAUSE TestIsValidUploadKey/listing846=== RUN TestIsValidUploadKey/build_log847=== PAUSE TestIsValidUploadKey/build_log848=== RUN TestIsValidUploadKey/build_log_home-manager_file849=== PAUSE TestIsValidUploadKey/build_log_home-manager_file850=== RUN TestIsValidUploadKey/build_log_plus_in_name851=== PAUSE TestIsValidUploadKey/build_log_plus_in_name852=== RUN TestIsValidUploadKey/build_log_question_mark853=== PAUSE TestIsValidUploadKey/build_log_question_mark854=== RUN TestIsValidUploadKey/build_log_equals855=== PAUSE TestIsValidUploadKey/build_log_equals856=== RUN TestIsValidUploadKey/realisation857=== PAUSE TestIsValidUploadKey/realisation858=== RUN TestIsValidUploadKey/realisation_plus_in_output859=== PAUSE TestIsValidUploadKey/realisation_plus_in_output860=== RUN TestIsValidUploadKey/nix-cache-info861=== RUN TestProxyWriteTimeout/10_GiB_nar862=== PAUSE TestProxyWriteTimeout/10_GiB_nar863=== PAUSE TestIsValidUploadKey/nix-cache-info864=== PAUSE TestPendingClosureWriteTimeout/empty865=== CONT TestTombstonedObjectOfferedWithoutWaiting866=== RUN TestProxyWriteTimeout/unknown_size867=== PAUSE TestProxyWriteTimeout/unknown_size868=== CONT TestGCSweepSkipsPendingObjects869=== RUN TestIsValidUploadKey/index.html870=== PAUSE TestIsValidUploadKey/index.html871=== RUN TestIsValidUploadKey/narinfo_key,_nar_type872=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type873=== RUN TestIsValidUploadKey/nar_key,_narinfo_type874=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type875=== RUN TestIsValidUploadKey/listing_key,_narinfo_type876=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type877=== RUN TestIsValidUploadKey/traversal878=== PAUSE TestIsValidUploadKey/traversal879=== RUN TestIsValidUploadKey/traversal_nar880=== PAUSE TestIsValidUploadKey/traversal_nar881=== RUN TestIsValidUploadKey/absolute882=== PAUSE TestIsValidUploadKey/absolute883=== RUN TestPendingClosureWriteTimeout/400_objects884=== PAUSE TestPendingClosureWriteTimeout/400_objects885=== RUN TestPendingClosureWriteTimeout/670k_objects886=== PAUSE TestPendingClosureWriteTimeout/670k_objects887=== RUN TestIsValidUploadKey/empty_key888=== PAUSE TestIsValidUploadKey/empty_key889=== RUN TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline890=== PAUSE TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline891=== RUN TestIsValidUploadKey/unknown_type892=== PAUSE TestIsValidUploadKey/unknown_type893=== CONT TestDeduplicatedObjectsRecordedAsPending894=== CONT TestPushDedupSurvivesConcurrentGC895--- PASS: TestSkippedUploadsHandler (0.07s)896=== CONT TestGCMetrics897=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart898=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart899=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts900=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts901=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure902=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure903=== CONT TestGCBugBareHashReferences9042026/11/12 00:09:33 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9052026/11/12 00:09:33 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst906--- PASS: TestCompleteMultipartUnregistered (0.33s)907=== CONT TestSweepRechecksPendingAfterLocking9082026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9092026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9102026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9112026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9122026/11/12 00:09:33 INFO Received cleanup request method=DELETE path=/api/pending_closures9132026/11/12 00:09:33 INFO Aborted multipart uploads count=0 kept=09142026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9152026/11/12 00:09:33 INFO Received cleanup request method=DELETE path=/api/pending_closures9162026/11/12 00:09:33 INFO Aborted multipart uploads count=1 kept=09172026/11/12 00:09:33 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete9182026-11-12 00:09:33.489 UTC [647] ERROR: Closure does not exist: id=19192026-11-12 00:09:33.489 UTC [647] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 13 at RAISE9202026-11-12 00:09:33.489 UTC [647] STATEMENT: -- name: CommitPendingClosure :one921 SELECT commit_pending_closure($1::bigint)::bigint922 923--- PASS: TestService_cleanupPendingClosuresHandler (0.43s)924=== CONT TestGCAdvisoryLockBlocksConcurrentRun925--- PASS: TestReadProxyInvalidPath (0.42s)926=== CONT TestLeadEndsOnShutdown9272026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9282026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9292026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures930--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.51s)931=== CONT TestLeadGenerationGrowsWithEveryGrant9322026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9332026/11/12 00:09:33 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst9342026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9352026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures936--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.53s)937=== CONT TestReadProxy404938--- PASS: TestService_Rustfstest (0.57s)939=== CONT TestLeadIncumbentAnnouncesAtOnce9402026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9412026/11/12 00:09:33 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9422026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9432026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures944--- PASS: TestDeduplicatedObjectsRecordedAsPending (0.58s)945=== CONT TestReadProxyNarStreaming9462026/11/12 00:09:33 INFO Aborted multipart uploads count=0 kept=09472026/11/12 00:09:33 WARN Force mode enabled - objects will be deleted immediately without grace period948--- PASS: TestGetExistingObjectsLocksRows (0.61s)949=== CONT TestLeadUnannouncedStandbyYieldsToIncumbent9502026/11/12 00:09:33 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."9512026/11/12 00:09:33 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."952--- PASS: TestFailedS3DeleteKeepsTombstone (0.69s)953=== CONT TestLeadElectsOneAndHandsOver9542026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9552026/11/12 00:09:33 INFO Aborted multipart uploads count=0 kept=09562026/11/12 00:09:33 INFO Aborted multipart uploads count=0 kept=09572026/11/12 00:09:33 WARN Force mode enabled - objects will be deleted immediately without grace period9582026/11/12 00:09:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09592026/11/12 00:09:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09602026/11/12 00:09:33 INFO Vacuumed table table=pending_closures9612026/11/12 00:09:33 INFO Vacuumed table table=pending_closures9622026/11/12 00:09:33 INFO Vacuumed table table=pending_objects9632026/11/12 00:09:33 INFO Vacuumed table table=multipart_uploads9642026/11/12 00:09:33 INFO Vacuumed table table=closures9652026/11/12 00:09:33 INFO Vacuumed table table=objects9662026/11/12 00:09:33 INFO Vacuumed table table=pending_objects9672026/11/12 00:09:33 INFO Vacuumed table table=multipart_uploads9682026/11/12 00:09:33 INFO Vacuumed table table=closures9692026/11/12 00:09:33 INFO Vacuumed table table=objects970--- PASS: TestGCMetrics (0.67s)971=== CONT TestReadProxyContentNegotiation9722026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9732026/11/12 00:09:33 INFO Aborted multipart uploads count=0 kept=09742026/11/12 00:09:33 WARN Force mode enabled - objects will be deleted immediately without grace period9752026/11/12 00:09:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=09762026/11/12 00:09:33 INFO Vacuumed table table=pending_closures9772026/11/12 00:09:33 INFO Vacuumed table table=pending_objects9782026/11/12 00:09:33 INFO Vacuumed table table=multipart_uploads9792026/11/12 00:09:33 INFO Vacuumed table table=closures9802026/11/12 00:09:33 INFO Vacuumed table table=objects981--- PASS: TestPushDedupSurvivesConcurrentGC (0.69s)982=== CONT TestConnectWaitsForAPeerMigration9832026/11/12 00:09:33 INFO Received complete multipart upload request method=POST path=/api/multipart/complete9842026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures9852026/11/12 00:09:33 INFO Received complete multipart upload request method=POST path=/api/multipart/complete986--- PASS: TestTombstonedObjectOfferedWithoutWaiting (0.72s)987=== CONT TestResolveDBConnectionString988=== RUN TestResolveDBConnectionString/flag_wins989=== PAUSE TestResolveDBConnectionString/flag_wins990=== RUN TestResolveDBConnectionString/file_when_flag_empty9912026/11/12 00:09:33 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YTVjMDZkZjMtNTEwMy00MjlkLTkxMjQtNTdhNDE4YjcyY2I2LjdkNTc4ZGFkLTkwNjgtNDM4MC1iOGQ5LTQ4NTQ0YjQ2OGVmZngxNzk0NDQyMTczODMyNDQwMTkz992=== PAUSE TestResolveDBConnectionString/file_when_flag_empty993=== RUN TestResolveDBConnectionString/missing_file_is_an_error994=== PAUSE TestResolveDBConnectionString/missing_file_is_an_error995=== RUN TestResolveDBConnectionString/PGHOST_allows_empty996=== PAUSE TestResolveDBConnectionString/PGHOST_allows_empty997=== RUN TestResolveDBConnectionString/nothing_configured998=== PAUSE TestResolveDBConnectionString/nothing_configured999=== CONT TestReadProxyNarinfoAlreadyDecompressed10002026/11/12 00:09:33 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=YTVjMDZkZjMtNTEwMy00MjlkLTkxMjQtNTdhNDE4YjcyY2I2LjdkNTc4ZGFkLTkwNjgtNDM4MC1iOGQ5LTQ4NTQ0YjQ2OGVmZngxNzk0NDQyMTczODMyNDQwMTkz parts=11001--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.74s)1002=== CONT TestConcurrentCommitsSharingObjectsDoNotDeadlock10032026/11/12 00:09:33 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10042026/11/12 00:09:33 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=YTVjMDZkZjMtNTEwMy00MjlkLTkxMjQtNTdhNDE4YjcyY2I2LjU4ZGVhMDg4LTk2NDUtNGZiYy1iNjRkLWU0YTlkYjFiOTI3OXgxNzk0NDQyMTczNDI1ODY5NjQ2 parts=1010052026/11/12 00:09:33 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10062026/11/12 00:09:33 INFO Completed upload id=1 objects=2 duration=5.298037ms10072026/11/12 00:09:33 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000010082026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures10092026/11/12 00:09:33 INFO Starting cleanup of old closures method=DELETE path=/api/closures10102026/11/12 00:09:33 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=YTVjMDZkZjMtNTEwMy00MjlkLTkxMjQtNTdhNDE4YjcyY2I2LmJjMGMzNjUwLTM3MmEtNDVkYS1iZWEwLWYzMjNjYjUwMDE2ZngxNzk0NDQyMTczNDU3ODg2OTcz parts=1010112026/11/12 00:09:33 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete10122026/11/12 00:09:33 INFO Completed upload id=1 objects=2 duration=4.272179ms10132026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures10142026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures10152026/11/12 00:09:33 INFO Aborted multipart uploads count=0 kept=010162026/11/12 00:09:33 INFO Received uploads request method=POST path=/api/pending_closures10172026/11/12 00:09:33 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo10182026/11/12 00:09:33 WARN Found objects in DB but missing from S3, will re-upload count=11019--- PASS: TestService_verifyS3Integrity (0.87s)1020=== CONT TestReadProxyNarinfo10212026/11/12 00:09:33 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010222026/11/12 00:09:33 INFO Vacuumed table table=pending_closures10232026/11/12 00:09:33 INFO Vacuumed table table=pending_objects10242026/11/12 00:09:33 INFO Vacuumed table table=multipart_uploads10252026/11/12 00:09:33 INFO Vacuumed table table=closures10262026/11/12 00:09:33 INFO Vacuumed table table=objects1027--- PASS: TestRegisterCompletedObjectNeedsPendingRow (0.81s)1028=== CONT TestDeleteClosuresRacingPins10292026/11/12 00:09:33 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001030--- PASS: TestService_createPendingClosureHandler (0.90s)1031=== CONT TestIsValidCachePath1032=== RUN TestIsValidCachePath/narinfo1033=== PAUSE TestIsValidCachePath/narinfo1034=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars1035=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars1036=== RUN TestIsValidCachePath/nar_zst1037=== PAUSE TestIsValidCachePath/nar_zst1038=== RUN TestIsValidCachePath/nar_xz1039=== PAUSE TestIsValidCachePath/nar_xz1040=== RUN TestIsValidCachePath/nar_bz21041=== PAUSE TestIsValidCachePath/nar_bz21042=== RUN TestIsValidCachePath/nar_uncompressed1043=== PAUSE TestIsValidCachePath/nar_uncompressed1044=== RUN TestIsValidCachePath/ls1045=== PAUSE TestIsValidCachePath/ls1046=== RUN TestIsValidCachePath/log1047=== PAUSE TestIsValidCachePath/log1048=== RUN TestIsValidCachePath/realisation1049=== PAUSE TestIsValidCachePath/realisation1050=== RUN TestIsValidCachePath/nix-cache-info1051=== PAUSE TestIsValidCachePath/nix-cache-info1052=== RUN TestIsValidCachePath/index.html1053=== PAUSE TestIsValidCachePath/index.html1054=== RUN TestIsValidCachePath/traversal_parent1055=== PAUSE TestIsValidCachePath/traversal_parent1056=== RUN TestIsValidCachePath/traversal_in_middle1057=== PAUSE TestIsValidCachePath/traversal_in_middle1058=== RUN TestIsValidCachePath/invalid_char_e1059=== PAUSE TestIsValidCachePath/invalid_char_e1060=== RUN TestIsValidCachePath/invalid_char_u1061=== PAUSE TestIsValidCachePath/invalid_char_u1062=== RUN TestIsValidCachePath/random_path1063=== PAUSE TestIsValidCachePath/random_path1064=== RUN TestIsValidCachePath/empty1065=== PAUSE TestIsValidCachePath/empty1066=== RUN TestIsValidCachePath/leading_slash1067=== PAUSE TestIsValidCachePath/leading_slash1068=== RUN TestIsValidCachePath/wrong_extension1069=== PAUSE TestIsValidCachePath/wrong_extension1070=== RUN TestIsValidCachePath/short_hash1071=== PAUSE TestIsValidCachePath/short_hash1072=== CONT TestClientFallsBackToClosures10732026/11/12 00:09:34 INFO Aborted multipart uploads count=0 kept=010742026/11/12 00:09:34 WARN Force mode enabled - objects will be deleted immediately without grace period10752026/11/12 00:09:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete10762026/11/12 00:09:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010772026/11/12 00:09:34 INFO Vacuumed table table=pending_closures10782026/11/12 00:09:34 INFO Vacuumed table table=pending_objects10792026/11/12 00:09:34 INFO Vacuumed table table=multipart_uploads10802026/11/12 00:09:34 INFO Vacuumed table table=closures10812026/11/12 00:09:34 INFO Vacuumed table table=objects10822026/11/12 00:09:34 INFO Aborted multipart uploads count=0 kept=010832026/11/12 00:09:34 WARN Force mode enabled - objects will be deleted immediately without grace period10842026/11/12 00:09:34 INFO Received uploads request method=POST path=/api/pending_closures10852026/11/12 00:09:34 INFO Aborted multipart uploads count=0 kept=010862026/11/12 00:09:34 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=YTVjMDZkZjMtNTEwMy00MjlkLTkxMjQtNTdhNDE4YjcyY2I2LjYyZThmYjhmLTliODItNDVjYi05ZDIzLWM1ZGI3ZmZjMjA1OHgxNzk0NDQyMTczNTMzMjcxMDIw parts=1210872026/11/12 00:09:34 WARN Force mode enabled - objects will be deleted immediately without grace period10882026/11/12 00:09:34 INFO Aborted multipart uploads count=0 kept=010892026/11/12 00:09:34 WARN Force mode enabled - objects will be deleted immediately without grace period10902026/11/12 00:09:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=010912026/11/12 00:09:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=010922026/11/12 00:09:34 INFO Vacuumed table table=pending_closures10932026/11/12 00:09:34 INFO Vacuumed table table=pending_objects10942026/11/12 00:09:34 INFO Vacuumed table table=multipart_uploads10952026/11/12 00:09:34 INFO Vacuumed table table=closures10962026/11/12 00:09:34 INFO Vacuumed table table=pending_closures10972026/11/12 00:09:34 INFO Vacuumed table table=pending_objects10982026/11/12 00:09:34 INFO Vacuumed table table=objects10992026/11/12 00:09:34 INFO Vacuumed table table=multipart_uploads11002026/11/12 00:09:34 INFO Vacuumed table table=closures1101--- PASS: TestSweepSkipsRowsLockedByPush (0.94s)1102=== CONT TestClientPushesUseOnePush11032026/11/12 00:09:34 INFO Vacuumed table table=objects1104--- PASS: TestGCSweepSkipsPendingObjects (0.95s)1105=== CONT TestParseSingleRange1106=== RUN TestParseSingleRange/none1107=== PAUSE TestParseSingleRange/none1108=== RUN TestParseSingleRange/unknown_unit1109=== PAUSE TestParseSingleRange/unknown_unit1110=== RUN TestParseSingleRange/multi-range_ignored1111=== PAUSE TestParseSingleRange/multi-range_ignored1112=== RUN TestParseSingleRange/malformed_no_dash1113=== PAUSE TestParseSingleRange/malformed_no_dash1114=== RUN TestParseSingleRange/malformed_both_empty1115=== PAUSE TestParseSingleRange/malformed_both_empty1116=== RUN TestParseSingleRange/malformed_end_before_start1117=== PAUSE TestParseSingleRange/malformed_end_before_start1118=== RUN TestParseSingleRange/closed1119=== PAUSE TestParseSingleRange/closed1120=== RUN TestParseSingleRange/open-ended1121=== PAUSE TestParseSingleRange/open-ended1122=== RUN TestParseSingleRange/end_clamped_to_size1123=== PAUSE TestParseSingleRange/end_clamped_to_size1124=== RUN TestParseSingleRange/suffix1125=== PAUSE TestParseSingleRange/suffix1126=== RUN TestParseSingleRange/suffix_exceeds_size1127=== PAUSE TestParseSingleRange/suffix_exceeds_size1128=== RUN TestParseSingleRange/single_byte1129=== PAUSE TestParseSingleRange/single_byte1130=== RUN TestParseSingleRange/start_past_EOF1131=== PAUSE TestParseSingleRange/start_past_EOF1132=== RUN TestParseSingleRange/start_far_past_EOF1133=== PAUSE TestParseSingleRange/start_far_past_EOF1134=== CONT TestPinProtectsFromGC11352026/11/12 00:09:34 INFO Aborted multipart uploads count=0 kept=011362026/11/12 00:09:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=011372026/11/12 00:09:34 INFO Vacuumed table table=pending_closures11382026/11/12 00:09:34 INFO Vacuumed table table=pending_objects11392026/11/12 00:09:34 INFO Vacuumed table table=multipart_uploads11402026/11/12 00:09:34 INFO Vacuumed table table=closures11412026/11/12 00:09:34 WARN Force mode enabled - objects will be deleted immediately without grace period11422026/11/12 00:09:34 INFO Vacuumed table table=objects1143--- PASS: TestGCSweepDeliversEachKeyOnce (0.96s)1144=== CONT TestPresentNotReportedWhileGCDeletesClosure1145=== RUN TestPresentNotReportedWhileGCDeletesClosure/stale1146=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/stale1147=== RUN TestPresentNotReportedWhileGCDeletesClosure/fresh1148=== PAUSE TestPresentNotReportedWhileGCDeletesClosure/fresh1149=== CONT TestPresentRefreshesOnlyStaleClosures11502026/11/12 00:09:34 ERROR failed to check GC advisory lock error="query pg_locks: failed to connect to `user=nixbld database=niks3`: /nonexistent/.s.PGSQL.5432 (/nonexistent): dial error: dial unix /nonexistent/.s.PGSQL.5432: connect: no such file or directory"1151--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.63s)1152=== CONT TestClientSharedPathCommittedMidPush1153--- PASS: TestCreatePendingClosureVerifyS3FailureReleasesConnection (0.98s)1154=== CONT TestClientWithDependencies11552026/11/12 00:09:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=011562026/11/12 00:09:34 INFO Vacuumed table table=pending_closures1157--- PASS: TestGCBugBareHashReferences (0.89s)1158=== CONT TestClientMultipleUploads11592026/11/12 00:09:34 INFO Vacuumed table table=pending_objects11602026/11/12 00:09:34 INFO Vacuumed table table=multipart_uploads11612026/11/12 00:09:34 INFO Vacuumed table table=closures11622026/11/12 00:09:34 INFO Vacuumed table table=objects1163--- PASS: TestSweepRechecksPendingAfterLocking (0.75s)1164=== CONT TestClientIntegration11652026/11/12 00:09:34 INFO lead: acquired remote=192.0.2.1:1234 generation=111662026/11/12 00:09:34 INFO lead: acquired remote=192.0.2.1:1234 generation=111672026/11/12 00:09:34 INFO lead: acquired remote=192.0.2.1:1234 generation=111682026/11/12 00:09:34 INFO lead: released remote=192.0.2.1:1234 generation=11169--- PASS: TestLeadIncumbentAnnouncesAtOnce (0.62s)1170=== CONT TestClientErrorHandling1171=== RUN TestClientErrorHandling/InvalidStorePath1172=== PAUSE TestClientErrorHandling/InvalidStorePath1173=== RUN TestClientErrorHandling/InvalidAuthToken1174=== PAUSE TestClientErrorHandling/InvalidAuthToken1175=== RUN TestClientErrorHandling/ServerNotAvailable1176=== PAUSE TestClientErrorHandling/ServerNotAvailable1177=== CONT TestPresentReportsOnlyClosureRoots1178--- PASS: TestReadProxyNarStreaming (0.57s)1179=== CONT TestClientCADerivations1180--- PASS: TestReadProxy404 (0.69s)1181=== CONT TestCreatePin_ReservedPins11822026/11/12 00:09:34 INFO lead: acquired remote=192.0.2.1:1234 generation=111832026/11/12 00:09:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11842026/11/12 00:09:34 INFO lead: acquired remote=192.0.2.1:1234 generation=111852026/11/12 00:09:34 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=YTVjMDZkZjMtNTEwMy00MjlkLTkxMjQtNTdhNDE4YjcyY2I2LmZiZjY5Njg4LTM2NzEtNDZiZi05ZGY0LTFlMDU2NDUzNDU5Y3gxNzk0NDQyMTczNzc1NzU1NDM3 parts=1211862026/11/12 00:09:34 INFO Received uploads request method=POST path=/api/pending_closures1187--- PASS: TestCompletedNarNotReofferedAcrossClosures (1.23s)1188=== CONT TestCacheStatsHandler11892026/11/12 00:09:34 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46463/oidc1190--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.55s)1191=== CONT TestCacheConfigHandler1192=== RUN TestCacheConfigHandler/full_config,_no_issuer1193=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1194=== RUN TestCacheConfigHandler/no_cache_url_configured1195=== PAUSE TestCacheConfigHandler/no_cache_url_configured1196=== RUN TestCacheConfigHandler/no_signing_keys1197=== PAUSE TestCacheConfigHandler/no_signing_keys1198=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1199=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1200=== CONT TestOrphanedObjectsGCStressTest1201--- PASS: TestReadProxyContentNegotiation (0.64s)1202=== CONT TestOrphanedObjectsGC12032026/11/12 00:09:34 ERROR Failed to decode narinfo key=3hcdxyjf9yiq7qf3i4548drb6sjmwa1v.narinfo limit=16777216 error=<nil>1204--- PASS: TestReadProxyNarinfo (0.69s)1205=== CONT TestService_ReadScope_PublicByDefault1206--- PASS: TestPresentRefreshesOnlyStaleClosures (0.53s)1207=== CONT TestObjectStatsUpdatedOncePerCommit12082026/11/12 00:09:34 INFO Received complete multipart upload request method=POST path=/api/multipart/complete12092026/11/12 00:09:34 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=YTVjMDZkZjMtNTEwMy00MjlkLTkxMjQtNTdhNDE4YjcyY2I2LmZmOGVjZTFlLTAxZjMtNGVlNS05YjkwLTU0ODMxM2MxMTJiOHgxNzk0NDQyMTczNTQyNjQ0Nzg2 parts=121210--- PASS: TestSharedNarBothUploadsComplete (1.62s)1211=== CONT TestService_RequireScope_OIDC1212=== NAME TestPinProtectsFromGC1213 client_integration_test.go:779: Pinned store path: /build/TestPinProtectsFromGC4115951193/001/store/jqn9malp487sbximvcfd333n0k9yz98c-pinned-file.txt1214 client_integration_test.go:780: Unpinned store path: /build/TestPinProtectsFromGC4115951193/001/store/3svy8v589n3nr3x6npzagab6hl3nz61f-unpinned-file.txt1215--- PASS: TestConcurrentCommitsSharingObjectsDoNotDeadlock (0.81s)1216=== CONT TestService_AuthMiddleware_OIDC12172026/11/12 00:09:34 INFO Received uploads request method=POST path=/api/pending_closures12182026/11/12 00:09:34 INFO Received uploads request method=POST path=/api/pending_closures1219=== NAME TestClientWithDependencies1220 client_integration_test.go:661: Built derivation: /build/TestClientWithDependencies2940472908/001/store/8j9fzg3hbjcq5kbxhawpzh3ndp9f3x74-test-script12212026/11/12 00:09:34 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)12222026/11/12 00:09:34 INFO Uploading srkimzq1dhd2b9iybhsri8vkfgi4bf89-a (216B)12232026/11/12 00:09:34 INFO Uploading nrm0shy48ggv6rg92xbm4fm4ifm4580k-shared-dep (136B)1224=== NAME TestClientMultipleUploads12252026/11/12 00:09:34 WARN Failed to register uploaded object key=nv80fw0cq0iabfgjwpbfigycbj0dpawg.ls error="server returned 404: 404 page not found\n"1226 client_integration_test.go:406: Created store path 0: /build/TestClientMultipleUploads1484375559/001/store/0jc1mz2aarwjximgrwqwwkf44yb6hrpv-test-file-0.txt12272026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/1w2iihvvhjn62qp9g1jhqgya6r18jgk9x80nx93sxxzi4gpy9a1d.nar.zst error="server returned 404: 404 page not found\n"12282026/11/12 00:09:34 WARN Failed to register uploaded object key=nrm0shy48ggv6rg92xbm4fm4ifm4580k.ls error="server returned 404: 404 page not found\n"12292026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"12302026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign12312026/11/12 00:09:34 WARN Failed to register uploaded object key=srkimzq1dhd2b9iybhsri8vkfgi4bf89.ls error="server returned 404: 404 page not found\n"12322026/11/12 00:09:34 INFO Signed narinfos id=1 count=212332026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign12342026/11/12 00:09:34 INFO Signed narinfos id=2 count=212352026/11/12 00:09:34 INFO Uploading 4 narinfos12362026/11/12 00:09:34 INFO Received push request method=POST path=/api/pushes12372026/11/12 00:09:34 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)12382026/11/12 00:09:34 INFO Uploading jqn9malp487sbximvcfd333n0k9yz98c-pinned-file.txt (128B)12392026/11/12 00:09:34 WARN Failed to register uploaded object key=nv80fw0cq0iabfgjwpbfigycbj0dpawg.narinfo error="server returned 404: 404 page not found\n"12402026/11/12 00:09:34 WARN Failed to register uploaded object key=nrm0shy48ggv6rg92xbm4fm4ifm4580k.narinfo error="server returned 404: 404 page not found\n"12412026/11/12 00:09:34 WARN Failed to register uploaded object key=srkimzq1dhd2b9iybhsri8vkfgi4bf89.narinfo error="server returned 404: 404 page not found\n"12422026/11/12 00:09:34 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37223/oidc1243--- PASS: TestPresentReportsOnlyClosureRoots (0.51s)12442026/11/12 00:09:34 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1245=== CONT TestObjectStatsTrigger12462026/11/12 00:09:34 WARN Failed to register uploaded object key=nrm0shy48ggv6rg92xbm4fm4ifm4580k.narinfo error="server returned 404: 404 page not found\n"12472026/11/12 00:09:34 WARN Failed to register uploaded object key=jqn9malp487sbximvcfd333n0k9yz98c.ls error="server returned 404: 404 page not found\n"12482026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign12492026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"12502026/11/12 00:09:34 INFO Signed narinfos id=1 count=112512026/11/12 00:09:34 INFO Uploading 1 narinfos12522026/11/12 00:09:34 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46589/oidc1253=== NAME TestClientWithDependencies1254 client_integration_test.go:663: Found 1 dependencies (including self)12552026/11/12 00:09:34 INFO Completed upload id=1 objects=6 duration=5.904517ms12562026/11/12 00:09:34 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete12572026/11/12 00:09:34 INFO Received complete push request method=POST path=/api/pushes/1/complete12582026/11/12 00:09:34 WARN Failed to register uploaded object key=jqn9malp487sbximvcfd333n0k9yz98c.narinfo error="server returned 404: 404 page not found\n"12592026/11/12 00:09:34 INFO Completed upload id=2 objects=6 duration=1.328219ms12602026/11/12 00:09:34 INFO Upload complete. (79ms)1261=== NAME TestClientFallsBackToClosures1262 client_pushes_test.go:112: Retrieved narinfo from S3:1263 StorePath: /build/TestClientFallsBackToClosures1253154433/001/store/nrm0shy48ggv6rg92xbm4fm4ifm4580k-shared-dep1264 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1265 Compression: zstd1266 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821267 NarSize: 1361268 References: 1269 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1270=== NAME TestClientIntegration1271 client_integration_test.go:292: Created store path: /build/TestClientIntegration3288560628/002/store/zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j-test-file.txt1272=== NAME TestClientFallsBackToClosures1273 client_pushes_test.go:112: Retrieved narinfo from S3:1274 StorePath: /build/TestClientFallsBackToClosures1253154433/001/store/srkimzq1dhd2b9iybhsri8vkfgi4bf89-a1275 URL: nar/1w2iihvvhjn62qp9g1jhqgya6r18jgk9x80nx93sxxzi4gpy9a1d.nar.zst1276 Compression: zstd12772026/11/12 00:09:34 INFO Received push request method=POST path=/api/pushes1278 NarHash: sha256:1w2iihvvhjn62qp9g1jhqgya6r18jgk9x80nx93sxxzi4gpy9a1d1279 NarSize: 2161280 References: /build/TestClientFallsBackToClosures1253154433/001/store/nrm0shy48ggv6rg92xbm4fm4ifm4580k-shared-dep1281 CA: text:sha256:1niminz8s2ap9mx3j3mnlwb2mk491h5mg8dsn2fza82hwhig53qb12822026/11/12 00:09:34 INFO Completed push id=1 objects=3 duration=8.241796ms12832026/11/12 00:09:34 INFO Upload complete. (66ms)1284 client_pushes_test.go:112: Retrieved narinfo from S3:1285 StorePath: /build/TestClientFallsBackToClosures1253154433/001/store/nv80fw0cq0iabfgjwpbfigycbj0dpawg-b1286 URL: nar/1w2iihvvhjn62qp9g1jhqgya6r18jgk9x80nx93sxxzi4gpy9a1d.nar.zst1287 Compression: zstd1288 NarHash: sha256:1w2iihvvhjn62qp9g1jhqgya6r18jgk9x80nx93sxxzi4gpy9a1d1289 NarSize: 2161290 References: /build/TestClientFallsBackToClosures1253154433/001/store/nrm0shy48ggv6rg92xbm4fm4ifm4580k-shared-dep1291 CA: text:sha256:1niminz8s2ap9mx3j3mnlwb2mk491h5mg8dsn2fza82hwhig53qb12922026/11/12 00:09:34 INFO Uploading 2 paths to 127.0.0.1 (1 already cached)12932026/11/12 00:09:34 INFO Uploading 3clm0463q7g05gjb1417n0ksv28np1cb-shared-dep (136B)12942026/11/12 00:09:34 INFO Uploading 0cqcxl9iyinrdxy5h1b7dav4d5fwyz2l-a (216B)1295=== NAME TestClientMultipleUploads1296 client_integration_test.go:406: Created store path 1: /build/TestClientMultipleUploads1484375559/001/store/mcrkg2jj7vs9ba9wws6n4yn4i8hyn9h8-test-file-1.txt1297--- PASS: TestClientFallsBackToClosures (0.83s)1298=== CONT TestService_ReadAuthMiddleware12992026/11/12 00:09:34 WARN Failed to register uploaded object key=vq36nkimv7qn70b0wl7yy7jv423f5zf4.ls error="server returned 404: 404 page not found\n"13002026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13012026/11/12 00:09:34 WARN Failed to register uploaded object key=0cqcxl9iyinrdxy5h1b7dav4d5fwyz2l.ls error="server returned 404: 404 page not found\n"13022026/11/12 00:09:34 WARN Failed to register uploaded object key=3clm0463q7g05gjb1417n0ksv28np1cb.ls error="server returned 404: 404 page not found\n"13032026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/0f7gjlczn5l5wsjz347avwyji2dikdfnbk9nlb2d1c2angxhc04b.nar.zst error="server returned 404: 404 page not found\n"13042026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13052026/11/12 00:09:34 INFO Signed narinfos id=1 count=313062026/11/12 00:09:34 INFO Uploading 3 narinfos13072026/11/12 00:09:34 WARN Failed to register uploaded object key=0cqcxl9iyinrdxy5h1b7dav4d5fwyz2l.narinfo error="server returned 404: 404 page not found\n"13082026/11/12 00:09:34 WARN Failed to register uploaded object key=3clm0463q7g05gjb1417n0ksv28np1cb.narinfo error="server returned 404: 404 page not found\n"13092026/11/12 00:09:34 INFO Received complete push request method=POST path=/api/pushes/1/complete13102026/11/12 00:09:34 WARN Failed to register uploaded object key=vq36nkimv7qn70b0wl7yy7jv423f5zf4.narinfo error="server returned 404: 404 page not found\n"13112026/11/12 00:09:34 INFO Received push request method=POST path=/api/pushes1312=== NAME TestClientMultipleUploads1313 client_integration_test.go:406: Created store path 2: /build/TestClientMultipleUploads1484375559/001/store/giqshv5ia3185jcmvwjlvrsx6vkbxfkj-test-file-2.txt13142026/11/12 00:09:34 INFO Completed push id=1 objects=8 duration=21.424441ms13152026/11/12 00:09:34 INFO Upload complete. (86ms)1316=== NAME TestClientPushesUseOnePush1317 client_pushes_test.go:97: Retrieved narinfo from S3:1318 StorePath: /build/TestClientPushesUseOnePush1321801652/001/store/3clm0463q7g05gjb1417n0ksv28np1cb-shared-dep1319 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1320 Compression: zstd1321 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821322 NarSize: 1361323 References: 1324 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n1325 client_pushes_test.go:97: Retrieved narinfo from S3:1326 StorePath: /build/TestClientPushesUseOnePush1321801652/001/store/0cqcxl9iyinrdxy5h1b7dav4d5fwyz2l-a1327 URL: nar/0f7gjlczn5l5wsjz347avwyji2dikdfnbk9nlb2d1c2angxhc04b.nar.zst1328 Compression: zstd1329 NarHash: sha256:0f7gjlczn5l5wsjz347avwyji2dikdfnbk9nlb2d1c2angxhc04b1330 NarSize: 2161331 References: /build/TestClientPushesUseOnePush1321801652/001/store/3clm0463q7g05gjb1417n0ksv28np1cb-shared-dep1332 CA: text:sha256:05z3nfjdabrs9cidkq4g4f0s5bcbkrk07czpiwjllcwx85k4syxw1333 client_pushes_test.go:97: Retrieved narinfo from S3:1334 StorePath: /build/TestClientPushesUseOnePush1321801652/001/store/vq36nkimv7qn70b0wl7yy7jv423f5zf4-b1335 URL: nar/0f7gjlczn5l5wsjz347avwyji2dikdfnbk9nlb2d1c2angxhc04b.nar.zst1336 Compression: zstd1337 NarHash: sha256:0f7gjlczn5l5wsjz347avwyji2dikdfnbk9nlb2d1c2angxhc04b1338 NarSize: 2161339 References: /build/TestClientPushesUseOnePush1321801652/001/store/3clm0463q7g05gjb1417n0ksv28np1cb-shared-dep1340 CA: text:sha256:05z3nfjdabrs9cidkq4g4f0s5bcbkrk07czpiwjllcwx85k4syxw13412026/11/12 00:09:34 INFO Uploading 2 paths to 127.0.0.1 (0 already cached)13422026/11/12 00:09:34 INFO Uploading 2r9c6n3nmhw83j2jyh12sh40a7dn2v8b-shared-dep (136B)13432026/11/12 00:09:34 INFO Uploading fk8zp4kwsqvybdmgy4r9dvzw6nm6w5ys-top (224B)1344--- PASS: TestCacheStatsHandler (0.46s)1345=== CONT TestPendingCleanupUsesOneCutoff1346--- PASS: TestClientPushesUseOnePush (0.76s)1347=== CONT TestService_AuthMiddleware_MTLSBoundSubjects13482026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/0y58d9hy095xfxaa1s2bd96iv8mygzfrv7yk18vp9v1d01mswf3p.nar.zst error="server returned 404: 404 page not found\n"13492026/11/12 00:09:34 WARN Failed to register uploaded object key=fk8zp4kwsqvybdmgy4r9dvzw6nm6w5ys.ls error="server returned 404: 404 page not found\n"13502026/11/12 00:09:34 INFO Received push request method=POST path=/api/pushes13512026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13522026/11/12 00:09:34 WARN Failed to register uploaded object key=2r9c6n3nmhw83j2jyh12sh40a7dn2v8b.ls error="server returned 404: 404 page not found\n"13532026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst error="server returned 404: 404 page not found\n"13542026/11/12 00:09:34 INFO Signed narinfos id=1 count=213552026/11/12 00:09:34 INFO Uploading 2 narinfos13562026/11/12 00:09:34 INFO Received push request method=POST path=/api/pushes13572026/11/12 00:09:34 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13582026/11/12 00:09:34 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux13592026/11/12 00:09:34 WARN Refused reserved pin name=worker-x86_64-linux13602026/11/12 00:09:34 INFO Uploading 3svy8v589n3nr3x6npzagab6hl3nz61f-unpinned-file.txt (128B)13612026/11/12 00:09:34 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux13622026/11/12 00:09:34 INFO Received create pin request method=POST path=/api/pins/my-app13632026/11/12 00:09:34 INFO Received create pin request method=POST path=/api/pins/worker-x86_64-linux13642026/11/12 00:09:34 WARN Failed to register uploaded object key=fk8zp4kwsqvybdmgy4r9dvzw6nm6w5ys.narinfo error="server returned 404: 404 page not found\n"13652026/11/12 00:09:34 INFO Received push request method=POST path=/api/pushes1366--- PASS: TestCreatePin_ReservedPins (0.56s)1367=== CONT TestMultipartUploadAbortedWhenCancelledBeforeRecorded13682026/11/12 00:09:34 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13692026/11/12 00:09:34 INFO Uploading 8j9fzg3hbjcq5kbxhawpzh3ndp9f3x74-test-script (136B)13702026/11/12 00:09:34 INFO Received complete push request method=POST path=/api/pushes/1/complete13712026/11/12 00:09:34 WARN Failed to register uploaded object key=2r9c6n3nmhw83j2jyh12sh40a7dn2v8b.narinfo error="server returned 404: 404 page not found\n"13722026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"13732026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign13742026/11/12 00:09:34 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)13752026/11/12 00:09:34 WARN Failed to register uploaded object key=3svy8v589n3nr3x6npzagab6hl3nz61f.ls error="server returned 404: 404 page not found\n"13762026/11/12 00:09:34 INFO Uploading zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j-test-file.txt (152B)13772026/11/12 00:09:34 INFO Signed narinfos id=2 count=113782026/11/12 00:09:34 INFO Uploading 1 narinfos13792026/11/12 00:09:34 INFO Completed push id=1 objects=6 duration=6.031568ms13802026/11/12 00:09:34 WARN Failed to register uploaded object key=8j9fzg3hbjcq5kbxhawpzh3ndp9f3x74.ls error="server returned 404: 404 page not found\n"13812026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"13822026/11/12 00:09:34 INFO Upload complete. (74ms)13832026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign13842026/11/12 00:09:34 WARN Failed to register uploaded object key=log/2qa477fw27w5xw32dgq4zmr17zqmsqy2-test-script.drv error="server returned 404: 404 page not found\n"13852026/11/12 00:09:34 INFO Signed narinfos id=1 count=113862026/11/12 00:09:34 INFO Uploading 1 narinfos1387=== NAME TestClientSharedPathCommittedMidPush1388 client_integration_test.go:728: Retrieved narinfo from S3:1389 StorePath: /build/TestClientSharedPathCommittedMidPush3885998300/001/store/2r9c6n3nmhw83j2jyh12sh40a7dn2v8b-shared-dep1390 URL: nar/1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y82.nar.zst1391 Compression: zstd1392 NarHash: sha256:1vnl0g6w4zgpfz06rb321mfd3jqsrs120jcl8dg2dxn014875y821393 NarSize: 1361394 References: 1395 CA: text:sha256:08nxw05k7b0pzzy0f3swvnw4ldxdbclkdnklpidwmrxdhvfrmv7n13962026/11/12 00:09:34 INFO Received complete push request method=POST path=/api/pushes/2/complete13972026/11/12 00:09:34 WARN Failed to register uploaded object key=3svy8v589n3nr3x6npzagab6hl3nz61f.narinfo error="server returned 404: 404 page not found\n"1398 client_integration_test.go:728: Retrieved narinfo from S3:1399 StorePath: /build/TestClientSharedPathCommittedMidPush3885998300/001/store/fk8zp4kwsqvybdmgy4r9dvzw6nm6w5ys-top1400 URL: nar/0y58d9hy095xfxaa1s2bd96iv8mygzfrv7yk18vp9v1d01mswf3p.nar.zst1401 Compression: zstd1402 NarHash: sha256:0y58d9hy095xfxaa1s2bd96iv8mygzfrv7yk18vp9v1d01mswf3p1403 NarSize: 2241404 References: /build/TestClientSharedPathCommittedMidPush3885998300/001/store/2r9c6n3nmhw83j2jyh12sh40a7dn2v8b-shared-dep1405 CA: text:sha256:1r0x0ynqi1qziq99sqj93rb7xpqfzsk3r7g1hqi433g5cgs1bmlp14062026/11/12 00:09:34 WARN Failed to register uploaded object key=zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j.ls error="server returned 404: 404 page not found\n"14072026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14082026/11/12 00:09:34 INFO Signed narinfos id=1 count=114092026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"14102026/11/12 00:09:34 INFO Uploading 1 narinfos14112026/11/12 00:09:34 INFO Completed push id=2 objects=3 duration=2.398109ms14122026/11/12 00:09:34 INFO Upload complete. (56ms)14132026/11/12 00:09:34 INFO Received complete push request method=POST path=/api/pushes/1/complete14142026/11/12 00:09:34 WARN Failed to register uploaded object key=8j9fzg3hbjcq5kbxhawpzh3ndp9f3x74.narinfo error="server returned 404: 404 page not found\n"14152026/11/12 00:09:34 WARN Failed to register uploaded object key=zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j.narinfo error="server returned 404: 404 page not found\n"1416--- PASS: TestClientSharedPathCommittedMidPush (0.75s)14172026/11/12 00:09:34 INFO Received complete push request method=POST path=/api/pushes/1/complete1418=== CONT TestMultipartCleanup1419=== NAME TestClientCADerivations1420 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations2812799262/001/store/cviiw4jwkk3li9hvlhfmpas46024wakm-ca-test14212026/11/12 00:09:34 INFO Completed push id=1 objects=4 duration=11.021195ms14222026/11/12 00:09:34 INFO Upload complete. (73ms)14232026/11/12 00:09:34 INFO Completed push id=1 objects=3 duration=8.747266ms14242026/11/12 00:09:34 INFO Upload complete. (70ms)1425=== NAME TestClientWithDependencies1426 client_integration_test.go:665: Skipping nix copy test - isolated store (/build/TestClientWithDependencies2940472908/001/store) requires matching store prefix1427--- PASS: TestClientWithDependencies (0.76s)1428=== CONT TestService_AuthMiddleware_MTLSProxyHeader14292026/11/12 00:09:34 INFO Received push request method=POST path=/api/pushes14302026/11/12 00:09:34 INFO Starting cleanup of old closures method=DELETE path=/api/closures14312026/11/12 00:09:34 INFO Garbage collection started14322026/11/12 00:09:34 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)14332026/11/12 00:09:34 INFO Uploading 0jc1mz2aarwjximgrwqwwkf44yb6hrpv-test-file-0.txt (160B)14342026/11/12 00:09:34 INFO Uploading mcrkg2jj7vs9ba9wws6n4yn4i8hyn9h8-test-file-1.txt (160B)14352026/11/12 00:09:34 INFO Uploading giqshv5ia3185jcmvwjlvrsx6vkbxfkj-test-file-2.txt (160B)1436=== NAME TestClientCADerivations1437 client_ca_test.go:139: Found 1 dependencies (including self)14382026/11/12 00:09:34 WARN Failed to register uploaded object key=0jc1mz2aarwjximgrwqwwkf44yb6hrpv.ls error="server returned 404: 404 page not found\n"14392026/11/12 00:09:34 WARN Failed to register uploaded object key=giqshv5ia3185jcmvwjlvrsx6vkbxfkj.ls error="server returned 404: 404 page not found\n"14402026/11/12 00:09:34 WARN Failed to register uploaded object key=mcrkg2jj7vs9ba9wws6n4yn4i8hyn9h8.ls error="server returned 404: 404 page not found\n"14412026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"14422026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"14432026/11/12 00:09:34 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign14442026/11/12 00:09:34 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"14452026/11/12 00:09:34 INFO Signed narinfos id=1 count=314462026/11/12 00:09:34 INFO Uploading 3 narinfos14472026/11/12 00:09:34 INFO Aborted multipart uploads count=0 kept=014482026/11/12 00:09:34 WARN Failed to register uploaded object key=mcrkg2jj7vs9ba9wws6n4yn4i8hyn9h8.narinfo error="server returned 404: 404 page not found\n"14492026/11/12 00:09:34 WARN Failed to register uploaded object key=giqshv5ia3185jcmvwjlvrsx6vkbxfkj.narinfo error="server returned 404: 404 page not found\n"14502026/11/12 00:09:34 INFO Received complete push request method=POST path=/api/pushes/1/complete14512026/11/12 00:09:34 WARN Failed to register uploaded object key=0jc1mz2aarwjximgrwqwwkf44yb6hrpv.narinfo error="server returned 404: 404 page not found\n"14522026/11/12 00:09:34 INFO All 1 paths already cached1453=== NAME TestClientIntegration1454 client_integration_test.go:318: Retrieved narinfo from S3:1455 StorePath: /build/TestClientIntegration3288560628/002/store/zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j-test-file.txt1456 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1457 Compression: zstd1458 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11459 NarSize: 1521460 References: 1461 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk114622026/11/12 00:09:34 WARN Force mode enabled - objects will be deleted immediately without grace period1463 client_integration_test.go:319: Retrieved .ls file from S3 (compressed size: 77 bytes)1464 client_integration_test.go:319: Decompressed .ls content (64 bytes):1465 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1466--- PASS: TestService_ReadScope_PublicByDefault (0.31s)1467=== CONT TestServerTLSConfig1468=== RUN TestServerTLSConfig/no_client_CA1469=== PAUSE TestServerTLSConfig/no_client_CA1470=== RUN TestServerTLSConfig/missing_CA_file1471=== PAUSE TestServerTLSConfig/missing_CA_file1472=== RUN TestServerTLSConfig/not_a_PEM_file1473=== PAUSE TestServerTLSConfig/not_a_PEM_file1474=== CONT TestService_AuthMiddleware14752026/11/12 00:09:34 INFO Completed push id=1 objects=9 duration=11.897035ms14762026/11/12 00:09:34 INFO Upload complete. (75ms)1477=== NAME TestClientMultipleUploads1478 client_integration_test.go:417: Uploaded 3 paths in 108.999864ms1479--- PASS: TestClientMultipleUploads (0.81s)1480=== CONT TestService_NativeMTLS14812026/11/12 00:09:34 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=014822026/11/12 00:09:34 INFO Vacuumed table table=pending_closures14832026/11/12 00:09:34 INFO Vacuumed table table=pending_objects14842026/11/12 00:09:34 INFO Vacuumed table table=multipart_uploads14852026/11/12 00:09:34 INFO Vacuumed table table=closures14862026/11/12 00:09:34 INFO Vacuumed table table=objects1487--- PASS: TestObjectStatsUpdatedOncePerCommit (0.36s)1488=== CONT TestGCTaskStore_Fail1489--- PASS: TestGCTaskStore_Fail (0.00s)1490=== CONT TestMetricsCommit1491=== RUN TestService_RequireScope_OIDC/builder_may_write1492=== PAUSE TestService_RequireScope_OIDC/builder_may_write1493=== RUN TestService_RequireScope_OIDC/builder_may_not_admin1494=== PAUSE TestService_RequireScope_OIDC/builder_may_not_admin1495=== RUN TestService_RequireScope_OIDC/ops_may_admin1496=== PAUSE TestService_RequireScope_OIDC/ops_may_admin1497=== RUN TestService_RequireScope_OIDC/ops_may_not_write1498=== PAUSE TestService_RequireScope_OIDC/ops_may_not_write1499=== RUN TestService_RequireScope_OIDC/reader_may_not_write1500=== PAUSE TestService_RequireScope_OIDC/reader_may_not_write1501=== RUN TestService_RequireScope_OIDC/static_token_may_admin1502=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin1503=== RUN TestService_RequireScope_OIDC/static_token_may_write1504=== PAUSE TestService_RequireScope_OIDC/static_token_may_write1505=== RUN TestService_RequireScope_OIDC/reader_may_read1506=== PAUSE TestService_RequireScope_OIDC/reader_may_read1507=== RUN TestService_RequireScope_OIDC/writer_implies_read1508=== PAUSE TestService_RequireScope_OIDC/writer_implies_read1509=== RUN TestService_RequireScope_OIDC/anonymous_may_not_read1510=== PAUSE TestService_RequireScope_OIDC/anonymous_may_not_read1511=== RUN TestService_RequireScope_OIDC/reader_may_read_with_netrc1512=== PAUSE TestService_RequireScope_OIDC/reader_may_read_with_netrc1513=== RUN TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1514=== PAUSE TestService_RequireScope_OIDC/static_token_may_admin_with_netrc1515=== RUN TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1516=== PAUSE TestService_RequireScope_OIDC/wrong_netrc_password_may_not_read1517=== RUN TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1518=== PAUSE TestService_RequireScope_OIDC/empty_netrc_password_may_not_read1519=== CONT TestGCTaskStore_PhaseUpdates1520--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1521=== CONT TestMetricsInventory15222026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes15232026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes1524=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1525=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1526=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1527=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1528=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1529=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1530=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1531=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1532=== CONT TestGCTaskStore_CompletedAllowsNewTask1533--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1534=== CONT TestNARDeduplicationMetadataUploadBug15352026/11/12 00:09:35 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15362026/11/12 00:09:35 INFO Uploading cviiw4jwkk3li9hvlhfmpas46024wakm-ca-test (144B)15372026/11/12 00:09:35 INFO Object in database but missing from S3 key=zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j.ls15382026/11/12 00:09:35 WARN Found objects in DB but missing from S3, will re-upload count=115392026/11/12 00:09:35 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)15402026/11/12 00:09:35 WARN Failed to register uploaded object key=log/r8ya2i7a80q7zi5sxal4yr7fmg41hci8-ca-test.drv error="server returned 404: 404 page not found\n"15412026/11/12 00:09:35 WARN Failed to register uploaded object key=cviiw4jwkk3li9hvlhfmpas46024wakm.ls error="server returned 404: 404 page not found\n"15422026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign15432026/11/12 00:09:35 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"15442026/11/12 00:09:35 INFO Signed narinfos id=1 count=115452026/11/12 00:09:35 INFO Uploading 1 narinfos15462026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/2/complete15472026/11/12 00:09:35 WARN Failed to register uploaded object key=zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j.ls error="server returned 404: 404 page not found\n"15482026/11/12 00:09:35 INFO Completed push id=2 objects=3 duration=2.990909ms15492026/11/12 00:09:35 INFO Upload complete. (55ms)1550=== NAME TestClientIntegration1551 client_integration_test.go:336: Retrieved .ls file from S3 (compressed size: 62 bytes)1552 client_integration_test.go:336: Decompressed .ls content (49 bytes):1553 {"version":1,"root":{"type":"regular","size":39}}15542026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/1/complete15552026/11/12 00:09:35 WARN Failed to register uploaded object key=cviiw4jwkk3li9hvlhfmpas46024wakm.narinfo error="server returned 404: 404 page not found\n"15562026/11/12 00:09:35 INFO Completed push id=1 objects=4 duration=10.186346ms15572026/11/12 00:09:35 INFO Upload complete. (111ms)1558=== NAME TestClientCADerivations1559 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations2812799262/001/store/cviiw4jwkk3li9hvlhfmpas46024wakm-ca-test1560 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1561 Compression: zstd1562 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1563 NarSize: 1441564 References: 1565 Deriver: /build/TestClientCADerivations2812799262/001/store/r8ya2i7a80q7zi5sxal4yr7fmg41hci8-ca-test.drv1566 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1567 client_ca_test.go:185: Checking for realisation files in S3...1568 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1569 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1570--- PASS: TestObjectStatsTrigger (0.30s)1571=== CONT TestGCTaskStore_GetReturnsLatest1572--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1573=== CONT TestCreatePendingClosureRejectsOversizedNAR15742026/11/12 00:09:35 INFO Received uploads request method=POST path=/api/pending_closures1575--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)1576=== CONT TestGCTaskStore_GetEmpty1577--- PASS: TestGCTaskStore_GetEmpty (0.00s)1578=== CONT TestCacheConfigHandlerMaxNarSize1579--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)1580=== CONT TestGCTaskStore_ConflictDifferentParams1581--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1582=== CONT TestReadRedirectUsesPublicS3URL1583--- PASS: TestService_ReadAuthMiddleware (0.27s)1584=== CONT TestGCTaskStore_DeduplicateSameParams1585--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1586=== CONT TestGCTaskStore_StartNew1587--- PASS: TestGCTaskStore_StartNew (0.00s)1588=== CONT TestGCEndsOnShutdown1589=== RUN TestGCEndsOnShutdown/before_the_run1590=== PAUSE TestGCEndsOnShutdown/before_the_run1591=== RUN TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1592=== PAUSE TestGCEndsOnShutdown/blocked_in_the_sweep's_S3_delete1593=== RUN TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1594=== PAUSE TestGCEndsOnShutdown/blocked_on_a_row_lock_in_the_mark1595=== CONT TestEnsureAbortUploadsRule15962026/11/12 00:09:35 INFO Received uploads request method=POST path=/api/pending_closures15972026/11/12 00:09:35 INFO Received uploads request method=POST path=/api/pending_closures15982026/11/12 00:09:35 INFO Received cleanup request method=DELETE path=/api/pending_closures15992026/11/12 00:09:35 INFO Received uploads request method=POST path=/api/pending_closures16002026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes16012026/11/12 00:09:35 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)16022026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/3/complete16032026/11/12 00:09:35 WARN Failed to register uploaded object key=zjfm3rz1rla4ffcjmj2hlb5pvpm0h61j.ls error="server returned 404: 404 page not found\n"1604--- PASS: TestMultipartUploadAbortedWhenCancelledBeforeRecorded (0.28s)1605=== CONT TestPush_SignsNarinfosOfItsPendingObjects16062026/11/12 00:09:35 INFO Completed push id=3 objects=3 duration=3.673048ms16072026/11/12 00:09:35 INFO Upload complete. (46ms)1608=== NAME TestClientIntegration1609 client_integration_test.go:353: Retrieved .ls file from S3 (compressed size: 62 bytes)1610 client_integration_test.go:353: Decompressed .ls content (49 bytes):1611 {"version":1,"root":{"type":"regular","size":39}}1612 client_integration_test.go:364: Testing garbage collection...16132026/11/12 00:09:35 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"16142026/11/12 00:09:35 WARN mTLS auth: bound subjects configured but subject DN unavailable16152026/11/12 00:09:35 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1616--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.31s)1617=== CONT TestGenerateLandingPage1618--- PASS: TestGenerateLandingPage (0.00s)1619=== CONT TestService_readinessHandler1620=== NAME TestClientCADerivations16212026/11/12 00:09:35 INFO Starting cleanup of old closures method=DELETE path=/api/closures1622 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1623 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1624 error: binary cache 's3://bucket51?endpoint=http://localhost:46211®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations2812799262/001/store'16252026/11/12 00:09:35 INFO Garbage collection started1626 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11627--- PASS: TestClientCADerivations (0.88s)1628=== CONT TestCommitRacingPendingCleanupKeepsObjects16292026/11/12 00:09:35 INFO Received uploads request method=POST path=/api/pending_closures16302026/11/12 00:09:35 INFO Aborted multipart uploads count=0 kept=016312026/11/12 00:09:35 WARN Force mode enabled - objects will be deleted immediately without grace period1632--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.32s)1633=== CONT TestService_healthCheckHandler16342026/11/12 00:09:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=016352026/11/12 00:09:35 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"1636--- PASS: TestService_AuthMiddleware (0.31s)1637=== CONT TestGracefulShutdownDrainsInflight16382026/11/12 00:09:35 INFO Starting HTTP server address=127.0.0.1:4299316392026/11/12 00:09:35 INFO Shutdown signal received, draining in-flight requests timeout=10s16402026/11/12 00:09:35 INFO Vacuumed table table=pending_closures16412026/11/12 00:09:35 INFO Vacuumed table table=pending_objects16422026/11/12 00:09:35 INFO Vacuumed table table=multipart_uploads16432026/11/12 00:09:35 INFO Vacuumed table table=closures16442026/11/12 00:09:35 INFO lead: released remote=192.0.2.1:1234 generation=11645--- PASS: TestLeadEndsOnShutdown (1.76s)1646=== CONT TestReadProxyDisabled16472026/11/12 00:09:35 INFO Vacuumed table table=objects16482026/11/12 00:09:35 WARN mTLS auth: subject not in bound subjects subject="CN=reader"16492026/11/12 00:09:35 WARN mTLS auth: subject not in bound subjects subject="CN=reader"1650--- PASS: TestService_NativeMTLS (0.32s)1651=== CONT TestReadProxyRangeRequest16522026/11/12 00:09:35 INFO Received cleanup request method=DELETE path=/api/pending_closures16532026/11/12 00:09:35 WARN Failed to abort upload, keeping its closure key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst error="Get \"http://127.0.0.1:1/bucket65/?location=\": dial tcp 127.0.0.1:1: connect: connection refused" code=""16542026/11/12 00:09:35 INFO Aborted multipart uploads count=0 kept=116552026/11/12 00:09:35 INFO Received cleanup request method=DELETE path=/api/pending_closures16562026/11/12 00:09:35 INFO Aborted multipart uploads count=1 kept=01657--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1658=== CONT TestReadRedirectKeepsNarinfoProxied1659--- PASS: TestMultipartCleanup (0.45s)1660=== CONT TestReadRedirectNar16612026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes16622026/11/12 00:09:35 INFO lead: released remote=192.0.2.1:1234 generation=11663=== NAME TestOrphanedObjectsGC1664 orphaned_objects_gc_test.go:287: GC Test Summary:1665 orphaned_objects_gc_test.go:288: - Kept: 2 objects from closure A1666 orphaned_objects_gc_test.go:289: - Deleted: 2 objects from closure B1667 orphaned_objects_gc_test.go:290: - Deleted: 6 orphaned chain objects (X1->X2->X3)1668 orphaned_objects_gc_test.go:291: - Deleted: 2 orphaned single objects (Y)1669 orphaned_objects_gc_test.go:292: - Total deleted: 10 objects16702026/11/12 00:09:35 INFO lead: acquired remote=192.0.2.1:1234 generation=21671--- PASS: TestOrphanedObjectsGC (0.88s)1672=== CONT TestReadProxyConditionalGet16732026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/1/complete16742026/11/12 00:09:35 INFO Completed push id=1 objects=4 duration=9.011226ms16752026/11/12 00:09:35 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete16762026/11/12 00:09:35 INFO Completed upload id=2 objects=2 duration=1.772279ms1677--- PASS: TestMetricsCommit (0.38s)1678=== CONT TestReadProxyRootRedirectsToIndexHTML1679=== NAME TestNARDeduplicationMetadataUploadBug1680 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug154988274/001/store/hf202b9h7hfs4kbrc0bdq5ngwmfxkd10-file1.txt1681--- PASS: TestMetricsInventory (0.41s)1682=== CONT TestPush_CompleteWaitsForCleanupOfThePush16832026/11/12 00:09:35 INFO Installed lifecycle rule aborting incomplete NAR uploads bucket=bucket72 days=21684--- PASS: TestEnsureAbortUploadsRule (0.37s)1685=== CONT TestPush_SkippedKeySurvivesGCBeforeCommit16862026/11/12 00:09:35 INFO lead: released remote=192.0.2.1:1234 generation=11687--- PASS: TestReadRedirectUsesPublicS3URL (0.40s)1688=== CONT TestPushWaitsForSweepHoldingRow16892026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes16902026/11/12 00:09:35 INFO lead: acquired remote=192.0.2.1:1234 generation=21691--- PASS: TestDeleteClosuresRacingPins (1.53s)1692=== CONT TestPush_RootsAreLockedInByteOrder16932026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes16942026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16952026/11/12 00:09:35 INFO Signed narinfos id=1 count=116962026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16972026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign16982026/11/12 00:09:35 ERROR Failed to sign narinfos signer="index 1" error="signing key cannot be nil"16992026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17002026/11/12 00:09:35 ERROR Failed to sign narinfos signer=niks3-test-1 error="signer failed"17012026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17022026/11/12 00:09:35 ERROR Signer returned unexpected signature count signer=niks3-test-1 got=0 want=117032026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17042026/11/12 00:09:35 ERROR Signer returned unexpected object key signer=niks3-test-1 object_key=unexpected.narinfo1705--- PASS: TestPush_SignsNarinfosOfItsPendingObjects (0.36s)1706=== CONT TestReadProxyOutlastsServerWriteTimeout17072026/11/12 00:09:35 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17082026/11/12 00:09:35 INFO Uploading hf202b9h7hfs4kbrc0bdq5ngwmfxkd10-file1.txt (160B)17092026/11/12 00:09:35 WARN Failed to register uploaded object key=hf202b9h7hfs4kbrc0bdq5ngwmfxkd10.ls error="server returned 404: 404 page not found\n"17102026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign17112026/11/12 00:09:35 WARN readiness check failed error="closed pool"1712--- PASS: TestService_readinessHandler (0.35s)17132026/11/12 00:09:35 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"1714=== CONT TestPush_CompleteCommitsEveryRoot17152026/11/12 00:09:35 INFO Signed narinfos id=1 count=117162026/11/12 00:09:35 INFO Uploading 1 narinfos17172026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/1/complete17182026/11/12 00:09:35 WARN Failed to register uploaded object key=hf202b9h7hfs4kbrc0bdq5ngwmfxkd10.narinfo error="server returned 404: 404 page not found\n"17192026/11/12 00:09:35 INFO lead: given back to the incumbent remote=192.0.2.1:1234 generation=217202026/11/12 00:09:35 INFO Completed push id=1 objects=3 duration=11.551705ms17212026/11/12 00:09:35 INFO Upload complete. (80ms)1722=== NAME TestNARDeduplicationMetadataUploadBug1723 metadata_upload_test.go:54: Retrieved narinfo from S3:1724 StorePath: /build/TestNARDeduplicationMetadataUploadBug154988274/001/store/hf202b9h7hfs4kbrc0bdq5ngwmfxkd10-file1.txt1725 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1726 Compression: zstd1727 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1728 NarSize: 1601729 References: 1730 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf17312026/11/12 00:09:35 INFO lead: acquired remote=192.0.2.1:1234 generation=31732 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1733 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1734 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}1735 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug154988274/001/store/sbkz7apbshgj9nw29xf3d40yfs349c1v-file2.txt1736--- PASS: TestService_healthCheckHandler (0.36s)1737=== CONT TestPush_RejectsBadRequests17382026/11/12 00:09:35 INFO Aborted multipart uploads count=0 kept=017392026/11/12 00:09:35 WARN Force mode enabled - objects will be deleted immediately without grace period17402026/11/12 00:09:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=017412026/11/12 00:09:35 INFO Vacuumed table table=pending_closures17422026/11/12 00:09:35 INFO Vacuumed table table=pending_objects17432026/11/12 00:09:35 INFO Vacuumed table table=multipart_uploads17442026/11/12 00:09:35 INFO Vacuumed table table=closures17452026/11/12 00:09:35 INFO Vacuumed table table=objects1746--- PASS: TestReadProxyDisabled (0.34s)1747=== CONT TestPush_RepeatedRootCommitsOnce1748--- PASS: TestCommitRacingPendingCleanupKeepsObjects (0.42s)1749=== CONT TestPush_OverlappingRootsStoreOneRowPerKey17502026/11/12 00:09:35 INFO lead: released remote=192.0.2.1:1234 generation=11751--- PASS: TestReadProxyRangeRequest (0.37s)1752=== CONT TestReadProxyHead17532026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes17542026/11/12 00:09:35 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17552026/11/12 00:09:35 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign17562026/11/12 00:09:35 WARN Failed to register uploaded object key=sbkz7apbshgj9nw29xf3d40yfs349c1v.ls error="server returned 404: 404 page not found\n"17572026/11/12 00:09:35 INFO Signed narinfos id=2 count=117582026/11/12 00:09:35 INFO Uploading 1 narinfos17592026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/2/complete17602026/11/12 00:09:35 WARN Failed to register uploaded object key=sbkz7apbshgj9nw29xf3d40yfs349c1v.narinfo error="server returned 404: 404 page not found\n"1761--- PASS: TestReadRedirectKeepsNarinfoProxied (0.35s)1762=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info17632026/11/12 00:09:35 INFO Received uploads request method=POST path=/1764=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key17652026/11/12 00:09:35 INFO Received complete multipart upload request method=POST path=/1766=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal17672026/11/12 00:09:35 INFO lead: acquired remote=192.0.2.1:1234 generation=217682026/11/12 00:09:35 INFO Received uploads request method=POST path=/1769=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key17702026/11/12 00:09:35 INFO Received request for more parts method=POST path=/1771--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1772 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1773 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1774 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1775 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1776=== CONT TestForceGCDuringPushOffersSweptObject/before_pending_rows17772026/11/12 00:09:35 INFO Completed push id=2 objects=3 duration=13.062654ms17782026/11/12 00:09:35 INFO Upload complete. (64ms)1779=== NAME TestNARDeduplicationMetadataUploadBug1780 metadata_upload_test.go:76: Retrieved narinfo from S3:1781 StorePath: /build/TestNARDeduplicationMetadataUploadBug154988274/001/store/sbkz7apbshgj9nw29xf3d40yfs349c1v-file2.txt1782 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1783 Compression: zstd1784 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1785 NarSize: 1601786 References: 1787 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1788 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1789 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1790 {"version":1,"root":{"type":"regular","size":44}}1791--- PASS: TestNARDeduplicationMetadataUploadBug (0.66s)1792=== CONT TestForceGCDuringPushOffersSweptObject/after_presence_check1793--- PASS: TestReadRedirectNar (0.36s)1794=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_other_NAR_pushed1795--- PASS: TestReadProxyConditionalGet (0.37s)1796=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost1797--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.36s)1798=== CONT TestService_verifyS3ReuploadsStaleNarinfo/intact,_other_NAR_pushed17992026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes18002026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes18012026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/1/complete18022026/11/12 00:09:35 INFO Received uploads request method=POST path=/api/pending_closures18032026/11/12 00:09:35 INFO Completed push id=1 objects=4 duration=10.990885ms18042026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes18052026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes18062026-11-12 00:09:35.880 UTC [2089] ERROR: could not obtain lock on row in relation "closures"18072026-11-12 00:09:35.880 UTC [2089] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18082026-11-12 00:09:35.880 UTC [2089] ERROR: could not obtain lock on row in relation "closures"18092026-11-12 00:09:35.880 UTC [2089] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18102026-11-12 00:09:35.881 UTC [2089] ERROR: could not obtain lock on row in relation "closures"18112026-11-12 00:09:35.881 UTC [2089] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18122026-11-12 00:09:35.881 UTC [2089] ERROR: could not obtain lock on row in relation "closures"18132026-11-12 00:09:35.881 UTC [2089] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT18142026-11-12 00:09:35.881 UTC [2089] ERROR: could not obtain lock on row in relation "closures"18152026-11-12 00:09:35.881 UTC [2089] STATEMENT: SELECT 1 FROM closures WHERE key = $1 FOR UPDATE NOWAIT1816--- PASS: TestPush_RootsAreLockedInByteOrder (0.40s)1817=== CONT TestService_verifyS3ReuploadsStaleNarinfo/lost,_same_NAR_pushed18182026/11/12 00:09:35 INFO Aborted multipart uploads count=0 kept=018192026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes18202026/11/12 00:09:35 WARN Force mode enabled - objects will be deleted immediately without grace period18212026/11/12 00:09:35 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=2 objects-failed-to-delete=018222026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/1/complete18232026/11/12 00:09:35 INFO Vacuumed table table=pending_closures18242026/11/12 00:09:35 INFO Vacuumed table table=pending_objects18252026/11/12 00:09:35 INFO Vacuumed table table=multipart_uploads18262026/11/12 00:09:35 INFO Completed push id=1 objects=6 duration=7.520027ms1827--- PASS: TestPush_CompleteCommitsEveryRoot (0.42s)1828=== CONT TestService_verifyS3ReuploadsStaleNarinfo/first_of_two_lost,_second_pushed1829=== RUN TestPush_RejectsBadRequests/root_not_in_objects1830=== PAUSE TestPush_RejectsBadRequests/root_not_in_objects1831=== RUN TestPush_RejectsBadRequests/no_roots1832=== PAUSE TestPush_RejectsBadRequests/no_roots18332026/11/12 00:09:35 INFO Vacuumed table table=closures1834=== RUN TestPush_RejectsBadRequests/no_objects1835=== PAUSE TestPush_RejectsBadRequests/no_objects1836=== RUN TestPush_RejectsBadRequests/bad_root1837=== PAUSE TestPush_RejectsBadRequests/bad_root1838=== CONT TestProxyWriteTimeout/narinfo1839=== CONT TestProxyWriteTimeout/10_GiB_nar1840=== CONT TestProxyWriteTimeout/unknown_size1841=== CONT TestProxyWriteTimeout/1_GiB_nar1842--- PASS: TestProxyWriteTimeout (0.09s)1843 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1844 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1845 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1846 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1847=== CONT TestPendingClosureWriteTimeout/empty1848=== CONT TestPendingClosureWriteTimeout/handler_extends_the_server's_deadline18492026/11/12 00:09:35 INFO Vacuumed table table=objects18502026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/2/complete18512026/11/12 00:09:35 INFO Completed push id=2 objects=4 duration=5.396888ms1852--- PASS: TestPush_SkippedKeySurvivesGCBeforeCommit (0.50s)1853=== CONT TestPendingClosureWriteTimeout/670k_objects1854=== CONT TestPendingClosureWriteTimeout/400_objects1855=== CONT TestIsValidUploadKey/narinfo1856=== CONT TestIsValidUploadKey/realisation1857=== CONT TestIsValidUploadKey/build_log_equals1858=== CONT TestIsValidUploadKey/build_log_question_mark1859=== CONT TestIsValidUploadKey/build_log_plus_in_name1860=== CONT TestIsValidUploadKey/build_log_home-manager_file1861=== CONT TestIsValidUploadKey/build_log1862=== CONT TestIsValidUploadKey/listing1863=== CONT TestIsValidUploadKey/nar_plain1864=== CONT TestIsValidUploadKey/nar_xz1865=== CONT TestIsValidUploadKey/nar_zst1866=== CONT TestIsValidUploadKey/traversal1867=== CONT TestIsValidUploadKey/unknown_type1868=== CONT TestIsValidUploadKey/empty_key1869=== CONT TestIsValidUploadKey/absolute1870=== CONT TestIsValidUploadKey/traversal_nar1871=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1872=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1873=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1874=== CONT TestIsValidUploadKey/index.html1875=== CONT TestIsValidUploadKey/nix-cache-info1876=== CONT TestIsValidUploadKey/realisation_plus_in_output1877--- PASS: TestIsValidUploadKey (0.09s)1878 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1879 --- PASS: TestIsValidUploadKey/realisation (0.00s)1880 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1881 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1882 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1883 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1884 --- PASS: TestIsValidUploadKey/build_log (0.00s)1885 --- PASS: TestIsValidUploadKey/listing (0.00s)1886 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1887 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1888 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1889 --- PASS: TestIsValidUploadKey/traversal (0.00s)1890 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1891 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1892 --- PASS: TestIsValidUploadKey/absolute (0.00s)1893 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1894 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1895 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1896 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1897 --- PASS: TestIsValidUploadKey/index.html (0.00s)1898 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1899 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1900=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart19012026/11/12 00:09:35 INFO Received complete multipart upload request method=POST path=/19022026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes1903=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure19042026/11/12 00:09:35 INFO Received uploads request method=POST path=/19052026/11/12 00:09:35 INFO Received complete push request method=POST path=/api/pushes/1/complete19062026/11/12 00:09:35 INFO Completed push id=1 objects=4 duration=6.521147ms1907--- PASS: TestPush_RepeatedRootCommitsOnce (0.38s)1908=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts19092026/11/12 00:09:35 INFO Received request for more parts method=POST path=/19102026/11/12 00:09:35 INFO Received push request method=POST path=/api/pushes1911--- PASS: TestPush_OverlappingRootsStoreOneRowPerKey (0.40s)1912=== CONT TestResolveDBConnectionString/flag_wins1913=== CONT TestResolveDBConnectionString/PGHOST_allows_empty1914=== CONT TestResolveDBConnectionString/nothing_configured1915=== CONT TestResolveDBConnectionString/missing_file_is_an_error1916=== CONT TestResolveDBConnectionString/file_when_flag_empty1917=== CONT TestIsValidCachePath/narinfo1918=== CONT TestIsValidCachePath/index.html1919=== CONT TestIsValidCachePath/nix-cache-info1920=== CONT TestIsValidCachePath/realisation1921=== CONT TestIsValidCachePath/log1922=== CONT TestIsValidCachePath/ls1923=== CONT TestIsValidCachePath/nar_uncompressed1924=== CONT TestIsValidCachePath/nar_bz21925=== CONT TestIsValidCachePath/nar_xz1926=== CONT TestIsValidCachePath/nar_zst1927=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1928--- PASS: TestResolveDBConnectionString (0.00s)1929 --- PASS: TestResolveDBConnectionString/flag_wins (0.00s)1930 --- PASS: TestResolveDBConnectionString/PGHOST_allows_empty (0.00s)1931 --- PASS: TestResolveDBConnectionString/nothing_configured (0.00s)1932 --- PASS: TestResolveDBConnectionString/missing_file_is_an_error (0.00s)1933 --- PASS: TestResolveDBConnectionString/file_when_flag_empty (0.00s)1934=== CONT TestIsValidCachePath/traversal_parent1935=== CONT TestIsValidCachePath/empty1936=== CONT TestIsValidCachePath/random_path1937=== CONT TestIsValidCachePath/invalid_char_u1938=== CONT TestIsValidCachePath/invalid_char_e1939=== CONT TestIsValidCachePath/traversal_in_middle1940=== CONT TestIsValidCachePath/wrong_extension1941=== CONT TestIsValidCachePath/short_hash1942=== CONT TestIsValidCachePath/leading_slash1943--- PASS: TestIsValidCachePath (0.00s)1944 --- PASS: TestIsValidCachePath/narinfo (0.00s)1945 --- PASS: TestIsValidCachePath/index.html (0.00s)1946 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1947 --- PASS: TestIsValidCachePath/realisation (0.00s)1948 --- PASS: TestIsValidCachePath/log (0.00s)1949 --- PASS: TestIsValidCachePath/ls (0.00s)1950 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1951 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1952 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1953 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1954 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1955 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1956 --- PASS: TestIsValidCachePath/empty (0.00s)1957 --- PASS: TestIsValidCachePath/random_path (0.00s)1958 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1959 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1960 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1961 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1962 --- PASS: TestIsValidCachePath/short_hash (0.00s)1963 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1964=== CONT TestParseSingleRange/none1965=== CONT TestParseSingleRange/open-ended1966=== CONT TestParseSingleRange/start_far_past_EOF1967=== CONT TestParseSingleRange/start_past_EOF1968=== CONT TestParseSingleRange/single_byte1969=== CONT TestParseSingleRange/suffix_exceeds_size1970=== CONT TestParseSingleRange/suffix1971=== CONT TestParseSingleRange/end_clamped_to_size1972=== CONT TestParseSingleRange/malformed_both_empty1973=== CONT TestParseSingleRange/closed1974=== CONT TestParseSingleRange/malformed_end_before_start1975=== CONT TestParseSingleRange/multi-range_ignored1976=== CONT TestParseSingleRange/malformed_no_dash1977=== CONT TestParseSingleRange/unknown_unit1978=== CONT TestPresentNotReportedWhileGCDeletesClosure/stale1979--- PASS: TestParseSingleRange (0.00s)1980 --- PASS: TestParseSingleRange/none (0.00s)1981 --- PASS: TestParseSingleRange/open-ended (0.00s)1982 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1983 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1984 --- PASS: TestParseSingleRange/single_byte (0.00s)1985 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1986 --- PASS: TestParseSingleRange/suffix (0.00s)1987 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1988 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1989 --- PASS: TestParseSingleRange/closed (0.00s)1990 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1991 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1992 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1993 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1994--- PASS: TestReadProxyHead (0.39s)1995=== CONT TestPresentNotReportedWhileGCDeletesClosure/fresh1996=== CONT TestClientErrorHandling/InvalidStorePath19972026/11/12 00:09:36 INFO Received uploads request method=POST path=/api/pending_closures19982026/11/12 00:09:36 INFO Aborted multipart uploads count=0 kept=019992026/11/12 00:09:36 WARN Force mode enabled - objects will be deleted immediately without grace period20002026-11-12 00:09:36.075 UTC [2080] ERROR: Push does not exist: id=1
loading 628 hidden lines…
26292026/11/12 00:09:49 INFO Vacuumed table table=pending_objects26302026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads26312026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=026322026/11/12 00:09:49 INFO Vacuumed table table=closures26332026/11/12 00:09:49 INFO Vacuumed table table=objects26342026/11/12 00:09:49 INFO Received uploads request method=POST path=/26352026/11/12 00:09:49 INFO Received complete upload request method=POST path=/26362026/11/12 00:09:49 INFO Completed upload id=7 objects=4 duration=1.771189ms26372026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period26382026/11/12 00:09:49 INFO Received complete upload request method=POST path=/26392026/11/12 00:09:49 INFO Completed upload id=4 objects=14 duration=3.061469ms26402026/11/12 00:09:49 INFO Received complete upload request method=POST path=/26412026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=026422026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period26432026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026442026/11/12 00:09:49 INFO Vacuumed table table=pending_closures26452026/11/12 00:09:49 INFO Completed upload id=10 objects=2 duration=3.205958ms26462026/11/12 00:09:49 INFO Vacuumed table table=pending_objects26472026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads26482026/11/12 00:09:49 INFO Vacuumed table table=closures26492026/11/12 00:09:49 INFO Vacuumed table table=objects26502026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=4 objects-failed-to-delete=026512026/11/12 00:09:49 INFO Vacuumed table table=pending_closures26522026/11/12 00:09:49 INFO Received uploads request method=POST path=/26532026/11/12 00:09:49 INFO Vacuumed table table=pending_objects26542026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads26552026/11/12 00:09:49 INFO Vacuumed table table=closures26562026/11/12 00:09:49 INFO Received uploads request method=POST path=/26572026/11/12 00:09:49 INFO Vacuumed table table=objects26582026/11/12 00:09:49 INFO Received complete push request method=POST path=/26592026/11/12 00:09:49 INFO Received complete upload request method=POST path=/26602026/11/12 00:09:49 INFO Received push request method=POST path=/26612026/11/12 00:09:49 INFO Completed push id=6 objects=7 duration=3.105209ms26622026/11/12 00:09:49 INFO Received push request method=POST path=/26632026/11/12 00:09:49 INFO Completed upload id=2 objects=26 duration=5.365648ms26642026/11/12 00:09:49 INFO Received push request method=POST path=/26652026/11/12 00:09:49 INFO Received complete upload request method=POST path=/26662026/11/12 00:09:49 INFO Completed upload id=5 objects=7 duration=3.193908ms26672026/11/12 00:09:49 INFO Received complete upload request method=POST path=/26682026/11/12 00:09:49 INFO Completed upload id=10 objects=6 duration=2.562709ms26692026/11/12 00:09:49 INFO Received push request method=POST path=/26702026/11/12 00:09:49 INFO Received push request method=POST path=/26712026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=026722026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=026732026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period26742026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=026752026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period26762026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026772026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026782026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=026792026/11/12 00:09:49 INFO Vacuumed table table=pending_closures26802026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=026812026/11/12 00:09:49 INFO Vacuumed table table=pending_closures26822026/11/12 00:09:49 INFO Vacuumed table table=pending_objects26832026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads26842026/11/12 00:09:49 INFO Received complete push request method=POST path=/26852026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period26862026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period26872026/11/12 00:09:49 INFO Vacuumed table table=pending_objects26882026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads26892026/11/12 00:09:49 INFO Vacuumed table table=closures26902026/11/12 00:09:49 INFO Vacuumed table table=closures26912026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026922026/11/12 00:09:49 INFO Vacuumed table table=objects26932026/11/12 00:09:49 INFO Completed push id=4 objects=7 duration=2.429659ms26942026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=026952026/11/12 00:09:49 INFO Vacuumed table table=objects26962026/11/12 00:09:49 INFO Received complete push request method=POST path=/26972026/11/12 00:09:49 INFO Received push request method=POST path=/26982026/11/12 00:09:49 INFO Received push request method=POST path=/26992026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27002026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27012026/11/12 00:09:49 INFO Vacuumed table table=pending_objects27022026/11/12 00:09:49 INFO Vacuumed table table=pending_objects27032026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads27042026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads27052026/11/12 00:09:49 INFO Vacuumed table table=closures27062026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=027072026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27082026/11/12 00:09:49 INFO Vacuumed table table=objects27092026/11/12 00:09:49 INFO Vacuumed table table=closures27102026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27112026/11/12 00:09:49 INFO Completed upload id=6 objects=9 duration=1.542229ms27122026/11/12 00:09:49 INFO Received uploads request method=POST path=/27132026/11/12 00:09:49 INFO Vacuumed table table=objects27142026/11/12 00:09:49 INFO Vacuumed table table=pending_objects27152026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads27162026/11/12 00:09:49 INFO Completed push id=3 objects=22 duration=7.424557ms27172026/11/12 00:09:49 INFO Vacuumed table table=closures27182026/11/12 00:09:49 INFO Vacuumed table table=objects27192026/11/12 00:09:49 INFO Received push request method=POST path=/27202026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27212026/11/12 00:09:49 INFO Completed upload id=2 objects=7 duration=5.408597ms27222026/11/12 00:09:49 INFO Received uploads request method=POST path=/27232026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027242026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027252026/11/12 00:09:49 INFO Received uploads request method=POST path=/27262026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27272026/11/12 00:09:49 INFO Vacuumed table table=pending_objects27282026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads27292026/11/12 00:09:49 INFO Vacuumed table table=closures27302026/11/12 00:09:49 INFO Vacuumed table table=objects27312026/11/12 00:09:49 INFO Received push request method=POST path=/27322026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27332026/11/12 00:09:49 INFO Received uploads request method=POST path=/27342026/11/12 00:09:49 INFO Received uploads request method=POST path=/27352026/11/12 00:09:49 INFO Completed upload id=1 objects=9 duration=6.114607ms27362026/11/12 00:09:49 INFO Received push request method=POST path=/27372026/11/12 00:09:49 INFO Received push request method=POST path=/27382026/11/12 00:09:49 INFO Received complete push request method=POST path=/27392026/11/12 00:09:49 INFO Completed push id=5 objects=3 duration=2.422449ms27402026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27412026/11/12 00:09:49 INFO Received uploads request method=POST path=/27422026/11/12 00:09:49 INFO Received complete push request method=POST path=/27432026/11/12 00:09:49 INFO Completed push id=1 objects=2 duration=2.833189ms27442026/11/12 00:09:49 INFO Completed upload id=1 objects=15 duration=6.679247ms27452026/11/12 00:09:49 INFO Received push request method=POST path=/27462026/11/12 00:09:49 INFO Received push request method=POST path=/27472026/11/12 00:09:49 INFO Received uploads request method=POST path=/27482026/11/12 00:09:49 INFO Received uploads request method=POST path=/27492026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27502026/11/12 00:09:49 INFO Completed upload id=9 objects=7 duration=2.493269ms27512026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027522026/11/12 00:09:49 INFO Received push request method=POST path=/27532026/11/12 00:09:49 INFO Received uploads request method=POST path=/27542026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027552026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27562026/11/12 00:09:49 INFO Vacuumed table table=pending_objects27572026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads27582026/11/12 00:09:49 INFO Received uploads request method=POST path=/27592026/11/12 00:09:49 INFO Vacuumed table table=closures27602026/11/12 00:09:49 INFO Vacuumed table table=objects27612026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27622026/11/12 00:09:49 INFO Received complete push request method=POST path=/27632026/11/12 00:09:49 INFO Received push request method=POST path=/27642026/11/12 00:09:49 INFO Completed push id=8 objects=13 duration=2.715448ms27652026/11/12 00:09:49 INFO Completed upload id=6 objects=24 duration=3.766978ms27662026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027672026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27682026/11/12 00:09:49 INFO Received uploads request method=POST path=/27692026/11/12 00:09:49 INFO Completed upload id=3 objects=5 duration=1.46575ms27702026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027712026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27722026/11/12 00:09:49 INFO Vacuumed table table=pending_objects27732026/11/12 00:09:49 INFO Received complete push request method=POST path=/27742026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads27752026/11/12 00:09:49 INFO Vacuumed table table=closures27762026/11/12 00:09:49 INFO Received complete upload request method=POST path=/27772026/11/12 00:09:49 INFO Received push request method=POST path=/27782026/11/12 00:09:49 INFO Vacuumed table table=objects27792026/11/12 00:09:49 INFO Completed push id=11 objects=8 duration=4.176078ms27802026/11/12 00:09:49 INFO Received uploads request method=POST path=/27812026/11/12 00:09:49 INFO Completed upload id=6 objects=7 duration=3.067019ms27822026/11/12 00:09:49 INFO Received push request method=POST path=/27832026/11/12 00:09:49 INFO Received push request method=POST path=/27842026/11/12 00:09:49 INFO Received push request method=POST path=/27852026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027862026/11/12 00:09:49 INFO Received complete push request method=POST path=/27872026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027882026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027892026/11/12 00:09:49 INFO Completed push id=5 objects=8 duration=3.777199ms27902026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=4 objects-deleted-after-grace-period=0 objects-failed-to-delete=027912026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027922026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027932026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027942026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27952026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=027962026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=027972026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27982026/11/12 00:09:49 INFO Vacuumed table table=pending_closures27992026/11/12 00:09:49 INFO Vacuumed table table=pending_objects28002026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028012026/11/12 00:09:49 INFO Vacuumed table table=pending_objects28022026/11/12 00:09:49 INFO Vacuumed table table=pending_objects28032026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads28042026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads28052026/11/12 00:09:49 INFO Vacuumed table table=pending_closures28062026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads28072026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=028082026/11/12 00:09:49 INFO Vacuumed table table=pending_objects28092026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period28102026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads28112026/11/12 00:09:49 INFO Vacuumed table table=closures28122026/11/12 00:09:49 INFO Vacuumed table table=closures28132026/11/12 00:09:49 INFO Vacuumed table table=closures28142026/11/12 00:09:49 INFO Vacuumed table table=objects28152026/11/12 00:09:49 INFO Vacuumed table table=pending_closures28162026/11/12 00:09:49 INFO Vacuumed table table=closures28172026/11/12 00:09:49 INFO Received push request method=POST path=/28182026/11/12 00:09:49 INFO Vacuumed table table=objects28192026/11/12 00:09:49 INFO Vacuumed table table=objects28202026/11/12 00:09:49 INFO Vacuumed table table=pending_objects28212026/11/12 00:09:49 INFO Vacuumed table table=objects28222026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads28232026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=028242026/11/12 00:09:49 INFO Vacuumed table table=closures28252026/11/12 00:09:49 INFO Vacuumed table table=objects28262026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=028272026/11/12 00:09:49 INFO Received push request method=POST path=/28282026/11/12 00:09:49 INFO Received push request method=POST path=/28292026/11/12 00:09:49 INFO Received complete upload request method=POST path=/28302026/11/12 00:09:49 INFO Received complete push request method=POST path=/28312026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period28322026/11/12 00:09:49 INFO Completed upload id=2 objects=2 duration=1.93089ms28332026/11/12 00:09:49 INFO Completed push id=8 objects=6 duration=2.409609ms28342026/11/12 00:09:49 INFO Received uploads request method=POST path=/28352026/11/12 00:09:49 INFO Received complete upload request method=POST path=/28362026/11/12 00:09:49 INFO Completed upload id=11 objects=6 duration=2.094219ms28372026/11/12 00:09:49 INFO Received uploads request method=POST path=/28382026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=028392026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=028402026/11/12 00:09:49 INFO Vacuumed table table=pending_closures28412026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28422026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28432026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28442026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28452026/11/12 00:09:49 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28462026/11/12 00:09:49 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28472026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28482026/11/12 00:09:49 INFO Received uploads request method=POST path=/28492026/11/12 00:09:49 INFO Vacuumed table table=pending_objects28502026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads28512026/11/12 00:09:49 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."28522026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28532026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28542026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=028552026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28562026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28572026/11/12 00:09:49 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28582026/11/12 00:09:49 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28592026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."28602026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."28612026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28622026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."28632026/11/12 00:09:49 INFO Vacuumed table table=closures28642026/11/12 00:09:49 INFO Received complete push request method=POST path=/28652026/11/12 00:09:49 INFO Received uploads request method=POST path=/28662026/11/12 00:09:49 INFO Vacuumed table table=objects28672026/11/12 00:09:49 INFO Received push request method=POST path=/28682026/11/12 00:09:49 INFO Received complete push request method=POST path=/28692026/11/12 00:09:49 INFO Completed push id=13 objects=6 duration=3.606608ms28702026/11/12 00:09:49 INFO Completed push id=6 objects=5 duration=2.29659ms28712026/11/12 00:09:49 INFO Received complete upload request method=POST path=/28722026/11/12 00:09:49 INFO Completed upload id=9 objects=3 duration=1.26308ms28732026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28742026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28752026/11/12 00:09:49 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."28762026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=028772026/11/12 00:09:49 INFO Received push request method=POST path=/28782026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=028792026/11/12 00:09:49 INFO Received uploads request method=POST path=/28802026/11/12 00:09:49 INFO Received complete upload request method=POST path=/28812026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period28822026/11/12 00:09:49 INFO Received push request method=POST path=/28832026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=028842026/11/12 00:09:49 INFO Completed upload id=10 objects=4 duration=2.802939ms28852026/11/12 00:09:49 INFO Received push request method=POST path=/28862026/11/12 00:09:49 INFO Received push request method=POST path=/28872026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=028882026/11/12 00:09:49 INFO Vacuumed table table=pending_closures28892026/11/12 00:09:49 INFO Received complete push request method=POST path=/28902026/11/12 00:09:49 INFO Vacuumed table table=pending_objects28912026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads28922026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."28932026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."28942026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."28952026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."28962026/11/12 00:09:49 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."28972026/11/12 00:09:49 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."28982026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."28992026/11/12 00:09:49 INFO Vacuumed table table=closures29002026/11/12 00:09:49 INFO Received push request method=POST path=/29012026/11/12 00:09:49 INFO Vacuumed table table=objects29022026/11/12 00:09:49 INFO Completed push id=2 objects=10 duration=3.687439ms29032026/11/12 00:09:49 INFO Received push request method=POST path=/29042026/11/12 00:09:49 INFO Received push request method=POST path=/29052026/11/12 00:09:49 INFO Received complete push request method=POST path=/29062026/11/12 00:09:49 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."29072026/11/12 00:09:49 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.narinfo error="Access Denied."29082026/11/12 00:09:49 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."29092026/11/12 00:09:49 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.narinfo error="Access Denied."29102026/11/12 00:09:49 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."29112026/11/12 00:09:49 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."29122026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.ls error="Access Denied."29132026/11/12 00:09:49 ERROR failed to remove object object=gggggggggggggggggggggggggggggggg.narinfo error="Access Denied."29142026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."29152026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."29162026/11/12 00:09:49 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."29172026/11/12 00:09:49 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."29182026/11/12 00:09:49 ERROR failed to remove object object=nar/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.nar.zst error="Access Denied."29192026/11/12 00:09:49 ERROR failed to remove object object=nar/dddddddddddddddddddddddddddddddddddddddddddddddddddd.nar.zst error="Access Denied."29202026/11/12 00:09:49 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."29212026/11/12 00:09:49 ERROR failed to remove object object=nar/gggggggggggggggggggggggggggggggggggggggggggggggggggg.nar.zst error="Access Denied."29222026/11/12 00:09:49 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."29232026/11/12 00:09:49 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."29242026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29252026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."29262026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."29272026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29282026/11/12 00:09:49 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."29292026/11/12 00:09:49 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."29302026/11/12 00:09:49 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."29312026/11/12 00:09:49 INFO Completed push id=2 objects=25 duration=3.381289ms29322026/11/12 00:09:49 INFO Received complete push request method=POST path=/29332026/11/12 00:09:49 INFO Received push request method=POST path=/29342026/11/12 00:09:49 INFO Received push request method=POST path=/29352026/11/12 00:09:49 INFO Completed push id=8 objects=2 duration=2.030299ms29362026/11/12 00:09:49 INFO Received push request method=POST path=/29372026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=029382026/11/12 00:09:49 INFO Received uploads request method=POST path=/29392026/11/12 00:09:49 INFO Received uploads request method=POST path=/29402026/11/12 00:09:49 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29412026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."29422026/11/12 00:09:49 INFO Received push request method=POST path=/29432026/11/12 00:09:49 INFO Received uploads request method=POST path=/29442026/11/12 00:09:49 INFO Received push request method=POST path=/29452026/11/12 00:09:49 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."29462026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."29472026/11/12 00:09:49 INFO Received complete push request method=POST path=/29482026/11/12 00:09:49 INFO Received push request method=POST path=/29492026/11/12 00:09:49 INFO Received push request method=POST path=/29502026/11/12 00:09:49 INFO Completed push id=9 objects=5 duration=2.477019ms29512026/11/12 00:09:49 INFO Received uploads request method=POST path=/29522026/11/12 00:09:49 INFO Received uploads request method=POST path=/29532026/11/12 00:09:49 INFO Received push request method=POST path=/29542026/11/12 00:09:49 INFO Received complete push request method=POST path=/29552026/11/12 00:09:49 INFO Completed push id=10 objects=2 duration=2.354099ms29562026/11/12 00:09:49 INFO Received push request method=POST path=/29572026/11/12 00:09:49 INFO Received complete upload request method=POST path=/29582026/11/12 00:09:49 INFO Completed upload id=3 objects=24 duration=1.87891ms29592026/11/12 00:09:49 INFO Received push request method=POST path=/29602026/11/12 00:09:49 INFO Received uploads request method=POST path=/29612026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=029622026/11/12 00:09:49 INFO Received complete push request method=POST path=/29632026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period29642026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029652026/11/12 00:09:49 INFO Completed push id=17 objects=6 duration=3.269978ms29662026/11/12 00:09:49 INFO Received uploads request method=POST path=/29672026/11/12 00:09:49 INFO Vacuumed table table=pending_closures29682026/11/12 00:09:49 INFO Received complete upload request method=POST path=/29692026/11/12 00:09:49 INFO Received complete push request method=POST path=/29702026/11/12 00:09:49 INFO Vacuumed table table=pending_objects29712026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads29722026/11/12 00:09:49 INFO Vacuumed table table=closures29732026/11/12 00:09:49 INFO Completed push id=8 objects=7 duration=2.054329ms29742026/11/12 00:09:49 INFO Completed upload id=3 objects=8 duration=2.615159ms29752026/11/12 00:09:49 INFO Vacuumed table table=objects29762026/11/12 00:09:49 INFO Received push request method=POST path=/29772026/11/12 00:09:49 INFO Received push request method=POST path=/29782026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=029792026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=029802026/11/12 00:09:49 INFO Received uploads request method=POST path=/29812026/11/12 00:09:49 INFO Vacuumed table table=pending_closures29822026/11/12 00:09:49 INFO Received uploads request method=POST path=/29832026/11/12 00:09:49 INFO Received complete upload request method=POST path=/29842026/11/12 00:09:49 INFO Vacuumed table table=pending_objects29852026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads29862026/11/12 00:09:49 INFO Received push request method=POST path=/29872026/11/12 00:09:49 INFO Completed upload id=12 objects=21 duration=1.497679ms29882026/11/12 00:09:49 INFO Vacuumed table table=closures29892026/11/12 00:09:49 INFO Received uploads request method=POST path=/29902026/11/12 00:09:49 INFO Received push request method=POST path=/29912026/11/12 00:09:49 INFO Vacuumed table table=objects29922026/11/12 00:09:49 INFO Received uploads request method=POST path=/29932026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=029942026/11/12 00:09:49 INFO Received uploads request method=POST path=/29952026/11/12 00:09:49 INFO Received push request method=POST path=/29962026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."29972026/11/12 00:09:49 INFO Received uploads request method=POST path=/29982026/11/12 00:09:49 INFO Received push request method=POST path=/29992026/11/12 00:09:49 INFO Received complete push request method=POST path=/30002026/11/12 00:09:49 INFO Received push request method=POST path=/30012026/11/12 00:09:49 INFO Received complete upload request method=POST path=/30022026/11/12 00:09:49 INFO Completed push id=12 objects=3 duration=2.428169ms30032026/11/12 00:09:49 INFO Received push request method=POST path=/30042026/11/12 00:09:49 INFO Completed upload id=8 objects=3 duration=2.382069ms30052026/11/12 00:09:49 INFO Received complete push request method=POST path=/30062026/11/12 00:09:49 INFO Completed push id=4 objects=7 duration=2.36246ms30072026/11/12 00:09:49 INFO Received complete upload request method=POST path=/30082026/11/12 00:09:49 INFO Received uploads request method=POST path=/30092026/11/12 00:09:49 INFO Completed upload id=11 objects=14 duration=1.908329ms30102026/11/12 00:09:49 INFO Received push request method=POST path=/30112026/11/12 00:09:49 INFO Received uploads request method=POST path=/30122026/11/12 00:09:49 INFO Received push request method=POST path=/30132026/11/12 00:09:49 INFO Received uploads request method=POST path=/30142026/11/12 00:09:49 INFO Received complete upload request method=POST path=/30152026/11/12 00:09:49 INFO Received uploads request method=POST path=/30162026/11/12 00:09:49 INFO Completed upload id=9 objects=16 duration=1.410959ms30172026/11/12 00:09:49 INFO Received uploads request method=POST path=/30182026/11/12 00:09:49 INFO Received push request method=POST path=/30192026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=030202026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period30212026/11/12 00:09:49 INFO Received push request method=POST path=/30222026/11/12 00:09:49 INFO Received complete push request method=POST path=/30232026/11/12 00:09:49 INFO Completed push id=13 objects=23 duration=2.077819ms30242026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=030252026/11/12 00:09:49 INFO Received push request method=POST path=/30262026/11/12 00:09:49 INFO Received push request method=POST path=/30272026/11/12 00:09:49 INFO Received complete push request method=POST path=/30282026/11/12 00:09:49 INFO Received uploads request method=POST path=/30292026/11/12 00:09:49 INFO Completed push id=7 objects=20 duration=2.760788ms30302026/11/12 00:09:49 INFO Received uploads request method=POST path=/30312026/11/12 00:09:49 INFO Received push request method=POST path=/30322026/11/12 00:09:49 INFO Received uploads request method=POST path=/30332026/11/12 00:09:49 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."30342026/11/12 00:09:49 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."30352026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30362026/11/12 00:09:49 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."30372026/11/12 00:09:49 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30382026/11/12 00:09:49 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30392026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30402026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30412026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."30422026/11/12 00:09:49 INFO Received push request method=POST path=/30432026/11/12 00:09:49 INFO Received push request method=POST path=/30442026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=030452026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=030462026/11/12 00:09:49 INFO Received push request method=POST path=/30472026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=030482026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period30492026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030502026/11/12 00:09:49 INFO Received uploads request method=POST path=/30512026/11/12 00:09:49 INFO Vacuumed table table=pending_closures30522026/11/12 00:09:49 INFO Vacuumed table table=pending_objects30532026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads30542026/11/12 00:09:49 INFO Received push request method=POST path=/30552026/11/12 00:09:49 INFO Vacuumed table table=closures30562026/11/12 00:09:49 INFO Received push request method=POST path=/30572026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=030582026/11/12 00:09:49 INFO Vacuumed table table=objects30592026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=030602026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=030612026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=9 objects-failed-to-delete=030622026/11/12 00:09:49 INFO Vacuumed table table=pending_closures30632026/11/12 00:09:49 INFO Vacuumed table table=pending_closures30642026/11/12 00:09:49 INFO Vacuumed table table=pending_closures30652026/11/12 00:09:49 INFO Vacuumed table table=pending_objects30662026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads30672026/11/12 00:09:49 INFO Vacuumed table table=pending_objects30682026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads30692026/11/12 00:09:49 INFO Vacuumed table table=pending_objects30702026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads30712026/11/12 00:09:49 INFO Vacuumed table table=closures30722026/11/12 00:09:49 INFO Received uploads request method=POST path=/30732026/11/12 00:09:49 INFO Vacuumed table table=closures30742026/11/12 00:09:49 INFO Vacuumed table table=objects30752026/11/12 00:09:49 INFO Vacuumed table table=closures30762026/11/12 00:09:49 INFO Received complete push request method=POST path=/30772026/11/12 00:09:49 INFO Vacuumed table table=objects30782026/11/12 00:09:49 INFO Received uploads request method=POST path=/30792026/11/12 00:09:49 INFO Vacuumed table table=objects30802026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=030812026/11/12 00:09:49 INFO Completed push id=6 objects=6 duration=2.315859ms30822026/11/12 00:09:49 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."30832026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."30842026/11/12 00:09:49 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."30852026/11/12 00:09:49 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.narinfo error="Access Denied."30862026/11/12 00:09:49 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."30872026/11/12 00:09:49 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.narinfo error="Access Denied."30882026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."30892026/11/12 00:09:49 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.narinfo error="Access Denied."30902026/11/12 00:09:49 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."30912026/11/12 00:09:49 ERROR failed to remove object object=nar/cccccccccccccccccccccccccccccccccccccccccccccccccccc.nar.zst error="Access Denied."30922026/11/12 00:09:49 ERROR failed to remove object object=nar/ffffffffffffffffffffffffffffffffffffffffffffffffffff.nar.zst error="Access Denied."30932026/11/12 00:09:49 ERROR failed to remove object object=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.nar.zst error="Access Denied."30942026/11/12 00:09:49 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."30952026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:aaaaaaaa!out.doi error="Access Denied."30962026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."30972026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:dddddddd!out.doi error="Access Denied."30982026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."30992026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:gggggggg!out.doi error="Access Denied."31002026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."31012026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."31022026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031032026/11/12 00:09:49 INFO Vacuumed table table=pending_closures31042026/11/12 00:09:49 INFO Vacuumed table table=pending_objects31052026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads31062026/11/12 00:09:49 INFO Vacuumed table table=closures31072026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=031082026/11/12 00:09:49 INFO Vacuumed table table=objects31092026/11/12 00:09:49 INFO Received complete push request method=POST path=/31102026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period31112026/11/12 00:09:49 INFO Received push request method=POST path=/31122026/11/12 00:09:49 INFO Completed push id=10 objects=3 duration=2.033939ms31132026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031142026/11/12 00:09:49 INFO Vacuumed table table=pending_closures31152026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=031162026/11/12 00:09:49 INFO Vacuumed table table=pending_objects31172026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads31182026/11/12 00:09:49 INFO Vacuumed table table=closures31192026/11/12 00:09:49 INFO Received complete push request method=POST path=/31202026/11/12 00:09:49 INFO Vacuumed table table=objects31212026/11/12 00:09:49 INFO Received push request method=POST path=/31222026/11/12 00:09:49 INFO Received uploads request method=POST path=/31232026/11/12 00:09:49 INFO Completed push id=4 objects=4 duration=3.328269ms31242026/11/12 00:09:49 INFO Received push request method=POST path=/31252026/11/12 00:09:49 INFO Received complete push request method=POST path=/31262026/11/12 00:09:49 INFO Received uploads request method=POST path=/31272026/11/12 00:09:49 INFO Completed push id=7 objects=7 duration=1.983709ms31282026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=031292026/11/12 00:09:49 INFO Vacuumed table table=pending_closures31302026/11/12 00:09:49 INFO Received complete upload request method=POST path=/31312026/11/12 00:09:49 INFO Received complete upload request method=POST path=/31322026/11/12 00:09:49 INFO Received uploads request method=POST path=/31332026/11/12 00:09:49 INFO Vacuumed table table=pending_objects31342026/11/12 00:09:49 INFO Completed upload id=16 objects=23 duration=1.418999ms31352026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads31362026/11/12 00:09:49 INFO Completed upload id=10 objects=23 duration=2.653239ms31372026/11/12 00:09:49 INFO Vacuumed table table=closures31382026/11/12 00:09:49 INFO Received uploads request method=POST path=/31392026/11/12 00:09:49 INFO Vacuumed table table=objects31402026/11/12 00:09:49 INFO Received uploads request method=POST path=/31412026/11/12 00:09:49 INFO Received complete push request method=POST path=/31422026/11/12 00:09:49 INFO Completed push id=5 objects=3 duration=3.894479ms31432026/11/12 00:09:49 INFO Received push request method=POST path=/31442026/11/12 00:09:49 INFO Received push request method=POST path=/31452026/11/12 00:09:49 INFO Received uploads request method=POST path=/31462026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=031472026/11/12 00:09:49 INFO Received complete push request method=POST path=/31482026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=031492026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period31502026/11/12 00:09:49 INFO Received complete upload request method=POST path=/31512026/11/12 00:09:49 INFO Completed upload id=15 objects=6 duration=1.10876ms31522026/11/12 00:09:49 INFO Completed push id=6 objects=6 duration=2.205089ms31532026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period31542026/11/12 00:09:49 INFO Received push request method=POST path=/31552026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031562026/11/12 00:09:49 INFO Vacuumed table table=pending_closures31572026/11/12 00:09:49 INFO Received push request method=POST path=/31582026/11/12 00:09:49 INFO Vacuumed table table=pending_objects31592026/11/12 00:09:49 INFO Received uploads request method=POST path=/31602026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads31612026/11/12 00:09:49 INFO Vacuumed table table=closures31622026/11/12 00:09:49 INFO Vacuumed table table=objects31632026/11/12 00:09:49 INFO Received push request method=POST path=/31642026/11/12 00:09:49 INFO Received complete upload request method=POST path=/31652026/11/12 00:09:49 INFO Received uploads request method=POST path=/31662026/11/12 00:09:49 INFO Completed upload id=10 objects=6 duration=2.018439ms31672026/11/12 00:09:49 INFO Received uploads request method=POST path=/31682026/11/12 00:09:49 INFO Received complete push request method=POST path=/31692026/11/12 00:09:49 INFO Received uploads request method=POST path=/31702026/11/12 00:09:49 INFO Completed push id=10 objects=15 duration=1.931519ms31712026/11/12 00:09:49 INFO Received push request method=POST path=/31722026/11/12 00:09:49 INFO Received complete push request method=POST path=/31732026/11/12 00:09:49 INFO Received complete push request method=POST path=/31742026/11/12 00:09:49 INFO Completed push id=15 objects=24 duration=3.926288ms31752026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=16 objects-failed-to-delete=031762026/11/12 00:09:49 INFO Completed push id=5 objects=9 duration=5.602607ms31772026/11/12 00:09:49 INFO Vacuumed table table=pending_closures31782026/11/12 00:09:49 INFO Received push request method=POST path=/31792026/11/12 00:09:49 INFO Received push request method=POST path=/31802026/11/12 00:09:49 INFO Received complete push request method=POST path=/31812026/11/12 00:09:49 INFO Vacuumed table table=pending_objects31822026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads31832026/11/12 00:09:49 INFO Vacuumed table table=closures31842026/11/12 00:09:49 INFO Vacuumed table table=objects31852026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=031862026/11/12 00:09:49 INFO Completed push id=14 objects=3 duration=3.044148ms31872026/11/12 00:09:49 INFO Received complete upload request method=POST path=/31882026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period31892026/11/12 00:09:49 INFO Completed upload id=18 objects=11 duration=1.732039ms31902026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=031912026/11/12 00:09:49 INFO Vacuumed table table=pending_closures31922026/11/12 00:09:49 INFO Vacuumed table table=pending_objects31932026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads31942026/11/12 00:09:49 INFO Vacuumed table table=closures31952026/11/12 00:09:49 INFO Vacuumed table table=objects31962026/11/12 00:09:49 INFO Received complete upload request method=POST path=/31972026/11/12 00:09:49 INFO Received push request method=POST path=/31982026/11/12 00:09:49 INFO Completed upload id=11 objects=6 duration=1.625339ms31992026/11/12 00:09:49 INFO Received push request method=POST path=/32002026/11/12 00:09:49 INFO Received push request method=POST path=/32012026/11/12 00:09:49 INFO Received uploads request method=POST path=/32022026/11/12 00:09:49 INFO Received uploads request method=POST path=/32032026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032042026/11/12 00:09:49 INFO Received push request method=POST path=/32052026/11/12 00:09:49 INFO Received complete upload request method=POST path=/32062026/11/12 00:09:49 INFO Received uploads request method=POST path=/32072026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032082026/11/12 00:09:49 INFO Completed upload id=17 objects=7 duration=1.263649ms32092026/11/12 00:09:49 INFO Vacuumed table table=pending_closures32102026/11/12 00:09:49 INFO Vacuumed table table=pending_objects32112026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads32122026/11/12 00:09:49 INFO Vacuumed table table=closures32132026/11/12 00:09:49 INFO Received push request method=POST path=/32142026/11/12 00:09:49 INFO Vacuumed table table=objects32152026/11/12 00:09:49 INFO Received push request method=POST path=/32162026/11/12 00:09:49 INFO Received uploads request method=POST path=/32172026/11/12 00:09:49 INFO Received complete push request method=POST path=/32182026/11/12 00:09:49 INFO Completed push id=8 objects=6 duration=1.805059ms32192026/11/12 00:09:49 INFO Received complete push request method=POST path=/32202026/11/12 00:09:49 INFO Received uploads request method=POST path=/32212026/11/12 00:09:49 INFO Received push request method=POST path=/32222026/11/12 00:09:49 INFO Received complete push request method=POST path=/32232026/11/12 00:09:49 INFO Received uploads request method=POST path=/32242026/11/12 00:09:49 INFO Completed push id=7 objects=27 duration=2.582879ms32252026/11/12 00:09:49 INFO Completed push id=5 objects=6 duration=2.269039ms32262026/11/12 00:09:49 INFO Received uploads request method=POST path=/32272026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032282026/11/12 00:09:49 INFO Received push request method=POST path=/32292026/11/12 00:09:49 INFO Received uploads request method=POST path=/32302026/11/12 00:09:49 INFO Received push request method=POST path=/32312026/11/12 00:09:49 ERROR failed to remove object object=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.ls error="Access Denied."32322026/11/12 00:09:49 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."32332026/11/12 00:09:49 INFO Received complete push request method=POST path=/32342026/11/12 00:09:49 INFO Completed push id=8 objects=6 duration=1.616519ms32352026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032362026/11/12 00:09:49 INFO Received complete upload request method=POST path=/32372026/11/12 00:09:49 INFO Received push request method=POST path=/32382026/11/12 00:09:49 INFO Received uploads request method=POST path=/32392026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period32402026/11/12 00:09:49 INFO Completed upload id=15 objects=4 duration=2.900109ms32412026/11/12 00:09:49 INFO Received complete push request method=POST path=/32422026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032432026/11/12 00:09:49 INFO Vacuumed table table=pending_closures32442026/11/12 00:09:49 INFO Completed push id=12 objects=15 duration=3.020558ms32452026/11/12 00:09:49 INFO Vacuumed table table=pending_objects32462026/11/12 00:09:49 INFO Received complete push request method=POST path=/32472026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads32482026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032492026/11/12 00:09:49 INFO Vacuumed table table=closures32502026/11/12 00:09:49 INFO Completed push id=13 objects=12 duration=2.106589ms32512026/11/12 00:09:49 INFO Vacuumed table table=objects32522026/11/12 00:09:49 INFO Received complete upload request method=POST path=/32532026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=032542026/11/12 00:09:49 INFO Received push request method=POST path=/32552026/11/12 00:09:49 INFO Vacuumed table table=pending_closures32562026/11/12 00:09:49 INFO Completed upload id=3 objects=9 duration=1.952249ms32572026/11/12 00:09:49 INFO Vacuumed table table=pending_objects32582026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads32592026/11/12 00:09:49 INFO Vacuumed table table=closures32602026/11/12 00:09:49 INFO Vacuumed table table=objects32612026/11/12 00:09:49 INFO Received uploads request method=POST path=/32622026/11/12 00:09:49 INFO Received uploads request method=POST path=/32632026/11/12 00:09:49 INFO Received uploads request method=POST path=/32642026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032652026/11/12 00:09:49 INFO Received complete push request method=POST path=/32662026/11/12 00:09:49 INFO Received push request method=POST path=/32672026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032682026/11/12 00:09:49 INFO Completed push id=12 objects=7 duration=1.48864ms32692026/11/12 00:09:49 INFO Received push request method=POST path=/32702026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period32712026/11/12 00:09:49 WARN Force mode enabled - objects will be deleted immediately without grace period32722026/11/12 00:09:49 INFO Received push request method=POST path=/32732026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032742026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=032752026/11/12 00:09:49 INFO Received push request method=POST path=/32762026/11/12 00:09:49 INFO Vacuumed table table=pending_closures32772026/11/12 00:09:49 INFO Vacuumed table table=pending_objects32782026/11/12 00:09:49 INFO Vacuumed table table=multipart_uploads32792026/11/12 00:09:49 INFO Vacuumed table table=closures32802026/11/12 00:09:49 INFO Vacuumed table table=objects32812026/11/12 00:09:49 INFO Received push request method=POST path=/32822026/11/12 00:09:49 INFO Received uploads request method=POST path=/32832026/11/12 00:09:49 INFO Received push request method=POST path=/32842026/11/12 00:09:49 INFO Received complete upload request method=POST path=/32852026/11/12 00:09:49 INFO Completed upload id=14 objects=6 duration=1.348849ms32862026/11/12 00:09:49 INFO Aborted multipart uploads count=0 kept=032872026/11/12 00:09:49 INFO Received push request method=POST path=/32882026/11/12 00:09:49 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=032892026/11/12 00:09:50 INFO Received uploads request method=POST path=/32902026/11/12 00:09:50 INFO Vacuumed table table=pending_closures32912026/11/12 00:09:50 INFO Vacuumed table table=pending_objects32922026/11/12 00:09:50 INFO Received uploads request method=POST path=/32932026/11/12 00:09:50 INFO Received complete upload request method=POST path=/32942026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads32952026/11/12 00:09:50 INFO Vacuumed table table=closures32962026/11/12 00:09:50 INFO Completed upload id=6 objects=6 duration=1.517349ms32972026/11/12 00:09:50 INFO Vacuumed table table=objects32982026/11/12 00:09:50 INFO Received push request method=POST path=/32992026/11/12 00:09:50 INFO Received push request method=POST path=/33002026/11/12 00:09:50 INFO Received push request method=POST path=/33012026/11/12 00:09:50 INFO Received uploads request method=POST path=/33022026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."33032026/11/12 00:09:50 INFO Received push request method=POST path=/33042026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=6 objects-failed-to-delete=033052026/11/12 00:09:50 INFO Received push request method=POST path=/33062026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33072026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33082026/11/12 00:09:50 INFO Received complete push request method=POST path=/33092026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033102026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33112026/11/12 00:09:50 INFO Received complete upload request method=POST path=/33122026/11/12 00:09:50 INFO Vacuumed table table=closures33132026/11/12 00:09:50 INFO Completed upload id=16 objects=3 duration=1.22952ms33142026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033152026/11/12 00:09:50 INFO Completed push id=5 objects=6 duration=2.368579ms33162026/11/12 00:09:50 INFO Vacuumed table table=objects33172026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033182026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period33192026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period33202026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033212026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period33222026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33232026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033242026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33252026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033262026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33272026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33282026/11/12 00:09:50 INFO Vacuumed table table=closures33292026/11/12 00:09:50 INFO Received push request method=POST path=/33302026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33312026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33322026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33332026/11/12 00:09:50 INFO Vacuumed table table=objects33342026/11/12 00:09:50 INFO Vacuumed table table=closures33352026/11/12 00:09:50 INFO Received push request method=POST path=/33362026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33372026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33382026/11/12 00:09:50 INFO Vacuumed table table=objects33392026/11/12 00:09:50 INFO Received uploads request method=POST path=/33402026/11/12 00:09:50 INFO Vacuumed table table=closures33412026/11/12 00:09:50 INFO Vacuumed table table=objects33422026/11/12 00:09:50 INFO Received push request method=POST path=/33432026/11/12 00:09:50 INFO Received complete upload request method=POST path=/33442026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033452026/11/12 00:09:50 INFO Completed upload id=7 objects=3 duration=1.10437ms33462026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033472026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033482026/11/12 00:09:50 INFO Received uploads request method=POST path=/33492026/11/12 00:09:50 INFO Received uploads request method=POST path=/33502026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period33512026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33522026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033532026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33542026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33552026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period33562026/11/12 00:09:50 INFO Vacuumed table table=closures33572026/11/12 00:09:50 INFO Vacuumed table table=objects33582026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033592026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33602026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33612026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33622026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033632026/11/12 00:09:50 INFO Vacuumed table table=closures33642026/11/12 00:09:50 INFO Received push request method=POST path=/33652026/11/12 00:09:50 INFO Vacuumed table table=objects33662026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033672026/11/12 00:09:50 INFO Received complete push request method=POST path=/33682026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33692026/11/12 00:09:50 INFO Received uploads request method=POST path=/33702026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33712026/11/12 00:09:50 INFO Completed push id=15 objects=4 duration=1.79077ms33722026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33732026/11/12 00:09:50 INFO Vacuumed table table=closures33742026/11/12 00:09:50 INFO Received complete upload request method=POST path=/33752026/11/12 00:09:50 INFO Vacuumed table table=objects33762026/11/12 00:09:50 INFO Completed upload id=10 objects=17 duration=1.591689ms33772026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033782026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=033792026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period33802026/11/12 00:09:50 INFO Received uploads request method=POST path=/33812026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period33822026/11/12 00:09:50 INFO Received complete push request method=POST path=/33832026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=033842026/11/12 00:09:50 INFO Vacuumed table table=pending_closures33852026/11/12 00:09:50 INFO Completed push id=18 objects=23 duration=1.923689ms33862026/11/12 00:09:50 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."33872026/11/12 00:09:50 INFO Vacuumed table table=pending_objects33882026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads33892026/11/12 00:09:50 INFO Received push request method=POST path=/33902026/11/12 00:09:50 INFO Vacuumed table table=closures33912026/11/12 00:09:50 INFO Vacuumed table table=objects33922026/11/12 00:09:50 INFO Received uploads request method=POST path=/33932026/11/12 00:09:50 INFO Received uploads request method=POST path=/33942026/11/12 00:09:50 INFO Received push request method=POST path=/33952026/11/12 00:09:50 INFO Received complete push request method=POST path=/33962026/11/12 00:09:50 INFO Completed push id=19 objects=6 duration=1.864279ms33972026/11/12 00:09:50 INFO Received complete push request method=POST path=/33982026/11/12 00:09:50 INFO Completed push id=9 objects=7 duration=3.998828ms33992026/11/12 00:09:50 INFO Received uploads request method=POST path=/34002026/11/12 00:09:50 INFO Received uploads request method=POST path=/34012026/11/12 00:09:50 INFO Received push request method=POST path=/34022026/11/12 00:09:50 INFO Received uploads request method=POST path=/34032026/11/12 00:09:50 INFO Received uploads request method=POST path=/34042026/11/12 00:09:50 INFO Received complete push request method=POST path=/34052026/11/12 00:09:50 INFO Completed push id=6 objects=26 duration=2.404439ms34062026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=034072026/11/12 00:09:50 INFO Received push request method=POST path=/34082026/11/12 00:09:50 INFO Received complete upload request method=POST path=/34092026/11/12 00:09:50 INFO Completed upload id=17 objects=6 duration=2.488199ms34102026/11/12 00:09:50 INFO Received complete push request method=POST path=/34112026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=034122026/11/12 00:09:50 INFO Vacuumed table table=pending_closures34132026/11/12 00:09:50 INFO Vacuumed table table=pending_objects34142026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads34152026/11/12 00:09:50 INFO Completed push id=11 objects=11 duration=4.369378ms34162026/11/12 00:09:50 INFO Vacuumed table table=closures34172026/11/12 00:09:50 INFO Received uploads request method=POST path=/34182026/11/12 00:09:50 INFO Vacuumed table table=objects34192026/11/12 00:09:50 INFO Received complete upload request method=POST path=/34202026/11/12 00:09:50 INFO Completed upload id=18 objects=4 duration=1.970549ms34212026/11/12 00:09:50 INFO Received uploads request method=POST path=/34222026/11/12 00:09:50 INFO Received complete upload request method=POST path=/34232026/11/12 00:09:50 INFO Completed upload id=14 objects=8 duration=2.421689ms34242026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=034252026/11/12 00:09:50 INFO Received complete upload request method=POST path=/34262026/11/12 00:09:50 INFO Received complete push request method=POST path=/34272026/11/12 00:09:50 INFO Received push request method=POST path=/34282026/11/12 00:09:50 INFO Completed upload id=10 objects=7 duration=1.767779ms34292026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period34302026/11/12 00:09:50 INFO Completed push id=11 objects=9 duration=1.433169ms34312026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=034322026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034332026/11/12 00:09:50 INFO Received push request method=POST path=/34342026/11/12 00:09:50 INFO Vacuumed table table=pending_closures34352026/11/12 00:09:50 INFO Vacuumed table table=pending_closures34362026/11/12 00:09:50 INFO Vacuumed table table=pending_objects34372026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads34382026/11/12 00:09:50 INFO Vacuumed table table=pending_objects34392026/11/12 00:09:50 INFO Vacuumed table table=closures34402026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads34412026/11/12 00:09:50 INFO Vacuumed table table=objects34422026/11/12 00:09:50 INFO Vacuumed table table=closures34432026/11/12 00:09:50 INFO Vacuumed table table=objects34442026/11/12 00:09:50 INFO Received complete push request method=POST path=/34452026/11/12 00:09:50 INFO Received uploads request method=POST path=/34462026/11/12 00:09:50 INFO Completed push id=14 objects=3 duration=1.99723ms34472026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=034482026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=034492026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period34502026/11/12 00:09:50 INFO Received uploads request method=POST path=/34512026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period34522026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034532026/11/12 00:09:50 INFO Received push request method=POST path=/34542026/11/12 00:09:50 INFO Received complete push request method=POST path=/34552026/11/12 00:09:50 INFO Vacuumed table table=pending_closures34562026/11/12 00:09:50 INFO Received push request method=POST path=/34572026/11/12 00:09:50 INFO Vacuumed table table=pending_objects34582026/11/12 00:09:50 INFO Completed push id=14 objects=6 duration=2.065819ms34592026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads34602026/11/12 00:09:50 INFO Vacuumed table table=closures34612026/11/12 00:09:50 INFO Vacuumed table table=objects34622026/11/12 00:09:50 INFO Received push request method=POST path=/34632026/11/12 00:09:50 INFO Received push request method=POST path=/34642026/11/12 00:09:50 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34652026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34662026/11/12 00:09:50 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34672026/11/12 00:09:50 INFO Received push request method=POST path=/34682026/11/12 00:09:50 INFO Received complete push request method=POST path=/34692026/11/12 00:09:50 INFO Completed push id=17 objects=6 duration=2.127049ms34702026/11/12 00:09:50 INFO Received complete push request method=POST path=/34712026/11/12 00:09:50 INFO Received uploads request method=POST path=/34722026/11/12 00:09:50 INFO Completed push id=4 objects=6 duration=2.766549ms34732026/11/12 00:09:50 INFO Received push request method=POST path=/34742026/11/12 00:09:50 INFO Received push request method=POST path=/34752026/11/12 00:09:50 INFO Received uploads request method=POST path=/34762026/11/12 00:09:50 INFO Received complete push request method=POST path=/34772026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=034782026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period34792026/11/12 00:09:50 INFO Received uploads request method=POST path=/34802026/11/12 00:09:50 INFO Completed push id=3 objects=21 duration=5.122447ms34812026/11/12 00:09:50 INFO Received uploads request method=POST path=/34822026/11/12 00:09:50 INFO Received uploads request method=POST path=/34832026/11/12 00:09:50 INFO Received uploads request method=POST path=/34842026/11/12 00:09:50 ERROR failed to remove object object=dddddddddddddddddddddddddddddddd.ls error="Access Denied."34852026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:bbbbbbbb!out.doi error="Access Denied."34862026/11/12 00:09:50 ERROR failed to remove object object=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.ls error="Access Denied."34872026/11/12 00:09:50 INFO Received uploads request method=POST path=/34882026/11/12 00:09:50 INFO Received push request method=POST path=/34892026/11/12 00:09:50 INFO Received complete upload request method=POST path=/34902026/11/12 00:09:50 INFO Completed upload id=15 objects=6 duration=1.136999ms34912026/11/12 00:09:50 INFO Received uploads request method=POST path=/34922026/11/12 00:09:50 INFO Received uploads request method=POST path=/34932026/11/12 00:09:50 INFO Received push request method=POST path=/34942026/11/12 00:09:50 INFO Received complete upload request method=POST path=/34952026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=034962026/11/12 00:09:50 INFO Completed upload id=20 objects=4 duration=1.059539ms34972026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period34982026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=034992026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35002026/11/12 00:09:50 INFO Received push request method=POST path=/35012026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35022026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35032026/11/12 00:09:50 INFO Vacuumed table table=closures35042026/11/12 00:09:50 INFO Vacuumed table table=objects35052026/11/12 00:09:50 INFO Received complete upload request method=POST path=/35062026/11/12 00:09:50 INFO Completed upload id=18 objects=4 duration=1.56602ms35072026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035082026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035092026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035102026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035112026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35122026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35132026/11/12 00:09:50 INFO Received push request method=POST path=/35142026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35152026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=1 objects-deleted-after-grace-period=0 objects-failed-to-delete=035162026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period35172026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035182026/11/12 00:09:50 INFO Received complete upload request method=POST path=/35192026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035202026/11/12 00:09:50 INFO Vacuumed table table=closures35212026/11/12 00:09:50 INFO Received push request method=POST path=/35222026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35232026/11/12 00:09:50 INFO Vacuumed table table=objects35242026/11/12 00:09:50 INFO Completed upload id=9 objects=2 duration=1.805049ms35252026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35262026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035272026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period35282026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35292026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period35302026/11/12 00:09:50 INFO Vacuumed table table=closures35312026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035322026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period35332026/11/12 00:09:50 INFO Vacuumed table table=objects35342026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035352026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035362026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35372026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period35382026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35392026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035402026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35412026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35422026/11/12 00:09:50 INFO Vacuumed table table=closures35432026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035442026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035452026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35462026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35472026/11/12 00:09:50 INFO Vacuumed table table=objects35482026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35492026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=035502026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period35512026/11/12 00:09:50 INFO Vacuumed table table=closures35522026/11/12 00:09:50 INFO Received push request method=POST path=/35532026/11/12 00:09:50 INFO Vacuumed table table=objects35542026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35552026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35562026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035572026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35582026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035592026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35602026/11/12 00:09:50 INFO Vacuumed table table=closures35612026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35622026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35632026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35642026/11/12 00:09:50 INFO Received complete push request method=POST path=/35652026/11/12 00:09:50 INFO Vacuumed table table=objects35662026/11/12 00:09:50 INFO Received push request method=POST path=/35672026/11/12 00:09:50 INFO Vacuumed table table=closures35682026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35692026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35702026/11/12 00:09:50 INFO Received complete push request method=POST path=/35712026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35722026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35732026/11/12 00:09:50 INFO Completed push id=8 objects=3 duration=2.075409ms35742026/11/12 00:09:50 INFO Vacuumed table table=objects35752026/11/12 00:09:50 INFO Vacuumed table table=closures35762026/11/12 00:09:50 INFO Vacuumed table table=closures35772026/11/12 00:09:50 INFO Completed push id=16 objects=5 duration=1.54816ms35782026/11/12 00:09:50 INFO Vacuumed table table=objects35792026/11/12 00:09:50 INFO Vacuumed table table=objects35802026/11/12 00:09:50 INFO Received uploads request method=POST path=/35812026/11/12 00:09:50 INFO Received uploads request method=POST path=/35822026/11/12 00:09:50 INFO Received complete push request method=POST path=/35832026/11/12 00:09:50 INFO Completed push id=15 objects=7 duration=1.37387ms35842026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=035852026/11/12 00:09:50 INFO Received uploads request method=POST path=/35862026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."35872026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=035882026/11/12 00:09:50 INFO Vacuumed table table=pending_closures35892026/11/12 00:09:50 INFO Received push request method=POST path=/35902026/11/12 00:09:50 INFO Vacuumed table table=pending_objects35912026/11/12 00:09:50 INFO Received uploads request method=POST path=/35922026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads35932026/11/12 00:09:50 INFO Vacuumed table table=closures35942026/11/12 00:09:50 INFO Vacuumed table table=objects35952026/11/12 00:09:50 INFO Received complete push request method=POST path=/35962026/11/12 00:09:50 INFO Received push request method=POST path=/35972026/11/12 00:09:50 INFO Completed push id=8 objects=6 duration=3.259849ms35982026/11/12 00:09:50 INFO Received complete push request method=POST path=/35992026/11/12 00:09:50 INFO Received uploads request method=POST path=/36002026/11/12 00:09:50 INFO Received push request method=POST path=/36012026/11/12 00:09:50 INFO Received push request method=POST path=/36022026/11/12 00:09:50 INFO Completed push id=19 objects=8 duration=3.466008ms36032026/11/12 00:09:50 INFO Received uploads request method=POST path=/36042026/11/12 00:09:50 INFO Received push request method=POST path=/36052026/11/12 00:09:50 INFO Received uploads request method=POST path=/36062026/11/12 00:09:50 INFO Received complete push request method=POST path=/36072026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036082026/11/12 00:09:50 INFO Completed push id=16 objects=27 duration=2.370269ms36092026/11/12 00:09:50 INFO Received uploads request method=POST path=/36102026/11/12 00:09:50 INFO Received uploads request method=POST path=/36112026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036122026/11/12 00:09:50 INFO Received uploads request method=POST path=/36132026/11/12 00:09:50 INFO Received uploads request method=POST path=/36142026/11/12 00:09:50 INFO Vacuumed table table=pending_closures36152026/11/12 00:09:50 INFO Vacuumed table table=pending_objects36162026/11/12 00:09:50 INFO Received uploads request method=POST path=/36172026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads36182026/11/12 00:09:50 INFO Vacuumed table table=closures36192026/11/12 00:09:50 INFO Vacuumed table table=objects36202026/11/12 00:09:50 INFO Received complete upload request method=POST path=/36212026/11/12 00:09:50 INFO Completed upload id=13 objects=10 duration=1.19947ms36222026/11/12 00:09:50 INFO Received push request method=POST path=/36232026/11/12 00:09:50 INFO Received push request method=POST path=/36242026/11/12 00:09:50 INFO Received uploads request method=POST path=/36252026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036262026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036272026/11/12 00:09:50 INFO Received complete push request method=POST path=/36282026/11/12 00:09:50 INFO Received uploads request method=POST path=/36292026/11/12 00:09:50 INFO Vacuumed table table=pending_closures36302026/11/12 00:09:50 INFO Vacuumed table table=pending_objects36312026/11/12 00:09:50 INFO Completed push id=15 objects=5 duration=2.390469ms36322026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036332026/11/12 00:09:50 INFO Received complete upload request method=POST path=/36342026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads36352026/11/12 00:09:50 INFO Received complete upload request method=POST path=/36362026/11/12 00:09:50 INFO Received complete push request method=POST path=/36372026/11/12 00:09:50 INFO Received complete upload request method=POST path=/36382026/11/12 00:09:50 INFO Vacuumed table table=closures36392026/11/12 00:09:50 INFO Completed upload id=15 objects=9 duration=1.16247ms36402026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036412026/11/12 00:09:50 INFO Vacuumed table table=objects36422026/11/12 00:09:50 INFO Completed upload id=19 objects=25 duration=2.807169ms36432026/11/12 00:09:50 INFO Completed upload id=16 objects=17 duration=2.863459ms36442026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036452026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period36462026/11/12 00:09:50 INFO Completed push id=9 objects=4 duration=3.613419ms36472026/11/12 00:09:50 INFO Vacuumed table table=pending_closures36482026/11/12 00:09:50 INFO Vacuumed table table=pending_objects36492026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads36502026/11/12 00:09:50 INFO Received complete upload request method=POST path=/36512026/11/12 00:09:50 INFO Vacuumed table table=closures36522026/11/12 00:09:50 INFO Received complete upload request method=POST path=/36532026/11/12 00:09:50 INFO Vacuumed table table=objects36542026/11/12 00:09:50 INFO Received push request method=POST path=/36552026/11/12 00:09:50 INFO Completed upload id=23 objects=16 duration=2.423019ms36562026/11/12 00:09:50 INFO Completed upload id=23 objects=15 duration=2.507419ms36572026/11/12 00:09:50 INFO Received complete push request method=POST path=/36582026/11/12 00:09:50 INFO Received uploads request method=POST path=/36592026/11/12 00:09:50 INFO Completed push id=12 objects=2 duration=3.487759ms36602026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036612026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period36622026/11/12 00:09:50 INFO Received complete push request method=POST path=/36632026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036642026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036652026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036662026/11/12 00:09:50 INFO Received uploads request method=POST path=/36672026/11/12 00:09:50 INFO Vacuumed table table=pending_closures36682026/11/12 00:09:50 INFO Completed push id=1 objects=11 duration=2.453349ms36692026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period36702026/11/12 00:09:50 INFO Vacuumed table table=pending_objects36712026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads36722026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period36732026/11/12 00:09:50 INFO Vacuumed table table=closures36742026/11/12 00:09:50 INFO Received push request method=POST path=/36752026/11/12 00:09:50 INFO Vacuumed table table=objects36762026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=036772026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period36782026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=036792026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=036802026/11/12 00:09:50 INFO Vacuumed table table=pending_closures36812026/11/12 00:09:50 INFO Vacuumed table table=pending_closures36822026/11/12 00:09:50 INFO Vacuumed table table=pending_objects36832026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads36842026/11/12 00:09:50 INFO Vacuumed table table=pending_objects36852026/11/12 00:09:50 INFO Vacuumed table table=closures36862026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads36872026/11/12 00:09:50 INFO Vacuumed table table=closures36882026/11/12 00:09:50 INFO Vacuumed table table=objects36892026/11/12 00:09:50 INFO Vacuumed table table=objects36902026/11/12 00:09:50 INFO Received uploads request method=POST path=/36912026/11/12 00:09:50 INFO Received complete push request method=POST path=/36922026/11/12 00:09:50 INFO Received complete push request method=POST path=/36932026/11/12 00:09:50 INFO Completed push id=18 objects=10 duration=1.999929ms36942026/11/12 00:09:50 INFO Completed push id=9 objects=24 duration=1.847809ms36952026/11/12 00:09:50 INFO Received complete upload request method=POST path=/36962026/11/12 00:09:50 INFO Received uploads request method=POST path=/36972026/11/12 00:09:50 INFO Received push request method=POST path=/36982026/11/12 00:09:50 INFO Received complete push request method=POST path=/36992026/11/12 00:09:50 INFO Completed upload id=22 objects=2 duration=1.01094ms37002026/11/12 00:09:50 INFO Completed push id=21 objects=17 duration=2.608229ms37012026/11/12 00:09:50 INFO Received push request method=POST path=/37022026/11/12 00:09:50 INFO Received complete upload request method=POST path=/37032026/11/12 00:09:50 INFO Completed upload id=16 objects=3 duration=1.422749ms37042026/11/12 00:09:50 INFO Received complete upload request method=POST path=/37052026/11/12 00:09:50 INFO Completed upload id=19 objects=6 duration=1.123609ms37062026/11/12 00:09:50 INFO Received uploads request method=POST path=/37072026/11/12 00:09:50 INFO Received push request method=POST path=/37082026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=11 objects-failed-to-delete=037092026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37102026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=037112026/11/12 00:09:50 INFO Received uploads request method=POST path=/37122026/11/12 00:09:50 INFO Received push request method=POST path=/37132026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37142026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37152026/11/12 00:09:50 INFO Received uploads request method=POST path=/37162026/11/12 00:09:50 INFO Vacuumed table table=closures37172026/11/12 00:09:50 INFO Vacuumed table table=objects37182026/11/12 00:09:50 INFO Received complete push request method=POST path=/37192026/11/12 00:09:50 INFO Completed push id=19 objects=7 duration=1.828909ms37202026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=037212026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37222026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37232026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37242026/11/12 00:09:50 INFO Vacuumed table table=closures37252026/11/12 00:09:50 INFO Received push request method=POST path=/37262026/11/12 00:09:50 INFO Vacuumed table table=objects37272026/11/12 00:09:50 INFO Received push request method=POST path=/37282026/11/12 00:09:50 INFO Received uploads request method=POST path=/37292026/11/12 00:09:50 INFO Received complete upload request method=POST path=/37302026/11/12 00:09:50 INFO Completed upload id=22 objects=7 duration=1.17092ms37312026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=037322026/11/12 00:09:50 INFO Received push request method=POST path=/37332026/11/12 00:09:50 INFO Received uploads request method=POST path=/37342026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037352026/11/12 00:09:50 INFO Received uploads request method=POST path=/37362026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37372026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37382026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37392026/11/12 00:09:50 INFO Vacuumed table table=closures37402026/11/12 00:09:50 INFO Received push request method=POST path=/37412026/11/12 00:09:50 INFO Vacuumed table table=objects37422026/11/12 00:09:50 INFO Received uploads request method=POST path=/37432026/11/12 00:09:50 INFO Received push request method=POST path=/37442026/11/12 00:09:50 INFO Received push request method=POST path=/37452026/11/12 00:09:50 INFO Received uploads request method=POST path=/37462026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=037472026/11/12 00:09:50 INFO Received push request method=POST path=/37482026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=8 objects-deleted-after-grace-period=8 objects-failed-to-delete=037492026/11/12 00:09:50 INFO Received uploads request method=POST path=/37502026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037512026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37522026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=037532026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37542026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37552026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37562026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37572026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37582026/11/12 00:09:50 INFO Vacuumed table table=closures37592026/11/12 00:09:50 INFO Received complete upload request method=POST path=/37602026/11/12 00:09:50 INFO Vacuumed table table=closures37612026/11/12 00:09:50 INFO Vacuumed table table=objects37622026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period37632026/11/12 00:09:50 INFO Vacuumed table table=objects37642026/11/12 00:09:50 INFO Completed upload id=10 objects=10 duration=1.47528ms37652026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=037662026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=5 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037672026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037682026/11/12 00:09:50 INFO Received push request method=POST path=/37692026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=037702026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37712026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37722026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37732026/11/12 00:09:50 INFO Received uploads request method=POST path=/37742026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37752026/11/12 00:09:50 INFO Vacuumed table table=closures37762026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37772026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period37782026/11/12 00:09:50 INFO Vacuumed table table=objects37792026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37802026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037812026/11/12 00:09:50 INFO Vacuumed table table=closures37822026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37832026/11/12 00:09:50 INFO Vacuumed table table=objects37842026/11/12 00:09:50 INFO Vacuumed table table=pending_objects37852026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads37862026/11/12 00:09:50 INFO Vacuumed table table=closures37872026/11/12 00:09:50 INFO Received push request method=POST path=/37882026/11/12 00:09:50 INFO Vacuumed table table=objects37892026/11/12 00:09:50 INFO Received complete push request method=POST path=/37902026/11/12 00:09:50 INFO Received push request method=POST path=/37912026/11/12 00:09:50 INFO Received push request method=POST path=/37922026/11/12 00:09:50 INFO Completed push id=14 objects=4 duration=3.526518ms37932026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=037942026/11/12 00:09:50 INFO Received complete push request method=POST path=/37952026/11/12 00:09:50 INFO Received uploads request method=POST path=/37962026/11/12 00:09:50 INFO Completed push id=17 objects=22 duration=3.097409ms37972026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=2 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=037982026/11/12 00:09:50 INFO Vacuumed table table=pending_closures37992026/11/12 00:09:50 INFO Received complete push request method=POST path=/38002026/11/12 00:09:50 INFO Vacuumed table table=pending_objects38012026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads38022026/11/12 00:09:50 INFO Vacuumed table table=closures38032026/11/12 00:09:50 INFO Received uploads request method=POST path=/38042026/11/12 00:09:50 INFO Completed push id=24 objects=9 duration=1.87433ms38052026/11/12 00:09:50 INFO Vacuumed table table=objects38062026/11/12 00:09:50 INFO Received uploads request method=POST path=/38072026/11/12 00:09:50 INFO Received push request method=POST path=/38082026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=038092026/11/12 00:09:50 INFO Received uploads request method=POST path=/38102026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period38112026/11/12 00:09:50 INFO Received uploads request method=POST path=/38122026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038132026/11/12 00:09:50 INFO Vacuumed table table=pending_closures38142026/11/12 00:09:50 INFO Received push request method=POST path=/38152026/11/12 00:09:50 INFO Vacuumed table table=pending_objects38162026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads38172026/11/12 00:09:50 INFO Vacuumed table table=closures38182026/11/12 00:09:50 INFO Vacuumed table table=objects38192026/11/12 00:09:50 INFO Received push request method=POST path=/38202026/11/12 00:09:50 INFO Received push request method=POST path=/38212026/11/12 00:09:50 INFO Received push request method=POST path=/38222026/11/12 00:09:50 INFO Received push request method=POST path=/38232026/11/12 00:09:50 INFO Received push request method=POST path=/38242026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=038252026/11/12 00:09:50 INFO Received complete push request method=POST path=/38262026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038272026/11/12 00:09:50 INFO Vacuumed table table=pending_closures38282026/11/12 00:09:50 INFO Received complete upload request method=POST path=/38292026/11/12 00:09:50 INFO Completed push id=7 objects=15 duration=2.882009ms38302026/11/12 00:09:50 INFO Vacuumed table table=pending_objects38312026/11/12 00:09:50 INFO Received push request method=POST path=/38322026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads38332026/11/12 00:09:50 INFO Vacuumed table table=closures38342026/11/12 00:09:50 INFO Received push request method=POST path=/38352026/11/12 00:09:50 INFO Completed upload id=15 objects=3 duration=2.103069ms38362026/11/12 00:09:50 INFO Vacuumed table table=objects38372026/11/12 00:09:50 INFO Received uploads request method=POST path=/38382026/11/12 00:09:50 INFO Received complete upload request method=POST path=/38392026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=038402026/11/12 00:09:50 INFO Received push request method=POST path=/38412026/11/12 00:09:50 INFO Completed upload id=12 objects=26 duration=2.806269ms38422026/11/12 00:09:50 INFO Received push request method=POST path=/38432026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period38442026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038452026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=038462026/11/12 00:09:50 INFO Vacuumed table table=pending_closures38472026/11/12 00:09:50 INFO Vacuumed table table=pending_objects38482026/11/12 00:09:50 INFO Received push request method=POST path=/38492026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads38502026/11/12 00:09:50 INFO Received uploads request method=POST path=/38512026/11/12 00:09:50 INFO Received complete push request method=POST path=/38522026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038532026/11/12 00:09:50 INFO Vacuumed table table=closures38542026/11/12 00:09:50 INFO Vacuumed table table=pending_closures38552026/11/12 00:09:50 INFO Completed push id=4 objects=6 duration=2.141079ms38562026/11/12 00:09:50 INFO Vacuumed table table=objects38572026/11/12 00:09:50 INFO Vacuumed table table=pending_objects38582026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads38592026/11/12 00:09:50 INFO Received uploads request method=POST path=/38602026/11/12 00:09:50 INFO Vacuumed table table=closures38612026/11/12 00:09:50 INFO Vacuumed table table=objects38622026/11/12 00:09:50 INFO Received complete push request method=POST path=/38632026/11/12 00:09:50 INFO Received push request method=POST path=/38642026/11/12 00:09:50 INFO Completed push id=22 objects=3 duration=2.850958ms38652026/11/12 00:09:50 INFO Received push request method=POST path=/38662026/11/12 00:09:50 INFO Received push request method=POST path=/38672026/11/12 00:09:50 INFO Received uploads request method=POST path=/38682026/11/12 00:09:50 INFO Received push request method=POST path=/38692026/11/12 00:09:50 INFO Received push request method=POST path=/38702026/11/12 00:09:50 INFO Received push request method=POST path=/38712026/11/12 00:09:50 INFO Received push request method=POST path=/38722026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=038732026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period38742026/11/12 00:09:50 INFO Received complete upload request method=POST path=/38752026/11/12 00:09:50 INFO Received complete upload request method=POST path=/38762026/11/12 00:09:50 INFO Completed upload id=7 objects=10 duration=1.699639ms38772026/11/12 00:09:50 INFO Completed upload id=14 objects=16 duration=2.04614ms38782026/11/12 00:09:50 INFO Received uploads request method=POST path=/38792026/11/12 00:09:50 INFO Received uploads request method=POST path=/38802026/11/12 00:09:50 INFO Received push request method=POST path=/38812026/11/12 00:09:50 INFO Received push request method=POST path=/38822026/11/12 00:09:50 INFO Received uploads request method=POST path=/38832026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=038842026/11/12 00:09:50 INFO Received push request method=POST path=/38852026/11/12 00:09:50 INFO Received uploads request method=POST path=/38862026/11/12 00:09:50 INFO Received push request method=POST path=/38872026/11/12 00:09:50 INFO Received complete push request method=POST path=/38882026/11/12 00:09:50 INFO Received uploads request method=POST path=/38892026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period38902026/11/12 00:09:50 INFO Completed push id=26 objects=14 duration=3.612779ms38912026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=038922026/11/12 00:09:50 INFO Vacuumed table table=pending_closures38932026/11/12 00:09:50 INFO Vacuumed table table=pending_objects38942026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads38952026/11/12 00:09:50 INFO Vacuumed table table=closures38962026/11/12 00:09:50 INFO Vacuumed table table=objects38972026/11/12 00:09:50 INFO Received complete push request method=POST path=/38982026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."38992026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=039002026/11/12 00:09:50 INFO Completed push id=16 objects=4 duration=3.124719ms39012026/11/12 00:09:50 INFO Received complete push request method=POST path=/39022026/11/12 00:09:50 INFO Completed push id=16 objects=8 duration=1.943349ms39032026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039042026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=039052026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=039062026/11/12 00:09:50 INFO Vacuumed table table=pending_closures39072026/11/12 00:09:50 INFO Vacuumed table table=pending_objects39082026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads39092026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period39102026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039112026/11/12 00:09:50 INFO Vacuumed table table=closures39122026/11/12 00:09:50 INFO Received push request method=POST path=/39132026/11/12 00:09:50 INFO Vacuumed table table=pending_closures39142026/11/12 00:09:50 INFO Vacuumed table table=objects39152026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039162026/11/12 00:09:50 INFO Received complete push request method=POST path=/39172026/11/12 00:09:50 INFO Vacuumed table table=pending_objects39182026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads39192026/11/12 00:09:50 INFO Vacuumed table table=pending_closures39202026/11/12 00:09:50 INFO Completed push id=14 objects=3 duration=1.34124ms39212026/11/12 00:09:50 INFO Vacuumed table table=closures39222026/11/12 00:09:50 INFO Received complete push request method=POST path=/39232026/11/12 00:09:50 INFO Vacuumed table table=pending_objects39242026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads39252026/11/12 00:09:50 INFO Vacuumed table table=objects39262026/11/12 00:09:50 INFO Vacuumed table table=closures39272026/11/12 00:09:50 INFO Received complete push request method=POST path=/39282026/11/12 00:09:50 INFO Vacuumed table table=objects39292026/11/12 00:09:50 INFO Completed push id=14 objects=3 duration=3.367119ms39302026/11/12 00:09:50 INFO Completed push id=17 objects=22 duration=2.515259ms39312026/11/12 00:09:50 INFO Received uploads request method=POST path=/39322026/11/12 00:09:50 INFO Received uploads request method=POST path=/39332026/11/12 00:09:50 INFO Received uploads request method=POST path=/39342026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=039352026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period39362026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039372026/11/12 00:09:50 INFO Received uploads request method=POST path=/39382026/11/12 00:09:50 INFO Vacuumed table table=pending_closures39392026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=039402026/11/12 00:09:50 INFO Received complete push request method=POST path=/39412026/11/12 00:09:50 INFO Vacuumed table table=pending_objects39422026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period39432026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads39442026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039452026/11/12 00:09:50 INFO Vacuumed table table=closures39462026/11/12 00:09:50 INFO Completed push id=17 objects=11 duration=1.735249ms39472026/11/12 00:09:50 INFO Vacuumed table table=pending_closures39482026/11/12 00:09:50 INFO Received push request method=POST path=/39492026/11/12 00:09:50 INFO Vacuumed table table=objects39502026/11/12 00:09:50 INFO Vacuumed table table=pending_objects39512026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads39522026/11/12 00:09:50 INFO Received uploads request method=POST path=/39532026/11/12 00:09:50 INFO Vacuumed table table=closures39542026/11/12 00:09:50 INFO Vacuumed table table=objects39552026/11/12 00:09:50 INFO Received uploads request method=POST path=/39562026/11/12 00:09:50 INFO Received uploads request method=POST path=/39572026/11/12 00:09:50 INFO Received complete upload request method=POST path=/39582026/11/12 00:09:50 INFO Received complete push request method=POST path=/39592026/11/12 00:09:50 INFO Completed upload id=20 objects=5 duration=1.16561ms39602026/11/12 00:09:50 INFO Completed push id=21 objects=9 duration=1.966029ms39612026/11/12 00:09:50 INFO Received complete push request method=POST path=/39622026/11/12 00:09:50 INFO Received push request method=POST path=/39632026/11/12 00:09:50 INFO Completed push id=16 objects=16 duration=4.057599ms39642026/11/12 00:09:50 INFO Received complete upload request method=POST path=/39652026/11/12 00:09:50 INFO Received uploads request method=POST path=/39662026/11/12 00:09:50 INFO Completed upload id=19 objects=2 duration=2.356669ms39672026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=039682026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=039692026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039702026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period39712026/11/12 00:09:50 INFO Vacuumed table table=pending_closures39722026/11/12 00:09:50 INFO Received complete push request method=POST path=/39732026/11/12 00:09:50 INFO Received uploads request method=POST path=/39742026/11/12 00:09:50 INFO Vacuumed table table=pending_objects39752026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads39762026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=039772026/11/12 00:09:50 INFO Received uploads request method=POST path=/39782026/11/12 00:09:50 INFO Vacuumed table table=closures39792026/11/12 00:09:50 INFO Vacuumed table table=pending_closures39802026/11/12 00:09:50 INFO Completed push id=11 objects=6 duration=3.379589ms39812026/11/12 00:09:50 INFO Vacuumed table table=objects39822026/11/12 00:09:50 INFO Vacuumed table table=pending_objects39832026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads39842026/11/12 00:09:50 INFO Vacuumed table table=closures39852026/11/12 00:09:50 INFO Vacuumed table table=objects39862026/11/12 00:09:50 INFO Received complete push request method=POST path=/39872026/11/12 00:09:50 INFO Received uploads request method=POST path=/39882026/11/12 00:09:50 INFO Completed push id=21 objects=3 duration=1.637259ms39892026/11/12 00:09:50 INFO Received push request method=POST path=/39902026/11/12 00:09:50 INFO Received push request method=POST path=/39912026/11/12 00:09:50 INFO Received complete upload request method=POST path=/39922026/11/12 00:09:50 INFO Received uploads request method=POST path=/39932026/11/12 00:09:50 INFO Completed upload id=11 objects=7 duration=913.959µs39942026/11/12 00:09:50 INFO Received push request method=POST path=/39952026/11/12 00:09:50 INFO Received uploads request method=POST path=/39962026/11/12 00:09:50 INFO Received complete push request method=POST path=/39972026/11/12 00:09:50 INFO Completed push id=19 objects=14 duration=1.998389ms39982026/11/12 00:09:50 INFO Received push request method=POST path=/39992026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040002026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period40012026/11/12 00:09:50 INFO Received uploads request method=POST path=/40022026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040032026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040042026/11/12 00:09:50 INFO Received complete push request method=POST path=/40052026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040062026/11/12 00:09:50 INFO Vacuumed table table=pending_closures40072026/11/12 00:09:50 INFO Completed push id=33 objects=7 duration=1.818399ms40082026/11/12 00:09:50 INFO Received uploads request method=POST path=/40092026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period40102026/11/12 00:09:50 INFO Vacuumed table table=pending_objects40112026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads40122026/11/12 00:09:50 INFO Vacuumed table table=closures40132026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040142026/11/12 00:09:50 INFO Received complete push request method=POST path=/40152026/11/12 00:09:50 INFO Vacuumed table table=objects40162026/11/12 00:09:50 INFO Vacuumed table table=pending_closures40172026/11/12 00:09:50 INFO Completed push id=18 objects=6 duration=1.281889ms40182026/11/12 00:09:50 INFO Vacuumed table table=pending_objects40192026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads40202026/11/12 00:09:50 INFO Vacuumed table table=closures40212026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040222026/11/12 00:09:50 INFO Vacuumed table table=objects40232026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040242026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period40252026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040262026/11/12 00:09:50 INFO Vacuumed table table=pending_closures40272026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040282026/11/12 00:09:50 INFO Vacuumed table table=pending_objects40292026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads40302026/11/12 00:09:50 INFO Vacuumed table table=closures40312026/11/12 00:09:50 INFO Vacuumed table table=objects40322026/11/12 00:09:50 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."40332026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period40342026/11/12 00:09:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."40352026/11/12 00:09:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."40362026/11/12 00:09:50 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."40372026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."40382026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."40392026/11/12 00:09:50 ERROR failed to remove object object=hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh.ls error="Access Denied."40402026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:cccccccc!out.doi error="Access Denied."40412026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."40422026/11/12 00:09:50 INFO Received push request method=POST path=/40432026/11/12 00:09:50 INFO Received push request method=POST path=/40442026/11/12 00:09:50 INFO Received complete upload request method=POST path=/40452026/11/12 00:09:50 INFO Received complete upload request method=POST path=/40462026/11/12 00:09:50 INFO Completed upload id=26 objects=6 duration=1.102729ms40472026/11/12 00:09:50 INFO Completed upload id=26 objects=3 duration=990.11µs40482026/11/12 00:09:50 INFO Received uploads request method=POST path=/40492026/11/12 00:09:50 INFO Received push request method=POST path=/40502026/11/12 00:09:50 INFO Received uploads request method=POST path=/40512026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040522026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040532026/11/12 00:09:50 INFO Vacuumed table table=pending_closures40542026/11/12 00:09:50 INFO Vacuumed table table=pending_objects40552026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads40562026/11/12 00:09:50 INFO Vacuumed table table=closures40572026/11/12 00:09:50 INFO Vacuumed table table=objects40582026/11/12 00:09:50 INFO Received uploads request method=POST path=/40592026/11/12 00:09:50 INFO Received complete upload request method=POST path=/40602026/11/12 00:09:50 INFO Completed upload id=17 objects=25 duration=2.882859ms40612026/11/12 00:09:50 INFO Received push request method=POST path=/40622026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040632026/11/12 00:09:50 INFO Received push request method=POST path=/40642026/11/12 00:09:50 INFO Received uploads request method=POST path=/40652026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period40662026/11/12 00:09:50 INFO Received complete push request method=POST path=/40672026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040682026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040692026/11/12 00:09:50 INFO Received push request method=POST path=/40702026/11/12 00:09:50 INFO Received uploads request method=POST path=/40712026/11/12 00:09:50 INFO Vacuumed table table=pending_closures40722026/11/12 00:09:50 INFO Completed push id=19 objects=14 duration=1.7739ms40732026/11/12 00:09:50 INFO Vacuumed table table=pending_objects40742026/11/12 00:09:50 INFO Received complete upload request method=POST path=/40752026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads40762026/11/12 00:09:50 INFO Vacuumed table table=closures40772026/11/12 00:09:50 INFO Vacuumed table table=objects40782026/11/12 00:09:50 INFO Completed upload id=23 objects=18 duration=2.551049ms40792026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=3 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=040802026/11/12 00:09:50 INFO Vacuumed table table=pending_closures40812026/11/12 00:09:50 INFO Vacuumed table table=pending_objects40822026/11/12 00:09:50 INFO Received push request method=POST path=/40832026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads40842026/11/12 00:09:50 INFO Vacuumed table table=closures40852026/11/12 00:09:50 INFO Vacuumed table table=objects40862026/11/12 00:09:50 INFO Received uploads request method=POST path=/40872026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=3 objects-failed-to-delete=040882026/11/12 00:09:50 INFO Vacuumed table table=pending_closures40892026/11/12 00:09:50 INFO Received complete upload request method=POST path=/40902026/11/12 00:09:50 INFO Received push request method=POST path=/40912026/11/12 00:09:50 INFO Received complete upload request method=POST path=/40922026/11/12 00:09:50 INFO Completed upload id=20 objects=3 duration=1.784969ms40932026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=040942026/11/12 00:09:50 INFO Vacuumed table table=pending_objects40952026/11/12 00:09:50 INFO Completed upload id=29 objects=5 duration=950.98µs40962026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads40972026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period40982026/11/12 00:09:50 INFO Received uploads request method=POST path=/40992026/11/12 00:09:50 INFO Vacuumed table table=closures41002026/11/12 00:09:50 INFO Received uploads request method=POST path=/41012026/11/12 00:09:50 INFO Vacuumed table table=objects41022026/11/12 00:09:50 INFO Received push request method=POST path=/41032026/11/12 00:09:50 INFO Received uploads request method=POST path=/41042026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=3 objects-failed-to-delete=041052026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41062026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41072026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41082026/11/12 00:09:50 INFO Vacuumed table table=closures41092026/11/12 00:09:50 INFO Vacuumed table table=objects41102026/11/12 00:09:50 INFO Received push request method=POST path=/41112026/11/12 00:09:50 INFO Received push request method=POST path=/41122026/11/12 00:09:50 INFO Received push request method=POST path=/41132026/11/12 00:09:50 INFO Received complete upload request method=POST path=/41142026/11/12 00:09:50 INFO Received push request method=POST path=/41152026/11/12 00:09:50 INFO Completed upload id=21 objects=2 duration=695.17µs41162026/11/12 00:09:50 INFO Received push request method=POST path=/41172026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=041182026/11/12 00:09:50 INFO Received uploads request method=POST path=/41192026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041202026/11/12 00:09:50 INFO Received push request method=POST path=/41212026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41222026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41232026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41242026/11/12 00:09:50 INFO Vacuumed table table=closures41252026/11/12 00:09:50 INFO Received complete upload request method=POST path=/41262026/11/12 00:09:50 INFO Vacuumed table table=objects41272026/11/12 00:09:50 INFO Received complete upload request method=POST path=/41282026/11/12 00:09:50 INFO Completed upload id=20 objects=6 duration=1.319329ms41292026/11/12 00:09:50 INFO Completed upload id=21 objects=7 duration=2.716468ms41302026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=041312026/11/12 00:09:50 INFO Received push request method=POST path=/41322026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period41332026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=041342026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041352026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41362026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period41372026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=041382026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41392026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period41402026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41412026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041422026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041432026/11/12 00:09:50 INFO Vacuumed table table=closures41442026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41452026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41462026/11/12 00:09:50 INFO Vacuumed table table=objects41472026/11/12 00:09:50 INFO Received uploads request method=POST path=/41482026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41492026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41502026/11/12 00:09:50 INFO Received complete push request method=POST path=/41512026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41522026/11/12 00:09:50 INFO Vacuumed table table=closures41532026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41542026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=041552026/11/12 00:09:50 INFO Vacuumed table table=objects41562026/11/12 00:09:50 INFO Vacuumed table table=closures41572026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period41582026/11/12 00:09:50 INFO Vacuumed table table=objects41592026/11/12 00:09:50 INFO Completed push id=9 objects=9 duration=3.269388ms41602026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041612026/11/12 00:09:50 INFO Received uploads request method=POST path=/41622026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41632026/11/12 00:09:50 INFO Received complete upload request method=POST path=/41642026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41652026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41662026/11/12 00:09:50 INFO Completed upload id=22 objects=22 duration=1.354729ms41672026/11/12 00:09:50 INFO Vacuumed table table=closures41682026/11/12 00:09:50 INFO Vacuumed table table=objects41692026/11/12 00:09:50 INFO Received complete push request method=POST path=/41702026/11/12 00:09:50 INFO Completed push id=30 objects=4 duration=3.118889ms41712026/11/12 00:09:50 INFO Received push request method=POST path=/41722026/11/12 00:09:50 INFO Received complete push request method=POST path=/41732026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=041742026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=041752026/11/12 00:09:50 INFO Received complete upload request method=POST path=/41762026/11/12 00:09:50 INFO Completed push id=13 objects=15 duration=2.047369ms41772026/11/12 00:09:50 INFO Completed upload id=24 objects=7 duration=1.26098ms41782026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041792026/11/12 00:09:50 INFO Received push request method=POST path=/41802026/11/12 00:09:50 INFO Received complete upload request method=POST path=/41812026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=041822026/11/12 00:09:50 INFO Received push request method=POST path=/41832026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41842026/11/12 00:09:50 INFO Received uploads request method=POST path=/41852026/11/12 00:09:50 INFO Received complete upload request method=POST path=/41862026/11/12 00:09:50 INFO Vacuumed table table=pending_closures41872026/11/12 00:09:50 INFO Completed upload id=21 objects=8 duration=1.58954ms41882026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41892026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41902026/11/12 00:09:50 INFO Vacuumed table table=pending_objects41912026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads41922026/11/12 00:09:50 INFO Vacuumed table table=closures41932026/11/12 00:09:50 INFO Completed upload id=33 objects=5 duration=2.348099ms41942026/11/12 00:09:50 INFO Vacuumed table table=closures41952026/11/12 00:09:50 INFO Vacuumed table table=objects41962026/11/12 00:09:50 INFO Vacuumed table table=objects41972026/11/12 00:09:50 INFO Received uploads request method=POST path=/41982026/11/12 00:09:50 INFO Received push request method=POST path=/41992026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=042002026/11/12 00:09:50 INFO Received complete push request method=POST path=/42012026/11/12 00:09:50 INFO Completed push id=22 objects=6 duration=1.167729ms42022026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=042032026/11/12 00:09:50 INFO Received uploads request method=POST path=/42042026/11/12 00:09:50 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042052026/11/12 00:09:50 INFO Vacuumed table table=pending_closures42062026/11/12 00:09:50 INFO Received complete push request method=POST path=/42072026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42082026/11/12 00:09:50 INFO Vacuumed table table=pending_objects42092026/11/12 00:09:50 INFO Vacuumed table table=multipart_uploads42102026/11/12 00:09:50 INFO Completed push id=26 objects=5 duration=1.42529ms42112026/11/12 00:09:50 INFO Vacuumed table table=closures42122026/11/12 00:09:50 INFO Received push request method=POST path=/42132026/11/12 00:09:50 INFO Vacuumed table table=objects42142026/11/12 00:09:50 INFO Received push request method=POST path=/42152026/11/12 00:09:50 INFO Received uploads request method=POST path=/42162026/11/12 00:09:50 INFO Aborted multipart uploads count=0 kept=042172026/11/12 00:09:50 WARN Force mode enabled - objects will be deleted immediately without grace period42182026/11/12 00:09:50 INFO Received complete push request method=POST path=/42192026/11/12 00:09:50 INFO Received complete push request method=POST path=/42202026/11/12 00:09:50 INFO Completed push id=24 objects=8 duration=1.37472ms42212026/11/12 00:09:50 INFO Received uploads request method=POST path=/42222026/11/12 00:09:50 INFO Received complete push request method=POST path=/42232026/11/12 00:09:50 INFO Completed push id=27 objects=25 duration=3.050379ms42242026/11/12 00:09:50 INFO Completed push id=7 objects=24 duration=3.449019ms42252026/11/12 00:09:50 INFO Received uploads request method=POST path=/42262026/11/12 00:09:50 INFO Received complete upload request method=POST path=/42272026/11/12 00:09:50 INFO Received complete upload request method=POST path=/42282026/11/12 00:09:50 INFO Received uploads request method=POST path=/42292026/11/12 00:09:50 INFO Completed upload id=24 objects=14 duration=936.759µs42302026/11/12 00:09:50 INFO Completed upload id=21 objects=2 duration=1.30161ms42312026/11/12 00:09:50 INFO Received push request method=POST path=/42322026/11/12 00:09:50 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."42332026/11/12 00:09:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.ls error="Access Denied."42342026/11/12 00:09:50 ERROR failed to remove object object=iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.narinfo error="Access Denied."42352026/11/12 00:09:50 ERROR failed to remove object object=nar/iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii.nar.zst error="Access Denied."42362026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."42372026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:hhhhhhhh!out.doi error="Access Denied."42382026/11/12 00:09:50 ERROR failed to remove object object=realisations/sha256:iiiiiiii!out.doi error="Access Denied."42392026/11/12 00:09:50 INFO Received complete upload request method=POST path=/42402026/11/12 00:09:50 INFO Completed upload id=25 objects=9 duration=1.011449ms42412026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42422026/11/12 00:09:51 INFO Received uploads request method=POST path=/42432026/11/12 00:09:51 INFO Received complete push request method=POST path=/42442026/11/12 00:09:51 INFO Completed upload id=21 objects=25 duration=970.8µs42452026/11/12 00:09:51 INFO Received uploads request method=POST path=/42462026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=042472026/11/12 00:09:51 INFO Completed push id=20 objects=3 duration=1.414989ms42482026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=3 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042492026/11/12 00:09:51 INFO Vacuumed table table=pending_closures42502026/11/12 00:09:51 INFO Vacuumed table table=pending_objects42512026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads42522026/11/12 00:09:51 INFO Vacuumed table table=closures42532026/11/12 00:09:51 INFO Vacuumed table table=objects42542026/11/12 00:09:51 INFO Received push request method=POST path=/42552026/11/12 00:09:51 INFO Received complete push request method=POST path=/42562026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42572026/11/12 00:09:51 INFO Completed push id=19 objects=3 duration=1.01332ms42582026/11/12 00:09:51 INFO Received uploads request method=POST path=/42592026/11/12 00:09:51 INFO Received uploads request method=POST path=/42602026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42612026/11/12 00:09:51 INFO Completed upload id=13 objects=8 duration=1.649859ms42622026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=042632026/11/12 00:09:51 INFO Completed upload id=31 objects=3 duration=1.75608ms42642026/11/12 00:09:51 INFO Received complete push request method=POST path=/42652026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period42662026/11/12 00:09:51 INFO Completed push id=32 objects=7 duration=1.22513ms42672026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42682026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042692026/11/12 00:09:51 INFO Vacuumed table table=pending_closures42702026/11/12 00:09:51 INFO Completed upload id=27 objects=3 duration=1.700619ms42712026/11/12 00:09:51 INFO Vacuumed table table=pending_objects42722026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads42732026/11/12 00:09:51 INFO Vacuumed table table=closures42742026/11/12 00:09:51 INFO Vacuumed table table=objects42752026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42762026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42772026/11/12 00:09:51 INFO Received uploads request method=POST path=/42782026/11/12 00:09:51 INFO Completed upload id=27 objects=4 duration=1.33382ms42792026/11/12 00:09:51 INFO Completed upload id=13 objects=14 duration=3.214018ms42802026/11/12 00:09:51 INFO Received uploads request method=POST path=/42812026/11/12 00:09:51 INFO Received complete push request method=POST path=/42822026/11/12 00:09:51 INFO Completed push id=24 objects=5 duration=2.761099ms42832026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42842026/11/12 00:09:51 INFO Received uploads request method=POST path=/42852026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42862026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=042872026/11/12 00:09:51 INFO Completed upload id=21 objects=3 duration=1.37859ms42882026/11/12 00:09:51 INFO Received complete upload request method=POST path=/42892026/11/12 00:09:51 INFO Received complete push request method=POST path=/42902026/11/12 00:09:51 INFO Completed upload id=22 objects=2 duration=2.089499ms42912026/11/12 00:09:51 INFO Completed upload id=28 objects=6 duration=785.389µs42922026/11/12 00:09:51 INFO Completed push id=26 objects=4 duration=1.325849ms42932026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period42942026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=042952026/11/12 00:09:51 INFO Vacuumed table table=pending_closures42962026/11/12 00:09:51 INFO Vacuumed table table=pending_objects42972026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads42982026/11/12 00:09:51 INFO Vacuumed table table=closures42992026/11/12 00:09:51 INFO Vacuumed table table=objects43002026/11/12 00:09:51 INFO Received complete push request method=POST path=/43012026/11/12 00:09:51 INFO Completed push id=13 objects=11 duration=1.722909ms43022026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=043032026/11/12 00:09:51 INFO Received uploads request method=POST path=/43042026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043052026/11/12 00:09:51 INFO Vacuumed table table=pending_closures43062026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=043072026/11/12 00:09:51 INFO Received complete push request method=POST path=/43082026/11/12 00:09:51 INFO Received uploads request method=POST path=/43092026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period43102026/11/12 00:09:51 INFO Vacuumed table table=pending_objects43112026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads43122026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043132026/11/12 00:09:51 INFO Completed push id=23 objects=28 duration=1.963489ms43142026/11/12 00:09:51 INFO Vacuumed table table=closures43152026/11/12 00:09:51 INFO Vacuumed table table=pending_closures43162026/11/12 00:09:51 INFO Vacuumed table table=objects43172026/11/12 00:09:51 INFO Vacuumed table table=pending_objects43182026/11/12 00:09:51 INFO Received complete push request method=POST path=/43192026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads43202026/11/12 00:09:51 INFO Vacuumed table table=closures43212026/11/12 00:09:51 INFO Received uploads request method=POST path=/43222026/11/12 00:09:51 INFO Completed push id=30 objects=4 duration=1.050649ms43232026/11/12 00:09:51 INFO Vacuumed table table=objects43242026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=043252026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=043262026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period43272026/11/12 00:09:51 INFO Received uploads request method=POST path=/43282026/11/12 00:09:51 INFO Received complete upload request method=POST path=/43292026/11/12 00:09:51 INFO Completed upload id=28 objects=13 duration=1.00033ms43302026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period43312026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043322026/11/12 00:09:51 INFO Vacuumed table table=pending_closures43332026/11/12 00:09:51 INFO Vacuumed table table=pending_objects43342026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads43352026/11/12 00:09:51 INFO Received uploads request method=POST path=/43362026/11/12 00:09:51 INFO Vacuumed table table=closures43372026/11/12 00:09:51 INFO Vacuumed table table=objects43382026/11/12 00:09:51 INFO Received push request method=POST path=/43392026/11/12 00:09:51 INFO Received complete push request method=POST path=/43402026/11/12 00:09:51 INFO Received complete upload request method=POST path=/43412026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=7 objects-failed-to-delete=043422026/11/12 00:09:51 INFO Completed push id=31 objects=7 duration=1.211019ms43432026/11/12 00:09:51 INFO Received uploads request method=POST path=/43442026/11/12 00:09:51 INFO Completed upload id=12 objects=6 duration=2.224559ms43452026/11/12 00:09:51 INFO Vacuumed table table=pending_closures43462026/11/12 00:09:51 INFO Vacuumed table table=pending_objects43472026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads43482026/11/12 00:09:51 INFO Vacuumed table table=closures43492026/11/12 00:09:51 INFO Vacuumed table table=objects43502026/11/12 00:09:51 INFO Received complete push request method=POST path=/43512026/11/12 00:09:51 INFO Received push request method=POST path=/43522026/11/12 00:09:51 INFO Received complete push request method=POST path=/43532026/11/12 00:09:51 INFO Completed push id=29 objects=3 duration=915.16µs43542026/11/12 00:09:51 INFO Completed push id=31 objects=6 duration=2.893138ms43552026/11/12 00:09:51 INFO Received uploads request method=POST path=/43562026/11/12 00:09:51 INFO Received push request method=POST path=/43572026/11/12 00:09:51 INFO Received complete upload request method=POST path=/43582026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=043592026/11/12 00:09:51 INFO Completed upload id=18 objects=2 duration=2.359279ms43602026/11/12 00:09:51 INFO Received uploads request method=POST path=/43612026/11/12 00:09:51 INFO Received complete upload request method=POST path=/43622026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period43632026/11/12 00:09:51 INFO Completed upload id=32 objects=5 duration=1.241699ms43642026/11/12 00:09:51 INFO Received complete push request method=POST path=/43652026/11/12 00:09:51 INFO Received uploads request method=POST path=/43662026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043672026/11/12 00:09:51 INFO Completed push id=17 objects=3 duration=1.575529ms43682026/11/12 00:09:51 INFO Vacuumed table table=pending_closures43692026/11/12 00:09:51 INFO Vacuumed table table=pending_objects43702026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads43712026/11/12 00:09:51 INFO Vacuumed table table=closures43722026/11/12 00:09:51 INFO Received complete upload request method=POST path=/43732026/11/12 00:09:51 INFO Vacuumed table table=objects43742026/11/12 00:09:51 INFO Received push request method=POST path=/43752026/11/12 00:09:51 INFO Completed upload id=27 objects=8 duration=1.14999ms43762026/11/12 00:09:51 INFO Received complete push request method=POST path=/43772026/11/12 00:09:51 INFO Completed push id=18 objects=6 duration=3.245598ms43782026/11/12 00:09:51 INFO Received complete upload request method=POST path=/43792026/11/12 00:09:51 INFO Received uploads request method=POST path=/43802026/11/12 00:09:51 INFO Completed upload id=31 objects=2 duration=771.33µs43812026/11/12 00:09:51 INFO Received push request method=POST path=/43822026/11/12 00:09:51 INFO Received push request method=POST path=/43832026/11/12 00:09:51 INFO Received push request method=POST path=/43842026/11/12 00:09:51 INFO Received complete push request method=POST path=/43852026/11/12 00:09:51 INFO Completed push id=13 objects=2 duration=1.345789ms43862026/11/12 00:09:51 INFO Received complete push request method=POST path=/43872026/11/12 00:09:51 INFO Completed push id=21 objects=26 duration=1.927499ms43882026/11/12 00:09:51 INFO Received complete upload request method=POST path=/43892026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=043902026/11/12 00:09:51 INFO Completed upload id=25 objects=4 duration=1.217009ms43912026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period43922026/11/12 00:09:51 INFO Received complete push request method=POST path=/43932026/11/12 00:09:51 INFO Received push request method=POST path=/43942026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=043952026/11/12 00:09:51 INFO Completed push id=23 objects=5 duration=1.365869ms43962026/11/12 00:09:51 INFO Vacuumed table table=pending_closures43972026/11/12 00:09:51 INFO Received push request method=POST path=/43982026/11/12 00:09:51 INFO Vacuumed table table=pending_objects43992026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads44002026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44012026/11/12 00:09:51 INFO Vacuumed table table=closures44022026/11/12 00:09:51 INFO Completed upload id=23 objects=9 duration=1.147119ms44032026/11/12 00:09:51 INFO Vacuumed table table=objects44042026/11/12 00:09:51 INFO Received complete push request method=POST path=/44052026/11/12 00:09:51 INFO Completed push id=33 objects=6 duration=1.200539ms44062026/11/12 00:09:51 INFO Received complete push request method=POST path=/44072026/11/12 00:09:51 INFO Received push request method=POST path=/44082026/11/12 00:09:51 INFO Completed push id=5 objects=28 duration=2.171899ms44092026/11/12 00:09:51 INFO Received complete push request method=POST path=/44102026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44112026/11/12 00:09:51 INFO Completed upload id=27 objects=13 duration=1.18484ms44122026/11/12 00:09:51 INFO Received uploads request method=POST path=/44132026/11/12 00:09:51 INFO Completed push id=24 objects=24 duration=3.413218ms44142026/11/12 00:09:51 INFO Received push request method=POST path=/44152026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44162026/11/12 00:09:51 INFO Completed upload id=24 objects=3 duration=1.28672ms44172026/11/12 00:09:51 INFO Received push request method=POST path=/44182026/11/12 00:09:51 INFO Received complete push request method=POST path=/44192026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44202026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44212026/11/12 00:09:51 INFO Completed upload id=28 objects=17 duration=1.213239ms44222026/11/12 00:09:51 INFO Completed upload id=13 objects=7 duration=1.448369ms44232026/11/12 00:09:51 INFO Completed push id=31 objects=14 duration=2.887419ms44242026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=044252026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=044262026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period44272026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44282026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44292026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044302026/11/12 00:09:51 INFO Completed upload id=34 objects=24 duration=1.335569ms44312026/11/12 00:09:51 INFO Completed upload id=22 objects=6 duration=739.99µs44322026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044332026/11/12 00:09:51 INFO Vacuumed table table=pending_closures44342026/11/12 00:09:51 INFO Vacuumed table table=pending_closures44352026/11/12 00:09:51 INFO Received complete push request method=POST path=/44362026/11/12 00:09:51 INFO Vacuumed table table=pending_objects44372026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads44382026/11/12 00:09:51 INFO Vacuumed table table=closures44392026/11/12 00:09:51 INFO Completed push id=10 objects=2 duration=2.516949ms44402026/11/12 00:09:51 INFO Vacuumed table table=pending_objects44412026/11/12 00:09:51 INFO Vacuumed table table=objects44422026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads44432026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=044442026/11/12 00:09:51 INFO Vacuumed table table=closures44452026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=044462026/11/12 00:09:51 INFO Vacuumed table table=objects44472026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period44482026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044492026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044502026/11/12 00:09:51 INFO Vacuumed table table=pending_closures44512026/11/12 00:09:51 INFO Received complete push request method=POST path=/44522026/11/12 00:09:51 INFO Vacuumed table table=pending_closures44532026/11/12 00:09:51 INFO Vacuumed table table=pending_objects44542026/11/12 00:09:51 INFO Received complete push request method=POST path=/44552026/11/12 00:09:51 INFO Completed push id=16 objects=14 duration=1.500789ms44562026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads44572026/11/12 00:09:51 INFO Received complete push request method=POST path=/44582026/11/12 00:09:51 INFO Vacuumed table table=closures44592026/11/12 00:09:51 INFO Completed push id=25 objects=4 duration=1.18111ms44602026/11/12 00:09:51 INFO Vacuumed table table=objects44612026/11/12 00:09:51 INFO Vacuumed table table=pending_objects44622026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads44632026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44642026/11/12 00:09:51 INFO Vacuumed table table=closures44652026/11/12 00:09:51 INFO Received push request method=POST path=/44662026/11/12 00:09:51 INFO Completed push id=22 objects=3 duration=3.068989ms44672026/11/12 00:09:51 INFO Received complete push request method=POST path=/44682026/11/12 00:09:51 INFO Completed upload id=20 objects=4 duration=856.65µs44692026/11/12 00:09:51 INFO Vacuumed table table=objects44702026/11/12 00:09:51 INFO Completed push id=29 objects=2 duration=1.202159ms44712026/11/12 00:09:51 INFO Received uploads request method=POST path=/44722026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=044732026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period44742026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044752026/11/12 00:09:51 INFO Vacuumed table table=pending_closures44762026/11/12 00:09:51 INFO Vacuumed table table=pending_objects44772026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads44782026/11/12 00:09:51 INFO Vacuumed table table=closures44792026/11/12 00:09:51 INFO Vacuumed table table=objects44802026/11/12 00:09:51 INFO Received complete push request method=POST path=/44812026/11/12 00:09:51 INFO Completed push id=1 objects=9 duration=1.422539ms44822026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44832026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=044842026/11/12 00:09:51 INFO Completed upload id=17 objects=5 duration=1.147819ms44852026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=044862026/11/12 00:09:51 INFO Received uploads request method=POST path=/44872026/11/12 00:09:51 INFO Vacuumed table table=pending_closures44882026/11/12 00:09:51 INFO Vacuumed table table=pending_objects44892026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads44902026/11/12 00:09:51 INFO Vacuumed table table=closures44912026/11/12 00:09:51 INFO Vacuumed table table=objects44922026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=044932026/11/12 00:09:51 INFO Received complete push request method=POST path=/44942026/11/12 00:09:51 INFO Received uploads request method=POST path=/44952026/11/12 00:09:51 INFO Received uploads request method=POST path=/44962026/11/12 00:09:51 INFO Received complete upload request method=POST path=/44972026/11/12 00:09:51 INFO Completed push id=20 objects=8 duration=1.4601ms44982026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period44992026/11/12 00:09:51 INFO Completed upload id=23 objects=7 duration=979.959µs45002026/11/12 00:09:51 INFO Received uploads request method=POST path=/45012026/11/12 00:09:51 INFO Received uploads request method=POST path=/45022026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045032026/11/12 00:09:51 INFO Received complete push request method=POST path=/45042026/11/12 00:09:51 INFO Vacuumed table table=pending_closures45052026/11/12 00:09:51 INFO Completed push id=31 objects=6 duration=1.282109ms45062026/11/12 00:09:51 INFO Vacuumed table table=pending_objects45072026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads45082026/11/12 00:09:51 INFO Vacuumed table table=closures45092026/11/12 00:09:51 INFO Vacuumed table table=objects45102026/11/12 00:09:51 INFO Received push request method=POST path=/45112026/11/12 00:09:51 INFO Received uploads request method=POST path=/45122026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45132026/11/12 00:09:51 INFO Received uploads request method=POST path=/45142026/11/12 00:09:51 INFO Completed upload id=25 objects=16 duration=1.518769ms45152026/11/12 00:09:51 INFO Received uploads request method=POST path=/45162026/11/12 00:09:51 INFO Received complete push request method=POST path=/45172026/11/12 00:09:51 INFO Received complete push request method=POST path=/45182026/11/12 00:09:51 INFO Completed push id=19 objects=3 duration=1.36288ms45192026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=045202026/11/12 00:09:51 INFO Completed push id=32 objects=9 duration=1.298269ms45212026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045222026/11/12 00:09:51 INFO Vacuumed table table=pending_closures45232026/11/12 00:09:51 INFO Vacuumed table table=pending_objects45242026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads45252026/11/12 00:09:51 INFO Vacuumed table table=closures45262026/11/12 00:09:51 INFO Vacuumed table table=objects45272026/11/12 00:09:51 INFO Received push request method=POST path=/45282026/11/12 00:09:51 INFO Received push request method=POST path=/45292026/11/12 00:09:51 INFO Received complete push request method=POST path=/45302026/11/12 00:09:51 INFO Completed push id=26 objects=23 duration=1.450689ms45312026/11/12 00:09:51 INFO Received push request method=POST path=/45322026/11/12 00:09:51 INFO Received push request method=POST path=/45332026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=045342026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period45352026/11/12 00:09:51 INFO Received push request method=POST path=/45362026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045372026/11/12 00:09:51 INFO Vacuumed table table=pending_closures45382026/11/12 00:09:51 INFO Vacuumed table table=pending_objects45392026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads45402026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45412026/11/12 00:09:51 INFO Vacuumed table table=closures45422026/11/12 00:09:51 INFO Received push request method=POST path=/45432026/11/12 00:09:51 INFO Completed upload id=24 objects=18 duration=874.11µs45442026/11/12 00:09:51 INFO Vacuumed table table=objects45452026/11/12 00:09:51 INFO Received complete push request method=POST path=/45462026/11/12 00:09:51 INFO Completed push id=7 objects=7 duration=1.76463ms45472026/11/12 00:09:51 INFO Received complete push request method=POST path=/45482026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45492026/11/12 00:09:51 INFO Completed upload id=33 objects=9 duration=1.034089ms45502026/11/12 00:09:51 INFO Completed push id=24 objects=3 duration=1.585519ms45512026/11/12 00:09:51 INFO Received complete push request method=POST path=/45522026/11/12 00:09:51 INFO Received uploads request method=POST path=/45532026/11/12 00:09:51 INFO Completed push id=20 objects=4 duration=1.01394ms45542026/11/12 00:09:51 INFO Received push request method=POST path=/45552026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45562026/11/12 00:09:51 INFO Completed upload id=2 objects=7 duration=1.982739ms45572026/11/12 00:09:51 INFO Received push request method=POST path=/45582026/11/12 00:09:51 INFO Received complete push request method=POST path=/45592026/11/12 00:09:51 INFO Completed push id=25 objects=2 duration=981.999µs45602026/11/12 00:09:51 INFO Received uploads request method=POST path=/45612026/11/12 00:09:51 INFO Received push request method=POST path=/45622026/11/12 00:09:51 INFO Received complete push request method=POST path=/45632026/11/12 00:09:51 INFO Received push request method=POST path=/45642026/11/12 00:09:51 INFO Completed push id=12 objects=6 duration=1.072829ms45652026/11/12 00:09:51 INFO Received complete push request method=POST path=/45662026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=045672026/11/12 00:09:51 INFO Completed push id=27 objects=15 duration=1.158049ms45682026/11/12 00:09:51 INFO Received uploads request method=POST path=/45692026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period45702026/11/12 00:09:51 INFO Received uploads request method=POST path=/45712026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045722026/11/12 00:09:51 INFO Vacuumed table table=pending_closures45732026/11/12 00:09:51 INFO Received complete push request method=POST path=/45742026/11/12 00:09:51 INFO Vacuumed table table=pending_objects45752026/11/12 00:09:51 INFO Completed push id=27 objects=11 duration=1.20039ms45762026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads45772026/11/12 00:09:51 INFO Received uploads request method=POST path=/45782026/11/12 00:09:51 INFO Received complete push request method=POST path=/45792026/11/12 00:09:51 INFO Vacuumed table table=closures45802026/11/12 00:09:51 INFO Vacuumed table table=objects45812026/11/12 00:09:51 INFO Received complete push request method=POST path=/45822026/11/12 00:09:51 INFO Completed push id=16 objects=26 duration=1.36111ms45832026/11/12 00:09:51 INFO Completed push id=27 objects=4 duration=578.509µs45842026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45852026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45862026/11/12 00:09:51 INFO Completed upload id=25 objects=6 duration=1.372199ms45872026/11/12 00:09:51 INFO Completed upload id=22 objects=8 duration=1.15068ms45882026/11/12 00:09:51 INFO Received push request method=POST path=/45892026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45902026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=045912026/11/12 00:09:51 INFO Completed upload id=29 objects=4 duration=1.920799ms45922026/11/12 00:09:51 INFO Received complete upload request method=POST path=/45932026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period45942026/11/12 00:09:51 INFO Received uploads request method=POST path=/45952026/11/12 00:09:51 INFO Completed upload id=29 objects=26 duration=919.079µs45962026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=045972026/11/12 00:09:51 INFO Vacuumed table table=pending_closures45982026/11/12 00:09:51 INFO Vacuumed table table=pending_objects45992026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads46002026/11/12 00:09:51 INFO Vacuumed table table=closures46012026/11/12 00:09:51 INFO Vacuumed table table=objects46022026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46032026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=046042026/11/12 00:09:51 INFO Completed upload id=8 objects=6 duration=1.2564ms46052026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=046062026/11/12 00:09:51 INFO Received complete push request method=POST path=/46072026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period46082026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period46092026/11/12 00:09:51 INFO Completed push id=23 objects=5 duration=1.004559ms46102026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046112026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=4 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046122026/11/12 00:09:51 INFO Vacuumed table table=pending_closures46132026/11/12 00:09:51 INFO Vacuumed table table=pending_closures46142026/11/12 00:09:51 INFO Vacuumed table table=pending_objects46152026/11/12 00:09:51 INFO Vacuumed table table=pending_objects46162026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads46172026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads46182026/11/12 00:09:51 INFO Vacuumed table table=closures46192026/11/12 00:09:51 INFO Vacuumed table table=closures46202026/11/12 00:09:51 INFO Vacuumed table table=objects46212026/11/12 00:09:51 INFO Vacuumed table table=objects46222026/11/12 00:09:51 INFO Received uploads request method=POST path=/46232026/11/12 00:09:51 INFO Received complete push request method=POST path=/46242026/11/12 00:09:51 INFO Completed push id=28 objects=26 duration=1.22627ms46252026/11/12 00:09:51 INFO Received complete push request method=POST path=/46262026/11/12 00:09:51 INFO Received uploads request method=POST path=/46272026/11/12 00:09:51 INFO Received complete push request method=POST path=/46282026/11/12 00:09:51 INFO Completed push id=13 objects=22 duration=1.497129ms46292026/11/12 00:09:51 INFO Completed push id=30 objects=8 duration=915.83µs46302026/11/12 00:09:51 INFO Received complete push request method=POST path=/46312026/11/12 00:09:51 INFO Received push request method=POST path=/46322026/11/12 00:09:51 INFO Completed push id=28 objects=19 duration=1.215339ms46332026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46342026/11/12 00:09:51 INFO Completed upload id=24 objects=8 duration=848.95µs46352026/11/12 00:09:51 INFO Received complete push request method=POST path=/46362026/11/12 00:09:51 INFO Received uploads request method=POST path=/46372026/11/12 00:09:51 INFO Completed push id=27 objects=6 duration=2.970429ms46382026/11/12 00:09:51 INFO Received complete push request method=POST path=/46392026/11/12 00:09:51 INFO Received complete push request method=POST path=/46402026/11/12 00:09:51 INFO Received uploads request method=POST path=/46412026/11/12 00:09:51 INFO Completed push id=30 objects=3 duration=808.529µs46422026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46432026/11/12 00:09:51 INFO Completed push id=21 objects=6 duration=1.22115ms46442026/11/12 00:09:51 INFO Received push request method=POST path=/46452026/11/12 00:09:51 INFO Completed upload id=31 objects=5 duration=850.75µs46462026/11/12 00:09:51 INFO Received uploads request method=POST path=/46472026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46482026/11/12 00:09:51 INFO Completed upload id=28 objects=5 duration=939.78µs46492026/11/12 00:09:51 INFO Received complete push request method=POST path=/46502026/11/12 00:09:51 INFO Completed push id=29 objects=10 duration=1.05599ms46512026/11/12 00:09:51 INFO Received uploads request method=POST path=/46522026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46532026/11/12 00:09:51 INFO Received complete push request method=POST path=/46542026/11/12 00:09:51 INFO Completed upload id=12 objects=9 duration=808.709µs46552026/11/12 00:09:51 INFO Completed push id=32 objects=19 duration=1.01845ms46562026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46572026/11/12 00:09:51 INFO Completed upload id=22 objects=7 duration=1.030339ms46582026/11/12 00:09:51 INFO Received uploads request method=POST path=/46592026/11/12 00:09:51 INFO Received complete push request method=POST path=/46602026/11/12 00:09:51 INFO Completed push id=32 objects=3 duration=617.92µs46612026/11/12 00:09:51 INFO Received complete push request method=POST path=/46622026/11/12 00:09:51 INFO Received complete push request method=POST path=/46632026/11/12 00:09:51 INFO Completed push id=11 objects=9 duration=1.73674ms46642026/11/12 00:09:51 INFO Completed push id=11 objects=6 duration=2.032849ms46652026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46662026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=046672026/11/12 00:09:51 INFO Completed upload id=23 objects=7 duration=3.087769ms46682026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46692026/11/12 00:09:51 INFO Completed upload id=30 objects=5 duration=986.82µs46702026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=046712026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period46722026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46732026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046742026/11/12 00:09:51 INFO Completed upload id=15 objects=2 duration=971.94µs46752026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=2 objects-failed-to-delete=046762026/11/12 00:09:51 INFO Vacuumed table table=pending_closures46772026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46782026/11/12 00:09:51 INFO Received complete push request method=POST path=/46792026/11/12 00:09:51 INFO Vacuumed table table=pending_closures46802026/11/12 00:09:51 INFO Vacuumed table table=pending_objects46812026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads46822026/11/12 00:09:51 INFO Completed push id=24 objects=4 duration=1.1775ms46832026/11/12 00:09:51 INFO Received complete upload request method=POST path=/46842026/11/12 00:09:51 INFO Received push request method=POST path=/46852026/11/12 00:09:51 INFO Vacuumed table table=closures46862026/11/12 00:09:51 INFO Vacuumed table table=pending_objects46872026/11/12 00:09:51 INFO Completed upload id=22 objects=5 duration=2.202079ms46882026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads46892026/11/12 00:09:51 INFO Completed upload id=30 objects=6 duration=910.989µs46902026/11/12 00:09:51 INFO Vacuumed table table=objects46912026/11/12 00:09:51 INFO Vacuumed table table=closures46922026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=046932026/11/12 00:09:51 INFO Vacuumed table table=objects46942026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period46952026/11/12 00:09:51 INFO Received complete push request method=POST path=/46962026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=046972026/11/12 00:09:51 INFO Received complete push request method=POST path=/46982026/11/12 00:09:51 INFO Completed push id=15 objects=24 duration=1.543569ms46992026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47002026/11/12 00:09:51 INFO Completed push id=20 objects=13 duration=1.51564ms47012026/11/12 00:09:51 INFO Vacuumed table table=pending_closures47022026/11/12 00:09:51 INFO Completed upload id=26 objects=5 duration=1.246809ms47032026/11/12 00:09:51 INFO Vacuumed table table=pending_objects47042026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads47052026/11/12 00:09:51 INFO Vacuumed table table=closures47062026/11/12 00:09:51 INFO Vacuumed table table=objects47072026/11/12 00:09:51 INFO Received complete push request method=POST path=/47082026/11/12 00:09:51 INFO Completed push id=31 objects=3 duration=1.34839ms47092026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47102026/11/12 00:09:51 INFO Received complete push request method=POST path=/47112026/11/12 00:09:51 INFO Completed upload id=31 objects=5 duration=641.99µs47122026/11/12 00:09:51 INFO Completed push id=25 objects=9 duration=1.016ms47132026/11/12 00:09:51 INFO Received complete push request method=POST path=/47142026/11/12 00:09:51 INFO Completed push id=28 objects=3 duration=1.840049ms47152026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47162026/11/12 00:09:51 INFO Completed upload id=19 objects=2 duration=1.115169ms47172026/11/12 00:09:51 INFO Received complete push request method=POST path=/47182026/11/12 00:09:51 INFO Completed push id=22 objects=4 duration=1.303919ms47192026/11/12 00:09:51 INFO Received complete push request method=POST path=/47202026/11/12 00:09:51 INFO Completed push id=27 objects=3 duration=1.615259ms47212026/11/12 00:09:51 INFO Received uploads request method=POST path=/47222026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47232026/11/12 00:09:51 INFO Completed upload id=34 objects=3 duration=690.879µs47242026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47252026/11/12 00:09:51 INFO Completed upload id=26 objects=3 duration=723.26µs47262026/11/12 00:09:51 INFO Received complete push request method=POST path=/47272026/11/12 00:09:51 INFO Completed push id=29 objects=15 duration=1.27461ms47282026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47292026/11/12 00:09:51 INFO Completed upload id=24 objects=7 duration=1.06048ms47302026/11/12 00:09:51 INFO Received complete push request method=POST path=/47312026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47322026/11/12 00:09:51 INFO Completed upload id=28 objects=5 duration=873.33µs47332026/11/12 00:09:51 INFO Completed push id=32 objects=6 duration=1.34928ms47342026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47352026/11/12 00:09:51 INFO Completed upload id=26 objects=2 duration=1.00014ms47362026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47372026/11/12 00:09:51 INFO Completed upload id=23 objects=7 duration=707.749µs47382026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=047392026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period47402026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047412026/11/12 00:09:51 INFO Received complete push request method=POST path=/47422026/11/12 00:09:51 INFO Received complete push request method=POST path=/47432026/11/12 00:09:51 INFO Vacuumed table table=pending_closures47442026/11/12 00:09:51 INFO Received complete push request method=POST path=/47452026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47462026/11/12 00:09:51 INFO Completed push id=26 objects=3 duration=1.10853ms47472026/11/12 00:09:51 INFO Completed push id=30 objects=6 duration=1.20786ms47482026/11/12 00:09:51 INFO Vacuumed table table=pending_objects47492026/11/12 00:09:51 INFO Completed upload id=27 objects=7 duration=814.96µs47502026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads47512026/11/12 00:09:51 INFO Completed push id=16 objects=7 duration=2.340129ms47522026/11/12 00:09:51 INFO Vacuumed table table=closures47532026/11/12 00:09:51 INFO Vacuumed table table=objects47542026/11/12 00:09:51 INFO Received complete push request method=POST path=/47552026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=047562026/11/12 00:09:51 INFO Completed push id=29 objects=17 duration=1.40476ms47572026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47582026/11/12 00:09:51 INFO Received push request method=POST path=/47592026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047602026/11/12 00:09:51 INFO Completed upload id=33 objects=7 duration=1.05177ms47612026/11/12 00:09:51 INFO Received complete push request method=POST path=/47622026/11/12 00:09:51 INFO Vacuumed table table=pending_closures47632026/11/12 00:09:51 INFO Completed push id=24 objects=5 duration=1.09822ms47642026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47652026/11/12 00:09:51 INFO Vacuumed table table=pending_objects47662026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads47672026/11/12 00:09:51 INFO Completed upload id=18 objects=22 duration=1.387049ms47682026/11/12 00:09:51 INFO Vacuumed table table=closures47692026/11/12 00:09:51 INFO Vacuumed table table=objects47702026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47712026/11/12 00:09:51 INFO Completed upload id=28 objects=8 duration=895.08µs47722026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47732026/11/12 00:09:51 INFO Completed upload id=28 objects=3 duration=964.609µs47742026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47752026/11/12 00:09:51 INFO Completed upload id=31 objects=3 duration=1.10606ms47762026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=047772026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47782026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period47792026/11/12 00:09:51 INFO Completed upload id=34 objects=3 duration=1.031699ms47802026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=1 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=047812026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47822026/11/12 00:09:51 INFO Vacuumed table table=pending_closures47832026/11/12 00:09:51 INFO Completed upload id=25 objects=2 duration=811.489µs47842026/11/12 00:09:51 INFO Received uploads request method=POST path=/47852026/11/12 00:09:51 INFO Vacuumed table table=pending_objects47862026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads47872026/11/12 00:09:51 INFO Vacuumed table table=closures47882026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47892026/11/12 00:09:51 INFO Vacuumed table table=objects47902026/11/12 00:09:51 INFO Completed upload id=21 objects=8 duration=896.91µs47912026/11/12 00:09:51 INFO Received complete push request method=POST path=/47922026/11/12 00:09:51 INFO Completed push id=23 objects=6 duration=1.389609ms47932026/11/12 00:09:51 INFO Received complete upload request method=POST path=/47942026/11/12 00:09:51 INFO Completed upload id=29 objects=10 duration=994.78µs47952026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=047962026/11/12 00:09:51 INFO Received complete push request method=POST path=/47972026/11/12 00:09:51 INFO Received complete push request method=POST path=/47982026/11/12 00:09:51 INFO Completed push id=35 objects=11 duration=1.138019ms47992026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period48002026/11/12 00:09:51 INFO Completed push id=29 objects=23 duration=1.51287ms48012026/11/12 00:09:51 INFO Received push request method=POST path=/48022026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048032026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48042026/11/12 00:09:51 INFO Completed upload id=26 objects=20 duration=1.208369ms48052026/11/12 00:09:51 INFO Vacuumed table table=pending_closures48062026/11/12 00:09:51 INFO Received uploads request method=POST path=/48072026/11/12 00:09:51 INFO Vacuumed table table=pending_objects48082026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads48092026/11/12 00:09:51 INFO Received complete push request method=POST path=/48102026/11/12 00:09:51 INFO Vacuumed table table=closures48112026/11/12 00:09:51 INFO Vacuumed table table=objects48122026/11/12 00:09:51 INFO Completed push id=24 objects=6 duration=1.345209ms48132026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48142026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48152026/11/12 00:09:51 INFO Completed upload id=30 objects=20 duration=811.18µs48162026/11/12 00:09:51 INFO Completed upload id=22 objects=7 duration=1.048599ms48172026/11/12 00:09:51 INFO Received complete push request method=POST path=/48182026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=048192026/11/12 00:09:51 INFO Completed push id=30 objects=6 duration=1.25824ms48202026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period48212026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48222026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=6 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048232026/11/12 00:09:51 INFO Completed upload id=26 objects=8 duration=1.34572ms48242026/11/12 00:09:51 INFO Vacuumed table table=pending_closures48252026/11/12 00:09:51 INFO Received complete push request method=POST path=/48262026/11/12 00:09:51 INFO Received complete push request method=POST path=/48272026/11/12 00:09:51 INFO Vacuumed table table=pending_objects48282026/11/12 00:09:51 INFO Received complete push request method=POST path=/48292026/11/12 00:09:51 INFO Completed push id=31 objects=11 duration=1.315259ms48302026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads48312026/11/12 00:09:51 INFO Completed push id=28 objects=7 duration=1.755279ms48322026/11/12 00:09:51 INFO Vacuumed table table=closures48332026/11/12 00:09:51 INFO Completed push id=24 objects=5 duration=1.6652ms48342026/11/12 00:09:51 INFO Vacuumed table table=objects48352026/11/12 00:09:51 INFO Received complete push request method=POST path=/48362026/11/12 00:09:51 INFO Completed push id=25 objects=7 duration=1.414479ms48372026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=048382026/11/12 00:09:51 INFO Received complete push request method=POST path=/48392026/11/12 00:09:51 INFO Completed push id=28 objects=14 duration=1.347739ms48402026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48412026/11/12 00:09:51 INFO Completed upload id=32 objects=2 duration=685.699µs48422026/11/12 00:09:51 INFO Received complete push request method=POST path=/48432026/11/12 00:09:51 INFO Completed push id=33 objects=3 duration=1.06886ms48442026/11/12 00:09:51 ERROR failed to remove object object=cccccccccccccccccccccccccccccccc.ls error="Access Denied."48452026/11/12 00:09:51 ERROR failed to remove object object=ffffffffffffffffffffffffffffffff.ls error="Access Denied."48462026/11/12 00:09:51 ERROR failed to remove object object=realisations/sha256:ffffffff!out.doi error="Access Denied."48472026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48482026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48492026/11/12 00:09:51 INFO Completed upload id=30 objects=6 duration=1.00422ms48502026/11/12 00:09:51 INFO Completed upload id=25 objects=8 duration=761.089µs48512026/11/12 00:09:51 INFO Received push request method=POST path=/48522026/11/12 00:09:51 INFO Received complete push request method=POST path=/48532026/11/12 00:09:51 INFO Completed push id=33 objects=3 duration=936.42µs48542026/11/12 00:09:51 INFO Received complete push request method=POST path=/48552026/11/12 00:09:51 INFO Completed push id=34 objects=6 duration=1.05598ms48562026/11/12 00:09:51 INFO Received push request method=POST path=/48572026/11/12 00:09:51 INFO Received complete push request method=POST path=/48582026/11/12 00:09:51 INFO Completed push id=31 objects=5 duration=1.023809ms48592026/11/12 00:09:51 INFO Received complete push request method=POST path=/48602026/11/12 00:09:51 INFO Completed push id=26 objects=4 duration=627.09µs48612026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48622026/11/12 00:09:51 INFO Completed upload id=26 objects=24 duration=831.34µs48632026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48642026/11/12 00:09:51 INFO Completed upload id=34 objects=3 duration=656.63µs48652026/11/12 00:09:51 INFO Received uploads request method=POST path=/48662026/11/12 00:09:51 INFO Received complete push request method=POST path=/48672026/11/12 00:09:51 INFO Completed push id=35 objects=8 duration=1.012409ms48682026/11/12 00:09:51 INFO Received complete push request method=POST path=/48692026/11/12 00:09:51 INFO Completed push id=30 objects=2 duration=718.58µs48702026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=048712026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period48722026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048732026/11/12 00:09:51 INFO Received push request method=POST path=/48742026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=048752026/11/12 00:09:51 INFO Vacuumed table table=pending_closures48762026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48772026/11/12 00:09:51 INFO Completed upload id=36 objects=9 duration=969.15µs48782026/11/12 00:09:51 INFO Received complete push request method=POST path=/48792026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period48802026/11/12 00:09:51 INFO Vacuumed table table=pending_objects48812026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads48822026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=048832026/11/12 00:09:51 INFO Completed push id=32 objects=26 duration=1.46343ms48842026/11/12 00:09:51 INFO Vacuumed table table=closures48852026/11/12 00:09:51 INFO Vacuumed table table=pending_closures48862026/11/12 00:09:51 INFO Vacuumed table table=objects48872026/11/12 00:09:51 INFO Vacuumed table table=pending_objects48882026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads48892026/11/12 00:09:51 INFO Vacuumed table table=closures48902026/11/12 00:09:51 INFO Vacuumed table table=objects48912026/11/12 00:09:51 INFO Received complete push request method=POST path=/48922026/11/12 00:09:51 INFO Received complete push request method=POST path=/48932026/11/12 00:09:51 INFO Completed push id=28 objects=11 duration=2.438559ms48942026/11/12 00:09:51 INFO Received complete upload request method=POST path=/48952026/11/12 00:09:51 INFO Completed push id=32 objects=7 duration=803.72µs48962026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=048972026/11/12 00:09:51 INFO Completed upload id=27 objects=26 duration=751.29µs48982026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period48992026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49002026/11/12 00:09:51 INFO Completed upload id=33 objects=2 duration=883.85µs49012026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049022026/11/12 00:09:51 INFO Vacuumed table table=pending_closures49032026/11/12 00:09:51 INFO Received uploads request method=POST path=/49042026/11/12 00:09:51 INFO Vacuumed table table=pending_objects49052026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads49062026/11/12 00:09:51 INFO Vacuumed table table=closures49072026/11/12 00:09:51 INFO Vacuumed table table=objects49082026/11/12 00:09:51 INFO Received complete push request method=POST path=/49092026/11/12 00:09:51 INFO Completed push id=28 objects=6 duration=696.949µs49102026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49112026/11/12 00:09:51 INFO Completed upload id=34 objects=5 duration=861.979µs49122026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49132026/11/12 00:09:51 INFO Completed upload id=33 objects=6 duration=687.989µs49142026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49152026/11/12 00:09:51 INFO Completed upload id=21 objects=9 duration=1.740589ms49162026/11/12 00:09:51 INFO Received complete push request method=POST path=/49172026/11/12 00:09:51 INFO Completed push id=29 objects=6 duration=958.97µs49182026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49192026/11/12 00:09:51 INFO Completed upload id=35 objects=11 duration=918.54µs49202026/11/12 00:09:51 INFO Received complete push request method=POST path=/49212026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49222026/11/12 00:09:51 INFO Completed upload id=34 objects=3 duration=565.57µs49232026/11/12 00:09:51 INFO Completed push id=29 objects=24 duration=1.633489ms49242026/11/12 00:09:51 INFO Received complete push request method=POST path=/49252026/11/12 00:09:51 INFO Completed push id=25 objects=10 duration=1.675049ms49262026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49272026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=049282026/11/12 00:09:51 INFO Completed upload id=36 objects=3 duration=915.83µs49292026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period49302026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=049312026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period49322026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049332026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49342026/11/12 00:09:51 INFO Completed upload id=29 objects=5 duration=970.34µs49352026/11/12 00:09:51 INFO Vacuumed table table=pending_closures49362026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049372026/11/12 00:09:51 INFO Vacuumed table table=pending_objects49382026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads49392026/11/12 00:09:51 INFO Vacuumed table table=pending_closures49402026/11/12 00:09:51 INFO Vacuumed table table=closures49412026/11/12 00:09:51 INFO Vacuumed table table=pending_objects49422026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads49432026/11/12 00:09:51 INFO Vacuumed table table=objects49442026/11/12 00:09:51 INFO Vacuumed table table=closures49452026/11/12 00:09:51 INFO Vacuumed table table=objects49462026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49472026/11/12 00:09:51 INFO Completed upload id=30 objects=8 duration=1.207789ms49482026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=049492026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period49502026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49512026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049522026/11/12 00:09:51 INFO Completed upload id=30 objects=2 duration=970.019µs49532026/11/12 00:09:51 INFO Vacuumed table table=pending_closures49542026/11/12 00:09:51 INFO Vacuumed table table=pending_objects49552026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads49562026/11/12 00:09:51 INFO Vacuumed table table=closures49572026/11/12 00:09:51 INFO Vacuumed table table=objects49582026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49592026/11/12 00:09:51 INFO Completed upload id=31 objects=6 duration=977.939µs49602026/11/12 00:09:51 INFO Received complete push request method=POST path=/49612026/11/12 00:09:51 INFO Completed push id=31 objects=20 duration=1.10609ms49622026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49632026/11/12 00:09:51 INFO Completed upload id=32 objects=4 duration=614.09µs49642026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49652026/11/12 00:09:51 INFO Completed upload id=32 objects=5 duration=863.07µs49662026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49672026/11/12 00:09:51 INFO Completed upload id=33 objects=3 duration=653.54µs49682026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49692026/11/12 00:09:51 INFO Completed upload id=33 objects=10 duration=719.5µs49702026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=049712026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period49722026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=049732026/11/12 00:09:51 INFO Vacuumed table table=pending_closures49742026/11/12 00:09:51 INFO Vacuumed table table=pending_objects49752026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads49762026/11/12 00:09:51 INFO Vacuumed table table=closures49772026/11/12 00:09:51 INFO Vacuumed table table=objects49782026/11/12 00:09:51 INFO Received complete upload request method=POST path=/49792026/11/12 00:09:51 INFO Completed upload id=34 objects=23 duration=1.3514ms49802026/11/12 00:09:51 INFO Received complete push request method=POST path=/49812026/11/12 00:09:51 INFO Completed push id=35 objects=6 duration=1.1212ms49822026/11/12 00:09:51 INFO Aborted multipart uploads count=0 kept=049832026/11/12 00:09:51 WARN Force mode enabled - objects will be deleted immediately without grace period49842026/11/12 00:09:51 INFO Garbage collection completed failed-uploads-deleted=2 old-closures-deleted=1 objects-marked-for-deletion=0 objects-deleted-after-grace-period=1 objects-failed-to-delete=049852026/11/12 00:09:51 INFO Vacuumed table table=pending_closures49862026/11/12 00:09:51 INFO Vacuumed table table=pending_objects49872026/11/12 00:09:51 INFO Vacuumed table table=multipart_uploads49882026/11/12 00:09:51 INFO Vacuumed table table=closures49892026/11/12 00:09:51 INFO Vacuumed table table=objects4990--- PASS: FuzzPushGC (0.00s)4991 --- PASS: FuzzPushGC/seed#14 (2.70s)4992 --- PASS: FuzzPushGC/seed#9 (2.72s)4993 --- PASS: FuzzPushGC/seed#0 (2.82s)4994 --- PASS: FuzzPushGC/seed#7 (2.92s)4995 --- PASS: FuzzPushGC/seed#3 (2.93s)4996 --- PASS: FuzzPushGC/seed#13 (2.98s)4997 --- PASS: FuzzPushGC/seed#1 (3.01s)4998 --- PASS: FuzzPushGC/seed#15 (3.03s)4999 --- PASS: FuzzPushGC/seed#8 (3.05s)5000 --- PASS: FuzzPushGC/seed#5 (3.11s)5001 --- PASS: FuzzPushGC/seed#4 (3.13s)5002 --- PASS: FuzzPushGC/seed#2 (3.18s)5003 --- PASS: FuzzPushGC/seed#10 (3.18s)5004 --- PASS: FuzzPushGC/seed#11 (3.20s)5005 --- PASS: FuzzPushGC/seed#6 (3.27s)5006 --- PASS: FuzzPushGC/seed#12 (3.34s)5007PASS5008{"timestamp":"2026-11-12T00:09:51.846525493Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:60126","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(396)"}5009{"timestamp":"2026-11-12T00:09:51.846711943Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:59436","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(395)"}5010{"timestamp":"2026-11-12T00:09:51.847125162Z","level":"ERROR","message":"HTTP transport failed","event":"http_transport_failed","component":"server","subsystem":"transport","peer_addr":"127.0.0.1:60142","error_kind":"io_error","error":"Cancelled","result":"transport_error","target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":2561,"threadName":"rustfs-worker","threadId":"ThreadId(387)"}50112026-11-12 00:09:52.273 UTC [129] LOG: received smart shutdown request50122026-11-12 00:09:52.278 UTC [129] LOG: background worker "logical replication launcher" (PID 139) exited with exit code 150132026-11-12 00:09:53.149 UTC [134] LOG: checkpoint complete: wrote 4001 buffers (24.4%), wrote 5 SLRU buffers; 0 WAL file(s) added, 0 removed, 33 recycled; write=3.846 s, sync=0.719 s, total=4.629 s; sync files=42364, longest=0.002 s, average=0.001 s; distance=534005 kB, estimate=534005 kB; lsn=0/26D36BA0, redo lsn=0/220E313050142026-11-12 00:09:53.152 UTC [134] LOG: shutting down50152026-11-12 00:09:53.152 UTC [134] LOG: checkpoint starting: shutdown immediate50162026-11-12 00:09:53.991 UTC [134] LOG: checkpoint complete: wrote 9458 buffers (57.7%), wrote 2 SLRU buffers; 0 WAL file(s) added, 0 removed, 4 recycled; write=0.165 s, sync=0.659 s, total=0.840 s; sync files=2820, longest=0.002 s, average=0.001 s; distance=78158 kB, estimate=488420 kB; lsn=0/26D36C18, redo lsn=0/26D36C1850172026-11-12 00:09:54.062 UTC [129] LOG: database system is shut down5018Running OIDC tests...5019=== RUN TestAudienceForIssuer5020=== PAUSE TestAudienceForIssuer5021=== RUN TestProviderClientGivesUpOnSilentIssuer5022=== PAUSE TestProviderClientGivesUpOnSilentIssuer5023=== RUN TestGlobMatch5024=== PAUSE TestGlobMatch5025=== RUN TestValidateToken_ValidToken5026=== PAUSE TestValidateToken_ValidToken5027=== RUN TestValidateToken_WrongAudience5028=== PAUSE TestValidateToken_WrongAudience5029=== RUN TestValidateToken_Expired5030=== PAUSE TestValidateToken_Expired5031=== RUN TestValidateToken_BoundClaimsMismatch5032=== PAUSE TestValidateToken_BoundClaimsMismatch5033=== RUN TestValidateToken_BoundSubjectMismatch5034=== PAUSE TestValidateToken_BoundSubjectMismatch5035=== RUN TestValidateToken_MultipleProviders5036=== PAUSE TestValidateToken_MultipleProviders5037=== RUN TestValidateToken_UnknownIssuerFetchesNoKeys5038=== PAUSE TestValidateToken_UnknownIssuerFetchesNoKeys5039=== RUN TestValidateToken_GoogleSchemelessIssuer5040=== PAUSE TestValidateToken_GoogleSchemelessIssuer5041=== RUN TestValidateToken_KubernetesServiceAccount5042=== PAUSE TestValidateToken_KubernetesServiceAccount5043=== RUN TestNewValidator_KubernetesRequiresCA5044=== PAUSE TestNewValidator_KubernetesRequiresCA5045=== RUN TestValidateToken_KubernetesIssuerFromOwnToken5046=== PAUSE TestValidateToken_KubernetesIssuerFromOwnToken5047=== RUN TestPins_ReservedForMatchingRule5048=== PAUSE TestPins_ReservedForMatchingRule5049=== RUN TestPins_TopLevelShorthand5050=== PAUSE TestPins_TopLevelShorthand5051=== RUN TestPins_ConfigValidation5052=== PAUSE TestPins_ConfigValidation5053=== RUN TestScopes_LegacyProviderDefaultsToWrite5054=== PAUSE TestScopes_LegacyProviderDefaultsToWrite5055=== RUN TestScopes_Rules5056=== PAUSE TestScopes_Rules5057=== RUN TestScopes_ConfigValidation5058=== PAUSE TestScopes_ConfigValidation5059=== CONT TestAudienceForIssuer5060=== CONT TestValidateToken_GoogleSchemelessIssuer5061--- PASS: TestAudienceForIssuer (0.00s)5062=== CONT TestGlobMatch5063=== CONT TestProviderClientGivesUpOnSilentIssuer5064=== CONT TestPins_TopLevelShorthand5065=== CONT TestScopes_ConfigValidation5066=== CONT TestScopes_Rules5067=== CONT TestScopes_LegacyProviderDefaultsToWrite5068=== CONT TestPins_ConfigValidation5069=== CONT TestValidateToken_WrongAudience5070=== CONT TestValidateToken_MultipleProviders5071=== CONT TestValidateToken_UnknownIssuerFetchesNoKeys5072=== CONT TestValidateToken_KubernetesIssuerFromOwnToken5073=== CONT TestPins_ReservedForMatchingRule5074=== CONT TestValidateToken_BoundSubjectMismatch50752026/11/12 00:09:56 INFO OIDC provider initialized name=known issuer=https://known.example5076=== CONT TestValidateToken_BoundClaimsMismatch5077=== CONT TestNewValidator_KubernetesRequiresCA5078=== CONT TestValidateToken_KubernetesServiceAccount5079=== CONT TestValidateToken_Expired5080=== CONT TestValidateToken_ValidToken5081=== RUN TestGlobMatch/foo_foo5082=== PAUSE TestGlobMatch/foo_foo5083=== RUN TestGlobMatch/foo_bar5084=== PAUSE TestGlobMatch/foo_bar5085=== RUN TestGlobMatch/*_5086=== PAUSE TestGlobMatch/*_5087=== RUN TestGlobMatch/*_anything5088=== PAUSE TestGlobMatch/*_anything5089=== RUN TestGlobMatch/foo*_foo5090=== PAUSE TestGlobMatch/foo*_foo5091--- PASS: TestProviderClientGivesUpOnSilentIssuer (0.00s)5092=== RUN TestGlobMatch/foo*_foobar5093--- PASS: TestScopes_ConfigValidation (0.00s)5094--- PASS: TestPins_ConfigValidation (0.00s)5095=== PAUSE TestGlobMatch/foo*_foobar5096=== RUN TestGlobMatch/foo*_bar5097=== PAUSE TestGlobMatch/foo*_bar5098=== RUN TestGlobMatch/*bar_bar5099=== PAUSE TestGlobMatch/*bar_bar5100=== RUN TestGlobMatch/*bar_foobar5101=== PAUSE TestGlobMatch/*bar_foobar5102=== RUN TestGlobMatch/*bar_foo5103=== PAUSE TestGlobMatch/*bar_foo5104=== RUN TestGlobMatch/foo*bar_foobar5105=== PAUSE TestGlobMatch/foo*bar_foobar5106=== RUN TestGlobMatch/foo*bar_foo123bar5107=== PAUSE TestGlobMatch/foo*bar_foo123bar5108=== RUN TestGlobMatch/foo*bar_foobarbaz5109=== PAUSE TestGlobMatch/foo*bar_foobarbaz5110=== RUN TestGlobMatch/*/*_foo/bar5111=== PAUSE TestGlobMatch/*/*_foo/bar5112=== RUN TestGlobMatch/*/*_foo5113=== PAUSE TestGlobMatch/*/*_foo5114=== RUN TestGlobMatch/refs/heads/*_refs/heads/main5115=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main5116=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.05117=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.05118=== RUN TestGlobMatch/refs/*/main_refs/heads/main5119=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main5120=== RUN TestGlobMatch/fo?_foo5121=== PAUSE TestGlobMatch/fo?_foo5122=== RUN TestGlobMatch/fo?_fo5123=== PAUSE TestGlobMatch/fo?_fo5124=== RUN TestGlobMatch/fo?_fooo5125=== PAUSE TestGlobMatch/fo?_fooo5126=== RUN TestGlobMatch/?oo_foo5127=== PAUSE TestGlobMatch/?oo_foo5128=== RUN TestGlobMatch/?oo_boo5129=== PAUSE TestGlobMatch/?oo_boo5130=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5131=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5132=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5133=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5134=== CONT TestGlobMatch/foo_foo5135=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main5136=== CONT TestGlobMatch/*bar_foobar5137=== CONT TestGlobMatch/*bar_bar5138=== CONT TestGlobMatch/foo*_bar5139=== CONT TestGlobMatch/foo*_foobar5140=== CONT TestGlobMatch/foo*_foo5141=== CONT TestGlobMatch/*_anything5142=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main5143=== CONT TestGlobMatch/*_5144=== CONT TestGlobMatch/?oo_boo5145=== CONT TestGlobMatch/foo_bar5146=== CONT TestGlobMatch/fo?_foo5147=== CONT TestGlobMatch/?oo_foo5148=== CONT TestGlobMatch/fo?_fo5149=== CONT TestGlobMatch/refs/*/main_refs/heads/main5150=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.05151=== CONT TestGlobMatch/*/*_foo5152=== CONT TestGlobMatch/*/*_foo/bar5153=== CONT TestGlobMatch/foo*bar_foobarbaz5154=== CONT TestGlobMatch/foo*bar_foo123bar5155=== CONT TestGlobMatch/foo*bar_foobar5156=== CONT TestGlobMatch/*bar_foo5157=== CONT TestGlobMatch/fo?_fooo5158=== CONT TestGlobMatch/refs/heads/*_refs/heads/main5159--- PASS: TestGlobMatch (0.01s)5160 --- PASS: TestGlobMatch/foo_foo (0.00s)5161 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)5162 --- PASS: TestGlobMatch/*bar_foobar (0.00s)5163 --- PASS: TestGlobMatch/*bar_bar (0.00s)5164 --- PASS: TestGlobMatch/foo*_bar (0.00s)5165 --- PASS: TestGlobMatch/foo*_foobar (0.00s)5166 --- PASS: TestGlobMatch/foo*_foo (0.00s)5167 --- PASS: TestGlobMatch/*_anything (0.00s)5168 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)5169 --- PASS: TestGlobMatch/*_ (0.00s)5170 --- PASS: TestGlobMatch/?oo_boo (0.00s)5171 --- PASS: TestGlobMatch/foo_bar (0.00s)5172 --- PASS: TestGlobMatch/fo?_foo (0.00s)5173 --- PASS: TestGlobMatch/?oo_foo (0.00s)5174 --- PASS: TestGlobMatch/fo?_fo (0.00s)5175 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)5176 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)5177 --- PASS: TestGlobMatch/*/*_foo (0.00s)5178 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)5179 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)5180 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)5181 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)5182 --- PASS: TestGlobMatch/*bar_foo (0.00s)5183 --- PASS: TestGlobMatch/fo?_fooo (0.00s)5184 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)51852026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36971/oidc5186--- PASS: TestValidateToken_BoundClaimsMismatch (0.03s)51872026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42537/oidc51882026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:36987/oidc5189--- PASS: TestValidateToken_BoundSubjectMismatch (0.04s)5190--- PASS: TestValidateToken_ValidToken (0.04s)51912026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:42407/oidc51922026/11/12 00:09:56 INFO OIDC provider initialized name=kubernetes issuer=https://oidc.eks.invalid/id/ABC12351932026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:46437/oidc5194--- PASS: TestPins_ReservedForMatchingRule (0.05s)5195--- PASS: TestValidateToken_WrongAudience (0.05s)5196--- PASS: TestValidateToken_KubernetesIssuerFromOwnToken (0.05s)51972026/11/12 00:09:56 http: TLS handshake error from 127.0.0.1:59878: remote error: tls: bad certificate5198--- PASS: TestNewValidator_KubernetesRequiresCA (0.05s)51992026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37185/oidc5200--- PASS: TestScopes_LegacyProviderDefaultsToWrite (0.07s)52012026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:41987/oidc52022026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:37121/oidc5203--- PASS: TestValidateToken_Expired (0.06s)5204--- PASS: TestPins_TopLevelShorthand (0.07s)5205--- PASS: TestValidateToken_UnknownIssuerFetchesNoKeys (0.07s)52062026/11/12 00:09:56 INFO OIDC provider initialized name=test issuer=http://127.0.0.1:40473/oidc5207--- PASS: TestScopes_Rules (0.09s)52082026/11/12 00:09:56 INFO OIDC provider initialized name=kubernetes issuer=https://127.0.0.1:453615209--- PASS: TestValidateToken_KubernetesServiceAccount (0.11s)52102026/11/12 00:09:56 INFO OIDC provider initialized name=provider2 issuer=http://127.0.0.1:35425/oidc52112026/11/12 00:09:56 INFO OIDC provider initialized name=provider1 issuer=http://127.0.0.1:46151/oidc5212--- PASS: TestValidateToken_MultipleProviders (0.13s)52132026/11/12 00:09:56 INFO OIDC provider initialized name=google issuer=https://accounts.google.com5214--- PASS: TestValidateToken_GoogleSchemelessIssuer (0.18s)5215PASS5216Running hook tests...5217=== RUN TestSendPathsEmpty5218=== PAUSE TestSendPathsEmpty5219=== RUN TestQueueEnqueueAndFetch5220=== PAUSE TestQueueEnqueueAndFetch5221=== RUN TestQueueDeduplication5222=== PAUSE TestQueueDeduplication5223=== RUN TestQueueRemove5224=== PAUSE TestQueueRemove5225=== RUN TestQueueFetchBatchLimit5226=== PAUSE TestQueueFetchBatchLimit5227=== RUN TestQueueRetryMovesToBack5228=== PAUSE TestQueueRetryMovesToBack5229=== RUN TestQueueFetchRemoveLifecycle5230=== PAUSE TestQueueFetchRemoveLifecycle5231=== RUN TestQueueConcurrentWriters5232=== PAUSE TestQueueConcurrentWriters5233=== RUN TestQueueRemoveLargeClosure5234=== PAUSE TestQueueRemoveLargeClosure5235=== RUN TestServerClientIntegration5236--- PASS: TestServerClientIntegration (0.00s)5237=== RUN TestServerQueueError52382026/11/12 00:09:56 ERROR Failed to queue paths error="permission denied" count=15239--- PASS: TestServerQueueError (0.00s)5240=== RUN TestGetListenerSocketActivation5241 server_test.go:220: === RUN TestGetListenerSocketActivation5242 --- PASS: TestGetListenerSocketActivation (0.00s)5243 PASS5244 5245--- PASS: TestGetListenerSocketActivation (0.01s)5246=== RUN TestServerStalledClientDoesNotBlockShutdown5247=== PAUSE TestServerStalledClientDoesNotBlockShutdown5248=== RUN TestServerBacksOffOnAcceptErrors5249=== PAUSE TestServerBacksOffOnAcceptErrors5250=== RUN TestDrainIsolatesPoisonPath5251=== PAUSE TestDrainIsolatesPoisonPath5252=== RUN TestRunNotBlockedByPoisonHead5253=== PAUSE TestRunNotBlockedByPoisonHead5254=== RUN TestDrainGivesUpWhenServerDown5255=== PAUSE TestDrainGivesUpWhenServerDown5256=== RUN TestFailedPathPrunedByLaterClosure5257=== PAUSE TestFailedPathPrunedByLaterClosure5258=== RUN TestWorkerUploadsAndRemoves5259=== PAUSE TestWorkerUploadsAndRemoves5260=== RUN TestWorkerSkipsGCdPaths5261=== PAUSE TestWorkerSkipsGCdPaths5262=== RUN TestWorkerPrunesClosureDeps5263=== PAUSE TestWorkerPrunesClosureDeps5264=== RUN TestDrainTimeout5265=== PAUSE TestDrainTimeout5266=== RUN TestWorkerRemoveFailureIsNotProgress5267=== PAUSE TestWorkerRemoveFailureIsNotProgress5268=== RUN TestWorkerKeepsPathItCannotStat5269=== PAUSE TestWorkerKeepsPathItCannotStat5270=== CONT TestSendPathsEmpty5271=== CONT TestDrainGivesUpWhenServerDown5272--- PASS: TestSendPathsEmpty (0.00s)5273=== CONT TestWorkerKeepsPathItCannotStat5274=== CONT TestWorkerRemoveFailureIsNotProgress5275=== CONT TestDrainTimeout5276=== CONT TestWorkerPrunesClosureDeps5277=== CONT TestWorkerSkipsGCdPaths5278=== CONT TestWorkerUploadsAndRemoves5279=== CONT TestFailedPathPrunedByLaterClosure5280=== CONT TestDrainIsolatesPoisonPath5281=== CONT TestQueueRetryMovesToBack5282=== CONT TestQueueFetchBatchLimit5283=== CONT TestQueueRemove5284=== CONT TestQueueDeduplication5285=== CONT TestQueueFetchRemoveLifecycle5286=== CONT TestQueueEnqueueAndFetch5287=== CONT TestServerBacksOffOnAcceptErrors5288=== CONT TestQueueRemoveLargeClosure5289=== CONT TestServerStalledClientDoesNotBlockShutdown52902000/01/01 00:00:00 ERROR Accept failed error="too many open files"5291=== CONT TestQueueConcurrentWriters52922000/01/01 00:00:00 ERROR Accept failed error="too many open files"52932000/01/01 00:00:00 ERROR Accept failed error="too many open files"52942000/01/01 00:00:00 ERROR Accept failed error="too many open files"52952000/01/01 00:00:00 ERROR Accept failed error="too many open files"5296=== CONT TestRunNotBlockedByPoisonHead52972000/01/01 00:00:00 ERROR Accept failed error="too many open files"52982000/01/01 00:00:00 ERROR Accept failed error="too many open files"52992000/01/01 00:00:00 ERROR Accept failed error="too many open files"5300--- PASS: TestServerBacksOffOnAcceptErrors (0.00s)53012026/11/12 00:09:56 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2954497746/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2954497746/002/locked/aaa: permission denied"53022026/11/12 00:09:56 INFO Uploading batch count=253032026/11/12 00:09:56 INFO Upload queue status pending=253042026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=253052026/11/12 00:09:56 INFO Uploading batch count=453062026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=453072026/11/12 00:09:56 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths4198903785/002/nonexistent53082026/11/12 00:09:56 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2019822238/002/a53092026/11/12 00:09:56 INFO Uploading batch count=153102026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=153112026/11/12 00:09:56 INFO Upload queue status pending=253122026/11/12 00:09:56 INFO Upload queue status pending=253132026/11/12 00:09:56 INFO Upload queue status pending=353142000/01/01 00:00:00 INFO Uploading batch count=253152000/01/01 00:00:00 ERROR Upload failed error="context deadline exceeded" count=253162026/11/12 00:09:56 INFO Uploading batch count=153172000/01/01 00:00:00 ERROR Drain finished with paths left in queue remaining=453182026/11/12 00:09:56 INFO Uploading batch count=253192026/11/12 00:09:56 INFO Uploading batch count=153202026/11/12 00:09:56 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2954497746/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2954497746/002/locked/aaa: permission denied"53212026/11/12 00:09:56 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath1772192122/002/bbb53222026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=153232026/11/12 00:09:56 INFO Uploading batch count=15324--- PASS: TestQueueEnqueueAndFetch (0.02s)53252026/11/12 00:09:56 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2019822238/002/b5326--- PASS: TestQueueFetchBatchLimit (0.02s)5327--- PASS: TestQueueDeduplication (0.02s)5328--- PASS: TestQueueFetchRemoveLifecycle (0.02s)53292026/11/12 00:09:56 INFO Uploading batch count=153302026/11/12 00:09:56 WARN Cannot stat store path, will retry later path=/build/TestWorkerKeepsPathItCannotStat2954497746/002/locked/aaa error="stat /build/TestWorkerKeepsPathItCannotStat2954497746/002/locked/aaa: permission denied"5331--- PASS: TestQueueRemove (0.02s)53322026/11/12 00:09:56 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1127977299/002/nonexistent53332026/11/12 00:09:56 INFO Uploading batch count=253342026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=253352026/11/12 00:09:56 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2019822238/002/c53362026/11/12 00:09:56 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=15337--- PASS: TestQueueRetryMovesToBack (0.02s)53382026/11/12 00:09:56 INFO Uploading batch count=153392026/11/12 00:09:56 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1127977299/002/nonexistent5340--- PASS: TestDrainTimeout (0.02s)53412026/11/12 00:09:56 ERROR Drain finished with paths left in queue remaining=153422026/11/12 00:09:56 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2019822238/002/d53432026/11/12 00:09:56 INFO Uploading batch count=153442026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=153452026/11/12 00:09:56 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153462026/11/12 00:09:56 INFO Uploading batch count=153472026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=153482026/11/12 00:09:56 INFO Uploading batch count=253492026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=253502026/11/12 00:09:56 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2019822238/002/e53512026/11/12 00:09:56 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerRemoveFailureIsNotProgress1127977299/002/nonexistent53522026/11/12 00:09:56 ERROR Failed to remove paths from queue error="deleting paths: attempt to write a readonly database (8)" count=153532026/11/12 00:09:56 ERROR Drain finished with paths left in queue remaining=15354--- PASS: TestWorkerKeepsPathItCannotStat (0.02s)53552026/11/12 00:09:56 INFO Uploading batch count=153562026/11/12 00:09:56 ERROR Upload failed error="upload failed" count=153572026/11/12 00:09:56 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown2019822238/002/f5358--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)53592026/11/12 00:09:56 ERROR Drain finished with paths left in queue remaining=15360--- PASS: TestWorkerRemoveFailureIsNotProgress (0.03s)53612026/11/12 00:09:56 ERROR Drain finished with paths left in queue remaining=105362--- PASS: TestDrainIsolatesPoisonPath (0.03s)5363--- PASS: TestDrainGivesUpWhenServerDown (0.03s)5364--- PASS: TestWorkerPrunesClosureDeps (0.04s)5365--- PASS: TestWorkerUploadsAndRemoves (0.04s)5366--- PASS: TestWorkerSkipsGCdPaths (0.04s)5367--- PASS: TestQueueRemoveLargeClosure (0.06s)5368--- PASS: TestQueueConcurrentWriters (0.17s)53692026/11/12 00:09:57 ERROR Failed to decode request error="read unix /build/hook1247772337/test.sock->@: i/o timeout"53702026/11/12 00:09:57 ERROR Failed to write response error="write unix /build/hook1247772337/test.sock->@: i/o timeout"5371--- PASS: TestServerStalledClientDoesNotBlockShutdown (0.20s)53722026/11/12 00:09:57 INFO Uploading batch count=153732026/11/12 00:09:57 INFO Uploading batch count=153742026/11/12 00:09:57 INFO Uploading batch count=153752026/11/12 00:09:57 ERROR Upload failed error="upload failed" count=153762026/11/12 00:09:57 INFO Uploading batch count=153772026/11/12 00:09:57 ERROR Upload failed error="upload failed" count=153782026/11/12 00:09:57 INFO Uploading batch count=153792026/11/12 00:09:57 ERROR Upload failed error="upload failed" count=153802026/11/12 00:09:57 INFO Uploading batch count=153812026/11/12 00:09:57 ERROR Upload failed error="upload failed" count=153822026/11/12 00:09:57 ERROR Drain finished with paths left in queue remaining=15383--- PASS: TestRunNotBlockedByPoisonHead (1.04s)5384PASS5385Running signing tests...5386=== RUN TestExternalSigner5387=== PAUSE TestExternalSigner5388=== RUN TestExternalSignerWrongPublicKey5389=== PAUSE TestExternalSignerWrongPublicKey5390=== RUN TestExternalSignerErrors5391=== PAUSE TestExternalSignerErrors5392=== RUN TestExternalSignerRestartsAfterCrash5393=== PAUSE TestExternalSignerRestartsAfterCrash5394=== RUN TestExternalSignerProgramErrorKeepsProcess5395=== PAUSE TestExternalSignerProgramErrorKeepsProcess5396=== RUN TestExternalSignerClose5397=== PAUSE TestExternalSignerClose5398=== RUN TestExternalSignerRejectsInvalidPublicKey5399=== PAUSE TestExternalSignerRejectsInvalidPublicKey5400=== RUN TestExternalSignerCancellation5401=== PAUSE TestExternalSignerCancellation5402=== RUN TestGenerateFingerprint5403=== PAUSE TestGenerateFingerprint5404=== RUN TestParseSigningKey5405=== PAUSE TestParseSigningKey5406=== RUN TestSign5407=== PAUSE TestSign5408=== RUN TestSignBatch5409=== PAUSE TestSignBatch5410=== RUN TestNilKeyPublicKey5411=== PAUSE TestNilKeyPublicKey5412=== RUN TestSignErrors5413=== PAUSE TestSignErrors5414=== CONT TestExternalSigner5415=== CONT TestExternalSignerCancellation5416=== CONT TestExternalSignerWrongPublicKey5417=== CONT TestExternalSignerRejectsInvalidPublicKey5418=== CONT TestExternalSignerClose5419=== CONT TestExternalSignerErrors5420=== CONT TestExternalSignerRestartsAfterCrash5421=== CONT TestParseSigningKey5422=== RUN TestParseSigningKey/valid_32-byte_key5423=== PAUSE TestParseSigningKey/valid_32-byte_key5424=== CONT TestSign5425=== CONT TestSignErrors5426=== CONT TestGenerateFingerprint5427=== CONT TestNilKeyPublicKey5428=== CONT TestSignBatch5429=== CONT TestExternalSignerProgramErrorKeepsProcess5430=== RUN TestExternalSignerRejectsInvalidPublicKey/#005431=== PAUSE TestExternalSignerRejectsInvalidPublicKey/#005432=== RUN TestExternalSignerRejectsInvalidPublicKey/no-colon5433=== PAUSE TestExternalSignerRejectsInvalidPublicKey/no-colon5434=== RUN TestExternalSignerErrors/too_few_signatures5435--- PASS: TestNilKeyPublicKey (0.00s)5436=== PAUSE TestExternalSignerErrors/too_few_signatures5437=== RUN TestExternalSignerErrors/exit_at_startup5438=== PAUSE TestExternalSignerErrors/exit_at_startup5439=== RUN TestParseSigningKey/valid_32-byte_key_with_different_name5440=== PAUSE TestParseSigningKey/valid_32-byte_key_with_different_name5441=== RUN TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5442=== PAUSE TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5443=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5444=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5445=== RUN TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5446=== RUN TestGenerateFingerprint/basic_with_references5447=== RUN TestExternalSignerErrors/invalid_JSON5448=== PAUSE TestExternalSignerErrors/invalid_JSON5449=== RUN TestParseSigningKey/no_colon5450=== PAUSE TestParseSigningKey/no_colon5451=== PAUSE TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5452=== CONT TestExternalSignerRejectsInvalidPublicKey/#005453=== CONT TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=5454=== CONT TestExternalSignerRejectsInvalidPublicKey/no-colon5455=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:invalid!5456=== RUN TestSignErrors/nil_key5457=== PAUSE TestSignErrors/nil_key5458=== RUN TestSignErrors/nil_narinfo5459=== PAUSE TestGenerateFingerprint/basic_with_references5460=== PAUSE TestSignErrors/nil_narinfo5461=== RUN TestGenerateFingerprint/no_references5462=== PAUSE TestGenerateFingerprint/no_references5463=== RUN TestParseSigningKey/empty_name5464=== PAUSE TestParseSigningKey/empty_name5465--- PASS: TestSign (0.00s)5466=== CONT TestExternalSignerRejectsInvalidPublicKey/test-key:YQ==5467=== RUN TestExternalSignerErrors/wrong_shape5468=== PAUSE TestExternalSignerErrors/wrong_shape5469=== RUN TestExternalSignerErrors/null5470=== RUN TestSignErrors/invalid_narinfo5471=== PAUSE TestSignErrors/invalid_narinfo5472=== RUN TestSignErrors/canceled5473=== PAUSE TestSignErrors/canceled5474=== CONT TestSignErrors/nil_key5475=== RUN TestGenerateFingerprint/unsorted_references_get_sorted5476=== PAUSE TestGenerateFingerprint/unsorted_references_get_sorted5477=== RUN TestParseSigningKey/invalid_base645478=== PAUSE TestParseSigningKey/invalid_base645479=== RUN TestParseSigningKey/wrong_length5480=== PAUSE TestParseSigningKey/wrong_length5481=== CONT TestParseSigningKey/valid_32-byte_key5482=== CONT TestParseSigningKey/no_colon5483--- PASS: TestSignBatch (0.00s)5484=== PAUSE TestExternalSignerErrors/null5485--- PASS: TestExternalSignerWrongPublicKey (0.00s)5486--- PASS: TestExternalSignerClose (0.00s)5487=== CONT TestSignErrors/invalid_narinfo5488=== CONT TestSignErrors/canceled5489=== CONT TestSignErrors/nil_narinfo5490=== RUN TestGenerateFingerprint/invalid_nar_hash_prefix5491=== PAUSE TestGenerateFingerprint/invalid_nar_hash_prefix5492=== RUN TestGenerateFingerprint/invalid_nar_hash_length5493=== CONT TestParseSigningKey/empty_name5494=== CONT TestParseSigningKey/wrong_length5495=== CONT TestParseSigningKey/invalid_base645496=== CONT TestParseSigningKey/valid_32-byte_key_with_different_name5497=== RUN TestExternalSignerErrors/null_signatures5498=== PAUSE TestExternalSignerErrors/null_signatures5499=== RUN TestExternalSignerErrors/wrong_signature_length5500--- PASS: TestExternalSignerRejectsInvalidPublicKey (0.00s)5501 --- PASS: TestExternalSignerRejectsInvalidPublicKey/#00 (0.00s)5502 --- PASS: TestExternalSignerRejectsInvalidPublicKey/:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= (0.00s)5503 --- PASS: TestExternalSignerRejectsInvalidPublicKey/no-colon (0.00s)5504 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:invalid! (0.00s)5505 --- PASS: TestExternalSignerRejectsInvalidPublicKey/test-key:YQ== (0.00s)5506=== PAUSE TestGenerateFingerprint/invalid_nar_hash_length5507=== RUN TestGenerateFingerprint/invalid_store_path_prefix5508=== PAUSE TestGenerateFingerprint/invalid_store_path_prefix5509=== RUN TestGenerateFingerprint/invalid_reference_prefix5510=== PAUSE TestGenerateFingerprint/invalid_reference_prefix5511=== PAUSE TestExternalSignerErrors/wrong_signature_length5512--- PASS: TestSignErrors (0.00s)5513 --- PASS: TestSignErrors/nil_key (0.00s)5514 --- PASS: TestSignErrors/invalid_narinfo (0.00s)5515 --- PASS: TestSignErrors/canceled (0.00s)5516 --- PASS: TestSignErrors/nil_narinfo (0.00s)5517=== CONT TestGenerateFingerprint/basic_with_references5518=== CONT TestGenerateFingerprint/invalid_store_path_prefix5519=== CONT TestGenerateFingerprint/invalid_nar_hash_length5520=== CONT TestGenerateFingerprint/invalid_nar_hash_prefix5521=== CONT TestGenerateFingerprint/unsorted_references_get_sorted5522=== CONT TestGenerateFingerprint/invalid_reference_prefix5523=== CONT TestGenerateFingerprint/no_references5524=== RUN TestExternalSignerErrors/wrong_id5525=== PAUSE TestExternalSignerErrors/wrong_id5526=== RUN TestExternalSignerErrors/missing_signatures5527--- PASS: TestParseSigningKey (0.00s)5528 --- PASS: TestParseSigningKey/valid_32-byte_key (0.00s)5529 --- PASS: TestParseSigningKey/no_colon (0.00s)5530 --- PASS: TestParseSigningKey/empty_name (0.00s)5531 --- PASS: TestParseSigningKey/wrong_length (0.00s)5532 --- PASS: TestParseSigningKey/invalid_base64 (0.00s)5533 --- PASS: TestParseSigningKey/valid_32-byte_key_with_different_name (0.00s)5534=== PAUSE TestExternalSignerErrors/missing_signatures5535=== RUN TestExternalSignerErrors/wrong_signatures_shape5536--- PASS: TestGenerateFingerprint (0.00s)5537 --- PASS: TestGenerateFingerprint/basic_with_references (0.00s)5538 --- PASS: TestGenerateFingerprint/invalid_store_path_prefix (0.00s)5539 --- PASS: TestGenerateFingerprint/invalid_nar_hash_length (0.00s)5540 --- PASS: TestGenerateFingerprint/invalid_nar_hash_prefix (0.00s)5541 --- PASS: TestGenerateFingerprint/unsorted_references_get_sorted (0.00s)5542 --- PASS: TestGenerateFingerprint/invalid_reference_prefix (0.00s)5543 --- PASS: TestGenerateFingerprint/no_references (0.00s)5544--- PASS: TestExternalSignerProgramErrorKeepsProcess (0.00s)5545=== PAUSE TestExternalSignerErrors/wrong_signatures_shape5546=== RUN TestExternalSignerErrors/unrequested_key5547=== PAUSE TestExternalSignerErrors/unrequested_key5548=== RUN TestExternalSignerErrors/wrong_signature_name5549=== PAUSE TestExternalSignerErrors/wrong_signature_name5550=== RUN TestExternalSignerErrors/invalid_signature_base645551=== PAUSE TestExternalSignerErrors/invalid_signature_base645552=== RUN TestExternalSignerErrors/exit_on_request5553=== PAUSE TestExternalSignerErrors/exit_on_request5554=== RUN TestExternalSignerErrors/missing_signature_name5555=== PAUSE TestExternalSignerErrors/missing_signature_name5556=== RUN TestExternalSignerErrors/program_error5557=== PAUSE TestExternalSignerErrors/program_error5558=== CONT TestExternalSignerErrors/too_few_signatures5559=== CONT TestExternalSignerErrors/wrong_id5560=== CONT TestExternalSignerErrors/invalid_signature_base645561=== CONT TestExternalSignerErrors/missing_signatures5562=== CONT TestExternalSignerErrors/missing_signature_name5563=== CONT TestExternalSignerErrors/null5564=== CONT TestExternalSignerErrors/exit_on_request5565=== CONT TestExternalSignerErrors/wrong_signature_name5566=== CONT TestExternalSignerErrors/unrequested_key5567=== CONT TestExternalSignerErrors/wrong_signatures_shape5568=== CONT TestExternalSignerErrors/null_signatures5569=== CONT TestExternalSignerErrors/invalid_JSON5570=== CONT TestExternalSignerErrors/wrong_shape5571=== CONT TestExternalSignerErrors/wrong_signature_length5572=== CONT TestExternalSignerErrors/exit_at_startup5573=== CONT TestExternalSignerErrors/program_error5574--- PASS: TestExternalSigner (0.01s)5575--- PASS: TestExternalSignerRestartsAfterCrash (0.01s)5576--- PASS: TestExternalSignerErrors (0.01s)5577 --- PASS: TestExternalSignerErrors/invalid_signature_base64 (0.00s)5578 --- PASS: TestExternalSignerErrors/too_few_signatures (0.00s)5579 --- PASS: TestExternalSignerErrors/wrong_id (0.00s)5580 --- PASS: TestExternalSignerErrors/missing_signatures (0.00s)5581 --- PASS: TestExternalSignerErrors/null (0.01s)5582 --- PASS: TestExternalSignerErrors/exit_on_request (0.01s)5583 --- PASS: TestExternalSignerErrors/unrequested_key (0.01s)5584 --- PASS: TestExternalSignerErrors/wrong_signature_name (0.01s)5585 --- PASS: TestExternalSignerErrors/missing_signature_name (0.01s)5586 --- PASS: TestExternalSignerErrors/wrong_signatures_shape (0.01s)5587 --- PASS: TestExternalSignerErrors/program_error (0.00s)5588 --- PASS: TestExternalSignerErrors/invalid_JSON (0.01s)5589 --- PASS: TestExternalSignerErrors/exit_at_startup (0.00s)5590 --- PASS: TestExternalSignerErrors/wrong_signature_length (0.01s)5591 --- PASS: TestExternalSignerErrors/null_signatures (0.01s)5592 --- PASS: TestExternalSignerErrors/wrong_shape (0.01s)5593--- PASS: TestExternalSignerCancellation (0.02s)5594PASS5595Running OpenBao signer tests...5596=== RUN TestSign5597=== PAUSE TestSign5598=== RUN TestSignErrors5599=== PAUSE TestSignErrors5600=== RUN TestServe5601=== PAUSE TestServe5602=== RUN TestServeRejectsGarbage5603=== PAUSE TestServeRejectsGarbage5604=== CONT TestSign5605=== CONT TestServe5606=== CONT TestSignErrors5607=== CONT TestServeRejectsGarbage5608=== RUN TestSignErrors/http_error5609=== PAUSE TestSignErrors/http_error5610--- PASS: TestServeRejectsGarbage (0.00s)5611=== RUN TestSignErrors/invalid_json5612=== PAUSE TestSignErrors/invalid_json5613=== RUN TestSignErrors/no_prefix5614=== PAUSE TestSignErrors/no_prefix5615=== RUN TestSignErrors/invalid_base645616=== PAUSE TestSignErrors/invalid_base645617=== CONT TestSignErrors/http_error5618--- PASS: TestServe (0.00s)5619=== CONT TestSignErrors/invalid_json5620=== CONT TestSignErrors/invalid_base645621=== CONT TestSignErrors/no_prefix5622--- PASS: TestSignErrors (0.00s)5623 --- PASS: TestSignErrors/http_error (0.00s)5624 --- PASS: TestSignErrors/invalid_base64 (0.00s)5625 --- PASS: TestSignErrors/invalid_json (0.00s)5626 --- PASS: TestSignErrors/no_prefix (0.00s)5627--- PASS: TestSign (0.00s)5628PASS