niks3-go-unit-tests
checks.x86_64-linux.go-unit-tests
· build #147
· raw
1tribuchet: building on jamie2Running client tests...3=== RUN TestDoServerRequestAttachesToken4=== PAUSE TestDoServerRequestAttachesToken5=== RUN TestCaseHackSuffix6=== PAUSE TestCaseHackSuffix7=== RUN TestFilterOversizedClosures8=== PAUSE TestFilterOversizedClosures9=== RUN TestPartSizeForNAR10=== PAUSE TestPartSizeForNAR11=== RUN TestUploadMultipart_SupersededByPeer12=== PAUSE TestUploadMultipart_SupersededByPeer13=== RUN TestDumpPathMatchesNix14=== PAUSE TestDumpPathMatchesNix15=== RUN TestDumpPathSingleFile16=== PAUSE TestDumpPathSingleFile17=== RUN TestDumpPathWriterError18=== PAUSE TestDumpPathWriterError19=== RUN TestEncodeNixBase3220=== PAUSE TestEncodeNixBase3221=== RUN TestEncodeNixBase32WithRealHash22=== PAUSE TestEncodeNixBase32WithRealHash23=== RUN TestConvertHashToNix3224=== PAUSE TestConvertHashToNix3225=== RUN TestGetStorePathHash26=== PAUSE TestGetStorePathHash27=== RUN TestPathInfoHashCompatibility28=== PAUSE TestPathInfoHashCompatibility29=== RUN TestParsePathInfoJSON30=== PAUSE TestParsePathInfoJSON31=== RUN TestParsePathInfoJSONMultiplePaths32=== PAUSE TestParsePathInfoJSONMultiplePaths33=== RUN TestPathInfoCACompatibility34=== PAUSE TestPathInfoCACompatibility35=== RUN TestRateLimiterFeedback36=== PAUSE TestRateLimiterFeedback37=== RUN TestRateLimiterFeedback_400DoesNotCountAsSuccess38=== PAUSE TestRateLimiterFeedback_400DoesNotCountAsSuccess39=== RUN TestResolveStorePath40=== PAUSE TestResolveStorePath41=== RUN TestDoWithRetry_BodyReplayedViaGetBody42=== PAUSE TestDoWithRetry_BodyReplayedViaGetBody43=== RUN TestShellSplit44=== PAUSE TestShellSplit45=== RUN TestShellSplitErrors46=== PAUSE TestShellSplitErrors47=== RUN TestSetClientTLS48=== PAUSE TestSetClientTLS49=== RUN TestSetClientTLSDoesNotMutateDefaultTransport50=== PAUSE TestSetClientTLSDoesNotMutateDefaultTransport51=== RUN TestSetClientTLSErrors52=== PAUSE TestSetClientTLSErrors53=== RUN TestStaticToken54=== PAUSE TestStaticToken55=== RUN TestFileTokenReadsAndCaches56=== PAUSE TestFileTokenReadsAndCaches57=== RUN TestFileTokenMissing58=== PAUSE TestFileTokenMissing59=== RUN TestFileTokenEmpty60=== PAUSE TestFileTokenEmpty61=== RUN TestScriptTokenNoExpiryRerunsEveryCall62=== PAUSE TestScriptTokenNoExpiryRerunsEveryCall63=== RUN TestScriptTokenCachesUntilRefresh64=== PAUSE TestScriptTokenCachesUntilRefresh65=== RUN TestScriptTokenEmptyToken66=== PAUSE TestScriptTokenEmptyToken67=== RUN TestScriptTokenBadJSON68=== PAUSE TestScriptTokenBadJSON69=== RUN TestScriptTokenScriptFails70=== PAUSE TestScriptTokenScriptFails71=== RUN TestScriptTokenEmptyCommand72=== PAUSE TestScriptTokenEmptyCommand73=== CONT TestDoServerRequestAttachesToken74=== CONT TestResolveStorePath75=== CONT TestConvertHashToNix3276=== CONT TestEncodeNixBase32WithRealHash77=== CONT TestRateLimiterFeedback_400DoesNotCountAsSuccess78--- PASS: TestEncodeNixBase32WithRealHash (0.00s)79=== CONT TestSetClientTLSErrors80=== CONT TestRateLimiterFeedback81=== RUN TestRateLimiterFeedback/429_enables_limiter82=== PAUSE TestRateLimiterFeedback/429_enables_limiter83=== CONT TestPathInfoCACompatibility84=== RUN TestPathInfoCACompatibility/null_ca_field852026/08/27 09:40:15 WARN Rate limiter enabled after throttle name=server-test rate=586=== CONT TestParsePathInfoJSONMultiplePaths87=== CONT TestParsePathInfoJSON88=== CONT TestPathInfoHashCompatibility89=== CONT TestGetStorePathHash90=== CONT TestFileTokenMissing91=== CONT TestScriptTokenCachesUntilRefresh92=== CONT TestScriptTokenNoExpiryRerunsEveryCall93=== CONT TestFileTokenEmpty94=== CONT TestDumpPathMatchesNix95=== CONT TestEncodeNixBase3296=== CONT TestDumpPathWriterError97=== CONT TestDumpPathSingleFile98=== CONT TestPartSizeForNAR99=== CONT TestUploadMultipart_SupersededByPeer100=== CONT TestSetClientTLSDoesNotMutateDefaultTransport101=== CONT TestFileTokenReadsAndCaches102=== CONT TestStaticToken103=== RUN TestConvertHashToNix32/SRI_format_to_Nix32104=== RUN TestRateLimiterFeedback/503_enables_limiter105=== RUN TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)106=== PAUSE TestRateLimiterFeedback/503_enables_limiter107=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)108=== RUN TestPathInfoHashCompatibility/old_string_format_with_colon109=== RUN TestRateLimiterFeedback/200_does_not_enable_limiter110=== PAUSE TestRateLimiterFeedback/200_does_not_enable_limiter111=== RUN TestRateLimiterFeedback/400_does_not_enable_limiter112=== PAUSE TestRateLimiterFeedback/400_does_not_enable_limiter113=== PAUSE TestConvertHashToNix32/SRI_format_to_Nix32114=== RUN TestConvertHashToNix32/already_Nix32_format115=== PAUSE TestConvertHashToNix32/already_Nix32_format116=== RUN TestConvertHashToNix32/invalid_format117=== PAUSE TestConvertHashToNix32/invalid_format118--- PASS: TestResolveStorePath (0.00s)119--- PASS: TestStaticToken (0.00s)120=== PAUSE TestPathInfoHashCompatibility/old_string_format_with_colon121=== RUN TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI122=== PAUSE TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI123=== RUN TestPathInfoHashCompatibility/new_structured_format_with_sha512124=== PAUSE TestPathInfoHashCompatibility/new_structured_format_with_sha512125=== CONT TestScriptTokenEmptyCommand126--- PASS: TestScriptTokenEmptyCommand (0.00s)127=== CONT TestShellSplitErrors128--- PASS: TestShellSplitErrors (0.00s)129=== CONT TestSetClientTLS130=== CONT TestScriptTokenBadJSON131=== PAUSE TestPathInfoCACompatibility/null_ca_field132--- PASS: TestFileTokenMissing (0.00s)133=== RUN TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths134=== RUN TestUploadMultipart_SupersededByPeer/exists135=== RUN TestGetStorePathHash/valid_store_path136=== CONT TestShellSplit137=== RUN TestPartSizeForNAR/zero_stays_at_minimum138=== CONT TestDoWithRetry_BodyReplayedViaGetBody139=== CONT TestScriptTokenScriptFails140=== RUN TestParsePathInfoJSON/Nix_format141=== CONT TestFilterOversizedClosures142=== CONT TestCaseHackSuffix143--- PASS: TestFileTokenEmpty (0.00s)144=== RUN TestPathInfoCACompatibility/old_string_format_-_text145--- PASS: TestFileTokenReadsAndCaches (0.00s)146=== RUN TestEncodeNixBase32/test_string_hash147=== PAUSE TestEncodeNixBase32/test_string_hash148=== RUN TestEncodeNixBase32/empty_input149=== PAUSE TestEncodeNixBase32/empty_input150=== CONT TestRateLimiterFeedback/429_enables_limiter151=== PAUSE TestUploadMultipart_SupersededByPeer/exists152=== PAUSE TestGetStorePathHash/valid_store_path153--- PASS: TestShellSplit (0.00s)154=== RUN TestGetStorePathHash/basename_without_hyphen_should_error155=== PAUSE TestGetStorePathHash/basename_without_hyphen_should_error156=== RUN TestUploadMultipart_SupersededByPeer/missing157=== PAUSE TestUploadMultipart_SupersededByPeer/missing158=== PAUSE TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths159=== RUN TestGetStorePathHash/hash_with_invalid_characters_should_error160=== PAUSE TestGetStorePathHash/hash_with_invalid_characters_should_error161=== PAUSE TestPartSizeForNAR/zero_stays_at_minimum162=== CONT TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI)163=== CONT TestConvertHashToNix32/SRI_format_to_Nix32164=== CONT TestRateLimiterFeedback/503_enables_limiter165=== PAUSE TestPathInfoCACompatibility/old_string_format_-_text166=== CONT TestConvertHashToNix32/already_Nix32_format167--- PASS: TestSetClientTLSDoesNotMutateDefaultTransport (0.01s)168=== CONT TestRateLimiterFeedback/400_does_not_enable_limiter169=== PAUSE TestParsePathInfoJSON/Nix_format170=== CONT TestRateLimiterFeedback/200_does_not_enable_limiter171=== RUN TestFilterOversizedClosures/no_limit_keeps_everything172=== CONT TestPathInfoHashCompatibility/old_string_format_with_colon173=== RUN TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths174=== RUN TestGetStorePathHash/hash_with_wrong_length_should_error175=== RUN TestPartSizeForNAR/small_stays_at_minimum176=== RUN TestSetClientTLSErrors/missing_cert_file177=== RUN TestPathInfoCACompatibility/old_string_format_-_fixed_recursive178=== CONT TestScriptTokenEmptyToken179=== RUN TestParsePathInfoJSON/Lix_format180--- PASS: TestDoServerRequestAttachesToken (0.01s)181=== PAUSE TestFilterOversizedClosures/no_limit_keeps_everything182=== PAUSE TestPartSizeForNAR/small_stays_at_minimum183=== RUN TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped184=== PAUSE TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths185=== CONT TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI186=== CONT TestPathInfoHashCompatibility/new_structured_format_with_sha512187=== CONT TestUploadMultipart_SupersededByPeer/exists188=== PAUSE TestGetStorePathHash/hash_with_wrong_length_should_error189=== CONT TestConvertHashToNix32/invalid_format190--- PASS: TestScriptTokenBadJSON (0.01s)191=== PAUSE TestPathInfoCACompatibility/old_string_format_-_fixed_recursive192--- PASS: TestScriptTokenScriptFails (0.00s)193=== PAUSE TestSetClientTLSErrors/missing_cert_file194=== CONT TestEncodeNixBase32/test_string_hash1952026/08/27 09:40:15 WARN Rate limiter enabled after throttle name=server-test rate=5196=== CONT TestEncodeNixBase32/empty_input197=== CONT TestUploadMultipart_SupersededByPeer/missing198=== PAUSE TestParsePathInfoJSON/Lix_format1992026/08/27 09:40:15 WARN Rate limiter enabled after throttle name=server-test rate=5200=== RUN TestPartSizeForNAR/80_GiB_fits_at_minimum201=== CONT TestGetStorePathHash/hash_with_wrong_length_should_error2022026/08/27 09:40:15 WARN Request returned retryable status, retrying attempt=1 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:35733203=== RUN TestPathInfoCACompatibility/new_structured_format_-_text204=== PAUSE TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped205=== RUN TestSetClientTLSErrors/missing_key_file206=== CONT TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths207=== CONT TestGetStorePathHash/valid_store_path2082026/08/27 09:40:15 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=429 url=http://127.0.0.1:345812092026/08/27 09:40:15 WARN Rate limiter enabled after throttle name=server-test rate=52102026/08/27 09:40:15 WARN Request returned retryable status, retrying attempt=1 max_attempts=2 backoff=0s status=503 url=http://127.0.0.1:35127211--- PASS: TestEncodeNixBase32 (0.00s)212 --- PASS: TestEncodeNixBase32/empty_input (0.00s)213 --- PASS: TestEncodeNixBase32/test_string_hash (0.00s)214=== PAUSE TestPartSizeForNAR/80_GiB_fits_at_minimum215=== RUN TestParsePathInfoJSON/empty_input216=== PAUSE TestParsePathInfoJSON/empty_input2172026/08/27 09:40:15 WARN Rate limiter backed off name=server-test rate=5218=== CONT TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths2192026/08/27 09:40:15 WARN Request returned retryable status, retrying attempt=2 max_attempts=6 backoff=0s status=503 url=http://127.0.0.1:357332202026/08/27 09:40:15 WARN Rate limiter backed off name=server-test rate=52212026/08/27 09:40:15 WARN Rate limiter backed off name=server-test rate=5222=== CONT TestGetStorePathHash/hash_with_invalid_characters_should_error223=== PAUSE TestSetClientTLSErrors/missing_key_file224--- PASS: TestConvertHashToNix32 (0.00s)225 --- PASS: TestConvertHashToNix32/SRI_format_to_Nix32 (0.00s)226 --- PASS: TestConvertHashToNix32/already_Nix32_format (0.00s)227 --- PASS: TestConvertHashToNix32/invalid_format (0.00s)228=== CONT TestGetStorePathHash/basename_without_hyphen_should_error229=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_text230=== RUN TestPathInfoCACompatibility/new_structured_format_-_nar_method231=== PAUSE TestPathInfoCACompatibility/new_structured_format_-_nar_method232=== RUN TestFilterOversizedClosures/all_closures_skipped233=== PAUSE TestFilterOversizedClosures/all_closures_skipped234=== CONT TestFilterOversizedClosures/no_limit_keeps_everything235=== RUN TestPartSizeForNAR/115_GiB_needs_larger_parts236=== CONT TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped2372026/08/27 09:40:15 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb-vm-image nar_size=5000 max_nar_size=2000238=== RUN TestParsePathInfoJSON/whitespace_only239=== PAUSE TestParsePathInfoJSON/whitespace_only240=== RUN TestParsePathInfoJSON/invalid_JSON241=== RUN TestSetClientTLS/rejects_connection_without_client_cert242=== PAUSE TestSetClientTLS/rejects_connection_without_client_cert243=== RUN TestSetClientTLSErrors/missing_ca_file244=== PAUSE TestSetClientTLSErrors/missing_ca_file245=== RUN TestSetClientTLSErrors/invalid_ca_file246=== PAUSE TestSetClientTLSErrors/invalid_ca_file247--- PASS: TestPathInfoHashCompatibility (0.00s)248 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_dash_(SRI) (0.00s)249 --- PASS: TestPathInfoHashCompatibility/old_string_format_with_colon (0.00s)250 --- PASS: TestPathInfoHashCompatibility/new_structured_format_-_converts_to_SRI (0.00s)251 --- PASS: TestPathInfoHashCompatibility/new_structured_format_with_sha512 (0.00s)252--- PASS: TestScriptTokenNoExpiryRerunsEveryCall (0.01s)253=== CONT TestPathInfoCACompatibility/null_ca_field254--- PASS: TestRateLimiterFeedback (0.00s)255 --- PASS: TestRateLimiterFeedback/200_does_not_enable_limiter (0.00s)256 --- PASS: TestRateLimiterFeedback/429_enables_limiter (0.01s)257 --- PASS: TestRateLimiterFeedback/400_does_not_enable_limiter (0.01s)258 --- PASS: TestRateLimiterFeedback/503_enables_limiter (0.01s)259=== CONT TestPathInfoCACompatibility/new_structured_format_-_nar_method260--- PASS: TestDoWithRetry_BodyReplayedViaGetBody (0.01s)261=== CONT TestPathInfoCACompatibility/new_structured_format_-_text262=== CONT TestPathInfoCACompatibility/old_string_format_-_fixed_recursive263=== CONT TestPathInfoCACompatibility/old_string_format_-_text264=== CONT TestFilterOversizedClosures/all_closures_skipped2652026/08/27 09:40:15 WARN Skipping closure: path exceeds server max NAR size top_level_path=/nix/store/cccccccccccccccccccccccccccccccc-wrapper oversized_path=/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-small nar_size=1000 max_nar_size=50266=== PAUSE TestPartSizeForNAR/115_GiB_needs_larger_parts267=== PAUSE TestParsePathInfoJSON/invalid_JSON268=== RUN TestSetClientTLS/succeeds_with_client_cert_and_CA269=== PAUSE TestSetClientTLS/succeeds_with_client_cert_and_CA270=== RUN TestSetClientTLS/preserves_debug_logging_transport271=== PAUSE TestSetClientTLS/preserves_debug_logging_transport272=== CONT TestSetClientTLS/rejects_connection_without_client_cert273=== CONT TestSetClientTLSErrors/missing_cert_file274=== CONT TestSetClientTLS/preserves_debug_logging_transport275=== CONT TestSetClientTLSErrors/invalid_ca_file276=== CONT TestSetClientTLSErrors/missing_ca_file277=== CONT TestSetClientTLSErrors/missing_key_file278--- PASS: TestParsePathInfoJSONMultiplePaths (0.01s)279 --- PASS: TestParsePathInfoJSONMultiplePaths/Nix_multiple_paths (0.00s)280 --- PASS: TestParsePathInfoJSONMultiplePaths/Lix_multiple_paths (0.00s)281=== RUN TestPartSizeForNAR/1_TiB282=== CONT TestParsePathInfoJSON/Nix_format283=== CONT TestParsePathInfoJSON/invalid_JSON284=== CONT TestParsePathInfoJSON/whitespace_only285=== CONT TestParsePathInfoJSON/empty_input286=== CONT TestParsePathInfoJSON/Lix_format287=== CONT TestSetClientTLS/succeeds_with_client_cert_and_CA288--- PASS: TestGetStorePathHash (0.01s)289 --- PASS: TestGetStorePathHash/hash_with_wrong_length_should_error (0.00s)290 --- PASS: TestGetStorePathHash/valid_store_path (0.00s)291 --- PASS: TestGetStorePathHash/hash_with_invalid_characters_should_error (0.00s)292 --- PASS: TestGetStorePathHash/basename_without_hyphen_should_error (0.00s)293--- PASS: TestUploadMultipart_SupersededByPeer (0.00s)294 --- PASS: TestUploadMultipart_SupersededByPeer/exists (0.01s)295 --- PASS: TestUploadMultipart_SupersededByPeer/missing (0.01s)296--- PASS: TestPathInfoCACompatibility (0.02s)297 --- PASS: TestPathInfoCACompatibility/null_ca_field (0.00s)298 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_nar_method (0.00s)299 --- PASS: TestPathInfoCACompatibility/new_structured_format_-_text (0.00s)300 --- PASS: TestPathInfoCACompatibility/old_string_format_-_fixed_recursive (0.00s)301 --- PASS: TestPathInfoCACompatibility/old_string_format_-_text (0.00s)302=== PAUSE TestPartSizeForNAR/1_TiB303=== RUN TestPartSizeForNAR/5_TiB_S3_max_object304--- PASS: TestFilterOversizedClosures (0.01s)305 --- PASS: TestFilterOversizedClosures/no_limit_keeps_everything (0.00s)306 --- PASS: TestFilterOversizedClosures/closure_with_oversized_dependency_is_skipped (0.00s)307 --- PASS: TestFilterOversizedClosures/all_closures_skipped (0.00s)308--- PASS: TestScriptTokenEmptyToken (0.01s)309=== PAUSE TestPartSizeForNAR/5_TiB_S3_max_object310=== RUN TestPartSizeForNAR/capped_at_5_GiB311=== PAUSE TestPartSizeForNAR/capped_at_5_GiB312=== CONT TestPartSizeForNAR/zero_stays_at_minimum313=== CONT TestPartSizeForNAR/1_TiB314=== CONT TestPartSizeForNAR/80_GiB_fits_at_minimum315=== CONT TestPartSizeForNAR/small_stays_at_minimum316=== CONT TestPartSizeForNAR/capped_at_5_GiB317=== CONT TestPartSizeForNAR/115_GiB_needs_larger_parts318--- PASS: TestParsePathInfoJSON (0.02s)319 --- PASS: TestParsePathInfoJSON/Nix_format (0.00s)320 --- PASS: TestParsePathInfoJSON/invalid_JSON (0.00s)321 --- PASS: TestParsePathInfoJSON/whitespace_only (0.00s)322 --- PASS: TestParsePathInfoJSON/empty_input (0.00s)323 --- PASS: TestParsePathInfoJSON/Lix_format (0.00s)324=== CONT TestPartSizeForNAR/5_TiB_S3_max_object325--- PASS: TestSetClientTLSErrors (0.02s)326 --- PASS: TestSetClientTLSErrors/missing_cert_file (0.00s)327 --- PASS: TestSetClientTLSErrors/missing_key_file (0.00s)328 --- PASS: TestSetClientTLSErrors/missing_ca_file (0.00s)329 --- PASS: TestSetClientTLSErrors/invalid_ca_file (0.00s)330--- PASS: TestPartSizeForNAR (0.02s)331 --- PASS: TestPartSizeForNAR/zero_stays_at_minimum (0.00s)332 --- PASS: TestPartSizeForNAR/1_TiB (0.00s)333 --- PASS: TestPartSizeForNAR/80_GiB_fits_at_minimum (0.00s)334 --- PASS: TestPartSizeForNAR/small_stays_at_minimum (0.00s)335 --- PASS: TestPartSizeForNAR/capped_at_5_GiB (0.00s)336 --- PASS: TestPartSizeForNAR/115_GiB_needs_larger_parts (0.00s)337 --- PASS: TestPartSizeForNAR/5_TiB_S3_max_object (0.00s)338--- PASS: TestScriptTokenCachesUntilRefresh (0.02s)3392026/08/27 09:40:15 http: TLS handshake error from 127.0.0.1:52750: remote error: tls: bad certificate340--- PASS: TestSetClientTLS (0.02s)341 --- PASS: TestSetClientTLS/succeeds_with_client_cert_and_CA (0.00s)342 --- PASS: TestSetClientTLS/preserves_debug_logging_transport (0.00s)343 --- PASS: TestSetClientTLS/rejects_connection_without_client_cert (0.01s)344--- PASS: TestDumpPathSingleFile (0.03s)345--- PASS: TestDumpPathWriterError (0.04s)346--- PASS: TestCaseHackSuffix (0.04s)347--- PASS: TestDumpPathMatchesNix (0.08s)348--- PASS: TestRateLimiterFeedback_400DoesNotCountAsSuccess (1.00s)349PASS350Running server tests...351The files belonging to this database system will be owned by user "nixbld".352This user must also own the server process.353354The database cluster will be initialized with locale "C".355The default database encoding has accordingly been set to "SQL_ASCII".356The default text search configuration will be set to "english".357358Data page checksums are enabled.359360creating directory /build/postgres1498508259/data ... ok361creating subdirectories ... ok362selecting dynamic shared memory implementation ... posix363selecting default "max_connections" ... 100364selecting default "shared_buffers" ... 128MB365selecting default time zone ... UTC366creating configuration files ... ok367running bootstrap script ... ok368performing post-bootstrap initialization ... ok369syncing data to disk ... ok370371initdb: warning: enabling "trust" authentication for local connections372initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.373374Success. You can now start the database server using:375376 pg_ctl -D /build/postgres1498508259/data -l logfile start377378/build/postgres1498508259:5432 - no response3792026-08-27 09:40:16.713 UTC [112] LOG: starting PostgreSQL 18.4 on x86_64-pc-linux-gnu, compiled by clang version 21.1.8, 64-bit3802026-08-27 09:40:16.714 UTC [112] LOG: listening on Unix socket "/build/postgres1498508259/.s.PGSQL.5432"3812026-08-27 09:40:16.719 UTC [119] LOG: database system was shut down at 2026-08-27 09:40:16 UTC3822026-08-27 09:40:16.723 UTC [112] LOG: database system is ready to accept connections383/build/postgres1498508259:5432 - accepting connections384=== RUN TestService_AuthMiddleware385=== PAUSE TestService_AuthMiddleware386=== RUN TestService_AuthMiddleware_MTLSProxyHeader387=== PAUSE TestService_AuthMiddleware_MTLSProxyHeader388=== RUN TestService_AuthMiddleware_MTLSBoundSubjects389=== PAUSE TestService_AuthMiddleware_MTLSBoundSubjects390=== RUN TestService_ReadAuthMiddleware391=== PAUSE TestService_ReadAuthMiddleware392=== RUN TestService_AuthMiddleware_OIDC393=== PAUSE TestService_AuthMiddleware_OIDC394=== RUN TestCacheConfigHandler395=== PAUSE TestCacheConfigHandler396=== RUN TestCacheStatsHandler397=== PAUSE TestCacheStatsHandler398=== RUN TestClientCADerivations399=== PAUSE TestClientCADerivations400=== RUN TestClientErrorHandling401=== PAUSE TestClientErrorHandling402=== RUN TestClientIntegration403=== PAUSE TestClientIntegration404=== RUN TestClientMultipleUploads405=== PAUSE TestClientMultipleUploads406=== RUN TestClientWithDependencies407=== PAUSE TestClientWithDependencies408=== RUN TestPinProtectsFromGC409=== PAUSE TestPinProtectsFromGC410=== RUN TestGCAdvisoryLockBlocksConcurrentRun4112026-08-27 09:40:17.204 UTC [906] ERROR: relation "goose_db_version" does not exist at character 364122026-08-27 09:40:17.204 UTC [906] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4132026/08/27 09:40:17 OK 20241026095416_initial_model.sql (7ms)4142026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.04ms)4152026/08/27 09:40:17 OK 20251218171726_add_pins.sql (2.42ms)4162026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (2.26ms)4172026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200004182026/08/27 09:40:17 OK 1_commit_pending_closure.sql (1.69ms)4192026/08/27 09:40:17 OK 2_object_stats_trigger.sql (603.66µs)4202026/08/27 09:40:17 goose: up to current file version: 2421--- PASS: TestGCAdvisoryLockBlocksConcurrentRun (0.13s)422=== RUN TestGCBugBareHashReferences423=== PAUSE TestGCBugBareHashReferences424=== RUN TestGCMetrics425=== PAUSE TestGCMetrics426=== RUN TestGCTaskStore_StartNew427=== PAUSE TestGCTaskStore_StartNew428=== RUN TestGCTaskStore_DeduplicateSameParams429=== PAUSE TestGCTaskStore_DeduplicateSameParams430=== RUN TestGCTaskStore_ConflictDifferentParams431=== PAUSE TestGCTaskStore_ConflictDifferentParams432=== RUN TestGCTaskStore_GetEmpty433=== PAUSE TestGCTaskStore_GetEmpty434=== RUN TestGCTaskStore_GetReturnsLatest435=== PAUSE TestGCTaskStore_GetReturnsLatest436=== RUN TestGCTaskStore_CompletedAllowsNewTask437=== PAUSE TestGCTaskStore_CompletedAllowsNewTask438=== RUN TestGCTaskStore_PhaseUpdates439=== PAUSE TestGCTaskStore_PhaseUpdates440=== RUN TestGCTaskStore_Fail441=== PAUSE TestGCTaskStore_Fail442=== RUN TestGracefulShutdownDrainsInflight443=== PAUSE TestGracefulShutdownDrainsInflight444=== RUN TestService_healthCheckHandler445=== PAUSE TestService_healthCheckHandler446=== RUN TestGenerateLandingPage447=== PAUSE TestGenerateLandingPage448=== RUN TestCacheConfigHandlerMaxNarSize449=== PAUSE TestCacheConfigHandlerMaxNarSize450=== RUN TestCreatePendingClosureRejectsOversizedNAR451=== PAUSE TestCreatePendingClosureRejectsOversizedNAR452=== RUN TestNARDeduplicationMetadataUploadBug453=== PAUSE TestNARDeduplicationMetadataUploadBug454=== RUN TestMetricsInventory455=== PAUSE TestMetricsInventory456=== RUN TestService_NativeMTLS457=== PAUSE TestService_NativeMTLS458=== RUN TestServerTLSConfig459=== PAUSE TestServerTLSConfig460=== RUN TestMultipartCleanup461=== PAUSE TestMultipartCleanup462=== RUN TestObjectStatsTrigger463=== PAUSE TestObjectStatsTrigger464=== RUN TestOrphanedObjectsGC465=== PAUSE TestOrphanedObjectsGC466=== RUN TestOrphanedObjectsGCStressTest467=== PAUSE TestOrphanedObjectsGCStressTest468=== RUN TestResurrectedObjectNotDeleted469=== PAUSE TestResurrectedObjectNotDeleted470=== RUN TestParseSingleRange471=== PAUSE TestParseSingleRange472=== RUN TestIsValidCachePath473=== PAUSE TestIsValidCachePath474=== RUN TestReadProxyNarinfo475=== PAUSE TestReadProxyNarinfo476=== RUN TestReadProxyNarinfoAlreadyDecompressed477=== PAUSE TestReadProxyNarinfoAlreadyDecompressed478=== RUN TestReadProxyNarStreaming479=== PAUSE TestReadProxyNarStreaming480=== RUN TestReadProxy404481=== PAUSE TestReadProxy404482=== RUN TestReadProxyInvalidPath483=== PAUSE TestReadProxyInvalidPath484=== RUN TestReadProxyHead485=== PAUSE TestReadProxyHead486=== RUN TestReadProxyConditionalGet487=== PAUSE TestReadProxyConditionalGet488=== RUN TestReadProxyRootRedirectsToIndexHTML489=== PAUSE TestReadProxyRootRedirectsToIndexHTML490=== RUN TestReadProxyDisabled491=== PAUSE TestReadProxyDisabled492=== RUN TestReadRedirectNar493=== PAUSE TestReadRedirectNar494=== RUN TestReadRedirectKeepsNarinfoProxied495=== PAUSE TestReadRedirectKeepsNarinfoProxied496=== RUN TestReadProxyRangeRequest497=== PAUSE TestReadProxyRangeRequest498=== RUN TestRedundantMultipartUpload499=== PAUSE TestRedundantMultipartUpload500=== RUN TestCompleteMultipartUpload_ErrorButObjectExists501=== PAUSE TestCompleteMultipartUpload_ErrorButObjectExists502=== RUN TestCompletedNarNotReofferedAcrossClosures503=== PAUSE TestCompletedNarNotReofferedAcrossClosures504=== RUN TestPresignedUploadRegisteredBeforeCommit505=== PAUSE TestPresignedUploadRegisteredBeforeCommit506=== RUN TestService_Rustfstest507=== PAUSE TestService_Rustfstest508=== RUN TestParseSize509=== PAUSE TestParseSize510=== RUN TestSkippedUploadsHandler511=== PAUSE TestSkippedUploadsHandler512=== RUN TestSystemdListenerNotActivated513--- PASS: TestSystemdListenerNotActivated (0.00s)514=== RUN TestWatchdogBeatsWhenHealthy515--- PASS: TestWatchdogBeatsWhenHealthy (0.02s)516=== RUN TestWatchdogSkipsWhenUnhealthy5172026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5182026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5192026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5202026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5212026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5222026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5232026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5242026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5252026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"5262026/08/27 09:40:17 WARN watchdog liveness check failed, skipping heartbeat error="context deadline exceeded"527--- PASS: TestWatchdogSkipsWhenUnhealthy (0.20s)528=== RUN TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle529=== PAUSE TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle530=== RUN TestProxyWriteTimeout531=== PAUSE TestProxyWriteTimeout532=== RUN TestIsValidUploadKey533=== PAUSE TestIsValidUploadKey534=== RUN TestUploadHandlersRejectInvalidKeys535=== PAUSE TestUploadHandlersRejectInvalidKeys536=== RUN TestUploadHandlersRejectOversizedBody537=== PAUSE TestUploadHandlersRejectOversizedBody538=== RUN TestService_cleanupPendingClosuresHandler539=== PAUSE TestService_cleanupPendingClosuresHandler540=== RUN TestService_createPendingClosureHandler541=== PAUSE TestService_createPendingClosureHandler542=== RUN TestService_verifyS3Integrity543=== PAUSE TestService_verifyS3Integrity544=== RUN TestCompleteMultipartUnregistered545=== PAUSE TestCompleteMultipartUnregistered546=== RUN TestCreatePendingClosure_SmallNARUsesSimplePUT547=== PAUSE TestCreatePendingClosure_SmallNARUsesSimplePUT548=== CONT TestCompleteMultipartUnregistered549=== CONT TestService_AuthMiddleware550=== CONT TestService_verifyS3Integrity551=== CONT TestService_createPendingClosureHandler552=== CONT TestService_cleanupPendingClosuresHandler553=== CONT TestUploadHandlersRejectOversizedBody554=== CONT TestUploadHandlersRejectInvalidKeys555=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info556=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info557=== CONT TestIsValidUploadKey558=== CONT TestProxyWriteTimeout559=== CONT TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle560=== CONT TestSkippedUploadsHandler561=== CONT TestParseSize562=== CONT TestService_Rustfstest563=== CONT TestPresignedUploadRegisteredBeforeCommit564=== CONT TestCompletedNarNotReofferedAcrossClosures565=== CONT TestCompleteMultipartUpload_ErrorButObjectExists566=== CONT TestRedundantMultipartUpload567=== CONT TestReadProxyRangeRequest568=== CONT TestReadRedirectKeepsNarinfoProxied569=== CONT TestReadRedirectNar570=== CONT TestReadProxyDisabled571=== CONT TestReadProxyRootRedirectsToIndexHTML572=== CONT TestReadProxyConditionalGet573=== CONT TestReadProxyHead574=== RUN TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal575=== PAUSE TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal576=== RUN TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key577=== PAUSE TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key578=== RUN TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key579=== PAUSE TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key580=== RUN TestProxyWriteTimeout/narinfo581=== RUN TestIsValidUploadKey/narinfo582--- PASS: TestParseSize (0.00s)583=== CONT TestReadProxyInvalidPath5842026/08/27 09:40:17 INFO Client skipped oversized paths paths=3 nar_bytes=5000000000585--- PASS: TestSkippedUploadsHandler (0.01s)586=== CONT TestReadProxyNarStreaming587=== CONT TestReadProxy404588=== PAUSE TestIsValidUploadKey/narinfo589=== PAUSE TestProxyWriteTimeout/narinfo590=== RUN TestProxyWriteTimeout/1_GiB_nar591=== PAUSE TestProxyWriteTimeout/1_GiB_nar592=== RUN TestProxyWriteTimeout/10_GiB_nar593=== PAUSE TestProxyWriteTimeout/10_GiB_nar594=== RUN TestProxyWriteTimeout/unknown_size595=== PAUSE TestProxyWriteTimeout/unknown_size596=== CONT TestReadProxyNarinfoAlreadyDecompressed597=== RUN TestIsValidUploadKey/nar_zst598=== PAUSE TestIsValidUploadKey/nar_zst599=== RUN TestIsValidUploadKey/nar_xz600=== PAUSE TestIsValidUploadKey/nar_xz601=== RUN TestIsValidUploadKey/nar_plain602=== PAUSE TestIsValidUploadKey/nar_plain603=== RUN TestIsValidUploadKey/listing604=== PAUSE TestIsValidUploadKey/listing605=== RUN TestIsValidUploadKey/build_log606=== PAUSE TestIsValidUploadKey/build_log607=== RUN TestIsValidUploadKey/build_log_home-manager_file608=== PAUSE TestIsValidUploadKey/build_log_home-manager_file609=== RUN TestIsValidUploadKey/build_log_plus_in_name610=== PAUSE TestIsValidUploadKey/build_log_plus_in_name611=== RUN TestIsValidUploadKey/build_log_question_mark612=== PAUSE TestIsValidUploadKey/build_log_question_mark613=== RUN TestIsValidUploadKey/build_log_equals614=== PAUSE TestIsValidUploadKey/build_log_equals615=== RUN TestIsValidUploadKey/realisation616=== PAUSE TestIsValidUploadKey/realisation617=== RUN TestIsValidUploadKey/realisation_plus_in_output618=== PAUSE TestIsValidUploadKey/realisation_plus_in_output619=== RUN TestIsValidUploadKey/nix-cache-info620=== PAUSE TestIsValidUploadKey/nix-cache-info621=== RUN TestIsValidUploadKey/index.html622=== PAUSE TestIsValidUploadKey/index.html623=== RUN TestIsValidUploadKey/narinfo_key,_nar_type624=== PAUSE TestIsValidUploadKey/narinfo_key,_nar_type625=== RUN TestIsValidUploadKey/nar_key,_narinfo_type626=== PAUSE TestIsValidUploadKey/nar_key,_narinfo_type627=== RUN TestIsValidUploadKey/listing_key,_narinfo_type628=== PAUSE TestIsValidUploadKey/listing_key,_narinfo_type629=== RUN TestIsValidUploadKey/traversal630=== PAUSE TestIsValidUploadKey/traversal631=== RUN TestIsValidUploadKey/traversal_nar632=== PAUSE TestIsValidUploadKey/traversal_nar633=== RUN TestIsValidUploadKey/absolute634=== PAUSE TestIsValidUploadKey/absolute635=== RUN TestIsValidUploadKey/empty_key636=== PAUSE TestIsValidUploadKey/empty_key637=== RUN TestIsValidUploadKey/unknown_type638=== PAUSE TestIsValidUploadKey/unknown_type639=== CONT TestReadProxyNarinfo6402026-08-27 09:40:17.735 UTC [976] ERROR: relation "goose_db_version" does not exist at character 366412026-08-27 09:40:17.735 UTC [976] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6422026-08-27 09:40:17.735 UTC [979] ERROR: relation "goose_db_version" does not exist at character 366432026-08-27 09:40:17.735 UTC [979] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6442026-08-27 09:40:17.735 UTC [978] ERROR: relation "goose_db_version" does not exist at character 366452026-08-27 09:40:17.735 UTC [978] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC6462026-08-27 09:40:17.735 UTC [977] ERROR: relation "goose_db_version" does not exist at character 366472026-08-27 09:40:17.735 UTC [977] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC648=== RUN TestUploadHandlersRejectOversizedBody/create_pending_closure649=== PAUSE TestUploadHandlersRejectOversizedBody/create_pending_closure650=== RUN TestUploadHandlersRejectOversizedBody/complete_multipart651=== PAUSE TestUploadHandlersRejectOversizedBody/complete_multipart652=== RUN TestUploadHandlersRejectOversizedBody/request_more_parts653=== PAUSE TestUploadHandlersRejectOversizedBody/request_more_parts654=== CONT TestIsValidCachePath655=== RUN TestIsValidCachePath/narinfo656=== PAUSE TestIsValidCachePath/narinfo657=== RUN TestIsValidCachePath/narinfo_all_nix_base32_chars658=== PAUSE TestIsValidCachePath/narinfo_all_nix_base32_chars659=== RUN TestIsValidCachePath/nar_zst660=== PAUSE TestIsValidCachePath/nar_zst661=== RUN TestIsValidCachePath/nar_xz662=== PAUSE TestIsValidCachePath/nar_xz663=== RUN TestIsValidCachePath/nar_bz2664=== PAUSE TestIsValidCachePath/nar_bz2665=== RUN TestIsValidCachePath/nar_uncompressed666=== PAUSE TestIsValidCachePath/nar_uncompressed667=== RUN TestIsValidCachePath/ls668=== PAUSE TestIsValidCachePath/ls669=== RUN TestIsValidCachePath/log670=== PAUSE TestIsValidCachePath/log671=== RUN TestIsValidCachePath/realisation672=== PAUSE TestIsValidCachePath/realisation673=== RUN TestIsValidCachePath/nix-cache-info674=== PAUSE TestIsValidCachePath/nix-cache-info675=== RUN TestIsValidCachePath/index.html676=== PAUSE TestIsValidCachePath/index.html677=== RUN TestIsValidCachePath/traversal_parent678=== PAUSE TestIsValidCachePath/traversal_parent679=== RUN TestIsValidCachePath/traversal_in_middle680=== PAUSE TestIsValidCachePath/traversal_in_middle681=== RUN TestIsValidCachePath/invalid_char_e682=== PAUSE TestIsValidCachePath/invalid_char_e683=== RUN TestIsValidCachePath/invalid_char_u684=== PAUSE TestIsValidCachePath/invalid_char_u685=== RUN TestIsValidCachePath/random_path686=== PAUSE TestIsValidCachePath/random_path687=== RUN TestIsValidCachePath/empty688=== PAUSE TestIsValidCachePath/empty689=== RUN TestIsValidCachePath/leading_slash690=== PAUSE TestIsValidCachePath/leading_slash691=== RUN TestIsValidCachePath/wrong_extension692=== PAUSE TestIsValidCachePath/wrong_extension693=== RUN TestIsValidCachePath/short_hash694=== PAUSE TestIsValidCachePath/short_hash695=== CONT TestParseSingleRange696=== RUN TestParseSingleRange/none697=== PAUSE TestParseSingleRange/none698=== RUN TestParseSingleRange/unknown_unit699=== PAUSE TestParseSingleRange/unknown_unit700=== RUN TestParseSingleRange/multi-range_ignored701=== PAUSE TestParseSingleRange/multi-range_ignored702=== RUN TestParseSingleRange/malformed_no_dash703=== PAUSE TestParseSingleRange/malformed_no_dash704=== RUN TestParseSingleRange/malformed_both_empty705=== PAUSE TestParseSingleRange/malformed_both_empty706=== RUN TestParseSingleRange/malformed_end_before_start707=== PAUSE TestParseSingleRange/malformed_end_before_start708=== RUN TestParseSingleRange/closed709=== PAUSE TestParseSingleRange/closed710=== RUN TestParseSingleRange/open-ended711=== PAUSE TestParseSingleRange/open-ended712=== RUN TestParseSingleRange/end_clamped_to_size713=== PAUSE TestParseSingleRange/end_clamped_to_size714=== RUN TestParseSingleRange/suffix715=== PAUSE TestParseSingleRange/suffix716=== RUN TestParseSingleRange/suffix_exceeds_size717=== PAUSE TestParseSingleRange/suffix_exceeds_size718=== RUN TestParseSingleRange/single_byte719=== PAUSE TestParseSingleRange/single_byte720=== RUN TestParseSingleRange/start_past_EOF721=== PAUSE TestParseSingleRange/start_past_EOF722=== RUN TestParseSingleRange/start_far_past_EOF723=== PAUSE TestParseSingleRange/start_far_past_EOF724=== CONT TestResurrectedObjectNotDeleted7252026-08-27 09:40:17.743 UTC [980] ERROR: relation "goose_db_version" does not exist at character 367262026-08-27 09:40:17.743 UTC [980] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7272026-08-27 09:40:17.743 UTC [981] ERROR: relation "goose_db_version" does not exist at character 367282026-08-27 09:40:17.743 UTC [981] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7292026-08-27 09:40:17.744 UTC [982] ERROR: relation "goose_db_version" does not exist at character 367302026-08-27 09:40:17.744 UTC [982] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7312026-08-27 09:40:17.745 UTC [983] ERROR: relation "goose_db_version" does not exist at character 367322026-08-27 09:40:17.745 UTC [983] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7332026-08-27 09:40:17.745 UTC [984] ERROR: relation "goose_db_version" does not exist at character 367342026-08-27 09:40:17.745 UTC [984] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7352026-08-27 09:40:17.763 UTC [985] ERROR: relation "goose_db_version" does not exist at character 367362026-08-27 09:40:17.763 UTC [985] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC7372026/08/27 09:40:17 OK 20241026095416_initial_model.sql (36.46ms)7382026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (14.55ms)7392026/08/27 09:40:17 OK 20241026095416_initial_model.sql (51.79ms)7402026/08/27 09:40:17 OK 20241026095416_initial_model.sql (39.75ms)7412026/08/27 09:40:17 OK 20241026095416_initial_model.sql (51.56ms)7422026/08/27 09:40:17 OK 20241026095416_initial_model.sql (30.39ms)7432026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.41ms)7442026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.73ms)7452026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.54ms)7462026/08/27 09:40:17 OK 20251218171726_add_pins.sql (10.73ms)7472026/08/27 09:40:17 OK 20241026095416_initial_model.sql (38.84ms)7482026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (4.24ms)7492026/08/27 09:40:17 OK 20241026095416_initial_model.sql (43.19ms)7502026/08/27 09:40:17 OK 20251218171726_add_pins.sql (6.47ms)7512026/08/27 09:40:17 OK 20241026095416_initial_model.sql (61.89ms)7522026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (4.9ms)7532026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (6.74ms)7542026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007552026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (4.49ms)7562026/08/27 09:40:17 OK 20241026095416_initial_model.sql (45.97ms)7572026/08/27 09:40:17 OK 20251218171726_add_pins.sql (8.53ms)7582026/08/27 09:40:17 OK 20251218171726_add_pins.sql (8.15ms)7592026/08/27 09:40:17 OK 20251218171726_add_pins.sql (6.33ms)7602026/08/27 09:40:17 OK 20241026095416_initial_model.sql (49.51ms)7612026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.71ms)7622026/08/27 09:40:17 OK 1_commit_pending_closure.sql (2.97ms)7632026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.26ms)7642026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (6.68ms)7652026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007662026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.22ms)7672026/08/27 09:40:17 OK 2_object_stats_trigger.sql (2.9ms)7682026/08/27 09:40:17 goose: up to current file version: 27692026/08/27 09:40:17 OK 20251218171726_add_pins.sql (7.95ms)7702026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (7.73ms)7712026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007722026/08/27 09:40:17 OK 20251218171726_add_pins.sql (8.1ms)7732026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (7.89ms)7742026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007752026/08/27 09:40:17 OK 1_commit_pending_closure.sql (5.14ms)7762026/08/27 09:40:17 OK 20251218171726_add_pins.sql (8.19ms)7772026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (9.64ms)7782026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007792026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.75ms)7802026/08/27 09:40:17 OK 20251218171726_add_pins.sql (8.43ms)7812026/08/27 09:40:17 OK 2_object_stats_trigger.sql (3.94ms)7822026/08/27 09:40:17 goose: up to current file version: 27832026/08/27 09:40:17 OK 1_commit_pending_closure.sql (5.68ms)7842026/08/27 09:40:17 OK 20251218171726_add_pins.sql (8.9ms)7852026/08/27 09:40:17 OK 2_object_stats_trigger.sql (3.86ms)7862026/08/27 09:40:17 goose: up to current file version: 27872026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (9.45ms)7882026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007892026/08/27 09:40:17 OK 1_commit_pending_closure.sql (5.12ms)7902026/08/27 09:40:17 OK 2_object_stats_trigger.sql (3.32ms)7912026/08/27 09:40:17 goose: up to current file version: 27922026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (9.42ms)7932026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007942026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (9.35ms)7952026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200007962026/08/27 09:40:17 OK 2_object_stats_trigger.sql (4.17ms)7972026/08/27 09:40:17 goose: up to current file version: 27982026/08/27 09:40:17 OK 1_commit_pending_closure.sql (4.3ms)7992026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (8.65ms)8002026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200008012026/08/27 09:40:17 OK 1_commit_pending_closure.sql (4.55ms)8022026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (9.85ms)8032026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200008042026/08/27 09:40:17 WARN Authentication failed token_preview=wrongtoken token_length=10 reason="static API token mismatch"805--- PASS: TestService_AuthMiddleware (0.38s)806=== CONT TestOrphanedObjectsGCStressTest8072026/08/27 09:40:17 OK 2_object_stats_trigger.sql (14.53ms)8082026/08/27 09:40:17 goose: up to current file version: 28092026/08/27 09:40:17 OK 1_commit_pending_closure.sql (17.67ms)8102026/08/27 09:40:17 OK 2_object_stats_trigger.sql (16.08ms)8112026/08/27 09:40:17 goose: up to current file version: 28122026/08/27 09:40:17 INFO Received complete multipart upload request method=POST path=/api/multipart/complete8132026/08/27 09:40:17 OK 1_commit_pending_closure.sql (17.52ms)8142026/08/27 09:40:17 OK 1_commit_pending_closure.sql (20.64ms)8152026/08/27 09:40:17 INFO Received cleanup request method=DELETE path=/api/pending_closures8162026/08/27 09:40:17 ERROR Failed to get multipart upload error="no rows in result set" upload_id=does-not-exist object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst817--- PASS: TestCompleteMultipartUnregistered (0.39s)818=== CONT TestOrphanedObjectsGC8192026/08/27 09:40:17 OK 2_object_stats_trigger.sql (7.06ms)8202026/08/27 09:40:17 goose: up to current file version: 28212026/08/27 09:40:17 OK 2_object_stats_trigger.sql (4.07ms)8222026/08/27 09:40:17 goose: up to current file version: 28232026/08/27 09:40:17 OK 2_object_stats_trigger.sql (4.2ms)8242026/08/27 09:40:17 goose: up to current file version: 28252026/08/27 09:40:17 INFO Aborted multipart uploads count=08262026/08/27 09:40:17 INFO Received uploads request method=POST path=/api/pending_closures827--- PASS: TestReadProxyDisabled (0.23s)828=== CONT TestObjectStatsTrigger8292026-08-27 09:40:17.884 UTC [998] ERROR: relation "goose_db_version" does not exist at character 368302026-08-27 09:40:17.884 UTC [998] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8312026-08-27 09:40:17.885 UTC [999] ERROR: relation "goose_db_version" does not exist at character 368322026-08-27 09:40:17.885 UTC [999] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8332026/08/27 09:40:17 INFO Received cleanup request method=DELETE path=/api/pending_closures8342026-08-27 09:40:17.889 UTC [1000] ERROR: relation "goose_db_version" does not exist at character 368352026-08-27 09:40:17.889 UTC [1000] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8362026-08-27 09:40:17.889 UTC [1001] ERROR: relation "goose_db_version" does not exist at character 368372026-08-27 09:40:17.889 UTC [1001] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8382026/08/27 09:40:17 INFO Received uploads request method=POST path=/api/pending_closures8392026/08/27 09:40:17 INFO Aborted multipart uploads count=18402026/08/27 09:40:17 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete8412026-08-27 09:40:17.896 UTC [982] ERROR: Closure does not exist: id=18422026-08-27 09:40:17.896 UTC [982] CONTEXT: PL/pgSQL function commit_pending_closure(bigint) line 16 at RAISE8432026-08-27 09:40:17.896 UTC [982] STATEMENT: -- name: CommitPendingClosure :exec844 SELECT commit_pending_closure($1::bigint)845 846--- PASS: TestService_cleanupPendingClosuresHandler (0.42s)847=== CONT TestMultipartCleanup8482026-08-27 09:40:17.905 UTC [1006] ERROR: relation "goose_db_version" does not exist at character 368492026-08-27 09:40:17.905 UTC [1006] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8502026-08-27 09:40:17.906 UTC [1005] ERROR: relation "goose_db_version" does not exist at character 368512026-08-27 09:40:17.906 UTC [1005] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8522026/08/27 09:40:17 INFO Received uploads request method=POST path=/api/pending_closures8532026/08/27 09:40:17 INFO Received uploads request method=POST path=/api/pending_closures8542026/08/27 09:40:17 INFO Received uploads request method=POST path=/api/pending_closures855--- PASS: TestReadProxyNarStreaming (0.42s)856=== CONT TestServerTLSConfig857=== RUN TestServerTLSConfig/no_client_CA858=== PAUSE TestServerTLSConfig/no_client_CA859=== RUN TestServerTLSConfig/missing_CA_file860=== PAUSE TestServerTLSConfig/missing_CA_file861=== RUN TestServerTLSConfig/not_a_PEM_file862=== PAUSE TestServerTLSConfig/not_a_PEM_file863=== CONT TestService_NativeMTLS8642026/08/27 09:40:17 OK 20241026095416_initial_model.sql (24.55ms)8652026/08/27 09:40:17 OK 20241026095416_initial_model.sql (25.11ms)8662026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (2.46ms)8672026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (4.62ms)8682026/08/27 09:40:17 OK 20241026095416_initial_model.sql (26.5ms)8692026/08/27 09:40:17 OK 20251218171726_add_pins.sql (5.69ms)8702026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.16ms)8712026/08/27 09:40:17 OK 20241026095416_initial_model.sql (29.12ms)8722026/08/27 09:40:17 OK 20251218171726_add_pins.sql (5.82ms)8732026-08-27 09:40:17.933 UTC [1010] ERROR: relation "goose_db_version" does not exist at character 368742026-08-27 09:40:17.933 UTC [1010] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8752026-08-27 09:40:17.934 UTC [1012] ERROR: relation "goose_db_version" does not exist at character 368762026-08-27 09:40:17.934 UTC [1012] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8772026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.81ms)8782026-08-27 09:40:17.935 UTC [1011] ERROR: relation "goose_db_version" does not exist at character 368792026-08-27 09:40:17.935 UTC [1011] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8802026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (5.29ms)8812026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200008822026/08/27 09:40:17 OK 20251218171726_add_pins.sql (5.17ms)883--- PASS: TestService_Rustfstest (0.45s)884=== CONT TestMetricsInventory8852026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (5.53ms)8862026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200008872026/08/27 09:40:17 OK 20251218171726_add_pins.sql (5.42ms)8882026/08/27 09:40:17 OK 1_commit_pending_closure.sql (4.17ms)8892026-08-27 09:40:17.939 UTC [1013] ERROR: relation "goose_db_version" does not exist at character 368902026-08-27 09:40:17.939 UTC [1013] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8912026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (5.59ms)8922026/08/27 09:40:17 OK 20241026095416_initial_model.sql (14.12ms)8932026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200008942026/08/27 09:40:17 OK 20241026095416_initial_model.sql (14.13ms)8952026-08-27 09:40:17.942 UTC [1014] ERROR: relation "goose_db_version" does not exist at character 368962026-08-27 09:40:17.942 UTC [1014] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC8972026/08/27 09:40:17 OK 2_object_stats_trigger.sql (2.47ms)8982026/08/27 09:40:17 goose: up to current file version: 28992026/08/27 09:40:17 OK 1_commit_pending_closure.sql (4ms)9002026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (2.17ms)9012026/08/27 09:40:17 OK 2_object_stats_trigger.sql (2.18ms)9022026/08/27 09:40:17 goose: up to current file version: 29032026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.31ms)9042026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (5.19ms)9052026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009062026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (3.64ms)9072026/08/27 09:40:17 OK 2_object_stats_trigger.sql (2.13ms)9082026/08/27 09:40:17 goose: up to current file version: 29092026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.29ms)9102026/08/27 09:40:17 OK 20251218171726_add_pins.sql (4.82ms)9112026/08/27 09:40:17 OK 20251218171726_add_pins.sql (4.34ms)9122026/08/27 09:40:17 OK 2_object_stats_trigger.sql (1.93ms)9132026/08/27 09:40:17 goose: up to current file version: 29142026-08-27 09:40:17.955 UTC [1017] ERROR: relation "goose_db_version" does not exist at character 369152026-08-27 09:40:17.955 UTC [1017] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC9162026-08-27 09:40:17.955 UTC [1018] ERROR: relation "goose_db_version" does not exist at character 369172026-08-27 09:40:17.955 UTC [1018] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC9182026/08/27 09:40:17 OK 20241026095416_initial_model.sql (15.34ms)9192026/08/27 09:40:17 OK 20241026095416_initial_model.sql (13.16ms)9202026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (8.78ms)9212026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009222026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (11.14ms)9232026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009242026/08/27 09:40:17 OK 20241026095416_initial_model.sql (15.93ms)9252026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (2.65ms)9262026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (2.94ms)9272026/08/27 09:40:17 OK 20241026095416_initial_model.sql (14.68ms)928--- PASS: TestReadRedirectNar (0.31s)929=== CONT TestNARDeduplicationMetadataUploadBug9302026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.43ms)9312026/08/27 09:40:17 OK 20241026095416_initial_model.sql (14.18ms)9322026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (1.9ms)9332026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.42ms)9342026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (2.5ms)9352026-08-27 09:40:17.965 UTC [1020] ERROR: relation "goose_db_version" does not exist at character 369362026-08-27 09:40:17.965 UTC [1020] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC937--- PASS: TestReadProxyNarinfoAlreadyDecompressed (0.32s)938=== CONT TestCreatePendingClosureRejectsOversizedNAR9392026/08/27 09:40:17 INFO Received uploads request method=POST path=/api/pending_closures940--- PASS: TestCreatePendingClosureRejectsOversizedNAR (0.00s)941=== CONT TestCacheConfigHandlerMaxNarSize942--- PASS: TestCacheConfigHandlerMaxNarSize (0.00s)943=== CONT TestCreatePendingClosure_SmallNARUsesSimplePUT944--- PASS: TestReadProxyRootRedirectsToIndexHTML (0.32s)945=== CONT TestGenerateLandingPage9462026/08/27 09:40:17 OK 2_object_stats_trigger.sql (11.94ms)9472026/08/27 09:40:17 goose: up to current file version: 29482026/08/27 09:40:17 OK 20251218171726_add_pins.sql (13.89ms)9492026/08/27 09:40:17 OK 20251218171726_add_pins.sql (13.84ms)9502026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (12.05ms)951--- PASS: TestGenerateLandingPage (0.01s)952=== CONT TestPinProtectsFromGC9532026/08/27 09:40:17 OK 2_object_stats_trigger.sql (14.99ms)9542026/08/27 09:40:17 goose: up to current file version: 2955--- PASS: TestReadProxyRangeRequest (0.33s)9562026/08/27 09:40:17 OK 20251218171726_add_pins.sql (14.87ms)957=== CONT TestClientWithDependencies9582026/08/27 09:40:17 OK 20241026095416_initial_model.sql (15.88ms)9592026/08/27 09:40:17 OK 20251218171726_add_pins.sql (16.16ms)9602026/08/27 09:40:17 OK 20241026095416_initial_model.sql (16.03ms)9612026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (2.87ms)9622026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (9.36ms)9632026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009642026/08/27 09:40:17 OK 20251218171726_add_pins.sql (9ms)9652026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (4.73ms)9662026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (6.6ms)9672026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009682026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (7.05ms)9692026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009702026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (11.19ms)9712026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009722026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.04ms)9732026/08/27 09:40:17 OK 20251218171726_add_pins.sql (5.24ms)9742026/08/27 09:40:17 OK 1_commit_pending_closure.sql (2.58ms)9752026/08/27 09:40:17 OK 20251218171726_add_pins.sql (4.42ms)9762026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.89ms)9772026/08/27 09:40:17 OK 1_commit_pending_closure.sql (4ms)9782026/08/27 09:40:17 OK 2_object_stats_trigger.sql (2.86ms)9792026/08/27 09:40:17 goose: up to current file version: 29802026/08/27 09:40:17 OK 20241026095416_initial_model.sql (11.14ms)9812026/08/27 09:40:17 OK 2_object_stats_trigger.sql (3.07ms)9822026/08/27 09:40:17 goose: up to current file version: 29832026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (7.86ms)9842026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009852026/08/27 09:40:17 OK 2_object_stats_trigger.sql (3.21ms)9862026/08/27 09:40:17 goose: up to current file version: 29872026/08/27 09:40:17 OK 2_object_stats_trigger.sql (3.07ms)9882026/08/27 09:40:17 goose: up to current file version: 29892026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (6.07ms)9902026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009912026/08/27 09:40:17 OK 20251210153512_drop_unused_gin_index.sql (2.79ms)9922026/08/27 09:40:17 OK 20260628120000_add_object_size_and_stats.sql (6.29ms)9932026/08/27 09:40:17 goose: successfully migrated database to version: 202606281200009942026/08/27 09:40:17 OK 1_commit_pending_closure.sql (4.33ms)995--- PASS: TestReadProxyConditionalGet (0.35s)996=== CONT TestGCBugBareHashReferences9972026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.65ms)9982026/08/27 09:40:17 OK 1_commit_pending_closure.sql (3.38ms)9992026/08/27 09:40:17 OK 20251218171726_add_pins.sql (4.75ms)10002026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.14ms)10012026/08/27 09:40:18 goose: up to current file version: 210022026/08/27 09:40:18 OK 2_object_stats_trigger.sql (3.23ms)10032026/08/27 09:40:18 goose: up to current file version: 210042026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.27ms)10052026/08/27 09:40:18 goose: up to current file version: 21006{"timestamp":"2026-08-27T09:40:18.002349473Z","level":"ERROR","duration":"353.04µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(582)"}1007{"timestamp":"2026-08-27T09:40:18.002470003Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"cd67dfe9-d7a0-496c-9d5a-ccf1dd916d31","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket23/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(582)"}1008{"timestamp":"2026-08-27T09:40:18.003192003Z","level":"ERROR","duration":"177.37µs","resp":"Response { status: 503, version: HTTP/1.1, headers: {\"content-type\": \"application/xml\"}, body: Body { once: b\"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>SlowDown</Code><Message>bucket creation concurrency limit reached; retry later</Message></Error>\" } }","target":"s3s::service","filename":"/build/rustfs-1.0.0-beta.12-vendor/source-git-1/s3s-0.14.1/src/service.rs","line_number":640,"threadName":"rustfs-worker","threadId":"ThreadId(766)"}1009{"timestamp":"2026-08-27T09:40:18.003219163Z","level":"ERROR","message":"HTTP request completed","event":"http_request_completed","component":"server","subsystem":"http","request_id":"39b6fa21-6225-41fe-9be5-c95d9017c896","trace_id":"unknown","span_id":"unknown","peer_addr":"127.0.0.1","method":"PUT","uri":"/bucket24/","status_code":503,"duration_ms":0,"result":"server_error","target":"rustfs::server::layer","filename":"rustfs/src/server/layer.rs","line_number":392,"threadName":"rustfs-worker","threadId":"ThreadId(766)"}10102026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures10112026-08-27 09:40:18.005 UTC [1030] ERROR: relation "goose_db_version" does not exist at character 3610122026-08-27 09:40:18.005 UTC [1030] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10132026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (7.27ms)10142026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000010152026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures1016--- PASS: TestReadProxyNarinfo (0.36s)1017=== CONT TestService_healthCheckHandler10182026-08-27 09:40:18.020 UTC [1032] ERROR: relation "goose_db_version" does not exist at character 3610192026-08-27 09:40:18.020 UTC [1032] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10202026/08/27 09:40:18 OK 1_commit_pending_closure.sql (14.29ms)10212026/08/27 09:40:18 OK 2_object_stats_trigger.sql (5.49ms)10222026/08/27 09:40:18 goose: up to current file version: 210232026-08-27 09:40:18.031 UTC [1035] ERROR: relation "goose_db_version" does not exist at character 3610242026-08-27 09:40:18.031 UTC [1035] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10252026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures10262026/08/27 09:40:18 OK 20241026095416_initial_model.sql (14.07ms)10272026-08-27 09:40:18.037 UTC [1036] ERROR: relation "goose_db_version" does not exist at character 3610282026-08-27 09:40:18.037 UTC [1036] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10292026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (3.42ms)10302026/08/27 09:40:18 INFO Registered completed upload object_key=nar/jjjjjjjjjjjjjjjjjjjjjjjjjjjjjj0100000000000000000000.nar.zst10312026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures10322026-08-27 09:40:18.042 UTC [1037] ERROR: relation "goose_db_version" does not exist at character 3610332026-08-27 09:40:18.042 UTC [1037] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC10342026/08/27 09:40:18 OK 20241026095416_initial_model.sql (11.34ms)1035--- PASS: TestPresignedUploadRegisteredBeforeCommit (0.39s)1036=== CONT TestClientMultipleUploads10372026/08/27 09:40:18 OK 20251218171726_add_pins.sql (5.78ms)10382026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (2.91ms)1039--- PASS: TestReadProxyHead (0.40s)1040=== CONT TestGracefulShutdownDrainsInflight10412026/08/27 09:40:18 INFO Starting HTTP server address=127.0.0.1:4660910422026/08/27 09:40:18 INFO Shutdown signal received, draining in-flight requests timeout=10s10432026/08/27 09:40:18 OK 20241026095416_initial_model.sql (10.5ms)10442026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (5.07ms)10452026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000010462026/08/27 09:40:18 OK 20251218171726_add_pins.sql (4.95ms)10472026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (2.06ms)10482026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures1049--- PASS: TestReadRedirectKeepsNarinfoProxied (0.40s)10502026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.98ms)1051=== CONT TestClientIntegration10522026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.41ms)10532026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000010542026/08/27 09:40:18 OK 20251218171726_add_pins.sql (4.37ms)10552026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.69ms)10562026/08/27 09:40:18 goose: up to current file version: 210572026/08/27 09:40:18 OK 20241026095416_initial_model.sql (12.15ms)10582026/08/27 09:40:18 OK 1_commit_pending_closure.sql (3.71ms)10592026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (3.57ms)10602026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000010612026/08/27 09:40:18 OK 20241026095416_initial_model.sql (10.75ms)10622026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.67ms)10632026/08/27 09:40:18 goose: up to current file version: 210642026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.51ms)10652026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (3.05ms)10662026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures10672026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (2.48ms)10682026-08-27 09:40:18.066 UTC [1042] ERROR: relation "goose_db_version" does not exist at character 3610692026-08-27 09:40:18.066 UTC [1042] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC1070--- PASS: TestReadProxyInvalidPath (0.59s)1071=== CONT TestGCTaskStore_Fail1072--- PASS: TestGCTaskStore_Fail (0.00s)1073=== CONT TestClientErrorHandling1074=== RUN TestClientErrorHandling/InvalidStorePath1075=== PAUSE TestClientErrorHandling/InvalidStorePath1076=== RUN TestClientErrorHandling/InvalidAuthToken1077=== PAUSE TestClientErrorHandling/InvalidAuthToken1078=== RUN TestClientErrorHandling/ServerNotAvailable1079=== PAUSE TestClientErrorHandling/ServerNotAvailable1080=== CONT TestGCTaskStore_PhaseUpdates1081--- PASS: TestGCTaskStore_PhaseUpdates (0.00s)1082=== CONT TestClientCADerivations1083--- PASS: TestReadProxy404 (0.43s)1084=== CONT TestGCTaskStore_CompletedAllowsNewTask1085--- PASS: TestGCTaskStore_CompletedAllowsNewTask (0.00s)1086=== CONT TestGCTaskStore_DeduplicateSameParams1087--- PASS: TestGCTaskStore_DeduplicateSameParams (0.00s)1088=== CONT TestCacheStatsHandler10892026/08/27 09:40:18 OK 2_object_stats_trigger.sql (16.82ms)10902026/08/27 09:40:18 goose: up to current file version: 210912026/08/27 09:40:18 OK 20251218171726_add_pins.sql (16.2ms)10922026/08/27 09:40:18 OK 20251218171726_add_pins.sql (16.96ms)10932026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (5.53ms)10942026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000010952026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (7.65ms)10962026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000010972026/08/27 09:40:18 OK 1_commit_pending_closure.sql (3.1ms)10982026/08/27 09:40:18 OK 2_object_stats_trigger.sql (3.57ms)10992026/08/27 09:40:18 goose: up to current file version: 211002026/08/27 09:40:18 OK 20241026095416_initial_model.sql (12.59ms)11012026/08/27 09:40:18 OK 1_commit_pending_closure.sql (4.57ms)11022026/08/27 09:40:18 OK 2_object_stats_trigger.sql (4.57ms)11032026/08/27 09:40:18 goose: up to current file version: 211042026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (4.53ms)11052026-08-27 09:40:18.101 UTC [1047] ERROR: relation "goose_db_version" does not exist at character 3611062026-08-27 09:40:18.101 UTC [1047] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11072026/08/27 09:40:18 OK 20251218171726_add_pins.sql (4.07ms)11082026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete11092026-08-27 09:40:18.109 UTC [1049] ERROR: relation "goose_db_version" does not exist at character 3611102026-08-27 09:40:18.109 UTC [1049] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11112026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (6.24ms)11122026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000011132026/08/27 09:40:18 WARN CompleteMultipartUpload errored but object exists; treating as success error="The specified key does not exist." object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MzdiYTliZTItZDg3NC00ZDJkLWEwZjgtN2ZjYTA0MzM5NjMzLjQxNjcxMjE2LTFiYmMtNGI2ZS1hNjI5LTY1OGFmMzJkZGMwMngxNzg3ODIzNjE4MDgxMTgzMzEy11142026-08-27 09:40:18.112 UTC [1050] ERROR: relation "goose_db_version" does not exist at character 3611152026-08-27 09:40:18.112 UTC [1050] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11162026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures11172026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.42ms)1118--- PASS: TestGracefulShutdownDrainsInflight (0.07s)1119=== CONT TestCacheConfigHandler1120=== RUN TestCacheConfigHandler/full_config,_no_issuer1121=== PAUSE TestCacheConfigHandler/full_config,_no_issuer1122=== RUN TestCacheConfigHandler/no_cache_url_configured1123=== PAUSE TestCacheConfigHandler/no_cache_url_configured1124=== RUN TestCacheConfigHandler/no_signing_keys1125=== PAUSE TestCacheConfigHandler/no_signing_keys1126=== RUN TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1127=== PAUSE TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator11282026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.25ms)1129=== CONT TestService_AuthMiddleware_OIDC11302026/08/27 09:40:18 goose: up to current file version: 211312026/08/27 09:40:18 INFO OIDC provider initialized name=test11322026/08/27 09:40:18 OK 20241026095416_initial_model.sql (10.12ms)11332026/08/27 09:40:18 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0300000000000000000000.nar.zst upload_id=MzdiYTliZTItZDg3NC00ZDJkLWEwZjgtN2ZjYTA0MzM5NjMzLjQxNjcxMjE2LTFiYmMtNGI2ZS1hNjI5LTY1OGFmMzJkZGMwMngxNzg3ODIzNjE4MDgxMTgzMzEy parts=11134--- PASS: TestCompleteMultipartUpload_ErrorButObjectExists (0.47s)1135=== CONT TestGCTaskStore_StartNew1136--- PASS: TestGCTaskStore_StartNew (0.00s)1137=== CONT TestService_ReadAuthMiddleware11382026-08-27 09:40:18.122 UTC [1051] ERROR: relation "goose_db_version" does not exist at character 3611392026-08-27 09:40:18.122 UTC [1051] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11402026-08-27 09:40:18.126 UTC [1052] ERROR: relation "goose_db_version" does not exist at character 3611412026-08-27 09:40:18.126 UTC [1052] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11422026/08/27 09:40:18 WARN mTLS auth: subject not in bound subjects subject="CN=reader"11432026/08/27 09:40:18 WARN mTLS auth: subject not in bound subjects subject="CN=writer"1144--- PASS: TestService_NativeMTLS (0.21s)1145=== CONT TestService_AuthMiddleware_MTLSBoundSubjects11462026/08/27 09:40:18 OK 20241026095416_initial_model.sql (19.04ms)11472026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (17.39ms)11482026/08/27 09:40:18 OK 20241026095416_initial_model.sql (21.81ms)11492026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (3ms)11502026/08/27 09:40:18 OK 20251218171726_add_pins.sql (3.74ms)11512026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (2.94ms)11522026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (5.04ms)11532026-08-27 09:40:18.147 UTC [1059] ERROR: relation "goose_db_version" does not exist at character 3611542026-08-27 09:40:18.147 UTC [1059] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC11552026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000011562026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures11572026/08/27 09:40:18 OK 20251218171726_add_pins.sql (6.18ms)11582026/08/27 09:40:18 OK 20251218171726_add_pins.sql (7.46ms)11592026/08/27 09:40:18 OK 1_commit_pending_closure.sql (3.85ms)11602026/08/27 09:40:18 OK 20241026095416_initial_model.sql (13.95ms)11612026/08/27 09:40:18 OK 20241026095416_initial_model.sql (13.93ms)11622026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.77ms)11632026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000011642026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.99ms)11652026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000011662026/08/27 09:40:18 OK 2_object_stats_trigger.sql (3.74ms)11672026/08/27 09:40:18 goose: up to current file version: 211682026/08/27 09:40:18 OK 1_commit_pending_closure.sql (3.51ms)11692026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (4.17ms)11702026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (4.3ms)11712026/08/27 09:40:18 OK 1_commit_pending_closure.sql (4.8ms)11722026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.8ms)11732026/08/27 09:40:18 goose: up to current file version: 211742026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.44ms)11752026/08/27 09:40:18 goose: up to current file version: 211762026/08/27 09:40:18 OK 20251218171726_add_pins.sql (5.28ms)1177--- PASS: TestObjectStatsTrigger (0.28s)11782026/08/27 09:40:18 OK 20251218171726_add_pins.sql (5.37ms)1179=== CONT TestGCMetrics11802026/08/27 09:40:18 OK 20241026095416_initial_model.sql (12.34ms)11812026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.5ms)11822026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000011832026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.72ms)11842026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000011852026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (2.82ms)11862026/08/27 09:40:18 OK 1_commit_pending_closure.sql (3.76ms)11872026/08/27 09:40:18 OK 1_commit_pending_closure.sql (3.54ms)11882026/08/27 09:40:18 OK 20251218171726_add_pins.sql (3.42ms)1189--- PASS: TestMetricsInventory (0.24s)1190=== CONT TestService_AuthMiddleware_MTLSProxyHeader11912026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.28ms)11922026/08/27 09:40:18 goose: up to current file version: 211932026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.2ms)11942026/08/27 09:40:18 goose: up to current file version: 211952026/08/27 09:40:18 INFO Created nix-cache-info in bucket bucket=bucket3211962026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.04ms)11972026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000011982026-08-27 09:40:18.183 UTC [1063] ERROR: relation "goose_db_version" does not exist at character 3611992026-08-27 09:40:18.183 UTC [1063] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12002026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures12012026-08-27 09:40:18.184 UTC [1066] ERROR: relation "goose_db_version" does not exist at character 3612022026-08-27 09:40:18.184 UTC [1066] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12032026/08/27 09:40:18 OK 1_commit_pending_closure.sql (13.55ms)12042026/08/27 09:40:18 OK 2_object_stats_trigger.sql (5.58ms)12052026/08/27 09:40:18 goose: up to current file version: 21206--- PASS: TestResurrectedObjectNotDeleted (0.46s)12072026/08/27 09:40:18 INFO Created nix-cache-info in bucket bucket=bucket341208=== CONT TestGCTaskStore_GetReturnsLatest1209--- PASS: TestGCTaskStore_GetReturnsLatest (0.00s)1210=== CONT TestGCTaskStore_GetEmpty1211--- PASS: TestGCTaskStore_GetEmpty (0.00s)1212=== CONT TestGCTaskStore_ConflictDifferentParams12132026-08-27 09:40:18.200 UTC [1067] ERROR: relation "goose_db_version" does not exist at character 3612142026-08-27 09:40:18.200 UTC [1067] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC1215--- PASS: TestGCTaskStore_ConflictDifferentParams (0.00s)1216=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info12172026/08/27 09:40:18 INFO Received uploads request method=POST path=/1218=== CONT TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key12192026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/1220=== CONT TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key12212026/08/27 09:40:18 INFO Received request for more parts method=POST path=/1222=== CONT TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal12232026/08/27 09:40:18 INFO Received uploads request method=POST path=/1224--- PASS: TestUploadHandlersRejectInvalidKeys (0.00s)1225 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_nix-cache-info (0.00s)1226 --- PASS: TestUploadHandlersRejectInvalidKeys/complete_multipart_rejects_non-NAR_key (0.00s)1227 --- PASS: TestUploadHandlersRejectInvalidKeys/request_more_parts_rejects_non-NAR_key (0.00s)1228 --- PASS: TestUploadHandlersRejectInvalidKeys/create_pending_closure_rejects_path_traversal (0.00s)1229=== CONT TestProxyWriteTimeout/narinfo1230=== CONT TestProxyWriteTimeout/unknown_size1231=== CONT TestProxyWriteTimeout/1_GiB_nar1232=== CONT TestIsValidUploadKey/narinfo1233=== CONT TestIsValidUploadKey/empty_key1234=== CONT TestIsValidUploadKey/absolute1235=== CONT TestIsValidUploadKey/traversal_nar1236=== CONT TestIsValidUploadKey/traversal1237=== CONT TestIsValidUploadKey/listing_key,_narinfo_type1238=== CONT TestIsValidUploadKey/nar_key,_narinfo_type1239=== CONT TestIsValidUploadKey/narinfo_key,_nar_type1240=== CONT TestIsValidUploadKey/index.html1241=== CONT TestIsValidUploadKey/nix-cache-info1242=== CONT TestIsValidUploadKey/realisation_plus_in_output1243=== CONT TestIsValidUploadKey/realisation1244--- PASS: TestCreatePendingClosure_SmallNARUsesSimplePUT (0.23s)1245=== CONT TestProxyWriteTimeout/10_GiB_nar1246=== CONT TestIsValidUploadKey/build_log_equals1247=== CONT TestIsValidUploadKey/build_log_plus_in_name1248=== CONT TestIsValidUploadKey/build_log_question_mark1249=== CONT TestIsValidUploadKey/build_log1250=== CONT TestIsValidUploadKey/listing1251=== CONT TestIsValidUploadKey/nar_plain1252--- PASS: TestProxyWriteTimeout (0.17s)1253 --- PASS: TestProxyWriteTimeout/narinfo (0.00s)1254 --- PASS: TestProxyWriteTimeout/unknown_size (0.00s)1255 --- PASS: TestProxyWriteTimeout/1_GiB_nar (0.00s)1256 --- PASS: TestProxyWriteTimeout/10_GiB_nar (0.00s)1257=== CONT TestIsValidUploadKey/build_log_home-manager_file1258=== CONT TestIsValidUploadKey/nar_zst1259=== CONT TestIsValidUploadKey/nar_xz1260=== CONT TestUploadHandlersRejectOversizedBody/create_pending_closure12612026/08/27 09:40:18 INFO Received uploads request method=POST path=/1262=== CONT TestIsValidUploadKey/unknown_type1263--- PASS: TestIsValidUploadKey (0.17s)1264 --- PASS: TestIsValidUploadKey/narinfo (0.00s)1265 --- PASS: TestIsValidUploadKey/empty_key (0.00s)1266 --- PASS: TestIsValidUploadKey/absolute (0.00s)1267 --- PASS: TestIsValidUploadKey/traversal_nar (0.00s)1268 --- PASS: TestIsValidUploadKey/traversal (0.00s)1269 --- PASS: TestIsValidUploadKey/listing_key,_narinfo_type (0.00s)1270 --- PASS: TestIsValidUploadKey/nar_key,_narinfo_type (0.00s)1271 --- PASS: TestIsValidUploadKey/narinfo_key,_nar_type (0.00s)1272 --- PASS: TestIsValidUploadKey/index.html (0.00s)1273 --- PASS: TestIsValidUploadKey/nix-cache-info (0.00s)1274 --- PASS: TestIsValidUploadKey/realisation_plus_in_output (0.00s)1275 --- PASS: TestIsValidUploadKey/realisation (0.00s)1276 --- PASS: TestIsValidUploadKey/build_log_equals (0.00s)1277 --- PASS: TestIsValidUploadKey/build_log_plus_in_name (0.00s)1278 --- PASS: TestIsValidUploadKey/build_log_question_mark (0.00s)1279 --- PASS: TestIsValidUploadKey/build_log (0.00s)1280 --- PASS: TestIsValidUploadKey/listing (0.00s)1281 --- PASS: TestIsValidUploadKey/nar_plain (0.00s)1282 --- PASS: TestIsValidUploadKey/build_log_home-manager_file (0.00s)1283 --- PASS: TestIsValidUploadKey/nar_zst (0.00s)1284 --- PASS: TestIsValidUploadKey/nar_xz (0.00s)1285 --- PASS: TestIsValidUploadKey/unknown_type (0.00s)1286=== CONT TestUploadHandlersRejectOversizedBody/request_more_parts12872026/08/27 09:40:18 INFO Received request for more parts method=POST path=/12882026/08/27 09:40:18 OK 20241026095416_initial_model.sql (13.32ms)12892026/08/27 09:40:18 OK 20241026095416_initial_model.sql (13.38ms)1290--- PASS: TestService_healthCheckHandler (0.19s)1291=== CONT TestUploadHandlersRejectOversizedBody/complete_multipart12922026-08-27 09:40:18.210 UTC [1085] ERROR: relation "goose_db_version" does not exist at character 3612932026-08-27 09:40:18.210 UTC [1085] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC12942026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/12952026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (3.13ms)12962026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (3.26ms)1297=== NAME TestNARDeduplicationMetadataUploadBug1298 metadata_upload_test.go:48: First store path: /build/TestNARDeduplicationMetadataUploadBug2778236546/001/store/95hsp8q3ar4mbq605agf24vb8ykvihry-file1.txt12992026/08/27 09:40:18 OK 20251218171726_add_pins.sql (4.93ms)13002026/08/27 09:40:18 OK 20251218171726_add_pins.sql (4.85ms)13012026/08/27 09:40:18 OK 20241026095416_initial_model.sql (10.34ms)13022026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.23ms)13032026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000013042026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (5.24ms)13052026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000013062026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (3.25ms)13072026/08/27 09:40:18 INFO Created nix-cache-info in bucket bucket=bucket3613082026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.44ms)13092026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.58ms)13102026/08/27 09:40:18 OK 2_object_stats_trigger.sql (1.94ms)13112026/08/27 09:40:18 goose: up to current file version: 213122026/08/27 09:40:18 OK 20251218171726_add_pins.sql (3.39ms)13132026/08/27 09:40:18 OK 2_object_stats_trigger.sql (1.73ms)13142026/08/27 09:40:18 goose: up to current file version: 213152026/08/27 09:40:18 OK 20241026095416_initial_model.sql (9.78ms)13162026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (1.74ms)13172026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (3.28ms)13182026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000013192026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.62ms)13202026/08/27 09:40:18 OK 20251218171726_add_pins.sql (4.77ms)13212026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete13222026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.03ms)13232026/08/27 09:40:18 goose: up to current file version: 213242026-08-27 09:40:18.237 UTC [1089] ERROR: relation "goose_db_version" does not exist at character 3613252026-08-27 09:40:18.237 UTC [1089] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13262026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (3.71ms)13272026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000013282026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.03ms)13292026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.15ms)13302026/08/27 09:40:18 goose: up to current file version: 213312026/08/27 09:40:18 INFO Created nix-cache-info in bucket bucket=bucket3813322026-08-27 09:40:18.246 UTC [1122] ERROR: relation "goose_db_version" does not exist at character 3613332026-08-27 09:40:18.246 UTC [1122] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13342026-08-27 09:40:18.247 UTC [1123] ERROR: relation "goose_db_version" does not exist at character 3613352026-08-27 09:40:18.247 UTC [1123] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC13362026/08/27 09:40:18 INFO Created nix-cache-info in bucket bucket=bucket3913372026/08/27 09:40:18 INFO Created nix-cache-info in bucket bucket=bucket4013382026/08/27 09:40:18 OK 20241026095416_initial_model.sql (17.78ms)13392026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (2.73ms)13402026/08/27 09:40:18 INFO Received cleanup request method=DELETE path=/api/pending_closures13412026/08/27 09:40:18 OK 20251218171726_add_pins.sql (2.24ms)13422026/08/27 09:40:18 INFO Aborted multipart uploads count=113432026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (3.5ms)13442026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000013452026/08/27 09:40:18 OK 20241026095416_initial_model.sql (8.14ms)13462026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.59ms)1347=== CONT TestIsValidCachePath/narinfo13482026/08/27 09:40:18 OK 20241026095416_initial_model.sql (9.16ms)1349=== CONT TestIsValidCachePath/index.html1350=== CONT TestIsValidCachePath/short_hash1351=== CONT TestIsValidCachePath/wrong_extension1352=== CONT TestIsValidCachePath/leading_slash1353=== CONT TestIsValidCachePath/empty1354=== CONT TestIsValidCachePath/random_path1355=== CONT TestIsValidCachePath/invalid_char_u1356=== CONT TestIsValidCachePath/invalid_char_e13572026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (1.55ms)1358=== CONT TestIsValidCachePath/traversal_in_middle1359=== CONT TestIsValidCachePath/traversal_parent1360=== CONT TestIsValidCachePath/nar_xz13612026/08/27 09:40:18 OK 2_object_stats_trigger.sql (748.79µs)1362=== CONT TestIsValidCachePath/nar_bz213632026/08/27 09:40:18 goose: up to current file version: 21364=== CONT TestIsValidCachePath/nar_uncompressed1365=== CONT TestIsValidCachePath/realisation1366=== CONT TestIsValidCachePath/log1367=== CONT TestIsValidCachePath/nix-cache-info1368=== CONT TestIsValidCachePath/ls13692026-08-27 09:40:18.273 UTC [1146] ERROR: relation "goose_db_version" does not exist at character 3613702026-08-27 09:40:18.273 UTC [1146] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC1371=== CONT TestIsValidCachePath/nar_zst1372=== CONT TestIsValidCachePath/narinfo_all_nix_base32_chars1373--- PASS: TestIsValidCachePath (0.00s)1374 --- PASS: TestIsValidCachePath/narinfo (0.00s)1375 --- PASS: TestIsValidCachePath/index.html (0.00s)1376 --- PASS: TestIsValidCachePath/short_hash (0.00s)1377 --- PASS: TestIsValidCachePath/wrong_extension (0.00s)1378 --- PASS: TestIsValidCachePath/leading_slash (0.00s)1379 --- PASS: TestIsValidCachePath/empty (0.00s)1380 --- PASS: TestIsValidCachePath/random_path (0.00s)1381 --- PASS: TestIsValidCachePath/invalid_char_u (0.00s)1382 --- PASS: TestIsValidCachePath/invalid_char_e (0.00s)1383 --- PASS: TestIsValidCachePath/traversal_in_middle (0.00s)1384 --- PASS: TestIsValidCachePath/traversal_parent (0.00s)1385 --- PASS: TestIsValidCachePath/nar_xz (0.00s)1386 --- PASS: TestIsValidCachePath/nar_bz2 (0.00s)1387 --- PASS: TestIsValidCachePath/nar_uncompressed (0.00s)1388 --- PASS: TestIsValidCachePath/realisation (0.00s)1389 --- PASS: TestIsValidCachePath/log (0.00s)1390 --- PASS: TestIsValidCachePath/nix-cache-info (0.00s)1391 --- PASS: TestIsValidCachePath/ls (0.00s)1392 --- PASS: TestIsValidCachePath/nar_zst (0.00s)1393 --- PASS: TestIsValidCachePath/narinfo_all_nix_base32_chars (0.00s)1394=== CONT TestParseSingleRange/none1395=== CONT TestParseSingleRange/open-ended1396--- PASS: TestMultipartCleanup (0.38s)13972026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (1.16ms)1398=== CONT TestParseSingleRange/start_far_past_EOF1399=== CONT TestParseSingleRange/single_byte1400=== CONT TestParseSingleRange/suffix_exceeds_size1401=== CONT TestParseSingleRange/suffix1402=== CONT TestParseSingleRange/end_clamped_to_size1403=== CONT TestParseSingleRange/malformed_both_empty1404=== CONT TestParseSingleRange/start_past_EOF1405=== CONT TestParseSingleRange/malformed_end_before_start1406=== CONT TestParseSingleRange/multi-range_ignored1407=== CONT TestParseSingleRange/malformed_no_dash1408=== CONT TestParseSingleRange/closed1409=== CONT TestServerTLSConfig/no_client_CA1410=== CONT TestParseSingleRange/unknown_unit1411=== CONT TestServerTLSConfig/not_a_PEM_file1412=== CONT TestServerTLSConfig/missing_CA_file1413--- PASS: TestParseSingleRange (0.00s)1414 --- PASS: TestParseSingleRange/none (0.00s)1415 --- PASS: TestParseSingleRange/open-ended (0.00s)1416 --- PASS: TestParseSingleRange/start_far_past_EOF (0.00s)1417 --- PASS: TestParseSingleRange/single_byte (0.00s)1418 --- PASS: TestParseSingleRange/suffix_exceeds_size (0.00s)1419 --- PASS: TestParseSingleRange/suffix (0.00s)1420 --- PASS: TestParseSingleRange/end_clamped_to_size (0.00s)1421 --- PASS: TestParseSingleRange/malformed_both_empty (0.00s)1422 --- PASS: TestParseSingleRange/start_past_EOF (0.00s)1423 --- PASS: TestParseSingleRange/malformed_end_before_start (0.00s)1424 --- PASS: TestParseSingleRange/multi-range_ignored (0.00s)1425 --- PASS: TestParseSingleRange/malformed_no_dash (0.00s)1426 --- PASS: TestParseSingleRange/closed (0.00s)1427 --- PASS: TestParseSingleRange/unknown_unit (0.00s)1428=== CONT TestClientErrorHandling/InvalidStorePath1429--- PASS: TestCacheStatsHandler (0.20s)1430=== CONT TestClientErrorHandling/ServerNotAvailable14312026/08/27 09:40:18 OK 20251218171726_add_pins.sql (2.59ms)1432--- PASS: TestServerTLSConfig (0.00s)1433 --- PASS: TestServerTLSConfig/no_client_CA (0.00s)1434 --- PASS: TestServerTLSConfig/missing_CA_file (0.00s)1435 --- PASS: TestServerTLSConfig/not_a_PEM_file (0.00s)1436=== CONT TestClientErrorHandling/InvalidAuthToken14372026/08/27 09:40:18 OK 20251218171726_add_pins.sql (2.07ms)1438=== CONT TestCacheConfigHandler/full_config,_no_issuer1439=== CONT TestCacheConfigHandler/no_signing_keys1440=== CONT TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator1441=== CONT TestCacheConfigHandler/no_cache_url_configured1442--- PASS: TestCacheConfigHandler (0.00s)1443 --- PASS: TestCacheConfigHandler/full_config,_no_issuer (0.00s)1444 --- PASS: TestCacheConfigHandler/no_signing_keys (0.00s)1445 --- PASS: TestCacheConfigHandler/issuer_requested_but_no_OIDC_validator (0.00s)1446 --- PASS: TestCacheConfigHandler/no_cache_url_configured (0.00s)14472026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (2.84ms)14482026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000014492026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (2.49ms)14502026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000014512026/08/27 09:40:18 OK 1_commit_pending_closure.sql (1.99ms)14522026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.01ms)14532026/08/27 09:40:18 OK 2_object_stats_trigger.sql (699.04µs)14542026/08/27 09:40:18 goose: up to current file version: 214552026/08/27 09:40:18 OK 2_object_stats_trigger.sql (706.51µs)14562026/08/27 09:40:18 goose: up to current file version: 21457=== RUN TestService_AuthMiddleware_OIDC/valid_OIDC_token1458=== PAUSE TestService_AuthMiddleware_OIDC/valid_OIDC_token1459=== RUN TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1460=== PAUSE TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected1461=== NAME TestPinProtectsFromGC1462 client_integration_test.go:646: Pinned store path: /build/TestPinProtectsFromGC2971549412/001/store/a42zdvs5clz69857xmz31iph8q5x7hhr-pinned-file.txt1463 client_integration_test.go:647: Unpinned store path: /build/TestPinProtectsFromGC2971549412/001/store/mj9ih70mj5znqmrnrhrddsl6mlcamiac-unpinned-file.txt1464=== RUN TestService_AuthMiddleware_OIDC/malformed_token_rejected1465=== PAUSE TestService_AuthMiddleware_OIDC/malformed_token_rejected1466=== RUN TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1467=== PAUSE TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1468=== CONT TestService_AuthMiddleware_OIDC/valid_OIDC_token1469=== CONT TestService_AuthMiddleware_OIDC/malformed_token_rejected14702026-08-27 09:40:18.284 UTC [1183] ERROR: relation "goose_db_version" does not exist at character 3614712026-08-27 09:40:18.284 UTC [1183] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC14722026/08/27 09:40:18 WARN Authentication failed token_preview=not-a-valid-jwt token_length=15 oidc_error="no provider could verify the token (signature or issuer mismatch)" oidc_provider="" tried_providers=[test]1473=== CONT TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured1474=== CONT TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected14752026/08/27 09:40:18 OK 20241026095416_initial_model.sql (7.59ms)1476=== NAME TestClientMultipleUploads1477 client_integration_test.go:338: Created store path 0: /build/TestClientMultipleUploads2451727888/001/store/gdfsi4qf1lqj4y94vrlii3kja5ss6fvs-test-file-0.txt14782026/08/27 09:40:18 INFO OIDC auth successful provider=test14792026/08/27 09:40:18 WARN Authentication failed token_preview=eyJhbGciOi...AbijIf2hxA token_length=702 oidc_error="bound claims validation failed: claim \"repository_owner\" value [otherorg] not in allowed patterns [myorg]" oidc_provider=test tried_providers=[test]1480--- PASS: TestService_AuthMiddleware_OIDC (0.17s)1481 --- PASS: TestService_AuthMiddleware_OIDC/malformed_token_rejected (0.00s)1482 --- PASS: TestService_AuthMiddleware_OIDC/static_token_still_works_with_OIDC_configured (0.00s)1483 --- PASS: TestService_AuthMiddleware_OIDC/valid_OIDC_token (0.00s)1484 --- PASS: TestService_AuthMiddleware_OIDC/OIDC_token_with_wrong_org_rejected (0.00s)14852026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (1.83ms)14862026/08/27 09:40:18 OK 20251218171726_add_pins.sql (3.2ms)1487=== NAME TestClientIntegration1488 client_integration_test.go:276: Created store path: /build/TestClientIntegration3420447798/002/store/pw6qfivgq9lskgdil3h3d0cw8rz87hh0-test-file.txt14892026/08/27 09:40:18 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted,O=Other"14902026/08/27 09:40:18 WARN mTLS auth: bound subjects configured but subject DN unavailable14912026/08/27 09:40:18 WARN mTLS auth: subject not in bound subjects subject="CN=untrusted"1492--- PASS: TestService_AuthMiddleware_MTLSBoundSubjects (0.17s)14932026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"14942026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.35ms)14952026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000014962026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.69ms)14972026/08/27 09:40:18 OK 20241026095416_initial_model.sql (9.8ms)14982026/08/27 09:40:18 OK 2_object_stats_trigger.sql (2.11ms)14992026/08/27 09:40:18 goose: up to current file version: 215002026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (2.49ms)15012026/08/27 09:40:18 WARN mTLS auth: subject not in bound subjects subject="CN=writer"1502--- PASS: TestService_ReadAuthMiddleware (0.18s)15032026/08/27 09:40:18 OK 20251218171726_add_pins.sql (3.13ms)1504=== NAME TestClientWithDependencies1505 client_integration_test.go:593: Built derivation: /build/TestClientWithDependencies1876973562/001/store/jbqx8fz11ffsrd6dixmmrbg774f2f1d9-test-script15062026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (4.59ms)15072026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000015082026/08/27 09:40:18 OK 1_commit_pending_closure.sql (2.99ms)15092026/08/27 09:40:18 OK 2_object_stats_trigger.sql (1.28ms)15102026/08/27 09:40:18 goose: up to current file version: 215112026/08/27 09:40:18 INFO Aborted multipart uploads count=015122026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15132026/08/27 09:40:18 WARN Force mode enabled - objects will be deleted immediately without grace period1514=== NAME TestClientMultipleUploads1515 client_integration_test.go:338: Created store path 1: /build/TestClientMultipleUploads2451727888/001/store/nvwzf7baxpzgzxdj7d4jc1nckpa71hzg-test-file-1.txt15162026/08/27 09:40:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=0 objects-marked-for-deletion=0 objects-deleted-after-grace-period=0 objects-failed-to-delete=015172026/08/27 09:40:18 INFO Vacuumed table table=pending_closures15182026/08/27 09:40:18 INFO Vacuumed table table=pending_objects1519--- PASS: TestService_AuthMiddleware_MTLSProxyHeader (0.15s)15202026/08/27 09:40:18 INFO Vacuumed table table=multipart_uploads15212026/08/27 09:40:18 INFO Vacuumed table table=closures15222026/08/27 09:40:18 INFO Vacuumed table table=objects15232026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures1524--- PASS: TestGCMetrics (0.17s)1525=== NAME TestClientCADerivations1526 client_ca_test.go:136: Built CA derivation: /build/TestClientCADerivations125866507/001/store/9aam2y77n5v4id3f3m9qrc22knz70lp4-ca-test15272026/08/27 09:40:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15282026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete15292026/08/27 09:40:18 INFO Uploading 95hsp8q3ar4mbq605agf24vb8ykvihry-file1.txt (160B)1530=== NAME TestClientWithDependencies1531 client_integration_test.go:595: Found 1 dependencies (including self)15322026/08/27 09:40:18 INFO Completed multipart upload object_key=nar/dadb44fdadb44fdadb44fdadb44f000000000000000000000000.nar.zst upload_id=MzdiYTliZTItZDg3NC00ZDJkLWEwZjgtN2ZjYTA0MzM5NjMzLmVhOWNkYThmLWJjOWYtNDcyMi1iZTczLTVmMjk0YWM4MzUxMHgxNzg3ODIzNjE3OTE4ODI4NzY1 parts=1015332026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15342026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"15352026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst error="server returned 404: 404 page not found\n"15362026/08/27 09:40:18 INFO Completed upload id=115372026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures15382026/08/27 09:40:18 WARN Failed to register uploaded object key=95hsp8q3ar4mbq605agf24vb8ykvihry.ls error="server returned 404: 404 page not found\n"15392026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign15402026/08/27 09:40:18 INFO Signed narinfos id=1 count=115412026/08/27 09:40:18 INFO Uploading 1 narinfos15422026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1543=== NAME TestClientMultipleUploads1544 client_integration_test.go:338: Created store path 2: /build/TestClientMultipleUploads2451727888/001/store/fvgj78w8glykjvi7byrlmli2rf3s5qzn-test-file-2.txt15452026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures15462026/08/27 09:40:18 WARN Failed to register uploaded object key=95hsp8q3ar4mbq605agf24vb8ykvihry.narinfo error="server returned 404: 404 page not found\n"15472026/08/27 09:40:18 INFO Object in database but missing from S3 key=dadb44fdadb44fdadb44fdadb44f0000.narinfo15482026/08/27 09:40:18 WARN Found objects in DB but missing from S3, will re-upload count=115492026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1550--- PASS: TestService_verifyS3Integrity (0.89s)15512026/08/27 09:40:18 INFO Completed multipart upload object_key=nar/0000000000000000000000000000000000000000000000000000.nar.zst upload_id=MzdiYTliZTItZDg3NC00ZDJkLWEwZjgtN2ZjYTA0MzM5NjMzLjRkY2QxYWVmLTk1MWYtNGVlZi05YTIzLTU1ZGZlZmM3NmExYXgxNzg3ODIzNjE3OTMxOTY0NjYz parts=1015522026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete15532026/08/27 09:40:18 INFO Completed upload id=115542026/08/27 09:40:18 INFO Upload complete. (110ms)1555=== NAME TestClientCADerivations1556 client_ca_test.go:139: Found 1 dependencies (including self)1557=== NAME TestNARDeduplicationMetadataUploadBug1558 metadata_upload_test.go:54: Retrieved narinfo from S3:1559 StorePath: /build/TestNARDeduplicationMetadataUploadBug2778236546/001/store/95hsp8q3ar4mbq605agf24vb8ykvihry-file1.txt1560 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1561 Compression: zstd1562 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1563 NarSize: 1601564 References: 1565 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf15662026/08/27 09:40:18 INFO Completed upload id=115672026/08/27 09:40:18 INFO Received get closure request method=GET path=/api/closures/000000000000000000000000000000001568 metadata_upload_test.go:55: Retrieved .ls file from S3 (compressed size: 77 bytes)1569 metadata_upload_test.go:55: Decompressed .ls content (64 bytes):1570 {"version":1,"root":{"type":"regular","size":44,"narOffset":96}}15712026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures15722026-08-27 09:40:18.381 UTC [1499] ERROR: relation "goose_db_version" does not exist at character 3615732026-08-27 09:40:18.381 UTC [1499] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC15742026/08/27 09:40:18 INFO Starting cleanup of old closures method=DELETE path=/api/closures15752026-08-27 09:40:18.386 UTC [1519] ERROR: relation "goose_db_version" does not exist at character 3615762026-08-27 09:40:18.386 UTC [1519] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC15772026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures15782026/08/27 09:40:18 INFO Aborted multipart uploads count=015792026/08/27 09:40:18 OK 20241026095416_initial_model.sql (9.02ms)15802026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (1.1ms)15812026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures15822026/08/27 09:40:18 OK 20241026095416_initial_model.sql (7.08ms)15832026/08/27 09:40:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15842026/08/27 09:40:18 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config15852026/08/27 09:40:18 INFO Uploading a42zdvs5clz69857xmz31iph8q5x7hhr-pinned-file.txt (128B)15862026/08/27 09:40:18 OK 20251210153512_drop_unused_gin_index.sql (887.73µs)15872026/08/27 09:40:18 OK 20251218171726_add_pins.sql (2.85ms)15882026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/0qmhsg42pqsz7x6hh02j8gznnac3yi3qpjhrqbwvz88i194jbb5s.nar.zst error="server returned 404: 404 page not found\n"15892026/08/27 09:40:18 OK 20251218171726_add_pins.sql (2.48ms)15902026/08/27 09:40:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=2 objects-deleted-after-grace-period=0 objects-failed-to-delete=015912026/08/27 09:40:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)15922026/08/27 09:40:18 INFO Uploading pw6qfivgq9lskgdil3h3d0cw8rz87hh0-test-file.txt (152B)15932026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (3.05ms)15942026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000015952026/08/27 09:40:18 OK 20260628120000_add_object_size_and_stats.sql (2.77ms)15962026/08/27 09:40:18 goose: successfully migrated database to version: 2026062812000015972026/08/27 09:40:18 OK 1_commit_pending_closure.sql (1.75ms)15982026/08/27 09:40:18 INFO Vacuumed table table=pending_closures15992026/08/27 09:40:18 WARN Failed to register uploaded object key=a42zdvs5clz69857xmz31iph8q5x7hhr.ls error="server returned 404: 404 page not found\n"16002026/08/27 09:40:18 OK 1_commit_pending_closure.sql (1.6ms)16012026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign16022026/08/27 09:40:18 INFO Signed narinfos id=1 count=116032026/08/27 09:40:18 OK 2_object_stats_trigger.sql (1.25ms)16042026/08/27 09:40:18 goose: up to current file version: 216052026/08/27 09:40:18 INFO Uploading 1 narinfos16062026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst error="server returned 404: 404 page not found\n"16072026/08/27 09:40:18 OK 2_object_stats_trigger.sql (739.61µs)16082026/08/27 09:40:18 goose: up to current file version: 216092026/08/27 09:40:18 INFO Vacuumed table table=pending_objects16102026/08/27 09:40:18 INFO Vacuumed table table=multipart_uploads16112026/08/27 09:40:18 WARN Failed to register uploaded object key=a42zdvs5clz69857xmz31iph8q5x7hhr.narinfo error="server returned 404: 404 page not found\n"16122026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16132026/08/27 09:40:18 WARN Failed to register uploaded object key=pw6qfivgq9lskgdil3h3d0cw8rz87hh0.ls error="server returned 404: 404 page not found\n"16142026/08/27 09:40:18 INFO Vacuumed table table=closures16152026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign16162026/08/27 09:40:18 INFO Signed narinfos id=1 count=116172026/08/27 09:40:18 INFO Uploading 1 narinfos16182026/08/27 09:40:18 INFO Vacuumed table table=objects16192026/08/27 09:40:18 WARN Failed to register uploaded object key=pw6qfivgq9lskgdil3h3d0cw8rz87hh0.narinfo error="server returned 404: 404 page not found\n"16202026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete16212026/08/27 09:40:18 INFO Completed upload id=116222026/08/27 09:40:18 INFO Upload complete. (96ms)16232026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"16242026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures16252026/08/27 09:40:18 INFO Completed upload id=116262026/08/27 09:40:18 INFO Upload complete. (93ms)1627=== NAME TestClientIntegration1628 client_integration_test.go:292: Retrieved narinfo from S3:1629 StorePath: /build/TestClientIntegration3420447798/002/store/pw6qfivgq9lskgdil3h3d0cw8rz87hh0-test-file.txt1630 URL: nar/11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk1.nar.zst1631 Compression: zstd1632 NarHash: sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk11633 NarSize: 1521634 References: 1635 CA: fixed:r:sha256:11l1b7hsy9g9pz8i2rcn4n128l6pyr2yf58d49r12hr5caw9dzk116362026/08/27 09:40:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16372026/08/27 09:40:18 INFO Uploading jbqx8fz11ffsrd6dixmmrbg774f2f1d9-test-script (136B)1638 client_integration_test.go:293: Retrieved .ls file from S3 (compressed size: 77 bytes)1639 client_integration_test.go:293: Decompressed .ls content (64 bytes):1640 {"version":1,"root":{"type":"regular","size":39,"narOffset":96}}1641 client_integration_test.go:296: Testing garbage collection...16422026/08/27 09:40:18 WARN Failed to register uploaded object key=log/66y8awqi5ms9pfq88fsgc098710frc7k-test-script.drv error="server returned 404: 404 page not found\n"16432026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/1s7yghixzpinyqm7g59kyn3sav81pmnsdi5v63m6x4r2vacy907j.nar.zst error="server returned 404: 404 page not found\n"16442026/08/27 09:40:18 WARN Failed to register uploaded object key=jbqx8fz11ffsrd6dixmmrbg774f2f1d9.ls error="server returned 404: 404 page not found\n"16452026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign1646=== NAME TestOrphanedObjectsGC1647 orphaned_objects_gc_test.go:290: GC Test Summary:1648 orphaned_objects_gc_test.go:291: - Kept: 2 objects from closure A1649 orphaned_objects_gc_test.go:292: - Deleted: 2 objects from closure B16502026/08/27 09:40:18 INFO Signed narinfos id=1 count=11651 orphaned_objects_gc_test.go:293: - Deleted: 6 orphaned chain objects (X1->X2->X3)1652 orphaned_objects_gc_test.go:294: - Deleted: 2 orphaned single objects (Y)1653 orphaned_objects_gc_test.go:295: - Total deleted: 10 objects16542026/08/27 09:40:18 INFO Uploading 1 narinfos1655--- PASS: TestOrphanedObjectsGC (0.56s)16562026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1657--- PASS: TestGCBugBareHashReferences (0.43s)16582026/08/27 09:40:18 INFO Received get closure request method=GET path=/api/closures/0000000000000000000000000000000016592026/08/27 09:40:18 WARN Failed to register uploaded object key=jbqx8fz11ffsrd6dixmmrbg774f2f1d9.narinfo error="server returned 404: 404 page not found\n"16602026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete1661--- PASS: TestService_createPendingClosureHandler (0.95s)16622026/08/27 09:40:18 INFO Completed upload id=116632026/08/27 09:40:18 INFO Upload complete. (57ms)1664=== NAME TestNARDeduplicationMetadataUploadBug1665 metadata_upload_test.go:64: Second store path (same content): /build/TestNARDeduplicationMetadataUploadBug2778236546/001/store/c34i0bq6raxjpw4jlgdgg5mms814bwb5-file2.txt1666=== NAME TestClientWithDependencies1667 client_integration_test.go:597: Skipping nix copy test - isolated store (/build/TestClientWithDependencies1876973562/001/store) requires matching store prefix16682026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1669--- PASS: TestClientWithDependencies (0.47s)16702026/08/27 09:40:18 INFO Starting cleanup of old closures method=DELETE path=/api/closures16712026/08/27 09:40:18 INFO Garbage collection started16722026/08/27 09:40:18 INFO Aborted multipart uploads count=016732026/08/27 09:40:18 WARN Force mode enabled - objects will be deleted immediately without grace period16742026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures16752026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures16762026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures16772026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures16782026/08/27 09:40:18 INFO Uploading 3 paths to 127.0.0.1 (0 already cached)16792026/08/27 09:40:18 INFO Uploading fvgj78w8glykjvi7byrlmli2rf3s5qzn-test-file-2.txt (160B)16802026/08/27 09:40:18 INFO Uploading gdfsi4qf1lqj4y94vrlii3kja5ss6fvs-test-file-0.txt (160B)16812026/08/27 09:40:18 INFO Uploading nvwzf7baxpzgzxdj7d4jc1nckpa71hzg-test-file-1.txt (160B)16822026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/0ahq3izw3h0b0c08p3xdfclh758fwxnp85ccvflasb4vnfsv50ri.nar.zst error="server returned 404: 404 page not found\n"16832026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/1823g6vx4q5zqvr73m7lkq1wpk40fin7bsc792di430yk7sg4lka.nar.zst error="server returned 404: 404 page not found\n"16842026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/13p9pqi0ga6fx2apcwls9hjnv7nmf740ggpn5zvfzdsp5vz2bbg4.nar.zst error="server returned 404: 404 page not found\n"16852026/08/27 09:40:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)16862026/08/27 09:40:18 INFO Uploading 9aam2y77n5v4id3f3m9qrc22knz70lp4-ca-test (144B)16872026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"16882026/08/27 09:40:18 WARN Failed to register uploaded object key=nvwzf7baxpzgzxdj7d4jc1nckpa71hzg.ls error="server returned 404: 404 page not found\n"16892026/08/27 09:40:18 WARN Failed to register uploaded object key=gdfsi4qf1lqj4y94vrlii3kja5ss6fvs.ls error="server returned 404: 404 page not found\n"16902026/08/27 09:40:18 WARN Failed to register uploaded object key=fvgj78w8glykjvi7byrlmli2rf3s5qzn.ls error="server returned 404: 404 page not found\n"16912026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign16922026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst error="server returned 404: 404 page not found\n"16932026/08/27 09:40:18 WARN Failed to register uploaded object key=log/ppw5ihjh8f6wf3m8v6rxjsxcsm8vndgc-ca-test.drv error="server returned 404: 404 page not found\n"16942026/08/27 09:40:18 INFO Signed narinfos id=1 count=116952026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign16962026/08/27 09:40:18 INFO Signed narinfos id=2 count=116972026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign16982026/08/27 09:40:18 INFO Signed narinfos id=3 count=116992026/08/27 09:40:18 INFO Uploading 3 narinfos17002026/08/27 09:40:18 WARN Failed to register uploaded object key=9aam2y77n5v4id3f3m9qrc22knz70lp4.ls error="server returned 404: 404 page not found\n"17012026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign17022026/08/27 09:40:18 INFO Signed narinfos id=1 count=117032026/08/27 09:40:18 INFO Uploading 1 narinfos17042026/08/27 09:40:18 WARN Failed to register uploaded object key=gdfsi4qf1lqj4y94vrlii3kja5ss6fvs.narinfo error="server returned 404: 404 page not found\n"17052026/08/27 09:40:18 WARN Failed to register uploaded object key=fvgj78w8glykjvi7byrlmli2rf3s5qzn.narinfo error="server returned 404: 404 page not found\n"17062026/08/27 09:40:18 WARN Failed to register uploaded object key=nvwzf7baxpzgzxdj7d4jc1nckpa71hzg.narinfo error="server returned 404: 404 page not found\n"17072026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete17082026/08/27 09:40:18 WARN Failed to register uploaded object key=9aam2y77n5v4id3f3m9qrc22knz70lp4.narinfo error="server returned 404: 404 page not found\n"17092026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17102026/08/27 09:40:18 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=195.870311ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config17112026/08/27 09:40:18 INFO Completed upload id=317122026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete17132026/08/27 09:40:18 INFO Completed upload id=117142026/08/27 09:40:18 INFO Upload complete. (94ms)17152026/08/27 09:40:18 INFO Completed upload id=117162026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17172026/08/27 09:40:18 INFO Completed upload id=217182026/08/27 09:40:18 INFO Upload complete. (105ms)1719=== NAME TestClientMultipleUploads17202026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"1721 client_integration_test.go:349: Uploaded 3 paths in 140.3254ms1722=== NAME TestClientCADerivations1723 client_ca_test.go:180: Narinfo contains CA field: StorePath: /build/TestClientCADerivations125866507/001/store/9aam2y77n5v4id3f3m9qrc22knz70lp4-ca-test1724 URL: nar/04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n.nar.zst1725 Compression: zstd1726 NarHash: sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1727 NarSize: 1441728 References: 1729 Deriver: /build/TestClientCADerivations125866507/001/store/ppw5ihjh8f6wf3m8v6rxjsxcsm8vndgc-ca-test.drv1730 CA: fixed:r:sha256:04jj7iiy79vhzp5kk3r2751n9x3xa44j98y5bk1ds4jp31kky74n1731 client_ca_test.go:185: Checking for realisation files in S3...1732 client_ca_test.go:207: No realisation files uploaded - this is expected for locally built CA derivations1733 client_ca_test.go:208: Locally built CA derivations don't have DrvOutput IDs until they're substituted from a cache1734--- PASS: TestClientMultipleUploads (0.47s)17352026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures17362026/08/27 09:40:18 INFO Uploading 1 paths to 127.0.0.1 (0 already cached)17372026/08/27 09:40:18 INFO Uploading mj9ih70mj5znqmrnrhrddsl6mlcamiac-unpinned-file.txt (128B)17382026/08/27 09:40:18 WARN Failed to register uploaded object key=nar/0gh0wdnr6gmk4r5sfzzh12isnl14rgmkrwy1ws41hpsrlndclg5h.nar.zst error="server returned 404: 404 page not found\n"17392026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17402026/08/27 09:40:18 WARN Failed to register uploaded object key=mj9ih70mj5znqmrnrhrddsl6mlcamiac.ls error="server returned 404: 404 page not found\n"17412026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign17422026/08/27 09:40:18 INFO Signed narinfos id=2 count=117432026/08/27 09:40:18 INFO Uploading 1 narinfos17442026/08/27 09:40:18 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: server returned 404: 404 page not found\n"17452026/08/27 09:40:18 WARN Failed to register uploaded object key=mj9ih70mj5znqmrnrhrddsl6mlcamiac.narinfo error="server returned 404: 404 page not found\n"17462026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete17472026/08/27 09:40:18 INFO Completed upload id=217482026/08/27 09:40:18 INFO Upload complete. (87ms)17492026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures17502026/08/27 09:40:18 INFO Uploading 0 paths to 127.0.0.1 (1 already cached)17512026/08/27 09:40:18 WARN Failed to register uploaded object key=c34i0bq6raxjpw4jlgdgg5mms814bwb5.ls error="server returned 404: 404 page not found\n"17522026/08/27 09:40:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign17532026/08/27 09:40:18 INFO Signed narinfos id=2 count=117542026/08/27 09:40:18 INFO Uploading 1 narinfos17552026/08/27 09:40:18 INFO Completed multipart upload object_key=nar/hhhhhhhhhhhhhhhhhhhhhhhhhhhhhh0100000000000000000000.nar.zst upload_id=MzdiYTliZTItZDg3NC00ZDJkLWEwZjgtN2ZjYTA0MzM5NjMzLjYzODcyNjQwLTMxMTktNDlkZi05MjdjLWU5YTRlM2VmNDljMHgxNzg3ODIzNjE4MDI2NzcyNzAw parts=1217562026/08/27 09:40:18 INFO Received uploads request method=POST path=/api/pending_closures17572026/08/27 09:40:18 WARN Failed to register uploaded object key=c34i0bq6raxjpw4jlgdgg5mms814bwb5.narinfo error="server returned 404: 404 page not found\n"17582026/08/27 09:40:18 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete1759--- PASS: TestCompletedNarNotReofferedAcrossClosures (0.89s)17602026/08/27 09:40:18 INFO Completed upload id=217612026/08/27 09:40:18 INFO Upload complete. (74ms)1762=== NAME TestNARDeduplicationMetadataUploadBug1763 metadata_upload_test.go:76: Retrieved narinfo from S3:1764 StorePath: /build/TestNARDeduplicationMetadataUploadBug2778236546/001/store/c34i0bq6raxjpw4jlgdgg5mms814bwb5-file2.txt1765 URL: nar/1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf.nar.zst1766 Compression: zstd1767 NarHash: sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1768 NarSize: 1601769 References: 1770 CA: fixed:r:sha256:1byf36hhc5sn8lvwqfkjh3ybgn76slmx8naakslrypxs6mna71rf1771 metadata_upload_test.go:77: Retrieved .ls file from S3 (compressed size: 62 bytes)1772 metadata_upload_test.go:77: Decompressed .ls content (49 bytes):1773 {"version":1,"root":{"type":"regular","size":44}}1774--- PASS: TestNARDeduplicationMetadataUploadBug (0.59s)17752026/08/27 09:40:18 WARN Authentication failed token_preview=invalid-token token_length=13 reason="static API token mismatch"17762026/08/27 09:40:18 INFO Received complete multipart upload request method=POST path=/api/multipart/complete17772026/08/27 09:40:18 INFO Completed multipart upload object_key=nar/gggggggggggggggggggggggggggggg0100000000000000000000.nar.zst upload_id=MzdiYTliZTItZDg3NC00ZDJkLWEwZjgtN2ZjYTA0MzM5NjMzLjdlZGRkYmQ4LWQ1YmUtNDJmZS1iZDI5LWZiZGNlNGRmOTUyZXgxNzg3ODIzNjE4MDQ3MTAyOTg3 parts=121778--- PASS: TestRedundantMultipartUpload (0.93s)17792026/08/27 09:40:18 INFO Received create pin request method=POST path=/api/pins/myapp17802026/08/27 09:40:18 INFO Created/updated pin name=myapp store_path=/build/TestPinProtectsFromGC2971549412/001/store/a42zdvs5clz69857xmz31iph8q5x7hhr-pinned-file.txt narinfo_key=a42zdvs5clz69857xmz31iph8q5x7hhr.narinfo17812026/08/27 09:40:18 INFO Starting cleanup of old closures method=DELETE path=/api/closures17822026/08/27 09:40:18 INFO Garbage collection started17832026/08/27 09:40:18 INFO Aborted multipart uploads count=017842026/08/27 09:40:18 WARN Force mode enabled - objects will be deleted immediately without grace period1785=== NAME TestClientCADerivations1786 client_ca_test.go:258: nix copy output: warning: you don't have Internet access; disabling some network-dependent features1787 warning: failed to create TLS context for AWS credential providers; SSO, STS WebIdentity, and ECS container authentication will be unavailable1788 error: binary cache 's3://bucket40?endpoint=http://localhost:34419®ion=eu-west-1' is for Nix stores with prefix '/nix/store', not '/build/TestClientCADerivations125866507/001/store'1789 client_ca_test.go:259: nix copy failed (might be expected with isolated stores): exit status 11790--- PASS: TestClientCADerivations (0.58s)17912026/08/27 09:40:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=017922026/08/27 09:40:18 INFO Vacuumed table table=pending_closures17932026/08/27 09:40:18 INFO Vacuumed table table=pending_objects17942026/08/27 09:40:18 INFO Vacuumed table table=multipart_uploads17952026/08/27 09:40:18 INFO Vacuumed table table=closures17962026/08/27 09:40:18 INFO Vacuumed table table=objects17972026/08/27 09:40:18 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=395.330553ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config1798--- PASS: TestUploadHandlersRejectOversizedBody (0.26s)1799 --- PASS: TestUploadHandlersRejectOversizedBody/complete_multipart (0.06s)1800 --- PASS: TestUploadHandlersRejectOversizedBody/request_more_parts (0.07s)1801 --- PASS: TestUploadHandlersRejectOversizedBody/create_pending_closure (0.73s)18022026/08/27 09:40:18 INFO Garbage collection completed failed-uploads-deleted=0 old-closures-deleted=1 objects-marked-for-deletion=3 objects-deleted-after-grace-period=2001 objects-failed-to-delete=018032026/08/27 09:40:18 INFO Vacuumed table table=pending_closures18042026/08/27 09:40:18 INFO Vacuumed table table=pending_objects18052026/08/27 09:40:18 INFO Vacuumed table table=multipart_uploads18062026/08/27 09:40:18 INFO Vacuumed table table=closures18072026/08/27 09:40:18 INFO Vacuumed table table=objects18082026/08/27 09:40:19 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=851.076882ms error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config1809=== NAME TestOrphanedObjectsGCStressTest1810 orphaned_objects_gc_test.go:425: Created 10 active closures, 5 to-delete closures, 20 orphaned chains1811 orphaned_objects_gc_test.go:446: Marked 210 objects for deletion1812 orphaned_objects_gc_test.go:509: Stress test completed successfully:1813 orphaned_objects_gc_test.go:510: - Active objects preserved: 201814 orphaned_objects_gc_test.go:511: - Objects deleted: 2101815 orphaned_objects_gc_test.go:512: - Total GC'd: 2101816--- PASS: TestOrphanedObjectsGCStressTest (1.63s)18172026/08/27 09:40:19 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.550381431s error="Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/cache-config18182026/08/27 09:40:20 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=01819=== NAME TestClientIntegration1820 client_integration_test.go:303: Objects in database after GC:1821 client_integration_test.go:303: Successfully deleted all objects with GC --force1822--- PASS: TestClientIntegration (2.41s)18232026/08/27 09:40:20 INFO Garbage collection progress phase="" failed_uploads_deleted=0 old_closures_deleted=1 objects_marked=3 objects_deleted=2001 objects_failed=01824=== NAME TestPinProtectsFromGC1825 client_integration_test.go:709: Pin successfully protected closure from garbage collection1826--- PASS: TestPinProtectsFromGC (2.62s)18272026/08/27 09:40:21 WARN Failed to fetch cache config, uploading without size limit error="fetching cache config: sending request: request failed after retries: Get \"http://localhost:19999/api/cache-config\": dial tcp [::1]:19999: connect: connection refused"18282026/08/27 09:40:21 WARN Request failed, retrying attempt=1 max_attempts=6 backoff=100ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures18292026/08/27 09:40:21 WARN Request failed, retrying attempt=2 max_attempts=6 backoff=194.091174ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures18302026/08/27 09:40:21 WARN Rate limiter enabled after throttle name=s3-test rate=518312026/08/27 09:40:21 WARN S3 rate limit hit operation="complete multipart upload" error="Please reduce your request rate."1832=== NAME TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle1833 throttle_test.go:213: Proxy stats: total=15, throttled=10, completeMultipart=101834 throttle_test.go:215: Rate limiter: enabled=true, rate=5.001835--- PASS: TestCompleteMultipartUploadHandler_RateLimitTriggersThrottle (4.33s)18362026/08/27 09:40:21 WARN Request failed, retrying attempt=3 max_attempts=6 backoff=437.012445ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures18372026/08/27 09:40:22 WARN Request failed, retrying attempt=4 max_attempts=6 backoff=750.562615ms error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures18382026/08/27 09:40:23 WARN Request failed, retrying attempt=5 max_attempts=6 backoff=1.515585985s error="Post \"http://localhost:19999/api/pending_closures\": dial tcp [::1]:19999: connect: connection refused" url=http://localhost:19999/api/pending_closures1839--- PASS: TestClientErrorHandling (0.00s)1840 --- PASS: TestClientErrorHandling/InvalidStorePath (0.18s)1841 --- PASS: TestClientErrorHandling/InvalidAuthToken (0.29s)1842 --- PASS: TestClientErrorHandling/ServerNotAvailable (6.27s)1843PASS18442026-08-27 09:40:24.792 UTC [112] LOG: received smart shutdown request18452026-08-27 09:40:24.797 UTC [112] LOG: background worker "logical replication launcher" (PID 122) exited with exit code 118462026-08-27 09:40:24.806 UTC [117] LOG: shutting down18472026-08-27 09:40:24.807 UTC [117] LOG: checkpoint starting: shutdown immediate18482026-08-27 09:40:26.026 UTC [117] LOG: checkpoint complete: wrote 11380 buffers (69.5%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 13 recycled; write=0.362 s, sync=0.850 s, total=1.220 s; sync files=15825, longest=0.007 s, average=0.001 s; distance=217981 kB, estimate=217981 kB; lsn=0/EC431A0, redo lsn=0/EC431A018492026-08-27 09:40:26.081 UTC [112] LOG: database system is shut down1850Running OIDC tests...1851=== RUN TestGlobMatch1852=== PAUSE TestGlobMatch1853=== RUN TestAudienceForIssuer1854=== PAUSE TestAudienceForIssuer1855=== RUN TestValidateToken_ValidToken1856=== PAUSE TestValidateToken_ValidToken1857=== RUN TestValidateToken_WrongAudience1858=== PAUSE TestValidateToken_WrongAudience1859=== RUN TestValidateToken_Expired1860=== PAUSE TestValidateToken_Expired1861=== RUN TestValidateToken_BoundClaimsMismatch1862=== PAUSE TestValidateToken_BoundClaimsMismatch1863=== RUN TestValidateToken_BoundSubjectMismatch1864=== PAUSE TestValidateToken_BoundSubjectMismatch1865=== RUN TestValidateToken_MultipleProviders1866=== PAUSE TestValidateToken_MultipleProviders1867=== RUN TestValidateToken_NoMatchingProvider1868=== PAUSE TestValidateToken_NoMatchingProvider1869=== CONT TestGlobMatch1870=== CONT TestValidateToken_NoMatchingProvider1871=== CONT TestValidateToken_ValidToken1872=== RUN TestGlobMatch/foo_foo1873=== PAUSE TestGlobMatch/foo_foo1874=== CONT TestValidateToken_BoundClaimsMismatch1875=== CONT TestAudienceForIssuer1876--- PASS: TestAudienceForIssuer (0.00s)1877=== CONT TestValidateToken_MultipleProviders1878=== CONT TestValidateToken_BoundSubjectMismatch1879=== CONT TestValidateToken_Expired1880=== CONT TestValidateToken_WrongAudience1881=== RUN TestGlobMatch/foo_bar1882=== PAUSE TestGlobMatch/foo_bar1883=== RUN TestGlobMatch/*_1884=== PAUSE TestGlobMatch/*_1885=== RUN TestGlobMatch/*_anything1886=== PAUSE TestGlobMatch/*_anything1887=== RUN TestGlobMatch/foo*_foo1888=== PAUSE TestGlobMatch/foo*_foo1889=== RUN TestGlobMatch/foo*_foobar1890=== PAUSE TestGlobMatch/foo*_foobar1891=== RUN TestGlobMatch/foo*_bar1892=== PAUSE TestGlobMatch/foo*_bar1893=== RUN TestGlobMatch/*bar_bar1894=== PAUSE TestGlobMatch/*bar_bar1895=== RUN TestGlobMatch/*bar_foobar1896=== PAUSE TestGlobMatch/*bar_foobar1897=== RUN TestGlobMatch/*bar_foo1898=== PAUSE TestGlobMatch/*bar_foo1899=== RUN TestGlobMatch/foo*bar_foobar1900=== PAUSE TestGlobMatch/foo*bar_foobar1901=== RUN TestGlobMatch/foo*bar_foo123bar1902=== PAUSE TestGlobMatch/foo*bar_foo123bar1903=== RUN TestGlobMatch/foo*bar_foobarbaz1904=== PAUSE TestGlobMatch/foo*bar_foobarbaz1905=== RUN TestGlobMatch/*/*_foo/bar1906=== PAUSE TestGlobMatch/*/*_foo/bar1907=== RUN TestGlobMatch/*/*_foo1908=== PAUSE TestGlobMatch/*/*_foo1909=== RUN TestGlobMatch/refs/heads/*_refs/heads/main1910=== PAUSE TestGlobMatch/refs/heads/*_refs/heads/main1911=== RUN TestGlobMatch/refs/heads/*_refs/tags/v1.01912=== PAUSE TestGlobMatch/refs/heads/*_refs/tags/v1.01913=== RUN TestGlobMatch/refs/*/main_refs/heads/main1914=== PAUSE TestGlobMatch/refs/*/main_refs/heads/main1915=== RUN TestGlobMatch/fo?_foo1916=== PAUSE TestGlobMatch/fo?_foo1917=== RUN TestGlobMatch/fo?_fo1918=== PAUSE TestGlobMatch/fo?_fo1919=== RUN TestGlobMatch/fo?_fooo1920=== PAUSE TestGlobMatch/fo?_fooo19212026/08/27 09:40:26 INFO OIDC provider initialized name=test1922=== RUN TestGlobMatch/?oo_foo1923=== PAUSE TestGlobMatch/?oo_foo1924=== RUN TestGlobMatch/?oo_boo19252026/08/27 09:40:26 INFO OIDC provider initialized name=provider11926=== PAUSE TestGlobMatch/?oo_boo1927=== RUN TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main1928=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main19292026/08/27 09:40:26 INFO OIDC provider initialized name=provider11930=== RUN TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main1931=== PAUSE TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main1932=== CONT TestGlobMatch/foo_foo19332026/08/27 09:40:26 INFO OIDC provider initialized name=test1934=== CONT TestGlobMatch/*/*_foo/bar19352026/08/27 09:40:26 INFO OIDC provider initialized name=test1936=== CONT TestGlobMatch/*_1937=== CONT TestGlobMatch/foo*_foobar1938=== CONT TestGlobMatch/*_anything19392026/08/27 09:40:26 INFO OIDC provider initialized name=test1940=== CONT TestGlobMatch/fo?_fo19412026/08/27 09:40:26 INFO OIDC provider initialized name=test1942=== CONT TestGlobMatch/*bar_bar1943=== CONT TestGlobMatch/foo*_bar1944=== CONT TestGlobMatch/foo_bar19452026/08/27 09:40:26 INFO OIDC provider initialized name=provider21946=== CONT TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main1947=== CONT TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main1948=== CONT TestGlobMatch/?oo_boo1949=== CONT TestGlobMatch/?oo_foo1950=== CONT TestGlobMatch/foo*bar_foobarbaz1951=== CONT TestGlobMatch/refs/heads/*_refs/tags/v1.01952=== CONT TestGlobMatch/foo*bar_foo123bar1953=== CONT TestGlobMatch/fo?_fooo1954=== CONT TestGlobMatch/fo?_foo1955=== CONT TestGlobMatch/foo*bar_foobar1956=== CONT TestGlobMatch/refs/*/main_refs/heads/main1957=== CONT TestGlobMatch/refs/heads/*_refs/heads/main1958=== CONT TestGlobMatch/*bar_foo1959=== CONT TestGlobMatch/foo*_foo1960=== CONT TestGlobMatch/*/*_foo1961=== CONT TestGlobMatch/*bar_foobar1962--- PASS: TestGlobMatch (0.01s)1963 --- PASS: TestGlobMatch/foo_foo (0.00s)1964 --- PASS: TestGlobMatch/*/*_foo/bar (0.00s)1965 --- PASS: TestGlobMatch/*_ (0.00s)1966 --- PASS: TestGlobMatch/foo*_foobar (0.00s)1967 --- PASS: TestGlobMatch/*_anything (0.00s)1968 --- PASS: TestGlobMatch/fo?_fo (0.00s)1969 --- PASS: TestGlobMatch/*bar_bar (0.00s)1970 --- PASS: TestGlobMatch/foo*_bar (0.00s)1971 --- PASS: TestGlobMatch/foo_bar (0.00s)1972 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:otherorg/myrepo:ref:refs/heads/main (0.00s)1973 --- PASS: TestGlobMatch/repo:myorg/*:*_repo:myorg/myrepo:ref:refs/heads/main (0.00s)1974 --- PASS: TestGlobMatch/?oo_boo (0.00s)1975 --- PASS: TestGlobMatch/?oo_foo (0.00s)1976 --- PASS: TestGlobMatch/foo*bar_foobarbaz (0.00s)1977 --- PASS: TestGlobMatch/refs/heads/*_refs/tags/v1.0 (0.00s)1978 --- PASS: TestGlobMatch/foo*bar_foo123bar (0.00s)1979 --- PASS: TestGlobMatch/fo?_fooo (0.00s)1980 --- PASS: TestGlobMatch/fo?_foo (0.00s)1981 --- PASS: TestGlobMatch/foo*bar_foobar (0.00s)1982 --- PASS: TestGlobMatch/refs/*/main_refs/heads/main (0.00s)1983 --- PASS: TestGlobMatch/refs/heads/*_refs/heads/main (0.00s)1984 --- PASS: TestGlobMatch/*bar_foo (0.00s)1985 --- PASS: TestGlobMatch/foo*_foo (0.00s)1986 --- PASS: TestGlobMatch/*/*_foo (0.00s)1987 --- PASS: TestGlobMatch/*bar_foobar (0.00s)1988--- PASS: TestValidateToken_ValidToken (0.01s)1989--- PASS: TestValidateToken_NoMatchingProvider (0.01s)1990--- PASS: TestValidateToken_BoundSubjectMismatch (0.01s)1991--- PASS: TestValidateToken_WrongAudience (0.01s)1992--- PASS: TestValidateToken_BoundClaimsMismatch (0.01s)1993--- PASS: TestValidateToken_Expired (0.01s)1994--- PASS: TestValidateToken_MultipleProviders (0.01s)1995PASS1996Running hook tests...1997=== RUN TestSendPathsEmpty1998=== PAUSE TestSendPathsEmpty1999=== RUN TestQueueEnqueueAndFetch2000=== PAUSE TestQueueEnqueueAndFetch2001=== RUN TestQueueDeduplication2002=== PAUSE TestQueueDeduplication2003=== RUN TestQueueRemove2004=== PAUSE TestQueueRemove2005=== RUN TestQueueFetchBatchLimit2006=== PAUSE TestQueueFetchBatchLimit2007=== RUN TestQueueRetryMovesToBack2008=== PAUSE TestQueueRetryMovesToBack2009=== RUN TestQueueFetchRemoveLifecycle2010=== PAUSE TestQueueFetchRemoveLifecycle2011=== RUN TestQueueConcurrentWriters2012=== PAUSE TestQueueConcurrentWriters2013=== RUN TestQueueRemoveLargeClosure2014=== PAUSE TestQueueRemoveLargeClosure2015=== RUN TestServerClientIntegration2016=== PAUSE TestServerClientIntegration2017=== RUN TestServerQueueError2018=== PAUSE TestServerQueueError2019=== RUN TestGetListenerSocketActivation2020 server_test.go:210: === RUN TestGetListenerSocketActivation2021 --- PASS: TestGetListenerSocketActivation (0.00s)2022 PASS2023 2024--- PASS: TestGetListenerSocketActivation (0.01s)2025=== RUN TestDrainIsolatesPoisonPath2026=== PAUSE TestDrainIsolatesPoisonPath2027=== RUN TestRunNotBlockedByPoisonHead2028=== PAUSE TestRunNotBlockedByPoisonHead2029=== RUN TestDrainGivesUpWhenServerDown2030=== PAUSE TestDrainGivesUpWhenServerDown2031=== RUN TestFailedPathPrunedByLaterClosure2032=== PAUSE TestFailedPathPrunedByLaterClosure2033=== RUN TestWorkerUploadsAndRemoves2034=== PAUSE TestWorkerUploadsAndRemoves2035=== RUN TestWorkerSkipsGCdPaths2036=== PAUSE TestWorkerSkipsGCdPaths2037=== RUN TestWorkerPrunesClosureDeps2038=== PAUSE TestWorkerPrunesClosureDeps2039=== CONT TestSendPathsEmpty2040=== CONT TestQueueEnqueueAndFetch2041=== CONT TestQueueConcurrentWriters2042--- PASS: TestSendPathsEmpty (0.00s)2043=== CONT TestWorkerPrunesClosureDeps2044=== CONT TestWorkerSkipsGCdPaths2045=== CONT TestWorkerUploadsAndRemoves2046=== CONT TestFailedPathPrunedByLaterClosure2047=== CONT TestDrainGivesUpWhenServerDown2048=== CONT TestRunNotBlockedByPoisonHead2049=== CONT TestDrainIsolatesPoisonPath2050=== CONT TestServerQueueError2051=== CONT TestQueueRemoveLargeClosure2052=== CONT TestQueueRemove2053=== CONT TestQueueFetchBatchLimit2054=== CONT TestQueueDeduplication2055=== CONT TestServerClientIntegration2056=== CONT TestQueueFetchRemoveLifecycle20572026/08/27 09:40:26 ERROR Failed to queue paths error="permission denied" count=12058=== CONT TestQueueRetryMovesToBack2059--- PASS: TestServerQueueError (0.00s)2060--- PASS: TestServerClientIntegration (0.00s)20612026/08/27 09:40:26 INFO Uploading batch count=420622026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=420632026/08/27 09:40:26 INFO Upload queue status pending=220642026/08/27 09:40:26 INFO Uploading batch count=220652026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=220662026/08/27 09:40:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901299949/002/a2067--- PASS: TestQueueDeduplication (0.01s)20682026/08/27 09:40:26 INFO Uploading batch count=120692026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=120702026/08/27 09:40:26 INFO Uploading batch count=220712026/08/27 09:40:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainIsolatesPoisonPath728948948/002/bbb2072--- PASS: TestQueueRemove (0.01s)2073--- PASS: TestQueueFetchRemoveLifecycle (0.01s)2074--- PASS: TestQueueEnqueueAndFetch (0.02s)20752026/08/27 09:40:26 INFO Upload queue status pending=220762026/08/27 09:40:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901299949/002/b20772026/08/27 09:40:26 INFO Upload queue status pending=32078--- PASS: TestQueueFetchBatchLimit (0.02s)20792026/08/27 09:40:26 INFO Uploading batch count=12080--- PASS: TestQueueRetryMovesToBack (0.02s)20812026/08/27 09:40:26 WARN Store path no longer exists (garbage collected?), removing from queue path=/build/TestWorkerSkipsGCdPaths2553378532/002/nonexistent20822026/08/27 09:40:26 INFO Upload queue status pending=220832026/08/27 09:40:26 INFO Uploading batch count=120842026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=120852026/08/27 09:40:26 INFO Uploading batch count=120862026/08/27 09:40:26 INFO Uploading batch count=120872026/08/27 09:40:26 INFO Uploading batch count=220882026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=220892026/08/27 09:40:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901299949/002/c20902026/08/27 09:40:26 INFO Uploading batch count=120912026/08/27 09:40:26 INFO Uploading batch count=120922026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=120932026/08/27 09:40:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901299949/002/d20942026/08/27 09:40:26 INFO Uploading batch count=220952026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=220962026/08/27 09:40:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901299949/002/e20972026/08/27 09:40:26 INFO Uploading batch count=120982026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=120992026/08/27 09:40:26 ERROR Upload failed, will retry later error="upload failed" path=/build/TestDrainGivesUpWhenServerDown1901299949/002/f21002026/08/27 09:40:26 INFO Uploading batch count=12101--- PASS: TestFailedPathPrunedByLaterClosure (0.02s)21022026/08/27 09:40:26 ERROR Upload failed error="upload failed" count=121032026/08/27 09:40:26 ERROR Drain finished with paths left in queue remaining=1021042026/08/27 09:40:26 ERROR Drain finished with paths left in queue remaining=12105--- PASS: TestDrainIsolatesPoisonPath (0.02s)2106--- PASS: TestDrainGivesUpWhenServerDown (0.02s)2107--- PASS: TestWorkerUploadsAndRemoves (0.03s)2108--- PASS: TestWorkerSkipsGCdPaths (0.04s)2109--- PASS: TestWorkerPrunesClosureDeps (0.04s)2110--- PASS: TestQueueRemoveLargeClosure (0.14s)2111--- PASS: TestQueueConcurrentWriters (0.47s)21122026/08/27 09:40:27 INFO Uploading batch count=121132026/08/27 09:40:27 INFO Uploading batch count=121142026/08/27 09:40:27 INFO Uploading batch count=121152026/08/27 09:40:27 ERROR Upload failed error="upload failed" count=121162026/08/27 09:40:27 INFO Uploading batch count=121172026/08/27 09:40:27 ERROR Upload failed error="upload failed" count=121182026/08/27 09:40:27 INFO Uploading batch count=121192026/08/27 09:40:27 ERROR Upload failed error="upload failed" count=121202026/08/27 09:40:27 INFO Uploading batch count=121212026/08/27 09:40:27 ERROR Upload failed error="upload failed" count=121222026/08/27 09:40:27 ERROR Drain finished with paths left in queue remaining=12123--- PASS: TestRunNotBlockedByPoisonHead (1.04s)2124PASS