python3.14-tokenizers-0.23.2
checks.x86_64-linux.nixos-eve
· build #837
· raw
1tribuchet: building on jamie2structuredAttrs is enabled3Sourcing python-remove-tests-dir-hook4Sourcing python-catch-conflicts-hook.sh5Sourcing python-remove-bin-bytecode-hook.sh6Sourcing pypa-build-hook7Using pypaBuildPhase8Sourcing python-runtime-deps-check-hook9Using pythonRuntimeDepsCheckHook10Sourcing pypa-install-hook11Using pypaInstallPhase12Sourcing python-imports-check-hook.sh13Using pythonImportsCheckPhase14Sourcing python-metadata-check-hook.sh15Using pythonMetadataCheckPhase16Sourcing python-namespaces-hook17Sourcing python-catch-conflicts-hook.sh18Sourcing setuptools-rust-hook19Sourcing pytest-check-hook20Using pytestCheckPhase
unpackPhase
21unpacking source archive /nix/store/fcjinchg0pm7s6bv2r7hh7l24ifvqzw0-source22source root is source/bindings/python23Executing cargoSetupPostUnpackHook24Finished cargoSetupPostUnpackHook25setting SOURCE_DATE_EPOCH to timestamp 315619200 of file "source/bindings/python/tools/stub-gen/src/main.rs"patchPhase
26Executing cargoSetupPostPatchHook27Validating consistency between /build/source/bindings/python/Cargo.lock and /build/tokenizers-0.23.2-vendor/Cargo.lock28Finished cargoSetupPostPatchHookconfigurePhase
29no configure script, doing nothingbuildPhase
30Executing maturinBuildHook31maturinBuildHook flags: --jobs=24 --offline --target x86_64-unknown-linux-gnu --manylinux off --strip --out /build/source/bindings/python/dist --interpreter python3.14 --release32🍹 Building a mixed python/rust project33🐍 Found CPython 3.14 at /nix/store/xahxrjs30p3mw0a42bqg9ikgkikskw8m-python3-3.14.8/bin/python3.1434🔗 Found pyo3 bindings with abi3-py3.10 support35📡 Using build options features, bindings from pyproject.toml36 Compiling proc-macro2 v1.0.10737 Compiling unicode-ident v1.0.2438 Compiling quote v1.0.4739 Compiling libc v0.2.18940 Compiling target-lexicon v0.13.541 Compiling autocfg v1.5.142 Compiling serde_core v1.0.22943 Compiling find-msvc-tools v0.1.1144 Compiling shlex v2.0.145 Compiling memchr v2.8.346 Compiling crossbeam-utils v0.8.2247 Compiling ident_case v1.0.148 Compiling fnv v1.0.749 Compiling crossbeam-epoch v0.9.2050 Compiling serde v1.0.22951 Compiling strsim v0.11.152 Compiling cfg-if v1.0.453 Compiling zerocopy v0.8.5654 Compiling getrandom v0.3.455 Compiling once_cell v1.21.456 Compiling crossbeam-deque v0.8.757 Compiling pkg-config v0.3.3458 Compiling rustversion v1.0.2359 Compiling regex-syntax v0.8.1160 Compiling rayon-core v1.13.061 Compiling heck v0.5.062 Compiling rawpointer v0.2.163 Compiling portable-atomic v1.15.064 Compiling cc v1.4.465 Compiling zmij v1.0.2366 Compiling version_check v0.9.567 Compiling either v1.18.068 Compiling minimal-lexical v0.2.169 Compiling pin-project-lite v0.2.1770 Compiling num-traits v0.2.1971 Compiling matrixmultiply v0.3.1172 Compiling unicode-width v0.2.273 Compiling utf8parse v0.2.274 Compiling itoa v1.0.1875 Compiling ahash v0.8.1276 Compiling thiserror v2.0.2077 Compiling serde_json v1.0.15178 Compiling futures-core v0.3.3479 Compiling log v0.4.3480 Compiling aho-corasick v1.1.581 Compiling paste v1.0.1582 Compiling nom v7.1.383 Compiling anstyle-parse v1.0.084 Compiling itertools v0.14.085 Compiling unicode-segmentation v1.13.386 Compiling futures-task v0.3.3487 Compiling unit-prefix v0.5.288 Compiling pastey v0.2.389 Compiling anstyle v1.0.1490 Compiling macro_rules_attribute-proc_macro v0.2.391 Compiling pyo3-build-config v0.29.292 Compiling ryu v1.0.2393 Compiling jiff-core v0.1.094 Compiling base64 v0.13.195 Compiling is_terminal_polyfill v1.70.296 Compiling smallvec v1.16.097 Compiling syn v3.0.498 Compiling syn v2.0.11999 Compiling static_assertions v1.1.0100 Compiling colorchoice v1.0.5101 Compiling anstyle-query v1.1.5102 Compiling castaway v0.2.4103 Compiling bitflags v2.13.1104 Compiling slab v0.4.12105 Compiling anstream v1.0.0106 Compiling futures-channel v0.3.34107 Compiling unicode-normalization-alignments v0.1.12108 Compiling macro_rules_attribute v0.2.3109 Compiling rustc-hash v2.1.3110 Compiling daachorse v3.0.3111 Compiling unicode_categories v0.1.1112 Compiling onig_sys v69.9.3113 Compiling esaxx-rs v0.1.10114 Compiling errno v0.3.14115 Compiling console v0.16.4116 Compiling mio v1.2.3117 Compiling signal-hook-registry v1.4.8118 Compiling pyo3-ffi v0.29.2119 Compiling pyo3 v0.29.2120 Compiling num-complex v0.4.6121 Compiling num-integer v0.1.47122 Compiling rand_core v0.9.5123 Compiling rayon v1.12.0124 Compiling numpy v0.29.0125 Compiling regex-automata v0.4.18126 Compiling indicatif v0.18.6127 Compiling jiff v0.2.35128 Compiling ndarray v0.17.2129 Compiling ndarray v0.16.1130 Compiling serde_derive v1.0.229131 Compiling futures-macro v0.3.34132 Compiling tokio-macros v2.7.2133 Compiling thiserror-impl v2.0.20134 Compiling darling_core v0.20.11135 Compiling pyo3-macros-backend v0.29.2136 Compiling rayon-cond v0.4.0137 Compiling monostate-impl v0.1.18138 Compiling regex v1.13.1139 Compiling ppv-lite86 v0.2.21140 Compiling env_filter v2.0.0141 Compiling rand_chacha v0.9.0142 Compiling monostate v0.1.18143 Compiling env_logger v0.11.11144 Compiling rand v0.9.5145 Compiling futures-util v0.3.34146 Compiling tokio v1.53.1147 Compiling darling_macro v0.20.11148 Compiling pyo3-macros v0.29.2149 Compiling darling v0.20.11150 Compiling dary_heap v0.3.9151 Compiling compact_str v0.9.1152 Compiling spm_precompiled v0.1.4153 Compiling derive_builder_core v0.20.2154 Compiling derive_builder_macro v0.20.2155 Compiling onig v6.5.3156 Compiling derive_builder v0.20.2157 Compiling tokenizers v0.23.2 (/build/source/tokenizers)158 Compiling pyo3-async-runtimes v0.29.0159 Compiling tokenizers-python v0.23.2 (/build/source/bindings/python)160 Finished `release` profile [optimized] target(s) in 1m 03s161🚉 Overriding platform tag from _PYTHON_HOST_PLATFORM environment variable as linux_x86_64.162📦 Built wheel for abi3 Python ≥ 3.10 to /build/source/bindings/python/dist/tokenizers-0.23.2-cp310-abi3-linux_x86_64.whl163Finished maturinBuildHook164buildPhase completed in 1 minutes 4 secondspythonRuntimeDepsCheckHook
165Executing pythonRuntimeDepsCheck166Checking runtime dependencies for tokenizers-0.23.2-cp310-abi3-linux_x86_64.whl167Finished executing pythonRuntimeDepsCheckinstallPhase
168Executing pypaInstallPhase169Successfully installed tokenizers-0.23.2-cp310-abi3-linux_x86_64.whl170Finished executing pypaInstallPhasepythonOutputDistPhase
171Executing pythonOutputDistPhase172Finished executing pythonOutputDistPhasefixupPhase
173shrinking RPATHs of ELF executables and libraries in /nix/store/2vpij4jimb6vf4xb171wa48g7xpdh7k0-python3.14-tokenizers-0.23.2-dist174checking for references to /build/ in /nix/store/2vpij4jimb6vf4xb171wa48g7xpdh7k0-python3.14-tokenizers-0.23.2-dist...175patching script interpreter paths in /nix/store/2vpij4jimb6vf4xb171wa48g7xpdh7k0-python3.14-tokenizers-0.23.2-dist176shrinking RPATHs of ELF executables and libraries in /nix/store/1yb1bnrcjbcpc2jifs9id6csri1f0lfh-python3.14-tokenizers-0.23.2177shrinking /nix/store/1yb1bnrcjbcpc2jifs9id6csri1f0lfh-python3.14-tokenizers-0.23.2/lib/python3.14/site-packages/tokenizers/tokenizers.abi3.so178checking for references to /build/ in /nix/store/1yb1bnrcjbcpc2jifs9id6csri1f0lfh-python3.14-tokenizers-0.23.2...179patching script interpreter paths in /nix/store/1yb1bnrcjbcpc2jifs9id6csri1f0lfh-python3.14-tokenizers-0.23.2180stripping (with command strip and flags -S -p) in /nix/store/1yb1bnrcjbcpc2jifs9id6csri1f0lfh-python3.14-tokenizers-0.23.2/lib181Executing pythonRemoveTestsDir182Finished executing pythonRemoveTestsDirinstallCheckPhase
183no Makefile or custom installCheckPhase, doing nothingpythonImportsCheckPhase
184Executing pythonImportsCheckPhase185Check whether the following modules can be imported: tokenizerspythonMetadataCheckPhase
186Executing pythonMetadataCheckPhasepytestCheckPhase
187Executing pytestCheckPhase188pytest flags: -m pytest --ignore-glob=benches/test_tiktoken.py -k not\ \(TestTrainFromIterators\)\ and\ not\ \(TestAsyncTokenizer\)\ and\ not\ \(test_decode_skip_special_tokens\)\ and\ not\ \(test_decode_stream_fallback\)\ and\ not\ \(test_encode_special_tokens\)\ and\ not\ \(test_from_pretrained\)\ and\ not\ \(test_from_pretrained_revision\)\ and\ not\ \(test_splitting\)189============================= test session starts ==============================190platform linux -- Python 3.14.8, pytest-9.1.1, pluggy-1.6.0191rootdir: /build/source/bindings/python192configfile: pytest.ini (WARNING: ignoring pytest config in setup.cfg!)193plugins: anyio-4.15.1, asyncio-1.4.0194asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function195collected 205 items / 21 deselected / 1 skipped / 184 selected196197tests/bindings/test_decoders.py ......................... [ 13%]198tests/bindings/test_encoding.py ........... [ 19%]199tests/bindings/test_models.py ........ [ 23%]200tests/bindings/test_normalizers.py ...................... [ 35%]201tests/bindings/test_pre_tokenizers.py ........................ [ 48%]202tests/bindings/test_processors.py ............. [ 55%]203tests/bindings/test_tokenizer.py .............................. [ 72%]204tests/bindings/test_trainers.py .................... [ 83%]205tests/documentation/test_pipeline.py .. [ 84%]206tests/documentation/test_quicktour.py . [ 84%]207tests/implementations/test_base_tokenizer.py . [ 85%]208tests/implementations/test_bert_wordpiece.py ... [ 86%]209tests/implementations/test_byte_level_bpe.py ..... [ 89%]210tests/implementations/test_char_bpe.py ..... [ 92%]211tests/implementations/test_sentencepiece.py ..... [ 95%]212tests/test_freethreaded.py ...s [ 97%]213tests/test_serialization.py ....s [100%]214215=============================== warnings summary ===============================216tests/test_freethreaded.py:32217 /build/source/bindings/python/tests/test_freethreaded.py:32: PytestUnknownMarkWarning: Unknown pytest.mark.timeout - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html218 pytestmark = pytest.mark.timeout(60) # any of these hanging means a deadlock219220tests/bindings/test_encoding.py::TestEncoding::test_sequence_ids221 /nix/store/1p2fja7y7dy76d7jbkrc51wrlssdz6zf-python3.14-pytest-9.1.1/lib/python3.14/site-packages/_pytest/fixtures.py:1313: PytestRemovedIn10Warning: Class-scoped fixture defined as instance method is deprecated.222 Instance attributes set in this fixture will NOT be visible to test methods,223 as each test gets a new instance while the fixture runs only once per class.224 Use @classmethod decorator and set attributes on cls instead.225 See https://docs.pytest.org/en/stable/deprecations.html#class-scoped-fixture-as-instance-method226 fixturefunc = resolve_fixture_function(fixturedef, request)227228tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism229tests/bindings/test_tokenizer.py::TestTokenizer::test_multiprocessing_with_parallelism230tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism231tests/implementations/test_bert_wordpiece.py::TestBertWordPieceTokenizer::test_multiprocessing_with_parallelism232tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism233tests/implementations/test_byte_level_bpe.py::TestByteLevelBPE::test_multiprocessing_with_parallelism234tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism235tests/implementations/test_char_bpe.py::TestCharBPETokenizer::test_multiprocessing_with_parallelism236 /nix/store/xahxrjs30p3mw0a42bqg9ikgkikskw8m-python3-3.14.8/lib/python3.14/multiprocessing/popen_fork.py:76: DeprecationWarning: This process (pid=3165) is multi-threaded, use of fork() may lead to deadlocks in the child.237 self.pid = os.fork()238239-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html240========= 182 passed, 3 skipped, 21 deselected, 10 warnings in 36.15s ==========241Finished executing pytestCheckPhase242pytestCheckPhase completed in 37 seconds