tribuchet: building on eliza Sourcing python-remove-tests-dir-hook Sourcing python-catch-conflicts-hook.sh Sourcing python-remove-bin-bytecode-hook.sh Sourcing pypa-build-hook Using pypaBuildPhase Sourcing python-runtime-deps-check-hook Using pythonRuntimeDepsCheckHook Sourcing pypa-install-hook Using pypaInstallPhase Sourcing python-imports-check-hook.sh Using pythonImportsCheckPhase Sourcing python-namespaces-hook Sourcing python-catch-conflicts-hook.sh Sourcing pytest-check-hook Using pytestCheckPhase Running phase: unpackPhase unpacking source archive /nix/store/z0k5i2qscw8lsdl7528sb1k01a0asxdq-nixbot source root is nixbot setting SOURCE_DATE_EPOCH to timestamp 315619200 of file "nixbot/pyproject.toml" Running phase: patchPhase Running phase: updateAutotoolsGnuConfigScriptsPhase Running phase: configurePhase no configure script, doing nothing Running phase: buildPhase Executing pypaBuildPhase Creating a wheel... pypa build flags: --no-isolation --outdir dist/ --wheel * Getting build dependencies for wheel... * Building wheel... Successfully built nixbot-0.0.1-py3-none-any.whl Finished creating a wheel... Finished executing pypaBuildPhase Running phase: pythonRuntimeDepsCheckHook Executing pythonRuntimeDepsCheck Checking runtime dependencies for nixbot-0.0.1-py3-none-any.whl Finished executing pythonRuntimeDepsCheck Running phase: installPhase Executing pypaInstallPhase Successfully installed nixbot-0.0.1-py3-none-any.whl Finished executing pypaInstallPhase Running phase: pythonOutputDistPhase Executing pythonOutputDistPhase Finished executing pythonOutputDistPhase Running phase: fixupPhase shrinking RPATHs of ELF executables and libraries in /nix/store/bsqgiravy19l2n0x49wvh5nm7qcxpjby-python3.14-nixbot-tests checking for references to /build/ in /nix/store/bsqgiravy19l2n0x49wvh5nm7qcxpjby-python3.14-nixbot-tests... patching script interpreter paths in /nix/store/bsqgiravy19l2n0x49wvh5nm7qcxpjby-python3.14-nixbot-tests stripping (with command strip and flags -S -p) in /nix/store/bsqgiravy19l2n0x49wvh5nm7qcxpjby-python3.14-nixbot-tests/lib /nix/store/bsqgiravy19l2n0x49wvh5nm7qcxpjby-python3.14-nixbot-tests/bin shrinking RPATHs of ELF executables and libraries in /nix/store/ix3c73i72lgwsbdjpn939dpi7gpx61v4-python3.14-nixbot-tests-dist checking for references to /build/ in /nix/store/ix3c73i72lgwsbdjpn939dpi7gpx61v4-python3.14-nixbot-tests-dist... patching script interpreter paths in /nix/store/ix3c73i72lgwsbdjpn939dpi7gpx61v4-python3.14-nixbot-tests-dist Rewriting #!/nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6/bin/python3.14 to #!/nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6 wrapping `/nix/store/bsqgiravy19l2n0x49wvh5nm7qcxpjby-python3.14-nixbot-tests/bin/nixbot'... Executing pythonRemoveTestsDir Finished executing pythonRemoveTestsDir Running phase: installCheckPhase no Makefile or custom installCheckPhase, doing nothing Running phase: pythonCatchConflictsPhase Running phase: pythonRemoveBinBytecodePhase Running phase: pythonImportsCheckPhase Executing pythonImportsCheckPhase Running phase: pytestCheckPhase Executing pytestCheckPhase pytest flags: -m pytest --numprocesses=24 /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) /nix/store/2xp5r99cnys85h806mpdw43bz48y05ds-python3.14-pytest-benchmark-5.2.3/lib/python3.14/site-packages/pytest_benchmark/logger.py:44: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment. warner(PytestBenchmarkWarning(text)) ============================= test session starts ============================== platform linux -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0 benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /build/nixbot configfile: pyproject.toml plugins: asyncio-1.4.0, timeout-2.4.0, xdist-3.8.0, benchmark-5.2.3, anyio-4.14.2 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function timeout: 60.0s timeout method: signal timeout func_only: False created: 24/24 workers 24 workers [845 items] ................s..................ssssssss............................. [ 8%] ........................................................................ [ 17%] ........................................................................ [ 25%] ........................................................................ [ 34%] ..................................................s..................... [ 42%] ..........F..................F.......................................... [ 51%] ........................................................................ [ 59%] .....................s..................................s............... [ 68%] .......................................................F................ [ 76%] ........................................................................ [ 85%] .....................................................F.................. [ 93%] ...............F...............................F..... [100%] =================================== FAILURES =================================== __________________________ test_prefetch_integration ___________________________ [gw6] linux -- Python 3.14.6 /nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6/bin/python3.14 tmp_path = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw6/test_prefetch_integration0') @needs_nix async def test_prefetch_integration(tmp_path: Path) -> None: """A locked git input is copied to the store and gc-rooted; after the input's source repo is gone, evaluation still succeeds purely from the store (the sandboxed eval has no credentials for private inputs).""" dep = init_upstream( tmp_path / "dep", {"flake.nix": '{ outputs = { self }: { greeting = "hello"; }; }'}, ) repo = init_upstream( tmp_path / "repo", { "flake.nix": f""" {{ inputs.dep.url = "git+file://{dep}"; outputs = {{ self, dep }}: {{ checks.x86_64-linux.ok = derivation {{ name = "ok-" + dep.greeting; system = "x86_64-linux"; builder = "/bin/sh"; args = [ "-c" "echo ok > $out" ]; }}; }}; }} """ }, ) _nix_flake_lock(repo) gcroots = tmp_path / "gcroots" await prefetch_flake_inputs(repo, BranchConfig(), gcroots) roots = list((gcroots / "inputs").iterdir()) assert len(roots) == 1 assert roots[0].is_symlink() assert roots[0].resolve().is_dir() # The eval must not need the input's origin anymore. shutil.rmtree(dep) if shutil.which("nix-eval-jobs") is None: pytest.skip("nix-eval-jobs not available") settings = EvalSettings( gc_roots_dir=tmp_path / "eval-gcroots", sandbox=False, systemd_scope=False ) > result = await EvalRunner().run(repo, BranchConfig(), settings) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nixbot/tests/test_flake_prefetch.py:147: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ nixbot/nix_eval.py:585: in run return await self._spawn( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = worktree_path = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw6/test_prefetch_integration0/repo') branch_config = BranchConfig(flake_dir='.', lock_file='flake.lock', attribute='checks', effects_on_pull_requests=False, effects_branches=[], build_branches=None, require_approval=True) settings = EvalSettings(gc_roots_dir=PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw6/test_prefetch_integration0/eval-gcroot...Path('/nix/var/nix/daemon-socket/socket'), extra_ro_paths=[], extra_args=[], eval_systems=[], legacy_attr_prefix=False) preexec_fn = None, on_jobs = None, on_stderr_line = None, eval_cgroup = None async def _spawn( # noqa: PLR0913 self, worktree_path: Path, branch_config: BranchConfig, settings: EvalSettings, preexec_fn: Callable[[], None] | None = None, on_jobs: JobBatchCallback | None = None, on_stderr_line: StderrLineCallback | None = None, eval_cgroup: Path | None = None, ) -> EvalResult: cmd = build_full_command(worktree_path, branch_config, settings) logger.info( "starting evaluation", extra={"worktree": str(worktree_path), "argv0": cmd[0]}, ) t0 = time.monotonic() proc = await asyncio.create_subprocess_exec( *cmd, cwd=worktree_path, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, preexec_fn=preexec_fn, # Own process group so a timeout can kill the whole tree. start_new_session=True, limit=STREAM_LIMIT, # Proxy/TLS/NIX_* must reach the evaluator: flake input # fetching fails in proxy/custom-CA deployments otherwise. env={ **passthrough_env(), "CLICOLOR_FORCE": "1", "PATH": os.environ.get("PATH", "/usr/bin:/bin"), # Fallback to /tmp when no `bwrap` "HOME": os.environ.get("HOME", "/tmp"), # noqa: S108 }, ) assert proc.stdout is not None # noqa: S101 assert proc.stderr is not None # noqa: S101 jobs: list[NixEvalJob] = [] parse_errors: list[str] = [] try: async with asyncio.timeout(settings.timeout): _, stderr_bytes = await asyncio.gather( _read_job_stream(proc.stdout, jobs, parse_errors, on_jobs), _read_stderr_tail(proc.stderr, on_line=on_stderr_line), ) returncode = await proc.wait() duration_ms = int((time.monotonic() - t0) * 1000) except BaseException as e: # Timeout, cancellation, or a reader bug: kill the evaluator # instead of leaking it (possibly blocked on a full pipe). _kill_process_tree(proc) await proc.wait() if isinstance(e, TimeoutError): msg = f"evaluation timed out after {settings.timeout:.0f} seconds" raise EvalError(msg) from None raise stderr_text = stderr_bytes.decode(errors="replace") if returncode != 0: tail = "\n".join(stderr_text.splitlines()[-50:]) # No stderr substring match: that would misclassify evals # whose trace merely mentions "out of memory". if returncode in OOM_RETURN_CODES or cgroup_oom_killed(eval_cgroup): msg = ( "evaluation ran out of memory (cgroup limit " f"{settings.memory_limit_mib} MiB, " f"{settings.worker_count} x {settings.max_memory_size_mib} " "MiB workers); this is a permanent failure, reduce " f"evaluation memory usage:\n{tail}" ) raise EvalOOMError(msg) msg = f"nix-eval-jobs failed with exit code {returncode}:\n{tail}" > raise EvalError(msg) E nixbot.nix_eval.EvalError: nix-eval-jobs failed with exit code 1: E warning: Git tree '/build/pytest-of-nixbld/pytest-0/popen-gw6/test_prefetch_integration0/repo' is dirty E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 1/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 2/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 224 ms (attempt 3/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 64 ms (attempt 4/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org E warning: Git tree '/build/pytest-of-nixbld/pytest-0/popen-gw6/test_prefetch_integration0/repo' is dirty E error: while starting worker, evaluation worker crashed with signal 7 (Bus error); enable coredumps for a backtrace nixbot/nix_eval.py:672: EvalError ----------------------------- Captured stderr call ----------------------------- warning: you don't have Internet access; disabling some network-dependent features warning: creating lock file "/build/pytest-of-nixbld/pytest-0/popen-gw6/test_prefetch_integration0/repo/flake.lock": • Added input 'dep': 'git+file:///build/pytest-of-nixbld/pytest-0/popen-gw6/test_prefetch_integration0/dep?ref=refs/heads/main&rev=b032898e336ba5b2904698936cd35f8810a3b4f8' (2026-09-23) ______________________________ test_full_pipeline ______________________________ [gw23] linux -- Python 3.14.6 /nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6/bin/python3.14 flake = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw23/flake0') tmp_path = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw23/test_full_pipeline0') async def test_full_pipeline(flake: Path, tmp_path: Path) -> None: > result, executor = await run_pipeline(flake, tmp_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nixbot/tests/test_pipeline.py:142: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ nixbot/tests/test_pipeline.py:135: in run_pipeline eval_result = await EvalRunner().run(flake, BranchConfig(), settings) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nixbot/nix_eval.py:585: in run return await self._spawn( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = worktree_path = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw23/flake0') branch_config = BranchConfig(flake_dir='.', lock_file='flake.lock', attribute='checks', effects_on_pull_requests=False, effects_branches=[], build_branches=None, require_approval=True) settings = EvalSettings(gc_roots_dir=PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw23/test_full_pipeline0/gcroots'), timeou...Path('/nix/var/nix/daemon-socket/socket'), extra_ro_paths=[], extra_args=[], eval_systems=[], legacy_attr_prefix=False) preexec_fn = None, on_jobs = None, on_stderr_line = None, eval_cgroup = None async def _spawn( # noqa: PLR0913 self, worktree_path: Path, branch_config: BranchConfig, settings: EvalSettings, preexec_fn: Callable[[], None] | None = None, on_jobs: JobBatchCallback | None = None, on_stderr_line: StderrLineCallback | None = None, eval_cgroup: Path | None = None, ) -> EvalResult: cmd = build_full_command(worktree_path, branch_config, settings) logger.info( "starting evaluation", extra={"worktree": str(worktree_path), "argv0": cmd[0]}, ) t0 = time.monotonic() proc = await asyncio.create_subprocess_exec( *cmd, cwd=worktree_path, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, preexec_fn=preexec_fn, # Own process group so a timeout can kill the whole tree. start_new_session=True, limit=STREAM_LIMIT, # Proxy/TLS/NIX_* must reach the evaluator: flake input # fetching fails in proxy/custom-CA deployments otherwise. env={ **passthrough_env(), "CLICOLOR_FORCE": "1", "PATH": os.environ.get("PATH", "/usr/bin:/bin"), # Fallback to /tmp when no `bwrap` "HOME": os.environ.get("HOME", "/tmp"), # noqa: S108 }, ) assert proc.stdout is not None # noqa: S101 assert proc.stderr is not None # noqa: S101 jobs: list[NixEvalJob] = [] parse_errors: list[str] = [] try: async with asyncio.timeout(settings.timeout): _, stderr_bytes = await asyncio.gather( _read_job_stream(proc.stdout, jobs, parse_errors, on_jobs), _read_stderr_tail(proc.stderr, on_line=on_stderr_line), ) returncode = await proc.wait() duration_ms = int((time.monotonic() - t0) * 1000) except BaseException as e: # Timeout, cancellation, or a reader bug: kill the evaluator # instead of leaking it (possibly blocked on a full pipe). _kill_process_tree(proc) await proc.wait() if isinstance(e, TimeoutError): msg = f"evaluation timed out after {settings.timeout:.0f} seconds" raise EvalError(msg) from None raise stderr_text = stderr_bytes.decode(errors="replace") if returncode != 0: tail = "\n".join(stderr_text.splitlines()[-50:]) # No stderr substring match: that would misclassify evals # whose trace merely mentions "out of memory". if returncode in OOM_RETURN_CODES or cgroup_oom_killed(eval_cgroup): msg = ( "evaluation ran out of memory (cgroup limit " f"{settings.memory_limit_mib} MiB, " f"{settings.worker_count} x {settings.max_memory_size_mib} " "MiB workers); this is a permanent failure, reduce " f"evaluation memory usage:\n{tail}" ) raise EvalOOMError(msg) msg = f"nix-eval-jobs failed with exit code {returncode}:\n{tail}" > raise EvalError(msg) E nixbot.nix_eval.EvalError: nix-eval-jobs failed with exit code 1: E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 1/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 2/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 360 ms (attempt 3/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 765 ms (attempt 4/5) E warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org E error: while evaluating 'checks.aarch64-linux.brokendep', evaluation worker crashed with signal 7 (Bus error); enable coredumps for a backtrace nixbot/nix_eval.py:672: EvalError ________________________ test_cached_skip_on_second_run ________________________ [gw23] linux -- Python 3.14.6 /nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6/bin/python3.14 flake = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw23/flake0') tmp_path = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw23/test_cached_skip_on_second_run0') async def test_cached_skip_on_second_run(flake: Path, tmp_path: Path) -> None: # First run built base/dependent (test ordering within module); # a second eval marks them local and the scheduler skips them. result, executor = await run_pipeline(flake, tmp_path) system = current_system() statuses = {r.attr: r.status for r in result.results} > assert statuses[f"checks.{system}.base"] == AttributeStatus.skipped_local E AssertionError: assert == E E - skipped_local E + succeeded nixbot/tests/test_pipeline.py:200: AssertionError __________________ test_inflight_share_fans_out_final_status ___________________ [gw1] linux -- Python 3.14.6 /nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6/bin/python3.14 make_env = .make at 0xffffed6d6140> upstream = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw1/test_inflight_share_fans_out_f0/upstream') async def test_inflight_share_fans_out_final_status( make_env: EnvFactory, upstream: Path ) -> None: sha = add_commit(upstream, "share") git(upstream, "branch", "copy", "main") executor = FakeExecutor(gate=asyncio.Event()) orchestrator, reporter, project = await make_env( FakeEvalRunner([mk_job("a")]), executor, "share", ) task = asyncio.create_task( orchestrator.handle_change_event( ChangeEvent(repo=project, branch="main", commit_sha=sha) ) ) await asyncio.wait_for(executor.started.wait(), timeout=10) build2 = await orchestrator.handle_change_event( ChangeEvent(repo=project, branch="copy", commit_sha=sha) ) assert executor.gate is not None executor.gate.set() build1 = await task assert build1 is not None assert build2 is not None assert build1.id_ == build2.id_ finished = [e for e in reporter.events if e[0] == "finished"] > assert len(finished) == 2 # one final status per context ^^^^^^^^^^^^^^^^^^^^^^^^^ E AssertionError: assert 1 == 2 E + where 1 = len([('finished', 27, 'succeeded', 1)]) nixbot/tests/test_orchestrator.py:1047: AssertionError _________________ test_linked_context_reported_on_eval_failure _________________ [gw1] linux -- Python 3.14.6 /nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6/bin/python3.14 make_env = .make at 0xffffed6d6350> upstream = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw1/test_linked_context_reported_o0/upstream') async def test_linked_context_reported_on_eval_failure( make_env: EnvFactory, upstream: Path ) -> None: sha = add_commit(upstream, "lef") git(upstream, "branch", "lef-copy", "main") eval_runner = FakeEvalRunner([], block=asyncio.Event(), error=EvalError("boom")) orchestrator, reporter, project = await make_env(eval_runner, FakeExecutor(), "lef") task = asyncio.create_task( orchestrator.handle_change_event( ChangeEvent(repo=project, branch="main", commit_sha=sha) ) ) await asyncio.wait_for(eval_runner.started.wait(), timeout=10) build2 = await orchestrator.handle_change_event( ChangeEvent(repo=project, branch="lef-copy", commit_sha=sha) ) assert eval_runner.block is not None eval_runner.block.set() build1 = await task assert build1 is not None assert build2 is not None assert build1.id_ == build2.id_ finished = [e for e in reporter.events if e[0] == "finished"] > assert len(finished) == 2 E AssertionError: assert 1 == 2 E + where 1 = len([('finished', 43, 'failed', 0)]) nixbot/tests/test_orchestrator.py:1505: AssertionError ------------------------------ Captured log call ------------------------------- WARNING nixbot.build_run:build_run.py:162 evaluation failed ____________ test_cancel_during_eval_resolves_linked_eval_contexts _____________ [gw1] linux -- Python 3.14.6 /nix/store/41m77i1296n33p6liin8ynr6wh3h6b7m-python3-3.14.6/bin/python3.14 make_env = .make at 0xffffed612980> upstream = PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw1/test_cancel_during_eval_resolv0/upstream') async def test_cancel_during_eval_resolves_linked_eval_contexts( make_env: EnvFactory, upstream: Path ) -> None: """Cancelling a build mid-eval must resolve the nix-eval context of every linked event, not only the primary one.""" add_commit(upstream, "linked-cancel") sha = git(upstream, "rev-parse", "HEAD") eval_runner = FakeEvalRunner([mk_job("a")], block=asyncio.Event()) orchestrator, reporter, project = await make_env( eval_runner, FakeExecutor(), name="linked-cancel", ) task1 = asyncio.create_task( orchestrator.handle_change_event( ChangeEvent(repo=project, branch="main", commit_sha=sha) ) ) await eval_runner.started.wait() build2 = await orchestrator.handle_change_event( ChangeEvent(repo=project, branch="main", commit_sha=sha, pr_number=4) ) assert build2 is not None orchestrator.cancel_events[build2.id_].set() await task1 cancelled_evals = [e for e in reporter.events if e[0] == "eval-cancelled"] assert len(cancelled_evals) == 2 finished = [e for e in reporter.events if e[0] == "finished"] > assert len(finished) == 2 E AssertionError: assert 1 == 2 E + where 1 = len([('finished', 69, 'cancelled', 0)]) nixbot/tests/test_orchestrator.py:2403: AssertionError =========================== short test summary info ============================ FAILED nixbot/tests/test_flake_prefetch.py::test_prefetch_integration - nixbo... FAILED nixbot/tests/test_pipeline.py::test_full_pipeline - nixbot.nix_eval.Ev... FAILED nixbot/tests/test_pipeline.py::test_cached_skip_on_second_run - Assert... FAILED nixbot/tests/test_orchestrator.py::test_inflight_share_fans_out_final_status FAILED nixbot/tests/test_orchestrator.py::test_linked_context_reported_on_eval_failure FAILED nixbot/tests/test_orchestrator.py::test_cancel_during_eval_resolves_linked_eval_contexts ================== 6 failed, 827 passed, 12 skipped in 31.83s ==================