syngen-bootstrap

Syngen Bootstrap

Purpose

Guarantee that the local Python environment has every dependency the test suite and quality gates need, and that pytest can collect tests without the structural break this repo had before 2026-05-27.

Historical context (load-bearing)

• 2026-05-27 — pytest collection break fixed. src/tests/__init__.py was added so tests/ is importable as a package; pyproject.toml now sets [tool.pytest.ini_options] pythonpath = ["src"] and testpaths = ["src/tests"]. Before that fix, python -m pytest src/tests/unit produced 20 ModuleNotFoundError: No module named 'tests' errors.
• 2026-05-27 — literal Fernet keys removed from src/tests/conftest.py. FERNET_KEY and FERNET_KEY_2 are now generated per pytest session via Fernet.generate_key().decode() and assigned via os.environ.setdefault(...). CI can still inject specific values by setting the env vars before invoking pytest.

If either regression returns, the test suite will silently degrade — restore the fixes before further work.

Workflow

Step 0 — repo root

REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "${REPO_ROOT}"

Step 1 — locate or create the project venv

The harness prefers a .venv at the repo root for parity with the other tdm-* repositories.

VENV_DIR="${REPO_ROOT}/.venv"
PY="${VENV_DIR}/bin/python"

Branch A — venv exists. Verify the interpreter still works:

test -x "${PY}" && "${PY}" --version

If the version is not 3.10.x or 3.11.x (the supported range), the venv is stale — escalate per the Approval contract before recreating.

Branch B — venv missing. Creating a venv is a workspace-state change. Emit:

APPROVAL REQUIRED: create .venv at ${REPO_ROOT}/.venv via
    python3.11 -m venv "${REPO_ROOT}/.venv"
    "${REPO_ROOT}/.venv/bin/python" -m pip install --upgrade pip
    "${REPO_ROOT}/.venv/bin/python" -m pip install -r "${REPO_ROOT}/requirements.txt"
    "${REPO_ROOT}/.venv/bin/python" -m pip install -e "${REPO_ROOT}"
    "${REPO_ROOT}/.venv/bin/python" -m pip install build twine

Wait for approved per CLAUDE.md "Approval contract". After approval, run each command in order and report the result of each. Notes:

• Use python3.11 (Python 3.10 also works; the project supports both).
• The pip install --upgrade pip step is a benign upgrade of pip itself inside the new venv — this is the only --upgrade use the harness permits, and it is bounded to the freshly created venv. All other pip install --upgrade invocations remain denied by .claude/settings.json.
• The -e . step installs the package in editable mode so the train, infer, and syngen console scripts are created on the venv's PATH.
• build and twine are dev-only release tools; install them with the venv since the project does not list them in requirements.txt.

Branch C — user denies venv creation. Fall back to the system Python:

PY="$(command -v python)"
"${PY}" --version

Continue with steps 2-5; quality gates that require missing dev tools (mypy, build, twine) will be skipped with an explicit note in the handoff.

Step 2 — probe runtime imports

"${PY}" - <<'PY'
groups = [
    ("runtime",  ["syngen", "marshmallow", "pandas", "numpy", "pyarrow", "loguru", "yaml"]),
    ("ml",       ["tensorflow"]),
    ("regex",    ["exrex"]),
    ("fernet",   ["cryptography.fernet"]),
    ("logging",  ["reportportal_client"]),
    ("dev",      ["pytest", "flake8", "build", "twine"]),
]
missing = []
for label, mods in groups:
    for m in mods:
        try:
            __import__(m)
        except Exception as e:
            missing.append((label, m, f"{type(e).__name__}: {e}"))
            print(f"  FAIL {label:8s} {m:30s} {type(e).__name__}: {e}")
        else:
            print(f"  OK   {label:8s} {m}")
print()
print("missing:" if missing else "all imports OK")
for label, m, e in missing:
    print(f"  - [{label}] {m}: {e}")
PY

Each missing module that you did not just resolve via the venv creation in step 1 is an escalation. Emit:

APPROVAL REQUIRED: install missing module(s) <list> via
    "${PY}" -m pip install <list>

Step 2b — installed-vs-source version parity

This library is consumed downstream by tdm_syngen, whose tests frequently run against this source tree via PYTHONPATH=<syngen>/src rather than an install. An installed syngen wheel of a different version then silently shadows or mismatches the source you are editing, producing failures (or false passes) that have nothing to do with your change. Before trusting any test result, compare the installed version to this source tree's src/syngen/VERSION:

"${PY}" - <<'PY'
from importlib.metadata import version, PackageNotFoundError
from pathlib import Path
src = Path("src/syngen/VERSION").read_text().strip()
try:
    inst = version("syngen")
except PackageNotFoundError:
    inst = "(not installed)"
print(f"source={src}  installed={inst}")
print("MATCH" if inst == src else "DRIFT — prefer PYTHONPATH=<syngen>/src, or reinstall the matching wheel (escalation)")
PY

On DRIFT, do not pip install --upgrade (denied by .claude/settings.json and a dependency change requiring escalation). Either run against the source tree with PYTHONPATH set, or escalate to reinstall the exact matching wheel. Report the drift in your handoff.

Step 3 — verify pytest collection

"${PY}" -m pytest --collect-only -q src/tests/unit 2>&1 | tail -5

Expected: a <N> tests collected line with no collection errors.

• If you see ModuleNotFoundError: No module named 'tests', the structural fix has been reverted. Restore src/tests/__init__.py (empty file) and the [tool.pytest.ini_options] block in pyproject.toml. This is a [[harness change]] — coordinate via syngen-harness-update.
• If you see ModuleNotFoundError: No module named 'exrex' (or any other runtime module), go back to Step 2.

Step 4 — verify entry-point binaries and imports

ls -la "${VENV_DIR}/bin/train" "${VENV_DIR}/bin/infer" "${VENV_DIR}/bin/syngen" 2>/dev/null
"${PY}" -c "import syngen; print(syngen.__file__)"

If the console scripts are missing, the editable install in step 1 was skipped or failed — re-run "${PY}" -m pip install -e ..

Step 5 — confirm runtime cache directories are absent or empty

ls -d "${REPO_ROOT}/model_artifacts" "${REPO_ROOT}/tmp_store" 2>/dev/null | head

These should not exist on a fresh checkout. If they do, they are leftover from a prior run and should be cleaned up before committing. They are denied for read by the harness (./model_artifacts/**, ./tmp_store/** in settings.json).

When to invoke

• Fresh git clone.
• Any python -m pytest … reports ModuleNotFoundError.
• Before running a release wheel build, mypy, or pip-audit.
• After a change to requirements.txt, setup.cfg, pyproject.toml, or MANIFEST.in.
• After a Python interpreter upgrade.

Do not

• Do not pip install --upgrade <package> for any package other than pip itself inside a freshly created venv. All other --upgrade / --force-reinstall / -U invocations are denied by .claude/settings.json because they constitute dependency changes that must go through syngen-release-engineer.
• Do not edit requirements.txt to satisfy a missing import — escalate via the Approval contract.
• Do not commit the .venv directory.
• Do not restore the literal Fernet keys that used to live in src/tests/conftest.py. The current os.environ.setdefault(..., Fernet.generate_key().decode()) pattern is the correct one. If a test fails because it expects a specific encrypted payload, the test is wrong — fix the test, not the fixture.

Handoff

State:

• Which Python interpreter was used (venv path or system).
• Whether the venv was created or already existed.
• Which (if any) modules were installed and the exact commands run.
• Whether pytest --collect-only reported zero collection errors and the number of tests collected.
• Whether train / infer / syngen console scripts are on the venv PATH.