testing

Testing

Test Types

1. Python unit tests (ut-py) (tests/ut/): Standard pytest tests for the Python compilation pipeline and nanobind bindings. Run with pytest tests/ut. Tests declaring @pytest.mark.requires_hardware[("<platform>")] auto-skip unless --platform points to a matching device.
2. C++ unit tests (ut-cpp) (tests/ut/cpp/): GoogleTest-based tests for pure C++ modules. Run with cmake -B tests/ut/cpp/build -S tests/ut/cpp && cmake --build tests/ut/cpp/build && ctest --test-dir tests/ut/cpp/build -LE requires_hardware --output-on-failure. Hardware-required tests carry a requires_hardware or requires_hardware_<platform> ctest label and are filtered via -LE.
3. Scene tests (examples/{arch}/*/, tests/st/{arch}/*/): End-to-end @scene_test classes declared inside test_*.py. Sim variants run cross-platform (Linux/macOS); hardware variants require the CANN toolkit and an Ascend device. Discovery is by pytest (batch) or python test_*.py (standalone); #591's parallel orchestrator handles device bin-packing and ChipWorker reuse automatically.

Running Tests

Important: Always read .github/workflows/ci.yml first for the current --pto-session-timeout values. Quarantines are marker-based, so mirror the a2a3 sweep with -m "not sdma" rather than copying a path list. PTO-ISA reproducibility comes from the repo-root pto_isa.pin.

CI does not run one flat sweep on a2a3. Marked tests are quarantined out of the general onboard sweep and run in a step of their own, after it, because they are only correct in isolation. A5 runs the non-pod corpus, including SDMA tests, on both x86_64 and ARM64. Reproducing a2a3 CI means reproducing that shape — a bare pytest examples tests/st --platform a2a3 is not what CI runs and will report failures that CI never sees:

The a2a3 SDMA demos provision 48 device-only STARS streams, which makes an AICore fault take ~306 s to tear down instead of ~0.3 s — so they must not share a sweep with the aicore_op_timeout fault-injection test (investigations/2026-07-a2a3-sdma-fault-teardown.md, issue #1425).

When an onboard test fails, run it alone before calling it a regression. Alone-passes plus sweep-fails is an isolation requirement, not a defect: check the test for a quarantine marker such as @pytest.mark.sdma. Do not "fix" such a test by reordering it earlier — that only moves the pollution onto whatever now runs after it.

Runtime rebuild decision

Before running tests, determine whether runtime binaries need recompilation:

In CI, pip install . pre-builds all runtimes before tests run.

# Python unit tests (no hardware)
pytest tests/ut

# Python unit tests (a2a3 hardware)
pytest tests/ut --platform a2a3

# C++ unit tests (no hardware)
cmake -B tests/ut/cpp/build -S tests/ut/cpp && cmake --build tests/ut/cpp/build
ctest --test-dir tests/ut/cpp/build -LE requires_hardware --output-on-failure

# C++ unit tests (a2a3 hardware)
ctest --test-dir tests/ut/cpp/build -L "^requires_hardware(_a2a3)?$" --output-on-failure

# All simulation scene tests (extract --pto-session-timeout from ci.yml)
pytest examples tests/st --platform a2a3sim \
    --pto-session-timeout <timeout>

# All hardware scene tests — mirror ci.yml: deselect the quarantined marker, or
# those tests fail here and nowhere else
pytest examples tests/st -m "not sdma" --platform a2a3 --device <range> \
    --pto-session-timeout <timeout>

# The quarantined tests, the way CI runs them — same corpus, selected by the
# marker, run after the sweep rather than inside it. Never a path list: that is
# what the marker replaced.
pytest examples tests/st -m sdma \
    --platform a2a3 --device <2 devs> --pto-session-timeout <timeout>

# A5 runs the non-pod corpus, including SDMA tests, on both host architectures.
pytest examples tests/st -m "not pod" --platform a5 --device <range> \
    --pto-session-timeout <timeout>

# Single runtime
pytest examples tests/st --platform a2a3sim --runtime host_build_graph

# Single example (pytest, uses pre-built binaries)
pytest examples/a2a3/host_build_graph/vector_example --platform a2a3sim

# Single example (standalone; re-run `pip install --no-build-isolation -e .` first if runtime C++ changed)
python examples/a2a3/host_build_graph/vector_example/test_vector_example.py \
    -p a2a3sim

Pre-Commit Testing Strategy

When changed files require testing (C++, Python, or CMake), follow these steps to decide what to test and how.

Step 1 — Platform Availability and Detection

command -v npu-smi &>/dev/null

When npu-smi is found, detect the platform by parsing chip name from npu-smi info output:

Use the detected platform for all subsequent --platform flags. If the chip name is unrecognized, warn and default to a2a3.

Step 2 — Test Scope

Run git diff --name-only (or git diff --cached --name-only for staged changes) and match the first applicable rule:

Note on runtime C++ changes: When changed paths include src/{arch}/runtime/ or src/{arch}/platform/, re-run pip install --no-build-isolation -e . before testing to rebuild the runtime binaries in build/lib/ (incremental via build/cache/). There is no rebuild-on-import — editable.rebuild = false.

Step 3 — Parallel Strategy

Parallelism is handled by the #591 scheduler (simpler_setup/parallel_scheduler.py) based on --device and --max-parallel:

Simulation (a2a3sim): --max-parallel auto = min(nproc, len(--device)). Pass --device 0-15 for a big virtual pool; auto caps in-flight at the CPU count. Override with --max-parallel N on CPU-constrained runners.

Hardware (a2a3): --max-parallel auto = len(--device). One in-flight subprocess per physical device — each device runs a dedicated ChipWorker (see docs/ci.md).

Step 4 — Device Detection (hardware only)

When testing on a2a3, detect idle devices:

npu-smi info

Pick devices whose HBM-Usage is 0 and find the longest consecutive sub-range (at most 4). Pass as --device <start>-<end> (or --device <id> if only one idle device). If no idle device is found, skip hardware testing and warn.

Decision Tree

git diff --name-only
  │
  ├─ Only docs/config? ──→ SKIP tests
  │
  └─ Code changed?
       │
       ├─ Determine SCOPE (Step 2)
       │    ├─ platform   → full (pytest --platform ...)
       │    ├─ runtime    → single runtime (--runtime ...)
       │    └─ example    → single example (standalone test_*.py or pytest <ex>)
       │
       ├─ Runtime C++ changed (src/{arch}/)? ──→ pip install --no-build-isolation -e . first
       │
       └─ npu-smi found?
            ├─ Yes → sim + hardware (idle devs, max 4)
            └─ No  → sim only

Adding a New Scene Test

1. Create a directory under the appropriate arch and runtime:
   • Examples: examples/{arch}/<runtime>/<name>/
   • Device-only scene tests: tests/st/{arch}/<runtime>/<name>/
2. Add test_<name>.py with a @scene_test-decorated class (see docs/testing.md for the full template: CALLABLE, CASES, generate_args, compute_golden). End with if __name__ == "__main__": SceneTestCase.run_module(__name__) so the file runs standalone.
3. Add kernel source files under kernels/aic/, kernels/aiv/, and/or kernels/orchestration/ — referenced by CALLABLE["orchestration"]["source"] / CALLABLE["incores"][*]["source"] as paths relative to the test file.
4. Pytest auto-discovers any test_*.py under examples/ and tests/st/; no registration needed.

Related Skills

• git-commit — Complete commit workflow (runs testing as a prerequisite)