bench

Benchmarking & Local Regression Testing

Benchmarks are the project's guard against silent performance regressions. Treat them like unit tests for speed: write a .bench.ts when you add or change a hot path, and run the suite to confirm a change didn't make things slower before you open a PR.

Invoke manually as /bench [tag|version] [report-only], or autonomously during issue development. For fixing a confirmed regression, hand off to the perf-fix skill (one hotspot, end-to-end) or the debug-perf skill (profiling reference).

Argument handling (slash-command use)

The text after /bench may contain:

• A git tag or bare version (grid-2.7.0, 2.7.0, grid-react-1.2.0) → current-vs-tag run. Resolve per Tag resolution, echo the resolved tag, then run bun .github/skills/bench/bench-vs-tag.ts --ref <tag>.
• Nothing → bun run bench (current-vs-previous-run).
• The literal report-only (alone or after the tag) → produce the report but skip the fix plan.

If a run fails, report the error verbatim and stop.

When to write a .bench.ts

Add or extend a benchmark as an integral part of issue development whenever you:

• Touch a hot path — render scheduler, virtualization, config merge, sort/filter, plugin hook dispatch, large-dataset row building, adapter reconciliation.
• Add a new feature or plugin that runs work per-row, per-frame, or per-keystroke.
• Fix a performance bug — add a bench that would have caught it, so it can't silently return.

Bench files live next to the code under test as *.bench.ts and are picked up by each project's bench target (vitest bench, include: ['src/**/*.bench.ts']). Keep each bench() case focused on one operation; avoid mixing setup cost into the measured body (use the bench harness's setup hooks only for immutable fixture construction). Mirror the style of existing benches under libs/*/src/**/*.bench.ts.

Stateful benchmark rules

• Do not treat Tinybench setup as a per-invocation reset. A stateful callback can become a cache hit or no-op after its first invocation. Restore or alternate the relevant state inside every timed invocation, and include that reset only when it is part of the operation being measured. If reset cost would contaminate the result, use prebuilt alternating fixtures.
• Prove that every invocation performs real work. Add a control case that exposes the expected algorithmic shape where practical, such as first/middle/last position updates.
• Use deterministic fixtures. Derive values and hit rates from indexes; never use Math.random() or node:crypto in fixture construction or timed callbacks. Current-vs-tag comparisons are meaningful only when both sides process the same data distribution.
• Keep unrelated work outside the callback. Prebuild rows, caches, and column definitions. If the operation under test requires invalidation, mutation, or cloning, include only the minimum reset that preserves a real operation on every invocation.

Two ways to run

bun run bench runs nx run-many --target=bench across the bench-enabled projects (grid, grid-react, grid-vue) and writes per-project JSON to the bench output directory.

The bench-vs-tag.ts orchestrator materialises the baseline ref in a sibling git worktree, runs vitest bench on both sides on this machine (same CPU / kernel / thermal state — the whole point of the methodology), merges max-of-N per side via tools/merge-bench-runs.ts, and compares via tools/compare-benches.ts. Artifacts land under tmp/ (bench-current-*.json, bench-baseline-*.json).

bench-vs-tag flags (most useful)

bun .github/skills/bench/bench-vs-tag.ts                    # default: last tag, all projects, 1 iter
bun .github/skills/bench/bench-vs-tag.ts --ref grid-2.7.0   # explicit tag/SHA baseline
bun .github/skills/bench/bench-vs-tag.ts --project grid     # subset (repeatable)
bun .github/skills/bench/bench-vs-tag.ts --iterations 3     # max-of-N per side (noisy suites)
bun .github/skills/bench/bench-vs-tag.ts --threshold 0.20   # tighter regression gate (default 0.30)
bun .github/skills/bench/bench-vs-tag.ts --keep-worktree    # don't delete ../base on exit
bun .github/skills/bench/bench-vs-tag.ts --fail-on-regression  # exit non-zero on regression (CI-style; off by default)
bun .github/skills/bench/bench-vs-tag.ts --help             # full flag list

Run with mode=sync and a generous timeout — bench runs are long. Do not pipe through tail/head/2>&1 (terminal hangs on this machine — see copilot-instructions.md).

Tag resolution

The user may pass a bare version like 2.7.0 instead of a fully-qualified tag. Resolve before invoking bench-vs-tag.ts:

1. If the input is already a tag (git rev-parse --verify "refs/tags/<input>"), use it verbatim.
2. If it looks like semver (^\d+\.\d+\.\d+), prepend the flagship prefix grid- and re-check (2.7.0 → grid-2.7.0).
3. Otherwise git tag --list "*<input>*" --sort=-creatordate. One match → use it. Multiple (e.g. 1.6.0 could be grid-vue-1.6.0 or grid-react-1.6.0) → stop and ask; do not guess.
4. Nothing resolves → surface the failure and candidate list; do not fall back silently.

Echo the resolved tag back before starting (e.g. "Resolved 2.7.0 → grid-2.7.0.").

Reading the results

Read the JSON / comparison-Markdown artifacts directly for precise numbers — do not parse console output. For each suite compare current vs baseline mean and flag any regression above the script's threshold (default 0.30 for vs-tag). When asked for a report, produce:

• Check comparability before diagnosing a flagged row. Diff the benchmark file and compare the Git blob hashes of the implementation and dependency lockfile against the baseline. If the implementation and lockfile are identical but fixture generation, lookup order, or measured work changed, classify the row as non-comparable benchmark drift rather than a code regression. Confirm with --iterations 3; do not optimize unchanged production code to compensate for fixture noise.

• Verdict — one line: Faster / Slower / Mixed vs the named baseline (grid-2.7.0 or previous local run @ <timestamp>), with the headline delta.

• Per-suite table — Suite | Baseline mean | Current mean | Δ | Regression?, sorted by |Δ%|.

• Hotspots — map each regressed suite to its *.bench.ts and the code under test; use git log --oneline <baseline>..HEAD -- <path> to surface the top 1–3 suspect commits. For grid-core, cross-reference .github/knowledge/grid-render-pipeline.md, .github/knowledge/grid-data-pipeline.md and .github/knowledge/data-flow-traces.md.

• Fix plan (skip under report-only) — per hotspot: Hypothesis (what's slower and why) · Investigation (minimal next step — debug-perf, microbench, bisect) · Candidate fix (concrete change, file + symbol) · Risk/scope (public API, plugin compat, bundle size). Order by impact-to-effort ratio.

Print the report inline (no Markdown file unless asked). End with the one-line verdict and the exact command to reproduce the run. If regressions exist, point to /perf-fix (executes one fix-plan item end-to-end: profile → patch → re-bench → verdict).

Constraints

• Never push, never commit. Benchmarking is read + analyze.
• Cite numbers; do not paraphrase deltas. Flag noisy-below-threshold suites explicitly.
• If a tag cannot be resolved, surface the error and stop — do not fall back silently.
• Do not "fix" the bench harness unless explicitly asked.
• Respect the bundle and perf invariants in .github/knowledge/build-and-deploy.md.