run-full-suite

Purpose

run-full-suite scans every specs/*/spec-integration-tests/ directory for a run-all.sh and/or tc-*.sh files, runs each suite in sequence, and produces a single consolidated report across all specs. This is the project-wide integration test runner equivalent of dotnet test for the speckit shell test layer.

User Input

$ARGUMENTS

• Empty: run every discovered spec suite
• Spec filter (e.g. 004, token, catalog): run only suites whose parent folder name contains the filter (case-insensitive)
• --list: print all discovered suites with TC counts, do not run
• --fail-fast: stop after the first spec suite that has any FAIL result

Execution Steps

Step 1 — Discover all spec suites

From the repo root, find all feature scripts directories:

find specs -maxdepth 3 -name "run-all.sh" | sort

For each found run-all.sh, derive:

• SPEC_DIR — the parent specs/{NNN}-{name}/ folder
• SPEC_ID — the {NNN}-{name} folder name
• TC_COUNT — number of tc-*.sh files in the same spec-integration-tests/ directory
• HAS_CONFIG — whether config.env exists in the scripts directory

If no run-all.sh files are found anywhere under specs/, stop and tell the user: "No runnable spec suites found. Run /speckit-scripttest on one or more features first."

Apply the $ARGUMENTS filter: keep only suites whose SPEC_ID contains the filter string (case-insensitive). If the filter matches nothing, tell the user which suites exist and stop.

Step 2 — Handle --list

If $ARGUMENTS contains --list:

Print a discovery table and stop without running:

Discovered speckit test suites:

  #   Spec                           TC Scripts   config.env   Runnable
  ─── ────────────────────────────── ──────────── ──────────── ────────
  1   004-token-wallet-service        27           ✓            ✓
  2   005-token-adjusted-checkout      0           ✗            ✗ (no tc-*.sh yet)
  …

Run '/run-full-suite' to execute all runnable suites.

Step 3 — Check shared prerequisites

Before running any suite, verify tools are available:

command -v curl  || echo "WARN: curl not found — HTTP tests will fail across all suites"
command -v jq    || echo "WARN: jq not found — JSON assertions will fail across all suites"
command -v psql  || echo "WARN: psql not found — DB tests will fail across all suites"
command -v docker || echo "WARN: docker not found — detect-ports.sh (env var refresh) will not be able to derive PG_CONN"

Print a one-line warning for any missing tool. Do not block execution — scripts will report their own FAILs.

Step 4 — Run each suite

For each suite in discovery order (filtered if $ARGUMENTS specified):

1. Print a suite header:

   ════════════════════════════════════════════════════════
   Suite: 004-token-wallet-service  (27 scripts)
   ════════════════════════════════════════════════════════
   

2. Refresh env vars before running. If specs/{SPEC_ID}/spec-integration-tests/detect-ports.sh exists, run it with --update first so config.env reflects the current AppHost run's ports and connection strings (Aspire reassigns ports/passwords on every restart, so a stale config.env will cause spurious failures):

bash "specs/{SPEC_ID}/spec-integration-tests/detect-ports.sh" --update

If the script reports it cannot detect a running project (e.g. the AppHost isn't running), surface that warning to the user before continuing — the suite will likely fail without it.

3. Run the suite using `run_in_terminal`:
```bash
bash "specs/{SPEC_ID}/spec-integration-tests/run-all.sh"

4. Parse stdout for the TEST SUMMARY block — extract Passed, Failed, Skipped, Total.

5. Store results as {SPEC_ID, passed, failed, skipped, total, duration_seconds}.

6. If --fail-fast is set and failed > 0: print the failure details for this suite, then stop and print the partial cross-spec summary (Step 5) with a note that execution was halted early.

7. Continue to the next suite regardless of failures (unless --fail-fast).

Step 5 — Cross-spec summary report

After all suites have run, print the consolidated report:

════════════════════════════════════════════════════════════════════
  FULL SUITE REPORT — {ISO datetime}
════════════════════════════════════════════════════════════════════

  Spec                           PASS   FAIL   SKIP   TOTAL   Status
  ─────────────────────────────  ─────  ─────  ─────  ─────   ──────
  004-token-wallet-service        25     2      0      27      ✗ FAIL
  005-token-adjusted-checkout      —     —      —       —      ○ NO SCRIPTS
  …

  ─────────────────────────────────────────────────────────────────
  TOTALS                          {N}   {N}    {N}    {N}
  Overall status: {PASS / FAIL}
════════════════════════════════════════════════════════════════════

Status key:

• ✓ PASS — all TCs in this suite passed
• ✗ FAIL — one or more TCs failed (list them below the table)
• ○ NO SCRIPTS — directory exists but has no tc-*.sh yet
• ⚠ SKIPPED — suite was excluded by the filter argument

Failure detail block (printed below table if any failures):

Failures:
  004-token-wallet-service
    ✗ tc-022.sh — FAIL: net_balance expected=0 actual=80
    ✗ tc-043.sh — FAIL: HTTP 409 expected, got 200

Step 6 — Exit guidance

• If all pass: "All {N} test cases passed across {M} spec suites."
• If failures exist:
  • "Run /speckit-runtests to re-run a single spec's suite."
  • "Run /run-full-suite 004 to isolate a specific spec."
• If any suites show NO SCRIPTS: "Run /speckit-scripttest on those features to generate scripts."
• Exit code: surface the failed count — if any FAIL > 0, treat the overall run as failed (use exit 1 semantics when running in terminal).