woo-ai-smoke

woo-ai-smoke — evaluation methodology

This skill evaluates the WooAIAssistant feature beyond surface smoke. It combines hard invariants (deterministic, must-hold) with a rubric scored by Claude across four dimensions (correctness, groundedness, tool appropriateness, recovery). Runs are stored append-only under runs/ so regressions over commits are detectable.

Delegation model (MANDATORY)

Main Claude never runs the pipeline itself. A single smoke run ingests ~70+ [smoke|...] lines plus thousands of xcodebuild log lines — that's a context firehose. Instead:

1. Main Claude parses $ARGUMENTS (suite/scenario/samples/mode) and picks the baseline to compare against.
2. Main Claude dispatches a single subagent via the Task tool with the self-contained prompt below. Use subagent_type: "general-purpose" so the subagent has full tool access (Bash, Read, Write, Edit, Grep, Glob).
3. The subagent does everything in the "Full execution checklist" — credential refresh, writing the Swift template, xcodebuild run, parsing, judging, JSONL write, cleanup.
4. The subagent returns only the final markdown report: the per-scenario table + the PASS/REGRESSION/FAIL/NEW/FLAKY summary + the JSONL run path + 1-2 lines per regression.
5. Main Claude relays that markdown verbatim to the user. Do not re-judge, re-parse, or re-print raw [smoke|...] lines in the main context.

Subagent prompt template

Fill in the placeholders (in ALL CAPS) before dispatching:

You are running the /woo-ai-smoke pipeline end-to-end. Follow the SKILL.md
at .claude/skills/woo-ai-smoke/SKILL.md as your reference for the Swift
template, parse protocol, hard invariants, rubric, JSONL format, and
reporting format. Everything below is your SCOPED task.

Inputs:
- mode: rest           # only "rest" is wired up; MCP support is deferred
- suite: SUITE          # "default" (24 scenarios × N samples) or ad-hoc "t1; t2"
- samples: N            # 1 for ad-hoc, 3 for default
- baseline: BASELINE    # path to baseline JSONL to compare against
- run_label: LABEL      # short tag for the stored run file, e.g. "post_prompt_revision"
- head_sha: SHA         # from `git rev-parse --short HEAD`
- branch: BRANCH        # from `git branch --show-current`

Pipeline (execute in this order, no skipping). Arm a `trap` cleanup at the start so a build crash never leaves the temp Swift file or log behind:

```bash
trap 'rm -f Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift /tmp/woo-ai-smoke.log /tmp/woo-ai-smoke-store.env' EXIT

1. Verify ~/.woo-ai-smoke/store.env exists with all five required keys (WOO_SITE_URL, WOO_SITE_ID, WOO_USERNAME, WOO_APP_PASSWORD, WOO_DOTCOM_ACCESS_TOKEN). On first run the file doesn't exist: scaffold it with placeholders per the Credentials section of SKILL.md, open it for editing, and stop with a message instructing the engineer to fill it in and re-run.
2. Load the scenario set from .claude/skills/woo-ai-smoke/baseline.json (or build ad-hoc from the SUITE arg).
3. Run the Scenario fixture preflight from SKILL.md for exactly the scenarios being executed. Inspect each scenario's fixtures block first, then infer obvious missing fixtures from the prompts/rubric. Use the WooCommerce REST API with the smoke credentials to verify fixtures exist and create/update only smoke-owned records when needed. If a required fixture cannot be created, stop before xcodebuild with a short fixture error report.
4. Write Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift from the template in SKILL.md, replacing SAMPLES_PLACEHOLDER with N and wiring each scenario's turns and derived autoDeclineWrites (default true when scenario.category == "write" or the scenario has any write tools in its hard invariants, false otherwise — unless the baseline scenario explicitly sets autoDeclineWrites on the turn). Mode "rest" uses the default WooAssistantHeadless tool source. The Swift template's specific API references (resolver typealias, Card.kind shape) may drift between trunk states — if the build fails on them, fix inline in the generated test file rather than the template.
5. Run xcodebuild with the command in SKILL.md's Running section. Tee full output to /tmp/woo-ai-smoke.log. You may run it in the background and poll the log, but you must wait for completion before parsing.
6. Parse every [smoke|...] line per SKILL.md Parse protocol.
7. Apply hard invariants deterministically. A hard-invariant failure is an automatic FAIL; do not rubric-score further.
8. For every remaining turn, judge yourself against the rubric in SKILL.md plus the scenario's rubric_notes from baseline.json. Score 0/1/2 per dim, write a one-sentence rationale.
9. Compute per-scenario means (over samples × turns) per dim.
10. Write the run to .claude/skills/woo-ai-smoke/runs/<ISO-timestamp>_SHA_LABEL.jsonl (one JSON record per turn per sample per mode, exactly as defined in SKILL.md Storage format).
11. Compare against BASELINE: classify each scenario PASS / REGRESSION / FAIL / NEW / FLAKY per the Outcome classification table.
12. Cleanup is automatic via the trap armed at step 0; verify the three artifacts are gone before returning.

Return ONLY this markdown (no tool logs, no chain-of-thought, no raw [smoke|...] lines). Main Claude will relay this verbatim:

Smoke result — MODE vs BASELINE

<the markdown table from SKILL.md Reporting section, one row per scenario>

PASS: X | REGRESSION: Y | FAIL: Z | NEW: W | FLAKY: V Run stored: .claude/skills/woo-ai-smoke/runs/<filename>.jsonl

<one or two lines per REGRESSION / FAIL with likely cause>

If the build fails or a hard harness error halts the run, return the short error + what you cleaned up, not a full log dump.

Keep the subagent dispatch in a single Task tool call. Never split the
pipeline into multiple subagent turns — the parse state has to stay
inside the subagent's context.

## How it works

1. **Load scenarios** — default suite (24 scenarios) from `baseline.json`, or ad-hoc via `scenario "turn1; turn2"`.
2. **Verify credentials** in `~/.woo-ai-smoke/store.env` (see "Credentials" below). Swift reads the dotenv directly each run.
3. **Preflight fixtures** for the selected scenarios. Verify/create smoke-owned products, orders, and customers through the WooCommerce REST API before running the model.
4. **Write `Modules/Tests/WooAIAssistantTests/SmokeRespondContractTests.swift`** using the template below. Scenarios get expanded into the `@Test(arguments:)` parametrised suite.
5. **Run the smoke via `xcodebuild`**, capture stdout.
6. **Parse each `[smoke|...]` line** into a turn record — prompt, tool names, tool arg snippets, tool results, assistant text, card kinds.
7. **Claude judges each turn** against the scenario's `rubric_notes` and the global rubric (details below). Fill in scores per dim.
8. **Apply hard invariants** (deterministic pass/fail).
9. **Write run** to `.claude/skills/woo-ai-smoke/runs/<ISO-timestamp>_<sha>.jsonl`.
10. **Compare to baseline** — flag REGRESSION when hard invariants fail or rubric mean drops below `rubric_pass_threshold`.
11. **Report** a markdown table + summary counts.
12. **Delete** the temp Swift file, `/tmp/woo-ai-smoke.log`, and the `/tmp/woo-ai-smoke-store.env` mirror (via the `trap` armed at the start of the run).

## Prerequisites

- Xcode + iOS simulator (the project's `bootstrap` skill covers this).
- A WooCommerce demo store with an admin **application password** (for the REST tool calls) and an authenticated iOS app session whose WPCOM OAuth bearer can be captured (for the woo-mobile-ai LLM calls).
- Required CLI tools (all macOS-default): `xcodebuild`, `xcrun simctl`, `open`.
- Store credentials in **`~/.woo-ai-smoke/store.env`** with `WOO_SITE_URL`, `WOO_SITE_ID`, `WOO_USERNAME`, `WOO_APP_PASSWORD`, and `WOO_DOTCOM_ACCESS_TOKEN`. On first run the skill scaffolds the file with placeholders and opens it for editing — see Credentials below.

The skill never commits credentials. Swift reads `~/.woo-ai-smoke/store.env` directly so nothing leaks to `/tmp`.

## Credentials

The engineer maintains `~/.woo-ai-smoke/store.env` (the source of truth, dotenv format). The skill stages a `/tmp/woo-ai-smoke-store.env` mirror at run-start because the iOS simulator process sandboxes `~` to its own container and can't read the host's home directly; the `trap` cleanup deletes the `/tmp` mirror at run-end. Swift reads from `/tmp/woo-ai-smoke-store.env`.

The harness sends LLM traffic through the wpcom `woo-mobile-ai` backend wrapper using a captured iOS-app WPCOM OAuth bearer (`WOO_DOTCOM_ACCESS_TOKEN`). For pre-merge testing the engineer can route locally via mitmproxy, `/etc/hosts`, or a temporary hardcoded URLSession in the harness (not committed); the committed code only ships production-URL routing because nginx on the wpcom sandbox vhost rejects requests whose `Host` header isn't `public-api.wordpress.com`. REST tool calls still hit the merchant store directly with the application password.

**First-run flow**: if `~/.woo-ai-smoke/store.env` doesn't exist, scaffold it with placeholders, open it for the engineer to fill in, then stop. The engineer saves the file and re-runs the skill.

```bash
ENV_FILE="$HOME/.woo-ai-smoke/store.env"
STAGED_ENV="/tmp/woo-ai-smoke-store.env"

# First run: scaffold the file with placeholders, open it for editing, stop.
if [ ! -f "$ENV_FILE" ]; then
  mkdir -p "$(dirname "$ENV_FILE")"
  cat > "$ENV_FILE" <<'TEMPLATE'
# Woo AI smoke credentials - fill these in, save, then re-run the smoke skill.
# WOO_SITE_ID is the WordPress.com blog id of the demo store. Find it in
# wp-admin/options-general.php?page=jetpack or via the Jetpack AI JWT mint.
WOO_SITE_URL=https://your-demo-store.example.com
WOO_SITE_ID=123456
WOO_USERNAME=your-admin-username
WOO_APP_PASSWORD=xxxx xxxx xxxx xxxx xxxx xxxx
# WPCOM OAuth bearer captured from an 

---

*Content truncated.*