woo-ai-smoke
Evaluates WooAIAssistant using automated scenarios and AI-driven rubric scoring.
Install
mkdir -p .claude/skills/woo-ai-smoke && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/19031" && unzip -o skill.zip -d .claude/skills/woo-ai-smoke && rm skill.zipInstalls to .claude/skills/woo-ai-smoke
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Evaluate WooAIAssistant against a structured scenario suite with hard invariants + LLM-as-judge rubric scoring. Runs live against the demo store + gpt-5.1 via the woo-mobile-ai backend wrapper, writes a JSONL run record, compares against stored baselines, and surfaces regressions. Always delegated to a subagent so the main context only sees the markdown report.Key capabilities
- →Evaluate WooAIAssistant against structured scenarios
- →Apply hard invariants for deterministic failures
- →Score responses using an LLM-as-judge rubric
- →Store run records in JSONL format
- →Compare runs against stored baselines
- →Surface regressions in assistant performance
How it works
The main Claude dispatches a subagent to run the evaluation pipeline, which involves setting up scenarios, running tests against the WooAIAssistant, applying hard invariants, and scoring responses with an LLM rubric.
Inputs & outputs
When to use woo-ai-smoke
- →Smoke-test AI assistant
- →Evaluate response quality
- →Detect regressions in assistant
About woo-ai-smoke
Runs evaluations against the demo store and evaluates responses using a rubric judged by an LLM. Writes JSONL run records and compares results against baselines to catch regressions.
Evaluate WooAIAssistant against a structured scenario suite with hard invariants + LLM-as-judge rubric scoring. Runs live against the demo store + gpt-5.1 via the woo-mobile-ai backend wrapper, writes a JSONL run record, compares against stored baselines, and surfaces regressions. Always delegated t
When not to use it
- →When the main Claude instance should run the pipeline directly
- →When re-judging or re-parsing raw smoke lines in the main context
- →When the subagent should not have full tool access
Limitations
- →Main Claude never runs the pipeline itself
- →Only 'rest' mode is currently wired up
- →Requires `~/.woo-ai-smoke/store.env` with five required keys
How it compares
This skill delegates the entire evaluation pipeline to a subagent, returning only a concise markdown report, which avoids context overload in the main session compared to running it directly.
Compared to similar skills
woo-ai-smoke side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| woo-ai-smoke (this skill) | 0 | 3mo | Caution | Advanced |
| agent-evaluation | 3 | 9mo | No flags | Advanced |
| phoenix-evals | 3 | 3mo | No flags | Advanced |
| mlops-validation | 2 | 9mo | No flags | Intermediate |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by woocommerce
View all by woocommerce →You might also like
agent-evaluation
davila7
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.
phoenix-evals
Arize-ai
Build and run evaluators for AI/LLM applications using Phoenix.
mlops-validation
fmind
Guide to implement rigorous validation layers including static analysis, automated testing, structured logging, and security scanning.
evaluating-machine-learning-models
jeremylongshore
Build this skill allows AI assistant to evaluate machine learning models using a comprehensive suite of metrics. it should be used when the user requests model performance analysis, validation, or testing. AI assistant can use this skill to assess model accuracy, p... Use when appropriate context detected. Trigger with relevant phrases based on skill purpose.
create-eval
HolmesGPT
This skill should be used when the user asks to "create an eval", "write an eval test", "add a new eval", "create a test case", "write a test for Holmes", or discusses LLM evaluation tests, eval fixtures, or test_case.yaml files for the HolmesGPT project.
evaluate
surus-lat
Run benchy evaluations against models or systems. Covers the canonical smoke→full workflow, config selection, task filtering, exit policies, and reading run_outcome.json. Use when asked to evaluate, benchmark, or run benchy against a model or system config.