evaluate-presets
Automated testing and validation of hat collection preset configurations via shell scripts.
Install
mkdir -p .claude/skills/evaluate-presets && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/4662" && unzip -o skill.zip -d .claude/skills/evaluate-presets && rm skill.zipInstalls to .claude/skills/evaluate-presets
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Use when testing Ralph's hat collection presets, validating preset configurations, or auditing the preset library for bugs and UX issues.Key capabilities
- →Evaluate hat collection presets
- →Validate preset configuration integrity
- →Audit preset library for bugs
- →Capture metrics on hat routing performance
How it works
It invokes shell scripts to run presets through Ralph, recording sessions and extracting metrics to verify correct hat routing and iteration behavior.
Inputs & outputs
When to use evaluate-presets
- →Verifying preset configuration integrity
- →Auditing the preset library for quality
- →Testing routing logic after updates
About this skill
Evaluate presets
Two jobs: run the evaluation scripts against the builtin presets, and audit a preset's design. For launching, monitoring, and measuring a single loop, or comparing two presets on one task, follow skills/ralph-loop/references/headless-runs.md.
Run the evaluation scripts
./tools/evaluate-preset.sh <preset> [claude|kiro] # one preset (default backend: claude)
./tools/evaluate-all-presets.sh claude # every preset in its PRESETS list
- The presets evaluated, and each one's test task, expected iterations, and timeout, live in
tools/evaluate-all-presets.sh(PRESETS=) andtools/preset-test-tasks.yml. Read them there rather than from this file. - Runs take minutes to hours, so start them as background jobs and wait for the exit notification.
- Each run writes
.eval/logs/<preset>/<timestamp>/(output.log,session.jsonl,metrics.json,environment.json,merged-config.yml). The suite writes.eval/results/<suite-id>/SUMMARY.mdplus one JSON file per preset. - Exit codes:
0means the completion promise was reached.124means it timed out. Anything else is a failure; readoutput.log.
Done when every preset in the run has an exit code and a metrics.json, and each failure or timeout has a cause written next to it.
Check hat routing
Each hat should run in its own iteration with fresh context: one hat, one published event, then stop.
Pattern in session.jsonl / output.log | Meaning |
|---|---|
| iterations ≈ events published | Routing works |
| iterations well below events | Several hats' work done in one iteration |
| iterations well above events | Hats ending without publishing; default_publishes or recovery is firing |
| 0 events | Events are not being read from the JSONL file |
grep -c "ITERATION" .eval/logs/<preset>/latest/output.log
grep -c "bus.publish" .eval/logs/<preset>/latest/session.jsonl
jq -r '.ts' .eval/logs/<preset>/latest/session.jsonl # identical stamps = same-iteration publishes
When routing is broken, look first at the prompt text in crates/ralph-core/src/hatless_ralph.rs: the Event Publishing Guide and the EVENT WRITING section tell a hat to publish one event and stop. Then check that the active hat's instructions and the ## PENDING EVENTS section reach the prompt.
Audit a preset's design
Apply every check in skills/ralph-hats/references/design.md to every hat in the preset. Built-in presets add two of their own:
- Tests assert protocol, not wording.
crates/ralph-cli/src/presets.rschecks events, triggers, paths, task keys, and commands, so a prompt rewrite doesn't break the tests. - Both copies stay in sync. Edit
presets/, then run./scripts/sync-embedded-files.sh, which mirrors tocrates/ralph-cli/presets/forinclude_str!.
Done when every hat has passed every check, or each miss is written down with its reason.
Fix what the evaluation finds
For each failure, create a code task (/code-task-generator, output to .ralph/tasks/preset-fixes/), implement it (/code-assist), then re-run ./tools/evaluate-preset.sh <preset>.
Prerequisites
- Cargo, to build Ralph.
yq(optional), to load test tasks from YAML.
When not to use it
- →When testing logic unrelated to Ralph's hat collection
Prerequisites
Limitations
- →Requires background execution for long-running suites
- →Depends on specific script output structures
How it compares
It automates the validation of complex agent routing logic, whereas manual testing is prone to missing iteration-based errors.
Compared to similar skills
evaluate-presets side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| evaluate-presets (this skill) | 1 | 8mo | Review | Intermediate |
| webapp-testing | 353 | 7mo | Review | Intermediate |
| ui-ux-expert-skill | 91 | 1y | Review | Advanced |
| skill-creator | 128 | 7mo | Review | Advanced |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by mikeyobrien
View all by mikeyobrien →You might also like
webapp-testing
anthropics
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
ui-ux-expert-skill
fercracix33
Technical workflow for implementing accessible React user interfaces with shadcn/ui, Tailwind CSS, and TanStack Query. Includes 6-phase process with mandatory Style Guide compliance, Context7 best practices consultation, Chrome DevTools validation, and WCAG 2.1 AA accessibility standards. Use after Test Agent, Implementer, and Supabase agents complete their work.
skill-creator
anthropics
Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.
python-testing-patterns
wshobson
Implement comprehensive testing strategies with pytest, fixtures, mocking, and test-driven development. Use when writing Python tests, setting up test suites, or implementing testing best practices.
dependency-upgrade
wshobson
Manage major dependency version upgrades with compatibility analysis, staged rollout, and comprehensive testing. Use when upgrading framework versions, updating major dependencies, or managing breaking changes in libraries.
playwright-mcp
sfc-gh-dflippo
Browser testing, web scraping, and UI validation using Playwright MCP. Use this skill when you need to test Streamlit apps, validate web interfaces, test responsive design, check accessibility, or automate browser interactions through MCP tools.