EV

evaluate-presets

Automated testing and validation of hat collection preset configurations via shell scripts.

Install

mkdir -p .claude/skills/evaluate-presets && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/4662" && unzip -o skill.zip -d .claude/skills/evaluate-presets && rm skill.zip

Installs to .claude/skills/evaluate-presets

Activation

This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.

Use when testing Ralph's hat collection presets, validating preset configurations, or auditing the preset library for bugs and UX issues.
137 chars✓ has a “when” trigger
Intermediate

Key capabilities

  • →Evaluate hat collection presets
  • →Validate preset configuration integrity
  • →Audit preset library for bugs
  • →Capture metrics on hat routing performance

How it works

It invokes shell scripts to run presets through Ralph, recording sessions and extracting metrics to verify correct hat routing and iteration behavior.

Inputs & outputs

You give it
Preset name and backend
You get back
Metrics report and session logs

When to use evaluate-presets

  • →Verifying preset configuration integrity
  • →Auditing the preset library for quality
  • →Testing routing logic after updates

About this skill

Evaluate presets

Two jobs: run the evaluation scripts against the builtin presets, and audit a preset's design. For launching, monitoring, and measuring a single loop, or comparing two presets on one task, follow skills/ralph-loop/references/headless-runs.md.

Run the evaluation scripts

./tools/evaluate-preset.sh <preset> [claude|kiro]   # one preset (default backend: claude)
./tools/evaluate-all-presets.sh claude              # every preset in its PRESETS list
  • The presets evaluated, and each one's test task, expected iterations, and timeout, live in tools/evaluate-all-presets.sh (PRESETS=) and tools/preset-test-tasks.yml. Read them there rather than from this file.
  • Runs take minutes to hours, so start them as background jobs and wait for the exit notification.
  • Each run writes .eval/logs/<preset>/<timestamp>/ (output.log, session.jsonl, metrics.json, environment.json, merged-config.yml). The suite writes .eval/results/<suite-id>/SUMMARY.md plus one JSON file per preset.
  • Exit codes: 0 means the completion promise was reached. 124 means it timed out. Anything else is a failure; read output.log.

Done when every preset in the run has an exit code and a metrics.json, and each failure or timeout has a cause written next to it.

Check hat routing

Each hat should run in its own iteration with fresh context: one hat, one published event, then stop.

Pattern in session.jsonl / output.logMeaning
iterations ≈ events publishedRouting works
iterations well below eventsSeveral hats' work done in one iteration
iterations well above eventsHats ending without publishing; default_publishes or recovery is firing
0 eventsEvents are not being read from the JSONL file
grep -c "ITERATION" .eval/logs/<preset>/latest/output.log
grep -c "bus.publish" .eval/logs/<preset>/latest/session.jsonl
jq -r '.ts' .eval/logs/<preset>/latest/session.jsonl   # identical stamps = same-iteration publishes

When routing is broken, look first at the prompt text in crates/ralph-core/src/hatless_ralph.rs: the Event Publishing Guide and the EVENT WRITING section tell a hat to publish one event and stop. Then check that the active hat's instructions and the ## PENDING EVENTS section reach the prompt.

Audit a preset's design

Apply every check in skills/ralph-hats/references/design.md to every hat in the preset. Built-in presets add two of their own:

  • Tests assert protocol, not wording. crates/ralph-cli/src/presets.rs checks events, triggers, paths, task keys, and commands, so a prompt rewrite doesn't break the tests.
  • Both copies stay in sync. Edit presets/, then run ./scripts/sync-embedded-files.sh, which mirrors to crates/ralph-cli/presets/ for include_str!.

Done when every hat has passed every check, or each miss is written down with its reason.

Fix what the evaluation finds

For each failure, create a code task (/code-task-generator, output to .ralph/tasks/preset-fixes/), implement it (/code-assist), then re-run ./tools/evaluate-preset.sh <preset>.

Prerequisites

  • Cargo, to build Ralph.
  • yq (optional), to load test tasks from YAML.

When not to use it

  • →When testing logic unrelated to Ralph's hat collection

Prerequisites

BashRalph environment

Limitations

  • →Requires background execution for long-running suites
  • →Depends on specific script output structures

How it compares

It automates the validation of complex agent routing logic, whereas manual testing is prone to missing iteration-based errors.

Compared to similar skills

evaluate-presets side by side with the closest alternatives in the catalog.

SkillInstallsUpdatedSafetyDifficulty
evaluate-presets (this skill)18moReviewIntermediate
webapp-testing3537moReviewIntermediate
ui-ux-expert-skill911yReviewAdvanced
skill-creator1287moReviewAdvanced

Try saying

Example prompts that trigger this skill in your AI assistant.

You might also like

webapp-testing

anthropics

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

353585

ui-ux-expert-skill

fercracix33

Technical workflow for implementing accessible React user interfaces with shadcn/ui, Tailwind CSS, and TanStack Query. Includes 6-phase process with mandatory Style Guide compliance, Context7 best practices consultation, Chrome DevTools validation, and WCAG 2.1 AA accessibility standards. Use after Test Agent, Implementer, and Supabase agents complete their work.

91244

skill-creator

anthropics

Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.

128200

python-testing-patterns

wshobson

Implement comprehensive testing strategies with pytest, fixtures, mocking, and test-driven development. Use when writing Python tests, setting up test suites, or implementing testing best practices.

77204

dependency-upgrade

wshobson

Manage major dependency version upgrades with compatibility analysis, staged rollout, and comprehensive testing. Use when upgrading framework versions, updating major dependencies, or managing breaking changes in libraries.

26240

playwright-mcp

sfc-gh-dflippo

Browser testing, web scraping, and UI validation using Playwright MCP. Use this skill when you need to test Streamlit apps, validate web interfaces, test responsive design, check accessibility, or automate browser interactions through MCP tools.

33197

Search skills

Search the agent skills registry