agent-eval-api
Operates the Omniloy API to run, manage, and report on agent evaluation test runs.
Install
mkdir -p .claude/skills/agent-eval-api && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/11111" && unzip -o skill.zip -d .claude/skills/agent-eval-api && rm skill.zipInstalls to .claude/skills/agent-eval-api
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Operate the Omniloy Agent Evaluator (Agent Testing Platform) API end-to-end — authenticate, resolve or create agents / personas / evaluators / test configs, launch a test run, poll it to a terminal state, and read back transcripts, scores and pass/fail. Use this whenever the user wants to test or evaluate an AI agent, run an eval, create or reuse a persona / evaluator / test config, launch or cancel a test run, check evaluation results, or mentions "agent evaluator", "evals platform", "test run", "persona", "evaluator", or "omniloy evals". It is also the execution half that `/live-testing-plan` hands off to (it produces the spec, this skill runs it). Usage - "/agent-eval-api test the booking agent" or just describe the agent + criteria to evaluate.Key capabilities
- →Authenticate with the testing platform
- →Resolve or create test configurations
- →Launch and monitor test runs
- →Retrieve and report evaluation results
How it works
The agent interacts with the Omniloy REST API to manage the lifecycle of agent testing, from resource resolution to result retrieval and reporting.
Inputs & outputs
When to use agent-eval-api
- →Run agent evaluation tests
- →Create test configurations
- →Retrieve agent performance scores
- →Evaluate agent personas
About agent-eval-api
Manages the end-to-end lifecycle of agent testing. It handles authentication, resource resolution, test execution, and result retrieval for the Agent Evaluator platform.
Operate the Omniloy Agent Evaluator (Agent Testing Platform) API end-to-end — authenticate, resolve or create agents / personas / evaluators / test configs, launch a test run, poll it to a terminal state, and read back transcripts, scores and pass/fail. Use this whenever the user wants to test or ev
When not to use it
- →Directly modifying shared resources
- →Running more than two simultaneous tests
Prerequisites
Limitations
- →Max 2 simultaneous runs
- →Shared resources cannot be modified
How it compares
It replaces manual API orchestration and polling with an automated, step-by-step evaluation workflow.
Compared to similar skills
agent-eval-api side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| agent-eval-api (this skill) | 0 | 1mo | Review | Advanced |
| agent-evaluation | 3 | 6mo | No flags | Advanced |
| phoenix-evals | 3 | 27d | No flags | Advanced |
| mlops-validation | 2 | 6mo | No flags | Intermediate |
Try saying
Example prompts that trigger this skill in your AI assistant.
You might also like
agent-evaluation
davila7
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.
phoenix-evals
Arize-ai
Build and run evaluators for AI/LLM applications using Phoenix.
mlops-validation
fmind
Guide to implement rigorous validation layers including static analysis, automated testing, structured logging, and security scanning.
evaluating-machine-learning-models
jeremylongshore
Build this skill allows AI assistant to evaluate machine learning models using a comprehensive suite of metrics. it should be used when the user requests model performance analysis, validation, or testing. AI assistant can use this skill to assess model accuracy, p... Use when appropriate context detected. Trigger with relevant phrases based on skill purpose.
create-eval
HolmesGPT
This skill should be used when the user asks to "create an eval", "write an eval test", "add a new eval", "create a test case", "write a test for Holmes", or discusses LLM evaluation tests, eval fixtures, or test_case.yaml files for the HolmesGPT project.
evaluate
surus-lat
Run benchy evaluations against models or systems. Covers the canonical smoke→full workflow, config selection, task filtering, exit policies, and reading run_outcome.json. Use when asked to evaluate, benchmark, or run benchy against a model or system config.