phoenix-evals
Tools for creating and executing evaluation suites for AI and LLM-based applications.
Install
mkdir -p .claude/skills/phoenix-evals && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/1054" && unzip -o skill.zip -d .claude/skills/phoenix-evals && rm skill.zipInstalls to .claude/skills/phoenix-evals
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Build and run evaluators for AI/LLM applications using Phoenix.Key capabilities
- →Build and run evaluation pipelines for LLM outputs
- →Compare model performance across different prompts
- →Automate quality checks for AI responses
- →Validate evaluator accuracy against human labels
- →Integrate evaluations into CI/CD test runners
How it works
The skill provides a framework to build deterministic code-based evaluators and LLM-based judges, allowing users to gate CI/CD pipelines based on quality signals.
Inputs & outputs
When to use phoenix-evals
- →Run eval suites on LLM outputs
- →Compare model performance across different prompts
- →Automate quality checks for AI responses
About this skill
Phoenix Evals
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Quick Reference
Workflows
Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
Gating CI: evaluators-{code|llm}-{python|typescript} → integrations-{pytest|vitest-jest} → production-continuous
Production: production-overview → production-guardrails → production-continuous
Reference Categories
| Prefix | Description |
|---|---|
fundamentals-* | Types, scores, anti-patterns |
observe-* | Tracing, sampling |
error-analysis-* | Finding failures |
axial-coding-* | Categorizing failures |
evaluators-* | Code, LLM, RAG evaluators |
experiments-* | Datasets, running experiments |
integrations-* | Run evals from test runners (pytest, Vitest, Jest) as a CI gate |
validation-* | Validating evaluator accuracy against human labels |
production-* | CI/CD, monitoring |
Key Principles
| Principle | Action |
|---|---|
| Error analysis first | Can't automate what you haven't observed |
| Custom > generic | Build from your failures |
| Code first | Deterministic before LLM |
| Validate judges | >80% TPR/TNR |
| Binary > Likert | Pass/fail, not 1-5 |
| Invariants gate, signals trend | assert/expect hard invariants (CI red); log LLM-judge quality signals and gate the aggregate (acceptance criteria), not every case |
When not to use it
- →Tasks not involving LLM output evaluation
Prerequisites
Limitations
- →Requires validation of judge models to ensure accuracy
- →Binary pass/fail is preferred over Likert scales
How it compares
It moves beyond manual review by providing a code-first framework to automate quality signals and track performance trends.
Compared to similar skills
phoenix-evals side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| phoenix-evals (this skill) | 3 | 29d | No flags | Advanced |
| agent-evaluation | 3 | 6mo | No flags | Advanced |
| mlops-validation | 2 | 6mo | No flags | Intermediate |
| evaluating-machine-learning-models | 1 | 27d | Review | Intermediate |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by Arize-ai
View all by Arize-ai →You might also like
agent-evaluation
davila7
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.
mlops-validation
fmind
Guide to implement rigorous validation layers including static analysis, automated testing, structured logging, and security scanning.
evaluating-machine-learning-models
jeremylongshore
Build this skill allows AI assistant to evaluate machine learning models using a comprehensive suite of metrics. it should be used when the user requests model performance analysis, validation, or testing. AI assistant can use this skill to assess model accuracy, p... Use when appropriate context detected. Trigger with relevant phrases based on skill purpose.
create-eval
HolmesGPT
This skill should be used when the user asks to "create an eval", "write an eval test", "add a new eval", "create a test case", "write a test for Holmes", or discusses LLM evaluation tests, eval fixtures, or test_case.yaml files for the HolmesGPT project.
openjudge
agentscope-ai
>
llm-evaluation
wshobson
Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.