model-change
Guidelines for modifying model logic, updating metrics, and validating changes with backtests.
Install
mkdir -p .claude/skills/model-change && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/14228" && unzip -o skill.zip -d .claude/skills/model-change && rm skill.zipInstalls to .claude/skills/model-change
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Use for prediction logic, feature usage, backtests, or metrics changes.Key capabilities
- →Update prediction logic in models
- →Modify feature usage for models
- →Run backtests for new features
- →Change performance metrics for models
How it works
The skill guides changes to model components, emphasizing adherence to a read order for relevant documentation and strict validation through tests and backtests.
Inputs & outputs
When to use model-change
- →Update model prediction logic
- →Run a backtest for a new feature
- →Modify performance metrics
- →Validate changes to model inputs
About this skill
Model Change
When To Use
Use for prediction logic, feature usage, backtests, or metrics changes.
Read Order
.agent/CONTEXT.mddocs/reference/models/index.mddocs/reference/specifications/transformations.mdscripts/generate_predictions.pyscripts/run_backtest.py
Rules
- Preserve the
reference_datecutoff. - Keep changes band-agnostic unless the issue is truly source-specific.
- Pair logic changes with tests or a narrow reproducible validation path.
Expected Validation
uv run pytest tests/test_models.py
uv run python scripts/run_backtest.py --band goose --model notebook --shows 10
Common Mistakes
- Leaking future data into features
- Changing metrics without updating docs
- Forgetting to validate downstream scripts
When not to use it
- →When the change is not related to prediction logic, feature usage, backtests, or metrics
- →When the issue is not truly source-specific and requires band-agnostic changes
- →When there is no intention to validate changes with tests or a reproducible path
Limitations
- →Changes must preserve the `reference_date` cutoff.
- →Changes should be band-agnostic unless the issue is source-specific.
- →Logic changes must be paired with tests or a narrow reproducible validation path.
How it compares
This skill provides a structured process for model changes, including specific validation steps, which differs from an ad-hoc approach to modifying model components.
Compared to similar skills
model-change side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| model-change (this skill) | 0 | 6mo | Review | Intermediate |
| llm-evaluation | 0 | 5mo | No flags | Intermediate |
| Plate Evaluation | 0 | 7mo | Review | Intermediate |
| llm-evaluation | 6 | 4mo | No flags | Advanced |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by connorkitchings
View all by connorkitchings →You might also like
llm-evaluation
H4D3ZS
Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
Plate Evaluation
ShArAvaNPai
Runs current model against validation set and returns JSON metrics with automated recommendations
llm-evaluation
wshobson
Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
evaluating-llms-harness
davila7
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
evaluating-machine-learning-models
jeremylongshore
Build this skill allows AI assistant to evaluate machine learning models using a comprehensive suite of metrics. it should be used when the user requests model performance analysis, validation, or testing. AI assistant can use this skill to assess model accuracy, p... Use when appropriate context detected. Trigger with relevant phrases based on skill purpose.
openjudge
agentscope-ai
>