training-check
Automated monitoring for ongoing spatial experiments to detect failures and stalls early.
Install
mkdir -p .claude/skills/training-check && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/11067" && unzip -o skill.zip -d .claude/skills/training-check && rm skill.zipInstalls to .claude/skills/training-check
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Monitors running spatial experiments. Checks output files, log files, and process status. Categorizes results as OK, STALLED, FAILED, or COMPLETE. Fires alerts by appending to output/PROJ_NOTES.md. Run every 15 minutes during Stage 3 of research-pipeline.Key capabilities
- →Monitor spatial experiment process status
- →Detect stalled experiments via file modification timestamps
- →Identify errors in log files using keyword matching
- →Classify experiment outcomes as OK, STALLED, FAILED, or COMPLETE
- →Append alerts to project notes
How it works
The skill reads experiment logs and checks output file timestamps to determine process health. It then categorizes the state and triggers appropriate recovery or logging actions.
Inputs & outputs
When to use training-check
- →Monitoring experiment progress
- →Detecting NaN values in logs
- →Managing stalled processes
- →Updating research logs
About this skill
Skill: training-check
You monitor spatial experiment execution and detect problems early to avoid wasting compute time.
Phase 1: Check Active Experiments
Read output/EXPERIMENT_LOG.md for experiments with status RUNNING or PENDING.
For each running experiment:
- Check if output file was recently modified:
python -c "import os, time; f='[output_path]'; age=(time.time()-os.path.getmtime(f))/60; print(f'Modified {age:.1f} min ago')"
- Check last few lines of log file for errors:
tail -20 [log_path]
- Check for error keywords:
Error,Traceback,NaN,inf,MemoryError,Killed - Check for stall: no file modification in > 30 min despite RUNNING status
Phase 2: Classify Status
| Signal | Classification | Action |
|---|---|---|
| Output file updated in last 15 min, no errors | OK | Continue monitoring |
| Log contains "NaN" or diverging loss | CLEARLY BAD | Terminate, log failure |
| Output file not modified in > 30 min | STALLED | Investigate process |
| Results file written, contains valid metrics | COMPLETE | Update EXPERIMENT_LOG |
| MemoryError in log | MEMORY FAIL | Retry with smaller sample |
| Process not running, no output file | DEAD | Re-queue if < 2 retries |
Phase 3: Actions
COMPLETE: Update output/EXPERIMENT_LOG.md status to SUCCESS. Append finding to output/PROJ_NOTES.md.
CLEARLY BAD: Kill job if possible. Mark as FAILED in log. Try to fix the issue and retry once.
STALLED: Check if process is still alive. If dead: re-queue. If alive: wait 15 more min.
Phase 3.5: Human Checkpoint — Data Synthesis
Honor the HUMAN_CHECKPOINT flag in CLAUDE.md (default: true). This is a monitoring skill, but a few automatic recovery actions silently produce or substitute experiment data. When HUMAN_CHECKPOINT is true, PAUSE and request explicit user approval before any of the following; when false, log the action to output/PROJ_NOTES.md with rationale and proceed.
| Trigger | Show before pausing |
|---|---|
| MEMORY FAIL → retry with smaller sample: about to subsample, downsample, or otherwise reduce the dataset to make the run fit | Original N, proposed N, sampling rule (random / stratified / spatial), seed, and the claim affected by the reduced sample |
| CLEARLY BAD → fix and retry: about to modify the run (reduce learning rate, drop a feature, change loss, swap optimizer) and re-launch | Diagnosed cause, the exact code/config change, and confirmation that the new run is still valid evidence for the original claim |
| DEAD → re-queue: about to re-launch a run whose previous outputs were partial, with carry-over from a checkpoint or warm-start | Which artifacts will be reused, which will be regenerated, and the risk of mixing pre-/post-fix data in the same result file |
| COMPLETE → mark SUCCESS: about to write a SUCCESS row using metrics that were imputed / interpolated across missing log lines, or backfilled from a different run | Which numbers were imputed, source, and whether the run should be re-executed instead |
Do not mark a run SUCCESS in EXPERIMENT_LOG.md based on synthesized metrics. If the user approves, append Synthesis approved: <action> — <user reason> — <date> to the run's notes.
Phase 4: Progress Report
Output to stdout:
Training Check — <timestamp>
Active experiments: N
COMPLETE: N
OK (running): N
STALLED: N
FAILED: N
[List any failures or alerts]
Append alerts to output/PROJ_NOTES.md: [ALERT] Experiment <name> failed: <reason>
When not to use it
- →When the HUMAN_CHECKPOINT flag is true and user approval is pending
Limitations
- →Requires manual intervention when HUMAN_CHECKPOINT is enabled
- →Limited to experiments tracked in EXPERIMENT_LOG.md
How it compares
It automates the classification and alerting process that would otherwise require manual log inspection and status checking.
Compared to similar skills
training-check side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| training-check (this skill) | 0 | 3mo | Review | Intermediate |
| model-usage | 5 | 2mo | Review | Beginner |
| tracking-crypto-derivatives | 4 | 27d | Review | Intermediate |
| weights-and-biases | 3 | 7mo | Review | Intermediate |
Try saying
Example prompts that trigger this skill in your AI assistant.
You might also like
model-usage
openclaw
Use CodexBar CLI local cost usage to summarize per-model usage for Codex or Claude, including the current (most recent) model or a full model breakdown. Trigger when asked for model-level usage/cost data from codexbar, or when you need a scriptable per-model summary from codexbar cost JSON.
tracking-crypto-derivatives
jeremylongshore
Track cryptocurrency futures, options, and perpetual swaps with funding rates, open interest, liquidations, and comprehensive derivatives market analysis. Use when monitoring derivatives markets, analyzing funding rates, tracking open interest, finding liquidation levels, or researching options flow. Trigger with phrases like "funding rate", "open interest", "perpetual swap", "futures basis", "liquidation levels", "options flow", "put call ratio", "derivatives analysis", or "BTC perps".
weights-and-biases
davila7
Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform
tooluniverse-pharmacovigilance
mims-harvard
Analyze drug safety signals from FDA adverse event reports, label warnings, and pharmacogenomic data. Calculates disproportionality measures (PRR, ROR), identifies serious adverse events, assesses pharmacogenomic risk variants. Use when asked about drug safety, adverse events, post-market surveillance, or risk-benefit assessment.
analyzing-mempool
jeremylongshore
Monitor blockchain mempools for pending transactions, gas analysis, and MEV opportunities. Use when analyzing pending transactions, optimizing gas prices, or researching MEV. Trigger with phrases like "check mempool", "scan pending txs", "find MEV", "gas price analysis", or "pending swaps".
agent-session-monitor
alibaba
Real-time agent conversation monitoring - monitors Higress access logs, aggregates conversations by session, tracks token usage. Supports web interface for viewing complete conversation history and costs. Use when users ask about current session token consumption, conversation history, or cost statistics.