homelab-investigator

Homelab Investigator

Investigate homelab infrastructure health by querying VictoriaMetrics (metrics) and VictoriaLogs (logs). Read-only analysis -- query, correlate, classify, recommend.

Principles

• Investigation-only: query and analyze, never modify infrastructure
• Evidence-based: every conclusion backed by query results
• Correlate across subsystems: node issues cause pod issues; network issues cause ingress errors; storage issues cause mount stalls
• Challenge your own conclusions: when you think you've found the root cause, ask "what else could cause these symptoms?" and look for evidence
• Human decides: present findings with confidence levels; the engineer makes the call

Services & Data

Data retention: VM 30 days, VL 7 days.

Status: !python3 .claude/skills/homelab-investigator/obs-query health 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); nodes=list(d['cpu_pct'].keys()); print(f'VM up -- {len(nodes)} nodes: {', '.join(nodes)}')" || echo "VictoriaMetrics not reachable" !curl -sf https://victorialogs.matthew-stratton.me/health >/dev/null 2>&1 && echo "VictoriaLogs up" || echo "VictoriaLogs not reachable"

References:

• Metrics catalog + query patterns: docs/appendix/victoriametrics-queries.md
• Query tool recipes: .claude/skills/homelab-investigator/query-recipes.md
• Known failure patterns: .claude/skills/homelab-investigator/known-patterns.md
• Stack architecture: docs/06-observability.md

Hardware Context

• 2 RPi 4B nodes (4-core ARM, 8GB RAM): k3-m1 (control-plane), k3-n1 (worker)
• MikroTik RB5009 router: 192.168.1.1
• Synology DS720+ NAS: 192.168.1.200 (NFS storage, iSCSI)

Query Tool

All queries go through .claude/skills/homelab-investigator/obs-query <command> [args...]. Outputs JSON lines (one JSON object per line).

Key Commands by Phase

See query-recipes.md for detailed usage and interpretation of each command.

Key Metrics

Investigation Workflow

Phase 0: Parse Input

Route by user request:

Phase 1: Health Snapshot

Run obs-query health. Scan for red flags:

If everything is green, say so. If red flags exist, proceed to Phase 2 with the most critical issue.

Phase 2: Identify Anomaly

Narrow to the specific subsystem:

• Node issue: node-health <node>, then cpu, memory, disk, temperature, network for detail
• Pod issue: pod-health <namespace>, pod-restarts <namespace>, resource-pressure <namespace>
• Storage issue: nas, nas-storage, disk <node>
• Network/ingress: ingress-status, ingress-errors, network
• Security: firewall-drops, modsecurity, ingress-errors
• Infrastructure: router, nas, deployments, node-conditions

Phase 3: Drill Down & Correlate

Cross-reference metrics with logs. Look for related symptoms:

• High CPU + pod restarts = possible OOM or crash loop
• High disk IO + processes_blocked = NFS mount stall or SD card degradation
• Ingress 5xx + deployment unhealthy = backend down
• High temperature + CPU spike = thermal throttling
• Node not Ready + multiple pod failures = node-level issue

Phase 4: Recommend

Summarize findings and propose remediation. Reference known-patterns.md for recognized failure signatures.

Output Format

## Investigation: [scope]

**Status**: [healthy | warning | critical]

### Findings
- [finding 1 with evidence]
- [finding 2 with evidence]

### Root Cause (if identified)
[Traced failure chain: symptom -> cause -> underlying issue]
**Confidence**: [high | medium | low]

### Recommendation
[Specific remediation steps]

Knowledge Capture

• Read known-patterns.md at the start of investigations.
• When you discover a stable failure pattern (observed across multiple incidents), append it to known-patterns.md.
• Don't capture session-specific findings -- that's conversation context.

Complementary Tools

• scripts/homelab-diagnose.py (via just diagnose): SSH-based diagnostics for when the observability stack is down. Gathers raw system state directly from nodes.
• Grafana dashboards: Visual investigation with historical trends. Use for time-correlated analysis.
• Both complement obs-query: diagnose.py works without the obs stack, Grafana provides visual context, obs-query provides programmatic querying for systematic investigation.