sre-templates

SRE Templates

Templates for SRE operational documents. All output goes to docs/sre/.

SLO/SLI Definition

Write to docs/sre/slos.md:

# Service Level Objectives

## Service: [Name]

### SLI: Availability
- **Definition:** Proportion of successful requests (HTTP 2xx/3xx) over total requests
- **Measurement:** `count(status < 500) / count(total)` over rolling 30-day window
- **SLO Target:** 99.9% (43.8 minutes/month error budget)
- **Data Source:** Load balancer access logs / Prometheus metrics

### SLI: Latency
- **Definition:** Proportion of requests served within threshold
- **Measurement:** `count(duration < 200ms) / count(total)` over rolling 30-day window
- **SLO Target:** 95% of requests < 200ms, 99% < 1000ms
- **Data Source:** Application metrics (histogram)

### SLI: Correctness
- **Definition:** Proportion of responses that return the expected result
- **Measurement:** Synthetic probe success rate
- **SLO Target:** 99.99%
- **Data Source:** Synthetic monitoring / canary checks

## Error Budget Policy

| Budget Remaining | Action |
|-----------------|--------|
| > 50% | Normal development velocity |
| 25-50% | Prioritize reliability work alongside features |
| 10-25% | Halt non-critical feature work; focus on reliability |
| < 10% | Freeze all changes except reliability fixes |

Alerting Rules

Write alert configurations to docs/sre/alerts.md or directly to monitoring config files:

# Alert: [Name]

**Severity:** critical / warning / info
**Condition:** [metric] [operator] [threshold] for [duration]
**Description:** What this alert means in plain language
**Impact:** What users experience when this fires
**Runbook:** docs/sre/runbooks/[name].md

## Examples

### High Error Rate
- **Condition:** `error_rate > 1%` for 5 minutes
- **Severity:** critical
- **Runbook:** docs/sre/runbooks/high-error-rate.md

### Elevated Latency
- **Condition:** `p99_latency > 2s` for 10 minutes
- **Severity:** warning
- **Runbook:** docs/sre/runbooks/elevated-latency.md

Runbook Format

Write runbooks to docs/sre/runbooks/<incident-type>.md:

# Runbook: [Incident Type]

## Overview
What this incident looks like and what typically causes it.

## Detection
How this incident is detected (alert name, dashboard, user report).

## Impact
What users experience during this incident.

## Severity Assessment
| Condition | Severity |
|-----------|----------|
| [condition] | SEV1 — critical |
| [condition] | SEV2 — major |
| [condition] | SEV3 — minor |

## Diagnosis Steps
1. Check [metric/dashboard] for [what to look for]
2. Run `[command]` to verify [condition]
3. Check [log source] for [pattern]

## Remediation Steps

### Immediate (Stop the Bleeding)
1. [Step with exact commands]
2. [Step with exact commands]

### Root Cause Fix
1. [Investigation steps]
2. [Fix steps]

## Escalation
- **If not resolved in 15 minutes:** Escalate to [team/person]
- **If customer-facing data loss:** Notify [stakeholder]

## Post-Incident
- [ ] Update incident timeline
- [ ] Create post-incident review document
- [ ] File follow-up tickets for permanent fixes

Capacity Plan

Write to docs/sre/capacity.md:

# Capacity Plan: [Service]

## Current Baseline
| Resource | Current Usage | Capacity | Utilization |
|----------|--------------|----------|-------------|
| CPU | ... | ... | ...% |
| Memory | ... | ... | ...% |
| Storage | ... | ... | ...% |
| Network | ... | ... | ...% |
| DB connections | ... | ... | ...% |

## Growth Projections
| Metric | Current | +3 months | +6 months | +12 months |
|--------|---------|-----------|-----------|------------|
| Requests/sec | ... | ... | ... | ... |
| Storage (GB) | ... | ... | ... | ... |
| Users | ... | ... | ... | ... |

## Scaling Thresholds
| Resource | Scale-Up Trigger | Scale-Down Trigger | Action |
|----------|-----------------|-------------------|--------|
| CPU | > 70% for 10min | < 30% for 30min | Add/remove instance |
| Memory | > 80% | < 40% for 30min | Add/remove instance |
| DB connections | > 80% pool | < 20% pool | Adjust pool size |

## Recommendations
[Specific actions to take based on projections]

Post-Incident Review

Write to docs/sre/post-incident/YYYY-MM-DD-<title>.md:

# Post-Incident Review: [Title]

**Date:** YYYY-MM-DD
**Duration:** [start time] — [end time] ([total duration])
**Severity:** SEV[1-4]
**Author:** [name]

## Summary
[1-2 sentence description of what happened]

## Impact
- **Users affected:** [number or percentage]
- **Duration of impact:** [time]
- **Data loss:** [yes/no, details]

## Timeline
| Time | Event |
|------|-------|
| HH:MM | [event] |
| HH:MM | [event] |

## Root Cause
[What actually broke and why]

## Contributing Factors
- [Factor 1]
- [Factor 2]

## What Went Well
- [Thing that helped]

## What Went Poorly
- [Thing that hurt]

## Action Items
| Action | Owner | Priority | Ticket |
|--------|-------|----------|--------|
| [action] | [owner] | P0/P1/P2 | [link] |

## Lessons Learned
[What we learned that applies beyond this incident]

Incident Severity Levels

Write to docs/sre/incident-process.md: