k8s-incident
Diagnose and respond to Kubernetes outages using standardized runbooks and inspection tools.
Install
mkdir -p .claude/skills/k8s-incident && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/7137" && unzip -o skill.zip -d .claude/skills/k8s-incident && rm skill.zipInstalls to .claude/skills/k8s-incident
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Respond to Kubernetes incidents with runbooks and diagnostics. Use for outages, pod failures, node issues, network problems, and emergency response.Key capabilities
- →Check control plane health
- →Assess node status
- →Gather event logs
- →Perform emergency pod deletion
- →Rollback deployments and Helm releases
How it works
The skill provides a structured triage framework using specific Kubernetes diagnostic tools to identify and resolve service outages or pod failures.
Inputs & outputs
When to use k8s-incident
- →Debugging pod crash loops
- →Assessing cluster-wide outages
- →Checking control plane component health
- →Gathering logs for failed deployments
About this skill
Kubernetes Incident Response
Runbooks and diagnostic workflows for common Kubernetes incidents.
Requires the
cilium,diagnostics,istio,securitytoolsets: start kubectl-mcp-server with--toolsets default,cilium,diagnostics,istio,security(orMCP_TOOLSETS=default,cilium,diagnostics,istio,security).
When to Apply
Use this skill when:
- User mentions: "incident", "outage", "emergency", "down", "not working"
- Operations: emergency response, production issues, service degradation
- Keywords: "urgent", "broken", "fix", "restore", "recover"
Priority Rules
| Priority | Rule | Impact | Tools |
|---|---|---|---|
| 1 | Check control plane first | CRITICAL | kubectl_get(resource_type="pods", namespace="kube-system") |
| 2 | Assess node health | CRITICAL | kubectl_get |
| 3 | Gather events before changes | HIGH | kubectl_get |
| 4 | Document timeline | HIGH | Manual notes |
| 5 | Rollback if safe | MEDIUM | kubectl_rollout |
Quick Reference
| Incident | First Tool | Next Steps |
|---|---|---|
| Pod failure | kubectl_logs(pod_name, previous=True) | kubectl_describe, kubectl_get |
| Node down | kubectl_describe | Check kubelet logs |
| Service unreachable | kubectl_get | analyze_network_policies |
| Control plane | kubectl_get(resource_type="pods", namespace="kube-system") | Check API server logs |
Incident Triage
Quick Health Check
kubectl_get(resource_type="nodes")
kubectl_get(resource_type="pods", namespace="kube-system")
kubectl_get(resource_type="events", namespace)
Severity Assessment
| Indicator | Severity | Action |
|---|---|---|
| Multiple nodes NotReady | Critical | Escalate immediately |
| kube-system pods failing | Critical | Control plane issue |
| Single pod CrashLoop | Medium | Debug pod |
| High latency | Medium | Check resources |
Runbook: Pod Failures
CrashLoopBackOff
kubectl_logs(name, namespace, previous=True)
kubectl_describe(name, namespace)
kubectl_get(resource_type="events", namespace, field_selector="involvedObject.name=<pod>")
kubectl_top(name, namespace)
Common Causes:
- OOMKilled → Increase memory limits
- Exit code 1 → Application error in logs
- Exit code 137 → Killed by OOM or SIGKILL
- Exit code 143 → Graceful SIGTERM
ImagePullBackOff
kubectl_describe(name, namespace)
kubectl_get(resource_type="secrets", namespace)
Pending Pod
kubectl_describe(name, namespace)
kubectl_get(resource_type="nodes")
kubectl_get(resource_type="events", namespace)
Runbook: Node Issues
Node NotReady
kubectl_describe(name)
kubectl_get(resource_type="events", namespace="", field_selector="involvedObject.name=<node>")
node_logs(name, "kubelet")
Node DiskPressure
kubectl_describe(name)
kubectl_get(resource_type="pods", field_selector="spec.nodeName=<node>")
Runbook: Network Issues
Service Not Accessible
kubectl_get(resource_type="services", namespace)
kubectl_get(resource_type="endpoints", namespace)
kubectl_get(resource_type="pods", namespace, label_selector="<service-selector>")
analyze_network_policies(namespace)
DNS Resolution Failures
kubectl_get(resource_type="pods", namespace="kube-system", label_selector="k8s-app=kube-dns")
kubectl_logs("coredns-xxx", "kube-system")
With Cilium
cilium_get_status()
cilium_list_endpoints(namespace)
get_hubble_flows(namespace)
With Istio
istio_analyze(namespace)
istio_proxy_status()
Runbook: Storage Issues
PVC Pending
kubectl_describe(name, namespace)
kubectl_get(resource_type="storageclasses")
kubectl_get(resource_type="events", namespace)
Pod Stuck in ContainerCreating
kubectl_describe(name, namespace)
kubectl_get(resource_type="pvc", namespace)
kubectl_get(resource_type="events", namespace)
Runbook: Control Plane Issues
API Server Unavailable
kubectl_get(resource_type="pods", namespace="kube-system", label_selector="component=kube-apiserver")
kubectl_get(resource_type="events", namespace="kube-system")
etcd Issues
kubectl_get(resource_type="pods", namespace="kube-system", label_selector="component=etcd")
kubectl_logs("etcd-xxx", "kube-system")
Emergency Actions
Delete a Stuck Pod
delete_resource(resource_type="pod", name=name, namespace=namespace)
Rollback Deployment
kubectl_rollout(action="undo", resource_type="deployment", name=name, namespace=namespace)
Helm Rollback
helm_rollback(name, namespace, revision=1)
Diagnostic Collection Script
For comprehensive incident diagnostics, see scripts/collect-diagnostics.py.
Multi-Cluster Incident Response
Check all clusters:
for context in ["prod-1", "prod-2", "staging"]:
kubectl_get(resource_type="nodes", context=context)
kubectl_get(resource_type="pods", namespace="kube-system", context=context)
kubectl_get(resource_type="events", namespace="kube-system", context=context)
Post-Incident
Document Timeline
- When did the incident start?
- What was the impact?
- What was the root cause?
- What fixed it?
Prevent Recurrence
- Add monitoring/alerting
- Improve resource limits
- Add readiness probes
- Document runbook
Related Skills
- k8s-troubleshoot - Detailed debugging
- k8s-security - Security incidents
When not to use it
- →Non-Kubernetes infrastructure issues
- →Security-specific incident response
Limitations
- →Requires manual notes for timeline documentation
- →Does not provide automated root cause analysis
How it compares
It automates the selection of diagnostic tools based on the incident type rather than requiring manual selection of kubectl commands.
Compared to similar skills
k8s-incident side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| k8s-incident (this skill) | 1 | 9mo | Review | Intermediate |
| k8s-troubleshoot | 1 | 9mo | Review | Intermediate |
| debug-cluster | 2 | 1y | Review | Intermediate |
| k8s-cilium | 1 | 9mo | Review | Advanced |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by rohitg00
View all by rohitg00 →You might also like
k8s-troubleshoot
rohitg00
Debug Kubernetes pods, nodes, and workloads. Use when pods are failing, containers crash, nodes are unhealthy, or users mention debugging, troubleshooting, or diagnosing Kubernetes issues.
debug-cluster
openshift
Provides systematic debugging approaches for HyperShift hosted-cluster issues. Auto-applies when debugging cluster problems, investigating stuck deletions, or troubleshooting control plane issues.
k8s-cilium
rohitg00
Cilium and Hubble network observability for Kubernetes. Use when managing network policies, observing traffic flows, or troubleshooting connectivity with eBPF-based networking.
agentstack-server-debugging
i-am-bee
Instructions for debugging agentstack-server during development
health-checks
dadbodgeoff
Implement health check endpoints for load balancers, Kubernetes, and monitoring. Covers liveness, readiness, and dependency checks.
k8s-diagnostics
rohitg00
Kubernetes diagnostics for metrics, health checks, resource comparisons, and cluster analysis. Use when analyzing cluster health, comparing environments, or gathering diagnostic data.