logging-monitoring

Logging & Monitoring

Purpose: Implement observability for production systems. Goal: Structured logs, correlation across requests, actionable metrics. Note: For implementation, see C# Development or Python Development.

When to Use This Skill

• Adding structured logging to applications
• Implementing request correlation IDs
• Configuring metrics collection
• Setting up distributed tracing (OpenTelemetry)
• Designing alerting rules and health checks

Prerequisites

• Logging framework installed
• Monitoring platform access

Decision Tree

Observability concern?
+- What to log?
| +- Request start/end -> INFO with correlation ID
| +- Expected errors -> WARN (validation, not-found)
| +- Unexpected errors -> ERROR with stack trace
| - Debug details -> DEBUG (disabled in production)
+- What NOT to log?
| - PII, passwords, tokens, credit cards -> NEVER
+- Metrics needed?
| +- RED metrics: Rate, Errors, Duration (for services)
| - USE metrics: Utilization, Saturation, Errors (for resources)
+- Distributed tracing?
| - OpenTelemetry for cross-service correlation
- Alerting?
 +- SLO-based: alert on error budget burn rate
 - Avoid alert fatigue: page only for actionable issues

Structured Logging

Concept

Log structured data (key-value pairs) instead of plain text for better searchability and analysis.

[FAIL] Unstructured (hard to parse):
 "User [email protected] logged in from 192.168.1.1 at 2024-01-15 10:30:00"

[PASS] Structured (machine-readable):
 {
 "event": "user_login",
 "user_email": "[email protected]",
 "ip_address": "192.168.1.1",
 "timestamp": "2024-01-15T10:30:00Z",
 "level": "INFO"
 }

Benefits

• Searchable: Query by any field
• Filterable: Show only errors, specific users, etc.
• Aggregatable: Count events, calculate averages
• Parseable: Tools can process automatically

Log Levels

Standard Levels

Level Configuration by Environment

Development: DEBUG or TRACE
 - See detailed information for debugging

Staging: INFO
 - Normal operations plus warnings/errors

Production: INFO (or WARN)
 - Reduce noise, focus on significant events
 - Keep ERROR/FATAL always enabled

Core Rules

Anti-Patterns

• Log and Forget: Writing logs but never querying or reviewing them -> Set up dashboards and alerts on ERROR/FATAL; review logs in incident postmortems
• PII in Logs: Logging email addresses, passwords, tokens, or credit card numbers -> Scrub sensitive fields before logging; use allowlists for loggable fields
• Unstructured Strings: Logging plain text messages that are hard to parse or search -> Use structured logging (JSON key-value pairs) for all log entries
• Missing Correlation: Logs from different services with no shared request ID -> Propagate W3C trace context or a correlation ID header across all service calls
• Alert Fatigue: Alerting on every warning or non-actionable metric -> Page only on SLO budget burn rate; group related alerts; include runbook links
• Debug in Production: Running production with DEBUG or TRACE level enabled -> Use INFO or WARN in production; enable DEBUG temporarily and only on specific components
• Metric Overload: Tracking hundreds of custom metrics with no clear purpose -> Focus on RED (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources

Observability Tools

See Also: Error Handling - C# Development - Python Development

Troubleshooting

References

• Logging Correlation Metrics
• Tracing Health Alerting