dotnet-observability

dotnet-observability

Modern observability for .NET applications using OpenTelemetry, structured logging, health checks, and custom metrics. Covers the three pillars of observability (traces, metrics, logs), integration with Microsoft.Extensions.Diagnostics and System.Diagnostics, and production-ready health check patterns.

Scope

• OpenTelemetry traces, metrics, and logs setup
• Health check endpoints and liveness/readiness patterns
• Custom metrics with System.Diagnostics.Metrics
• Structured logging integration with OTel export

Out of scope

• DI container mechanics and service lifetimes -- see [skill:dotnet-csharp-dependency-injection]
• Async/await patterns -- see [skill:dotnet-csharp-async-patterns]
• Testing observability output -- see [skill:dotnet-integration-testing]
• Middleware pipeline patterns (request logging, exception handling) -- see [skill:dotnet-middleware-patterns]

Cross-references: [skill:dotnet-csharp-dependency-injection] for service registration, [skill:dotnet-csharp-async-patterns] for async patterns in background exporters, [skill:dotnet-resilience] for Polly telemetry integration, [skill:dotnet-middleware-patterns] for request/exception logging middleware.

---

OpenTelemetry Setup

OpenTelemetry is the standard observability framework in .NET. The .NET SDK includes native support for System.Diagnostics.Activity (traces) and System.Diagnostics.Metrics (metrics), which OpenTelemetry collects and exports.

Package Landscape

Package	Purpose
OpenTelemetry.Extensions.Hosting	Host integration, lifecycle management
OpenTelemetry.Instrumentation.AspNetCore	Automatic HTTP server trace/metric instrumentation
OpenTelemetry.Instrumentation.Http	Automatic HttpClient trace/metric instrumentation
OpenTelemetry.Instrumentation.Runtime	GC, thread pool, assembly metrics
OpenTelemetry.Exporter.OpenTelemetryProtocol	OTLP exporter (gRPC/HTTP) for collectors
OpenTelemetry.Exporter.Console	Console exporter for local development

Install the core stack:

<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.*" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.*" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.*" />

```text

### Aspire Service Defaults Integration

If using .NET Aspire, the `ServiceDefaults` project configures OpenTelemetry automatically. This is the recommended approach for Aspire apps -- do not duplicate this configuration manually:

```csharp

// ServiceDefaults/Extensions.cs (generated by Aspire)
public static IHostApplicationBuilder AddServiceDefaults(
    this IHostApplicationBuilder builder)
{
    builder.ConfigureOpenTelemetry();
    builder.AddDefaultHealthChecks();
    // ... other defaults
    return builder;
}

```text

For non-Aspire apps, configure OpenTelemetry explicitly as shown below.

### Full Configuration (Non-Aspire)

```csharp

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: builder.Environment.ApplicationName,
            serviceVersion: typeof(Program).Assembly
                .GetCustomAttribute<AssemblyInformationalVersionAttribute>()
                ?.InformationalVersion ?? "unknown"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("MyApp.*")          // Custom ActivitySources
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("MyApp.*")           // Custom Meters
        .AddOtlpExporter());

```text

### OTLP Configuration via Environment Variables

The OTLP exporter reads standard environment variables -- no code changes needed between environments:

```bash

# Collector endpoint (gRPC default)
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# Or HTTP/protobuf
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Resource attributes
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=myapp

# Service name (overrides code-based configuration)
OTEL_SERVICE_NAME=order-api

```text

---


For detailed code examples (custom traces, metrics, structured logging, health checks, production configuration), see `examples.md` in this skill directory.

## Key Principles

- **Use OpenTelemetry as the standard** -- it provides vendor-neutral instrumentation that works with any backend (Prometheus, Grafana, Datadog, Azure Monitor, AWS X-Ray)
- **Use `IMeterFactory` from DI** -- do not create `Meter` instances directly; the factory integrates with the DI lifecycle and OpenTelemetry registration
- **Use source-generated `LoggerMessage`** for hot paths -- zero allocation when the log level is disabled
- **Separate liveness from readiness** -- liveness checks should not include dependency health; readiness checks should
- **Configure via environment variables** -- OTLP endpoint, service name, and resource attributes should not be hardcoded
- **Enrich logs with trace context** -- structured logging with `TraceId` and `SpanId` enables log-to-trace correlation
- **Follow OpenTelemetry semantic conventions** for metric and span names

---

## Agent Gotchas

1. **Do not create `Meter` or `ActivitySource` via `new` in DI-registered services without using `IMeterFactory`** -- instruments created outside the factory are not collected by the OpenTelemetry SDK. Use `IMeterFactory.Create()` for `Meter` instances. `ActivitySource` is static and registered via `.AddSource()`.
2. **Do not add dependency checks to liveness endpoints** -- a database outage should not restart the app. Only the readiness endpoint should check dependencies.
3. **Do not use `ILogger.LogInformation("message: " + value)` or string interpolation `$"message: {value}"`** -- use structured logging templates: `ILogger.LogInformation("message: {Value}", value)`. String concatenation and interpolation bypass structured logging and prevent log indexing.
4. **Do not configure OTLP endpoints in code for production** -- use environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`) so the same image works across environments.
5. **Do not forget to register custom `ActivitySource` names** with `.AddSource("MyApp.*")` -- unregistered sources are silently ignored and produce no traces.

---

## References

- [OpenTelemetry .NET documentation](https://opentelemetry.io/docs/languages/net/)
- [.NET observability with OpenTelemetry](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/observability-with-otel)
- [ASP.NET Core health checks](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks)
- [System.Diagnostics.Metrics](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/metrics)
- [High-performance logging in .NET](https://learn.microsoft.com/en-us/dotnet/core/extensions/high-performance-logging)
- [Logging in .NET: Log filtering](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging#how-filtering-rules-are-applied)
- [Serilog OpenTelemetry sink](https://github.com/serilog/serilog-sinks-opentelemetry)

---

## Attribution

Adapted from [Aaronontheweb/dotnet-skills](https://github.com/Aaronontheweb/dotnet-skills) (MIT license).

Code Navigation (Serena MCP)

Primary approach: Use Serena symbol operations for efficient code navigation:

1. Find definitions: serena_find_symbol instead of text search
2. Understand structure: serena_get_symbols_overview for file organization
3. Track references: serena_find_referencing_symbols for impact analysis
4. Precise edits: serena_replace_symbol_body for clean modifications

When to use Serena vs traditional tools:

• Use Serena: Navigation, refactoring, dependency analysis, precise edits
• Use Read/Grep: Reading full files, pattern matching, simple text operations
• Fallback: If Serena unavailable, traditional tools work fine

Example workflow:

# Instead of:
Read: src/Services/OrderService.cs
Grep: "public void ProcessOrder"

# Use:
serena_find_symbol: "OrderService/ProcessOrder"
serena_get_symbols_overview: "src/Services/OrderService.cs"