Operating distributed .NET systems: OpenTelemetry tracing, health checks, message versioning

Distributed tracing with OpenTelemetry: the trace context (W3C traceparent) flows automatically through HTTP calls and Service Bus messages, so you can see the whole path of an order across all services.

csharp
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// ======================================================================
// Q: How do you debug and operate a distributed system?
//
// - Distributed tracing with OpenTelemetry: the trace context (W3C traceparent)
//   flows automatically through HTTP calls and Service Bus messages, so you can
//   see the whole path of an order across all services.
// - Structured logs with OrderId and MessageId as fields, not inside the message
//   text.
// - Metrics and alerts: queue length, age of the oldest message, DLQ size,
//   consumer lag.
// - Health checks (/health/live, /health/ready) for Kubernetes / Container Apps.
// - Versioning of message contracts: additive changes only (new optional
//   fields). The consumer is a tolerant reader (ignores unknown fields). For a
//   breaking change: a new message type (OrderPlacedV2), and for a while you
//   publish both.
// - API Gateway (YARP, Azure API Management): a single entry point for auth,
//   rate limiting and routing. It contains no business logic.
// - .NET Aspire: spins up all services, the database and the broker locally,
//   with service discovery and a dashboard for traces and logs.
// A good closing line: "Microservices solve a team-organization and scaling
// problem, not a code problem. If the code is tangled in a monolith, it will be
// tangled when distributed too."
// ======================================================================

// OpenTelemetry (Azure Monitor) and health checks
builder.Services.AddOpenTelemetry()
    .UseAzureMonitor()                                   // Application Insights
    .WithTracing(t => t.AddSource("Shop.Ordering"))      // your own ActivitySource
    .WithMetrics(m => m.AddMeter("Shop.Ordering"));

builder.Services.AddHealthChecks()
    .AddDbContextCheck<AppDbContext>(tags: ["ready"]);