Graceful shutdown of ASP.NET Core pods in Kubernetes: SIGTERM and grace period (C#)

The pod is removed from the Service endpoints and, at the same time, the container receives SIGTERM.

csharp
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
// ======================================================================
// Q: What happens when Kubernetes stops a pod? How do you shut down gracefully?
//
// - The pod is removed from the Service endpoints and, at the same time, the
//   container receives SIGTERM.
// - The .NET host triggers ApplicationStopping. Kestrel stops accepting new
//   requests and completes the in-flight ones. BackgroundServices get
//   cancellation through the stoppingToken.
// - If it doesn't finish within terminationGracePeriodSeconds (default 30s),
//   SIGKILL follows.
// - Workers: honor the CancellationToken. A message that was not finished is not
//   completed, so it gets redelivered (Q30), which is why the consumer must be
//   idempotent (Q31).
// - Memory: .NET reads the container's memory limit (cgroups) and sizes the GC
//   heap accordingly (default ~75% of the limit). If the pod gets OOMKilled,
//   look for a leak first (Q44) and only then raise the limit.
// - CPU limits cause throttling, which looks like random latency. We often set
//   only CPU requests, without limits.
// ======================================================================

// The .NET timeout must be shorter than the Kubernetes grace period
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));