Graceful shutdown of ASP.NET Core pods in Kubernetes: SIGTERM and grace period (C#)
The pod is removed from the Service endpoints and, at the same time, the container receives SIGTERM.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | // ====================================================================== // Q: What happens when Kubernetes stops a pod? How do you shut down gracefully? // // - The pod is removed from the Service endpoints and, at the same time, the // container receives SIGTERM. // - The .NET host triggers ApplicationStopping. Kestrel stops accepting new // requests and completes the in-flight ones. BackgroundServices get // cancellation through the stoppingToken. // - If it doesn't finish within terminationGracePeriodSeconds (default 30s), // SIGKILL follows. // - Workers: honor the CancellationToken. A message that was not finished is not // completed, so it gets redelivered (Q30), which is why the consumer must be // idempotent (Q31). // - Memory: .NET reads the container's memory limit (cgroups) and sizes the GC // heap accordingly (default ~75% of the limit). If the pod gets OOMKilled, // look for a leak first (Q44) and only then raise the limit. // - CPU limits cause throttling, which looks like random latency. We often set // only CPU requests, without limits. // ====================================================================== // The .NET timeout must be shorter than the Kubernetes grace period builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25)); |