.NET Microservices, Performance & Docker

// ======================================================================
// Q: A pod is in CrashLoopBackOff. What do you do?
//
// - Exception at startup (missing config or secret): logs --previous shows
//   InvalidOperationException or OptionsValidationException.
// - OOMKilled: describe shows Last State: Terminated, Reason: OOMKilled.
// - Liveness probe too strict: Events show Liveness probe failed, with restarts
//   and no exception in the logs.
// - Wrong port: the probe fails; the app listens on 8080 but the manifest says
//   80.
// - ImagePullBackOff: wrong tag or missing access to the registry (ACR).
// ValidateOnStart() on Options makes a bad setting show up immediately at
// startup with a clear message, instead of later on the first request.
// ======================================================================

// Fail fast on invalid configuration
builder.Services.AddOptions<ErpOptions>()
    .BindConfiguration("Erp")
    .ValidateDataAnnotations()
    .ValidateOnStart();          // the pod fails immediately, with a clear message in the logs

# ======================================================================
# Q: A pod is in CrashLoopBackOff. What do you do?
#
# - Exception at startup (missing config or secret): logs --previous shows
#   InvalidOperationException or OptionsValidationException.
# - OOMKilled: describe shows Last State: Terminated, Reason: OOMKilled.
# - Liveness probe too strict: Events show Liveness probe failed, with restarts
#   and no exception in the logs.
# - Wrong port: the probe fails; the app listens on 8080 but the manifest says
#   80.
# - ImagePullBackOff: wrong tag or missing access to the registry (ACR).
# ValidateOnStart() on Options makes a bad setting show up immediately at
# startup with a clear message, instead of later on the first request.
# ======================================================================

# Triage a crashing pod
kubectl get pods -n shop
kubectl describe pod shop-api-7d9f-x2k -n shop      # Events: OOMKilled, failed probes, image pull
kubectl logs shop-api-7d9f-x2k -n shop --previous   # logs of the container that crashed
kubectl get events -n shop --sort-by=.lastTimestamp
kubectl rollout undo deployment/shop-api -n shop    # roll back first if users are affected
kubectl port-forward svc/shop-api 8080:80 -n shop   # access from your machine for debugging

# ======================================================================
# Q: How do you handle configuration, secrets and scaling in Kubernetes?
#
# - ConfigMap for configuration (Erp__BaseUrl), exposed as env vars and read by
#   IConfiguration as usual.
# - Kubernetes Secrets are only base64, not encryption. Better: Azure Key Vault
#   through the Secrets Store CSI driver, or directly from the application with
#   Workload Identity (Q15), with no secrets in the cluster at all.
# - Stateless pods: sessions and cache in Redis, shared Data Protection keys
#   (Q38), files in Blob Storage and not on the pod's disk.
# - Migrations as a Kubernetes Job or a pipeline step before the rollout, not at
#   the startup of every pod (Q8).
# - PodDisruptionBudget, so that a node upgrade doesn't take down all replicas
#   at once.
# ======================================================================

# Worker: scaling based on queue length (KEDA), even down to 0
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: erp-sync-worker }
spec:
  scaleTargetRef: { name: erp-sync-worker }
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
    - type: azure-servicebus
      metadata:
        topicName: order-events
        subscriptionName: erp-sync
        messageCount: "50"                  # ~1 replica per 50 waiting messages
      authenticationRef: { name: servicebus-workload-identity }

# ======================================================================
# Q: How do you handle configuration, secrets and scaling in Kubernetes?
#
# - ConfigMap for configuration (Erp__BaseUrl), exposed as env vars and read by
#   IConfiguration as usual.
# - Kubernetes Secrets are only base64, not encryption. Better: Azure Key Vault
#   through the Secrets Store CSI driver, or directly from the application with
#   Workload Identity (Q15), with no secrets in the cluster at all.
# - Stateless pods: sessions and cache in Redis, shared Data Protection keys
#   (Q38), files in Blob Storage and not on the pod's disk.
# - Migrations as a Kubernetes Job or a pipeline step before the rollout, not at
#   the startup of every pod (Q8).
# - PodDisruptionBudget, so that a node upgrade doesn't take down all replicas
#   at once.
# ======================================================================

# API: CPU-based scaling (HPA)
kubectl autoscale deployment shop-api --cpu-percent=70 --min=3 --max=10

# ======================================================================
# Q: What happens when Kubernetes stops a pod? How do you shut down gracefully?
#
# - The pod is removed from the Service endpoints and, at the same time, the
#   container receives SIGTERM.
# - The .NET host triggers ApplicationStopping. Kestrel stops accepting new
#   requests and completes the in-flight ones. BackgroundServices get
#   cancellation through the stoppingToken.
# - If it doesn't finish within terminationGracePeriodSeconds (default 30s),
#   SIGKILL follows.
# - Workers: honor the CancellationToken. A message that was not finished is not
#   completed, so it gets redelivered (Q30), which is why the consumer must be
#   idempotent (Q31).
# - Memory: .NET reads the container's memory limit (cgroups) and sizes the GC
#   heap accordingly (default ~75% of the limit). If the pod gets OOMKilled,
#   look for a leak first (Q44) and only then raise the limit.
# - CPU limits cause throttling, which looks like random latency. We often set
#   only CPU requests, without limits.
# ======================================================================

# Deployment: a short wait so the load balancer stops sending traffic before SIGTERM
spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      lifecycle:
        preStop:
          # native sleep (Kubernetes 1.30+): the chiseled image has no /bin/sleep
          sleep: { seconds: 5 }