Debugging a .NET pod in CrashLoopBackOff on Kubernetes (kubectl)

Exception at startup (missing config or secret): logs --previous shows InvalidOperationException or OptionsValidationException.

bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# ======================================================================
# Q: A pod is in CrashLoopBackOff. What do you do?
#
# - Exception at startup (missing config or secret): logs --previous shows
#   InvalidOperationException or OptionsValidationException.
# - OOMKilled: describe shows Last State: Terminated, Reason: OOMKilled.
# - Liveness probe too strict: Events show Liveness probe failed, with restarts
#   and no exception in the logs.
# - Wrong port: the probe fails; the app listens on 8080 but the manifest says
#   80.
# - ImagePullBackOff: wrong tag or missing access to the registry (ACR).
# ValidateOnStart() on Options makes a bad setting show up immediately at
# startup with a clear message, instead of later on the first request.
# ======================================================================

# Triage a crashing pod
kubectl get pods -n shop
kubectl describe pod shop-api-7d9f-x2k -n shop      # Events: OOMKilled, failed probes, image pull
kubectl logs shop-api-7d9f-x2k -n shop --previous   # logs of the container that crashed
kubectl get events -n shop --sort-by=.lastTimestamp
kubectl rollout undo deployment/shop-api -n shop    # roll back first if users are affected
kubectl port-forward svc/shop-api 8080:80 -n shop   # access from your machine for debugging