Deployments
Give every container that serves traffic a readiness probe, and keep its liveness probe to the process itself. Let it serve a few more seconds after Kubernetes asks it to stop. A rolling update then replaces pods only as fast as new ones turn ready, and Kubernetes restarts or replaces pods that fail. It doesn't roll back a rollout that gets stuck, so the deploy job has to wait for the rollout and fail when it doesn't finish.
Rolling updates
Only a change to the pod template starts a rollout; a new replica count doesn't (updating a Deployment). The Deployment controller then creates a new ReplicaSet, scales it up and scales the old one down. maxSurge sets how many pods may run above the replica count, and maxUnavailable how many may be missing. Both default to 25%, with the surge rounded up and the missing pods rounded down (rolling update). For one to three replicas that's one extra pod and none missing, so an old pod only stops once a new one is available.
A new pod is available once it's ready, as minReadySeconds defaults to 0 (min ready seconds). A container without a readiness probe counts as ready as soon as it starts (probe results). Without one, the rollout stops old pods while the new ones are still booting.
Shutdown
When a pod is deleted, two things start at the same time (pod termination). The kubelet stops the pod, and the control plane marks its endpoints in the Services' EndpointSlices as not ready. Load balancers that haven't processed that change yet still send new requests to the pod (KEP-3960). A preStop hook runs before the SIGTERM, so a short sleep there keeps the app serving until they have:
spec:
terminationGracePeriodSeconds: 30
containers:
- name: app
lifecycle:
preStop:
sleep:
seconds: 5The kubelet runs the sleep action itself, so the image needs no sleep binary (hook handlers). The action is on by default since Kubernetes 1.30 (feature gates). The grace period covers the hook and the app's shutdown together, and the kubelet kills what still runs when it ends. With the 5-second sleep, the app has 25 seconds to finish open requests after its SIGTERM. The runtime sends that signal to the container's PID 1, which is why an entrypoint script has to exec the app (see Startup scripts).
Volumes
A ReadWriteOnce volume can be mounted by one node at a time (access modes). When the extra pod of a rolling update lands on another node, it can't mount the volume while the old pod's node has it. The rollout then stalls. Give a single-replica Deployment with such a volume strategy.type: Recreate. It stops the old pod before it starts the new one, which means downtime on every deploy (Recreate).
Stuck rollouts
A rollout stalls when the new pods never turn ready, or when their image doesn't pull. After progressDeadlineSeconds, 600 by default, the Deployment gets the condition Progressing with status False and reason ProgressDeadlineExceeded. Kubernetes only reports it: the old pods that still run stay, and nothing rolls back (failed Deployment).
Make the deploy job wait for the rollout, so a stuck rollout fails the pipeline. kubectl rollout status exits with 1 once the deadline passes. helm upgrade --wait fails when the resources aren't ready within its --timeout, 5 minutes by default. Helm 4's --rollback-on-failure then rolls back to the last release that succeeded (helm upgrade).
Probes
The kubelet on the pod's node runs the probes. It runs a command in the container or connects to the pod's IP over HTTP, TCP or gRPC. A probe never goes through the Service. By default it probes every 10 seconds, gives up on an answer after 1 second and acts after 3 failures in a row (configuration fields). It reads at most 10 KiB of an HTTP response, so point probes at an endpoint that answers fast with a small body (HTTP probes).
containers:
- name: app
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
readinessProbe:
httpGet:
path: /ready
port: 8080
livenessProbe:
httpGet:
path: /healthz
port: 8080Readiness
A failing readiness probe takes the pod out of traffic. The kubelet sets the pod's Ready condition to false. The EndpointSlice controller then removes the pod's IP from the EndpointSlices of every Service that selects it (readiness probe). The container keeps running.
The Kubernetes docs suggest checking required backends in this probe, so no traffic goes to pods that can only answer with errors (when to use a readiness probe). When such a backend is down for every pod, the Service has no ready endpoints left. Requests then fail before they reach the app, including the ones that don't need the backend.
Liveness
A failing liveness probe restarts the container, so check only the process itself, such as whether it's deadlocked (liveness probe). Never check a database or another service. When that service goes down, every pod fails the probe and restarts, which doesn't bring the service back. The restart backoff can then keep the pods down for up to 5 minutes after the service has recovered (see Self-healing).
An app that exits on its own when it can't recover may not need a liveness probe, because the kubelet restarts it anyway (when to use a liveness probe).
Startup
Liveness and readiness probes wait until the startup probe has succeeded. An app that loads data or runs migrations when it starts needs one. Point it at the liveness endpoint, and set its failureThreshold so that failureThreshold times periodSeconds covers the longest start (when to use a startup probe): 30 in the example above allows 5 minutes at the default periodSeconds of 10. A long initialDelaySeconds on the liveness probe would also wait, but it waits that long after every start, even a fast one.
Self-healing
Kubernetes restarts failed containers, and replaces pods that are deleted or that run on a failed node (self-healing):
| Failure | What Kubernetes does | Delay |
|---|---|---|
| A container exits or fails its liveness probe | The kubelet restarts it in the same pod | 10 seconds, doubling up to 5 minutes |
| A pod is deleted or evicted | Its ReplicaSet creates a replacement, which the scheduler places on a node | Right away |
| A node stops reporting | The node controller marks it unhealthy, then its pods are evicted and replaced | 50 seconds, then 5 minutes |
Restarts
kubectl get pods shows a container that waits for its next restart as CrashLoopBackOff. The wait resets once the container has run for 10 minutes without a crash.
Nodes
A pod on a failed node is replaced only after about 6 minutes. Run two or more replicas of anything that has to stay up. The scheduler prefers to put them on different nodes, but doesn't require it (default spreading).
StatefulSets
A StatefulSet doesn't replace a pod on an unreachable node, because it runs at most one pod per identity. The StatefulSet controller waits until the node responds again or its Node object is deleted, by you or by the node controller (StatefulSet pods).