I'm keeping this post around for reference, but treat it as a snapshot of where I was a while back rather than current advice — some of the defaults and recommended API versions have moved on since I wrote this. The core mistakes are still worth remembering, though.
No Readiness Probe, No Warning
The very first version of our deployment manifest had a livenessProbe but no readinessProbe. Kubernetes started routing traffic to pods the instant the container process started, not when the application was actually ready to serve requests — which for us meant a few seconds of failed requests on every rollout while the app finished its startup work.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
template:
spec:
containers:
- name: api
image: registry.example.com/api:1.4.2
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
Adding the readinessProbe alone eliminated the failed-request blip almost entirely, because the Service stopped sending traffic to a pod until /healthz actually returned success.
No Resource Limits Meant No Predictability
The other early mistake was deploying without resources.requests or resources.limits at all. Without requests, the scheduler has no idea how much room a pod actually needs, so it happily packed too many pods onto the same node. Without limits, one pod with a memory leak (see the Node.js leak I tracked down here) could starve every other pod on that node instead of just getting killed on its own.
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
Setting real numbers, even rough ones based on actual observed usage, turned "the whole node fell over" incidents into "one pod got OOMKilled and restarted," which is a categorically better failure mode.
Rolling Update Strategy, Left at Defaults
I also left strategy.rollingUpdate at its defaults, which meant Kubernetes could take down more pods at once during a rollout than we could actually afford at our (small, at the time) replica count. Tightening maxUnavailable to 0 and giving maxSurge a bit more room fixed rollouts that occasionally dropped capacity below what traffic needed:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
The Mistake That Took Longest to Diagnose
The one that cost the most debugging time wasn't in the manifest at all — it was assuming kubectl rollout status finishing meant the rollout was actually healthy. It only confirms that the new ReplicaSet reached its desired replica count; it says nothing about whether those pods are handling real traffic correctly. We had a rollout "succeed" while quietly serving elevated error rates for several minutes, because the readiness probe checked a /healthz endpoint that didn't actually exercise the database connection the app depended on.
readinessProbe:
httpGet:
path: /healthz/deep
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
Splitting the probe into a shallow /healthz for the liveness check and a deeper /healthz/deep — one that actually pings the database — for readiness meant a pod with a broken DB connection never started receiving traffic in the first place, instead of passing a check that only proved the process was running.
Why I'm Leaving This As-Is
I've since moved a lot of this workload to simpler infrastructure for the traffic level we actually have, so I don't maintain these manifests day-to-day anymore. I'm keeping the post up because the underlying lessons — readiness probes are not optional, set real resource requests, don't trust the rollout defaults, and don't trust "rollout status" as a proxy for "actually healthy" — are still exactly the advice I'd give anyone standing up their first cluster today, even if my own current setup looks different.