Resource Requests and Limits

The one-sentence version

Requests = what the scheduler uses to place your Pod. Limits = what the kernel uses to constrain your container at runtime. They're used at different times by different parts of the system.

⚠️ The Mi vs M confusion has caused real incidents

Setting memory: 1G gives 1,000,000,000 bytes. Setting memory: 1Gi gives 1,073,741,824 bytes — about 7% more. When engineers copy manifests and switch between M and Mi without thinking, the memory limit ends up slightly different from what was intended. Always use Mi and Gi — they match how the Linux kernel and JVM report memory.

What just happened?

How the scheduler used requests — When you applied this Deployment, the scheduler looked at each available node and checked: does this node have at least 150m CPU and 200Mi memory unallocated (i.e. not already promised to other Pods via their requests)? Nodes that didn't meet the bar were rejected. The scheduler doesn't know or care what CPU or memory the running containers are actually using — only what they've claimed via requests.

CPU throttling vs memory OOM-kill — CPU and memory behave fundamentally differently when a limit is exceeded. CPU is compressible: if a container tries to use more than 500m, Linux throttles it — it runs slower, but it keeps running. Memory is incompressible: if a container tries to use more than 350Mi, the Linux OOM killer immediately terminates the process. The container restarts. This is why memory limits need more headroom than CPU limits.

kubectl describe pod — Limits/Requests section — The kubectl describe pod output always shows the resolved limits and requests for each container. This is your source of truth for what's actually configured — not the Deployment YAML, which may have been modified since the Pod started.

What just happened?

Exit Code 137 — Exit code 137 means the process was killed by signal 9 (SIGKILL). When you see Reason: OOMKilled alongside exit code 137, the Linux OOM killer terminated the container because it exceeded its memory limit. The fix is to increase limits.memory — not to restart the Pod, which just gets OOM-killed again at the same usage level.

kubectl top — the actual vs requested gap — The top output shows real current usage. The Pods are consuming 338–344Mi of memory, but the limit is 350Mi. They're running at 97% of the memory limit — they will be OOM-killed on the next memory spike. The right fix here is to raise the memory limit to at least 500Mi and raise the request proportionally.

Restart Count: 4 — Combined with Last State: OOMKilled, a high restart count means the container has been getting killed and restarted repeatedly. Users are experiencing intermittent failures every time this happens. This is an active production incident, not just a warning.

What just happened?

Actual usage vs requests — The notification service is actually using 38–41m CPU and 94–97Mi memory. We set requests of 100m CPU and 128Mi memory — a reasonable buffer. This tells us requests are well-calibrated. The limits (500m CPU, 512Mi memory) have huge headroom, which is fine for now but wastes schedulable capacity. After a few weeks of observing peak load, tighten them to about 2x actual peak usage.

Over-committed limits (130% CPU, 112% memory) — The node's limits exceed 100% of physical capacity. This is normal and expected — Kubernetes allows over-commitment on limits because not all containers hit their limits simultaneously. But if they do, the OOM killer starts evicting BestEffort and Burstable Pods to recover memory. This is why QoS class matters.

Requests at 40% CPU, 38% memory — Requests reflect how much of the node is allocated — not how much is being used. When requests hit around 80–90% of capacity, the scheduler will start struggling to place new Pods even if actual CPU usage is only 30%. This is the "noisy neighbour" problem — over-inflated requests lock up schedulable capacity. Right-sizing requests is just as important as right-sizing limits.

Teacher's Note: The two failure modes and how to avoid both

There are two opposite ways to get resources wrong. Under-requesting: setting requests too low. The scheduler packs too many Pods onto nodes because it thinks they're lightweight. Actual usage exceeds node capacity, memory pressure builds, evictions cascade, and suddenly 40% of your Pods are being killed at once. This is the "noisy neighbour" that took down three services mentioned in Lesson 18 — except at the node level rather than the namespace level.

Over-requesting: setting requests too high. Every Pod claims far more than it uses. Nodes look "full" to the scheduler even when they're running at 20% actual utilisation. New Pods can't schedule. Cluster autoscaler spins up new nodes unnecessarily. You end up paying for 3x the cloud compute you actually need. I've seen clusters where actual CPU utilisation was 15% but every node was "full" because request values were copied from a legacy VM that had peak specs.

The right answer is continuous tuning: measure → observe peak usage → set requests to give a healthy buffer above p95 actual usage → set limits to give safety headroom above that → repeat. Tools like Vertical Pod Autoscaler (Lesson 50) can do this automatically.

Up Next · Lesson 23

Health Checks

Liveness, readiness, and startup probes — how Kubernetes knows when your container is actually healthy and ready to serve traffic, not just running.