API Server Survival Guide at Scale

26.05.2026 — kubernetes, api-server, etcd, scalability, platform-engineering — 7 min read

The kube-apiserver is the one component everything else talks to, and on a large cluster it is usually the first thing to fall over. It goes down the same way almost every time. A controller lists every Pod, the server buffers the whole response, memory spikes, and the process gets an out-of-memory (OOM) kill. It restarts with a cold cache, re-reads everything from etcd, that spike slows every write, and the next list finishes the job. This post follows that loop and shows how each fix breaks a link in it. It targets Kubernetes 1.36.

Most of the relief landed in one release. Consistent reads from cache and streaming list responses both went GA in 1.34, the snapshottable API server cache went beta in 1.34 behind ListFromCacheSnapshot, and API Priority and Fairness (APF) has been GA since 1.29. Together they mean most reads never touch etcd, large lists no longer buffer whole in memory, and the server can shed load before it crashes. Four metrics tell you when any of that is slipping.

For how requests get prioritized under load, see the companion post on API Priority and Fairness. For how the control plane fits together, see the architecture deep dive.

The failure loop

etcd is the cluster's database, but clients do not talk to it. They talk to the API server, which reads from etcd and caches. The trouble is reads that return a lot of data. A controller that lists every Pod in a 5,000-pod cluster asks the API server to produce a response that can be hundreds of megabytes. If several such lists arrive at once, and the server buffers each one whole before sending, memory climbs fast. Under network backpressure the buffers linger, and the server OOMs. When it restarts, its cache is cold, so it re-reads everything from etcd, which spikes etcd, which slows every write. That is the failure loop large clusters hit.

The moving parts

Watch cache: an in-memory, eventually consistent copy of a resource's state inside the API server, populated from etcd.
Consistent read from cache: serving a strongly consistent read from the watch cache rather than etcd, while still guaranteeing it reflects etcd.
list vs watch: a list returns a snapshot of many objects at once; a watch streams changes over time. Lists are the expensive one.
API Priority and Fairness (APF): the mechanism that classifies requests into priority levels and queues under load.
Informer: the client-side cache most controllers use, which does one list then a long watch.

Tracing one request

Every request runs the same gauntlet before it returns data. Knowing the path tells you where latency and memory come from.

client ─▶ authn ─▶ authz ─▶ APF (classify + queue) ─▶ handler
                                                          │
                                             read? ───────┤
                                              │           │
                                        watch cache   mutating ─▶ etcd (write)
                                        (most reads)
                                              │
                                        serialize + stream to client

Caption: reads are usually served from the watch cache; writes and some reads go to etcd.

Two things on this path dominate cost at scale. Serialization of a large list is where memory goes. And etcd is where write latency and cold-cache reads go. APF sits early, so it can shed load before the expensive part runs.

Why lists are the threat, and what defused them

A list used to be built as one contiguous buffer: the server serialized every object into memory, then wrote the whole blob. For a big collection that is a large allocation held until the client finishes reading it, and Go's buffer reuse keeps oversized buffers around afterward. Three features, all landed by 1.34, cut this down.

Consistent reads from cache means a strongly consistent get or list can be served from the watch cache instead of etcd, while still reflecting etcd's state. This removed the old reason to bypass the cache and hit etcd for consistency, which was a major source of etcd load.

Streaming list responses changes serialization: the server encodes and sends items one at a time, freeing each as it goes, so memory stays flat regardless of list size.

Before: serialize whole list ─▶ [ one big buffer ] ─▶ send   (can OOM)
After:  encode item ─▶ send ─▶ free ─▶ next item             (flat memory)

Caption: streaming encoding keeps API server memory flat on large lists.

The snapshottable cache lets the server answer paginated reads and reads at an older resource version from in-memory snapshots, instead of falling back to etcd. Before it, pagination and point-in-time reads bypassed the cache and loaded etcd directly.

You do not configure these per request; they are on by default in 1.36. The action item is to upgrade, and to prefer well-behaved list patterns (see the gotchas below).

Why watch scales and list does not

The reason watch scales and list does not is state. A watch is a long-lived stream that sends deltas, so the server holds little per client. A list is a point-in-time snapshot that must be fully materialized. This is why the entire controller ecosystem is built on informers: one list to seed the cache, then a watch forever. When you see a controller repeatedly full-listing instead of watching, that is a bug, and at scale it is an outage waiting to happen. The streaming and snapshot features made the unavoidable lists cheaper; they did not make a badly written full-list-every-second controller safe.

The overload guard

The old controls were two flags, --max-requests-inflight and --max-mutating-requests-inflight, which cap total concurrent requests but cannot tell a critical request from a noisy one. Under those flags, a runaway client could crowd out the scheduler. APF replaces them with priority levels and fair queuing, so leader-election and node heartbeats keep flowing while a batch client is throttled. APF is GA and on by default. It has its own deep dive; here it matters as the thing that sheds load before serialization, returning 429 with a Retry-After instead of OOMing.

Four signals to watch

At scale you watch four signals. Each maps to a distinct failure, and each has a specific metric from the metrics reference.

Symptom	Metric	Points at
Slow reads/writes overall	`apiserver_request_duration_seconds`	API server latency by verb/resource
Writes slow, `/readyz/etcd` flaps	`etcd_request_duration_seconds`, `etcd_disk_wal_fsync_duration_seconds`	etcd disk or load
429s, controllers lagging	`apiserver_flowcontrol_rejected_requests_total`, `apiserver_flowcontrol_current_inqueue_requests`	APF saturation
Mutations stall on one resource	`apiserver_admission_webhook_admission_duration_seconds`	a slow admission webhook

Source: Kubernetes metrics reference. The single most useful early-warning signal is etcd WAL fsync latency; when it climbs past a few tens of milliseconds, every write slows and the rest follows.

When it still falls over

A list without resourceVersion or pagination. A client that lists everything with strong consistency and no paging is the classic memory spike. Prefer informers (list once, then watch) and paginate large lists with limit.
etcd disk latency. etcd_disk_wal_fsync_duration_seconds p99 above roughly 100 ms means slow disks; every mutating request waits on the WAL fsync. Give etcd fast, dedicated disks.
Cold cache after a restart. When the API server restarts, its watch cache re-initializes from etcd, briefly spiking etcd. Several API servers restarting together can overwhelm etcd; stagger rollouts.
A slow admission webhook. apiserver_admission_webhook_admission_duration_seconds plateauing at the webhook timeout means a webhook is the bottleneck, and every matching mutation waits on it. Set tight timeouts and a sane failurePolicy.
APF starving a level. apiserver_flowcontrol_rejected_requests_total rising with 429s means a priority level's concurrency is exhausted. Do not raise the global limits blindly; tune the FlowSchema for the offending client.

Put it under load yourself

Chart apiserver_request_duration_seconds and etcd_request_duration_seconds side by side; find which leads the other during load.
Run a large kubectl get pods -A and watch API server memory; then compare with --chunk-size (paginated).
Alert on etcd_disk_wal_fsync_duration_seconds p99 above 100 ms.
Inspect apiserver_flowcontrol_current_inqueue_requests by priority level to see who queues under load.
Audit your controllers for full lists that should be informers.

When you want to tune which requests win under that pressure, the APF deep dive covers the knobs.