API Priority and Fairness: Keeping the Control Plane Fair Under Load

When the API server is overloaded, the question is not just how many requests it drops but which ones. If a runaway controller's list requests crowd out node heartbeats and leader election, the cluster does not just slow down, it loses its control plane. API Priority and Fairness (APF) is the mechanism that decides who wins under pressure.

This post targets Kubernetes 1.36, where APF is the flowcontrol.apiserver.k8s.io/v1 API, GA since 1.29 and enabled by default (source: API Priority and Fairness). It replaces the blunt --max-requests-inflight flags with priority levels and fair queuing: a request is matched by a FlowSchema to a PriorityLevelConfiguration, then queued within that level by a flow key, and it costs seats, with a large list costing several. The fix for a noisy client is a FlowSchema that routes it to a low-share priority level, not a raised global limit. What follows walks that path from the incident back to the tuning.

This is the companion to the API server survival guide, which covers the memory and etcd side of scale.

Why counting requests was never enough

The API server has always had a self-protection valve: --max-requests-inflight and --max-mutating-requests-inflight cap total concurrent requests. The problem is that a cap counts requests without judging them. Under those flags, a misbehaving client hammering list can fill the inflight budget, and the API server then rejects the next request regardless of what it is, including the kubelet renewing a node lease or the controller manager holding its leader-election lock. The result is nodes going NotReady and controllers losing leadership, from a problem that started as one noisy client.

APF fixes the judgment part. It classifies requests into priority levels, gives each level a share of concurrency, and queues fairly within a level, so a flood in one level cannot starve another.

How APF classifies and queues a request

Before any request runs, APF answers three things: which rule matches it, which pool of concurrency it belongs to, and where it waits if that pool is busy. A short vocabulary covers the moving parts.

• FlowSchema: a rule that matches requests (by user, group, or service account, and by resource) and assigns them to a priority level.
• PriorityLevelConfiguration: a pool of concurrency with its own share and queuing behaviour.
• Seat: one unit of concurrency. A request holds one or more seats while it executes.
• Flow: a subdivision within a priority level (by user or namespace) used for fair queuing, so one user cannot dominate the level.
• Seat cost / work estimate: how many seats a request takes; a large list takes more.

Every request is classified before it runs. The path is match, then queue, then dispatch when seats are free.

Copycopy code to clipboard
request ─▶ match FlowSchema (lowest matchingPrecedence wins)
              │
              ▼
        PriorityLevelConfiguration (its own seat budget)
              │
              ▼
        queue by flow key (fair queuing across flows)
              │
       seats free? ── yes ──▶ dispatch (hold seats while executing)
              │
              no ──▶ wait in queue, or 429 + Retry-After if full

Caption: a request is matched to a priority level, queued by flow, and dispatched when seats free up.

The two mandatory levels bound the behaviour. exempt is never limited, for the most critical system traffic. catch-all is the last resort for anything unmatched, deliberately given a small share so unclassified traffic cannot take over.

Why a big list costs more than one seat

A seat is one unit of concurrency, and in the simple case each request holds one seat while it runs. The important refinement is that a list estimated to return many objects is charged multiple seats, proportional to that estimate, because it will do proportionally more work serializing the response. Watch requests hold a seat only during their initial burst, and writes are charged extra seat-time for the notification work they trigger.

This is what stops a single expensive list from quietly consuming all concurrency: the work estimator makes it pay for what it will do, so it queues instead of monopolizing the level.

The two numbers that decide a level's capacity are its share of total concurrency and its queue depth. Adjust them below to see how many seats a level gets and how many requests it can hold before rejecting.

Fencing off a noisy client

The common task is isolating a client, say a batch controller, so its bursts cannot affect the rest of the cluster. First define a priority level with a small share and queuing enabled.

Copycopy code to clipboard
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata: { name: batch-low }
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10
    limitResponse:
      type: Queue
      queuing:
        queues: 64
        handSize: 6
        queueLengthLimit: 50

Notice type: Limited means this level is subject to concurrency limits (the other value, Exempt, means no limiting). nominalConcurrencyShares sets its slice of total concurrency relative to other levels, and limitResponse.type: Queue makes excess requests wait rather than be rejected outright.

Then a FlowSchema routes the batch service account into that level.

Copycopy code to clipboard
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata: { name: batch-low }
spec:
  priorityLevelConfiguration: { name: batch-low }
  matchingPrecedence: 900
  distinguisherMethod: { type: ByUser }
  rules:
    - subjects:
        - kind: ServiceAccount
          serviceAccount: { name: batch-runner, namespace: batch }
      resourceRules:
        - verbs: ["*"]
          apiGroups: ["*"]
          resources: ["*"]
          namespaces: ["*"]

Notice matchingPrecedence: 900 places this rule after the built-in high-priority schemas (lower numbers match first), so system traffic is classified before this one. distinguisherMethod: ByUser gives each user its own flow within the level, so two batch clients cannot starve each other.

The effect: the batch client now runs inside a 10-share level. When it bursts, its own level queues and it gets 429 Retry-After, while leader election and node heartbeats sit in their own levels, untouched.

How the queuing stays fair

The queues and handSize fields in that config are not arbitrary. APF's fair queuing uses shuffle sharding: each flow is hashed to a small random subset of the level's queues (handSize of them), so two heavy flows are unlikely to collide on all the same queues, and a light flow almost always finds an uncontended queue. Increasing queues lowers collision probability at the cost of memory; a queues of 1 disables fair queuing while still allowing queueing. Levels can also lend unused concurrency to busy levels through lendablePercent, so idle capacity is not wasted, while each level keeps a guaranteed floor. This is the mechanism behind "fair": not equal shares, but no flow able to starve another.

What Kubernetes ships by default

You rarely start from scratch. Kubernetes ships mandatory and suggested levels; most tuning is adding a FlowSchema that steers a specific client into an existing or new level.

Source: flow control. The reason leader-election and node-high exist as separate levels is exactly the failure this whole system prevents: those requests must never be crowded out by workload traffic.

Signals that one client needs its own lane

APF exposes its state through apiserver_flowcontrol_* metrics. The ones to watch:

Source: metrics reference. Rising rejected_requests_total on one level with the rest quiet is the signature of a client that needs its own FlowSchema.

Traps to avoid

• Raising global limits to fix 429s. Bumping --max-requests-inflight treats the symptom and can push the real overload onto etcd. Route the offending client to a low-share level instead.
• Wrong matchingPrecedence. A catch-all FlowSchema with too low a precedence can capture traffic you meant for a specific level. Lower numbers match first; order matters.
• Putting a client in exempt. Exempt bypasses limiting entirely, so an exempt noisy client is back to the old problem. Reserve exempt for genuine system traffic.
• A big list starving a level. A client doing large lists consumes many seats each. The work estimator queues it, but if it shares a level with important traffic, that traffic waits too. Isolate heavy-list clients.
• Queue full, not just slow. When current_inqueue_requests hits queues * queueLengthLimit for a level, new requests are rejected immediately. Tune the queuing config or the client, not the global cap.

Try it on a test service account

1. List the active config: kubectl get flowschemas and kubectl get prioritylevelconfigurations.
2. Create the batch-low level and FlowSchema above for a test service account and watch it get its own level.
3. Generate a burst from that account and confirm apiserver_flowcontrol_rejected_requests_total rises only for batch-low.
4. Compare current_executing_seats for a normal request versus a large list to see the seat cost.
5. Chart request_wait_duration_seconds by level to find which traffic is queuing.

APF makes overload fair: it classifies requests, gives each class a seat budget, and queues within a class, so a noisy client waits instead of taking down leader election. The right lever is almost always a FlowSchema that isolates the offender, not a bigger global limit. Pair this with the API server survival guide for the memory and etcd side of the same story.

Further reading

• API Priority and Fairness
• KEP-1040: priority and fairness
• Kubernetes metrics reference
• API server survival guide at scale