Kueue Deep Dive: Job Queueing and Quotas for Kubernetes

02.09.2026 — kubernetes, kueue, batch, scheduling, ml-infra — 9 min read

Kubernetes has a scheduler, but it has no queue. Submit more jobs than the cluster can run and it does not line them up; it admits them all, and their pods sit Pending forever, half-scheduled, holding nothing and finishing nothing. For batch and machine-learning (ML) workloads on shared GPUs, that is the whole problem. Kueue is the project that adds the missing layer: quota, queueing, and fair sharing on top of the scheduler. This post is a deep dive into how it works. It targets Kubernetes 1.37 and the Kueue v1beta2 API.

This builds on two earlier posts: gang scheduling for why all-or-nothing admission matters, and JobSet for the job shape Kueue admits.

The object shapes here follow the Kueue v1beta2 reference and concept docs (Kueue concepts, ClusterQueue, admission). What follows has a short spine. A job targets a namespaced LocalQueue that points at a cluster-scoped ClusterQueue holding the quota, and Kueue tracks each job internally as a Workload. It admits that Workload in two phases, reserving quota first and running admission checks second, and nothing runs until quota is reserved. ClusterQueues grouped in a cohort borrow unused quota from each other within borrowingLimit and lendingLimit, and reclaim it through preemption. Because a Workload is admitted atomically, a distributed job gets gang admission instead of starting half its pods.

No queue behind the scheduler

The default scheduler answers one question: where does this pod go? It does not answer: should this job start now, or wait? So when a team submits ten training jobs to a cluster that can run three, Kubernetes creates all ten jobs' pods. The scheduler places what fits and leaves the rest Pending. Now the cluster is full of jobs that each have some pods running and some pending, none of them able to finish, all of them holding partial resources. There is no notion of "this job waits until that job frees its GPUs," and no notion of "team A gets half the cluster, team B the other half." Kueue adds both.

The objects that build a queue

A working queue is not one object but six that reference each other.

ResourceFlavor: a description of a kind of node, for example A100 GPUs, tied to nodes by labels. Quota is counted per flavor.
ClusterQueue: a cluster-scoped pool of quota with sharing and preemption rules. This is where admins set how much each team gets.
LocalQueue: a namespaced pointer to a ClusterQueue. Jobs reference a LocalQueue, not the ClusterQueue directly.
Workload: the internal object Kueue creates for each job, capturing its total resource request. Kueue queues and admits Workloads, not pods.
Cohort: a group of ClusterQueues that can share unused quota.
AdmissionCheck: an extra gate that must pass, after quota is reserved, before a Workload is admitted.

Admission in two phases

A Kueue-managed job starts life suspended. Kueue creates a Workload for it, queues that Workload, and only unsuspends the job once it has both reserved quota and cleared every admission check.

Job (suspend: true) ─▶ Kueue creates a Workload
                            │
                            ▼
                   queue in the LocalQueue ─▶ ClusterQueue
                            │
             phase 1: reserve quota (borrow from cohort if needed)
                            │
             phase 2: run admission checks concurrently
                            │
               all positive? ── yes ──▶ admit: unsuspend the Job
                            │
                            no ──▶ stay queued

Caption: Kueue's two-phase admission; the job's pods never exist until the Workload is admitted.

This two-phase shape is the key design choice. Phase one is logical: is there quota? Phase two is everything else: admission checks, and physical placement when topology-aware scheduling is used. Because the job is suspended until both pass, Kueue never leaves a job half-started. That is also what makes gang admission natural: the whole Workload is admitted at once or not at all.

Quota for one team, in four objects

The minimum setup is four objects. First, a ResourceFlavor describing the GPU nodes.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata: { name: gpu-a100 }
spec:
  nodeLabels:
    accelerator: nvidia-a100

Notice nodeLabels ties this flavor to a subset of nodes, so quota is tracked separately per hardware type rather than as one undifferentiated pool.

Then a ClusterQueue that owns the quota, in a cohort so it can share.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata: { name: team-a }
spec:
  namespaceSelector: {}
  cohortName: research
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: gpu-a100
          resources:
            - name: cpu
              nominalQuota: 100
            - name: memory
              nominalQuota: 400Gi
            - name: "nvidia.com/gpu"
              nominalQuota: 8
              borrowingLimit: 8
              lendingLimit: 4

Notice nominalQuota: 8 is what team A is guaranteed, borrowingLimit: 8 lets it use up to 8 more GPUs borrowed from the cohort when they are idle, and lendingLimit: 4 caps how many of its own it will lend out. Borrowing only exists because of cohortName: research.

Then a namespaced LocalQueue that jobs point at, and a Job that targets it.

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata: { name: team-a, namespace: team-a }
spec:
  clusterQueue: team-a
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: team-a
spec:
  suspend: true
  # ... pod template requesting nvidia.com/gpu ...

Notice the kueue.x-k8s.io/queue-name label is how the job selects its LocalQueue, and suspend: true hands start control to Kueue. Kueue flips suspend to false when it admits the Workload.

How idle quota moves through a cohort

The three quota numbers are the whole economy. nominalQuota is guaranteed. lendingLimit bounds how much a queue will give away when idle. borrowingLimit bounds how much it can take from the cohort's idle capacity on top of its nominal.

Cohort: research
 ┌───────────────────────────────────────────┐
 │ ClusterQueue team-a   nominal 8, lend ≤ 4  │
 │ ClusterQueue team-b   nominal 8, lend ≤ 4  │
 └───────────────────────────────────────────┘
   team-b idle ─▶ team-a borrows up to +8 (its borrowingLimit)
   team-b returns ─▶ team-a's borrowed GPUs are reclaimed

Caption: within a cohort, idle quota flows to busy queues and is reclaimed when the owner needs it.

The point of the two limits is bounded generosity. Without a borrowingLimit, a busy queue could consume the whole cohort; without a lendingLimit, a queue could lend so much that its own jobs starve when they arrive. Quotas can also be defined at the cohort level and shared down, which is how you model a shared pool plus per-team floors.

Taking borrowed quota back

Borrowing raises an obvious question: what happens when the lender wants its quota back? Preemption. Kueue preempts a Workload in two situations. Within one ClusterQueue, a higher-priority pending Workload can preempt a lower-priority running one. Across a cohort, a ClusterQueue below its nominal quota can reclaim resources that another queue borrowed, preempting the borrower (preemption).

Fair sharing changes the ordering so that a queue's history matters: workloads from a LocalQueue that has consumed less over time are preferred, so a team that has been quiet is not permanently behind a team that floods the queue. Kueue's fair-sharing preemption is designed so two Workloads in different ClusterQueues cannot enter a loop preempting each other (fair sharing).

Extra gates and queues across clusters

An AdmissionCheck is a second gate that runs after quota is reserved. All checks configured on a ClusterQueue run concurrently, and the Workload is admitted only when every one returns positive (admission check). This is the extension point for "quota says yes, but also confirm X."

MultiKueue is the largest user of that extension point. A manager cluster holds ClusterQueues whose quota mirrors the total across worker clusters, and a MultiKueue AdmissionCheck dispatches an admitted Workload to a worker cluster that acts as a standalone Kueue (MultiKueue). That is how Kueue spreads one queue across several clusters without the submitter knowing which cluster runs the job.

Exercises on a real cluster

Run these in order:

Install Kueue, define one ResourceFlavor and one ClusterQueue, and submit a job through a LocalQueue.
Submit more jobs than the quota allows and watch them queue as Workloads instead of stranding pods.
Put two ClusterQueues in a cohort with borrowing and confirm an idle queue's quota flows to a busy one.
Set priorities and trigger a reclaim: submit to the lender and watch the borrower get preempted.
Inspect kubectl get workloads and read the admission and preemption conditions.

When a job will not run

Job stuck Suspended. Kueue has not admitted the Workload. Check the Workload's conditions: kubectl get workloads -n <ns> and describe it. Usually quota is unavailable or an admission check is pending.
Never gets quota. The ClusterQueue's nominalQuota for the requested flavor is smaller than the job's request, or the request maps to no flavor. A single pod requesting more than the whole nominal quota will never fit.
Borrowing does nothing. borrowingLimit is set but the ClusterQueue has no cohortName, so there is nothing to borrow from. Borrowing requires a cohort.
Flavor matches no nodes. The ResourceFlavor's nodeLabels do not match any node, so admitted pods still cannot schedule. Kueue quota and actual node capacity are separate; keep them consistent.
Preemption thrash. Aggressive priorities plus borrowing can cause repeated preemptions. Fair sharing dampens this, but check the preemption reasons on Workloads if jobs keep getting evicted.

Why atomic admission is the point

The reason Kueue matters for distributed training is that it admits a Workload atomically. The default scheduler decides pod by pod, which is why a JobSet or a multi-node job can end up with half its pods running and half Pending, each half holding GPUs the other needs, deadlocked. Kueue reserves quota for the entire Workload before any pod exists, so either the whole job has room or none of it starts. That is gang admission at the quota layer, and it composes with gang scheduling at the placement layer, covered in the gang scheduling post. Together they close both halves of the all-or-nothing problem: enough quota to run, and enough nodes to place.

So the whole design comes back to one property. Jobs wait as Workloads, quota belongs to ClusterQueues and flows through cohorts, and admission stays atomic so distributed jobs never deadlock half-started. Three quota numbers and a preemption policy are the levers, and cohorts and fair sharing are the guardrails. Pair it with gang scheduling to cover both quota and placement for large jobs.