Skip to content
Yuvraj 🧢
Github - yindiaContact

Kueue Deep Dive: Job Queueing and Quotas for Kubernetes

— kubernetes, kueue, batch, scheduling, ml-infra — 13 min read

Kubernetes has a scheduler, but it has no queue. Submit more jobs than the cluster can run and it does not line them up; it admits them all, and their pods sit Pending forever, half-scheduled, holding nothing and finishing nothing. For batch and machine-learning (ML) workloads on shared GPUs, that is the whole problem. Kueue is the project that adds the missing layer: quota, queueing, and fair sharing on top of the scheduler. This post is a deep dive into how it works. It targets Kubernetes 1.37 and the Kueue v1beta2 API, which is the only served version from Kueue v0.20.0 on (v0.20.0 upgrade notes).

This builds on two earlier posts: gang scheduling for why all-or-nothing admission matters, and JobSet for the job shape Kueue admits.

The object shapes here follow the Kueue v1beta2 reference and concept docs (Kueue concepts, ClusterQueue, admission). What follows has a short spine. A job targets a namespaced LocalQueue that points at a cluster-scoped ClusterQueue holding the quota, and Kueue tracks each job internally as a Workload. It admits that Workload in two phases: it reserves quota first (including physical placement when topology-aware scheduling is on), then runs admission checks. Nothing runs until both pass. ClusterQueues grouped in a Cohort borrow unused quota from each other within borrowingLimit and lendingLimit, and reclaim it through preemption. Because a Workload is admitted atomically, a distributed job gets gang admission instead of starting half its pods.

No queue behind the scheduler

The default scheduler answers one question: where does this pod go? It does not answer: should this job start now, or wait? So when a team submits ten training jobs to a cluster that can run three, Kubernetes creates all ten jobs' pods. The scheduler places what fits and leaves the rest Pending. Now the cluster is full of jobs that each have some pods running and some pending, none of them able to finish, all of them holding partial resources. There is no notion of "this job waits until that job frees its GPUs," and no notion of "team A gets half the cluster, team B the other half." Kueue adds both.

The objects that build a queue

A working queue is not one object but six that reference each other.

  • ResourceFlavor: a description of a kind of node, for example A100 GPUs, tied to nodes by labels, with optional taints and tolerations. Quota is counted per flavor.
  • ClusterQueue: a cluster-scoped pool of quota with sharing and preemption rules. This is where admins set how much each team gets.
  • LocalQueue: a namespaced pointer to a ClusterQueue. Jobs reference a LocalQueue, not the ClusterQueue directly.
  • Workload: the internal object Kueue creates for each job, capturing its total resource request. Kueue queues and admits Workloads, not pods.
  • Cohort: a cluster-scoped object that groups ClusterQueues so they can share unused quota. It can also hold shared quota of its own, and cohorts can nest.
  • AdmissionCheck: an extra gate that must pass, after quota is reserved, before a Workload is admitted.

Admission in two phases

A Kueue-managed job starts life suspended. Kueue creates a Workload for it, queues that Workload, and only unsuspends the job once it has reserved quota and cleared every admission check.

Job (suspend: true) ─▶ Kueue creates a Workload
                            │
                            ▼
                   queue in the LocalQueue ─▶ ClusterQueue
                            │
   phase 1: reserve quota  ├─ logical: does the quota fit (borrow from cohort if needed)?
                            └─ physical: does it place (Topology-Aware Scheduling, if on)?
                            │
   phase 2: admission checks ─ run concurrently (MultiKueue, ProvisioningRequest, plugins)
                            │
     all checks Ready? ── yes ──▶ admit: unsuspend the Job
                            │
                            no ──▶ stay queued

Caption: Kueue's two-phase admission. Quota reservation covers both logical fit and physical placement; the job's pods never exist until the Workload is admitted.

The two-phase shape is the key design choice. Phase one reserves quota: is there enough, and when topology-aware scheduling is enabled, does it physically place on the chosen topology? Phase two runs the admission checks. The Workload is admitted only when every AdmissionCheckState is Ready (admission). Because the job is suspended until all of that passes, Kueue never leaves a job half-started, and gang admission falls out for free: the whole Workload is admitted at once or not at all.

Quota for one team, in four objects

The minimum setup is four objects. First, a ResourceFlavor describing the GPU nodes.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata: { name: gpu-a100 }
spec:
  nodeLabels:
    accelerator: nvidia-a100

Notice nodeLabels ties this flavor to a subset of nodes, so quota is tracked separately per hardware type rather than as one undifferentiated pool. When Kueue admits a Workload against this flavor, it copies these labels into the pod template's nodeSelector so the pods land on the right nodes (ResourceFlavor).

Then a ClusterQueue that owns the quota, in a cohort so it can share.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata: { name: team-a }
spec:
  namespaceSelector: {}
  cohortName: research
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: gpu-a100
          resources:
            - name: cpu
              nominalQuota: 100
            - name: memory
              nominalQuota: 400Gi
            - name: "nvidia.com/gpu"
              nominalQuota: 8
              borrowingLimit: 8
              lendingLimit: 4

Notice nominalQuota: 8 is what team A is guaranteed, borrowingLimit: 8 lets it use up to 8 more GPUs borrowed from the cohort when they are idle, and lendingLimit: 4 caps how many of its own it will lend out. Borrowing only exists because of cohortName: research. These limits work without fair sharing; fair sharing changes ordering, not whether borrowing is allowed.

Then a namespaced LocalQueue that jobs point at, and a Job that targets it.

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata: { name: team-a, namespace: team-a }
spec:
  clusterQueue: team-a
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: team-a
spec:
  suspend: true
  # ... pod template requesting nvidia.com/gpu ...

Notice the kueue.x-k8s.io/queue-name label is how the job selects its LocalQueue, and suspend: true hands start control to Kueue. Kueue flips suspend to false when it admits the Workload.

How idle quota moves through a cohort

The three quota numbers are the whole economy. nominalQuota is guaranteed. lendingLimit bounds how much a queue will give away when idle (stable since Kueue v0.17). borrowingLimit bounds how much it can take from the cohort's idle capacity on top of its nominal. Leave borrowingLimit unset and a queue can borrow the sum of every other queue's nominal quota; leave lendingLimit unset and a queue lends all of its idle quota.

Cohort: research
 ┌───────────────────────────────────────────┐
 │ ClusterQueue team-a   nominal 8, lend ≤ 4  │
 │ ClusterQueue team-b   nominal 8, lend ≤ 4  │
 └───────────────────────────────────────────┘
   team-b idle ─▶ team-a borrows up to +8 (its borrowingLimit)
   team-b returns ─▶ team-a's borrowed GPUs are reclaimed

Caption: within a cohort, idle quota flows to busy queues and is reclaimed when the owner needs it.

The point of the two limits is bounded generosity. Without a borrowingLimit, a busy queue could consume the whole cohort's idle capacity; without a lendingLimit, a queue could lend so much that its own jobs wait when they arrive.

The Cohort object and nesting

In v1beta2 a cohort is a real object, not just a string. The simplest form is empty and exists only so ClusterQueues can share:

apiVersion: kueue.x-k8s.io/v1beta2
kind: Cohort
metadata: { name: research }

A Cohort can also hold quota of its own. Quota defined on the Cohort is an extra shared pool on top of what its ClusterQueues own, and any ClusterQueue that wants to draw from it must still declare the resource and flavor, even at nominalQuota: 0 (Cohort).

apiVersion: kueue.x-k8s.io/v1beta2
kind: Cohort
metadata: { name: research }
spec:
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: gpu-a100
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 16     # shared pool for the whole cohort

Notice this models a shared pool plus per-team floors: each team keeps its own nominalQuota, and the 16 GPUs on the cohort are the shared top-up.

Cohorts nest through parentName, forming a tree Kueue calls a CohortTree. ClusterQueues anywhere in the tree can draw on quota within it, bounded by the borrowing and lending limits at each level. This is how you model an org: a root pool, then per-department sub-pools.

root-cohort
 ├── important-org  (fairSharing.weight 0.75)
 └── regular-org    (fairSharing.weight 0.25)

Caption: a two-level CohortTree. With fair sharing on, the weights steer each branch toward 75% and 25% of shared resources.

A cycle in parentName references disables every affected cohort member until you remove it, so keep the tree acyclic.

Who goes first: queueing strategy

When more Workloads are pending than quota allows, .spec.queueingStrategy on the ClusterQueue decides ordering. Both strategies order by priority first, then creation time. They differ on head-of-line blocking.

  • BestEffortFIFO (default): a Workload that cannot be admitted does not block newer Workloads that fit. Throughput stays high; a large job can wait while small ones pass it.
  • StrictFIFO: an older Workload that cannot be admitted blocks everything behind it, even Workloads that would fit. Use it when submission order must be respected.

Taking borrowed quota back

Borrowing raises an obvious question: what happens when the lender wants its quota back? Preemption. The ClusterQueue's preemption block has three knobs (ClusterQueue preemption).

  • withinClusterQueue: whether a pending Workload can preempt admitted Workloads in the same ClusterQueue. Never (default), LowerPriority, or LowerOrNewerEqualPriority.
  • reclaimWithinCohort: whether a pending Workload can reclaim quota from other ClusterQueues in the cohort that are over their nominal quota. Never (default), LowerPriority, or Any.
  • borrowWithinCohort: whether a Workload that itself needs to borrow may preempt to do so. Classic preemption only; it cannot combine with fair sharing.
spec:
  preemption:
    withinClusterQueue: LowerPriority
    reclaimWithinCohort: Any

Notice reclaimWithinCohort: Any lets a team below its nominal quota reclaim its share from a borrower regardless of the borrower's priority, which is what guarantees each team its floor.

Fair sharing changes the ordering so history matters: Workloads from a LocalQueue that has consumed less over time are preferred, so a team that has been quiet is not permanently behind a team that floods the queue. Kueue's fair-sharing preemption carries a proof that two Workloads in different ClusterQueues cannot enter a loop preempting each other, because each preemption strictly lowers the preemptor's dominant resource share (fair sharing).

When a flavor is full: fungibility

A ClusterQueue can list several flavors for one resource, say spot then on-demand GPUs. flavorFungibility decides whether Kueue settles for the first flavor it can use or keeps looking for a better one.

  • whenCanBorrow: MayStopSearch (default) takes the current flavor if borrowing makes it fit; TryNextFlavor keeps looking.
  • whenCanPreempt: TryNextFlavor (default) prefers a flavor that needs no preemption; MayStopSearch preempts in the current flavor rather than moving on.

A flavor that fits with neither borrowing nor preemption always wins, whatever these are set to. The knobs only decide the tie-break between borrowing here and trying the next flavor.

Pausing a queue

stopPolicy lets an admin freeze a ClusterQueue without deleting it. Hold stops admitting new Workloads but lets running ones finish; HoldAndDrain also evicts the admitted Workloads. Set it back to None to resume. This is the clean lever for draining a queue before maintenance.

Extra gates: admission checks

An AdmissionCheck is a second gate that runs after quota is reserved. All checks configured on a ClusterQueue run concurrently, and the Workload is admitted only when every one reports Ready (admission check). Two built-in checks matter most.

ProvisioningRequest asks the cluster autoscaler to stand up nodes before the Workload runs, so a job that reserved quota does not then sit pending for want of machines. If provisioning fails for a transient reason, Kueue releases the reserved quota, requeues the Workload, and retries with backoff.

MultiKueue spreads one queue across clusters. A manager cluster holds ClusterQueues whose quota mirrors the total across worker clusters, and a MultiKueue check dispatches an admitted Workload to a worker cluster that acts as a standalone Kueue (MultiKueue). The submitter never picks a cluster.

Keeping the group together: topology-aware scheduling

Quota says a job can run; it does not say the pods land close to each other. For collective communication that matters: a training job split across two racks is slower than one on a single rack. Topology-aware scheduling (TAS, beta and enabled by default since Kueue v0.14) folds physical placement into the quota-reservation phase, so Kueue only reserves quota it can actually place on the requested topology, for example all pods within one rack or block (TAS). This is the quota-layer counterpart to scheduler-level gang placement.

When admission fails

Reserving quota is not a promise the job runs. If an admission check fails, Kueue handles it by cause (admission).

  • Transient (a cloud capacity shortage during a ProvisioningRequest): Kueue releases the reserved quota at once, requeues the Workload, and retries with exponential backoff. Quota is never held while waiting on a retry.
  • Permanent: Kueue marks the check Rejected, evicts the Workload, releases quota, and deactivates it. A deactivated Workload does not requeue on its own; a user sets .status.active back to true to try again.

Exercises on a real cluster

Run these in order:

  1. Install Kueue, define one ResourceFlavor and one ClusterQueue, and submit a job through a LocalQueue.
  2. Submit more jobs than the quota allows and watch them queue as Workloads instead of stranding pods.
  3. Put two ClusterQueues in a cohort with borrowing and confirm an idle queue's quota flows to a busy one.
  4. Set priorities and reclaimWithinCohort, submit to the lender, and watch the borrower get preempted.
  5. Set stopPolicy: Hold and confirm new Workloads stop admitting while running ones finish.
  6. Inspect kubectl get workloads and read the admission and preemption conditions.

When a job will not run

  • Job stuck Suspended. Kueue has not admitted the Workload. Check its conditions: kubectl get workloads -n <ns> and describe it. Usually quota is unavailable or an admission check is pending.
  • Workload deactivated. A permanent admission-check failure deactivates it. It will not requeue until you set .status.active: true.
  • Never gets quota. The ClusterQueue's nominalQuota for the requested flavor is smaller than the job's request, or the request maps to no flavor. A single pod requesting more than the whole nominal quota will never fit.
  • Borrowing does nothing. borrowingLimit is set but the ClusterQueue has no cohortName, so there is nothing to borrow from. Borrowing requires a cohort.
  • Flavor matches no nodes. The ResourceFlavor's nodeLabels do not match any node, so admitted pods still cannot schedule. Kueue quota and actual node capacity are separate; keep them consistent.
  • Preemption thrash. Aggressive priorities plus borrowing can cause repeated preemptions. Fair sharing dampens this, but check the preemption reasons on Workloads if jobs keep getting evicted.

Why atomic admission is the point

The reason Kueue matters for distributed training is that it admits a Workload atomically. The default scheduler decides pod by pod, which is why a JobSet or a multi-node job can end up with half its pods running and half Pending, each half holding GPUs the other needs, deadlocked. Kueue reserves quota for the entire Workload before any pod exists, so either the whole job has room or none of it starts. That is gang admission at the quota layer, and it composes with gang scheduling at the placement layer, covered in the gang scheduling post. Together they close both halves of the all-or-nothing problem: enough quota to run, and enough nodes to place.

So the whole design comes back to one property. Jobs wait as Workloads, quota belongs to ClusterQueues and flows through cohorts, and admission stays atomic so distributed jobs never deadlock half-started. Three quota numbers and a preemption policy are the levers, and cohorts and fair sharing are the guardrails. Pair it with gang scheduling to cover both quota and placement for large jobs.

Sources