Gang Scheduling in Kubernetes 1.35 and 1.36: The Road to All-or-Nothing

A distributed training job needs all its pods or none. If the scheduler places half a gang and the rest cannot fit, those pods sit idle holding GPUs while the job makes no progress, and other jobs starve behind them. At its extreme, two half-placed jobs deadlock, each holding what the other needs. Gang scheduling prevents that: it admits the whole group together or not at all.

Three mechanisms give you that all-or-nothing behaviour today: the scheduler-plugins coscheduling plugin (out-of-tree, a PodGroup CRD), Kueue (admission at the quota layer), and the upstream Workload API (in-tree, still alpha). This post traces how each matured across Kubernetes 1.35 and 1.36, with every feature dated to the release that introduced it. The upstream effort is KEP-4671 (Workload Aware Scheduling) under the scheduling.k8s.io group; the out-of-tree option is scheduler-plugins coscheduling (PodGroup, scheduling.x-k8s.io/v1alpha1), and Kueue uses kueue.x-k8s.io/v1beta2 (1.36 release).

Two facts shape everything below. The upstream API moved from scheduling.k8s.io/v1alpha1 to v1alpha2 in 1.36, so a manifest written for 1.35 does not apply on 1.36. And upstream gang scheduling is alpha in both releases, behind a feature gate and not production-ready, so today you reach for Kueue or the coscheduling plugin and watch the upstream API as it moves toward beta.

This builds on the rest of the series: JobSet and Kueue for the workload side, scheduler plugins for the mechanism, and DRA for the devices a gang competes for.

Why a half-placed job deadlocks

The default scheduler places one pod at a time and greedily. For most workloads that is correct. For a tightly coupled distributed job it is a trap. Say a training job needs eight pods, each on an 8-GPU node. The scheduler places five, then the cluster fills up. Those five pods are running, holding forty GPUs, and doing nothing, because the job cannot start until all eight are up. Worse, another job that could have used those GPUs is stuck behind them. This is resource fragmentation, and at its extreme, two half-placed jobs can deadlock, each holding what the other needs.

Gang scheduling fixes this by making the group the unit of admission. Either all eight pods can be placed, in which case they all start, or none are, in which case none hold anything. The job either runs or waits cleanly. The cost is latency: a large gang may wait longer for enough room to open up, and if the timeout is too short it thrashes between waiting and rejection. That trade, a little scheduling latency for no fragmentation, is almost always worth it for expensive GPU jobs.

Terms this post uses

• Gang: a set of pods that must be scheduled together, all or nothing.
• PodGroup: the object that names a gang and its minimum member count.
• Workload: the upstream in-tree representation of a schedulable group.
• Permit: the scheduler extension point where a pod can wait for its gang before being bound.
• Coscheduling: the scheduler-plugins implementation of gang scheduling using PodGroup.

Copycopy code to clipboard
8 pods want to run          gang admission
        │                           │
   place 5, block?  ── no ──▶  place all 8 together (all-or-nothing)
        │
       yes (old greedy way) ──▶ 5 idle pods hold GPUs, job stuck

Caption: gang admission places the whole group or nothing, avoiding idle-but-held resources.

How pods wait at Permit

All three mechanisms share the same core idea, implemented at the scheduler's Permit extension point (see the scheduler plugins post). As each pod of a gang reaches Permit, it does not bind immediately. It waits until enough of its group has also reached Permit, then they are all released together. If the group does not assemble within a timeout, the waiting pods are rejected and retried later.

Copycopy code to clipboard
pod 1 reaches Permit ──▶ wait
pod 2 reaches Permit ──▶ wait
...
pod N reaches Permit ──▶ group complete ──▶ release all N to bind
        │
   timeout before N ──▶ reject waiting pods, retry later

Caption: pods wait at Permit until the gang is complete, then bind together, or time out.

The difference between the mechanisms is where the group is defined and counted, not the Permit mechanic. That is worth internalizing: coscheduling, Kueue, and the upstream API are three front ends to the same "wait at Permit until the group is ready" idea, differing in what defines the group (a PodGroup CRD, a Kueue Workload, or a native Workload object) and what else they layer on (quota, preemption, priority).

The three mechanisms

scheduler-plugins coscheduling is the long-standing out-of-tree answer. You install the scheduler-plugins build, define a PodGroup (scheduling.x-k8s.io/v1alpha1) with a minMember, and label pods to join it. The coscheduling plugin holds pods at Permit until minMember are ready. It works on any recent Kubernetes version because it is a plugin, not a core feature, which is also its cost: you run a custom scheduler build and re-qualify it on every upgrade.

Copycopy code to clipboard
apiVersion: scheduling.x-k8s.io/v1alpha1
kind: PodGroup
metadata: { name: training-gang }
spec:
  minMember: 8

Notice minMember is the gang size the plugin waits for before releasing any pod.

Pods join the group with a label, and they must also target the scheduler that runs the coscheduling plugin. A Job template makes this concrete:

Copycopy code to clipboard
spec:
  template:
    metadata:
      labels:
        scheduling.x-k8s.io/pod-group: training-gang
    spec:
      schedulerName: scheduler-plugins-scheduler
      containers:
        - name: trainer
          resources:
            limits: { nvidia.com/gpu: "8" }

Notice two things: the scheduling.x-k8s.io/pod-group label ties the pod to the PodGroup, and schedulerName points it at the scheduler build that runs the coscheduling plugin. Both are required, or the pod is scheduled by the default scheduler with no gang behaviour.

Kueue provides gang behaviour at the quota layer, as covered in the JobSet post. It represents a whole workload (a JobSet, a Job, a set of pods) as one Workload, its unit of admission, and admits it only when the quota can fit the entire thing. Until then the workload is suspended and holds nothing. This is all-or-nothing admission, plus quota, fair sharing, and preemption. For most teams running batch or training on shared clusters, Kueue is the practical choice today, because you almost always want the quota model anyway.

Upstream Workload API is the in-tree effort to make gang scheduling a native scheduler feature, so you do not need an out-of-tree build. It is still alpha as of 1.36 and is covered next.

What the upstream API did between releases

The upstream API changed between the two releases. This table is what existed and its stage per version.

The change that bites: in 1.36, the scheduling.k8s.io/v1alpha1 Workload API was removed and v1alpha2 introduced, so any object stored under the old version had to be recreated before upgrading (CHANGELOG-1.36). This is normal for an alpha API, and it is exactly why you do not build production workflows on one: the group, version, and gate names move between releases.

The lesson if you follow alpha features: do not copy a 1.35 manifest onto a 1.36 cluster; check the API version for your release first.

Which to use today

As of 1.36, the upstream gang scheduling is alpha, which means test clusters only, behind a feature gate, with an API that is still changing. For production:

• Running JobSet or batch Jobs on shared GPU: use Kueue. It gives gang admission plus quota and preemption, which you need anyway.
• Need gang scheduling without Kueue's quota model: use the scheduler-plugins coscheduling plugin.
• Building toward the native feature: pilot the upstream Workload API on a test cluster behind its feature gate, and plan to adopt it as it graduates to beta in a later release.

Do not stack all three. Pick the layer that owns admission for your workloads. Kueue and coscheduling can coexist if they govern different workloads, but two gang mechanisms fighting over the same pods causes confusing waits, where a pod is held by one layer for a reason the other cannot see.

Traps that waste GPU hours

• Version-skewed manifests. A scheduling.k8s.io/v1alpha1 object fails on 1.36. Match the manifest to the cluster version using the table.
• Gang timeout too short. If the group cannot assemble within the Permit timeout, pods are rejected and retried, which can loop under contention. Tune the timeout to your realistic startup time, including image pull and device preparation.
• minMember mismatch. With coscheduling, a minMember that does not match the real pod count means the gang never completes, or admits too early with missing members. Keep it in sync with the workload's pod count.
• Two mechanisms at once. Kueue admitting a workload while coscheduling also gates its pods leads to double waiting. Let one layer own admission.
• Preemption of gangs. Preempting one pod of a running gang can break the whole job, since the collective loses a member. Gang-aware preemption is part of the upstream Workload Aware Scheduling effort but is not production-ready yet; on today's setups, protect running gangs with priority and PodDisruptionBudgets.

Where gang scheduling is still unfinished

Gang scheduling is converging but not finished. The open frontiers as of 1.36: the upstream native API is still alpha and changing between releases; gang-aware preemption is being designed under Workload Aware Scheduling; topology-aware gang placement (keeping a gang on one rack or fast fabric) is not fully native; the interaction between gangs and DRA (a gang competing for DRA devices, see the DRA driver post) is still being worked out; and cross-namespace gangs are limited. The next releases are expected to move the upstream API toward beta and consolidate its feature gates; track KEP-4671 and the Workload Aware Scheduling KEPs for how these resolve.

Try it on a test cluster

1. Install scheduler-plugins, create a PodGroup with minMember, and watch pods wait at Permit until the gang completes.
2. Shrink the cluster below the gang size and confirm no pods bind, rather than a half-placed job.
3. Run the same job through Kueue and compare gang admission plus quota behaviour.
4. On a 1.36 test cluster, enable the upstream gang feature gate and try the v1alpha2 Workload API; note it is alpha.
5. Inspect a waiting pod's events to see the Permit wait and the gang timeout in action.

Where this leaves you

Gang scheduling turns a distributed job into a unit that is admitted whole or not at all, which is what keeps GPU clusters from fragmenting and deadlocking. Across 1.35 and 1.36 the upstream API stayed alpha and changed its group version, while Kueue and the coscheduling plugin remained the production-ready options. Use Kueue or coscheduling today, and pilot the upstream feature as it stabilizes.

This concludes the DRA and scheduling series. Start at Writing a DRA driver for the device foundation.

Further reading

• Scheduling framework
• scheduler-plugins coscheduling
• KEP-4671: Workload Aware Scheduling
• Kueue documentation, Kueue workload concept
• Kueue: run a JobSet
• 1.36 release, CHANGELOG-1.36
• JobSet and Kueue, scheduler plugins