JobSet for Distributed Training: Groups, Restarts, and Kueue Quota

30.06.2026 — kubernetes, jobset, kueue, distributed-training, gang-scheduling — 9 min read

Distributed training runs many pods that must find each other, start together, and be treated as one job. A plain Kubernetes Job does not model that: it has one pod template and no notion of a launcher plus workers, or of several jobs that live and die together. JobSet does. This post shows how to run a multi-node training job with JobSet, how its restart policies work, and how Kueue adds the quota and all-or-nothing admission that training on shared clusters needs.

The plan is to build one distributed training job from the ground up and watch what each piece buys you. A JobSet groups one or more ReplicatedJobs, each a Job template with a replica count, and manages their lifecycle together. It gives every pod a stable DNS hostname through an auto-managed headless Service so training frameworks can rendezvous. Its failurePolicy, startupPolicy, and successPolicy control when the whole set restarts, in what order it starts, and when it is done. What it does not do is gang-schedule: that falls to Kueue, which treats the JobSet as one Workload and admits it all-or-nothing against a quota.

This connects to two other posts: LeaderWorkerSet solves the same "group of pods as a unit" problem for inference, and a companion post covers the upstream gang-scheduling API that Kueue and JobSet build on.

This post targets Kubernetes 1.36, the JobSet API jobset.x-k8s.io/v1alpha2, and the Kueue API kueue.x-k8s.io/v1beta2. JobSet and Kueue are independent operators with their own release cadence, so pin the exact chart versions current on your cluster rather than assuming they track the Kubernetes minor version (JobSet docs, Kueue docs).

What a plain Job cannot express

A training job on N nodes is not N independent pods. Every worker needs a stable address to reach its peers for collective communication, all workers must be running before training starts, and if one worker dies the job usually has to restart as a whole because the collective is broken. A Kubernetes Job cannot express any of this: it has a single pod template, no per-worker identity beyond an index, and no way to bind several jobs into one unit.

Teams worked around this with hand-written operators, StatefulSets, and scripts. JobSet is the shared answer: a single object that creates a group of Jobs, wires up their networking, and manages their combined lifecycle (JobSet overview).

The building blocks you get

JobSet: the top-level object that owns a group of Jobs and their shared lifecycle.
ReplicatedJob: a named Job template with a replicas count; a JobSet has one or more.
Job and Pod defaults: JobSet defaults each Job's completionMode to Indexed and each pod's restartPolicy to OnFailure.
Headless Service: auto-created so pods get stable DNS names.
Coordinator: an optional designated pod (a specific replicatedJob, job index, and pod index) that others treat as the rendezvous point.

JobSet
 ├── ReplicatedJob "leader"  (replicas: 1) ──▶ Job ──▶ Pod (coordinator)
 └── ReplicatedJob "worker"  (replicas: 4) ──▶ Jobs ─▶ Pods
        (all share one headless Service and DNS subdomain)

Caption: a JobSet composes named ReplicatedJobs, each expanding to Jobs and Pods under one Service.

Giving every pod an address

The hard part of multi-node training is rendezvous. JobSet solves it with stable hostnames. Set enableDNSHostnames: true and JobSet creates a headless Service and gives each pod a predictable DNS name (JobSet concepts).

The hostname pattern is <jobset>-<replicatedJob>-<jobIndex>-<podIndex>.<subdomain>. So worker 0 of a JobSet named train reaches worker 2 at a name it can compute without any discovery service.

train-worker-0-0.train   ──▶  train-worker-0-2.train
   (pod index 0)                 (pod index 2)
   resolved via the auto-created headless Service

Caption: JobSet gives each pod a computable, stable DNS name for peer rendezvous.

Wiring the leader and workers together

Here is the shape of a two-group JobSet: a single leader (the rendezvous coordinator) and four workers. Networking is enabled so all pods share a DNS subdomain.

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: train
spec:
  network:
    enableDNSHostnames: true      # stable pod hostnames
  replicatedJobs:
    - name: leader
      replicas: 1
    - name: worker
      replicas: 4

Notice replicatedJobs is a list, each with its own replicas. The leader and workers are separate ReplicatedJobs so they can use different pod templates and counts.

Each ReplicatedJob carries a normal Job template. The worker template requests GPUs and runs the training entrypoint, which uses the DNS names to join the group.

- name: worker
      replicas: 4
      template:
        spec:
          parallelism: 1
          completions: 1
          template:
            spec:
              containers:
                - name: trainer
                  image: my-trainer:1.0
                  resources:
                    limits: { nvidia.com/gpu: "8" }

Notice each worker Job here is a single pod (parallelism: 1), so the JobSet has one leader plus four worker pods, five nodes worth of GPUs acting as one training run. You size replicas and parallelism to your topology.

Deciding when the group restarts, starts, and finishes

Three policies control the group's lifecycle. This is where JobSet earns its place over a plain Job.

Policy	Field	Controls
Failure	`failurePolicy`	What happens when a child Job fails: restart the whole set, fail, or apply per-rule actions
Startup	`startupPolicy`	Whether ReplicatedJobs start in order (`InOrder`) or all at once
Success	`successPolicy`	When the JobSet counts as complete (all, or a chosen subset)

The failure policy is rule-based. Rules are evaluated in order and the first match wins; if none match, the default action restarts the whole JobSet and counts toward maxRestarts (failure policy).

spec:
  failurePolicy:
    maxRestarts: 3

Notice maxRestarts bounds how many times the set restarts before it is declared failed. Without a bound, a job that fails fast would restart forever.

The restart lifecycle is a small state machine.

[running] ──child Job fails──▶ [evaluate failurePolicy rules]
        ▲                               │
        │                        first match / default
        │                               ▼
   [restart whole JobSet] ◀── under maxRestarts ── [restart?]
                                        │
                                 over maxRestarts
                                        ▼
                                   [failed]

Caption: a child failure evaluates the failure policy, and restarts the whole set until maxRestarts is hit.

Startup policy matters when the leader must be ready before workers start. startupPolicyOrder: InOrder starts ReplicatedJobs in list order, so the coordinator comes up first. Success policy lets you declare the run complete when a chosen ReplicatedJob finishes, rather than requiring every job to succeed.

Why JobSet will not pack the cluster for you

This is the point people get wrong, so be precise. JobSet groups pods and manages their lifecycle, but it does not gang-schedule. It will happily create all the pods even if the cluster can only fit some of them, which on a busy GPU cluster leads to a half-placed job holding GPUs while it waits, the classic resource-fragmentation deadlock.

All-or-nothing admission comes from Kueue. Kueue represents the whole JobSet as a single Workload, its unit of admission, and admits it only when the quota can fit the entire thing. Until then the JobSet is suspended and holds nothing. This is gang behaviour at the quota layer (Kueue workload). A companion post covers the separate upstream scheduler-level gang API and how it relates.

Letting Kueue gate on quota

Kueue uses three objects to manage quota, then you point the JobSet at a queue. First the admin defines the quota pool.

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata: { name: gpu-cq }
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: default
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 40

Notice the ClusterQueue is cluster-scoped and defines the GPU quota. It admits a Workload only if the whole thing fits within nominalQuota, which is what makes admission all-or-nothing.

A LocalQueue in the team's namespace maps to that ClusterQueue.

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata: { name: team-lq, namespace: team-a }
spec: { clusterQueue: gpu-cq }

Notice LocalQueue is namespaced; it is what users submit to, and it routes to the shared ClusterQueue.

Finally, label the JobSet with the queue name. Kueue suspends it on creation and admits it when the quota fits.

metadata:
  name: train
  labels:
    kueue.x-k8s.io/queue-name: team-lq

Notice this one label is the whole integration. Kueue watches for it, gates the JobSet on quota, and preempts lower-priority workloads if configured. Without it, the JobSet runs immediately with no quota control.

Where this bites

Half-placed job without Kueue. JobSet alone creates all pods; if the cluster cannot fit them, some pend while others hold GPUs. Use Kueue for gang admission on shared clusters.
DNS readiness race. Workers may resolve peers before those pods are ready. Publishing not-ready addresses (a network option) and retrying the rendezvous in the trainer avoids startup flakes.
Worker restart without a full restart. If a worker dies and only it restarts, the collective is broken and the run hangs. Set a failure policy that restarts the whole set for tightly coupled training.
Quota starvation. A large JobSet can wait indefinitely if its quota is always partly used by smaller jobs. Kueue preemption and fair sharing address this; without them, big gangs starve.
maxRestarts too low or unset. A fast-failing container either burns all restarts immediately or, if unbounded, loops forever. Set a sensible maxRestarts.

The edges that are still rough

JobSet and Kueue split the work cleanly: JobSet owns lifecycle and networking, Kueue owns quota and gang admission. The remaining rough edges are topology-aware placement (keeping a group on one rack or fast fabric) and the interaction with the upstream scheduler-level gang API, which is evolving separately. A companion post traces that evolution and explains which gang mechanism to use today.

Put it together on a cluster

Deploy the JobSet operator and apply the leader-plus-workers manifest; watch pods get stable DNS names.
Kill a worker pod and observe the failure policy restart the whole set.
Install Kueue, create a ClusterQueue and LocalQueue, and label the JobSet; watch it stay suspended until quota fits.
Shrink the quota below the job's need and confirm the JobSet holds nothing while it waits.
Set startupPolicyOrder: InOrder and confirm the leader starts before the workers.

Two operators, one training run

Built up from a plain Job, the distributed run is now one object: grouped ReplicatedJobs, stable hostnames, and lifecycle policies that restart the set as a unit. It deliberately leaves quota and gang admission to Kueue, which admits the whole JobSet or none of it. Use both together on shared GPU clusters. Companion posts go inside the scheduler framework and cover gang scheduling in depth.