Writing a DRA Driver: From ResourceSlice to kubelet Plugin

Dynamic Resource Allocation (DRA) is how Kubernetes now handles GPUs and other specialized devices. Instead of a fixed integer count like nvidia.com/gpu: 1, a driver publishes devices with attributes, and a user claims them by describing what they need. This post builds a DRA driver end to end: the objects involved, the allocation flow, and the actual Go code that publishes devices and prepares them for a pod. The code is adapted from the upstream reference driver and verified against its source; it is not executed in this post.

This is part of a DRA series. A companion post compares DRA with the older device plugins, and another covers gang scheduling, which often runs alongside DRA for AI jobs.

Here is the shape of the work before the first line of code. A DRA driver does two things: it publishes devices as ResourceSlices, and it prepares allocated devices on the node through a kubelet plugin. The scheduler allocates devices using structured parameters (device attributes plus CEL selectors) without ever calling the driver; the driver is only called on the node, to prepare the device. You implement a small Go interface (PrepareResourceClaims, UnprepareResourceClaims) and let kubeletplugin.Start wrap it into the gRPC service the kubelet expects. Device access reaches the container through CDI (Container Device Interface): the driver returns CDI device IDs, and the runtime injects the device.

This post targets Kubernetes 1.35, the resource.k8s.io/v1 API (GA since 1.34), and the DRA modules at v0.35.0. The core DRA API graduated to GA in 1.34 and the DynamicResourceAllocation feature gate was locked on in 1.35, so there is nothing to enable (1.34 release, KEP-4381). Because the driver library and the resource.k8s.io/v1 types travel with the Kubernetes minor version, keep the driver's module versions in lockstep with the cluster you deploy to.

What you are building and why

Before DRA, a device was a counted resource. A pod asked for nvidia.com/gpu: 2 and the scheduler found a node with two free. That works when GPUs are interchangeable. It falls apart when they are not. You cannot say "a GPU with at least 40 GB, connected to this NIC, that I can share with another pod." The device plugin API had no vocabulary for attributes, topology, or sharing, and the scheduler could not reason about any of it.

DRA fixes this by making devices first-class objects with attributes, and by letting the scheduler match a request against those attributes. A driver describes what each device is; a user describes what they need; the scheduler does the matching. The rest of this post is how a driver participates in that.

The objects you will wire together

Five objects and one interface. Each is one sentence.

• DeviceClass: a category of devices and how to select them, written by the driver or admin.
• ResourceSlice: the driver's advertisement of the devices on one node, with their attributes and capacity.
• ResourceClaim: a user's request for devices with the capabilities they need.
• ResourceClaimTemplate: a stamp that produces a per-pod ResourceClaim.
• DRA driver: your code, which publishes ResourceSlices and prepares allocated devices on the node.
• CDI (Container Device Interface): the standard the runtime uses to inject a device into a container.

Copycopy code to clipboard
DeviceClass ── selects ──▶ devices in ResourceSlices (per node, from the driver)
        ▲                                  │
        │ references                       │ scheduler matches
   ResourceClaim ◀── stamped by ── ResourceClaimTemplate
        │
   Pod (resourceClaims + resources.claims)

Caption: the driver publishes ResourceSlices; a claim references a DeviceClass; the pod references the claim.

Allocation in the scheduler, preparation on the node

The key idea is a split: allocation happens in the scheduler, preparation happens on the node. The scheduler never calls your driver.

Copycopy code to clipboard
1. Driver ──▶ publishes ResourceSlice (devices + attributes) to the API server
2. User   ──▶ creates a Pod + ResourceClaim(Template)
3. Scheduler ──▶ reads ResourceSlices, matches the claim with CEL, writes the
                 allocation onto the ResourceClaim, binds the Pod to a node
4. Kubelet (on that node) ──▶ calls the driver: NodePrepareResources
5. Driver ──▶ returns CDI device IDs; runtime injects the device; container starts

Caption: allocation is done by the scheduler from ResourceSlices; the driver is only called on the node to prepare devices.

For the kubelet to call your driver, the driver must register its socket. The kubeletplugin helper does this: it serves the DRA gRPC API on a socket under the kubelet plugins directory and registers it through the registrar directory.

Copycopy code to clipboard
driver pod starts
   │  kubeletplugin.Start serves gRPC on /var/lib/kubelet/plugins/<driver>/...
   ▼
registers via /var/lib/kubelet/plugins_registry/...
   ▼
kubelet discovers the plugin and sends NodePrepareResources when a claim lands

Caption: the kubelet plugin registration flow via the registrar and plugins directories.

Building the driver, piece by piece

All code below is adapted from the reference driver and matches its API at module v0.35.0. It is source-verified, not executed here. The kubeletplugin and resourceslice helpers keep the same shape across recent minor versions, so the structure carries forward; always build against your cluster's version.

The DeviceClass

The DeviceClass names your driver and gives a CEL (Common Expression Language) selector that matches its devices. This is what a claim references.

Copycopy code to clipboard
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example.com
spec:
  selectors:
    - cel:
        expression: "device.driver == 'gpu.example.com'"

Notice the selector is a CEL expression over device fields. The scheduler evaluates it against every device in the ResourceSlices to find candidates, with no call to your driver.

Starting the kubelet plugin

Your driver is a Go type that implements the DRA plugin interface. kubeletplugin.Start wraps it into the gRPC service the kubelet talks to. The options tell it the driver name, node name, and the two socket directories.

Copycopy code to clipboard
helper, err := kubeletplugin.Start(ctx, driver,
    kubeletplugin.KubeClient(config.coreclient),
    kubeletplugin.NodeName(config.flags.nodeName),
    kubeletplugin.DriverName(config.flags.driverName),        // "gpu.example.com"
    kubeletplugin.RegistrarDirectoryPath(config.flags.kubeletRegistrarDirectoryPath),
    kubeletplugin.PluginDataDirectoryPath(config.DriverPluginPath()),
    kubeletplugin.RollingUpdate(types.UID(config.flags.podUID)),
)

Notice RollingUpdate takes the pod UID. It lets a new driver pod take over from an old one without disrupting claims that are already prepared, which matters during upgrades.

Publishing devices as a ResourceSlice

The driver advertises its node's devices by handing the helper a set of DriverResources, organized into pools of slices, each slice holding devices with their attributes and capacity.

Copycopy code to clipboard
// state.driverResources is a resourceslice.DriverResources value:
//   Pools[nodeName].Slices[].Devices[]  (each device: Name, Attributes, Capacity)
if err := helper.PublishResources(ctx, state.driverResources); err != nil {
    return nil, err
}

Notice you build the device list once and call PublishResources; the helper reconciles the ResourceSlice objects in the API server for you, including their naming and ordering. Attributes (like memory or productName) are what a claim's CEL selector matches against.

Preparing a claim on the node

When a pod with an allocated claim lands on the node, the kubelet calls NodePrepareResources, which the helper turns into a call to your PrepareResourceClaims. You return, per claim, the devices you prepared and their CDI device IDs.

Copycopy code to clipboard
func (d *driver) PrepareResourceClaims(ctx context.Context,
    claims []*resourceapi.ResourceClaim,
) (map[types.UID]kubeletplugin.PrepareResult, error) {
    result := make(map[types.UID]kubeletplugin.PrepareResult)
    for _, claim := range claims {
        result[claim.UID] = d.prepareResourceClaim(ctx, claim)
    }
    return result, nil
}

Notice the return is keyed by claim UID, so one call can prepare several claims. Each value is a PrepareResult, which carries either an error or the prepared devices.

The per-claim work turns the allocation into concrete devices and reports their CDI IDs.

Copycopy code to clipboard
var prepared []kubeletplugin.Device
for _, dev := range preparedDevices {
    prepared = append(prepared, kubeletplugin.Device{
        Requests:     dev.GetRequestNames(),
        PoolName:     dev.GetPoolName(),
        DeviceName:   dev.GetDeviceName(),
        CDIDeviceIDs: dev.GetCdiDeviceIds(),
    })
}
return kubeletplugin.PrepareResult{Devices: prepared}

Notice CDIDeviceIDs is the bridge to the runtime. The driver has already written a CDI spec describing how to inject each device; here it just names the IDs, and the container runtime does the injection.

Unpreparing

When the pod is gone, the kubelet calls the unprepare path so the driver can release the device.

Copycopy code to clipboard
func (d *driver) UnprepareResourceClaims(ctx context.Context,
    claims []kubeletplugin.NamespacedObject,
) (map[types.UID]error, error) {
    result := make(map[types.UID]error)
    for _, claim := range claims {
        result[claim.UID] = d.state.Unprepare(claim.UID)
    }
    return result, nil
}

Notice unprepare receives lightweight NamespacedObject references, not full claims, because the claim may already be deleted by the time cleanup runs.

The DaemonSet

The driver runs as a DaemonSet on nodes that have the devices. It mounts three host paths: the kubelet registrar directory, the kubelet plugins directory (where it serves its socket), and the CDI directory (where it writes CDI specs).

Copycopy code to clipboard
volumeMounts:
  - { name: plugins-registry, mountPath: /var/lib/kubelet/plugins_registry }
  - { name: plugins,          mountPath: /var/lib/kubelet/plugins }
  - { name: cdi,              mountPath: /var/run/cdi }
volumes:
  - { name: plugins-registry, hostPath: { path: /var/lib/kubelet/plugins_registry } }
  - { name: plugins,          hostPath: { path: /var/lib/kubelet/plugins } }
  - { name: cdi,              hostPath: { path: /var/run/cdi } }

Notice the CDI mount: the driver writes CDI specs to /var/run/cdi on the host, and the runtime reads them there when it sees the CDI device IDs your driver returned.

go.mod

Pin the modules to the target version so the API matches the cluster.

Copycopy code to clipboard
require (
    k8s.io/api v0.35.0
    k8s.io/client-go v0.35.0
    k8s.io/dynamic-resource-allocation v0.35.0
)

Notice all three track the Kubernetes minor version (v0.35.x for 1.35). Mixing versions is the most common source of subtle API mismatches, because the resource.k8s.io Go types and the kubelet gRPC contract are versioned together. When you upgrade the cluster, bump all three and rebuild; the driver is coupled to the node's kubelet, not loosely decoupled like a typical client.

What the user writes

Finally, the consumer side. A ResourceClaimTemplate references the DeviceClass, and the pod references the template. Note the doubly-nested spec.spec and the exactly wrapper, which are specific to the v1 API.

Copycopy code to clipboard
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata: { name: single-gpu }
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.example.com
---
apiVersion: v1
kind: Pod
metadata: { name: pod0 }
spec:
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: single-gpu
  containers:
    - name: ctr0
      image: ubuntu:22.04
      resources:
        claims:
          - name: gpu

Notice the container opts in through resources.claims. Only containers that list the claim get the device, so a pod can hold a device that only one of its containers sees.

Why the scheduler never calls your driver

Early DRA designs let a vendor driver make the allocation decision through a callback. That was rejected because it made allocation opaque: the scheduler could not reason about devices, and the cluster autoscaler could not predict whether adding a node would help (KEP-4381).

Structured parameters flipped it. The driver publishes device attributes and capacity in ResourceSlices, and a claim selects with CEL over those attributes. Now the scheduler owns allocation with full visibility, and it can simulate placement for the autoscaler. The cost is that your driver must describe devices in the structured model rather than run arbitrary logic. That trade is what let DRA reach a stable, core API.

Where it still hurts

Most of the trouble you hit running a driver is one of these:

• Plugin never registers. If the DaemonSet cannot write to the registrar or plugins directory, the kubelet never sees the driver and claims stay pending. Check the two hostPath mounts and the kubelet plugin logs.
• Claim stuck in pending. Usually no device matches the claim's selector. Inspect the ResourceSlice (kubectl get resourceslice -o yaml) and confirm the device attributes satisfy the CEL expression.
• ResourceSlice staleness. If the driver crashes, its slices can go stale and the scheduler may allocate a device that is gone. The helper reconciles slices while running; a healthy DaemonSet is what keeps them current.
• CDI mismatch. If NodePrepareResources returns a CDI device ID that has no matching entry in /var/run/cdi, the runtime fails to start the container. The CDI spec write and the returned IDs must agree.
• Driver upgrade with allocated claims. Restarting the driver while claims are prepared can disrupt running pods. Use the RollingUpdate option so a new driver pod takes over cleanly.

Beyond the day-to-day failures, parts of DRA itself are still landing. DRA graduates in stages. As of 1.35, the core structured-parameters API is GA, while several extensions beyond the core are still maturing: admin access and prioritized lists are beta, and device taints, partitionable devices, and consumable capacity are alpha or not yet present. The companion post on DRA versus device plugins has the full stage-by-version table. If you are building a driver today, target the stable core API and adopt extensions only as they reach beta in the version you run, because an alpha extension can be disabled or change shape between minor releases.

Ship it

The driver itself is small: publish devices as ResourceSlices, implement prepare and unprepare, and let the helper handle the kubelet gRPC and socket registration. To get it running and watch the pieces move:

1. Clone the reference driver and run it on a kind cluster with the DRA feature enabled; watch kubectl get resourceslice -o yaml.
2. Apply the ResourceClaimTemplate and pod above; watch the claim move to allocated and the pod schedule.
3. Add a device attribute (like memory) to the published devices and write a claim CEL selector that requires it.
4. Kill the driver DaemonSet pod and observe how pending claims behave until it recovers.
5. Trace one NodePrepareResources call in the driver logs and match the returned CDI ID to /var/run/cdi.

The scheduler does allocation from your published attributes, which is what makes DRA schedulable and autoscaler-friendly. The companion post compares DRA with device plugins and covers when to migrate.

Further reading

• DRA concepts
• Install and use DRA
• KEP-4381: DRA structured parameters
• kubeletplugin package
• Reference driver: dra-example-driver
• Allocate devices with DRA