Kubernetes 1.36: PSI Metrics GA, User Namespaces GA, and a Quieter Cleanup Release

Kubernetes 1.36 is a cleanup and maturation release with two genuinely useful graduations: Pressure Stall Information (PSI) metrics and user namespaces both reach GA. It also matures Dynamic Resource Allocation (DRA) further and stabilizes mutating admission policies. The headline for upgrade planning is what did not happen: the containerd 1.x removal that earlier releases scheduled for 1.36 slipped to 1.37. This post covers what changed and what to do.

Previous release: Kubernetes 1.35. Next: 1.37 (coming soon).

Here is the short version before the detail. PSI metrics (KEP #4205) and user namespaces (KEP #127) are both GA. DRA keeps maturing: admin access and prioritized lists go GA, while partitionable devices and consumable capacity land as beta. The release removes the gitRepo volume plugin (now impossible to re-enable), the in-tree Portworx plugin, and kubeadm flex-volume support, and it drops several v1alpha1 APIs. Service .spec.externalIPs is deprecated, with removal scheduled for 1.43. And the containerd 1.x removal was deferred from 1.36 to 1.37, so 1.36 does not force the runtime upgrade, though you should plan it anyway.

Root in the pod stops being root on the node

The most consequential graduation in 1.36 is the one that shrinks what a compromised container can reach. Until now, a container running as root was root on the host too. A user namespace maps the user and group IDs inside a container to a different, unprivileged range on the host, so root inside the Pod is not root on the node. This graduated to GA in 1.36 (KEP #127). You opt in per Pod.

Copycopy code to clipboard
spec:
  hostUsers: false   # container UIDs/GIDs map to unprivileged host ranges

Notice existing Pods are unaffected; you set hostUsers: false to enable it. It is one of the strongest cheap wins for reducing container-escape blast radius.

Adopt for untrusted or multi-tenant workloads. Confirm your runtime and kernel support it.

The other graduations in 1.36

Beyond user namespaces, 1.36 finishes a batch of long-running graduations and moves a few newer features forward.

PSI metrics reach GA

PSI (Pressure Stall Information) is a Linux cgroup v2 feature that measures how long tasks are stalled waiting for CPU, memory, or I/O. Raw CPU and memory utilization tell you how much a node uses, not how much a workload is stalling while it waits for a resource. It is a sharper contention signal than utilization: a node can be at 60% CPU and still have tasks stalling. In 1.36 the kubelet exports PSI metrics at GA (KEP #4205).

Copycopy code to clipboard
cgroup v2 (kernel) ──▶ some/full pressure for cpu, memory, io
        │
        ▼
   kubelet reads PSI ──▶ exposes metrics ──▶ your monitoring / autoscaling

Caption: PSI flows from the kernel's cgroup v2 accounting through the kubelet to your metrics.

Adopt if you run autoscaling or capacity planning and want a leading indicator of contention. PSI needs cgroup v2 nodes, which you should already be on after 1.35.

DRA keeps filling out

DRA (Dynamic Resource Allocation), whose core API went GA in 1.34, keeps filling out. Two sub-features reach GA in 1.36: admin access, which lets authorized users inspect devices already in use for monitoring (KEP #5018), and prioritized lists, which let a claim accept alternatives in priority order (KEP #4816). A beta wave adds partitionable devices, consumable capacity, and device taints and tolerations.

Adopt the GA pieces if you run DRA; treat the beta wave as evaluate-on-test-clusters.

DRA has been graduating in stages since 1.30, and that staged approach is worth noting. Rather than one big GA, DRA graduates sub-features release by release: core API in 1.34, admin access and prioritized lists in 1.36, with partitionable devices and consumable capacity following in beta. If you adopt DRA, track which sub-features you depend on and their individual stages.

Mutating admission policies go GA

MutatingAdmissionPolicy is a declarative, in-process alternative to mutating webhooks, using CEL (Common Expression Language) and JSON Patch. Beta since 1.34, it is GA in 1.36 (KEP #3962). Moving simple defaulting off a webhook removes a network hop and a failure mode.

Adopt for defaulting logic you currently run in a mutating webhook.

Fine-grained kubelet authorization goes GA

Previously, granting access to the kubelet's HTTPS API meant the broad nodes/proxy permission. KubeletFineGrainedAuthz lets you grant precise, least-privilege access to specific kubelet endpoints, and it is GA (KEP #2862). Adopt if anything scrapes the kubelet API and you want to tighten its RBAC (Role-Based Access Control).

Node log query and OCI volume source go GA

Two smaller GAs worth knowing. Node log query lets you read a node's system logs through the kubelet API and kubectl, gated by NodeLogQuery and the kubelet enableSystemLogQuery setting (KEP #2258). And the OCI (Open Container Initiative) volume source, which mounts an image or artifact as a read-only volume, is now GA (KEP #4639).

Memory QoS with cgroup v2 reaches beta

The kubelet can now program cgroup v2 memory.min and memory.high to give Pods tiered memory protection aligned to their requests and limits, reducing thrashing between Pods on a node (KEP #2570). Beta. Adopt on test clusters if you have noisy-neighbor memory contention.

Workload-aware and gang scheduling stay in alpha

For AI and batch, 1.36 advances workload-aware scheduling with a decoupled PodGroup API. Gang scheduling, which binds a group of Pods all-or-nothing so a distributed job does not get half-placed, evaluates the group atomically (KEP #4671 and related WAS KEPs).

Copycopy code to clipboard
PodGroup (needs 4 pods)
   │  scheduler evaluates the group as a unit
   ▼
all 4 fit?  ── yes ──▶ bind all 4
   │
   no ──▶ bind none, retry later   (no half-scheduled job)

Caption: gang scheduling binds a PodGroup atomically.

Adopt only on test clusters; this is alpha.

Declarative API validation and the gogo removal

Two internal but meaningful changes. Declarative validation with validation-gen lets API validation be generated from IDL marker comments in the type definitions rather than hand-written functions (KEP #5073). And the unmaintained gogoprotobuf dependency was removed from the API types, closing a security and maintenance liability (KEP #5589). Nothing to do; both improve the codebase you depend on.

Before you upgrade: what breaks and what is gone

Part of 1.36 is finishing long-running cleanups, and some of that will break workloads that still lean on removed plugins or APIs. The gitRepo volume plugin, for example, has been a known remote-code-execution risk since it was deprecated in 1.11, and it is now gone for good. 1.36 lands the final steps for several of these cleanups, so read this section before you upgrade.

Removed in 1.36

Deprecated in 1.36 (removed later)

To find Services using the deprecated externalIPs field, which is a CVE-2020-8554 man-in-the-middle risk:

Copycopy code to clipboard
kubectl get svc -A -o json \
  | jq -r '.items[] | select(.spec.externalIPs != null) | "\(.metadata.namespace)/\(.metadata.name)"'

Notice these all need to move to LoadBalancer, NodePort, or Gateway API before 1.43.

Behaviour that shifts under you

From the changelog's Urgent Upgrade Notes and deprecation section:

• A metric was renamed. volume_operation_total_errors became volume_operation_errors_total in kube-controller-manager. Update dashboards and alerts.
• Scheduler PreBind changed. PreBind plugins can now run in parallel; out-of-tree plugins must return a PreBindPreFlightResult from PreBindPreFlight (returning nil keeps sequential behaviour). Rebuild custom scheduler plugins.
• containerd 1.x removal was deferred from 1.36 to 1.37 (changelog). It then slipped again from 1.37 to 1.38 in the 1.37 cycle, so 1.36 does not force a containerd upgrade. Do the migration this cycle regardless; the deadline keeps moving but is coming.
• gitRepo is gone for good. Any Pod still using a gitRepo volume will fail. There is no gate to re-enable it.
• SELinux volume relabeling changed. The faster recursive SELinux relabel is GA, with behaviour implications flagged for 1.37; audit clusters that rely on SELinux volume labels.

Version skew rules are unchanged: upgrade the control plane before nodes and stay within the supported kubelet skew (version skew policy).

The containerd non-event

The interesting non-event is the containerd deferral. The 1.34 and 1.35 release blogs both said 1.36 would be the release that dropped containerd 1.x. The 1.36 changelog moved that to 1.37, and the 1.37 changelog moved it again to 1.38, both times to align with containerd 1.7's end of support. The lesson for release planning: treat "removed in version N" as intent until the release notes for version N confirm it. This series flags that difference so your migration timeline is based on what shipped, not what was announced.

The upgrade order

Copycopy code to clipboard
1. remove gitRepo volumes and in-tree Portworx / flex-volume usage
2. delete v1alpha1 objects (Workload API, WebhookAdmissionConfiguration, SnapshotMetadataService)
3. update dashboards for the renamed volume metric and removed cAdvisor metrics
4. rebuild out-of-tree scheduler plugins for the PreBind change
5. upgrade control plane, then nodes
6. plan the containerd 2.0+ upgrade now (required for 1.37)
7. keep 1.35 control-plane images for rollback

Caption: recommended 1.35 to 1.36 upgrade order.

Specifics:

1. Scan for gitRepo volumes and Portworx/flex-volume plugin usage; these no longer work.
2. Clear the removed v1alpha1 APIs before upgrading.
3. containerd 1.x still works on 1.36, but 1.37 removes support. Do the runtime upgrade during this cycle so 1.37 is uneventful.
4. Rollback: keep 1.35 control-plane images.

Where this leaves you

Kubernetes 1.36 is a maturation release: PSI metrics and user namespaces go GA, DRA and mutating policies stabilize further, and a batch of old plugins and alpha APIs are removed. The single most important planning takeaway is that the containerd 1.x removal moved to 1.37, so use this cycle to finish the runtime upgrade. Migrate off gitRepo and externalIPs now.

A few concrete moves for this week:

1. Scrape PSI metrics from the kubelet and chart some/full pressure for CPU and memory on a busy node.
2. Enable user namespaces on a workload: set hostUsers: false and confirm it schedules.
3. Move one mutating webhook's defaulting logic to a MutatingAdmissionPolicy.
4. Grep for externalIPs Services and plan their migration.
5. Confirm your container runtime plan: are all nodes ready to move to containerd 2.0+ before 1.37?

Next in this series: Kubernetes 1.37.

Sources and specs

Release and dates:

• Kubernetes v1.36: Haru
• Releases: 1.36
• CHANGELOG-1.36

Features:

• PSI metrics: KEP #4205
• User namespaces: KEP #127
• DRA admin access: KEP #5018; prioritized lists: KEP #4816
• MutatingAdmissionPolicy: KEP #3962
• Fine-grained kubelet authz: KEP #2862
• Node log query: KEP #2258; OCI volume source: KEP #4639
• Memory QoS cgroup v2: KEP #2570
• Gang scheduling / WAS: KEP #4671
• Declarative validation: KEP #5073; gogo removal: KEP #5589

Deprecations:

• Service externalIPs: KEP #5707
• gitRepo removal: KEP #5040