How the kubelet Actually Starts a Pod

11.03.2026 — kubernetes, kubelet, cri, cni, node — 8 min read

A pod is Scheduled, then it sits in ContainerCreating for a while, then it runs. What happens in that gap is the kubelet's job, and when a pod is stuck, knowing the exact sequence tells you which component to look at. This post walks the real order the kubelet follows to turn a pod spec into running containers, and maps each step to the event or signal that shows it worked or failed. It targets Kubernetes 1.35 and the CRI (Container Runtime Interface) v1 API, with containerd as the runtime. The behaviour here comes from the kubelet sync loop reference and the init containers docs.

This is a companion to the Kubernetes architecture deep dive, which covers how the pod got scheduled to the node in the first place.

Pending with no node means scheduling, and that is a control-plane problem. But ContainerCreating is the kubelet, and it covers half a dozen distinct steps: mounting a volume, creating the network sandbox, pulling an image, running an init container. "It's stuck in ContainerCreating" is not a diagnosis, it is a prompt to ask which step. The rest of this post gives you the steps, in the order the kubelet runs them, so you can ask the right question and read the right log. The short version: the kubelet admits the pod, mounts volumes, creates the sandbox and sets up CNI, pulls images, runs init containers in order, starts app containers, and runs probes. The pod has no IP until CNI finishes during sandbox creation, init containers run one at a time to completion, and app containers start concurrently after them.

How the kubelet is wired

The kubelet is one Go process with a main sync loop and a set of worker goroutines around it. A few terms carry the rest of this post:

Sync loop: the kubelet's main control loop; it reconciles desired pod specs against actual running containers.
Pod worker: a per-pod goroutine that processes that pod's changes in order (FIFO by pod UID).
Pod sandbox: the pod-level environment the runtime creates, anchored by a pause container that holds the shared network namespace.
CRI (Container Runtime Interface): the gRPC API the kubelet uses to talk to the runtime (RunPodSandbox, CreateContainer, StartContainer).
CNI (Container Network Interface): the plugin the runtime calls to give the sandbox an IP.
Volume manager: the kubelet subsystem that attaches and mounts volumes, including calling CSI (Container Storage Interface) drivers.

The sync loop reads pod changes from several sources (the API server, static pod files, an HTTP endpoint) and a periodic timer, then hands each pod to its own worker (kubelet sync loop).

API server ─┐
static files ┼─▶ sync loop ─▶ per-pod worker (one goroutine per pod UID)
HTTP ───────┤                    │
periodic ───┘                    ▼
                        SyncPod: reconcile this pod to desired state

Caption: the sync loop aggregates pod sources and dispatches each pod to its own worker.

Each worker runs one of three paths depending on the pod's phase: SyncPod for setup and steady state, and two teardown paths for termination and cleanup. Starting a pod is the SyncPod path.

The per-pod worker model is why one bad pod does not stall the node: each pod UID gets its own goroutine processing its changes in FIFO order, so a pod stuck mounting a volume does not block another pod's startup. The sync loop also runs on a periodic timer, not only on events, so the kubelet re-reconciles even without an API change, which is how it recovers a container that died without an event reaching it.

The order SyncPod follows

For a fresh pod on the node, SyncPod runs these steps in order. Containers do not start until both the volumes are mounted and the sandbox is ready.

1. Admit         check the pod fits the node (resources, limits)
2. Mount volumes volume manager attaches + mounts, incl. CSI (can take seconds)
3. Sandbox + CNI  RunPodSandbox creates the pause container; CNI assigns the IP
4. Pull images    per container, if not already cached
5. Init containers run in order, one at a time, each must exit 0
6. App containers start concurrently
7. postStart + probes  hooks run, then startup/readiness/liveness probes
8. Status         report container states back to the API server

Caption: the fixed order from a scheduled pod to running containers.

Two ordering facts matter most. Volumes are mounted before the sandbox and containers, so a CSI driver that is slow or missing blocks everything downstream. And the pod gets no IP until step 3, because the IP comes from CNI during sandbox creation. That is why a pod with a networking problem sits in ContainerCreating with no address.

The container start itself, step 4 through 7, is per container: the runtime pulls the image, creates the container in the sandbox, starts it, then runs the postStart hook (kuberuntime source). Init containers repeat this loop one at a time; app containers run it concurrently.

Reading the sequence with kubectl and crictl

Every phase surfaces as a pod event. This is the same sequence from kubectl describe pod, in the order the kubelet emits it.

Scheduled              scheduler bound the pod (not the kubelet)
SuccessfulAttachVolume volume manager attached a volume
Pulling / Pulled       image pull started / finished
Created                container created in the sandbox
Started                container started

Notice Scheduled comes from the scheduler; everything after it is the kubelet. The gap between Scheduled and Pulling is where volume mounting and sandbox creation happen, and it is silent unless something fails.

The pod's conditions track the same progress, and are more precise than the phase. Check them with a status query.

kubectl get pod mypod -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'
# PodScheduled=True
# PodReadyToStartContainers=True   <- sandbox + volumes + CNI done
# Initialized=True                 <- all init containers succeeded
# ContainersReady=True
# Ready=True

Notice PodReadyToStartContainers is the signal that volumes and the network sandbox are set up (KEP-3085). If it is False, the problem is before containers: volumes or CNI. If Initialized is False, an init container has not finished. That condition is worth knowing because it split a previously invisible phase into an observable one. Before it, the window between "scheduled" and "containers starting" was one opaque ContainerCreating. Now you can tell whether the delay is sandbox-and-volume setup or something later, from the condition alone.

On the node itself, you can see the sandbox and containers directly through the runtime.

crictl pods --name mypod        # the sandbox (one per pod)
crictl ps --pod <sandbox-id>    # containers inside it

Notice there is one sandbox per pod and one entry per container. If crictl pods shows the sandbox but crictl ps shows no containers, you are stuck between sandbox creation and container start, usually an image pull.

Why volumes mount first and the sandbox stands alone

Why a sandbox and a separate pause container at all? The alternative would be to attach networking to the first application container. Kubernetes chose a dedicated sandbox so the network namespace and IP outlive any single container: an app container can crash and restart without losing the pod's IP or its peers' connections. The cost is one extra tiny container per pod and an extra CRI call. That trade is why a restarting container keeps its address.

Why mount volumes before creating the sandbox? So a container never starts without its storage, which would race the application against its own data. The cost is that a slow or broken CSI driver blocks the pod at a stage with no container logs to read, which is exactly why the volume phase is a common source of confusing hangs.

Where a pod hangs

Each way a pod stalls maps back to one step in the sequence, and each has its own signal.

Stuck at volumes. Event FailedMount or FailedAttachVolume, and PodReadyToStartContainers=False. The CSI driver is missing, slow, or the volume cannot attach. Check the CSI driver pods and the node's kubelet log; there are no container logs yet.
Stuck at the sandbox. Event FailedCreatePodSandBox, often mentioning CNI. The network plugin failed to assign an IP. Check the CNI plugin (Calico, Cilium) pods on the node and /etc/cni/net.d.
Stuck at image pull. ImagePullBackOff or ErrImagePull. Wrong image name, missing pull secret, or registry unreachable. The sandbox exists but no app container does.
Stuck at an init container. Initialized=False and one init container not Completed. Init containers run one at a time and must exit 0; a failing init container blocks the whole pod. Read that init container's logs.
Runs but never Ready. Containers started but ContainersReady=False. A readiness probe is failing. The pod holds its IP but is not an endpoint, so Services will not route to it.

Try it on a live node

The sequence is easiest to trust once you watch it break. Each of these forces one phase to show itself.

Create a pod and watch conditions flip in order: kubectl get pod -w -o wide plus the jsonpath above.
Point a pod at a missing image and confirm the sandbox exists (crictl pods) but no container does (crictl ps).
Add a failing init container and watch Initialized stay False while the app containers never start.
On a node, run crictl pods and crictl ps for a running pod and match them to kubectl describe.
Break a readiness probe and confirm the pod is Running but not an endpoint in its Service.

Next time a pod hangs in ContainerCreating, read the conditions first, then the event, then the log for that phase.