Kubernetes Architecture: How the Pieces Actually Fit Together
— kubernetes, architecture, platform-engineering, cloud-native — 9 min read
Most people learn Kubernetes as a set of YAML recipes. This post is about the machine underneath: what each control-plane component does, the one pattern that ties them together, and exactly what happens when you run kubectl apply. If you understand the reconciliation model and the request pipeline, most of Kubernetes stops being mysterious.
To make it concrete, we are going to follow a single Deployment all the way through. It begins as desired state you declare, and controllers work continuously to make reality match it. On the way it meets the four parts of the control plane (etcd the store, the API server the gateway, the controller manager the reconcilers, and the scheduler that places pods on nodes), where only the API server ever talks to etcd. It lands on a node, where the kubelet runs pods through the container runtime and kube-proxy implements Services. And every write it triggers passes through authentication, then authorization, then admission before anything is stored. That whole trip is the architecture.
One idea: declarative state and reconciliation
Kubernetes objects describe a desired state. A Deployment says "I want 3 replicas of this pod." Nothing makes that true instantly. Instead, a controller watches the object and keeps nudging the cluster toward it: if there are 2 pods, it creates one; if there are 4, it deletes one. This is the reconciliation loop, and nearly every component is a variation of it (controllers).
desired state (spec) ──┐ ▼ controller compares │ observed state (status) ◀── cluster reality │ take action to close the gap │ └──────▶ (repeat forever)Caption: a controller continuously drives observed state toward desired state.
This is why Kubernetes is resilient: there is no one-shot "deploy" step that can half-finish. If a node dies, the gap reappears and the controller closes it again. Understanding this makes the rest of the architecture read as "who watches what, and what gap do they close."
The control plane behind every apply
Four components sit between your kubectl apply and a running pod, each with one job (components).
etcd: the store
etcd is a consistent, highly available key-value store that holds all cluster state. It is the single source of truth. It uses the Raft consensus algorithm, so a majority of etcd members must agree on every write, which is why you run an odd number (3 or 5) for high availability. etcd chooses consistency over availability: if it loses quorum, it stops accepting writes rather than serve stale data.
The practical rule: protect etcd. Back it up, give it fast disks, and never let anything but the API server touch it.
API server: the gateway
The kube-apiserver exposes the Kubernetes HTTP API and is the only component that reads and writes etcd. Every other component, and every kubectl command, goes through it. That centralization is deliberate: it gives one place for authentication, authorization, admission, validation, and audit.
Components do not poll the API server in a loop. They open a watch, a streaming connection that pushes changes as they happen. This is how the scheduler learns about new pods and how controllers learn their objects changed, cheaply and in near real time.
Controller manager: the reconcilers
The kube-controller-manager runs the built-in controllers, each a reconciliation loop for one kind of object. The Deployment controller manages ReplicaSets, the ReplicaSet controller manages pods, the node controller reacts to nodes going unready, and so on. They all follow the same watch-compare-act pattern.
Scheduler: placement
The kube-scheduler watches for pods that have no node assigned and picks a node for each one. It works in two phases: filtering removes nodes that cannot run the pod (not enough CPU, a taint the pod does not tolerate, a node selector that does not match), and scoring ranks the survivors (spread, resource balance, affinity) to pick the best. The scheduler does not start the pod; it just writes the chosen node onto the pod object. The kubelet on that node takes it from there.
Following a Deployment from apply to running
Here is the whole trip, and it shows every component playing its part.
kubectl apply │ 1. HTTP to API server ▼API server: authn → authz → admission → validate → write to etcd (Deployment) │ 2. watch event ▼Deployment controller: creates a ReplicaSet │ReplicaSet controller: creates 3 Pods (no node assigned yet) │ 3. watch event: pods are Pending, unscheduled ▼Scheduler: filters + scores nodes, writes nodeName onto each Pod │ 4. watch event: a Pod on my node ▼Kubelet (on the chosen node): calls the container runtime via CRI to start containers │ 5. reports status back ▼API server updates Pod status in etcd; Deployment shows 3/3 readyCaption: creating a Deployment cascades through controllers, the scheduler, and the kubelet, each closing one gap.
Notice no component orchestrates the others. Each watches the API server, does its one job, and writes the result back. The Deployment becoming ready is an emergent result of several loops running independently.
Where the pod actually runs
The scheduler wrote a node name onto each pod. Now that node takes over.
kubelet
The kubelet is the agent on every node. It watches for pods assigned to its node and makes them real: it pulls images, starts containers through the runtime, runs liveness and readiness probes, and reports pod status back to the API server. It is itself a reconciliation loop, closing the gap between "pods assigned to me" and "pods actually running here."
Container runtime and CRI
The kubelet does not run containers directly. It talks to a container runtime through the Container Runtime Interface (CRI), a gRPC API. containerd is the common runtime. This abstraction is why Kubernetes could remove the old Docker shim: anything that speaks CRI works.
kube-proxy
kube-proxy programs the node's network rules so that traffic to a Service's virtual IP is spread across the Service's pods. Modern clusters use its nftables or IPVS backend for this. It, too, watches the API server (for Services and EndpointSlices) and reconciles the node's rules.
Giving pods a reachable address
The pods are running, but nothing can find them yet. Kubernetes has one firm rule: every pod gets its own IP, and every pod can reach every other pod's IP directly, without NAT (Network Address Translation). How that flat network is built is left to a CNI (Container Network Interface) plugin like Calico or Cilium, which the kubelet calls when setting up a pod's network.
Pods are ephemeral and their IPs change, so you do not talk to pods directly. A Service gives a stable virtual IP and DNS name in front of a set of pods.
client ──▶ Service (stable ClusterIP / DNS) │ kube-proxy load-balances ┌───────┼───────┐ ▼ ▼ ▼ pod pod pod (tracked by EndpointSlices)Caption: a Service is a stable front for a changing set of pods.
The Service types stack: ClusterIP is reachable only inside the cluster, NodePort opens a port on every node, and LoadBalancer provisions an external load balancer. For HTTP routing by host and path, an Ingress controller or the newer Gateway API sits in front and routes to Services.
Giving pods durable storage
Our Deployment might also need a disk. Storage separates the request from the supply. A pod uses a PersistentVolumeClaim (PVC), which says "I need 20Gi, read-write-once." A PersistentVolume (PV) is the actual piece of storage. A StorageClass describes how to create PVs on demand.
Pod ──▶ PVC (a request) ──bound──▶ PV (real storage) │ StorageClass ──▶ CSI driver provisions the PV dynamicallyCaption: a claim is matched to a volume, created on demand by a CSI driver.
Dynamic provisioning is the normal path: you create a PVC referencing a StorageClass, and a CSI (Container Storage Interface) driver creates the backing disk and the PV automatically. CSI is the storage analogue of CRI and CNI: a standard plugin interface so Kubernetes core does not embed vendor code.
The gate every write passes through
Every gap the Deployment closed became a write to the API server, and each write took the same path. Every write to the API server passes through three stages in order. Understanding this order explains most "why was I denied" questions.
request ──▶ Authentication ──▶ Authorization ──▶ Admission ──▶ etcd who are you? are you allowed? mutate/validate (certs, tokens, (RBAC rules) (policies, defaults) OIDC)Caption: authentication, then authorization, then admission, then storage.
Authentication answers "who are you," using client certificates, bearer tokens, or an OIDC (OpenID Connect) identity provider. It never decides permissions, only identity. Authorization answers "are you allowed to do this," almost always through RBAC (Role-Based Access Control): Roles grant verbs on resources, and RoleBindings attach them to users or groups. Admission runs last, after you are authenticated and authorized: mutating admission can change the object (inject defaults, sidecars) and validating admission can reject it (policy engines like Kyverno, or the built-in Pod Security admission). Only after all three does the object reach etcd.
Holding up at scale and through failure
One Deployment is easy. A cluster running thousands needs the same loops to survive failures. For a production control plane, run etcd with 3 or 5 members for quorum, and run multiple API servers behind a load balancer. The controller manager and scheduler run as several replicas but use leader election, so only one is active at a time; the others stand by. The API server is stateless, so it scales horizontally; etcd is the component that limits cluster size, which is why large clusters tune etcd hard and minimize write churn.
Reading a symptom back to its loop
Because each component owns one gap, most failures point straight at the loop that owns them.
- etcd quorum loss stops writes. Lose a majority of etcd members and the cluster goes read-only. Always keep an odd member count and current backups.
- The scheduler only assigns; the kubelet runs. A pod stuck in
Pendingis a scheduling problem (no fitting node); a pod stuck inContainerCreatingis a kubelet, image, or CNI problem. The distinction points you at the right component. - Nothing talks to etcd but the API server. If you are tempted to read etcd directly, use the API instead; direct access bypasses authorization and audit.
- Admission runs after authorization. A request denied by a policy engine was already authenticated and authorized; look at admission webhooks, not RBAC.
- Services front EndpointSlices, not pods directly. If traffic misses a healthy pod, check the EndpointSlices and readiness probes, since only ready pods are endpoints.
The trip in one view
Kubernetes is a set of independent reconciliation loops coordinating through one API server backed by etcd. The control plane stores desired state and drives reality toward it; the kubelet runs the pods; CNI, CSI, and CRI are the standard plugin seams for networking, storage, and runtime. Trace a Deployment through the system once and the architecture clicks: no orchestrator, just many controllers each closing one gap. That model is also your debugging map, since every symptom points at the one loop that owns it.