vLLM on Kubernetes: Serving LLMs Without the Bloat

14.01.2025 — vllm, kubernetes, llm, inference, gpu — 7 min read

vLLM is the most widely used engine for serving open large language models (LLMs), because it makes GPU memory go much further and keeps the GPU busy. Running it on Kubernetes is straightforward once you know the few things that actually matter: a correct GPU Deployment, shared memory for tensor parallelism, and autoscaling on vLLM's own metrics. This post skips the custom glue code and shows the built-in mechanisms that do the job.

The speed comes from two ideas, PagedAttention and continuous batching, and both are built in, so you configure them rather than write them. You serve a model with vllm serve <model>, which starts an OpenAI-compatible HTTP server on port 8000 with /health and /metrics. On Kubernetes that becomes a Deployment requesting nvidia.com/gpu, mounting a large /dev/shm, and probing /health. When traffic climbs, vLLM already exposes Prometheus metrics like vllm:num_requests_waiting and vllm:num_requests_running, so you autoscale on queue depth instead of CPU and skip the custom exporter. The rest of this post follows that path from one GPU to many.

How vLLM gets more out of a GPU

Serving an LLM means holding a KV cache (key-value cache), the attention state for every token in every active request. It is large and its size changes as generation proceeds. Naive serving reserves a fixed contiguous block per request sized for the maximum sequence length, which wastes most of the GPU memory most of the time.

PagedAttention borrows the operating-system idea of paging: the KV cache is split into fixed-size blocks that do not need to be contiguous, allocated on demand as a request grows. That removes the fragmentation and over-reservation, so a GPU can hold many more concurrent requests.

Naive:  [req A: reserved for max length ................ mostly empty ]
        [req B: reserved for max length ................ mostly empty ]

Paged:  KV cache split into blocks, allocated as needed
        A: [blk][blk][blk]      B: [blk][blk]      free: [blk][blk]...

Caption: PagedAttention allocates KV-cache blocks on demand instead of reserving a max-length slab per request.

The second win is continuous batching. Instead of waiting to assemble a fixed batch, vLLM adds and removes requests from the running batch every step, so a finished request frees its slot immediately and a new one takes it. The GPU stays saturated even under bursty, variable-length traffic.

You do not implement either of these. They are why vLLM exists; you just configure the server.

Serving one model on one GPU

vllm serve launches the OpenAI-compatible server. Point any OpenAI client at it.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.92

Notice --gpu-memory-utilization (default 0.92) is the fraction of GPU memory vLLM claims for weights plus KV cache; lower it if the GPU is shared. --tensor-parallel-size (-tp) shards one model across multiple GPUs in a pod. The server listens on port 8000 and serves /v1/chat/completions, /v1/completions, plus /health and /metrics.

Getting it onto Kubernetes

Run vLLM as a Deployment that requests a GPU. The important, easy-to-miss parts are the shared-memory volume, the Hugging Face token for gated models, and probing /health.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-3-8b
spec:
  replicas: 1
  selector: { matchLabels: { app: llama-3-8b } }
  template:
    metadata: { labels: { app: llama-3-8b } }
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "meta-llama/Llama-3.1-8B-Instruct", "--gpu-memory-utilization", "0.92"]
          ports: [{ containerPort: 8000 }]
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom: { secretKeyRef: { name: hf-token, key: token } }
          resources:
            limits: { nvidia.com/gpu: "1" }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 10
          volumeMounts:
            - { name: shm, mountPath: /dev/shm }
      volumes:
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 2Gi }

Notice three things. The nvidia.com/gpu limit requires the NVIDIA device plugin on your nodes. The /dev/shm volume matters because tensor parallelism uses shared memory between GPU workers, and the default 64 MB /dev/shm causes hangs; back it with memory. The readiness probe waits on /health, and initialDelaySeconds is generous because loading weights takes time.

Expose it with a Service so clients and the autoscaler can reach it.

apiVersion: v1
kind: Service
metadata: { name: llama-3-8b }
spec:
  selector: { app: llama-3-8b }
  ports: [{ port: 8000, targetPort: 8000 }]

Notice this is a normal ClusterIP Service on 8000; put an Ingress or Gateway in front for external access.

Splitting a big model across GPUs

A model too large for one GPU is split across several with tensor parallelism. Set --tensor-parallel-size to the number of GPUs and request that many in the pod. All the GPUs must be in the same pod (same node), so schedule accordingly.

args: ["--model", "meta-llama/Llama-3.1-70B-Instruct", "--tensor-parallel-size", "4"]
resources:
  limits: { nvidia.com/gpu: "4" }

Notice --tensor-parallel-size must equal the GPU count, and this is where the /dev/shm volume becomes essential, since the workers coordinate through shared memory.

Scaling on queue depth, not CPU

vLLM already exposes Prometheus metrics at /metrics with the vllm: prefix. You do not need to write an exporter. The metrics that matter for scaling and health:

Metric	Meaning
`vllm:num_requests_running`	Requests currently decoding on the GPU
`vllm:num_requests_waiting`	Requests queued, waiting for a slot
`vllm:kv_cache_usage_perc`	Fraction of KV cache in use

Source: vLLM production metrics. Scrape /metrics with a ServiceMonitor if you run the Prometheus operator.

For autoscaling, scale on queue depth, not CPU. CPU utilization is meaningless for a GPU server; vllm:num_requests_waiting tells you demand is outrunning capacity. Feed that metric to an HPA through the Prometheus Adapter or KEDA.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: llama-3-8b }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: llama-3-8b }
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: Pods
      pods:
        metric: { name: vllm_num_requests_waiting }
        target: { type: AverageValue, averageValue: "5" }

Notice the target is an average queued-requests value per pod, so new replicas are added when requests start backing up. Each replica needs a GPU, so pair this with a GPU-aware cluster autoscaler or Karpenter.

Shrinking the footprint with quantization

Before you reach for a second GPU, see whether the model fits in less memory. To run a larger model on a given GPU, serve a quantized checkpoint. vLLM supports formats like AWQ, GPTQ, and FP8 through --quantization, which cuts weight memory at some quality cost.

vllm serve TheModel-AWQ --quantization awq

Notice quantization trades a small accuracy drop for a large memory saving, which often lets you drop from two GPUs to one.

When it breaks

Most production trouble comes from the same handful of causes.

CUDA out of memory at startup. Lower --gpu-memory-utilization, or the model plus your --max-model-len does not fit. Reduce context length or use a smaller or quantized model.
Tensor parallelism hangs. The default 64 MB /dev/shm is too small. Mount a memory-backed /dev/shm as shown.
Pod flaps as NotReady on start. Weight download and load take minutes for large models. Raise initialDelaySeconds or pre-bake weights into a volume or image.
Gated model 401. Set HUGGING_FACE_HUB_TOKEN from a Secret; models like Llama require accepting a license.
Autoscaling on CPU does nothing useful. Scale on vllm:num_requests_waiting. A GPU server can be 100% busy at low CPU.
Slow first token under load. Watch vllm:num_requests_waiting and vllm:kv_cache_usage_perc; if the cache is saturated, you need more replicas or a bigger GPU, not tuning.

The whole production path

vLLM is fast because of PagedAttention and continuous batching, and on Kubernetes the job is mostly getting the Deployment right: request GPUs, give it a memory-backed /dev/shm, probe /health, and provide a Hugging Face token. Use vLLM's built-in /metrics rather than custom exporters, and autoscale on vllm:num_requests_waiting. That is the whole production path, without the glue code.