Kubernetes GPU Basics: Why GPUs Aren't Scheduled Like CPUs

When you first learn Kubernetes, you learn it through CPU and memory. You write a resource block like this:

yaml
Copy
1resources:
2  requests:
3    cpu: "500m"
4    memory: "512Mi"
5  limits:
6    cpu: "1"
7    memory: "1Gi"

Then the scheduler figures out where the pod goes. Easy enough.

GPUs break that intuition.

Kubernetes does not treat a GPU as "extra CPU." It doesn't understand CUDA out of the box, it has no idea how much VRAM your model needs, and it doesn't care whether your container is running PyTorch, TensorRT, vLLM, a hand-written CUDA kernel, or just nvidia-smi. To Kubernetes, a GPU is a special piece of hardware that gets exposed through a device plugin.

That's the first mental model to fix in your head.

Every node already reports CPU and memory to the control plane. Run this:

bash
Copy
1kubectl describe node worker-1

and you'll see allocatable resources like:

Copy
cpu:     8
memory: 32Gi

A GPU is different because it's vendor-specific hardware. NVIDIA GPUs, AMD GPUs, FPGAs, fast NICs, and other accelerators all need their own setup. Kubernetes handles this through the device plugin framework: the vendor ships a plugin that advertises the hardware to the kubelet.

For NVIDIA, the plugin usually exposes GPUs under this name:

Copy
nvidia.com/gpu

So once the NVIDIA device plugin is installed, a GPU node starts reporting something like:

Copy
Capacity:
  nvidia.com/gpu: 1

Allocatable:
  nvidia.com/gpu: 1

Now Kubernetes can schedule pods that ask for:

yaml
Copy
1nvidia.com/gpu: 1

Skip the plugin and the GPU stays invisible. The card is physically sitting in the box, but the scheduler has no idea it exists.

Think of it as a small agent that runs on each node. It's deployed as a DaemonSet, so you get one copy per GPU node. NVIDIA's own docs describe it plainly: it reports how many GPUs are on the node, tracks their health, and lets GPU-enabled containers run inside Kubernetes.

The flow is roughly this:

Copy
GPU node has NVIDIA driver installed
        ↓
NVIDIA container runtime/toolkit is configured
        ↓
NVIDIA device plugin runs on the node
        ↓
Plugin registers with kubelet
        ↓
Kubelet advertises nvidia.com/gpu to the API server
        ↓
Scheduler can place GPU-requesting pods on that node

Here's the part people get wrong. The scheduler never touches the physical GPU. All it does is answer one question:

"This pod wants 1 nvidia.com/gpu. Which node has one free?"

Once the pod lands, the kubelet and the device plugin do the low-level work of wiring the actual device into the container.

CPU is fractional. You can ask for half a core:

yaml
Copy
1requests:
2  cpu: "500m"

Memory is a byte count:

yaml
Copy
1requests:
2  memory: "1Gi"

GPUs are whole devices. You request them as integers, and you put them in limits:

yaml
Copy
1resources:
2  limits:
3    nvidia.com/gpu: 1

The docs are strict about this. Specify a GPU limit with no request and Kubernetes copies the limit into the request for you. Specify both and they have to match. You can't set a GPU request without a limit at all.

So the normal pattern is this:

yaml
Copy
1resources:
2  limits:
3    nvidia.com/gpu: 1

Not this:

yaml
Copy
1resources:
2  requests:
3    nvidia.com/gpu: 1

And usually not this:

yaml
Copy
1resources:
2  requests:
3    nvidia.com/gpu: 1
4  limits:
5    nvidia.com/gpu: 2

For GPUs the scheduler wants a clean integer count, nothing clever.

This is about the smallest thing you can run to prove GPU scheduling works:

yaml
Copy
1apiVersion: v1
2kind: Pod
3metadata:
4  name: cuda-gpu-test
5spec:
6  restartPolicy: Never
7  containers:
8    - name: cuda
9      image: nvidia/cuda:12.4.1-base-ubuntu22.04
10      command: ["nvidia-smi"]
11      resources:
12        limits:
13          nvidia.com/gpu: 1

Apply it:

bash
Copy
1kubectl apply -f cuda-gpu-test.yaml

Check where it landed:

bash
Copy
1kubectl get pod cuda-gpu-test -o wide

Then read the logs:

bash
Copy
1kubectl logs cuda-gpu-test

If everything is wired up, you'll get nvidia-smi output showing the GPU from inside the container. That single result proves three separate things at once: Kubernetes saw a GPU resource, the scheduler put the pod on a GPU node, and the runtime actually gave the container access to the card. That's your hello world for GPU workloads.

Say you have three nodes:

Copy
node-a: 0 GPUs
node-b: 1 GPU
node-c: 4 GPUs

You deploy a pod asking for one GPU:

yaml
Copy
1resources:
2  limits:
3    nvidia.com/gpu: 1

The scheduler filters. node-a is out because it has no GPU. node-b works if its GPU is free. node-c works if at least one of its four is free. Then the scheduler picks a winner using its usual logic.

Look at everything the scheduler is not thinking about:

Copy
Does this model need 12GB VRAM?
Does this GPU support the right CUDA version?
Is this an A10, T4, A100, H100, or L40S?
Is this workload latency-sensitive?
Will this pod fight another pod for memory bandwidth?

By default it mostly sees one thing:

Copy
nvidia.com/gpu: available count

That gap is exactly why real GPU clusters need labels, taints, node selectors, affinity rules, monitoring, and sometimes a custom scheduler. The defaults are honest about how little they know.

In a real cluster the GPUs are not interchangeable. A T4 is not an A100. An A10 is not an H100. A 24GB card is a completely different animal from an 80GB card. If you don't tell Kubernetes the difference, it won't invent one for you.

The docs suggest labeling nodes and selecting on those labels when you have mixed hardware:

bash
Copy
1kubectl label node gpu-node-1 accelerator=nvidia-t4
2kubectl label node gpu-node-2 accelerator=nvidia-a100

Then a pod can target a specific class of GPU:

yaml
Copy
1apiVersion: v1
2kind: Pod
3metadata:
4  name: a100-job
5spec:
6  restartPolicy: Never
7  nodeSelector:
8    accelerator: nvidia-a100
9  containers:
10    - name: cuda
11      image: nvidia/cuda:12.4.1-base-ubuntu22.04
12      command: ["nvidia-smi"]
13      resources:
14        limits:
15          nvidia.com/gpu: 1

This is how you stop a heavy inference job from landing on a weak node by accident.

Doing this by hand doesn't scale though. You don't want engineers memorizing node names and guessing. You want the platform to label GPU nodes automatically and hand out clean workload classes like:

Copy
gpu-type=a10
gpu-type=a100
gpu-memory=80gb
workload=inference
workload=training

The common assumption is that setting nvidia.com/gpu: 1 means Kubernetes will manage GPU performance for you. It won't. All that line does is reserve access to a GPU as far as the scheduler is concerned.

It doesn't guarantee good utilization. It doesn't guarantee you have enough VRAM. It won't autoscale on GPU usage, it won't save you from bad batching, and it won't make your inference server efficient on its own.

A pod can grab a whole GPU and then use 5% of it. Kubernetes still counts that GPU as fully allocated and hands you the bill. That's why weak platform tooling makes GPU infrastructure so expensive. The hardware is busy on paper and idle in reality.

By default, no. Extended resources are integer resources and Kubernetes doesn't overcommit them. The device plugin docs are clear that in the basic model a device can't be split across containers. So what you get out of the box is:

Copy
1 pod requests 1 GPU
1 full GPU is allocated
no other pod can touch that advertised GPU

In practice, plenty of workloads don't need a whole card:

Copy
small embeddings model
small image model
internal batch job
low-QPS inference endpoint

So teams reach for sharing. NVIDIA gives you two main options: time-slicing and MIG.

Time-slicing lets workloads interleave on an oversubscribed GPU. It's simple, but NVIDIA is upfront that it gives you no memory or fault isolation between the replicas sharing the card. If one workload misbehaves, everyone on that GPU feels it.

MIG, on the cards that support it, splits one physical GPU into smaller hardware-isolated instances. Stronger boundaries, at the cost of only working on certain hardware.

A rough way to hold it in your head:

Copy
Full GPU allocation:
  strong isolation
  simplest to reason about
  wasteful if the workload is small

Time-slicing:
  good sharing
  weak isolation
  fine for small or bursty workloads

MIG:
  hardware-level partitioning
  better isolation than time-slicing
  only on supported GPUs

If you're just starting, use full GPU allocation. Get comfortable with that first, then earn the right to complicate things with sharing.

OpenAI has written about running Kubernetes across thousands of nodes for large models and fast research iteration. In their scaling post they described pushing clusters to 7,500 nodes, and the point they kept coming back to was that researchers got infrastructure that scaled without them rewriting their code.

That's the real reason Kubernetes matters for AI infra, and it has nothing to do with Kubernetes being magic. It gives you one control plane for a set of boring but essential jobs:

Copy
submit workload
request resources
schedule on the right machine
collect logs
restart failed jobs
scale infrastructure
standardize deployment

For an AI team, that's the difference between researchers SSH-ing into random GPU boxes forever and having an actual platform. Past a certain size you need the platform, and Kubernetes ends up being it. Just don't confuse "we run Kubernetes" with "GPU scheduling is solved." GPU Kubernetes is a different operational problem, not normal Kubernetes with pricier nodes.

Don't stop at "it ran." Poke at it the way you would if it were on fire.

1. Check the node's allocatable GPUs

bash
Copy
1kubectl describe node <gpu-node-name> | grep -A5 -i allocatable

You're looking for:

Copy
nvidia.com/gpu: 1

If it's not there, the device plugin isn't advertising anything.

2. Confirm where the pod landed

bash
Copy
1kubectl get pod cuda-gpu-test -o wide

Make sure it's actually on a GPU node.

3. Describe the pod

bash
Copy
1kubectl describe pod cuda-gpu-test

Read the Node, Events, and Limits fields. If the pod is stuck pending, the events almost always tell you why. The classic one is:

Copy
Insufficient nvidia.com/gpu

which means no node had a free GPU to give it.

4. Check GPU visibility from inside the container

bash
Copy
1kubectl logs cuda-gpu-test

If nvidia-smi printed a GPU, the container can see it. If it failed, work through the usual suspects:

Copy
driver not installed
NVIDIA runtime not configured
device plugin not running
wrong container image
pod scheduled on the wrong node

5. Check the device plugin itself

bash
Copy
1kubectl get pods -A | grep nvidia

You should see the plugin running on your GPU nodes. If it's missing, the node can have a GPU bolted in and Kubernetes still won't expose nvidia.com/gpu.

A production GPU cluster is a lot more than one YAML file. You end up caring about all of this:

Copy
driver versions
CUDA compatibility
container runtime setup
GPU node labels
taints and tolerations
GPU health checks
pod scheduling constraints
GPU utilization metrics
model memory usage
batching
autoscaling
cost per request
failure recovery

For training, add:

Copy
multi-GPU placement
node locality
network bandwidth
checkpointing
job retries
distributed training coordination

For inference, add:

Copy
tokens per second
time to first token
p95 latency
GPU memory pressure
batch size
queue depth
cold starts
model loading time

This is roughly where the line sits between "I deployed a pod" and "I run GPU infrastructure."

If you remember nothing else:

Copy
CPU and memory are native Kubernetes resources.
GPUs are extended hardware resources.
A device plugin advertises them.
The scheduler places pods based on availability.
The kubelet and plugin make the device usable inside the container.

That's GPU scheduling in one paragraph.

I'd keep it to three steps.

Deploy the NVIDIA device plugin on a GPU node. Run a small CUDA container that asks for one GPU in limits. Then walk the whole scheduling path and watch what each command tells you:

bash
Copy
1kubectl describe node
2kubectl get pod -o wide
3kubectl describe pod
4kubectl logs

And when you write it up, don't just paste the final working YAML. Write down the journey, because that's where the understanding lives:

Copy
Before the plugin, the node showed no nvidia.com/gpu.
After the plugin, it advertised nvidia.com/gpu: 1.
After requesting the GPU in limits, the pod only scheduled on the GPU node.
Inside the container, nvidia-smi confirmed access.

Most write-ups skip the middle and jump straight to the answer. The middle is the part worth reading.

Kubernetes doesn't understand GPUs the way people assume. It understands resources. A GPU only becomes a Kubernetes resource once a device plugin exposes it, and once it's exposed you request it through limits, usually like this:

yaml
Copy
1resources:
2  limits:
3    nvidia.com/gpu: 1

From there the scheduler can place the pod. But scheduling is only the first step. Running GPU workloads well means owning the whole loop:

Copy
advertise → schedule → allocate → run → monitor → optimize

That loop is the real foundation. The YAML is just where it starts.