Before You Scale LLMs: The NVIDIA GPU Operator and DCGM Layer

When people talk about running LLMs, they go straight to the interesting part. Tensor parallelism. Batching. KV caches. Throughput numbers you can put on a slide.

Fair enough, that's the part that feels like engineering. But none of it runs on a box that can't see its own GPU, and that's the part nobody wants to think about. Before any of the serving logic matters, you have to get a much less exciting layer working:

• NVIDIA drivers
• CUDA access inside containers
• Kubernetes device discovery
• GPU scheduling
• GPU node labels
• Health validation
• GPU monitoring

I'm calling it boring because it is, but this is the layer that decides whether your expensive cluster behaves like infrastructure or like eight machines that fail in eight different ways. Get it wrong and you'll spend your first week of "LLM work" debugging why a pod says nvidia.com/gpu doesn't exist.

This post walks through how the NVIDIA GPU Operator manages that layer, and how DCGM Exporter lets you actually see what your GPUs are doing.

Say you rack a new GPU worker and join it to your cluster. The card is physically there. Kubernetes has no idea.

It doesn't know what kind of GPU it is, whether the right driver is installed, whether containers can touch it, how many are available, whether any of them are healthy, which pod is using one, how much memory is in use, whether it's cooking itself, or whether the workload is even doing anything with it. None of that comes for free.

The way Kubernetes handles specialized hardware is the device plugin framework. A vendor ships a device plugin that advertises its hardware to the kubelet, and Kubernetes then exposes that hardware as a schedulable resource. For NVIDIA cards the resource name is nvidia.com/gpu.

Until those NVIDIA components exist, this pod spec is just text:

yaml
Copy
1resources:
2  limits:
3    nvidia.com/gpu: 1

The scheduler can only honor that request after the GPU has been discovered and registered as an extended resource. Several pieces have to line up for that to happen.

When a container runs a CUDA workload, the call travels down through a stack:

Copy
LLM application
      ↓
CUDA libraries inside the container
      ↓
NVIDIA Container Toolkit
      ↓
Container runtime
      ↓
NVIDIA kernel driver
      ↓
Physical GPU

Kubernetes wraps another set of components around that:

Copy
Kubernetes scheduler
      ↓
NVIDIA Device Plugin
      ↓
GPU resource: nvidia.com/gpu
      ↓
NVIDIA Container Toolkit
      ↓
Driver
      ↓
GPU

And monitoring adds its own path:

Copy
GPU
 ↓
NVIDIA DCGM
 ↓
DCGM Exporter
 ↓
Prometheus
 ↓
Grafana

Three stacks, and every one of them has to be reliable before you put production traffic on the thing. That's what the rest of this is about.

A Kubernetes operator is a controller that keeps some application in a desired state. The GPU Operator does that for the whole NVIDIA stack. Instead of hand-installing and babysitting every GPU component on every node, you install one operator and declare how GPU nodes should look.

It manages a long list of things for you:

• NVIDIA GPU drivers
• NVIDIA Container Toolkit
• NVIDIA Kubernetes Device Plugin
• GPU Feature Discovery
• Node Feature Discovery
• DCGM
• DCGM Exporter
• MIG Manager
• Validation workloads

NVIDIA's own framing is that the operator automates the NVIDIA components needed to provision GPUs in Kubernetes: drivers, the container toolkit, device plugins, GPU labels, and DCGM monitoring. A default install today lays down the driver, Container Toolkit, Device Plugin, DCGM Exporter, and MIG Manager on your GPU workers.

The part that matters is that it does this continuously. It isn't a one-shot install script. It watches the cluster and reconciles reality against what you declared. Add a new GPU node next month and the operator configures the NVIDIA components on it without you touching the machine. That property is the entire reason to use it instead of Ansible and hope.

This is the lowest software layer that talks to the card. Your application never speaks to the hardware directly. CUDA calls make their way down to the kernel driver, and the driver submits the work to the GPU. If the driver is missing or the versions don't match, everything above it falls over.

First thing I check on any GPU box:

bash
Copy
1nvidia-smi

If that works, the OS can talk to the card. But here's the trap that catches everyone at least once: nvidia-smi working on the host does not mean your containers can use the GPU. The host driver can be perfectly fine while the container runtime still has no idea how to hand the device and libraries into a container. That gap is exactly what the next component exists to close.

Containers can't see host devices by default. That's the whole point of a container. The Container Toolkit configures the container runtime so a GPU workload actually gets what it needs:

• GPU device files
• NVIDIA driver libraries
• Runtime hooks
• Container Device Interface configuration

Recent GPU Operator releases enable CDI by default for device injection, though the exact behavior depends on your operator version and runtime config. The mental model is simple: the toolkit is the bridge between a normal OCI container and one that can see a GPU. Skip it and your container will start up fine, then CUDA will politely tell you there's no compatible device. That error message has cost the industry a lot of collective hours.

Kubernetes has no built-in notion of NVIDIA hardware. The Device Plugin runs on GPU nodes and registers the cards with the kubelet. Once that's done, the node advertises something like:

yaml
Copy
1status:
2  allocatable:
3    nvidia.com/gpu: "4"

Now the scheduler understands the node has four GPUs to hand out. The plugin is what turns a physical card into a Kubernetes resource, so a workload can ask for one:

yaml
Copy
1resources:
2  limits:
3    nvidia.com/gpu: 1

With the classic device-plugin model you specify GPUs as limits, and the plugin gives the allocated card to the chosen container.

Now the part junior engineers consistently miss: the Device Plugin does not care whether your app uses the GPU well. It only handles discovery, allocation, and access. Allocation and utilization are two completely different problems. A pod can reserve an entire H100 and then use five percent of it while you pay for the whole thing. The plugin will report that GPU as fully allocated and be perfectly correct. Spotting the waste is DCGM's job, which is why we get to it later.

Real clusters are not uniform. You'll have CPU-only nodes, L4 nodes, A100 nodes, H100 nodes, some with MIG, some with NVLink, some with more memory than others. The scheduler needs labels to tell them apart.

Node Feature Discovery detects hardware features and labels the nodes. GPU Feature Discovery adds the NVIDIA-specific detail. GPU workers get picked out using NVIDIA's PCI vendor ID, and the operator uses that to decide where its GPU operands should run. Those labels are what let you write node selectors and affinity rules:

yaml
Copy
1spec:
2  nodeSelector:
3    nvidia.com/gpu.product: NVIDIA-L4

The exact label depends on your GPU Feature Discovery version and the actual hardware, so do not copy a label out of a blog post (including this one) and assume it exists. Look at the node:

bash
Copy
1kubectl get node <gpu-node-name> --show-labels

Or:

bash
Copy
1kubectl describe node <gpu-node-name>

Verify, then use. Every time.

DCGM is the Data Center GPU Manager. It handles monitoring and management for NVIDIA data-center GPUs. DCGM Exporter takes those metrics and turns them into Prometheus format, served over an HTTP /metrics endpoint that Prometheus can scrape. It runs as a standalone container or, more commonly, as a DaemonSet across your GPU nodes.

What you get out of it:

• GPU utilization
• GPU memory used and free
• GPU temperature
• Power consumption
• SM clock and memory clock
• PCIe and NVLink traffic
• Tensor and DRAM activity
• XID errors
• Energy consumption

The metric names you'll actually query:

Copy
DCGM_FI_DEV_GPU_UTIL
DCGM_FI_DEV_FB_USED
DCGM_FI_DEV_FB_FREE
DCGM_FI_DEV_GPU_TEMP
DCGM_FI_DEV_POWER_USAGE
DCGM_FI_DEV_XID_ERRORS
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE
DCGM_FI_PROF_DRAM_ACTIVE

Depending on how it's configured and what Kubernetes can map, the exporter can attach dimensions like GPU ID, GPU UUID, hostname, namespace, pod, and container to each metric. That mapping is what lets you go from "GPU 3 is idle" to "the pod on GPU 3 is idle," which is the question you'll actually be asking at 2am.

Before you install anything, make sure you have a working cluster, at least one NVIDIA GPU worker, kubectl access, Helm, a supported runtime like containerd or CRI-O, network access to pull the images, and an OS and Kubernetes combination NVIDIA actually supports.

That last point is not a formality. GPU Operator releases pin specific versions of the driver, device plugin, Container Toolkit, DCGM, and the rest. Check the current platform support and release notes before a production install. I've watched people lose a day to a driver that didn't match their kernel.

The operator can install NFD for you, but installing it twice causes conflicts. NVIDIA recommends checking for existing NFD labels first:

bash
Copy
1kubectl get nodes -o json |
2jq '.items[].metadata.labels |
3keys |
4any(startswith("feature.node.kubernetes.io"))'

If that returns true, install with NFD disabled:

bash
Copy
1--set nfd.enabled=false

bash
Copy
1helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
2helm repo update

NVIDIA's docs currently use GPU Operator v26.3.3. Swap in whatever version you've actually validated for your environment.

bash
Copy
1helm install gpu-operator \
2  --wait \
3  --namespace gpu-operator \
4  --create-namespace \
5  nvidia/gpu-operator \
6  --version v26.3.3

If your cluster enforces Pod Security Admission, the operator namespace probably needs privileged enforcement, because the driver and device-management components need host-level access:

bash
Copy
1kubectl create namespace gpu-operator
2
3kubectl label --overwrite namespace gpu-operator \
4  pod-security.kubernetes.io/enforce=privileged

Run that before the Helm install if your security config requires it.

Some managed platforms and custom images ship the NVIDIA driver before the node ever joins the cluster. In that case, tell the operator not to install its own:

bash
Copy
1helm install gpu-operator \
2  --wait \
3  --namespace gpu-operator \
4  --create-namespace \
5  nvidia/gpu-operator \
6  --version v26.3.3 \
7  --set driver.enabled=false

Do not reach for driver.enabled=false on reflex. Only use it when you know who owns the host driver and how upgrades to it will be handled. Otherwise you've just created a driver nobody is responsible for.

A Helm chart installing cleanly proves that Helm ran. It proves nothing about your GPUs. Validate each layer.

bash
Copy
1kubectl get pods -n gpu-operator

You should see something like:

Copy
gpu-feature-discovery
gpu-operator
node-feature-discovery
nvidia-container-toolkit-daemonset
nvidia-cuda-validator
nvidia-dcgm-exporter
nvidia-device-plugin-daemonset
nvidia-driver-daemonset
nvidia-operator-validator

Most should be Running or Completed. The exact list shifts with your config and operator version.

The operator uses a ClusterPolicy custom resource to hold the desired GPU config.

bash
Copy
1kubectl get clusterpolicy

You want:

Copy
NAME             STATUS
cluster-policy   ready

If it's not ready, describe it and read the events:

bash
Copy
1kubectl describe clusterpolicy cluster-policy

bash
Copy
1kubectl get nodes -o json |
2jq '.items[] |
3select(.status.allocatable["nvidia.com/gpu"] != null) |
4{
5  name: .metadata.name,
6  gpus: .status.allocatable["nvidia.com/gpu"]
7}'

Expected:

json
Copy
1{
2  "name": "gpu-worker-01",
3  "gpus": "1"
4}

The Device Plugin registers GPUs under the extended resource nvidia.com/gpu. If that resource isn't there, the scheduler can't place a single GPU request, no matter how correct your pod spec is. You can also read it off the node directly:

bash
Copy
1kubectl describe node gpu-worker-01

Looking for:

Copy
Capacity:
  nvidia.com/gpu: 1

Allocatable:
  nvidia.com/gpu: 1

Green pods are reassuring. They are not proof. The real test is whether a CUDA kernel runs.

Create cuda-vectoradd.yaml:

yaml
Copy
1apiVersion: v1
2kind: Pod
3metadata:
4  name: cuda-vectoradd
5spec:
6  restartPolicy: OnFailure
7
8  containers:
9    - name: cuda-vectoradd
10      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
11
12      resources:
13        limits:
14          nvidia.com/gpu: 1

Apply it, see where it landed, read the logs:

bash
Copy
1kubectl apply -f cuda-vectoradd.yaml
2kubectl get pod cuda-vectoradd -o wide
3kubectl logs cuda-vectoradd

A good run ends with:

Copy
Test PASSED
Done

That one line quietly confirms the entire chain: Kubernetes discovered the GPU, the scheduler placed the pod on a GPU node, the Device Plugin allocated the card, the runtime injected it, the container reached the host driver, and a CUDA kernel executed. NVIDIA uses this same VectorAdd job as the install verification test in their docs. If it passes, your foundation is real.

A default install turns DCGM Exporter on unless you explicitly disabled it. Find its pods:

bash
Copy
1kubectl get pods -n gpu-operator |
2grep dcgm-exporter

Since it usually runs as a DaemonSet, you get one exporter pod per eligible GPU node. Confirm that and check its logs:

bash
Copy
1kubectl get daemonset -n gpu-operator |
2grep dcgm
3
4kubectl logs \
5  -n gpu-operator \
6  <dcgm-exporter-pod-name>

Port-forward an exporter pod:

bash
Copy
1kubectl port-forward \
2  -n gpu-operator \
3  pod/<dcgm-exporter-pod-name> \
4  9400:9400

Then in another terminal:

bash
Copy
1curl http://localhost:9400/metrics

Grep for what you care about:

bash
Copy
1curl -s http://localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL
2curl -s http://localhost:9400/metrics | grep DCGM_FI_DEV_FB_USED

At this point you've proven the exporter works. You have not proven Prometheus is scraping it. Keep those two facts separate in your head, because conflating them is how people waste an afternoon debugging Grafana when the actual problem is a service selector.

The path you're building is:

Copy
DCGM Exporter
      ↓
Prometheus scrape
      ↓
Prometheus time-series storage
      ↓
Grafana dashboard

DCGM Exporter hooks into Kubernetes pod resource info and serves GPU metrics in a format Prometheus understands. When you're running Prometheus Operator, NVIDIA's telemetry docs describe using a ServiceMonitor to pick the exporter up.

How you finish the wiring depends entirely on how Prometheus got installed in your cluster. Common setups: kube-prometheus-stack, standalone Prometheus, a managed Prometheus, Grafana Alloy, VictoriaMetrics Operator, or an external monitoring cluster. Start by looking at what the operator actually created:

bash
Copy
1kubectl get service -n gpu-operator | grep dcgm
2
3kubectl get servicemonitor --all-namespaces | grep -i dcgm

Then go into the Prometheus UI, find the DCGM Exporter target, and confirm its state is UP. Once it is, run:

promql
Copy
1DCGM_FI_DEV_GPU_UTIL

If that returns data, the whole collection path is alive. Now you can build dashboards on top of something you trust.

A dashboard should answer operational questions. It should not be a wall of every metric the exporter can produce, because a dashboard nobody can read is just decoration. Here's the starter set I'd build.

promql
Copy
1DCGM_FI_DEV_GPU_UTIL

How busy the card is. For a smoother view:

promql
Copy
1avg_over_time(DCGM_FI_DEV_GPU_UTIL[5m])

A GPU that's allocated but consistently idle is telling you something: an input pipeline bottleneck, slow CPU preprocessing, batches that are too small, network latency, poor concurrency, too much synchronization, or a card you didn't need to provision in the first place. The number doesn't diagnose the cause, but it tells you to go looking.

promql
Copy
1DCGM_FI_DEV_FB_USED

Framebuffer usage, which is what people mean when they say VRAM. High usage is not automatically a problem. An inference server will deliberately hold model weights, CUDA graphs, the KV cache, and memory pools resident. That's it doing its job. The question worth asking is whether growth is expected and stable, or whether something is leaking.

promql
Copy
1DCGM_FI_DEV_GPU_TEMP

Watch for cooling and airflow problems, sustained thermal pressure, a dying fan, or thermal imbalance on dense nodes. Do not copy a temperature threshold off the internet. Use the limits and guidance for your specific card and chassis, because a number that's fine for one platform is an alert on another.

promql
Copy
1DCGM_FI_DEV_POWER_USAGE

Power gives you context the other metrics can't on their own. High utilization plus high power usually means real work is happening. An allocated card with low utilization and low power is probably idle. Low reported utilization but high power is worth a closer look. Repeated power drops can line up with throttling or an unstable workload.

promql
Copy
1DCGM_FI_DEV_XID_ERRORS

XID errors are GPU driver or hardware events, and this is the one metric on the dashboard you should never treat as just another line on a graph. A non-zero XID means go investigate.

What the right response is depends on the specific XID. Some are workload-related. Some want a pod restart, some a GPU reset, some a node drain, and some are just hardware quietly failing. When you're digging in, NVIDIA's troubleshooting docs point you at the kernel log:

bash
Copy
1sudo dmesg | grep -i NVRM
2sudo dmesg | grep -i Xid

Where it's supported and collected:

promql
Copy
1DCGM_FI_PROF_PIPE_TENSOR_ACTIVE

This one is gold for ML workloads. A GPU can look busy overall while barely touching its Tensor Cores. Low Tensor Core activity can be completely expected when the workload isn't matrix-heavy, the model uses an unsupported dtype, the kernels aren't Tensor Core optimized, the job is memory-bound, or the card doesn't support the profiling metric at all.

Which brings me to the one rule that matters most here: never read a single metric in isolation. Correlate Tensor Core activity with utilization, DRAM activity, memory usage, batch size, request rate, latency, and tokens per second. Any one number lies. The pattern across several tells the truth.

A node-level GPU graph is fine, but the question you actually debug against is: which workload is on this card? Depending on config, DCGM Exporter can tag metrics with namespace, pod, container, GPU UUID, node, and GPU index. That's the difference between a graph and an answer.

The operator can also fold selected pod labels into the exported metrics:

bash
Copy
1--set dcgmExporter.enablePodLabels=true

Resist the urge to turn on every label. High-cardinality labels will quietly wreck your Prometheus. These are the kind that generate an unbounded number of time series:

Copy
request_id
job_id
commit_sha
timestamp
session_id

Stick to stable infrastructure dimensions instead:

Copy
team
environment
model
workload
service

NVIDIA recommends restricting labels with an allowlist for exactly this reason. Observability only helps if you can still afford to run it.

Copy
nvidia-device-plugin-daemonset   0/1   Init:0/1
nvidia-dcgm-exporter             0/1   Init:0/1

Do not start with the exporter. A lot of the operator's components wait on the driver and Container Toolkit layers before they'll come up, and NVIDIA notes that operands sit in Init when the driver DaemonSet hasn't started cleanly. Work bottom-up:

bash
Copy
1kubectl get pods -n gpu-operator
2
3kubectl logs -n gpu-operator <driver-pod-name> -c nvidia-driver-ctr
4kubectl logs -n gpu-operator <driver-pod-name> -c k8s-driver-manager

Fix the driver and the rest usually unblocks itself.

Check the Device Plugin, then the driver behind it:

bash
Copy
1kubectl get pods -n gpu-operator | grep device-plugin
2kubectl logs -n gpu-operator <device-plugin-pod-name>
3
4kubectl exec -n gpu-operator <driver-pod-name> \
5  -c nvidia-driver-ctr -- nvidia-smi

Likely culprits: a failed driver install, a failed Device Plugin, an unsupported GPU or platform, wrong node labels, a GPU passed incorrectly into the VM, a broken runtime config, or a card that's genuinely unhealthy.

Copy
no runtime for "nvidia" is configured

This almost always means the Container Toolkit didn't configure the runtime properly. Read its logs, then inspect the runtime config:

bash
Copy
1kubectl logs -n gpu-operator <container-toolkit-pod-name> \
2  -c nvidia-container-toolkit-ctr
3
4containerd config dump

Newer CDI-based deployments and older runtime-handler setups differ here, so troubleshoot against your installed operator version rather than a generic guide. NVIDIA's troubleshooting docs classify this as a Container Toolkit or runtime problem.

Good news: your GPU layer is probably fine and the problem is the monitoring integration. Walk it one hop at a time:

1. Does the Service exist?
2. Does its selector match the exporter pods?
3. Does a ServiceMonitor exist?
4. Does Prometheus select that ServiceMonitor?
5. Is the target visible in Prometheus?
6. Is it UP?
7. Can Prometheus reach port 9400?
8. Is a NetworkPolicy blocking the scrape?
9. Are the expected metrics even enabled in the DCGM config?

Copy
1Exporter process
2Exporter Pod
3Kubernetes Service
4ServiceMonitor or scrape config
5Prometheus target
6Prometheus query
7Grafana

Do not open Grafana until you've confirmed the metric exists in Prometheus. Grafana can't show you data that was never scraped.

This one panics junior engineers, and it usually shouldn't. Zero utilization is not automatically a failure. The app might still be loading weights. The pod might be waiting for requests. CPU preprocessing or storage might be slow. The batch might be tiny. The workload might be blocked on the network. It might have grabbed the card and not submitted a kernel yet. The metric interval might have simply missed a very short burst. Or the process crashed after allocating and is holding memory while doing nothing.

Read several signals together:

promql
Copy
1DCGM_FI_DEV_GPU_UTIL
2DCGM_FI_DEV_FB_USED
3DCGM_FI_DEV_POWER_USAGE

Then line those up against application-level numbers: requests per second, queue depth, batch size, time to first token, inter-token latency, tokens per second, model loading state. Infrastructure metrics tell you what the GPU is doing. Application metrics tell you why. You need both.

Pin your versions and test the combination. Never run latest in production. GPU infrastructure has compatibility relationships running from the Linux kernel through the driver, Container Toolkit, CUDA runtime, Kubernetes, container runtime, GPU Operator, Device Plugin, DCGM, and the hardware itself. Validate the exact stack before rollout, not after.

Keep GPU nodes in a dedicated pool. Separating them from general workers makes autoscaling, taints and tolerations, driver upgrades, scheduling, capacity planning, security, draining, and cost reporting all easier to reason about. Taint the nodes:

bash
Copy
1kubectl taint nodes gpu-worker-01 workload=gpu:NoSchedule

And tolerate the taint on GPU workloads:

yaml
Copy
1tolerations:
2  - key: workload
3    operator: Equal
4    value: gpu
5    effect: NoSchedule

Alert on symptoms, not noise. Good early alerts: DCGM Exporter gone, a node missing its expected GPU capacity, repeated XID errors, sustained high temperature, unexpected memory exhaustion, an unavailable Device Plugin, a GPU allocated and idle for a long stretch, or a workload stuck pending because nothing is allocatable. Don't alert on every little utilization dip. Inference traffic is bursty by nature and you'll train your team to ignore the pager.

Write a short runbook. Your team should be able to answer, in order: Is the node ready? Does the host see the GPU? Is the driver loaded? Does Kubernetes advertise nvidia.com/gpu? Is the Device Plugin healthy? Can a test pod run CUDA? Is DCGM Exporter producing metrics? Is Prometheus scraping them? Are there XID errors? And finally, do we restart the pod, reset the GPU, drain the node, or replace the machine? A dashboard tells you something is wrong. A runbook tells you what to do about it. You want both, and the runbook is the one people forget to write.

A junior engineer looks at a GPU node and thinks: I have a GPU, I can deploy an LLM. A production engineer looks at the same node and runs down a different list.

Copy
Can the OS see the GPU?
Can the driver talk to it?
Can the runtime inject it into a container?
Can Kubernetes discover and allocate it?
Can the scheduler pick the right GPU node?
Can we tell which pod owns it?
Can we measure memory, utilization, power, and errors?
Can we respond safely when it fails?

That's the whole job of the GPU Operator and DCGM layer. The operator runs the lifecycle of the NVIDIA stack. The Device Plugin makes GPUs schedulable. GPU Feature Discovery describes the hardware. The Container Toolkit gets the card into your containers. DCGM Exporter makes all of it observable.

Only once that layer is stable do you get to move up to the fun stuff: model serving, autoscaling, batching, MIG, GPU sharing, distributed inference, multi-node training, tensor parallelism, cost optimization.

The boring layer isn't separate from your LLM performance. It's the thing your LLM performance sits on. Build it first, build it well, and most of the mysteries above the line stop being mysteries.