How DeepSeek serves V3/R1 on H800s, and what its 545% margin actually means

A 545% margin on inference sounds strange if your starting assumption is that serving models loses money.

Yet that is the theoretical cost-profit margin DeepSeek reported in its March 1, 2025 inference system overview, published on GitHub during Open Source Week.

The calculation: assume an NVIDIA H800 costs $2 per GPU-hour. Daily GPU cost comes to about $87K. Price all the tokens served at R1 rates, and theoretical daily revenue reaches about $562K.

Before getting into the architecture, we need to deal with the word doing the most work here: theoretical.

01The caveat belongs next to the number

DeepSeek explicitly says its actual revenue is substantially lower.

Its web and app are free. V3 is priced lower than R1. Night-time discounts reduce what customers pay during off-peak hours.

The calculation takes traffic across those services and asks what it would generate if every token were billed at R1 pricing. That is a hypothetical billing scenario, not reported revenue.

There is also a denominator worth watching. DeepSeek's cost-profit margin measures the difference between theoretical revenue and GPU cost relative to GPU cost: ($562K − $87K) / $87K ≈ 545%. It is not a profit margin measured against revenue, and GPU cost alone is not the full cost of running the service.

I read the number as an invitation to inspect the serving architecture. How does DeepSeek get enough useful work out of its H800s to support that calculation?

The answer involves sparse computation, expert placement, separate serving stages, caching, and several kinds of load balancing.

02Mixture of experts reduces the work per token

V3/R1 has 256 routed experts per layer. Each token activates only 8 of them.

Most routed experts therefore sit idle for that particular token. The model has access to a large collection of experts without running every token through all of them.

That distinction matters when thinking about inference cost. Model size alone does not describe how much expert computation happens for each token.

But sparsity introduces a scheduling problem.

Imagine a batch of tokens arriving together. Each token needs its selected experts. Once those tokens are divided among experts, an individual expert might receive relatively little work.

DeepSeek's overview makes this connection explicit: high sparsity requires a large overall batch so that each expert receives enough tokens to process efficiently.

This is where the serving system starts to matter as much as the model structure.

Activating fewer experts reduces work per token. Feeding those experts efficiently requires gathering and distributing enough work across the system. You can save computation and still waste hardware if that distribution is poor.

03Spread the experts out, then pay for the network

DeepSeek uses expert parallelism across GPUs and nodes.

Instead of placing every expert on every GPU, it spreads experts across many GPUs. Each GPU holds a small subset. Tokens travel to the GPUs holding their selected experts.

For a backend engineer, the placement problem looks familiar. Work has destinations determined by the data it needs, and those destinations live on different machines.

There is a benefit beyond fitting the experts somewhere. DeepSeek says having each GPU process fewer experts reduces its memory access demands. Large-scale expert parallelism also supports the larger batches needed to feed the experts efficiently.

The catch is communication.

A token's expert computation can require crossing the network. Spreading work more widely creates more opportunities for communication to become part of the critical path.

DeepSeek addresses this by overlapping communication with computation. The useful idea is to keep computation moving while communication for other work happens.

That does not make the network free. It makes the execution schedule important. An efficient expert placement still needs an efficient path through the machines holding those experts.

04Prefill and decode get separate machines

Reading a prompt and generating a response are different workloads.

Prefill processes the prompt and produces the state needed to continue generation. Decode generates tokens using that state.

DeepSeek separates these stages onto different machines, with different parallelism configurations:

The KV cache is handed from prefill to decode so generation can continue from the processed prompt.

The engineering reason for separating them is that the stages have different work to distribute. They do not need to share a configuration just because they belong to the same request.

DeepSeek's balancing objectives make this concrete. Prefill needs to account for input token counts and attention computation. Decode needs to account for request counts and KV cache usage.

Counting requests alone misses those differences. Requests with different prompt lengths do not create equal prefill work. Decode placement also has to consider the state associated with ongoing generation.

The handoff adds coordination, but it gives each stage its own placement and balancing decisions. That is the tradeoff I find interesting: more distributed-system machinery in exchange for control over workloads with different bottlenecks.

05Load balancing happens at several levels

An evenly distributed request count does not guarantee evenly distributed GPU work.

DeepSeek describes separate load balancers for prefill, decode, and expert parallelism. They address different sources of imbalance.

The prefill balancer considers attention computation and input token distribution. The decode balancer considers KV cache usage and request distribution. The expert balancer deals with uneven demand for experts.

That last problem is easy to miss.

You can distribute incoming requests evenly and still overload a GPU because many tokens select experts hosted there. Other GPUs may have spare capacity, but that capacity is in the wrong place.

DeepSeek duplicates hot experts onto additional GPUs, allowing their work to be spread across more locations.

This is close to a hot-key problem in a backend service. The initial placement can be balanced while the access pattern is skewed. Once demand concentrates, the mapping between work and capacity needs to change.

The objective is to prevent the busiest GPU from slowing the wider system. Average utilization can hide that bottleneck. The overloaded destination is what matters to work waiting on it.

06Reuse prompt work and move capacity at night

Expert scheduling is only part of the picture. DeepSeek also avoids recomputing input it can reuse.

About 56% of input tokens in the reported period hit an on-disk KV cache.

That is a large share of input processing served from previously computed state. When reading throughput numbers, this distinction matters: serving an input token from cache and computing its state again are different amounts of work.

For me, this is a reminder to ask what a throughput metric includes. Cache reuse is useful system behavior, but the hit rate belongs beside the result if you want to understand it.

DeepSeek also changes how it allocates nodes over the day.

During peak daytime traffic, all nodes serve inference. At night, when traffic drops, it reduces inference capacity and moves nodes to research and training.

The cluster allocation follows demand. Capacity that is no longer needed for serving can do other work.

That choice belongs in the economics discussion too. Efficient inference depends on both processing requests well and deciding how much hardware should be assigned to serving at a given time.

07What I built to understand hot experts

I wrote a Go simulation to isolate the expert-routing problem.

It routes tokens to experts placed across simulated GPUs. I wanted to see what happens when the distribution of expert demand changes while the available GPUs stay the same.

With demand spread across experts, work spreads across their hosting GPUs. Then I make a few experts hot.

Throughput drops.

The GPUs hosting those experts become bottlenecks. Spare capacity elsewhere does not help enough because the tokens need those particular experts.

The throughput change under concentrated expert demand: result pending.

Next, I duplicate the hot experts onto additional simulated GPUs and route work across those copies. Throughput recovers as the concentrated work gets more places to execute.

The result after duplicating hot experts: result pending.

This is a simulation of routing and contention. It does not reproduce DeepSeek's GPU execution or establish its production performance.

What it makes visible is the placement problem: having spare capacity somewhere in the system does not mean the work causing congestion can use it. Expert replication changes that mapping.

08What I take back to backend work

I would look for the same problems in an ordinary backend: hot keys that overwhelm an otherwise balanced fleet, stages whose different bottlenecks justify separate capacity, and repeated work that can be cached. DeepSeek's theoretical revenue calculation needs its caveat attached. Its serving design gives me concrete scheduling and placement decisions to examine in my own systems.