A common DynamoDB misunderstanding is: “My table has 1,000 WCU provisioned. Why am I getting throttled at only 100 writes/sec?”
The answer is partition-level limits.
DynamoDB does not distribute traffic magically evenly. Your provisioned capacity is divided across physical partitions, and real traffic is never perfectly uniform.
For example:
Table capacity: 1,000 WCU
10 physical partitions
Each partition:
100 WCUThe theoretical capacity is still:
10 partitions × 100 WCU = 1,000 WCUBut imagine this traffic distribution:
Partition 1 → 100 writes/sec ❌ throttling
Partition 2 → 20 writes/sec
Partition 3 → 10 writes/sec
Partition 4 → 5 writes/sec
...
Partition 10 → 2 writes/secThe table has hundreds of unused WCUs, but the hot partition cannot accept more traffic. The result is that the table may have 900 WCU available overall while the hot key range has only 100 WCU available.
This is the hot partition problem.
01The evolution of DynamoDB capacity management
Amazon did not solve this problem overnight. The system evolved through multiple generations of capacity management.
02Burst capacity
Normally, unused capacity is not immediately lost. If a table is provisioned for 1,000 WCU but consumes only 400 WCU, DynamoDB can retain some unused capacity and let partitions temporarily consume from that burst pool during a spike.
Minute 1:
Provisioned: 1000 WCU
Used: 400 WCU
Unused: 600 WCU
Later traffic spike:
Required: 1500 WCU
Normal: 1000 WCU
Burst: +500 WCU
Total: 1500 WCUThe limitation is that burst capacity is still constrained by partition ownership. A hot partition can hit its local limit even while other partitions have unused capacity.
03Adaptive capacity
Adaptive capacity improves on the assumption that every partition should be treated identically. DynamoDB observes traffic patterns and can give a consistently hot partition more of the table's available capacity.
Before:
Partition A: 100 WCU limit
Partition B: 100 WCU limit
Partition C: 100 WCU limitTraffic:
Partition A: 300 WCU needed
Partition B: 20 WCU
Partition C: 10 WCUAdaptive capacity can shift resources toward Partition A. The system learns that it is consistently hot and gives it more capacity when the overall table has room.
There is still a fundamental limitation: the system has to observe the pattern first.
Traffic spike
↓
Partition overloaded
↓
Throttle happens
↓
Adaptive capacity reactsThe first requests in a sudden spike can still suffer. That leads to a deeper question: why should requests reach a partition if the system already knows the table is overloaded?
04Admission control moves the decision earlier
Instead of letting every partition independently reject traffic, a router can decide whether a request should enter the system before it reaches the partition.
Before:
Client → Partition → “Sorry, throttled”With admission control:
Client → Router admission control → PartitionThe router has a broader view of table capacity. It can protect the system at the entry point and make capacity decisions before a partition becomes overloaded.
05Token-bucket based admission control
The idea comes from token buckets, a familiar rate-limiting algorithm. Every request requires a token, and tokens are continuously refilled at a configured rate.
Bucket: [Token][Token][Token][Token][Token]
Refill rate: 1000 tokens/secEach request consumes one token. If the bucket is empty, the request waits or is throttled. In a DynamoDB-style design, the router maintains a table-level view of available tokens:
Table capacity: 1,000,000 tokens/sec
Request
↓
Router: does the table have tokens?
↓ yes
PartitionThis does not make capacity infinite, but it makes overload behavior deliberate and visible before traffic reaches the storage layer.
06Why table-level admission works better
Previously, partitions had a fragmented view:
Partition A: “I am full”
Partition B: “I am empty”
No communication.A global router can see that Partition A is hot while Partition B has spare capacity. It can allow more traffic when the table has room and reject excess traffic early when the table is genuinely at its limit.
The system stops wasting available capacity simply because it is stranded behind an overloaded partition.
07Prime Day scale
At Amazon scale, small inefficiencies become huge failures. During Prime Day 2021, DynamoDB handled a reported peak of 89.2 million requests per second.
At that scale, a 1% mistake means roughly 890,000 requests per second are affected. Admission control is not a minor optimization when the system is operating at that volume; it is part of making overload predictable.
08Predictable systems beat optimized systems
One of the most interesting engineering decisions in a large distributed system is sometimes about metadata caching rather than raw capacity.
Distributed databases constantly need metadata:
- Which partition owns this key?
- Where should this request go?
- What configuration applies?
Naturally, you build a cache. A cache hit is extremely fast, but high cache hit rates can create a dangerous failure mode.
09The 99.75% cache hit trap
Imagine a normal workload:
1000 requests/sec
995 requests → cache hit
5 requests → metadata serviceThe metadata service sees only 5 requests per second. Everything looks healthy.
Then the cache fails:
1000 requests/sec → metadata serviceThat is a 200x traffic spike caused by a small cache problem. A failure in an optimization suddenly becomes an outage in the system the cache was protecting.
10Always calling the metadata service
The counterintuitive solution is to keep the metadata service in the request path even on a cache hit. The cache still provides the fast answer, while the metadata service validates that the answer remains correct.
Request
↓
Metadata cache
↓
Metadata service
↓
PartitionThe goal is not to avoid every unit of work. It is to keep the downstream load approximately constant.
Before:
Normal: 5 requests/sec
Failure: 1000 requests/sec
After:
Normal: 1000 requests/sec
Failure: 1000 requests/secDoing unnecessary work forever can be cheaper than handling an unpredictable spike.
11A simplified implementation
A small implementation can model the same ideas with four components:
+-------------+
| Client |
+------+------+
|
v
+----------------+
| Router Layer |
| Token Bucket |
+-------+--------+
|
+-----------+-----------+
| |
v v
+-------------+ +-------------+
| Partition 1 | | Partition 2 |
+-------------+ +-------------+Router: tracks table capacity, maintains the token bucket, rejects excess traffic early, and routes requests.
Token bucket: for example, a capacity of 10,000 tokens with a refill rate of 1,000 tokens per second. Each write consumes one token.
Metadata service: stores partition ownership, such as User#123 → Partition 5 and Order#456 → Partition 2.
Metadata cache: reduces latency for hot keys while background validation keeps downstream load predictable.
12Measuring the improvement
Without admission control, a hot partition can dominate the result:
Traffic: 1000 req/sec
Hot partition limit: 100 req/sec
Throttle: 90%With a router distributing work across four partitions:
Partition 1: 250 req/sec
Partition 2: 250 req/sec
Partition 3: 250 req/sec
Partition 4: 250 req/secThe throttle rate can approach zero, assuming the partitions have enough capacity and the keys are distributed evenly. The measurement should include per-partition utilization, admission rejections, end-to-end latency, and the number of requests served from stale or refreshed metadata.
13Final takeaway
DynamoDB's scaling story is not just about adding more machines. The deeper lesson is that large distributed systems fail at the boundaries between components.
A partition does not know the whole table. A cache does not know future traffic. A router can make an earlier decision with a broader view.
The biggest systems move intelligence closer to the entry point. Sometimes the best optimization is doing extra work forever, because predictable systems survive unpredictable traffic.
References: Amazon DynamoDB adaptive capacity documentation; Amazon Prime Day DynamoDB scaling architecture discussions; token bucket rate-limiting algorithms; AWS re:Invent distributed database talks.
