Discord Moved Trillions of Messages in Nine Days. The Database Swap Wasn't the Clever Part.

Everyone tells this story the same way: Discord had a Cassandra problem, they switched to ScyllaDB, latency dropped. End of story. It's a clean arc and it's mostly true — but it buries the two things actually worth learning from.

The first is that Discord fixed most of the pain before they touched the database, by putting a layer of Rust in front of it. The second is that when the official migration tool quoted three months, an engineer decided three months was three months too long, rewrote the migrator in Rust in about a day, and finished the whole thing in nine.

Let me walk through what really happened, because the interesting decisions are the ones that don't fit on a slide.

Discord stores messages in a cluster they called cassandra-messages. In 2017 that was twelve Cassandra nodes holding billions of messages. By the start of 2022 it was 177 nodes holding trillions. (They'd run MongoDB before that, but it fell over early — Cassandra was the grown-up system.)

A 177-node cluster sounds like success, and in a way it was. But it had quietly become a high-toil system. Latency was unpredictable. The on-call team got paged constantly. And some routine maintenance operations had gotten so expensive to run that the team had started skipping them — which is the kind of thing that feels fine right up until it isn't.

The root cause was hot partitions, and to see why, you have to look at how a message is stored. Discord partitions messages by the channel they're sent in, plus a time bucket — a static window of time. Every ID is a Snowflake, so it's chronologically sortable. The practical effect is that every message for a given channel-and-bucket lives together on the same partition, replicated across three nodes.

Now picture a giant server making an announcement. It pings tens of thousands of people. They all open the app in the same few seconds, and all read from the same channel — which means they all hammer the same partition, which lives on the same three replicas. Cassandra had no ceiling on concurrency here, so the requests piled up, and each new query waited behind the growing backlog. Latency didn't just spike — it cascaded. One hot channel could drag the tail latency of the whole cluster.

Two structural facts about Cassandra made this worse. The first is the JVM. Cassandra runs on the JVM, which means garbage collection, which means GC pauses — and GC pauses show up as latency spikes at exactly the wrong moment. The team spent real energy fighting the collector. The second is the storage engine. Cassandra uses an LSM-tree, which makes reads more expensive than writes, leans on compaction (which itself adds latency), and — for Discord specifically — was slow at reverse queries. Discord sometimes needs to scan message history in ascending order, the opposite of the default, and that path was painful.

So you have a workload that reads hot, a runtime that stutters under GC, and a storage engine that's expensive on exactly the access pattern you need. That's not a tuning problem. That's a "we need to change something structural" problem.

Here's the move I want you to internalize. When a database hurts, the instinct — especially as a junior engineer — is to go shopping for a better database. Discord's first real win came from the opposite instinct: protect the database you already have.

They were rightly suspicious that bolting on a new engine would magically fix everything. Hot partitions can happen on any database. So before the migration, they built an intermediary layer that sits between the API monolith and the database clusters — they call it data services — written in Rust and spoken to over gRPC. Its whole reason to exist is to shield the database from the traffic shapes that were hurting it.

That layer does two things, and both matter.

Go back to the announcement. Ten thousand clients ask for the same message in the same instant. Done naively, that's ten thousand identical reads slamming one hot partition. But they're identical. There's no reason to ask the database the same question ten thousand times.

Request coalescing fixes this. The first request to arrive spins up a worker that goes to the database. The other 9,999 requests for that same key don't fire their own queries — they attach to the in-flight one and wait. One query goes out, one answer comes back, and everybody gets the same result. N concurrent reads of the same key collapse into a single database hit. A thundering herd becomes a single footstep.

But coalescing only works if the duplicate requests actually meet each other on the same machine. If they scatter across a dozen data-service instances, each instance still sees "one" request and still hits the database — you've coalesced nothing. So the second job of the data services layer is consistent-hash routing: route by a key like the channel id, so every request for a given channel lands on the same instance. That's what makes the coalescing real.

This is the part people skip when they retell the story, and it's the part that bought Discord time. With coalescing and routing in front of Cassandra, hot partitions got less frequent, the firefighting eased, and the team finally had the breathing room to prepare the migration properly instead of doing it in a panic.

I built a small version of this to feel the difference for myself — a Rust gRPC data service that coalesces concurrent reads of the same key in front of a local database, with a benchmark that runs the same load with coalescing on and off. The prototype is here: https://github.com/Bhup-GitHUB/request-coalescing. The gap between "one query" and "N queries" under a synthetic thundering herd is not subtle.

With the database protected, they could pick the right engine without a gun to their head — and they picked ScyllaDB.

ScyllaDB is Cassandra-compatible, which keeps the migration sane, but it's written in C++ instead of running on the JVM. No JVM means no garbage collector, which means the entire GC-pause latency tail — the thing the team had been fighting for years — simply doesn't exist. On top of that, ScyllaDB uses a shard-per-core architecture: each CPU core owns its own slice of the data and handles its own requests, share-nothing. You get much better isolation between workloads and you avoid the coordination bottlenecks you hit with a multi-threaded JVM design.

It wasn't free. The one real blocker was that reverse-query problem — ascending scans of message history, which Discord depends on, were slow on ScyllaDB at first. Discord worked directly with the ScyllaDB team to make that path fast enough for production. And notice how they de-risked the whole bet: by 2020 they'd already migrated every other database to ScyllaDB and left the messages cluster for last, on purpose. As Bo Ingram put it, they didn't want to learn all their lessons on the messages database. By the time they came for the big one, they'd been running ScyllaDB in anger for two years.

There's also a nice hardware footnote. On GCP, local NVMe SSDs are fast but ephemeral, while persistent disks are durable but slower. Discord wanted both, so they built a hybrid — local SSDs mirrored to a persistent disk via RAID — to get local-SSD speed with persistent-disk durability. They called it the Superdisk, and they tuned ScyllaDB to play nicely with it.

The plan was textbook. Stand up the ScyllaDB cluster. Turn on dual writes, so every new message gets written to both Cassandra and ScyllaDB. Serve recent data from ScyllaDB and backfill the historical messages behind it.

For that backfill, ScyllaDB ships a Spark-based migrator. Discord tuned it and got ready to run. The ETA came back: three months.

By May 2022, Ingram had had enough of Cassandra firefighting, and three months of running two systems in parallel — with all the risk that doubling implies — was three months too long. So instead of waiting, the team rewrote the migrator in Rust by extending the data-service library they already had. The rewrite took Ingram and two colleagues about a day.

The Rust migrator read token ranges out of Cassandra, checkpointed its progress in SQLite so it could resume if it died, and wrote into ScyllaDB with heavy concurrency. It pushed 3.2 million records per second. Three months collapsed to nine days. And going faster actually simplified the plan — it was now quick enough to migrate everything at once, so they didn't have to maintain separate logic for "old" data and "new" data.

The detail I love is the ending. The migration got stuck at 99.9999% complete. The very last token ranges were stuffed with tombstones — deletion markers — and wouldn't move. The fix was to compact those token ranges, and then it finished. Almost done is its own special kind of not done.

To make sure nothing silently corrupted, they mirrored a sample of live read traffic to both databases and compared the responses until they consistently matched. Then they flipped the switch. In May 2022, ScyllaDB became the primary store for Discord messages.

The cluster went from 177 Cassandra nodes to 72 ScyllaDB nodes, with each ScyllaDB node holding around 9 TB of disk versus the roughly 4 TB average per Cassandra node — denser, smaller, calmer. Those figures come straight from Discord's own writeup.

On latency, write p99 went from a jittery 5–70ms on Cassandra to a steady 5ms on ScyllaDB, and historical reads went from 40–125ms down to a chill 15ms p99 — again, Discord's own numbers. You'll see the headline "200 milliseconds to 5 milliseconds" repeated everywhere; that framing comes from the coverage of Ingram's ScyllaDB Summit talk, not the blog, so if you're quoting it, I'd reach for the precise per-operation numbers instead — they're both more defensible and more impressive in their specificity.

But the real result wasn't a number on a dashboard. It was a quiet on-call rotation. Later that year, during the World Cup, message volume spiked hard and the system didn't break a sweat — the kind of non-event that, for the people carrying the pager, is the whole point.

Three things, and they outlast the specific technologies.

First: when a datastore hurts, your first move is not always a new datastore. Discord's biggest early relief came from a layer in front of the database — coalescing and routing — not from swapping the engine. The engine swap mattered, but it came second, after the bleeding had already been stopped. Protect what you have, buy yourself time, and then make the big change deliberately instead of in a fire.

Second: request coalescing is one of the highest-leverage, lowest-cost patterns you'll ever deploy. Collapsing N identical concurrent reads into a single query costs you a worker and a little bookkeeping, and in return it turns your worst traffic shape — everyone wanting the same thing at the same moment — into a non-problem. Reach for it before you reach for more nodes.

Third: be willing to throw out the official tool. A three-month vendor migrator became a nine-day in-house one because someone refused to accept the estimate and knew the problem well enough to beat a generic solution. That's not recklessness — it works precisely because they'd already lived with ScyllaDB for two years and had a Rust data-service library to build on. The "meme-driven engineering" joke about rewriting things in Rust has a serious core: a team that genuinely loves its tools moves faster, because the rewrite that reads as insane on a slide is a fun afternoon for the people who actually do it.

Sources, by title: Discord Engineering, "How Discord Stores Trillions of Messages"; Bo Ingram's ScyllaDB Summit 2023 talk, as covered by The New Stack in "How Discord Migrated Trillions of Messages to ScyllaDB"; and InfoQ, "Discord Migrates Trillions of Message Cassandra to ScyllaDB." The node counts, disk sizes, and per-operation p99 latencies are from Discord's blog; the 3.2 million records/second throughput, the nine-day timeline, and the roughly one-day rewrite are from the talk coverage.