Storing trillions of your Discord messages is waaaaay harder than you thought

On big public Discord servers, most of the the displayed messages are recent and cached which is good for performance. But smaller servers used by just a few friends can often involve old messages that aren't cached being displayed or read from the database every time a user logs in.

The specifics of the company's struggles have been outlined in forensic detail in a blog by Discord's Bo Ingram, a senior software engineer for the platform. 

On big public Discord servers, most of the the displayed messages are recent and cached, which is good for performance. But smaller servers used by just a few friends can often involve old messages that aren't cached being displayed or read from the database every time a user logs in.

According to numbers from 2019, Discord handled something like 25 billion messages per month, 850 million every day, and 600,000 per minute. You can expect those numbers to be even bigger now.

Discord started off using the MongoDB database to support this activity, but then moved to Cassandra. Its appeal was as a NoSQL database that had support for clustering, which means you can have multiple instances of a given database working as one to improve performance.

Both Apple and Netflix, among other big tech companies, both reportedly use Cassandra. So, it's got proper blue chip cred. Surely it's good enough for Discord? Initially, yes. Once Discord had ironed out a few issues, the Cassandra databases were writing new messages within one millisecond and read back old messages to users in five milliseconds. Quick enough for your typical text chat, that's for sure.

But the good times didn't last. Discord now has something in the region of 150 monthly active users. By early 2022, Discord was running 177 Cassandra nodes storing trillions of messages.

The result was hotspots within the database when users interacted with Discord servers in certain patterns. The upshot is latency as the server falls ever further behind with user queries—in other words, users sending and reading text messages.

The way that Cassandra marks data for deletion before eventually removing it via garbage collection routines also added dramatically to latency when reading messages. In really rough layman's terms, the tombstones marking the location of data marked for eventual deletion slowed down the process of getting at actual live data for reading.

The solution for Discord is another extremely difficult migration to a new database, known as ScyllaDB. It has several advangtages, including being written in C++, which is much faster than the Java coding language used by Cassandra. It's also Cassandra-compatible and also deletes data directly rather than using a garbage collector.

Anyway, the switch over to ScyllaDB happened in May last year, and no doubt you didn't even notice when it happened. Which is a good thing. According to Bo, latencies are much improved compared to the tail end of the Cassandra implementation.

"It's been a quiet, well-behaved database (it's okay to say this because I'm not on-call this week). We're not having weekend-long firefights, nor are we juggling nodes in the cluster to attempt to preserve uptime. It's a much more efficient database—we're going from running 177 Cassandra nodes to just 72 ScyllaDB nodes," he says.

"Our tail latencies have also improved drastically. For example, fetching historical messages had a p99 of between 40-125ms on Cassandra, with ScyllaDB having a nice and chill 15ms p99 latency, and message insert performance going from 5-70ms p99 on Cassandra, to a steady 5ms p99 on ScyllaDB."

So, there you have it, the slightly-more-complicated-than-you-might-have-thought story of how to keep a message service from collapsing under the weight of millions of users. You can read about it all in much more detail over on the official Discord blog.