Discord’s messages store is usually told as a database-swap story: Cassandra was slow, ScyllaDB was fast, 177 nodes became 72. Read all the primary sources together, from the CTO’s 2017 post to the 2026 automation post, and something else stands out. The table never changed. The primary key chosen in 2016, ((channel_id, bucket), message_id), carried the product from billions to trillions of messages across two storage engines, and Discord’s engineers say plainly that the 2022 migration did not touch it. What changed, three times, was everything around the table: the process that deletes data, the layer that decides how many identical reads reach a partition at once, and the block device under the database. The thesis of this case: Discord’s hard problem was never storing trillions of rows. It was bounding three kinds of unbounded work (concurrent reads on one hot partition, tombstones a read must step over, and maintenance that grows with data written), and the ScyllaDB swap only paid off because it arrived last: after a coalescing layer had already capped the first, and together with a storage substrate that took disk queueing out of the read path.
177 → 72
messages nodes, Cassandra → ScyllaDB [S2]
~9 days
Rust migrator, vs 3-month Spark estimate [S2]
40–125 → 15 ms
p99 historical read [S2]
0
schema or partitioning changes in 2022 [S14]
Source tags like [S2] refer to the list in Section 12. S3, S9 and S10 are conference talks quoted from auto-generated transcripts. S13–S15 are Hacker News comments by Discord engineers (users jhgg and zorkian), treated as insider testimony rather than official statements. Anything labelled Our analysis, Reconstruction or Counterfactual is ours, not Discord’s.
The original Discord was built “in just under two months in early 2015”, with everything in a single MongoDB replica set[S1]. Bo Ingram’s 2023 talk corrects the detail: it was actually TokuMX, “an open source sister of MongoDB”[S3]. The choice was deliberately temporary. Stanislav Vishnevskiy, Discord’s CTO, describes the culture as “build quickly to prove out a product feature, but always with a path to a more robust solution”, and the team refused MongoDB sharding outright because it was “complicated to use and not known for stability”[S1]. Messages lived in one collection with one compound index on (channel_id, created_at). Around November 2015, at 100 million stored messages, data and index stopped fitting in RAM and latency became unpredictable[S1].
Two product decisions set every constraint that followed. First, Discord “decided early on to store all chat history forever”, available on any device[S1]. There is no retention horizon to hide behind: every channel is an append-only log that grows for the life of the company. Second, reads are random in a specific way. The 2017 post’s workload analysis is the most useful paragraph in the whole corpus, and it is rarely quoted. It splits servers into three populations[S1]:
The read/write ratio was “about 50/50”, and the team expected more random reads from features on the roadmap: 30-day mentions, jump-to-pinned, full-text search[S1]. By 2024 an engineer says reads are “much higher than write traffic”, driven by short-lived mobile sessions re-syncing history, search executing “1 query per result”, and GDPR data exports[S14]. The 2022 super-disk post also calls the workloads “read-heavy”[S5]. The system was chosen for a balanced workload and ended up read-dominated. That drift matters for the rest of this study because it is reads, not writes, that an LSM engine finds expensive.
Early 2015
Discord built in under two months on one MongoDB replica set (TokuMX, per the talk); index on (channel_id, created_at)
S1 S3
~Nov 2015
100M stored messages; data + index no longer fit in RAM, latency unpredictable
S1
~early 2016
Dual read/write to MongoDB + Cassandra; Cassandra made primary, MongoDB gone “within a week”
S1
~mid 2016
Puzzles & Dragons: millions of tombstones, 10 s stop-the-world GC, 20 s channel load
S1
Jan 2017
“Billions” post: 12 nodes, RF 3, ~1 TB compressed/node; Scylla named as a long-term option
S1
Mar 2019
ScyllaDB already serving some production data at Discord
S8
By 2020
Every Cassandra cluster except cassandra-messages moved to ScyllaDB
S2 S7
Early 2022
cassandra-messages: 177 nodes, trillions of messages, “a high-toil system”
S2
2022
Rust data services in front of Cassandra; super-disk Scylla cluster; dual writes; Rust migrator runs ~9 days
S2 S3
May 2022
ScyllaDB becomes the primary messages store, flipped at a team onsite
S2 S3
Dec 2022
World Cup Final: nine spikes on the send graph; Discord and the database “aren’t breaking a sweat”
S2
Nov 2023
Auth cluster (not messages) drops to ~25% availability: degraded disk + 45-min RAID rebuild
S10
May 2026
7-person team, dozens of Scylla clusters, hundreds of nodes; 36 h ops now under 2 h
S16
The requirements for MongoDB’s replacement were written down in 2016 and are worth reading as a specification that the next six years tested[S1]: linear scalability with no manual resharding; automatic failover; low maintenance (“It should just work once we set it up”); proven technology; predictable performance, with alerts on API p95 above 80 ms and an explicit refusal to “cache messages in Redis or Memcached”; not a blob store, because appending to serialized blobs at thousands of writes a second would not work; and open source. “Cassandra was the only database that fulfilled all of our requirements.”[S1] No alternatives are named. Ingram’s talk adds one motive the post doesn’t, framed as fault tolerance: “We didn’t want to lose the Postgres database and then be completely down”[S3].
Our analysis — the 2016 spec predicted the 2022 pain
The data model is where the Cassandra choice became concrete. Vishnevskiy describes Cassandra to newcomers as “a KKV store”[S1]: partition key, clustering key, value. channel_id became the partition key because every query is scoped to a channel. created_at could not be the clustering key because two messages can share a timestamp, but every Discord ID is a Snowflake, chronologically sortable, so message_id took that role and a channel load became a single range scan[S1].
The import immediately produced warnings for partitions over 100 MB, though Cassandra “advertises that it can support 2GB partitions”. The CTO’s verdict: “Apparently, just because it can be done, it doesn’t mean it should.”[S1] Large partitions put GC pressure on compaction and cluster expansion, and a partition cannot be spread across nodes. Because a channel “can exist for years and perpetually grow”, the partition had to be bounded by something other than the channel. Discord looked at its largest channels and chose a static bucket of about 10 days, derivable from the message ID or a timestamp, so those channels would stay comfortably under 100 MB[S1]. The key became ((channel_id, bucket), message_id). To read recent messages, the code generates a bucket range from now back to the channel’s own ID (also a Snowflake, so older than any message in it) and queries partitions one at a time until it has enough[S1].
Our analysis — one constant, three populations
Model: messages per bucket k = R × 10/365, Poisson-spread; buckets walked ≈ 50/k, non-empty among them ≈ (50/k)(1 − e−k). The walk also stops at the channel’s creation (its Snowflake ID), so a young quiet channel reads fewer. At an illustrative 2 ms per serial partition read, 17 reads cost ~34 ms before queueing: the long tail pays in latency, the head pays in concentration.
Our analysis — the decision rule a bucket width encodes
The dark launch exposed the other defining property of the engine. Discord double-read and double-wrote MongoDB and Cassandra, and immediately saw errors that author_id, a required field, was null[S1]. Cassandra writes are upserts, conflicts resolve with “last write wins” per column, and read-before-write is an anti-pattern. So a user editing a message while another user deletes it can leave a row holding only the primary key and the new text[S1].
channel_id
bucket
message_id
author_id
content
…12 more
stored row
C
B
M
u42
hi
…
t1 · user A deletes
C
B
M
∅
∅
∅
t2 · user B edits text
C
B
M
·
hi!
·
merged result
C
B
M
∅
hi!
∅
Option rejected: write the whole message on edit
re-writes every column with a newer timestamp → can resurrect a deleted message, widens conflict surface
Option chosen: treat null author_id as corruption
a required column is null → the read path deletes the row. Detection, not prevention
Discord considered writing the whole message back on every edit and rejected it, because a full rewrite with a newer timestamp can resurrect a deleted message and widens the conflict surface for concurrent writes to other columns. They chose detection: treat a null in a required column (author_id) as corruption and delete the row[S1]. The same investigation found the team “writing 12 tombstones into Cassandra most of the time for no reason”. The schema had 16 columns, the average message set 4, and writing a null is a delete. The fix was to write only non-null values[S1].
Our analysis — repair on read is a design pattern, not a hack
author_id rule is worth generalising because it is cheap and almost never written down. In any store with per-column last-write-wins and no read-before-write, pick one column that every valid write path sets and no update path touches. Its absence becomes a tombstone detector you get for free on every read. The rejected alternative, whole-row writes, converts a column-level race into a row-level one and makes resurrection possible, which is worse than a transient zombie row because it is silent. At 2017’s rate of more than 120 million messages a day, the null-write fix alone avoided on the order of 12 × 120M ≈ 1.4 billion tombstone cells a day (our arithmetic, Section 8). That is the kind of number that disappears into “we tuned the schema” in most write-ups.Cassandra passed a week of testing with sub-millisecond writes and reads under 5 ms, went primary, and MongoDB was “phased out” within a week. “It continued to work flawlessly…for about 6 months until that one day where Cassandra became unresponsive.”[S1] Nodes were running 10-second stop-the-world GCs constantly. The trail led to a channel that took 20 seconds to load: the Puzzles & Dragons Subreddit server, whose channel contained exactly one message, because its owners had deleted millions through the API[S1]. Loading that channel made Cassandra “effectively scan millions of message tombstones (generating garbage faster than the JVM could collect it)”[S1].
Two fixes shipped[S1]. Tombstone lifetime (gc_grace_seconds) went from the default 10 days to 2, justified by the fact that Discord ran repairs “every night on our message cluster”. And the read path learned to track empty buckets for a channel and skip them, so a repeat of the query would at worst scan the newest bucket. It is easy to read this as a tuning footnote. It is actually the first appearance of an invariant that shapes the next six years.
Our analysis — why gc_grace = 2 days bet the cluster on repair
Tombstones came back three more times. The first was in 2022, when the Rust migrator reached “99.9999% complete (no, really)” and stalled. It was “timing out reading the last few token ranges of data because they contain gigantic ranges of tombstones that were never compacted away in Cassandra”. The team compacted that token range, and “seconds later, the migration is complete”[S2]. In other words, after six years the messages cluster still held regions where deletes had never been cleaned up, and nobody knew until a full scan hit them.
The second return is architectural. Stephen Ma’s 2023 talk on KKV, Discord’s generic key-key-value store on ScyllaDB (in production in 2022), attacks the invariant from the other side[S9]. Deletes are issued at consistency level ALL. If that succeeds, the tombstone is on all three replicas and there is nothing for repair to fix. If it fails, KKV writes an application tombstone into the entity blob at QUORUM and publishes a retry message to Google Pub/Sub. The subscriber retries at ALL and nacks until it succeeds; reads that see an application tombstone re-publish the retry. This “allows us to set GC grace seconds to zero”[S9]. Instead of relying on a periodic repair to deliver the delete to every replica, KKV guarantees delivery at write time with a durable retry queue. That removes the dependency on repair cadence that the 2016 fix created.
The third is the engine. Asked in 2024 what made ScyllaDB behave better on the same schema, jhgg answered that “when Scylla is processing a bunch of tombstones it’s able to do so in a way that doesn’t choke the whole server”, and that recent versions can return “partial/empty pages to the client to limit the amount of work per query”[S14]. ScyllaDB’s co-founder added, as a vendor claim, a tombstone_gc=repair mode and CPU/IO schedulers that put queries ahead of compaction[S14]. Tombstones did not go away on ScyllaDB. What changed is that a tombstone-heavy read became a slow query rather than a server-wide GC event.
Reconstruction (inferred) — why tombstones hurt the JVM more than the data does
In January 2017 the messages cluster was 12 nodes, RF 3, with “nearly 1TB of compressed data on each node”, run by a company with “only 4 backend engineers” and no dedicated DevOps[S1]. The post’s near-term plans were a Cassandra 3 upgrade (a storage format that “can reduce storage size by more than 50%”) and denser nodes at 2 TB. Its long-term options were “Explore using Scylla”, for its lower repair times, and archiving unused channels to flat files on Google Cloud Storage, which the team wanted “to avoid”[S1]. At the beginning of 2022, cassandra-messages had 177 nodes and trillions of messages, and “it was a high-toil system — our on-call team was frequently paged for issues with the database”[S2].
The 2023 post and talk describe four interacting failure modes[S2][S3]:
(channel, bucket) made “latency in the node … increase as the node tried harder and harder to serve traffic and fell further and further behind”. With QUORUM reads and writes, “all queries to the nodes that serve the hot partition suffer latency increases”[S2].The operators’ workaround was a script called the “gossip dance”: take a node out of rotation so it can compact without serving traffic, bring it back to pick up hints through hinted handoff, repeat until the backlog is gone[S2]. The talk adds that it could only run on one node at a time, while “the other nodes would have to struggle”[S3].
Our analysis — what the gossip dance tells you about capacity
Two details are easy to miss. First, jhgg, writing in October 2022, put the root cause in terms of tenants: “pathological read / write patterns by some tenants on our system causing outsized availability impact due to triggering garbage collection pressure”[S15]. His list of fixes begins with “working with the relevant product teams to implement appropriate rate limiting or improving data modeling”, before the Rust query layer and ScyllaDB[S15]. Second, the messages cluster was also the oldest. jhgg says “we were on old version of Cassandra for our most problematic cluster”, and because it was the last cluster to migrate, the team “didn’t really do much to explore the problem space of JVM tuning” beyond improvements like switching some clusters from CMS to G1[S15]. The version is not published.
Our analysis — the failure was a positive-feedback loop, so the fix had to cut it somewhere
ScyllaDB was not a 2022 discovery. The 2017 post already listed “Explore using Scylla” as a long-term option, specifically because repairs were CPU-bound and “Scylla advertises significantly lower repair times”[S1]. By March 2019 ScyllaDB was serving production data at Discord; Mark Smith, then Director of Engineering, said it powered the Activity tab, “the homepage of Discord”[S8]. A July 2020 ScyllaDB press release announced Discord moving production data to ScyllaDB Enterprise[S7], and the 2023 post confirms that “by 2020, we had migrated every database but one to ScyllaDB”[S2]. The stated reasons were better performance, faster repairs, “stronger workload isolation via its shard-per-core architecture”, and “a garbage collection-free life”[S2]. jhgg adds the one that probably mattered most economically: moving to something without those issues “while also not having to rewrite a bunch of code was a positive”[S14]. ScyllaDB speaks CQL, so the application, drivers and schema could carry over.
Smith’s 2019 talk gives a less-quoted motive: operational opinionation. Coming from running Kafka at Dropbox, he argued that the hard part of open-source data systems is the year it takes to “productionize” and the “Wild West” of configuration. With ScyllaDB, “you don’t choose how to configure ScyllaDB for the most part. You run some tools. They write out config files”, and monitoring ships as prescribed Prometheus and Grafana images and dashboards[S8]. For a small team, the value is that community help starts from a shared baseline rather than an hour of explaining your config.
Messages went last on purpose. The cluster was big (“trillions of messages and nearly 200 nodes”), the team wanted to tune the new database before moving its most important data, and it wanted “more experience with ScyllaDB in production, using it in anger and learning its pitfalls”[S2]. The talk puts it more bluntly: “We didn’t want to learn all our lessons with our messages database”[S3]. One technical blocker had to be fixed by the vendor first: reverse-query performance. Discord runs a reverse query “when we scan messages in ascending order”, and in testing ScyllaDB’s reverse queries were “insufficient for our needs”; the ScyllaDB team “implemented performant reverse queries, removing the last database blocker”[S2].
Reconstruction (inferred) — what the reverse-query blocker tells you about the table
message_id: newest first, which is what “load the latest 50” wants. Ascending scans come from the less common paths: jump to a message and read forward, or read history after a pinned or mentioned message, the random-read features the 2017 post anticipated[S1]. So the migration was gated not on the hot path but on the tail of the product’s access patterns. That is a generalisable lesson for engine swaps between “compatible” systems: the query language matching is necessary, but the thing to benchmark is the distribution of access orders and shapes you actually issue, including the rare ones that a product feature depends on. Discord found this in testing, before migrating, which is the point of the “messages last” sequencing.Hot-partition concurrency
GC pauses
Compaction / repair cost
Code rewrite
Evidence it was weighed
Tune / upgrade Cassandra (G1, newer JVM)
CMS→G1 on some clusters; old version on messages; little JVM exploration [S15]
ScyllaDB, same schema
chosen; reverse queries fixed first [S2][S14]
Rust data services only
built first; “incidents go down, but not away” [S3]
Archive cold channels to GCS
listed in 2017 as “want to avoid” [S1]
Bigtable
rejected for KKV: single-zone, no failover with RYW [S9]
Per-server Postgres
“few hundred million” servers; DMs don’t fit [S14]
No single row fixes the first three columns. Discord’s answer was two rows at once (data services plus ScyllaDB) on a third change, the storage substrate. That is why no source can isolate how much of the latency gain belongs to each.
Some alternatives appear only in fragments, and are worth collecting. Stay on Cassandra and tune harder: jhgg concedes the most problematic cluster ran an old version and the team did not deeply explore JVM tuning once it had decided to leave[S15]. Archive cold channels to GCS was listed in 2017 and never pursued[S1]. Bigtable was evaluated, for KKV rather than messages: GCP told the team a Bigtable cluster is single-zone, and read-your-writes consistency needs single-cluster routing, which gives up automatic failover[S9]. Per-server relational databases come up in HN debates; jhgg’s answer is that there are “a few hundred million discord servers… do you really want to run that many Postgres instances? And even so what do you do about DM/GDMs?”[S14]. No official Discord source evaluates Postgres for messages.
Counterfactual — a modern JVM on the same cluster
The language choice for the new layer had specific antecedents. Discord had used Rust in Elixir native implemented functions (NIFs) on real-time hot paths and had rewritten its Read States service from Go[S3]. That rewrite is directly relevant: the Go service, which kept an LRU cache in front of a Cassandra cluster handling “tens of thousands of database writes per second”, showed latency spikes “roughly every 2 minutes” because Go forces a GC at least every two minutes and the collector had to scan the whole LRU cache[S12]. The Rust port had “no latency spikes”[S12]. The data services inherited the conclusion: “We really don’t like garbage collection”[S3]. The stated reasons also included Tokio for async IO, native Cassandra and ScyllaDB drivers, and “fearless concurrency”, plus a joke about “meme cred”[S2].
In about 2020, a Python API monolith talked “directly to databases”[S3]. Discord inserted a tier between the two: Rust services on Tokio, speaking gRPC, with “roughly one gRPC endpoint per database query” and “intentionally … no business logic”[S2]. It started as a single “message service”, the prototype that became a shared data service library[S3]. jhgg’s October 2022 description adds that the layer also does “query throttling/load shedding to reduce work the database must do for hot keys”, and that exposing gRPC “lets us centralize control of the client driver and tune it appropriately” while reusing gRPC routing tools such as Envoy[S15]. The talk says the services used the Cassandra C++ driver crate and were moving to the ScyllaDB Rust driver[S3].
API tier
Python API monolith
formerly talked to the DB directly [S2]
Clients
mobile sync, search (1 query per hit), GDPR exports [S14]
Data services · Rust · Tokio [S2][S3]
~1 endpoint per query
no business logic [S2]
Request coalescing
in-flight dedup, not a cache [S13][S14]
Throttling / shedding
for hot keys [S15]
Centralised driver
C++ driver → Scylla Rust driver [S3][S15]
Shared row types
reused by the migrator [S13]
Metrics built in
metrics / telemetry come with the library [S13]
scylla-messages · 72 nodes · 9 TB disk each [S2]
Same schema
((channel_id, bucket), message_id) [S14]
ICS compaction
per jhgg [S14]
Own row cache
direct IO, no page cache [S14]
Super-disk
RAID0 of local NVMe SSDs
375 GB each, lost with the host [S5]
Persistent Disk, write-mostly
durable, snapshots, ~ms latency [S5]
What is not in the path
No message cache
“For messages, we don’t really cache” [S13]
No schema change
“We did not change schema or partitioning strategy” [S14]
No per-tenant DBs
one “mega cluster” for messages [S14]
Instance counts, ring implementation and coalescing hit rates are not published. The zone layout is described only for the auth cluster: RF 3 across three zones [S10].
The feature that matters is request coalescing. “If multiple users are requesting the same row at the same time, we’ll only query the database once.” The first request spawns a worker task; later requests check for that task and subscribe; the worker queries the database and returns the row to all subscribers[S2]. The motivating case is an @everyone ping on a large server, when users “pick up their phone, tab to the app” and all read the same message[S3]. The second half is upstream: consistent-hash routing on a routing key, which for messages is the channel ID, so “all requests for the same channel go to the same instance of the service”[S2].
Discord’s engineers insist this is not a cache. “No cache. Just read coalescing. There is a big difference.” If an identical query arrives while one is executing, it waits for that result; if it arrives after, “it will be issued against the database”. The benefit is avoiding “cache invalidation/consistency issues while also being able to handle thundering herds”[S14]. zorkian, who works on Discord infrastructure, gives the reason a cache would not help much for messages: the hot channel buckets “end up in memory on the database, which is NVMe backed for reads anyway”; caching is used in other data services, where one user query fans out into many database queries[S13]. jhgg himself, in 2022, called coalescing “a form of micro-caching”[S15], which is a fair description if the cache lifetime is exactly one query.
Model: an instance receiving λ requests/s for one key, with query time L, issues λ/(1+λL) queries/s. The 8-instance fleet and the 15 ms service time are illustrative; Discord has not published instance counts or coalescing hit rates.
Our analysis — coalescing turns an unbounded rate into a bounded one
Our analysis — the staleness contract is the design decision
Two more properties of the layer shaped what came next. It gave the migration a free, well-tested data-access library (Section 7). And it became the substrate for later tooling: Discord’s 2026 automation post credits the data services with enabling shadow clusters, short-lived full replicas that receive the same reads and writes as production, used before any risky change to OS, hardware or ScyllaDB version[S16]. A proxy that owns every query is also the natural place to mirror them.
The third change was below the database. Oakley’s 2022 post names the mechanism: with limited database parallelism and “disks that take a millisecond or two”, queries back up into “an ever-expanding queue of disk operations whose queries time out by the time the disk is available”, which was “exactly what we were seeing”[S5]. This is the loop of Section 3 entered from the bottom: IO latency raises in-flight work exactly as a hot partition does. GCP offered local NVMe (about 0.5 ms, no snapshots, data “gone forever” on host problems) or Persistent Disk (replicated, snapshot-able, “a couple milliseconds”)[S5]. The requirements ranked “low-latency disk reads over all other disk metrics”, with write latency explicitly secondary for a read-heavy workload[S5]. A kernel block cache (dm-cache, lvm-cache, bcache) failed one test: a bad sector on the cache device failed the whole read, and ScyllaDB halts on I/O errors “until operator intervention”[S5]. The adopted design is Linux md: RAID0 across 375 GB local SSDs, mirrored in RAID1 with a Persistent Disk flagged write-mostly; the talk’s example is 1.5 TB of each[S5][S3].
It did not work out of the box. Reads “are actually still occurring at the speed of persistent disk writes”, because ScyllaDB by default used one IO bandwidth channel shared by reads and writes, so the slowest operation (PD writes) set the pace. ScyllaDB implemented duplex IO, separate read and write budgets[S3]. The md write-intent bitmap, which speeds up resyncs, was “too expensive to maintain” (“IO would back up”) and was turned off[S3]. RAID5/6 was rejected because a single local SSD cannot be replaced in GCP; if one fails, the whole server migrates and all local data goes[S5].
Our analysis — why RAID1 with write-mostly beats a cache here
Complexity was contained by reducing the number of states the system can be in. A preflight script runs before ScyllaDB starts and recognises three states: a brand-new node (build the array, then green-light join or replace); data on PD but not on the local SSDs, meaning an unclean stop or host error (wait for the resync to finish); and data on both (health-check and start)[S3]. For anything unfamiliar, the team biases “towards removing and replacing nodes”, using ScyllaDB’s replace primitive so peers stream the data, rather than performing “gnarly RAID operations”[S3]. The argument for dropping the write-intent bitmap follows the same logic: in practice a resync is either tiny and instant, or a full rebuild that must stream everything anyway, so the bitmap accelerates nothing that matters[S3].
A · new node
means
no RAID array yet
action
build RAID0 + RAID1, then green-light join or replace
B · PD has data, NVMe empty
means
host error or unclean stop wiped local SSDs
action
wait for md resync from PD; do not start ScyllaDB until done
C · both have data
means
ordinary restart
action
health-check the array, then start
Anything else → remove and replace the node
ScyllaDB streams replicas to the new node; no bespoke RAID surgery [S3]
Cost of state B, seen in 2023
auth cluster: ~45 min resync before a node can serve [S10]
Our analysis — the rebuild time is the new MTTR, and it is not small
The first plan was the textbook one. Dual-write new messages to both clusters, serve recent data from ScyllaDB after a time cutover, and backfill history with ScyllaDB’s Spark migrator. The backfill estimate after “a lot of tuning” was three months[S2]. The team then wrote a Rust migrator on top of the data-service library “in an afternoon”, re-estimated at nine days, and replaced the split read path with a single flip of everything in May 2022, at a team onsite[S2][S3]. Validation was a small percentage of reads sent to both databases and compared[S2].
Plan v1 · abandoned
Dual-write new data to both clusters
Serve recent data from ScyllaDB after a time cutover
two read paths, split by timestamp
ScyllaDB Spark migrator backfills history
“a lot of tuning” → ETA three months [S2]
Plan v2 · executed
Keep dual-writes
Rust migrator copies everything
written “in an afternoon”, ETA nine days [S2]
Shadow-compare a small % of reads, then flip all at once
May 2022, at a team onsite [S2][S3]
Migrator internals (jhgg, S13)
int64 token ring
split into N equal ranges
SQLite work DB
range, stats, status, last page token
“sunder”
split the SQLite DB into N shards
5 large GCP VMs
binary run in screen
N Tokio tasks / VM
own connection, one range each
Throughput
bursts to 3.2M msgs/s [S2]; ~3M inserts/s saturating ScyllaDB with write-tuned compaction and no reads [S14]
The last 0.0001%
final token ranges time out on never-compacted tombstone runs; compact that range in Cassandra, done “seconds later” [S2]
The migrator itself is the most transferable artefact in the case, and only the HN thread describes it. The int64 token space is split into N equal ranges; a SQLite file holds each work item with its stats, status and last pagination token, so any range can pause and resume; N Tokio tasks each hold their own connection; a sunder command splits the SQLite file into shards so the work can be spread across machines; it ran on “5 chonky VMs” in screen[S13]. It reused the same Rust row types as the live service[S13]. jhgg’s explanation of why it beat Spark is blunt: the Spark migrator was “needlessly complicated in pursuit of being general” and slower[S13].
Our analysis — the 10× came from specificity, not from Rust
The run did not finish cleanly. It stalled at “99.9999% complete (no, really)” because the last token ranges contained huge runs of tombstones that had never been compacted in Cassandra, and reads over them timed out. The fix was to compact that range in Cassandra; the migration finished “seconds later”[S2]. Section 2’s antagonist made one final appearance on the way out.
Reconstruction (inferred) — why a flip-at-once cutover was defensible
1 · Dual-write on
2 · Backfill (~9 d)
3 · Last 0.0001%
4 · Shadow reads
5 · Flip · May 2022
6 · Dual-write tail
Reads served by
Cassandra
Cassandra
Cassandra
Cassandra (+ sample to both)
ScyllaDB
ScyllaDB
Writes go to
both
both + migrator firehose
both
both
both (inferred)
both, until retired
Failure mode (our analysis)
one leg fails → silent divergence
migrator writes a row read before a live edit/delete → stale overwrite or resurrection
range reads time out on tombstone runs [S2]
sample too small to see rare divergence
cold row cache for history reads
no disclosed rollback [S2][S3]
What catches it
later backfill re-copies; shadow compare
preserving source write timestamps (not disclosed)
per-range status in SQLite makes the stuck range visible [S13]
size the sample: ~3/p comparisons for rate p
hot buckets already warm from dual writes (inferred)
reads can flip back only while writes still land in both
Reported: dual writes, the Rust migrator, the tombstone stall, a small percentage of shadow-compared reads and the May 2022 flip [S2][S3][S13]. Not reported: shadow-sample size and mismatch rate, whether the migrator preserved write timestamps, how long dual writes outlived the flip, and the rollback plan.
Our analysis — the two races a copy-and-flip must close
The headline results come from the 2023 post: 177 Cassandra nodes became 72 ScyllaDB nodes, with average disk per node rising from 4 TB to 9 TB; p99 historical-message reads went from 40–125 ms to 15 ms, and p99 inserts from 5–70 ms to a steady 5 ms[S2]. The talk adds a 53% reduction in disk usage across the cluster (“no, we didn’t delete half the messages”) and quotes reads as 5–200 ms falling to “5 milliseconds almost every single time”, without a percentile[S3]. We treat the blog’s p99 figures as authoritative and the talk’s as informal.
Provisioned disk is nodes × average disk per node (177 × 4 TB, 72 × 9 TB): capacity, not data. The latency rows bundle data services, the ScyllaDB engine, super-disk storage and ICS; none of the sources isolates one of them. The insert row’s gain is mostly the removal of the 70 ms tail, not a faster typical write.
Our analysis — the node count is mostly a density story
The rates reported across seven years can be checked against each other. They are consistent, which is itself useful: it means the 2017, 2022 and 2024 sources describe the same system rather than different ones.
Quantity
Arithmetic
Result
Src
Live write rate, Jan 2017
120M/day ÷ 86,400
≈ 1.4k msgs/s
S1
Live write rate, Aug 2022
4B/day ÷ 86,400
≈ 46k msgs/s
S5
Growth between them
(4B / 120M) over ~5.6 years
≈ 33× · ≈ 1.9× per year
ours
Migrator, sustained
~3M inserts/s
≈ 65× the 2022 live write rate
S14
Rows moved in 9 days
3M/s × 86,400 × 9
≈ 2.3 trillion
ours
Spark vs Rust
3 months ÷ 9 days
≈ 10× slower, whatever the row count
ours
Null-column tombstones avoided
12 per msg × 120M/day
≈ 1.4B tombstone cells/day in 2017
ours
Our analysis — the migration was a bandwidth problem, not a volume problem
Counterfactual — how much was ScyllaDB, and how much was everything else?
This case is cited constantly in database arguments, usually through a headline or a conference summary. Here are the recurring claims, ordered by how badly they would mislead someone planning the same move.
“ScyllaDB cut latency from 200 ms to 5 ms.”
MisattributedA press write-up’s compression of the talk[S6]. The blog’s p99 figures are 40–125 → 15 ms for reads and 5–70 → 5 ms for inserts[S2], and they bundle data services, super-disk storage and ICS with the engine change[S3][S14].
“They redesigned the data model for ScyllaDB.”
False“We did not change schema or partitioning strategy,” says jhgg; the changes were coalescing and ICS[S14].
“The data service is a cache.”
Mostly false“No cache. Just read coalescing. There is a big difference”[S14]; “For messages, we don’t really cache”[S13]. A late identical request still reaches the database. jhgg did once call coalescing “a form of micro-caching”[S15].
“It was just GC; a modern JVM would have fixed it.”
Half trueThe 2017 GC storm was caused by scanning millions of tombstones[S1], and the 2023 post and talk list hot partitions, compaction backlog and un-runnable repairs alongside GC[S2][S3]. jhgg concedes the worst cluster ran an old Cassandra and that JVM tuning was not explored much once they decided to leave[S15]. GC was the amplifier, not the root.
“ScyllaDB is so fast they needed 60% fewer nodes.”
MisleadingPer-node disk more than doubled, 4 → 9 TB[S2], and on-disk size fell 53%[S3]. Density drove the count; CPU, RAM and cost per node are undisclosed.
“Messaging is write-heavy, hence an LSM store.”
DatedThe 2016 analysis found about 50/50[S1]. By 2024 reads are “much higher than write traffic” (mobile sync, search, exports)[S14], and the super-disk post calls the workloads read-heavy[S5].
“ScyllaDB was a bold 2022 bet on the core table.”
FalseNamed as an option in 2017[S1], in production by 2019[S8], every other cluster moved by 2020[S2][S7]. Messages went last, on purpose.
Our analysis — the error that matters most
The first public load test was involuntary. During the December 2022 World Cup Final the message-send graph shows nine spikes, each an event in the match (goals, penalties, half-time), and the post says Discord and the database “aren’t breaking a sweat”, alongside a graph of coalesced messages per second[S2]. That is the workload coalescing was built for: a burst of readers on the same channels at the same instant.
The tombstone lesson was also turned into a platform. The Key-Key-Value store, rolled out on ScyllaDB in 2022 and serving more than 20 use cases by 2023 with about a day to onboard one, deletes at consistency ALL, falls back to an application-level tombstone with a Pub/Sub retry when that fails, and can therefore run with gc_grace_seconds = 0; it also doubles partitions automatically as they grow[S9]. Bigtable was considered and rejected because it is single-zone and its single-cluster routing lacks automatic failover for read-your-writes[S9].
Our analysis — KKV is the 2017 fix, generalised
The super-disk’s cost arrived on 6 November 2023, on the authentication cluster rather than messages. Discord had been running an OS upgrade since 3 November, one zone at a time: take a zone offline, keep the persistent disks, and let each node rebuild its RAID onto fresh local NVMe, about 45 minutes per node[S10]. The cluster had six nodes, two per zone. A practice run looked clean and the automated health check passed, but node B2 had a degraded disk. When zone C was drained at 10:52, quorum was lost for much of the key space and API availability fell to about 25%. Restarting B2 was ruled out while C was still rebuilding, because restarting the wrong node would have taken the cluster to 100% down; the team waited for C to finish, then restarted B, and recovered at 11:41[S10].
Our analysis — the incident is Section 6’s arithmetic, realised
Our analysis — resolving the density tension with a number
By 2026 the Persistence Infrastructure team was seven people running “dozens of clusters, with hundreds of database nodes” on ScyllaDB, holding messages, channels, servers and most user data[S16]. Their control plane turned a roughly 36-hour cluster operation into under two hours, most of it waiting for nodes to bootstrap[S16]. The post also notes they want to expand clusters in smaller batches with repairs in between, “so we’re never far behind on repair coverage”[S16]: repair remains the constraint it was in 2017.
The Discord record is unusually complete, which makes it tempting to copy whole. The parts that transfer are narrower than the headline, and each comes with a condition.
Bound concurrency per key before you change engines. Coalescing plus consistent-hash routing by channel_id works because Discord’s hot keys are hot for reads of identical rows at the same moment[S2]. If your hot keys are hot for writes, or for many distinct rows in one partition, coalescing absorbs little and you need write batching or partition splitting (which KKV later automated[S9]). Measure the duplicate in-flight read ratio on your hottest keys before building a data-service tier.
Keep the schema fixed if you want a fast migration. The nine-day copy, the single flip and the reuse of production row types all depend on the table being identical on both sides[S13][S14]. If you also need a new partition key, run that as a separate migration, inside one engine, with its own validation.
The super-disk transfers only with spare replicas. Mirroring local NVMe with a write-mostly network disk trades small, frequent failures for large, rare ones: a full rebuild measured in tens of minutes to hours[S5][S10]. That is sound at RF 3 with enough nodes per zone to run a zone down under peak load; it is dangerous on small clusters, which is exactly where the 2023 incident happened[S10]. On a cloud with durable local NVMe, or a managed service, the whole mechanism is unnecessary.
Tombstones are a data-model property, not an engine property. A table with “keep forever” retention, user deletes and time-bucketed partitions will generate tombstone runs on any LSM store[S1][S2]. ScyllaDB copes better with them[S14], but the migration still stalled on them[S2]. If your deletes are rare, bucket-skipping and short gc_grace suffice; if they are common, design for delete-at-ALL or range deletes from the start.
Our analysis — questions to ask in the design review
Counterfactual — a 10-node Cassandra team reads this post
Our analysis — the one-paragraph version
Primary sources are Discord’s engineering blog (S1, S2, S5, S12, S16–S18) and talks by Discord engineers at ScyllaDB Summit (S3, S9, S10), quoted from auto-generated transcripts. S13–S15 are Hacker News comments by Discord engineers, treated as informal insider testimony. S4, S6–S8, S11 and S19 are vendor or press material, cited only for what they report; S7’s TCO claim carries no figures. Instance types, costs, shadow-read mismatch rates and the cutover rollback plan are not disclosed in any source.
How Discord Stores Billions of Messages — Stanislav Vishnevskiy, Discord Blog, 13 Jan 2017
How Discord Stores Trillions of Messages — Bo Ingram, Discord Blog, 6 Mar 2023
How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB (talk) — Bo Ingram, ScyllaDB Summit 2023, Feb 2023
How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB (session page and abstract) — ScyllaDB, 2023
How Discord Supercharges Network Disks for Extreme Low Latency — Glen Oakley, Discord Blog, 15 Aug 2022
How Discord Migrated Trillions of Messages to ScyllaDB — Loraine Lawson, The New Stack, 21 Feb 2023
Discord Chooses ScyllaDB as Its Core Storage Layer (press release) — ScyllaDB, 22 Jul 2020
Discord, on the Joy of Opinionated Systems — ScyllaDB Blog, summarising a talk by Mark Smith (Discord), 20 Mar 2019
Key-Key-Value Store: Generic NoSQL Datastore with Tombstone Reduction and Automatic Partition Splitting (talk) — Stephen (Siling) Ma, ScyllaDB Summit 2023, Feb 2023
So You’ve Lost Quorum: Lessons From Accidental Downtime (keynote) — Bo Ingram, ScyllaDB Summit 2024, Feb 2024
ScyllaDB User Talk Takeaways: CEO Dor Laor’s Perspective — Dor Laor, ScyllaDB Blog, 12 Apr 2023
Why Discord is switching from Go to Rust — Jesse Howarth, Discord Blog, 4 Feb 2020
HN: How Discord Stores Trillions of Messages (comments by Discord engineers jhgg, zorkian) — Hacker News, 6 Mar 2023
HN: How Discord stores trillions of messages (2023) (jhgg replies; ScyllaDB co-founder dorlaor) — Hacker News, 28 Sep 2024
HN: comments by jhgg on running Cassandra, in “Cassandra at Apple” thread — Hacker News, 8–9 Oct 2022
How Discord Automates ScyllaDB Clusters at Scale — Peter French, Discord Blog, 8 May 2026
How Discord Indexes Billions of Messages — Jake Heinz, Discord Blog, 16 Mar 2017
How Discord Indexes Trillions of Messages — Vicki Niu, Discord Blog, 24 Apr 2025
ScyllaDB Summit 2024 Recap: An Inside Look — ScyllaDB Blog, 16 Feb 2024