Most resharding stories are about a one-off migration: a team spends a year getting off a single database and writes it up. Stripe’s is about turning data movement into a routine operation of the database platform. Moving a chunk between replica sets happens with no change to the application, with cutovers shorter than a primary failover. Stripe used it to split shards, to bin-pack thousands of underused databases (1.5 PB moved, shard count cut by about three quarters), and to upgrade the database engine by copying data rather than upgrading nodes in place. The thesis of this case: the design works because Stripe did not create a new availability contract. It reused one its applications already had to honour. That contract is the retry budget that lets them survive a replica-set failover, underwritten by a decade of making writes idempotent. The mechanisms are good engineering. The contract is what makes them safe.
>5M
queries per second
2,000+
shards (2024)
1.5 PB
moved in 2023 bin-packing
< 2 s
traffic switch
The primary source is a single 2024 engineering post by two members of Stripe’s Database Infrastructure team[1]. On its own it is a tidy summary that many newsletters have re-summarised. What follows puts it back in context. That means the 2012 MongoSF talk by Stripe’s then-CTO[8], the 2013 MoSQL release[9], the 2017 posts on online migrations and idempotency[2][3], the July 2019 outage RCA and the staff comments on it[5][6], the Ledger post[11], and two 2026 posts about the fleet that the platform now manages[16][17]. Where we reconstruct or infer, the text says so. Stripe publishes very little about its internals, and several things an architect would most want to know are simply not public. Section 10 lists them.
Stripe launched in 2011 on self-managed MongoDB (“better developer productivity than standard relational databases”; Atlas “didn’t exist in 2011”)[1][7], and was building infrastructure on it from the start: a 2011 talk abstract describes “Monster”, an event-processing framework on a MongoDB event store[10]. Self-hosting is the root of this case: every capability that follows had to be built because nobody was selling it. Read in sequence, the older sources are less a prehistory than a parts list.
2012 — the client contract. Brockman’s MongoSF write-up treats failover as routine: during reconfiguration “the servers close all open connections”, the driver cannot know whether a write landed, so “we make all of our writes idempotent” and every Mongo call is wrapped in retry “with an appropriate backoff and timeout”[8]. His caveat is the one people still get wrong: “make sure a conflicting write can’t sneak in between tries.” Elections under flapping took “a minute or more”; by 2024 a planned failover is “typically lasting a few seconds”[1][8]. That order-of-magnitude shrink in the event clients must survive (our comparison) is what later made “shorter than a failover” a tight target rather than a loose one. The same talk shows bulk operations as the enemy of availability (per-record, throttleable migrations because of the then “global write lock”; replacement nodes synced from a secondary to protect the primary’s cache)[8]. Both instincts reappear in 2024 as “no performance impact to source shard”.
2013 — the data-movement shape. MoSQL did “an initial import” and then applied changes by “tailing the MongoDB oplog”[9]: bulk-copy-then-stream-the-log, the same two phases as steps 2 and 3 of the 2024 platform. No source draws that line and we do not claim ancestry; it shows only that the oplog-as-change-stream idea was native at Stripe a decade before the platform was written up.
2011
Stripe launches on self-managed MongoDB in the cloud; Atlas “didn’t exist in 2011”
[1]
2011
“Monster”: in-house event-processing framework on a MongoDB event store (talk)
[10]
May 2012
CTO at MongoSF: all writes idempotent, every Mongo call wrapped in retry; per-record migrations
[8]
Feb 2013
MoSQL open-sourced: initial import, then tail the oplog to Postgres in near-real-time
[9]
Feb 2017
Online migrations at scale: four-phase dual-write for hundreds of millions of Subscriptions
[2]
Feb 2017
Idempotency-Key header; exponential backoff with jitter recommended for clients
[3]
Jul 2019
Latent election fault + remediation: API severely degraded 27 min, then 93 min
[5]
2023
$1T volume at 99.999%; 1.5 PB bin-packed, shards cut ~¾; fork-lift major upgrade
[1]
Jun 2024
DocDB / Data Movement Platform published: >5M QPS, 5,000+ collections, 2,000+ shards
[1]
2025
“One of 2,000” shards that together processed $1.9T
[16]
Jul 2026
Graph-search auto-remediation on Temporal across 40+ shard layouts
[16]
Aug 2026
Stated reliability target: 99.9995%
[17]
Not in any source: when the Data Movement Platform was started or first used. It sits somewhere between the 2019 layout and the 2023 consolidation.
2017 — verification, but in application code. “Online migrations at scale” moved hundreds of millions of Subscriptions out of Customer records with a four-phase dual write: Scalding backfill over snapshots, reads moved under GitHub’s Scientist, writes moved, old data removed[2]. The operating rule (“never … more than a few hundred lines of code at one time”) tells you the cost unit: every migration was a product team’s engineering project.
One detail from the 2017 HN thread is worth keeping. A reader asked about a race: a subscription deleted after the snapshot could be resurrected by the snapshot-driven backfill. The Stripe reply said only that the final verification job “is a safety check”[15]. That is the snapshot/stream seam every such pipeline has. In 2017 it was closed after the fact by re-verification; in 2024 it is designed out by starting log replay at the snapshot’s own time T, with a snapshot comparison as backstop[1]. Same seam, moved from “detect” to “prevent”.
The last pre-platform snapshot is the 2019 RCA’s single sentence on layout: “Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes”[5].
Our analysis
The platform was “built originally as a horizontal scaling solution to overcome vertical scaling limits of MongoDB compute and storage”[1]. The before-and-after is stated plainly. Stripe went from “a small number of database shards (each with tens of terabytes of data)” to “thousands of database shards (each with a fraction of the original data)”[1]. Stripe does not publish a date, a growth curve or the trigger. What it does publish lets us work out why tens of terabytes per replica set is where the pain starts.
Every node-level operation scales with data per node. Replacing a failed secondary means copying the whole data set to it. So does adding a replica, and so does restoring a backup. Stripe’s 2026 fleet post gives one ratio: “a node rebuild takes roughly 33 times longer than a config change”[16]. That is on today’s smaller shards. On a shard holding tens of terabytes, the time a replica set runs with reduced redundancy after a node loss is set by how fast you can copy that data. The shard gets more fragile exactly as it gets bigger.
Our analysis — what splitting does, and does not, buy for redundancy
Illustrative numbers (Stripe publishes no rebuild rate): at an effective 250 MB/s copy rate, rebuilding a 30 TB node takes about 33 hours; a 1.5 TB node takes under two hours. For those hours the replica set runs one copy short, and a second node failure in the window can cost the shard its majority.
The non-obvious part is what happens fleet-wide. Split the same data D over S times as many shards and you get roughly S times as many node failures a year, each with a window about 1/S as long. The expected number of double-fault events is (failures a year) × (chance of a second failure in the window) ∝ S × D/S, so it stays about the same. What changes is the consequence of each one: 1/S of the data is at risk, for 1/S of the time, behind a much smaller set of dependent applications. Splitting makes double faults smaller and shorter, not rarer. That is the right trade for a payments company, where the cost of an incident grows with how much of the API it takes down. But the argument only holds if the dependency graph splits too, which is the caution in the callout below.
Blast radius scales with what the shard is shared by. The 2019 outage is the clearest public view of Stripe’s pre-platform failure mode. The timeline is from the RCA[5]:
1 · 16:35 Primary node for the database cluster fails; first degradation begins
2 · 16:36 Alert fires; incident response begins (detection “within a minute”)
3 · 16:50 Team determines the cluster cannot elect a primary
4 · 17:00 All nodes in the cluster restarted; election succeeds
5 · 17:02 API fully recovered — 27 minutes
6 · 20:13 Likely faulty code path found in the new minor version’s election protocol
7 · 20:42 Election protocol rolled back to the previous minor version (deployed in ~4 min)
8 · 21:14 High CPU; shards unavailable; second severe degradation begins
9 · 21:26 Rollback identified as interacting with a recent config setting; fix slowed by CPU contention
10 · 22:34 New configuration rolled out, nodes restarted
11 · 22:47 API fully recovered — 93 minutes; traffic ramped back, API-critical services first
Three things in the RCA and its HN thread matter for this case. First, the trigger was a latent fault from an in-place minor-version upgrade. It had been rolled out in phases, from less critical to more critical clusters, and ran for three months “including many successful failovers”. It only showed up “in the presence of multiple stalled nodes”[5]. Second, the stalled nodes stopped reporting replication lag while still passing active health checks. The infra lead’s summary: “The missing bit was verifying the absence of lag updates”[6]. Third, the blast radius: “Because of widespread use of this shard across applications, including the API, the unavailability of this shard starved compute resources for the API and cascaded”[5]. The second outage was self-inflicted. The rollback of the election protocol “interacted poorly with a recently-introduced configuration setting”, which caused CPU starvation. Responders at first assumed a repeat and reran the first playbook[5]. The CTO wrote on HN that the revert “was reviewed by several engineers; however we lacked an efficient way to fully validate this change on the order of minutes”[6].
The RCA’s remediation list is explicit[5]: alert when nodes stop reporting lag; work with “the database maintainers” on the election fault; add circuit breaking on failed operations per cluster; pursue “fault isolation techniques to contain the impact of a single failed shard”; “limit resource consumption by clients attempting repeated retries of failed requests”; and add tooling to make rapid configuration changes safer.
Our analysis — reading 2019 against 2024
No Stripe source connects the 2019 outage to the Data Movement Platform, and we do not know which came first. Read side by side, though, the 2019 remediation list and the 2024 platform line up closely. “Contain the impact of a single failed shard” is an argument for many small shards, and that is only operable if moving data between shards is cheap. The upgrade story is the sharper parallel. In 2019 an in-place minor upgrade across a cluster carried a latent fault for three months, and the rollback caused the second outage. In 2023 Stripe upgraded the fleet’s major version by “fork-lifting data to a later version of MongoDB in one step”[1]: data is copied onto new-version shards, and each chunk keeps a path back to its old-version source until it is deregistered. Whatever the chronology, the 2024 design answers the 2019 failure.
One caution. Smaller shards reduce blast radius in data terms. They reduce it in request terms only if a typical request touches few shards. A shard “used across applications, including the API” is a key-design problem as much as a size problem. Splitting it by key range does not help if every API call reads a chunk from each piece.
The 2024 post also lists what Stripe wanted from a database service that nothing off the shelf offered[1]: the highest availability, durability and performance; “exposing a minimal set of database functions to avert self-inflicted issues due to suboptimal queries from client applications”; horizontal scaling through sharding; “first-class support for multitenancy with enforced quotas”; and security through enforced authorization policies. Three of the five are about protecting the database from its clients, not about storage. That tells you what the proxy tier in the next section is for.
Stripe’s DocDB is “an extension of MongoDB Community” plus “a set of services that we built in-house,” with MongoDB described as the “underlying storage engine”[1]. (Not to be confused with AWS DocumentDB, which people often also call “DocDB”. The two are unrelated.) As of June 2024 it served “over five million queries per second” from product applications, with “10,000+ distinct query shapes” over “petabytes” in “5,000+ collections distributed over 2,000+ database shards”[1]. Two of the post’s four figures (F1 and F3) and its description of the components, combined, give the picture below. The layout and the dashed migration-only paths are our reconstruction.
Solid: steady-state paths. Dashed: paths that exist only while a chunk is moving. Every read the platform makes comes from a snapshot or the CDC archive, never from the source primary, which is how it meets the “no performance impact to source shard” requirement. The Desired state store and the Validation service appear in Stripe’s figure F3 but are never discussed in the text; their roles here are read from the diagram.
The proxy fleet is the load-bearing decision. Applications never talk to a shard. They talk to proxies “developed in-house in Go” to “enforce concerns of reliability, scalability, admission control, and access control”[1]. A proxy parses the query, routes it “to one or more shards”, combines the results and returns them. It asks a chunk metadata service which shard holds which chunk[1]. Every mechanism in this case depends on this layer. Without an interposed router you cannot flip a route without touching the clients, and you cannot stamp every request with a version token. You also cannot enforce the “minimal set of database functions” that keeps 10,000+ query shapes from turning into 10,000+ ways to melt a shard.
Logical versus physical. Product teams provision a logical database through the “document database control plane”. It holds one or more collections, and each collection’s data is spread across physical databases, each holding “a small chunk of the collection”. Physical databases live on shards, which are replica sets with “a primary node and several secondary nodes with replication and automated failover”[1]. Figure F2 of the post makes it concrete with an example. core_payments.payment_intents is split into key ranges 1–10, 11–20 … 91–100, and each range is its own physical database (core_paymentsdb_0000 … _0009) on its own shard (shard_a … shard_j).
logical db: core_payments
provisioned by a product team via the control plane
collection: payment_intents
one of 5,000+ collections
key range
physical database
shard (today)
after a move
1 – 10
core_paymentsdb_0000
shard_a
unchanged
11 – 20
core_paymentsdb_0001
shard_b
→ shard_x
…
…
…
unchanged
91 – 100
core_paymentsdb_0009
shard_j
unchanged
Stripe’s figure uses key ranges and one physical database per chunk. We read it as an example, not a production layout. The real shard-key scheme, chunk size and split policy are not published. One implication the figure does support: a chunk that is its own physical database can be imported, validated and dropped as a unit.
Reconstruction (inferred)
The change log is already a platform. Each shard’s oplog is shipped “to Kafka” and then archived “to a cloud object storage service such as Amazon S3”[1]. The post presents this as standard infrastructure (“in line with typical database infrastructure stacks”). For data movement it is the most important dependency in the design. Section 5 explains why.
The control plane has more boxes than the text explains. Figure F3 shows a DocDB control plane containing the Data Movement Platform. The platform has four services: a Coordinator, a Bulk import service, a Replication service and a Validation service. The figure also shows a Desired state store with a two-way link to the control plane, snapshots from both source and target shards feeding in, and CDC from each shard[1]. The text never mentions the desired state store or names the validation service. A desired-state store alongside a coordinator suggests a reconcile-toward-target-state design, as opposed to a fire-and-forget script. That is our reading of a box label and nothing more.
Our analysis — the fan-out tax
Reconstruction (inferred) — the chunk metadata service is off the hot path until it is not
F4 shows proxies fetching routes from the metadata service after a rejection, and the text says proxies fetch updated routes once the switch completes[1]. Put that together with the shard-side token check and a property falls out that the post never states: the proxies’ route caches can be arbitrarily stale without ever being wrong. Correctness is enforced at the shard. Cache staleness only decides how many requests eat one rejection before the proxy refreshes. So no TTL has to be tuned for safety, and no invalidation broadcast has to reach every proxy.
The same reasoning shows where the metadata service is critical. In steady state, if proxies serve from cached routes (our inference; the caching model is not published), a metadata outage stops migrations but not queries. During a switch it is on the critical path of the moving chunk: the fence is up, and only a metadata write lets traffic resume. And under churn (thousands of databases bin-packed in a year) the refresh load is roughly switches × proxies holding the route. That is small per switch but grows with both fleet sizes at once. A design that tolerates stale caches is what keeps that product cheap.
The post lists four requirements for the platform[1]. Consistency and completeness of data across source and target. Availability. Granularity and adaptability: any number of chunks from any number of sources to any targets, with “no restrictions on the number of in-flight database chunk migrations in the fleet, and no restrictions on the number of migrations any given shard can participate in”, at high throughput for chunks of varying size, since “several of our database shards contain terabytes of data”. And no performance impact on the source shard. The availability requirement is the one that shapes the whole design:
“Our goal is to keep the key phase of the migration process shorter than the duration of a planned database primary failover—typically lasting a few seconds, and in line with the retry budget of our product applications.”
Stripe, 2024 [1]
Read that as an architect. Stripe did not set a target such as “p99 cutover under 500 ms” and then engineer toward it. It set the target relative to an event the application fleet already has to survive. Every client of a replica-set database must tolerate a primary failover: connections drop, in-flight writes have unknown outcomes, and the only correct response is retry-with-backoff on idempotent operations. Stripe has built its clients this way since at least 2012[8]. If a chunk cutover is shorter than a planned failover, a migration adds no new failure class to any client. It only adds more occurrences of one they already handle. Stripe describes the migrations as “client-transparent”[1]. Our reading is that no application team had to review, test or opt into the platform, and that this property is what made “no application changes” achievable rather than aspirational.
Assumptions (ours): backoff base 0.5 s, doubling (Stripe recommends 2ⁿ backoff with jitter and its Ruby library retries with increasing backoff and jitter [3], but the base is not published); the fence lasts 1.9 s. The pink client, which gives up after 1.05 s, fails the request even though nothing is wrong. The design is only “zero downtime” for callers whose retry envelope is at least as long as the fence.
The hidden dependency is idempotency. The post says failed reads and writes “succeed on retries”[1]. It does not say why retrying a write is safe. The rest of Stripe’s record does. “At Stripe, we make all of our writes idempotent” (2012[8]). At the public API, Stripe’s Idempotency-Key header saves “the resulting status code and body of the first request … regardless of whether it succeeds or fails”, and “subsequent requests with the same key return the same result, including 500 errors”[4]. Keys can be up to 255 characters and may be pruned once they are at least 24 hours old. Results are not saved if validation fails or the request “conflicts with another request that’s executing concurrently”[4]. The 2017 idempotency post recommends exponential backoff with jitter, and says Stripe’s Ruby library retries automatically with an idempotency key “using increasing backoff times and jitter”[3]. Linking the migration protocol to this discipline is our inference. The post never makes the link. But without it, “succeed on retries” would be a much weaker statement.
Our analysis — the fence produces the clean kind of failure
Whose retry budget? The requirement says “our product applications”, meaning Stripe’s own services calling DocDB through the proxies. Figure F4 shows the proxy returning a 5xx to the “client app”[1], and that client app is an internal service, not a merchant. Whether a merchant’s API call ever sees a cutover depends on whether the internal service retries within the API request’s own deadline. That is likely by design, but not stated. The distinction matters when you copy this pattern. Your “product applications” may be third parties whose retry behaviour you do not control, and then the whole contract fails.
The Coordinator orchestrates each migration by calling the purpose-built services in order[1]. The table below lays out the steps against two questions the post answers only in passing: how much load each step puts on the source shard, and who serves the chunk while it runs. In our reading, the first four steps are invisible to clients and can be abandoned cheaply, almost all the risk sits in step 5, and step 6 is the one you cannot undo.
#
Step
What happens
Source shard load
Serves chunk
1
Register + build indexes
Intent recorded in chunk metadata; indexes built on target for the moving chunks
none
source
2
Bulk import at T
Snapshot at time T, filtered to the chunks, loaded in index-sorted order (10x)
snapshot read
source
3
Async replication
Replay oplog from T out of Kafka / S3; checkpointed; pausable; lag exposed; bidirectional + tagged
none (reads CDC)
source
4
Correctness check
Compare point-in-time snapshots of source and target once replication is in sync
snapshot read
source
5
Traffic switch
Bump token on source → drain → update route + token in metadata; under 2 s
fenced
none → target
6
Deregister + drop
Mark complete in metadata; drop chunk data from the source
delete
target
The Coordinator first registers the intent “to migrate database chunks from their source shards to arbitrary target shards” in the chunk metadata service, and then builds indexes on the targets for the moving chunks[1]. The post gives no reason for building indexes before loading. The common advice runs the other way: load first, then build indexes in one sorted pass. ByteByteGo gives reasons (query performance, consistency)[13], but those are its own, not Stripe’s.
Our analysis
The import reads “a snapshot of the chunks on the source shards at a specific time, denoted as time T” and loads it “onto one or more database shards”. The import service “accepts various data filters, and only imports the chunks of data that satisfy the filtering criteria”[1]. Filters plus one-or-more targets are what make both directions of elasticity the same operation. A split is one source feeding two targets under two filters. A bin-pack is many sources feeding one target. The post does not say how the snapshot is taken, for example from a secondary or from backups.
This step “appeared simple at first”, but loading a DocDB shard hit throughput limits. Batching writes and tuning “DocDB engine parameters” gave “little success”. The breakthrough was insertion order. DocDB stores data in a B-tree, so sorting the data “based on the most common index attributes in the collections and inserting it in sorted order” improved “the proximity of writes—boosting write throughput by 10x”[1].
Stripe reports only the outcome: batching and engine tuning gave “little success”, and sorting on “the most common index attributes” gave 10x write throughput [1]. The model is ours and deliberately crude (it counts leaf touches, ignores caching of upper levels and the document store itself). Its point: with k indexes of which one matches the sort, pages touched per batch of B fall from kB to about 1 + (k−1)B, so the ceiling on the gain shrinks quickly as k grows, unless the other indexes share the sort key’s prefix or are correlated with it. A reported 10x therefore suggests either collections with one dominant access path, or a sort key chosen as the common prefix of several indexes. The plural “attributes” fits the second reading; we cannot tell which.
The lesson here is broader than MongoDB. When batching and knob-tuning fail, the bottleneck is usually not per-write overhead. It is the working set each write touches. Random-order inserts into a large B-tree touch leaf pages across the whole key space, so the page cache thrashes and pages split everywhere. Sorted inserts touch one leaf at a time and fill it before moving on. The post names neither the storage engine nor which index the sort favoured. As the model above shows, data can be sorted in only one order, so the gain is a property of each collection’s index set, not of the platform. Expect it to vary widely per collection, and do not promise a migration duration from a fleet-average import rate.
The 10x matters operationally because of step 3. Every hour the import runs is another hour of change log the replication step must replay before the target catches up. The chart in the next subsection shows why.
Once the import finishes, the replication service applies every mutation to the moving chunks from time T onward. Stripe describes this service as reading from the CDC systems, not from the source shard: it replicates “using the oplog events in Kafka and Amazon S3”[1]. The post gives two reasons: to avoid slowing user queries “by consuming read throughput that would otherwise be available to user queries on the source shard”, and “to avoid being constrained by the size of the oplog on the source shard”[1].
The second reason is the more important one, and it is easy to skim past. A MongoDB oplog is a bounded collection on the shard. Its window, the time span it covers, is its size divided by the shard’s write rate. A migration that replays from the source oplog must finish import, validation and catch-up before the oplog rolls past time T, or it has to start over. The larger and busier the shard, the shorter the window and the longer the import. The shards that most need splitting are exactly the ones that make source-oplog replay fail.
All rates and the 24 h window are assumptions for illustration; Stripe publishes neither. The shape is the point. Replaying from the source oplog puts a ceiling on how much data one migration can move, and that ceiling falls exactly when the shard is busiest. Replaying from an archived CDC stream removes the ceiling. The 10x import speed-up then shortens how long the migration runs rather than deciding whether it can run at all.
Replaying from Kafka and S3 separates migration duration from anything on the source. Kafka covers recent events and the object store covers everything older, so the replay window is as long as the archive. The post lists the properties that follow[1]: the service is “resilient to target shard unavailability”, supports “starting, pausing, and resuming synchronization from a checkpoint at any point in time”, and exposes its replication lag. The checkpoint property changes the operational character of a migration. A target shard failing over halfway through a 20-hour catch-up becomes a pause, not a restart.
Our analysis — what replicating from CDC costs
Catch-up is a rate inequality. Removing the oplog ceiling does not make convergence automatic. While the import runs, the chunk keeps taking writes, and they pile up as a backlog in the log. Once replay starts, that backlog drains only at the difference between the replicator’s apply rate and the chunk’s live write rate.
Every number is ours; Stripe publishes no rates, lag thresholds or migration durations. The shape is general: catch-up time is backlog ÷ (apply − write), so it is hyperbolic in the margin, not linear. Cutting the margin from 2w to 0.2w multiplies catch-up time by ten. When several sources pour into one bin-pack target, their write rates add up while the target’s apply capacity does not. Past the point where Σw exceeds apply, the lag never closes and the switch can never start.
Our analysis — “no restrictions on in-flight migrations” meets a finite target
Stripe’s requirement is “no restrictions on the number of in-flight database chunk migrations in the fleet, and no restrictions on the number of migrations any given shard can participate in”[1]. Read as a scheduling claim, that holds: nothing in steps 1–4 loads the source, so there is nothing to ration. Read as a convergence claim, it cannot hold unconditionally. A bin-pack target receiving chunks from many underused sources has to apply the sum of their write rates plus its own. “Underutilized” is what keeps that sum small, which is probably why bin-packing at the scale Stripe reports (thousands of databases) worked. The dangerous case is the reverse: splitting a hot shard “during traffic surges”[1]. The chunk’s write rate is at its highest exactly when the move is urgent, so the margin (apply − w) is at its thinnest. From the model above, a margin of 0.2w turns a three-hour catch-up into more than a day.
Two decision rules follow (ours). First, admit a migration onto a target only if the target’s spare apply capacity exceeds the summed write rate of everything already converging onto it, with a margin of at least 2x. Below that, catch-up time is dominated by noise in w. Second, split hot shards before the surge, when the margin is wide, not during it. That second rule is the real argument for the “heat management system” Stripe said it was prototyping[1]: placement decided while there is headroom, not at the peak.
Bidirectional, with tagged writes. Mutations to chunks under migration are replicated “bidirectionally—from the source shards to the target shards and vice versa”, and “the replication service tags the writes it issues to avert cyclical asynchronous replication”. The stated purpose is “the flexibility to revert traffic to the source shards if any issues emerge when directing traffic to the target shards”[1]. How the tag works (a field, operation metadata or something else) is not published.
Our analysis — why bidirectional replication is safe here
When replication is in sync, the platform runs “a comprehensive check for data completeness and correctness by comparing point-in-time snapshots—a deliberate design choice we made in order to avoid impacting shard throughput”[1]. Figure F3 draws snapshots from both shards feeding the control plane, where the Validation service sits.
What is compared (full documents, hashes, counts, samples), how the two snapshots are aligned to the same logical point while replication is live, and what happens on a mismatch are all unpublished. The design principle is still readable. Validation runs on snapshots, never against the live primaries, for the same reason replication reads from CDC: the source shard is still serving, and the platform’s fourth requirement is no impact on it. Every read the platform needs comes from a copy.
Our analysis — correctness at two layers
The migration ends by “marking the migration as complete in the chunk metadata service and subsequently dropping the chunk data from the source shard”[1]. The post gives no delay between switch and drop. It matters. Reverse replication keeps rollback cheap only while the source copy exists. The 2019 fault took three months to surface[5], and no one keeps a reverse replica for three months. So the rollback window is a policy choice that trades storage cost against how late a problem can be found and still be reversed cheaply. For a bin-pack, where the point is to free source shards, that pressure pushes toward dropping early.
To move reads and writes, Stripe has to “stop the traffic on the source shard for a brief period of time, update the routes in the chunk metadata service, and have the proxy servers redirect reads and writes to the target shards”[1]. The protocol is “based on the idea of versioned gating”. In steady state each proxy annotates requests to shards with a version token number. Stripe “added a custom patch to MongoDB” so a shard can enforce that the token on incoming requests is newer than the one it knows of, and serve only requests that pass[1]. The Coordinator then runs three steps[1]:
Proxies then fetch the updated route and latest token and send reads and writes to the target. “The entire traffic switch protocol takes less than two seconds to execute, and all failed reads and writes directed to the source shard succeed on retries”[1]. Figure F4 of the post draws it as a sequence diagram, reproduced and annotated below.
Why this order, and no other. Each step makes the next one safe:
Fencing first makes the set of writes to drain finite. While the source still accepts writes, “wait until replication catches up” is a moving target. Once it is fenced, the lag can only go down, and the drain is guaranteed to finish. Draining before publishing makes sure the target holds every write the source ever acknowledged before any client can read from the target. Reverse the two steps and a client could write to the target and then read back a state that is missing an earlier write still in the replication pipeline. That is a lost update. Publishing last means that, during the gap, proxies with the old route get a clean rejection rather than stale data.
Figure F4 contains a detail the text does not mention. The Coordinator asks the replication service “is replication in sync?” before bumping the token, and again after[1]. The first check limits the drain to what is in flight at that instant. The fence never goes up while the target is behind by minutes, so the time in the red band of the sequence diagram is set by pipeline latency, not by backlog.
Reconstruction (inferred) — what “under two seconds” says about the CDC pipeline
Generalise that first “in sync?” check and you get the part of the protocol that decides whether the sub-two-second number holds: a go/no-go gate. The asymmetry drives the design. Refusing to fence is free, because replication keeps running and the Coordinator can try again. Fencing at the wrong moment costs the chunk’s entire traffic for as long as the fence lasts.
Check
Basis in sources
Go if (illustrative)
What it prevents
Replication lag on the moving chunks
Figure F4’s first “is replication in sync?” [1]
lag × drain rate < fence budget; e.g. lag < 200 ms
Fence length = drain time; drain time = backlog at fence
CDC pipeline latency (source oplog → target)
Implied by “< 2 s” (section 6 callout)
p99 over the last minutes well under the fence budget
A slow Kafka partition becomes a long fence
Validation result
Step 4 snapshot comparison [1]
Clean compare since the last divergence
Switching onto a copy you have not proven
Target shard health
“resilient to target shard unavailability” (replication) [1]
Primary stable; no election in the last N minutes
Switch lands traffic on a shard mid-failover
Other controllers on source or target
Not published; remediator runs on the same fleet [16]
No rebuild, vote or priority change in flight
Election during the fence; cross-controller race
Metadata service write path
Route + token publish is the only way out of the fence [1]
Healthy, and the Coordinator holds a lease
The stuck fence of the crash matrix
Only the first check appears in Stripe’s material. The rest follow from the constraint that the fence must end inside the shortest retry envelope among callers. Every check turns an unbounded fence into a refusal to start, and a refused switch costs nothing because steps 1–4 keep running.
Why fence at the node and not at the proxies. This is the design decision most worth taking away. There is a fleet of proxies, each caching routes. To switch a chunk safely without a node-side fence, the Coordinator would need every proxy to confirm it has stopped sending the chunk to the source before the drain could begin. That is a distributed barrier across the whole proxy fleet. It costs more as the fleet grows, and it is only as fast as the slowest or most partitioned proxy. It also fails unsafe: a proxy that missed the message keeps writing to the old location. Enforcing the token at the shard turns that N-party barrier into a single conditional check at the one place every write must pass. Stale proxies cannot do harm even if they have not been told anything: their requests to the source after the bump are rejected. In F4 the proxy returns a 5xx and then fetches new routes; the text says only that proxies fetch the updated routes “upon completion”[1]. Whether the rejection itself triggers the refresh is not stated, but in our reading it can serve as the cache-invalidation signal.
This is the fencing-token pattern from distributed locking (covered in the Distributed Systems Deep Dive track), applied to routing: the resource checks a monotonically increasing token, so a holder with an old view cannot act on it. Storing the token “in a document in a special collection in DocDB” has a consequence the post does not spell out. We infer that the token is written and replicated through the replica set like ordinary data. A primary failover on the source shard in the middle of a switch would then keep the fence, because the new primary sees the same token document. A fence held only in a proxy’s or primary’s memory would disappear exactly when it is needed.
Unresolved in the sources
> or ≥, whether tokens are per chunk or per shard, and how the target shard learns its token are not specified[1]. ByteByteGo restates it as “newer than the earlier one”[13], which does not settle it either. Do not build on either reading without testing it.Where it can go wrong. The post describes the happy path. The matrix below is our analysis of where a Coordinator crash, or a lost message, leaves the system.
Coordinator dies…
State left behind
Client impact
Recovery
Steps 1–4
Target partly loaded or replicating; source serves normally
None
Resume from checkpoint, or abandon and drop the target copy
After fence, before route update
Source rejects the chunk; metadata still routes to source
Every request for the chunk fails until someone acts
Finish the drain and publish the route (forward), or publish route = source at the new version (back). Either way it is one metadata write
After route update, proxies not yet refreshed
Stale proxies send the old token to the source
One rejected attempt per stale proxy, then a refresh
Self-healing: the rejection is the invalidation signal
After switch, problem found on target
Reverse replication keeps the source current
Another sub-2 s switch
Run the switch in reverse (inferred from the stated purpose of bidirectional replication)
After deregistration
Source chunk data dropped
—
No fast path back: rollback is now a new forward migration
Stripe does not describe Coordinator failure handling, where migration state is persisted, or how a stuck fence is detected. The red row is the protocol’s only window in which availability depends on the control plane. Any implementation needs a watchdog that notices a fenced chunk with no route change.
The red row is the one to design for. Between the bump and the route update, the chunk is unavailable by construction. The source refuses it and the metadata still points at the source. Normally that lasts under two seconds. If the Coordinator dies in that window, or the metadata service is unavailable, the fence stays up until something finishes the protocol. A useful property of versioned gating is that recovery in either direction is a single metadata write. Publish the route to the target and finish forward. Or publish the route back to the source at the new version, so proxies send a token the source will accept. Neither requires touching the shards. The post does not say how Stripe persists Coordinator progress or detects a stuck fence. The desired-state store in F3 is a hint, not an answer.
Both client profiles and all rates are ours, carried over from the retry-envelope diagram in section 4. The model ignores jitter and give-up-then-resubmit at higher layers. What it shows: the protocol has a cliff, not a slope. Below the shortest retry envelope in the caller population a fence costs zero errors. Above it, failures rise quickly toward 100% of the chunk’s traffic, and retries multiply the rejected load on the source by three to four times while they do.
Our analysis — how a stuck fence unfolds, and what the budget really is
The failure has a threshold. For the first second or two a stuck fence looks exactly like a normal switch: rejections that retries absorb. Once it outlasts the shortest retry envelope in the caller population, requests start failing outright, and the callers with the tightest budgets go first. Meanwhile every blocked request makes three to four attempts against the source, and each one has to be parsed, routed and rejected. That load lands on the proxy tier and the fenced shard, both shared with healthy chunks. This is the same shape as the 2019 cascade, where one shard’s unavailability “starved compute resources for the API”[5], but here it is limited to one chunk’s traffic, not a whole shard’s.
So the number that matters is not “under two seconds”. It is the gap between the fence’s tail latency and the shortest retry envelope among the callers. Stripe states the target as “in line with the retry budget of our product applications”[1], which implies someone knows that envelope per caller. If you copy the design, the watchdog on a stuck fence should fire and resolve well inside that minimum envelope, not at a round-number timeout. With the illustrative profiles above, that means 1 second for the pink client, not 30.
Counterfactual — retry inside the proxy?
2,500
avg queries/s per shard
≈ 48 MB/s
1.5 PB over a year
≈ $60K/s
2025 payment volume
≈ 23x
2019 vs five-nines budget
Per-shard load is modest, and that is the point. Five million queries per second over 2,000+ shards averages about 2,500 queries per second per replica set[1]. Traffic is certainly skewed, so the hottest shards will be far above that. Still, the average is well inside what one MongoDB replica set handles, which suggests Stripe sizes shards for blast radius, recovery time and headroom, not to the ceiling of a node. That fits the “fraction of the original data” framing and the 2026 figure that a node rebuild takes about 33 times as long as a config change[16]. When rebuild time is the expensive recovery path, you keep the data each node must rebuild small.
Data per shard, bounded. Stripe says “petabytes” without a figure[1]. At 2 to 5 PB over 2,000 shards, the average is 1 to 2.5 TB per shard, which fits the note that “several” shards still hold terabytes. Before the platform, shards held “tens of terabytes”. At 20 to 50 TB each, petabytes of data implies somewhere between about 40 and 250 shards, which fits “a small number”. Both ranges are ours. The order of magnitude is a roughly tenfold drop in data per replica set, with the fleet getting much wider.
What 1.5 PB in a year means. Stripe bin-packed “thousands of underutilized databases by migrating 1.5 petabytes of data transparent to product applications” in 2023[1]. Spread over a year that is about 4.1 TB a day, or about 48 MB/s sustained, which is trivial as an aggregate. The constraint was never bandwidth. It was per-target ingest (which the 10x addressed), per-source impact (designed out), and the number of switches the fleet can absorb, which is effectively unlimited, as the next paragraph shows. The post gives no duration for the programme. It could have taken a quarter or the whole year.
Reading A · 2,000+ is post-consolidation
Consistent with the 2026 count and with “thousands of database shards”. We think this is the likelier reading.
Reading B · 2,000+ is pre-consolidation
This would require the fleet to grow fourfold again by 2025 and contradicts “thousands”. It is possible, but nothing supports it.
Stripe uses two different units: “thousands of underutilized databases” were bin-packed (physical databases), and shard count (replica sets) fell by about three quarters. Neither the ~8,000 nor the ~500 is a reported number.
What the consolidation says about the shards it emptied. Under the more consistent reading above (about 8,000 shards reduced to about 2,000), about 6,000 shards were emptied. If the 1.5 PB was mostly their data, the emptied shards held on average about 250 GB each. That is small next to the “tens of terabytes” of the pre-platform era, and it matches Stripe’s word “underutilized”. The post’s own framing (split “during traffic surges”, consolidate “when traffic is low”) treats this as a normal cycle, not a mistake to be corrected. Elasticity that only splits leaves a fleet of mostly idle replica sets, each paying for a primary and several secondaries. The bin-pack is the other half of the same capability. We do not know how much it saved, because no cost figure is published.
The comparison flatters the switch in one way: a switch is scoped to one chunk’s traffic, while the budgets and 2019 bars are fleet-wide. Counting a switch against the fleet-wide budget would need that chunk’s share of traffic, which is not published.
Switches against the availability budget. Five nines is about 5.3 minutes of unavailability a year. Stripe’s newer 99.9995% target[17] is about 2.6 minutes. A switch blocks one chunk for under two seconds. Assume, for illustration, that a chunk carries a tenth of its shard’s traffic and the shard carries 1/2,000 of the fleet’s. Then one switch costs about 2 × 1/20,000 = 0.0001 fleet-seconds of failed requests, before retries recover them. The five-nines budget would fit several million such switches a year. Migrations are not what threatens Stripe’s availability budget. A single 2019-style event is: 27 plus 93 minutes of severe, API-wide degradation is about 7,200 seconds, roughly 23 times a full year’s five-nines allowance[5]. The platform is worth building because it makes the routine case nearly free and reduces the blast radius of the catastrophic case. It does not rule the catastrophic case out.
Money in flight. Stripe processed $1 trillion in 2023[1][12] and the 2,000-shard fleet handled $1.9 trillion in 2025[16]. Averaged, that is about $32,000 per second in 2023 and $60,000 per second in 2025. Spread evenly over 2,000 shards, 2025 averages about $30 per second per shard. A two-second fence on a tenth of one shard’s chunks delays single-digit dollars of payment volume, and the retries recover them. Uneven distribution changes the constant, not the conclusion. For scale on the peak: over Black Friday to Cyber Monday 2023, Stripe handled 300 million transactions and $18.6 billion in volume with API availability above 99.999%[11]. Over those four days that averages roughly 870 transactions per second, three to four orders of magnitude below the database query rate. The 5M QPS figure is not a BFCM number, so the ratio is loose. It still tells you that most of DocDB’s load is not the payment write itself but everything around it.
Our analysis — the headline number that matters least
The post does not discuss alternatives. Everything in this section is our framing of the options a team in Stripe’s position would realistically have weighed, tested against what the sources do say.
App changes per move
Source shard load
Cutover correctness
Rollback
Fleet-scale ops
App-level dual write (Stripe 2017 pattern)
per migration, code
double writes
app-defined; Scientist
flip flags back
one team at a time
Replicate from source oplog
none
reads on source
same as built
same
capped by oplog window
Pause at proxies, no node fence
none
none
needs barrier across all proxies
same
barrier cost grows with fleet
Dual-serve during switch
none
none
multi-writer; conflicts
easy
hard to reason about
In-place engine upgrades
none
rolling restarts
n/a
per-version, fleet-wide
every hop repeated
Built: snapshot + CDC + node-enforced fence
none
snapshot reads only
single writer by construction
reverse replication
any-to-any, unlimited concurrency
Keep using application-level dual writes (the 2017 pattern). It works, and Stripe proved it on hundreds of millions of objects[2]. But each migration needs code changes, Scientist experiments and a product team’s attention for weeks. That is fine for a schema change, which only the owning team can do anyway. It is not workable for moving thousands of databases a year. The two are tools for different layers. Dual writes change what the data is. The platform changes where it lives. No source says the platform replaced dual writes for schema changes, and it almost certainly did not.
Replicate from the source oplog. This is simpler, needs no Kafka dependency, and has lower latency at the fence. Stripe rejected it for the two reasons in section 5: read load on the source, and the oplog-window ceiling. The counterfactual is useful. Had they done this, migrations of the biggest and busiest shards (the ones that most needed to move) would have been the ones most likely to fail with an oplog rollover. The platform would have worked best where it was needed least.
Pause at the proxies instead of fencing at the shard. This avoids patching MongoDB, which is a real cost. A custom patch has to be carried forward through every engine upgrade, including the fork-lift upgrade the platform itself performed. But correctness then depends on a barrier across the whole proxy fleet, and one proxy with a stale route and a missed message writes to a location that is no longer authoritative. Stripe chose to carry a patch rather than depend on a fleet-wide barrier. That is the right call once the proxy fleet is large and correctness failures mean lost money.
Dual-serve during the switch. Let both shards accept writes briefly and reconcile. This gives zero failed requests and replaces them with conflict resolution on financial records. Nobody who has operated multi-writer replication would pick this for payment intents. The fence gives up a two-second hard stop for chunk-level single-writer semantics, and that is a good trade.
Upgrade engines in place. This is the default everywhere. The 2019 RCA shows its failure mode at Stripe: a phased, well-tested minor upgrade that hid a fault for three months[5]. In 2012 there were similar problems, such as a production upgrade that ran faster than staging and left “no eligible primary node”[8]. Fork-lifting “to a later version of MongoDB in one step—without going through intermediate major and minor versions”[1] removes the intermediate hops, each of which is a fleet-wide change. It also gives every chunk a rollback target on the old version while the source copy exists. The cost is temporary capacity: the fleet must run old and new shards side by side for the duration.
Buy instead of build. In 2011 there was nothing to buy[1]. By 2024 managed MongoDB existed, but Stripe’s list of requirements (admission control, a minimal function surface, enforced quotas per tenant, fine-grained authorization) describes a platform shaped around Stripe’s own applications, not a hosting product. Moving to a different database entirely would have meant rewriting every data-access path across 5,000+ collections. Brandur Leach’s 2017 personal-blog article on Postgres idempotency keys remarks that “none of this is possible on a non-ACID store like MongoDB”[14]. That was one engineer’s view of a reference design, not a description of Stripe’s production. The July 2026 fleet post describes a fleet that is still MongoDB-based[16].
Counterfactual — no platform at all
Most of these have come up in earlier sections. They are collected here because they are the claims you will hear repeated in design reviews.
1. “Zero downtime” means zero failed requests. It does not. The moving chunk is hard-stopped, requests are rejected, and F4 shows a 5xx to the client app[1]. The precise claim is “the interruption fits inside the retry budget” (section 4). Without retrying clients, the same design is a two-second outage per chunk move.
2. DocDB runs at 99.999%. The figure is Stripe’s 2023 uptime; the post credits the databases with supporting it[1]. No DocDB or migration SLA is published. The 99.9995% is a 2026 target from a service-mesh post[17].
3. “5 million QPS” is the achievement. Divided across 2,000+ shards[1], it averages about 2,500 queries per second per replica set (our arithmetic, section 7). The achievement is moving data between them.
4. Stripe shards by customer ID or timestamp, and builds indexes first for query performance. Both are ByteByteGo’s illustrations[13]. Stripe has not published its shard key; F2 is an example[1].
5. Stripe moved off MongoDB / runs on Postgres. The 2024 and 2026 posts both describe a MongoDB-based fleet[1][16]. The Postgres idempotency article usually cited is a personal-blog reference design[14].
6. MongoDB caused the 2019 outage. The RCA never names the database; it blames “two different database bugs and a configuration change”[5]. HN commenters named MongoDB and no Stripe staffer confirmed it[6]. Consistent with, not attributed to.
7. This is the 2017 dual-write pattern at scale. Different layer. 2017 changed what the data is, with code in the owning service[2]; 2024 changes where it lives, with no application change[1]. (The 2017 post originally said “tables” and was edited after readers pointed out Stripe ran MongoDB[15].)
8. The clever part is the cutover. The cutover is a textbook fencing token. What the retellings skip is the precondition that makes a two-second fence possible at all: a CDC pipeline trustworthy enough to sit in the write path of every moving chunk, with end-to-end latency well under the fence (section 6), and retention long enough that migration duration is unbounded (section 5). Teams that copy the fence onto a best-effort CDC feed get the hard part wrong while getting the famous part right. This is our analysis, not a Stripe statement.
The 2024 post ends with two plans: a “heat management system that proactively balances data across shards based on size and throughput” (then being prototyped), and investment in “shard autoscaling that dynamically responds to changes in traffic patterns”[1]. Both are natural next steps. Once a chunk move is a cheap, safe primitive, the next question is who decides when to move. Neither has been confirmed as shipped in any later source we have. The 2026 fleet post covers remediation, not placement or autoscaling[16].
What the July 2026 post does show is the fleet the platform produced and the next bottleneck it exposed. A MongoDB shard is described as “one of 2,000 that collectively processed $1.9 trillion in payments in 2025”. The fleet spans “over 40 distinct shard layouts”[16]. The pain had moved from moving data to keeping some 2,000 replica sets healthy. In one six-month window the control plane paged 124 times for misconfigured shards and 32 times for single-node-down[16]. The answer was to model shard configuration as a state graph and search it for a path to health. Breadth-first search comes first, then Dijkstra with weighted operation costs when full health cannot be reached (a node rebuild being about 33 times as expensive as a config change). Remediation workflows run on Temporal, with operations such as vote assignment, priority adjustment, node rebuild and oplog resizing[16]. In the post’s words, “recovery becomes a search problem”, and “what you simulate is what you execute”[16]. The reported result: a 30% reduction in pager volume (about 200 fewer pages a year) and “12 fewer days of shards stuck in unhealthy states annually”[16].
Our analysis — the 2026 numbers, worked through
Paging rate. If “about 200 fewer pages” is 30%, the baseline was about 670 pages a year and the remainder about 470. Over roughly 2,000 shards that is about 0.23 pages per shard-year. Low per shard, but still more than one page a day for the team. The two categories Stripe names (124 + 32 in six months, about 310 a year) are close to half the implied baseline on their own. Width is the cost driver: every per-shard rate, however small, is multiplied by the fleet size the platform made cheap.
Unhealthy time. Twelve days a year is about 290 shard-hours, or under nine minutes per shard-year across 2,000 shards. “Unhealthy” here mostly means misconfigured or a node down, which is reduced redundancy, not unavailability. By the argument in section 2, though, exposure time is exactly what the expected number of double faults scales with. Cutting it is the fleet-level form of the same trade the platform made per shard: it does not prevent failures, it shrinks the window in which a second one hurts.
The interaction the posts do not cover. Two controllers now mutate the same shards: the Coordinator (imports, fences, drops) and the remediator (votes, priorities, rebuilds, oplog resizing). A remediation that triggers an election on a source shard mid-fence, or rebuilds a target mid-import, is exactly the kind of cross-controller race that neither post addresses. The token-in-a-replicated-document design (section 6) suggests the fence survives an election; whether the two controllers coordinate beyond that is not published.
Our analysis — the arc
What is still not public, and what you should not assume when citing this case: when the platform was built and by how many people (an HN question on staffing went unanswered[7]); the shard-key design, chunk size and split policy; how the chunk metadata service is built, how proxies cache routes, and what happens when it is unavailable; the MongoDB patch itself and the exact token semantics; how write tags are implemented; what the correctness check compares and what happens on a mismatch; lag thresholds and cutover latency distributions beyond the upper bound; how long a migration takes per terabyte, and how long the 1.5 PB programme ran; the MongoDB versions involved in the fork-lift; cloud and hardware; Kafka topology and retention; cost savings; and any incident caused by a migration, or any rollback that used the reverse path. None has been reported.
Copy it when
Do not when
Routing layer
Every query already passes through a proxy you own and can change weekly
Clients connect straight to the database with driver-side routing
Client contract
Writes are idempotent and every caller retries with a budget of seconds
Some callers give up after one fast retry, or retries can double-apply
Change log
A durable, ordered, replayable per-shard log already feeds Kafka-like storage
CDC is best-effort, lossy, or read from the primary
Enforcement point
You can fence at the storage node (patch, permission, or per-chunk flag)
Your only lever is telling N proxies to stop and hoping they all heard
Scale of movement
Hundreds to thousands of shards; moves are routine, not quarterly events
A handful of shards; a managed resharding feature would do
Team
A dedicated database-infrastructure team for the life of the platform
Nobody owns the Coordinator once the builders move on
Lesson 1: set cutover targets against an event your clients already survive. Measure your clients’ real retry envelope (attempts × backoff, per caller, including the ones that give up after one retry). Then design the cutover to fit inside it. If you cannot state the envelope, you do not have one, and “zero downtime” is not available to you until you do. The SRE / Reliability track’s material on retry budgets and backoff is the prerequisite here, not an extra.
Lesson 2: idempotency is infrastructure, not an API nicety. Stripe’s decision in 2012 to make all writes idempotent[8] is what made the 2024 cutover protocol cheap. If you are starting a system that will one day need online resharding, this is the cheapest time to make that decision. Retrofitting idempotency across hundreds of services later is a bigger project than the migration platform itself. The API & Integration Design track covers the key-storage and conflict semantics.
Lesson 3: enforce the fence where the data lives. If your switch depends on every router hearing a message, it will one day fail when one router does not. The node-side check can be a patch (as at Stripe), a per-chunk permission revocation, or a conditional write against a version stored in the partition itself. The requirement is that the authoritative copy refuses stale writers on its own, and that the fence is replicated with the data so it survives a failover.
Lesson 4: replicate from a durable log you already trust, not from the source. This only works if your CDC pipeline is correct enough to be part of the write path: per-shard ordering, no loss, an exact snapshot/log seam, and retention longer than your longest migration. If your CDC today is a best-effort analytics feed, fix that first (the Data Engineering track’s CDC chapter covers the failure modes). Otherwise your migration platform inherits that pipeline’s bugs as data corruption.
Lesson 5: make every step before the switch reversible and cheap to abandon. Stripe’s steps 1 to 4 touch the source only through snapshots and can be dropped without clients noticing. That is what makes “no restriction on in-flight migrations” safe to promise. If your pre-switch steps load the source, concurrency limits come back, and with them the scheduling problem the platform was supposed to remove.
Lesson 6: treat deregistration as a policy decision. Decide how long to keep the source copy based on how late a problem could plausibly surface, and write that down. Stripe’s own history includes a fault that took three months to appear[5].
Pitfalls when copying this
The stuck fence. Build the watchdog before the first production switch: alert on any chunk that is fenced at the source with no route change after a few seconds, and give it an automatic resolution (forward or back).
The long-running request at the fence. Decide and test what happens to an operation already executing on the source when the token is bumped. Stripe has not published its answer.
The index that did not get sorted. Sorted bulk load helps the index you sorted for. Measure import throughput per collection before promising a migration duration for a collection with several unrelated secondary indexes.
Loop tags that leak. If the tag marking replicated writes can be lost (a field stripped by a transform, an operation type that carries no metadata), bidirectional replication turns into an infinite echo. Test the reverse path as thoroughly as the forward one. It is the path you will use under pressure.
Splitting into fan-out. Each split can raise the shard count of queries that span it. Track the fan-out distribution of queries alongside shard sizes, or splitting will turn a single-shard dependency into a many-shard one.
The patch you now own. A custom engine patch has to be ported to every future version, including the one you fork-lift to. Budget for it permanently.
Our analysis — questions to ask in the design review
1. What is the shortest retry envelope among your callers, and who measured it? The answer should be a number per caller class, not “clients retry”. The fence’s p99.9 has to sit under the minimum, not the median.
2. Is every mutation behind the router idempotent, or only the ones behind the public API? Internal writers that are not idempotent turn the clean rejection into an unsafe retry the moment someone adds a timeout path.
3. Where does the fence live, and does it survive a failover of the fenced node? If the token is not replicated with the data, an election during the switch removes the fence.
4. What is the target’s spare apply capacity, and what admission rule stops ten migrations converging on it at once? “Unlimited concurrency” without a convergence rule gives you lag that never closes.
5. Show me the snapshot/log seam. How is the snapshot tied to a log position, what makes replay across the overlap idempotent, and what does the validator do on a mismatch: block the switch, restart, or page?
6. Which controllers can touch a shard mid-migration, and how do they exclude each other? Remediation, upgrades and autoscaling all mutate the same replica sets as the Coordinator.
7. How long is the source kept after the switch, and what is the rollback drill? If the reverse path has never been exercised in production, it is not a rollback plan.
8. What does a query’s shard fan-out distribution look like before and after the split? A split that raises the median k trades a size problem for an availability one (section 3).
Who should not copy this. If you run a handful of shards, use whatever resharding your managed database offers and spend the savings elsewhere. The design pays off when data movement is continuous (splits, merges, upgrades and tenant moves happening all the time) and the fleet is large enough that a fleet-wide barrier or per-migration application work would dominate. The platform also needs owners. Stripe has a dedicated database-infrastructure team and has run MongoDB since 2011. A version of this built by a team that moves on after it ships will fail at step 5 at some point, and nobody will remember how the fence works. The Architecture Judgement essays on build-versus-buy and the Technical Leadership essays on platform ownership cover that decision.
The short version. Stripe did not solve online data movement by finding a clever algorithm. It started from a client contract (idempotent writes, retry through a failover), a durable change log and a routing layer it owned, and put a single node-enforced fence on top. Each part is well understood on its own. What is uncommon is having all of them in place at once, and that took Stripe about a decade.
Numbers in square brackets throughout the text refer to this list. “Our analysis”, “Reconstruction (inferred)” and “Counterfactual” callouts are Algoroq’s reasoning, not claims by these sources.