Shopify gets cited for two claims that seem to contradict each other: that a Rails monolith can carry the world’s largest flash sales, and that surviving them takes hundreds of isolated database shards. Both are true because they answer different questions. From roughly 2015 on, Shopify made the shop the unit of data placement, failure, disaster recovery and load balancing. It deliberately did not make the shop, or the business domain, the unit of deployment. This case study assembles eleven years of Shopify’s engineering posts, talks and retrospectives into one account and runs the numbers they publish. Our thesis is that the transferable idea is not “pods”. It is the invariant underneath them: every unit of work touches exactly one tenant’s datastores. Everything else (the Lua routing layer, the shop mover, the modular monolith, even a 2026 move of inventory reservations out of Redis) follows from protecting that invariant.
100+
Pods by Aug 2018 [13]
< 10 s
Write pause per shop move [5]
$5.1M/min
Peak sales, BFCM 2025 [21]
2.8M
Lines of Ruby in the monolith (2020) [9]
A Rails monolith in continuous development since 2006 [9] is not unusual. The workload is. Shopify’s first major flash sale was in 2007 [6]; by 2016–17 Kylie Cosmetics sales came “every seven or eight days” [6]. A flash sale is not a traffic spike in the ordinary sense. It is a single tenant becoming, for minutes, a large share of the platform’s write load, concentrated on a few products and one code path. The 2017 posts say why that path was expensive: “every checkout session created a new record in MySQL and… every step in the flow modified that same record” [6]. Every buyer cost one insert plus an update per step, all on a primary. Reads can be cached and replicated; these writes cannot.
Shop-owned tables carry shop_id, and it is the sharding key [2]. New merchants are “assigned to an arbitrary shard” at signup [2]. In data terms the product is millions of small, almost disjoint databases sharing a schema. That premise is what made everything below possible, and it also contains the trap described next.
Our analysis: the shard key that isolates you also caps you
shop_id is the shard key, every checkout for one shop lands on one MySQL writer. Sharding can spread a thousand shops across a thousand writers; it cannot spread one shop across two. A single tenant’s peak checkout rate is therefore bounded by one writer’s write capacity, minus what its pod-mates consume, divided by the writes per checkout. Fleet size does not appear in that expression.Our model; every number on the axes is illustrative, not Shopify’s. The two structural levers compound multiplicatively: cutting Δ writes from w multiplies the ceiling by w/(w−Δ); evicting pod-mates that hold share r multiplies it by 1/(1−r). At 4 writes per checkout, removing two doubles the ceiling, the same as evicting pod-mates that use half the writer. At 12 writes, removing two buys only 1.2×, less than evicting a 20% pod-mate share (1.25×). Work out which regime your largest tenant is in before you choose which lever to fund.
2006
Rails monolith begins; storefront code will live in it for 15+ years
[9 · 28]
2007
First major flash sale (Colts vs. Bears Super Bowl)
[6]
2014
Sharding across “dozens of database partitions”; BFCM resiliency push yields Semian + Toxiproxy
[13 · 23]
2015
“no longer possible to continue buying a larger database server” — shard
[1]
Feb 2016
Kylie Cosmetics sale takes down every store on its shard
[6]
2016
Runtime architecture reorganised into pods; monolith recognised as a problem
[1 · 8]
Early 2017
Componentization starts: ~6,000 classes catalogued
[8]
Feb 2017
Posts describe the edge checkout throttle (built after the 2016 outage) + fair queue in OpenResty/Lua
[6 · 7]
2017
Bare metal → GCP + Kubernetes
[15 · 31]
2018
Over 100 pods; BFCM: ~100K RPS peak, 99.98% uptime
[13 · 14]
2019
PD-snapshot backups, RTO under 30 min; autoscaler saves “a few $M every month”
[24 · 31]
2020
Limits hit “within a shard”; storefront renderer extracted; Packwerk open-sourced; 37 components
[31 · 28 · 9 · 10]
2021
Fleet upgraded to Percona MySQL 5.7.32 (Aug–Sep); shard balancing with Ghostferry published (Sep)
[2 · 25]
Jan 2024
Packwerk retrospective (privacy checks gone); Shop app picks Vitess, not pods
[11 · 27]
BFCM 2025
$14.6B; peak $5.1M/min; five scale tests Apr–Oct up to 200M RPM
[19 · 21]
May 2026
Inventory reservations Redis → MySQL, rolled out pod by pod
[26]
Source disagreement
One organisational fact matters for transfer. Before BFCM 2014 the infrastructure team prioritised resiliency, producing Semian (circuit breakers plus a SysV-semaphore bulkhead) and Toxiproxy (TCP fault injection) [23]. Fault injection predates pods at Shopify. Pods move failures into smaller boxes; something still has to prove the walls hold, and Shopify had that tooling first.
The pods post states the problem plainly. After sharding, code like Sharding.with_each_shard do … end was scattered through the platform, and “If any of our shards went down, that entire action would be unavailable across the platform” [1]. Sharding removed the capacity ceiling. It did not remove shared fate, and because every shard added to the fleet was another dependency of every fan-out action, each new shard made the fan-out actions less available.
Left: the pattern Shopify named as the problem. Right, our arithmetic: if a fan-out action needs every shard and each shard is independently up a fraction a of the time, the action is up aᴺ. At 100 shards and 99.95% each, it is 95.1%, worse than any single shard by a factor of about 100 in downtime. Sharding multiplies capacity and, without isolation, multiplies exposure too.
The arithmetic is worth doing because it is the whole argument. Suppose each shard is independently available 99.95% of the time. That is about 22 minutes of unavailability in a 730-hour month. A fan-out action that needs all 100 shards is available 0.9995¹⁰⁰ ≈ 95.1% of the time, about 36 hours a month, which is roughly a hundred times worse than any single shard. Shard failures are not independent in real life (shared hardware, shared deploys, shared migrations), so the true number differs. It is not better in the way that matters, though: correlated failures do not make fan-out actions safe, they make every shard fail together. Growing the fleet makes this worse at exactly the rate the business is succeeding.
Then in February 2016 a Kylie Cosmetics sale “took down not just her store, but all others on the database shard” [6]. This is the second failure mode, and it is different from the first. The first is fan-out coupling: an action that spans shards fails when any one shard fails. The second is co-tenancy: one hot tenant saturates the shared resource its neighbours depend on. Sharding causes the first problem. It narrows the second without fixing it, because the Kylie sale damaged one shard rather than the whole database.
Our analysis
The pods post names the resulting design goals: “Horizontal scalability”, “Adding a new pod won’t cause unexpected interference with other, pre-existing pods”, and “Serving a request only requires a single pod to be online” [1]. Put in failure-domain terms, the third goal matters most. It changes the platform’s availability from a product over shards into, for any one shop, the availability of a single pod plus the global pieces. In 2016 Shopify “sat down to reorganize Shopify’s runtime architecture” around it [1].
Three official definitions (2018 “a set of shops that live on a fully isolated set of datastores” [1]; 2021 “an individual MySQL database shard, along with other datastores like Redis and Memcached” [2]; 2022 “basically a complete version of Shopify” [5]) read together give a picture narrower than the slogan. A pod is one MySQL shard, its Redis and its Memcached [2][13]. It is not a copy of the compute.
The compute is shared. “Outside of the isolated datastores, pods also have shared resources such as job workers, app servers, and load balancers” [1]. The 2022 talk repeats it: “while the datastores are isolated the stateless workers are shared” [5], and those workers are shared across pods in a region, which lets load be spread when one shop is running a flash sale [5]. What stops that sharing from recreating shared fate is a rule that is easy to state and hard to enforce: “All shared resources can only ever communicate to a single pod at a time” [1]. In practice every request and every background job operates on one shop [5].
Edge · OpenResty (NGINX + Lua) · shared
[5][6][7]
Bot filtering
before app servers [5]
Checkout throttle + queue
leaky bucket · signed cookies [6][7]
Sorting Hat
host → pod_id · adds header [1][5]
Routing table
separate, non-sharded DB [2]
Stateless tier · shared across pods in a region
[1][5]
Rails app servers
read pod header → connect to one pod
Job workers
one shop per job [5]
Rule
“only ever communicate to a single pod at a time” [1]
Pods · isolated datastores · active in one region
[1][2][13][25]
pod 1
MySQL writer + 5+ replicas
Redis · Memcached
shops: 1..many
pod 2 · flash sale
MySQL writer + 5+ replicas
Redis · Memcached
shops: 1..many
pod 3
MySQL writer + 5+ replicas
Redis · Memcached
shops: 1..many
… “more than a hundred” (2018)
hundreds of shards (2021)
Passive copy in a second region
replication active → passive; whole-pod failover [5]
Pod Mover
to recovery DC “in a minute” [1]
Outside the pod model
[5][9][13][27][28]
Storefront Renderer
separate app · read replicas [28]
CardSink / CardServer
card data never enters monolith [5]
Shop Pay · Shop app
cross-shop · Shop app on Vitess [5][27]
Data warehouse (Presto)
cross-shop analytics [5]
100+ other services
other production services (2018) [13]
Solid facts are cited. The arrangement into lanes, and the assumption that the stateless tier is regional, is our reconstruction: [5] says the stateless workers are shared so Shopify can “balance the load in a region”; no source describes the routing table’s caching or distribution to the load balancers.
Routing. Something has to decide, before the application runs, which pod a request belongs to. That is Sorting Hat, “a piece of software in our load balancers” that “matches every request to a pod and adds a header to that request” [1]. In the 2022 description it is “a Lua module called Sorting Hat that looks up the host in the routing table and finds the corresponding pod that this shop belongs to” [5], running at an OpenResty (NGINX + Lua) edge that also blocks bots [5]. Lua at the edge was not new. By 2017 Shopify already used it for multi-DC HTTP routing, request fan-out across shards and DDoS mitigation [6]. The routing table itself “lives in a separate database that isn’t sharded”, and moving a shop means changing that shop’s pod_id there [2][5].
Geography. In the 2018 post each pod was assigned “a pair of data centers”, with one active at a time [1]. After the 2017 move to Google Cloud and Kubernetes [15][31], the 2022 description is regions: a pod is “active in a single region at a time but exists in two with replication set up from the active to the non-active region. We can failover an entire pod to another region” [5]. The MySQL fleet behind this was “thousands of VMs across hundreds of shards” in 2021, with a writer and five or more replicas per shard and up to four replication hops [25]. In 2019 it spanned three GCP regions [24], and us-central, us-east and europe-west4 are the regions named for BFCM 2025 [19].
Resource
Scope
Evidence
Blast radius if it fails
MySQL shard
1:1 with pod in [2]; writer + 5+ replicas [25]
Contained to the pod’s shops
Redis
[2][13]; shop-move copies Redis jobs/state [5]
Contained
Memcached
[2][13]
Contained
App servers
[1][5]
Every pod; absorbs a hot shop’s burst
Job workers
[1][5]; each job touches one shop
Every pod; queue backlogs are shared
Load balancers / edge
[1][5]
Every pod; also the throttle point
Routing table
Separate non-sharded DB [2]
Every request depends on it
Kafka · Elasticsearch
Used [13][17]; scope per pod unknown
Kafka overload once needed a regional drain [29]
A pod is a data failure domain, not a compute failure domain. The popular picture of “a full copy of Shopify per pod” is Bart de Water’s conceptual framing [5]. The runtime reality in [1] and [5] is shared stateless compute that is only allowed to talk to one pod per unit of work.
Reconstruction (inferred)
Reconstruction (inferred): the connection bill for pooled compute
Our model; process counts, pool size and the proxy cap are illustrative. The dangerous property is the slope: a peak scale-up of the shared tier triples connections into every writer, including every pod whose traffic did not change. The failure then shows up on a quiet pod, and a dashboard keyed to per-pod QPS will not explain it. Budget the connection count as (processes × pool × fraction of pods touched), and alert on it per writer.
The cell boundary sits around state only; compute stays pooled. Section 4 argues that this apparent compromise is the most important economic decision in the design.
The rule is simple: any request or job touches the datastores of exactly one pod [1][5]. What it bans matters more than what it allows. It rules out any feature that needs two shops’ data in one code path, and a commerce platform has plenty of those. Shopify’s answer, in the 2022 talk, is that cross-shop features are built as separate applications (Shop Pay is the example given), and cross-shop analytics goes through a data warehouse (Presto) [5]. The Shop app, a consumer product whose users span every merchant, went further. In 2024 it scaled on Vitess instead of pods, because “While Shopify has millions of merchants, Shop app has an order of magnitude more users. The impact of one user is small compared to the impact of a large merchant” [27].
That Shop app sentence is the best statement in any Shopify source of when pods fit. Pods work for tenants that are few relative to the load each one can generate, and lumpy. A flash-sale merchant can be a large share of a pod’s write traffic. A single Shop app user never can. For uniform, fine-grained keys, a keyspace-sharding proxy that hides the topology is simpler and good enough. For lumpy tenants you want to be able to name the tenant, route it, move it and account for it, and a transparent proxy works against all of that.
In return the rule keeps checkout a single local transaction. Bart de Water gives the reason checkout was never extracted from the monolith: it “needs to write to a lot of things”, so extracting it would be “a monumental undertaking” [5]. Under the single-pod rule, all the things checkout writes for a shop sit in one MySQL database. Orders, inventory and payment state can commit or roll back together without a saga, a two-phase commit or an outbox. The 2026 inventory-reservation rewrite, covered in Section 12, is Shopify reinforcing this invariant a decade later. Reservations moved into MySQL because “reservations and the inventory ledger lived in two different systems” and could not be committed atomically [26].
Our analysis: why shared compute, not full cells
Two extractions are useful counterexamples because neither is about tenancy. The storefront renderer (2020) is read-only, runs on read replicas, and responds 4–6× faster than the code it replaced [28][5]. CardSink/CardServer keeps raw card data out of the monolith so the Rails app stays out of PCI scope [5]. One follows a workload shape, the other a compliance boundary. Our reading: those are the two extraction criteria that survive scrutiny in a system built on a local-transaction invariant, because neither service needs to join a write transaction with the core. A candidate service that must commit atomically with checkout fails both tests, however clean its domain boundary looks.
After the February 2016 outage, nothing yet solved the problem that Kylie Cosmetics posed every week or so, and the durable fix, making checkout cheaper, was out of reach. Rewriting checkout “wouldn’t be feasible in time, especially since the next crippling sale was the following week” [6]. What Shopify did have was a programmable edge. Lua in OpenResty let “a small team of production engineers” change the behaviour of the entire Rails application without touching it [6].
The first version was a leaky-bucket throttle on checkout that reset every 5 seconds. Buyers over the limit got a queue page cached in NGINX, which polled /checkout with JavaScript and fell back to a meta refresh. Buyers who got through received a securely signed cookie that let them skip the throttle for the rest of the session [6]. Once admitted, a buyer was never throttled mid-purchase, which is the right invariant for a funnel whose value is at the end. It protected the database. It was also unfair in a way customers noticed before dashboards did. Polling for a random slot meant “you were essentially placed into a lottery with everyone else” [6]. Some buyers waited up to 40 minutes, for a sale that itself lasted only 40 minutes, and then found the product sold out, and the team found out through “Twitter (which functions as a pretty good real-time customer-sentiment monitoring service)” [6].
Our model, not Shopify’s data: a 5-second poll that passes with probability q = 0.5% makes each buyer’s wait geometric and independent of when they arrived. The first buyer in line has the same 1-in-20 chance of waiting about 50 minutes as the last. The reported symptom, waits of up to 40 minutes ending in “sold out” [6], is exactly this tail. [7] reports that after the change, median and p95 queue times “are more in sync”.
Part II of the posts fixed fairness without adding shared state [7]. Each buyer gets a signed cookie with the timestamp of their first checkout attempt. Admission has two phases. First, a buyer’s timestamp is compared against a “Now Serving” threshold, “like the virtual version of the ‘Now Serving: 42’ counters at delis or the DMV”. Buyers early enough then face the leaky bucket. The threshold is “calculated independently on each load balancer to maintain the statelessness of our edge tier” [7]. A central queue store would have been a single point of failure with cross-datacenter replication problems [7], and at 2017 multi-DC scale it would have put a shared dependency in front of every checkout.
The threshold moves under feedback control. The target is for exactly the second phase’s limit to pass the first phase. The error is “the bucket limit minus the number of requests passed to the second phase”, and a PID controller adjusts the threshold to drive it to zero [7]. After MATLAB simulations the team kept only the proportional term [7]. The reported result is that median and p95 queue times “are more in sync”, with no further unfairness complaints [7].
Our analysis: why P-only is the right call here
The posts do not say whether this exact controller still runs. Later sources describe the edge as filtering bots before app servers [5], note that “Bots also hammer our systems much more than real buyers do” [5], and explain that flash-sale goods “could fetch double or triple the original price on the secondary market” [5]. So admission control at the edge became a product feature (protecting merchants from resellers) as well as a capacity tool. The architectural lesson carries over to other systems. Admission control belongs in front of the shared tier, it should be stateless, and it should be fair by arrival order. The part that is Shopify-specific is where the state lives: in signed cookies held by the client. The same idea applies to any bursty, single-resource contention problem, such as ticket drops, sneaker launches or vaccine appointment booking.
The pods architecture raises a question that sharding alone never forced into the open: which shops should share a pod? Initial placement is arbitrary [2], shops grow at very different rates, and a merchant that was small at signup can become a celebrity brand. By 2021 the gap was measurable. “The least used shard and most used shard vary by almost four times” [2]. Hot shards risk failure, and cold shards waste the capacity bought for them [2].
Only the two ratios are reported [2]. Bar shapes are our illustration (linear spread, equal total load in both panels); the headroom and 1.2× figure follow from that assumption, not from Shopify data. Capacity is bought for the hottest shard, so imbalance is paid for in idle capacity on every other shard. New shops are “assigned to an arbitrary shard” [2], so without an active mover the skew grows back.
The placement method is deliberately simple. Historical database utilisation and traffic put shards into cohorts such as high_traffic and low_traffic, candidate moves are simulated (for instance, moving “every Nth shop” from high- to low-traffic shards), and the simulated spread drops to “closer to two times” [2]. The numbers are worth working through. Assume utilisation is spread roughly linearly between the coldest and hottest shard. At a 4× spread the mean shard runs at 62.5% of the hottest one, and at 2× it runs at 75%. The fleet is sized so the hottest shard survives the peak, so halving the spread lets the same hardware carry about 20% more total load (0.75 / 0.625 = 1.2). That gain came from moving data, not buying machines, which is why a shop mover is a capacity tool as well as an operations tool.
The mechanism is Ghostferry, a Go library (MIT licensed, with Ruby integration tests) that Shopify built and open-sourced. It is inspired by GitHub’s gh-ost, but gh-ost works within one database, and Ghostferry copies across databases [3]. A shop move has three phases [2]:
1 · Copy and tail. Tables are iterated, selecting the shop’s rows by shop_id with SELECT … FOR UPDATE locking reads, several tables at once in threads. At the same time, the binlog in row-based format is streamed, filtered down to the shop’s rows, and applied to the target. The storefront keeps being served from the source pod throughout [2]. The locking read closes the race in which a row is copied and then updated before the tailer has seen that update. The binlog tailer is what makes an online copy converge. The Databases from the Inside track covers the binlog mechanics if you want the replication-protocol details.
2 · Cutover. When the binlog replay queue is small, the mover takes the writer side of an application-level multi-reader-single-writer lock backed by Redis. Every web request and job that mutates the shop’s data must hold a reader lock, so once the writer lock is granted, “no writes for the shop are executed on that pod” [2]. Ghostferry records the final binlog coordinate and replays up to it. Bart de Water’s version of the sequence adds that writes are queued during the lock, and that the shop’s Redis jobs and state are copied before pod_id changes [5].
3 · Flip, verify, prune. The routing table is updated, the lock is released, verifiers confirm the copy, and, once the move is deemed successful, stale data on the old pod is pruned [2]. Verification includes checking that no queries are routed to the source shard “for a period of time” after the move [2]. Ghostferry ships several verifiers (inline, iterative, compression and target) [3], which run before, during and after the move [2].
Ordering from [2] and [5]; lane layout, relative widths and the pause budget are our reconstruction (no per-step durations are published). The idea that carries the design: the lock is taken at the application level around units of work, not in MySQL. So the pause is the sum of ①–④, and every term is set by the shop’s current write activity and tailing lag, none directly by table size. Size enters only indirectly: bigger shops tend to have more in-flight work and a longer replay tail, which fits the reported under-20 s for larger stores versus under-10 s [5]. One slow lock-holding job can blow the budget for any shop. The practical rule that follows: bound the maximum duration of any lock-holding job before you rely on the mover.
The result: “We can do this for a lot of stores with less than 10 seconds of downtime”, and less than 20 seconds for larger ones [5]. The 2021 post’s title says “zero-downtime”, and the body says “virtually zero consumer-facing downtime” [2]. Both are accurate descriptions of different things. Storefront reads never stop. Writes pause for seconds. Ghostferry’s own documentation, written for the generic case, gives the downtime as “on the order of seconds to minutes” [4]. The gap between that and Shopify’s under-10-seconds figure comes from the application-level lock, which lets Shopify drain precisely the units of work that matter instead of stopping the database.
Our analysis: what the mover depends on
The same property shows up as a rollout strategy. When Shopify moved inventory reservations from Redis to MySQL in 2026, the rollout went “pod by pod, starting with low-traffic pods and working up to our highest-volume merchants” [26]. A pod is a data boundary and a failure boundary, and it also serves as a canary cohort whose members you choose. Once tenants can be moved, you can decide which tenants a risky change reaches first.
The 2018 post argued that isolation pays off twice. Pods are “a convenient atom for our disaster recovery tooling” [1]. Pod Mover moves “a pod to its recovery data center in a minute without dropping requests or jobs”, and whole data centres can be evacuated one pod at a time [1]. The post promised a follow-up on how Pod Mover worked. We could not find one.
The cloud-era version replaces data centre pairs with regions: a pod is active in one region with replication to a second, and “We can failover an entire pod to another region” [5]. Traffic shifts at the DNS layer as well. Shopify’s DNS traffic management covers 40+ domains and 100M+ requests a day, and when Kafka brokers in one region were overloaded during a major flash sale, it moved half the traffic out of that region within five minutes [29]. Backups are the last line. In 2019 MySQL backups moved to persistent-disk snapshots taken every 15 minutes. A multi-terabyte initial snapshot takes about 20 minutes and an incremental one under 10, more than a petabyte is verified daily, and restore RTO dropped to under 30 minutes from more than 6 hours per shard [24].
What fails
Who it reaches
What Shopify built
Reported number
One hot shop
Its pod’s datastores; the shared app tier
Edge throttle + fair queue; bot blocking; move the shop off a hot pod
Queue waits up to 40 min (2017) [6]
One dependency
Features that call it
Semian breakers + bulkheads, scoped (e.g. payments per country); degrade: logged-out view if session Redis dies
“A degraded service is better than being completely down” [5]
One pod / shard
Shops on that pod only (by design)
Single-pod rule; Pod Mover
Pod to recovery DC “in a minute” [1]
One region
Pods active there + regional shared tier
Whole-pod failover to passive region; DNS traffic management
Kafka overload: half the traffic moved out of the region by DNS within 5 min [29]
Data loss / corruption
A shard’s data
PD-snapshot backups every 15 min
RTO under 30 min, from 6+ h [24]
Global shared pieces
Everyone
Routing table, edge, deploys: capacity planning, scale tests, freezes
Five scale tests up to 200M RPM before BFCM 2025 [19]
The rungs are ours; each mechanism and number is cited. The last rung matters most for the design review. Pods shrink every rung except the global one, and the global rung (routing table, edge fleet, shared compute, the deploy pipeline) is where a pod architecture still has a single failure domain.
Put side by side, these numbers show a system with a recovery mechanism at each blast radius, and the mechanism gets heavier as the radius grows. Circuit breakers act in milliseconds on a single dependency. Semian’s payment breakers are scoped by country, so an upstream outage for Canadian merchants does not open the breaker for American or Mexican ones [5]. A Redis session failure degrades to a logged-out storefront instead of a 500 [5]. A pod moves in about a minute. A region drains in minutes. A shard restores from snapshot in under half an hour. Shopify states the principle directly: “A degraded service is better than being completely down” [5].
Gap in the record
Reconstruction (inferred): three different events hide behind “failover”
pod_id for every shop on the pod, resume. Its RPO is zero and its RTO is dominated by replication lag at the moment of the call, which is why a pod evacuation is cheap at 3 a.m. and expensive mid-sale.Our analysis: the decision you have to pre-commit
The pod boundary also shapes fleet-wide change. The 2021 upgrade to Percona MySQL 5.7.32 took about eight months of preparation and roughly two weeks of production rollout (25 August to early September) [25]. Our arithmetic: “hundreds of shards” [25] in about 14 days is tens of shards a day, each a small, independently reversible rollout, which is what makes a two-week window survivable. The cost of that shape showed up in downgrade testing: a rollback triggered full table rebuilds lasting over a day on some shards, because tables created before 5.7.23 carried a pack-key flag that confused has_index_def_changed; Shopify patched its Percona fork [25].
The general point is worth stating because it inverts a common intuition. Many shards are usually sold as reducing risk per change, and they do. They also guarantee that the fleet contains every historical state the schema and engine have ever been in, so a bug keyed to table creation date, row format or an old default will find a shard to fire on. A rollback that takes a day on the worst shard is not a rollback. The mitigation Shopify used, testing the downgrade path before the upgrade, should be a precondition for any fleet change in a sharded system, and it only works if the test fleet is sampled across table ages, not built fresh.
Pods fixed the runtime. They did nothing for the codebase, which by 2016 was recognised as a problem [8]: “extremely fragile”, highly coupled, with slow CI and hard onboarding [8]. Measured in 2020, before cleanup “every component depended on over half of all the other components” [9]. Shopify’s choice, argued in 2019, was a modular monolith: “a system where all of the code powers a single application and there are strictly enforced boundaries between different domains” [8].
The reasons they give for not moving to microservices are the familiar ones, stated with unusual precision. Microservices mean multiple test and deploy pipelines, per-service infrastructure overhead, network latency, a loss of reliability at every cross-service call, coordinated deploys for refactors, and service boundaries drawn before the domain is understood [8]. They cite Martin Fowler’s warnings against starting with microservices [8]. Two lines from the Packwerk announcement make the point that carries over: “Distributing code over multiple machines is a topology change, not an architectural change”, and “A poorly designed API within the monolith will still be a poorly designed API within a service” [10].
Code: one tangled app
Code: enforced domain boundaries
Data: one DB
2006 – ~2014
One Rails app on bigger and bigger database servers until buying a larger one was “no longer possible” [1].
(Not a stage Shopify went through)
Components on a single database. Fixes coupling but not capacity or blast radius.
Data: isolated per tenant set (pods)
2014 – 2016: shards → pods
Capacity and isolation fixed at runtime; the code is still “extremely fragile” [8].
2017 → today
Pods for data and failure; a componentized monolith for code; services only where the workload shape differs (renderer, card vault, Shop app).
Microservices split both axes along the same seam: domain. Shopify split data along the tenant seam and code along the domain seam, and deploys it all as one unit. This is our framing, but every cell is sourced. It is the main reason “Shopify is a monolith” and “Shopify runs on hundreds of isolated shards” are both true.
Here is the dependency density in numbers. With 37 components [9] each depending on more than half of the other 36, each has at least 19 dependencies, so there are at least 703 directed edges out of 1,332 possible. (The two figures come from the same 2020 post, but it does not say the 37-component count and the “over half” measurement describe the same moment, so treat this as an order-of-magnitude illustration.) If those had been network calls, and each synchronous dependency were independently 99.95% available (an illustrative figure), a component calling 19 others would be available at most 0.9995¹⁹ ≈ 99.05%. That is about 7 hours a month of unavailability from dependencies alone, before any bug of its own. This is the “cross-service reliability loss” [8] in concrete terms. Extracting services from a graph this dense would have been Section 2’s shared-fate problem again, on the code axis instead of the data axis.
The tooling went through three generations (Wedge’s runtime call graphs, Packwerk’s static constant-reference checks, then functional packages), summarised below with sources. Two details matter more than the tools themselves. Packwerk records existing violations in a todo list and fails only on new ones [10], which in 2022 the payments team described as the point: “at least it breaks the build” [5]. And contracts between components moved from Component::Schema to Sorbet types [9], so the boundary became something the type checker, not only a linter, could see.
Wedge
described 2019
What code calls at runtime
Packwerk
2020 – 2023
What code names statically
Functional packages
2024 →
What code needs in order to run
Organisational scale alongside the tools: 2.8M lines of Ruby, 500K commits, 37 components with public entrypoints (about a third on Packwerk) [9]; 48 packages and 30 boundary enforcements in the core monolith [10]; an Architecture Guild of ~400 members [9].
The 2024 retrospective is the most candid document in this set of sources and the least read [11]. Its findings in one line each: privacy checks were removed in Packwerk 3.0 because they turned the tool into an “API design tool” that fought Rails conventions (“privacy is sugar”) [11][12]; static analysis misses dynamic constants, require/autoload and engine routes and initializers [11]; Shopify was “likely the first Packwerk user to completely work through an entire package todo file”, years after release [11]; and Platform Essentials, at zero violations, was still “broken and unusable in isolation” until months of inversion-of-control work and isolated CI runs [11].
Our analysis
The development numbers show what the monolith bought in throughput. In 2018 it ran about 100,000 unit tests in a 15–20 minute CI build on hundreds of parallel workers, and the merge queue shipped 5–10 commits per deploy batch [13]. Later sources put deploys at about 40 a day [5][22]. The figures come from different years, but combined they suggest roughly 200–400 commits a day reaching production through one pipeline. That is the order of magnitude that makes “one deployable” feasible for a thousand engineers, and it depends on heavy investment in test parallelism and merge-queue tooling. The monolith is not free. What it does is replace the microservices tax (network, versioning, distributed transactions) with a build-infrastructure tax, which scales far better.
Shopify publishes BFCM numbers every year, and people compare them carelessly. The units change over time: requests per second in 2018 and 2020, requests per minute from 2021. The layer changes too: app-server requests in some years, edge requests in others. Several years are missing from the primary sources we have. Before drawing conclusions, it helps to put them side by side.
BFCM sales, $B
1.5
5.1
6.3
11.5
14.6
18
19
20
21
22
23
24
25
Hatched: no primary figure in our sources for that year, so none is shown. Sources: [14][16][5][20][21].
Peak request rate (per second, log scale 50K → 6.3M)
Aug 2018 · peak
80K/s · app, “80K RPS” [13]
BFCM 2018
100K/s · ~100K RPS [14]
BFCM 2020
170K/s · spikes “up to 170k” RPS [22]
BFCM 2021
533K/s · 32M RPM · layer unstated [5]
BFCM 2024 app
1.33M/s · 80M+ RPM app servers [20]
2025 test 5
3.33M/s · p99 test at 200M RPM [19]
BFCM 2024 edge
4.73M/s · 284M RPM at the edge [20]
These figures are measured at different layers (edge vs app), as a peak vs a spike, and on test vs event days. Read the bars as orders of magnitude, not as a growth series.
Peak orders grew much less than sales did. The peak-minute figure is what the architecture has to survive, and it can be derived from the published numbers. BFCM 2025 peaked at $5.1M per minute with an average cart of $114.70 [21][26]. That is about 44,500 orders a minute, roughly 740 a second. The caveat is that the weekend-average cart stands in for the peak-minute cart. BFCM 2018 peaked at about 11,000 orders a minute [14]. So the peak-minute order rate grew about 4× from 2018 to 2025, while total BFCM sales grew 9.7× ($1.5B to $14.6B) [14][21]. BFCM 2020 fits the same curve. Its peak hour was $102M+ at an $89.20 average cart [22], about 19,000 orders a minute averaged over that hour, which is a lower bound for the peak minute. The reading is that growth has increasingly come from a wider event (more merchants, more time zones, more hours at high volume), not a sharper spike. That favours an architecture that scales out by adding pods over one that has to absorb a single global surge.
The test headroom is about 1.8×. In the fourth of five 2025 scale tests, Shopify drove 146M requests per minute and more than 80,000 checkouts per minute [19]. Against the derived real peak of about 44,500 orders a minute, that is roughly 1.8× headroom on the metric that matters. The fifth and last test “tested p99, which was 200M RPM” [19]. Tests ran from April to October [19], which continues the “faux BFCM” practice Shopify adopted in 2018, when it decided that a mid-year rehearsal “would increase our resilience as an organization”, with drills such as “Mayday” and “Spooky scale-up” (2019) and “Oktoberfest scale-up” (2020) [15].
The edge absorbs about 70%. BFCM 2024 peaked at 284M requests per minute at the edge and more than 80M per minute at app servers [20], a ratio of about 3.5 to 1. Roughly 70% of edge requests never reach the Rails tier. The sources do not break that down between caching, static assets, rejected bots and queue pages, but as a class this is the admission-control layer from Section 5 at work.
Sensitivity: BFCM 2024 database load per shard (our arithmetic)
Reported: 10.5T database queries and 1.17T writes over the event [20], so writes are 11%. We assume a four-day window (345,600 s), which the newsroom does not state. That gives a fleet mean of about 30M queries/s and 3.4M writes/s. The 2024 shard count is not published. “Hundreds” is the last figure (2021) [25].
| Assumed shards | Writes/s per writer (mean) | Reads/s per instance (6 per shard) | Plausibility |
|---|---|---|---|
| 200 | ~17,000 | ~22,500 | hot for a single MySQL writer |
| 400 | ~8,500 | ~11,200 | comfortable |
| 800 | ~4,200 | ~5,600 | lots of headroom |
These are means. The peak minute is several times higher, and the hottest pod is higher again (Section 6 reports a 4× spread before rebalancing). The newsroom’s “database queries” may also include datastores other than MySQL. The takeaway is directional: the published totals only fit comfortably if the fleet is in the high hundreds of shards or the definition of “query” is broad, and either way the peak lands on individual pods, not on the fleet average.
Scale limits keep moving from the database to the plumbing. The failures found in scale tests are more revealing than the headline numbers. Before BFCM 2022, when the platform was scaled up about 200% from baseline, tests found that ingress-nginx allowed only 1,000 endpoints per cluster against a target of about 1,200 Kubernetes pods per cluster, uneven request distribution that overloaded some nginx pods during regional failover, and memory pressure and timeouts from the growing number of TCP connections as Storefront Renderer and streaming services scaled up [18]. (These “pods” are Kubernetes pods, not Shopify pods, a collision of terms that confuses many readers.) The 2025 tests found Kafka partitions that needed increasing, Analytics API memory and timeout problems, checkout queues backing up, rate-limit paths exposed once authenticated checkout flows were added to the tests, and, in test 4, delays when bringing regions back online during rebalancing [19]. None of these is “MySQL ran out”. Once data is partitioned into pods, the bottlenecks move to shared connection-count and fan-in limits: ingress endpoints, proxies, partitions, rate limiters. That fits the failure ladder in Section 7, where the global rung is the one left.
~44.5K
Derived peak orders/min, 2025
~4×
Peak orders/min growth 2018→25
~1.8×
Test 4 checkouts vs derived peak
~70%
Edge requests not reaching app tier (2024)
Source caveat
Shopify’s posts explain the choices they made far more than the ones they rejected. The matrix below reconstructs the options that were realistically available from 2015 to 2017, scored on the criteria the sources show Shopify cared about. Cells with citations are Shopify’s own reasoning. Everything else is our assessment.
Write scale
Shard outage reach
Hot-tenant isolation
Unit of DR / move
Checkout = one local txn
Cost to adopt / run
Bigger box
pre-2015
capped [1]
n/a: one DB
none
whole DB
yes
cheap until impossible
Shards + fan-out
2014–16
linear
platform-wide actions [1]
shard-wide [6]
shard (data only)
yes
moderate
Pods, shared compute
chosen
linear
one pod’s shops
pod datastores; edge gate for app tier
pod (in a minute) [1]
yes
routing + movers + discipline
Full cells (compute per pod)
not chosen
linear
one cell
full
cell
yes
idle compute per cell; no burst sharing
Transparent sharding (Vitess)
Shop app, 2024
linear
keyspace shard
weak for lumpy tenants [27]
resharding, not tenant moves
if keyed well
proxy tier to run
Domain microservices
rejected for core
per service
depends on call graph
none by default
service
no: distributed txn [5]
pipelines, network, ops [8]
What they gave up. The pods design has real costs, and people who admire it rarely list them. Cross-tenant queries leave the transactional system altogether, going to separate apps or to Presto [5]. Every cross-shop feature is therefore a data-pipeline problem with eventual consistency built in. The routing table becomes a global dependency with its own availability engineering [2]. Tenant moves need custom infrastructure (Ghostferry, the MRSW lock, verifiers) and schema constraints: no foreign keys, numeric auto-increment keys, full row-based binlog [4]. The single-pod rule is a discipline, not a type system. Nothing in Ruby stops a job from opening a second pod’s connection. And the shared app tier stays a co-tenancy domain, so edge admission control is mandatory rather than optional.
Counterfactual 1: microservices in 2016
Counterfactual 2: Vitess (or any transparent sharding) for the core
Counterfactual 3: rewrite checkout instead of throttling at the edge
“Each pod is a full copy of Shopify.”
Only the datastores are isolated. App servers, job workers and load balancers are shared [1][5]. “A complete version of Shopify” is a conceptual framing [5]. The difference decides whether a runaway worker pool is a pod incident or a platform incident.
“Shopify is a monolith, so it has no services.”
In 2018 Shopify ran 100+ other production services [13]. Storefront rendering, card vaulting and Shop Pay run separately [5][9][28]. The core is a modular monolith. The platform is not a single process.
“Pods solved flash sales.”
Pods limit the blast radius to a pod. A hot shop still hurts its pod-mates and the shared tier. Flash sales were handled by edge admission control [6][7] and tenant moves [2].
“Shop moves are zero-downtime.”
Reads continue. Writes pause for under 10 s, or under 20 s for larger shops [5]. The generic Ghostferry figure is seconds to minutes [4]. “Zero-downtime” in the 2021 title is shorthand [2].
“Ghostferry is formally verified.”
It is specified and model-checked in TLA+. The README says the proofs are incomplete and the spec “may not be entirely correct” [3].
“Packwerk solved modularity.”
Shopify’s own retrospective removed privacy checks, found violations piling up faster than teams fixed them, and moved to packages that must run in isolation [11].
“Shopify handles N million requests per second.”
The quoted numbers mix edge and app-server layers, RPS and RPM, and peaks, spikes and tests (Section 9). 284M RPM at the edge and 80M+ RPM at app servers describe the same weekend [20].
One more deserves more than a card: the claim that the Kylie Cosmetics sale in 2016 “took Shopify down”. It did not. It took down her store and every other store on the same shard [6]. That distinction is the whole case for pods. A platform-wide outage would have argued for more capacity. A shard-wide outage argues for isolation plus tenant mobility, and that is what Shopify built.
Pods moved the ceiling. They did not remove it. By 2020 Kir Shatrov reported limits being hit “within a shard”, which led to work on data access patterns and caching [31]. Shopify runs a ProxySQL tier in front of MySQL [30][18], and Shatrov writes, in general terms rather than about a specific Shopify rollout, that he has “seen 20x reductions on DB load after introducing ProxySQL” [32]. The same year Shopify deployed globally, cutting latency in Australia by 200 ms [31]. Our inference: moving compute closer to users while each pod’s writes stay in its active region only pays off because most traffic is reads that can be served near the user.
The clearest post-pods decision is the 2026 migration of inventory reservations from Redis to MySQL [26]. The Redis design used one key per item with DECR/INCR. It was fast, but it sat outside the ledger’s transaction and had no multi-location awareness. The replacement stores one row per sellable unit in a bounded pool of 1,000 rows per item/location, a cap “large enough to absorb bursts without running dry, but small enough to keep the table compact and the SKIP LOCKED scan fast”. Reservations claim rows with SELECT … FOR UPDATE SKIP LOCKED, so concurrent buyers take different rows instead of queueing on one, and the pool is replenished inline under a lock to avoid a thundering herd [26]. Getting there needed a composite primary key (shop_id, inventory_item_id, inventory_group_id, id) to go from two row locks per reservation to one, READ COMMITTED to avoid gap-lock deadlocks, and a consistent statement order for reserve and claim (always DELETE from the units table before INSERT) to prevent deadlocks [26]. The rollout was shadow writes to both stores with Redis as the source of truth, then a switch, then pod by pod from low-traffic to highest-volume [26].
The instructive part is the bottleneck. Throughput hit a ceiling that was neither CPU nor query cost. It was connection exhaustion caused by other checkout code holding connections too long. Tagging every SQL statement with a comment such as /* conn_tag:checkout_completion */ and tracking hold time per tag in ProxySQL found the culprits. Fixing them removed 50% of reads and 33% of transactions on the primary, and at peak the writer stayed under 50% CPU and readers under 16% [26]. In the post’s words: “The real bar was sustaining throughput without degrading database health for everything else” [26].
Our analysis
When to copy it. The conditions follow from the mechanisms above, not from Shopify’s size.
Copy it when
Do not when
Tenant shape
Multi-tenant, and nearly all work is scoped to one tenant (a shop, an account, a workspace)
Consumer social graph or a marketplace where most reads cross tenants
Load distribution
Lumpy: one tenant can briefly be a large share of the platform (flash sales)
Many small users with even load: a keyspace proxy is simpler (Shop app → Vitess [27])
Cross-tenant features
Few, and can live in separate apps or a warehouse
Core product features need cross-tenant joins or transactions
Transactional core
The money path must commit atomically across many tables for one tenant
Your write paths are already eventually consistent across services
Ops maturity
You can build and own a tenant mover (binlog tailing, app-level lock, verifiers)
No one owns data movement; tenants will be stuck where they signed up
Engine constraints
Numeric auto-inc PKs, full row-based binlog, tenant id on every table
Foreign keys everywhere, no tenant column on child tables
The pitfalls in copying it, in the order teams usually hit them:
1. Adopting pods without the single-pod rule. If jobs still iterate across all pods for “just this one report”, you have Section 2’s fan-out coupling back again. Enforce the rule mechanically. Scope connection handling so a unit of work can only get a connection for the pod named in its context, and make a second pod unreachable rather than merely discouraged.
2. Treating the routing table as configuration. It is the one global, per-request dependency you have left. Give it an availability design at least as strong as a pod’s, meaning edge caching and a known behaviour when it is stale. Test that behaviour in game days, as Shopify does with its resiliency matrix [16].
3. Building the mover last. Without tenant mobility, arbitrary placement at signup [2] hardens into a permanent 4× imbalance, and your first celebrity tenant becomes a pod-wide incident you cannot fix quickly. The mover also dictates schema: tenant id on every table, no cross-table foreign keys, a replication format you can tail [4]. Those decisions are cheap in year one and ruinously expensive in year eight.
4. Skipping admission control because “we have isolation now”. The shared compute tier is what makes pods economical, and it is also a co-tenancy domain. Build a stateless, fair, arrival-ordered gate in front of the expensive write path before you need it [7].
5. Measuring modularity by a violation count. Package checkers are valuable leading indicators and misleading final metrics [11]. The finish line is a package whose tests run with nothing else loaded.
6. Confusing Shopify’s scale with the reason for Shopify’s design. The design follows from lumpy, write-heavy, single-tenant transactions. A B2B SaaS with a few hundred large accounts and occasional bulk imports has the same shape at a thousandth of the scale and should consider tenant-scoped datastores early. A consumer app with 50 million even users has a different shape at any scale, and should do what Shopify did for the Shop app [27].
Our analysis: questions to ask in the design review
In summary: choose the partition key your transactions and your failure domains share, make it impossible for work to cross it, keep compute pooled and put a fair gate in front of it, and build the tool that moves tenants before you need it. Keep the code as one deployable for as long as your build infrastructure can carry it, and judge modularity by what can run on its own.
Every factual claim above is traceable to one of these. Bracketed numbers in the text refer to this list. Analysis, reconstructions and counterfactuals are ours and labelled as such.