Slack’s infrastructure story is usually told as two unrelated blog posts. In the first, from 2020, Slack moved its MySQL fleet onto Vitess. In the second, from 2023, Slack moved to “cells”. Read the twenty-odd primary sources together, from Mike Demmer’s 2018 Percona slides to Cooper Bethea’s 2024 QCon talk, and they turn out to be one story told in two halves. The old design had two kinds of isolation that nobody had engineered: every workspace lived on one shard, and every shard was a pair of writable masters. Vitess gave up both to get flexible sharding keys and strong consistency. The cellular program was the bill for that trade. It redrew blast radius on purpose, along availability-zone lines, with Vitess as the one component that could not simply be fenced in. The thesis of this case: Slack scaled by trading isolation it got for free for isolation it had to build. Both programs finally succeeded the same way, once a re-architecture was recast as a mechanical, reversible migration that could be done one shard or one service at a time.
99%
MySQL traffic on Vitess, Dec 2020 [S1]
2.3M
peak QPS, 2 ms median, 11 ms p99 [S1]
4 eng · ~1 yr
moved 1,000+ legacy shards (VIFL) [S10]
60 s
to drain siloed services from an AZ [S3]
Source tags like [S3] refer to the list in Section 12. S14–S16 are podcast summaries by a third party and are flagged wherever they are the only source. Slide decks (S7, S9, S10) are cited from their text. Anything labelled Our analysis, Reconstruction or Counterfactual is ours, not Slack’s.
Slack began as a LAMP application, and every piece of customer data lived in MySQL across three kinds of cluster[S1]. Shards held almost everything (messages, channels, DMs), partitioned by workspace id, with thousands of workspaces per shard. A metadata cluster, called “mains”, mapped each workspace to its shard. A kitchen sink cluster held tables with no workspace id, such as the app directory[S1][S10]. The PHP (later Hack) monolith, webapp, did all the routing. It looked up the workspace in mains and then opened a connection straight to the shard[S1][S10]. Demmer’s 2018 QCon talk adds the operational texture: hosts were “a herd of pets” known by number, and the monolith carried a hand-maintained map from shard number to hostname[S8].
The unusual part was replication. Each shard had at least two MySQL hosts in different datacenters[S1], and each host was both a writable master and an asynchronous, statement-based replica of the other, with no GTIDs. The application preferred one side using team_id % 2 and switched sides on failure. Conflicts were kept rare with upserts and globally unique IDs from a dedicated ID service[S7]. Demmer’s slide describes this with unusual candour: “Yes, this is a bit odd... BUT it yields Availability >> Consistency”[S7].
2016
~4M DAU; workspace-sharded MySQL, master/master pairs; largest orgs ~10,000 users
S7 S8
Fall 2016
“Hundreds of thousands” of QPS, “thousands” of sharded hosts: a new approach is needed
S1
Mid 2017
First Vitess use case in production (RSS feeds into channels) after ~7 months of work
S1 S7 S11
Apr 2018
~10% of queries on Vitess; 8 keyspaces, 316 tablets, 32 vtgates; orgs now 125,000+ users
S7 S8
Nov 2019
~40% on Vitess; Vitess topology split into one cell per AZ, each its own Consul dc
S9
Mar 2020
COVID: query rates +50% in one week; a hot keyspace split with Vitess workflows
S1
Apr 2020
`message` table migrated (channel-sharded): 70% of load on Vitess
S10
May–Sep 2020
VIFL: Vitess in front of 1,000+ legacy shards; +30% of load in under 6 months
S10
Dec 2020
99% of MySQL traffic on Vitess, 2.3M QPS peak; last 1% “in the next two months”
S1
30 Jun 2021
Gray failure: one us-east-1 AZ link flaps twice; ~1% sitewide errors; nobody drains
S2 S3
22 Feb 2022
Consul rollout + Mcrib churn + scatter queries: metastable cache/Vitess cascade
S5
Oct 2022
Bulk user removal floods one shard: OOM → promote → replace loop
S17
Aug 2023
“Over the last 1.5 years” critical user-facing services moved to AZ cells
S2
QCon SF 2024
Siloed services drain in 60 s; Vitess reparents away “as fast as the replication permits”
S3
webapp (PHP → Hack/HHVM)
owns all routing; hand-maintained shard → hostname map [S8]
Globally unique IDs
dedicated ID service [S7]
Many short-lived connections
webapp → MySQL directly, no proxy [S7]
① SELECT db_shard FROM teams WHERE id = ? [S10]
② connect straight to that shard
non-workspace data
mains / metadata
workspace id → shard id; webapp looks this up before connecting to the shard [S1][S7].
team shards · thousands of workspaces each [S1]
Shard 1a (even)
writable master AND replica of b
Shard 1b (odd)
writable master AND replica of a
async · statement-based · no GTIDs · app picks side by team_id % 2, switches on failure · conflicts mitigated by upserts + global IDs [S7]
kitchen sink / aux
tables with no workspace id, e.g. the app directory [S1][S10].
Free properties: a host failure is a retry to the other side; a shard outage touches only its workspaces; debugging maps a customer report to one host “within minutes” [S1].
Hidden limits: one workspace can never exceed one host; no usable replicas; conflicts repaired by hand; Enterprise Grid needs N + 1 shards per org [S1][S7].
It is worth being precise about what this design gave Slack for free, because both programs in this case study are about losing those properties and then paying to get them back. The first was host-failure invisibility. A write that failed on side a could be retried on side b at once, with no failover step, because b already accepted writes. Slack’s 2020 retrospective lists this first: the application “could retry a request to the other host without any visible customer impact”[S1]. The second was tenant-shaped blast radius. When a shard died, only the workspaces on it were affected. That was a full outage for them[S1], but nobody else noticed. The third was debuggability: an engineer could map a customer report to a database host “within minutes”[S1].
The price was paid elsewhere. Operators repaired replication conflicts by hand, split hot shards with custom scripts, and replaced failed hosts manually[S7]. Because both hosts were masters and the application routed to hosts directly, there were no replicas that could safely serve reads without reworking routing[S1]. The fleet was also big and mostly idle. In April 2018 Demmer reported about 1,400 database hosts serving 100,000 to 400,000 QPS “with very high bursts”, about 24 billion queries a day, and average load per host of about 200 QPS, “much lower than capacity to handle spikes”[S7].
Our analysis — the 200 QPS figure is not a typo, it is the design
Slack’s 2020 post frames the problem as a question: “What if a single team and all of their Slack data doesn’t fit our largest shard?”[S1]. By the fall of 2016 the fleet was handling “hundreds of thousands of MySQL queries per second” on “thousands of sharded MySQL hosts”, and application teams were regularly designing around the workspace model’s limits[S1]. Four distinct pressures appear in the sources, and they are worth separating because they don’t all point to the same fix.
Hot spots. Demmer showed a chart of the top 100 databases by daily query time in which one shard was “5 to 10 times as busy as all the rest”, and said this “would just keep happening”: a customer would bulk-provision users or delete a batch of channels, clients would react, and the load would land on one shard[S8]. A shard “can’t scale up past the capabilities of a single database host”[S7]. A data model stuck on workspaces. Enterprise Grid federated many workspaces into one org using “N + 1 shards”, and shared channels had to read across workspace shards[S7]. Code worked around this by “checking multiple shards explicitly”[S1]. Waste, as above. Operations: a non-standard topology that needed a large amount of in-house tooling, with conflict repair and shard splits done by hand[S1][S7].
The secondary sources put a face on the ceiling. According to a 2025 podcast summary, by late 2019 Slack’s largest customer (IBM, about 300,000 employees) lived on a single legacy shard nicknamed “542”, and capacity models said it would start dropping writes in 12 to 14 months[S14]. Treat the specific numbers as unverified. They come from an interview summary, not a Slack post. But the shape matches exactly what the primary sources describe as the fundamental limit.
Our analysis — runway is measured in growth rates, and a step change ignores them
Our analysis — the trigger was the shape of the distribution, not its size
The fix had to deliver something specific. The canonical example in Slack’s post is messages: instead of storing all of a workspace’s messages on its shard, shard them by channel_id, “so we would no longer be forced to serve all message data for our largest customer on the same database shard”[S1]. The requirement was not “more capacity” but a different key per table. That requirement ruled out most of the easy answers.
The team framed the choice as “should we evolve our approach in place or replace it?”[S1]. Three constraints narrowed the options before any technology was compared. There were thousands of distinct queries in the application, some of them MySQL-specific. There were years of MySQL-specific operations (deploys, durability, backups, warehouse ETL, compliance)[S1]. And the culture was “generally conservative” about new technology on the critical path, with a small datastores team that “valued simple solutions with few moving parts”[S1].
Keep MySQL + tooling
Flexible shard key
Fixes ops (failover, replicas)
Migration cost
Verdict
Shard-by-X inside webapp
yes
yes
no — “didn't address any of our operational issues” [S1]
lowest: “no new components” [S7]
prototyped, rejected: app/storage coupling [S1]
NoSQL (DynamoDB, Cassandra)
no
yes
partly
“major change to app” [S7]
“pretty much ruled out” [S1]
NewSQL (Spanner, CockroachDB)
no
yes
yes
“least well known” [S7]
ruled out as too disruptive [S1]
Vitess
yes, MySQL at the core
yes, per-table vindexes
yes, topology + failover
new moving parts; “not magic” [S7]
chosen; proven at YouTube [S1][S7]
The rejected option that deserves the most attention is the one Slack actually prototyped: building “shard by X” into webapp. Demmer’s slide lists its advantages honestly (“no new components”, “easiest migration”)[S7], and it is what many teams do. Slack rejected it for three reasons. Application logic and storage layout were already entangled; the post’s example is counting messages in a channel, which depended on assumptions about the channel’s workspace. It wouldn’t fix the operational problems or make read replicas usable. And it would hit the same limits again, for instance a single team’s shard getting “surprisingly hot on the write path”[S1].
Vitess won because it kept MySQL as the storage and replication engine, offered per-table sharding through vindexes, took over topology, failover and backups, was open-source Go with an active community, and had been proven at YouTube[S1][S7][S8]. Demmer’s summary of what came with it is the most quoted line in the deck: “Vitess is magical but not magic”[S7]. The same slide lists what was missing in 2018: no way yet to change a sharding key, some unsupported queries, and the fact that “scalability / efficiency requires stale reads from replica”[S7].
Our analysis — the real decision was to move routing out of the monolith
A second decision was made at the same time and is easy to miss: which consistency model to adopt. Vitess meant moving from master/master to a single primary per shard with row-based, semi-synchronous replication, GTIDs, and Orchestrator for failover “in seconds”[S7][S8]. Demmer’s “Change All The Things” slide lists what else changed in the same move: MySQL 5.7, new i3 instances, a new Ubuntu release, HHVM’s async MySQL driver, and reads from replicas[S7]. Its “Networking Matters” slide states the trade outright: Vitess “is intrinsically more network dependent”, and improved consistency “comes at the expense of availability and performance”[S7]. That sentence, written in April 2018, is the seed of the 2021 cellular project.
The third decision was the migration approach. In 2018 the slide offered two options: migrate tables one by one (✅), or run Vitess in front of the existing databases (🚫). The second was rejected because table-by-table was “the only approach that enables resharding (for now)”, because the legacy tier ran master/master, and because fronting it “doesn’t help to avoid hot spots”[S7]. Remember that 🚫. Two years later the same team reversed it and moved about 30% of Slack’s query load in under six months. The same deck also shows, in its closing slides, a “Legacy Databases (Future)” diagram with a vtgate and a vtqueryserver in front of a legacy shard[S7]. The idea was already on the whiteboard in 2018. What it lacked was the right moment.
Counterfactual — had Slack bought NewSQL in 2016
No single source draws Slack’s Vitess deployment end to end. The picture below is assembled from seven of them. Applications speak the MySQL protocol to vtgate, which uses each table’s vindex to route to one shard, or scatters to all shards when the key isn’t in the predicate[S7][S8]. Tables are grouped into keyspaces (logical databases), and different keyspaces are sharded by different keys: channel membership exists both sharded by user and sharded by channel[S5], and messages are sharded by channel id[S1][S18]. Each shard has one primary tablet and several replicas, and a replica whose lag exceeds one hour is taken out of service[S17]. Everything runs on EC2 with autoscaling groups for the stateless parts, Percona MySQL 5.7, and ephemeral NVMe with no EBS[S9].
AZ 1 · Vitess cell
webapp (Hack)
HHVM async MySQL driver [S7]
↓ MySQL protocol · AZ-aware [S7]
vtgate
routes by vindex; scatter if no key
↓ gRPC
vttablet · replica
reads; pulled if lag > 1 h [S17]
local topo: Consul dc for this AZ [S9]
AZ 2 · Vitess cell
webapp (Hack)
HHVM async MySQL driver [S7]
↓ MySQL protocol · AZ-aware [S7]
vtgate
routes by vindex; scatter if no key
↓ gRPC
vttablet · PRIMARY
all writes for this shard
local topo: Consul dc for this AZ [S9]
AZ 3 · Vitess cell
webapp (Hack)
HHVM async MySQL driver [S7]
↓ MySQL protocol · AZ-aware [S7]
vtgate
routes by vindex; scatter if no key
↓ gRPC
vttablet · replica
reads; pulled if lag > 1 h [S17]
local topo: Consul dc for this AZ [S9]
Writes from AZ a and AZ c cross the AZ boundary to the shard primary; the primary waits for a semi-sync ack from another host before it acknowledges a commit [S7][S9][S24]. Reads can stay cell-local.
Keyspace: channel membership by user
cheap: “what am I in?” [S5]
Keyspace: membership by channel
cheap: “who is in this?” [S5]
messages by channel_id
largest customer no longer on one shard [S1][S18]
Control + durability
Global topology
its own Consul dc, separate from the per-AZ ones [S9]
Orchestrator
primary failover “in seconds” [S7]
MySQL settings [S9]
rpl_semi_sync_master_timeout
= 9999999999999
rpl_semi_sync_master_wait_no_slave = 1
sync_binlog = OFF
innodb_flush_log_at_trx_commit = 2
Row-based replication with GTIDs; MySQL 5.7 on i3 instances with ephemeral NVMe, no EBS; EC2 + ASGs, not Kubernetes [S7][S9].
The deployment went through three routing generations in its first year: vtgates behind an ELB, then client-side load balancing, then AZ-aware routing with webapp and vtgate in each of us-east-1a/b/d/e[S7]. Even the third was “improved… but still not great”. Short-lived connections had to be opened and closed rapidly, and to mitigate packet loss the application quickly failed over to another vtgate or shard, which “under load… causes delays, brownouts”[S7]. Remember this detail too. It is the 2018 version of the failure mode that triggered cells in 2021: a client-side retry policy against a network that is only partly broken.
The durability model inverted the old one. At KubeCon 2019 Slack showed its MySQL settings[S9]:
rpl_semi_sync_master_timeout = 9999999999999 rpl_semi_sync_master_wait_no_slave = 1 sync_binlog = OFF innodb_flush_log_at_trx_commit = 2
Read together, these say: never fall back to asynchronous replication, keep waiting even if no semi-sync replica is connected, and don’t fsync locally on every commit. Durability comes from another host having received the transaction, not from this host’s disk. That fits ephemeral NVMe, where a lost instance takes its disk with it anyway[S9].
Our analysis — the timeout says “availability is not our problem any more”
Reconstruction (inferred) — how a flapping AZ link reaches a user through semi-sync
The most under-appreciated element in the picture is the 2019 change to the Vitess topology. Slack’s first deployment was “a single cell across multiple AZ’s”, with the global and local topology in the same Consul cluster. It was rebuilt as “isolated topologies (one dc for each AZ and one for the global topo)” so that “blast radius is mapped to physical infrastructure”. The slide’s proof: an AZ failure during backup time affected only a single cell[S9]. In Vitess’s own terms a cell is a datacenter or AZ with its own local topology service; reads are served locally and writes go to the shard primary wherever it is[S24]. So the datastore team had already cellularised along AZ lines two years before the rest of Slack, and had documented the result. The 2021 project generalised a pattern the database tier had already adopted.
Two figures from the end of the migration fix the steady state: 2.3M QPS at peak (2M reads, 300K writes), with a median query latency of 2 ms and a p99 of 11 ms[S1]. Vitess also carries International Data Residency, with clusters in six regions, which Slack credits with shipping that product “in record time”[S1]. Some of the upstream work Slack contributed was about isolation itself. The first item on its list is a refactor of the topology metadata service “for scalability across isolation regions”[S1].
The first Vitess use case took about seven months “from whiteboard to production”[S11][S7]: a small feature that integrates an RSS feed into a channel[S1]. Reconciling S1 (“2017”), S10 (July 2017) and S7 (ten months in production by April 2018, after seven months of effort) puts evaluation in late 2016 and first production in mid 2017[S1][S7][S10]. By April 2018 Vitess carried about 10% of all queries, around 30,000 QPS at peak with spikes above 50,000, across 8 keyspaces, 316 tablets and 32 vtgates, and every new feature was being built on it[S7].
For the next two and a half years the method was the per-table state machine below. For each table the team analysed its queries, picked a keyspace and sharding key, double-wrote and backfilled, then switched reads[S7] (the same Backfill/Dark/Light/Sunset modes reappear in Refactoring at Scale’s channels_members chapter[S13]). The piece that did the real work was a double-read diffing system[S1]. Note where the one-way door sits: only at SUNSET, after users have already been served from Vitess in LIGHT. Every risk before that is paid in engineer time, not in customer exposure, which is precisely why the method was slow.
Stage
Writes go to
Reads from
App sees
What the stage buys
PASSTHROUGH
legacy
legacy
legacy
convert call sites to the new access layer
BACKFILL
legacy + Vitess
legacy
legacy
bulk copy history while double-writing
DARK
legacy + Vitess
both, diffed
legacy
diff engine finds semantic gaps; zero user exposure
LIGHT
legacy + Vitess
both, diffed
Vitess
users now see Vitess; legacy still a full rollback
SUNSET
Vitess
Vitess
Vitess
stop legacy writes; the one-way door
Every stage before SUNSET is reversible by configuration. The expensive part is not the machine; it is that each table also needed a new sharding key, new queries for every joined table, and often a new data model [S7] — the reason S10 later calls it “a re-architecture”.
The method was correct and it was slow. At QCon in 2018 Demmer said the migration had been “slower than I would’ve liked”[S8]. Slack’s 2020 post on how large technical changes happen reports “hundreds of tables totaling more than 50% of our query workload” done by February 2020, with the hard tables and a long tail still left[S12]. The KubeCon 2019 slide set a target of 70% by the end of 2019 from about 40%[S9]. The target was missed by about four months. The message table, sharded by channel, landed in April 2020 and took Vitess to 70%[S10].
The messages migration is the one the secondary sources tell as a race. In a 2025 podcast summary, the engineer who led it describes a ten-month project with four engineers, started because shard 542 was on course to drop writes, with 45,000 diffs surfaced during double-writes and daily replication lag on 542 through the COVID surge until the largest customer was flipped to Vitess early[S14]. The primary record agrees on the outline: March 2020 raised query rates 50% in a single week, and one of the busiest keyspaces was split horizontally with Vitess’s splitting workflows while it happened[S1]. The episode shows what the separation of routing from the application bought. A key change and a live split were operations on a proxy tier, not application deploys.
Then the method changed. The KubeCon 2020 deck states the arithmetic that forced it. About 70% of the workload had taken about 2.5 years and about 10 tables. The remaining 30% was spread over more than 200 tables, and at “1 month / 1 table / 1 engineer” that was 16.5 years[S10]. The deck’s verdict on the old approach is the most useful sentence in the whole record: “It was not really a migration, it was a re-architecture”[S10].
The new method, codenamed VIFL (Vitess in front of legacy), is the one Slack had marked 🚫 in 2018: seed a Vitess shard per legacy shard from backup, keep it in sync with VReplication’s “vtshovel” workflow, validate, and move traffic behind vtgate (steps in the diagram below)[S10]. The automation was built from January to April 2020, migrations began in May, and 99% of VIFL migrations were done by September[S10]. Four engineers took about a year, including design. The deck estimates the legacy path would have needed more than 70 engineers for the same result in a similar time[S10]. The outcome: “hundreds of terabytes”, “over 1000 MySQL shards”, and “zero downtime, not a single outage”[S10].
Consistent cut across two live DBs
Both snapshots describe the same logical point α. The lock is held only long enough to open a consistent read, so writes stall for milliseconds, not for the length of the copy.
Expected cost: +1 RTT per read (vtgate hop), +2 RTT per write (hop + semi-sync ack).
Two details transfer beyond Slack. First, the validation diffs two live databases at one logical instant: a millisecond table lock on the source pins a binlog position, and VReplication is stopped at exactly that position on the target before it is read[S10]. That turns “are these the same?” from a moving-target comparison into a snapshot comparison, and more than two months of application-level query diffs then covered the semantic gaps a row diff cannot see[S10]. Second, the guardrails encode the failure modes of running 1,000 migrations with a small team: refuse to run if other automation holds the shard, and refuse if the shard being cut over is not the one that was validated (the slide’s example: migrating 0123 after validating 4567)[S10]. At that volume, the likeliest outage is not a Vitess bug but an operator pointing the right tool at the wrong shard.
Our analysis — why the 🚫 became the winning move
Our analysis — the two staffing figures don’t agree, and that is informative
VIFL had costs, and the deck lists them. Slack expected one extra round trip per read (the vtgate hop) and two per write (the hop plus the semi-sync acknowledgement)[S10]. On a few of the busiest shards it also hit things it had not expected: a MySQL 5.7 optimizer regression, noticeable vttablet overhead, Go garbage-collection overhead, and latency regressions at p99 and p999[S10]. Application diffs came from 5.6-to-5.7 changes in row order for queries with no ORDER BY, and from code that assumed read-after-write[S10]. The last one is the familiar cost of adding replicas under a system that never had any. In December 2020 Slack reported 99% of MySQL traffic on Vitess and expected the last 1% “in the next two months”[S1]. No source in our set confirms the date the last 1% moved.
Slack published an unusual number of figures over five years, but in different units, windows and venues. Put together, they tell a story that none of the posts states.
Apr 2018
Nov 2019
Nov–Dec 2020
Queries / day
~24B, legacy [S7]
53B+ total [S9]
65B+ total [S10]
Peak QPS
100K–400K legacy; ~30K Vitess [S7]
~500K on Vitess [S11]
2.3M, 2M reads + 300K writes [S1]
Share on Vitess
~10% [S7]
~40–45% [S9][S11]
99% [S1]
Fleet / storage
~1,400 DB hosts [S7]
7.5+ PB provisioned [S9]
9+ PB; “thousands” of servers [S10]
Users
8M+ DAU (2018) [S8]
—
12M DAU [S10]
Our analysis — queries grew far faster than users
Our analysis — the peak-to-average ratio and what 2.3M QPS means
The read/write mix moved too. In April 2018, Vitess traffic was about 80/20 reads to writes, but about 75% of queries went to primaries[S7]. By December 2020 the split was 2M reads to 300K writes, about 87/13[S1]. Slack said early on that efficiency “requires stale reads from replica”[S7]. The same deck lists replica reads among the things changed at once, and VIFL later found code that assumed read-after-write[S10]. The practical point is that on a single-primary system the write path is the scarce one, since every write costs a cross-AZ semi-sync acknowledgement. The more reads that can safely leave the primary, the more headroom the primary keeps. That is also why the 2-22-22 fix of reading immutable data from replicas mattered (Section 7).
Our analysis — the proxy tax, in proportion
Demmer named the cost in 2018, before most of the migration had happened. With fine-grained sharding, one user depends on many shards and servers, and failures became “more widespread”[S8]. His mitigations were to accept partial results and to bound the time spent waiting on slow shards[S8]. The arithmetic behind that sentence is simple and unforgiving.
Assumes the k shards are chosen uniformly and independently. S = 100 and the k axis are illustrative, not Slack figures. The curve is why Demmer's 2018 mitigation was to accept partial results and bound the wait per shard [S8]: once almost everyone touches a failed shard, the only way to keep the product usable is to degrade rather than fail.
Under workspace sharding, a dead shard is a total outage for a known, small set of customers, and invisible to everyone else[S1]. Under per-table keys, the same failure is a partial degradation for most users, because most sessions touch it. Neither is strictly better. Slack clearly preferred the second for its enterprise customers, whose whole company was one workspace. But the second changes what “an incident” looks like. It is less often an outage and more often a site-wide rise in error rate. That is exactly what 30 June 2021 looked like.
The clearest example of the new failure shape is 22 February 2022[S5]. A Consul agent upgrade was rolled out in 25% steps across the fleet. On the third step, restarting agents deregistered memcached nodes from the catalog. Mcrib, the control loop that manages the memcached fleet, did its job well: it quickly swapped in spare nodes. The spares were empty, so the cache hit rate fell[S5]. The query that populated group-DM (GDM) membership, which clients load when they boot, ran against a keyspace sharded by user. Its cache misses therefore became scatter queries to every shard. Load rose “superlinearly” with the miss rate. Cache fills timed out, so the cache couldn’t refill, and client retries kept the load up. It was a metastable failure: the trigger ended, but the loop kept itself alive[S5].
P(scatter fill succeeds) = (1 − q)^S
q = chance any one shard query times out. Shard counts are illustrative; S5 does not give the keyspace's size. A single-shard fill (the by-channel keyspace) succeeds with probability 1 − q.
The fixes read like a list of places where the new architecture had been leaning on the cache without anyone deciding it should. Throttle client boot. Read only the keys that are missing. Let this query read from replicas as well as primaries, since GDM membership is immutable. Move it to the keyspace sharded by channel. Audit other cache-fronted scatter queries. Change how the Consul rollout and Mcrib interact[S5]. Slack’s own summary of the automation’s role is one of the more honest sentences in any post-mortem: “Mcrib’s efficiency increased the churn in the cache tier and contributed to the severity of the incident”[S5].
Our analysis — a sharding key is also a statement about which cache misses you can afford
The October 2022 incident showed the old hot-spot problem coming back under a new key[S17]. A customer bulk-removed users. The job that processes removed users flooded one shard that held 6% of the users’ subscription data. The primary was OOM-killed and a replica promoted. The replacement replica couldn’t catch up, so automation deprovisioned it, and the loop repeated[S17]. Mitigation included temporarily disabling Thread View, then adopting the Vitess tablet throttler and circuit breakers, with Slack contributing the throttler changes upstream[S17].
Reconstruction (inferred) — two automations with no shared model of capacity
The 6% figure is worth a second look. If the keyspace were evenly balanced, one shard holding 6% would imply about 16 or 17 shards. Either the keyspace is small or the data is skewed. The sources don’t say which. Either way, one customer’s bulk operation concentrated on one shard, which is the problem the migration set out to fix, now appearing at a different key. Re-sharding moves hot spots; it doesn’t abolish them. What Vitess changed is that a hot spot can be split without touching application code, which it could not have been in 2016.
Two earlier incidents shaped the infrastructure the cellular project would later use, but neither was its trigger. On 12 May 2020, database load from a config change in the morning scaled webapp up 75%, past the number of backend slots configured in HAProxy. A bug in the program that synced HAProxy state (it tried to fill new slots before freeing old ones) left backend lists stale, and 503s began at 4:45pm PT[S6]. On 4 January 2021, the first working day after the holidays, an AWS Transit Gateway failed to scale for the cold-cache surge and dropped packets. Web-tier threads blocked on backends, autoscaling scaled down on low CPU and then tried to add 1,200 instances, and the provisioning service ran into the Linux open-files limit and an AWS quota[S4]. Recovery came partly through Envoy’s panic mode, retries and circuit breaking, and then AWS manually raising Transit Gateway capacity[S4]. In March 2021 Slack described moving its websocket traffic from HAProxy to Envoy[S20]. That HAProxy-to-Envoy/xDS migration is what later made AZ drains cheap[S2].
The trigger was 30 June 2021. A network link between one us-east-1 AZ and the others had intermittent faults from 11:45am to 12:33pm PDT, and again from 5:22pm to 5:31pm[S2]. The status page lists only the first window, with an end time of 12:36pm[S23]. Bethea describes about 1% sitewide errors[S3]. The diagnosis was easy: TCP retransmits, grouped by AZ, showed one AZ equal to the other two combined[S3]. “Nearly every graph we had aggregated by target AZ looked similar”[S2]. The responders talked about draining the AZ and didn’t. Bethea gives the reasons: “It’s really scary to do things for the first time in an incident”, they feared overloading the other two AZs, and AWS fixed the link while they were still talking[S3]. Then it came back at 5:22pm[S2][S3].
The post-mortem question Slack asked was the right one: “why, in fact, this outage was visible to our users at all”[S2]. The answer runs through fan-out. A single API request “may fan out into hundreds of RPCs”[S2]. Every one must succeed for the response to be correct. And the frontends, which continuously try to detect and exclude failed backends, must record some failures before they can exclude one[S2].
Assumes independent failures and no retries. The point is the gap: a user-visible error rate that breaks a four-nines SLO can come from a per-backend error rate around one in ten thousand, far below what any outlier detector will act on without also ejecting healthy hosts on noise.
Our analysis — why automatic exclusion can’t win this one
Slack’s conclusion was not to build a better detector. It was to accept that humans could see the failure in seconds on a graph, and to give them a lever: “If we had a button that told all our systems ‘This AZ is bad; avoid it.’ we would absolutely have smashed it!”[S2]. The explicit decision was to rely on “the power of human judgment” rather than try to solve automatic remediation of gray failures[S2]. After the incident Bethea wrote the proposal whose title became the project’s thesis: we should be able to drain an AZ[S3][S16]. In the talk’s words, the project “was born from fear, anger, lost revenue, and a bad graph”[S3].
Our analysis — the SLA arithmetic that set the design goal
Slack worked backwards from one operation. Bethea’s definition is short: “A cell is a set of services in an AZ. That set may be drained or removed from service as a unit”[S3]. Four goals followed[S2][S3]. Remove most traffic from an AZ within five minutes. Cause no user-visible errors, so that a drain can be used as a generic, experimental mitigation. Make drains and undrains incremental down to 1%, so an operator can send a trickle back to test recovery. And make the drain independent of the AZ being drained.
The last goal rules out most obvious designs. The earliest attempts used SSH to make the reverse proxies in the bad AZ lame-duck themselves[S3][S16]. That needs the sick AZ to be reachable, which is the one thing you can’t count on “at the end of a bad network link in a real big incident”, as Bethea put it[S3]. It also ran into Envoy’s panic routing. When roughly a third or more of an upstream’s hosts look unhealthy, Envoy stops trusting health checks and spreads traffic across all hosts, which sent traffic back across AZs[S16]. Slack already knew this protection well: it had helped recovery on 4 January[S4].
v0 · lame-duck via health checks
SSH into reverse proxies in the AZ
needs the sick AZ to be reachable (violates goal 4) [S2]
Fail their health checks
Envoy sees ~⅓ of backends unhealthy
Envoy panic routing
ignores health, sprays across all backends, all AZs [S16]
Health is an observation. Envoy is designed to distrust a mass observation of unhealthiness, the same protection that helped recovery on 4 Jan 2021 [S4]. A drain expressed as fake unhealthiness trips it.
v1 · weighted clusters + RTDS
Operator: drain AZ 2
one signal, not per-host actions
Rotor (in-house xDS control plane)
must not depend on the drained AZ (goal 4) [S2]
Edge Envoys in other regions
RTDS runtime weights; seconds to propagate; 1% steps [S2]
normal
drained
Weight is intent. In-flight requests finish; new ones go elsewhere [S2]. Health checking keeps doing its job underneath.
The mechanism Slack settled on is almost anticlimactic. The edge load balancers run Envoy, configured by Rotor, Slack’s in-house xDS control plane. Each AZ is a weighted cluster, with weights set at runtime through Envoy’s runtime discovery service (RTDS). By default each AZ gets 33%; a drained AZ gets 0%[S2][S3]. Weight changes propagate “on the order of seconds” with 1% granularity, and in-flight requests finish while new ones go elsewhere[S2]. Bethea’s summary: “It just works”[S3].
Our analysis — health is an observation, weight is an intention
A drain at the edge only works if traffic that enters an AZ stays there. This is the hard part, and it is where the sources spend most of their words. Slack calls a service siloed if it receives traffic only from its own AZ and sends traffic upstream only within that AZ, so that it behaves as “N virtual services, one per AZ”[S2]. Before the project, most backends were reached across AZs. Bethea: “We were counting on failover happening”[S3].
Before, a webapp in any AZ called backends in every AZ, so one AZ's bad link showed up as errors in all three. After, each service is “N virtual services, one per AZ” [S2]; removing user traffic from an AZ quiesces everything behind it. Vitess is the deliberate exception: the primary for a shard lives in exactly one AZ, so draining that AZ requires a reparent, not just less traffic [S3].
Slack rejected the alternative of putting drain awareness into every RPC client, and gave concrete reasons. User-path services were written in Hack, Go, Java and C++, so the logic would need four implementations. Discovery happened through Envoy xDS, the Consul API and DNS, and DNS has no AZ concept. And Vitess would have needed a fork or an upstream change[S2]. Siloing puts the only traffic-shifting logic in one place, the edge, and makes every other service’s job the simpler one of staying AZ-local[S2].
Not everything can be siloed, and the talk is clearest on how Slack decided what to do with each service. The organising question is where a service sits between stateless and strongly consistent[S3].
Stateless services, above all the Hack webapp, were siloed. Vitess was not. A shard has one primary in one AZ, so draining that AZ means reparenting primaries elsewhere. Slack does that shard by shard, in parallel, with its existing autohealing automation, and in Bethea’s words the reparent happens “as fast as the replication permits”[S3]. Memcache was the awkward middle case. Slack found many places where code had been using it “as a strongly consistent datastore”, because reading back from the node you just wrote to behaves that way[S3]. The fix was per-cell Memcache clusters plus a global Memcache layer, with call sites moved from the old system to the new one at a time[S3]. Consul had been deployed as many functionally sharded clusters, each spanning all AZs, and a Consul cluster “tends to fail as a unit and totally”[S3]. They were collapsed into high- and low-priority clusters per AZ, with the global xDS control plane assembling discovery data above them as an eventually consistent layer[S3]. Bethea’s advice from this is general: if you have to run global services, make them eventually consistent where you can[S3].
Reconstruction (inferred) — what a drain does to Vitess write paths
Our analysis — a drain narrows the durability quorum while it runs
rpl_semi_sync_master_wait_no_slave = 1 and an effectively infinite timeout[S9], a primary stalls rather than commit without an ack. In 2018 Slack ran three replicas per shard[S7]. Suppose, illustratively, one replica per AZ. Draining AZ 2 moves its primaries to AZ 1 or AZ 3, and every shard’s AZ 2 replica now sits behind the bad link. For the length of the drain, each shard’s write availability rests on one remaining ack path, the replica in the one surviving AZ that doesn’t hold the primary. A second, unrelated fault in that path (a slow host, a backup, an xtrabackup stream) stops that shard’s writes; it doesn’t degrade them. Three consequences for anyone copying this. Place replicas so each surviving AZ pair still has two ack candidates, i.e. more than one replica per AZ for hot shards. Pause scheduled replica-heavy work (backups, resharding, schema changes) for the duration of any drain, the same guardrail idea VIFL used against competing automation[S10]. And alert on “shards with one semi-sync candidate”, which is a steady-state healthy-looking condition that only a drain exposes.Draining also changed how Slack deploys. Deploys were reworked to fill one AZ first before spreading to the others, so a bad deploy can be reverted by draining that AZ[S3]. The same drain service used by the edge was opened to internal service-mesh users, who can drain an AZ, deploy into it, and undrain at 1%, 10%, 20% and upward[S3]. Siloing also contains “poisonous” deploys, where a service is itself fine but causes another service to crash: the damage stays in one AZ[S3].
Slack had tried cells before. An earlier effort to build empty, isolated cells with hard network boundaries and repartitioned data, then cut over, “never really got off the ground”[S3]. Bethea’s diagnosis is that it wasn’t incremental. It needed two environments and double cost, delivered no value until the cutover, and required every team to move together[S3]. The second attempt worked from the bottom up, one service at a time, with a one-page plan per service[S3]. It borrowed the idea of “roofshots” from Alex Komoroske’s “slime mold” deck: instead of a direct shot at an ambitious goal, take a series of steps that each lock in value[S3]. Services were prioritised by criticality (does chat.postMessage touch it?) against difficulty, and cellular compliance went into each team’s roadmap and OKRs[S3][S16].
The measurement was the drain itself. For a while, every Friday around noon Pacific, the team drained an AZ as far as it could and watched how much bandwidth left it: 10%, then 20%, “then keep pulling traffic until we got scared”[S3]. Bethea: “I think we only broke stuff one time”[S3]. By April 2022, Slack’s continuous load-testing clients were being used to watch how systems and monitoring responded to removing traffic from a single AZ, “without putting any customer traffic at risk”[S25]. The weekly drains found services that didn’t drain, Vitess needing zonal failover orchestration, Memcache’s global-state assumptions, and old discovery logic that crossed AZs[S16].
Our analysis — the drain percentage is a rare thing: a progress metric that is also a capability
Slack's default weights (33% each [S3]) imply N = 3, so the survivors take 1.5x. To stay under an 80% ceiling at the moment you drain, which is usually peak because that is when gray failures hurt, each AZ must run at or below ~53% at peak. N = 4 relaxes that to 60%, N = 6 to ~67%. Every point of that gap is capacity you pay for daily to use a few times a year. Slack published no headroom figure; the ceiling and N = 4 and 6 are illustrative.
Our analysis — the 1.5x is a floor, not the bill
Bethea is candid about where risk lives in a project like this: “it’s the messy middle where all the complexity and risk is lying”[S3]. He is equally direct about the debt it leaves behind: “tech debt is the price of progress. It is not some engineering sin”[S3]. The program also knew where to stop. Data warehousing used by analysts was left non-cellular[S3]. “We didn’t get all the critical services there 100%, but they’ve all got their roadmaps”[S3]. The side benefits Slack names are modest and specific: a Consul cleanup it had been putting off, “a little bit” of cross-AZ data-transfer savings, AZ-by-AZ deploys with rollback by drain, and containment of poisonous deploys[S3]. The cellular post’s own summary was that critical user-facing services had moved to cells over “the last 1.5 years”[S2].
Our analysis — the two programs succeeded for the same reason
Seven common readings of this story don’t survive the primary sources.
“It’s just load-balancer draining.” The top critique in the Hacker News thread on the cellular post was that draining is a decades-old load-balancer feature[S21]. That is true of the drain, and Slack said so first: the mechanism is two stock Envoy features[S2], and “we didn’t even invent an algorithm”[S3]. The replies in the same thread got it right: the hard part is keeping every internal RPC inside the AZ and handling stateful systems[S21]. The multi-year work was siloing, Vitess failover, Memcache and Consul[S3][S16]. A drain on an unsiloed stack moves user traffic and leaves the backend mesh exactly as exposed as before.
“Slack adopted a cell-based architecture.” Three different things share the word. Vitess cells are per-AZ topology domains, which Slack set up by November 2019[S9][S24]. Slack’s cells are services grouped by AZ in one region[S3]. Neither is the pattern often called “cell-based architecture”, in which each cell is an independent full stack serving a subset of customers. Slack’s cells share one Vitess with cross-AZ writes[S24][S3]. They isolate against zonal infrastructure failures, not against bad tenants or bad global deploys.
“The January 2021 outage triggered cells.” It didn’t. Both primary accounts name 30 June 2021[S2][S3]. The January outage was a regional Transit Gateway capacity failure that affected every AZ[S4]; draining one AZ would have done nothing for it.
“June 30 was a Transit Gateway misconfiguration.” Only a secondary podcast summary says so[S16]. Slack’s post says a cloud provider’s network link between one AZ and others[S2], and the status page says “a network error in part of our infrastructure”[S23]. Don’t merge it with January.
“Vitess solved Slack’s scaling and was done by 2020.” It solved the tenant-size ceiling. It also removed natural isolation, and the 2022 incidents are its bill[S8][S5][S17]. On timing, the December 2020 figure was 99%, with the last 1% expected within two months[S1]. We found no primary confirmation of the completion date. And the migration took three to four years depending on where you start counting, not the one year of VIFL.
“COVID tripled traffic.” That headline comes from a podcast summary[S14]. The primary figure is database query rates up 50% in one week[S1]. They measure different things over different windows; use the primary one.
“VIFL shows that you should always front the legacy system.” Slack rejected fronting in 2018 for good reasons, and those reasons still held in 2020 for the tables that had them[S7]. VIFL worked because the re-keying had already been done for the head (Section 5).
What happened after. The re-keying kept paying out. In 2024 Slack described Unified Grid, which lets users of large Enterprise Grid orgs work across all their workspaces in one view. Its authors credit the Vitess migration: data sharded along axes other than workspace meant the workspace was “no longer required to route queries” for the most important tables. Because messages are sharded by channel, fetching a channel’s messages needed no significant change[S18]. That is the flexible sharding model S1 wanted in 2016, doing product work eight years later. Upstream, Slack’s contributions included refactoring the topology service for scaling across isolation regions, migration tooling, and later the tablet-throttler work[S1][S17]. On cells, Bethea said in 2024 that siloed services drain in 60 seconds and that Slack does drain in practice, for single-AZ cloud problems and for deploys[S3]. The cellular post promised a second part on siloing internal services and on services that can’t be siloed[S2]. As of September 2026 the Slack Engineering networking tag lists no such follow-up[S28]. Slack has published no numbers for the program’s team size, services converted, drainable bandwidth over time or per-AZ headroom cost.
Counterfactual — cells without Vitess, or Vitess without cells
One more correction concerns scope rather than fact. Slack’s engineering blog documents five major incidents across this period. Lay them against the lever each one needed, and the drain button turns out to address exactly one class of incident: a zonal, gray, infrastructure fault. That is the class that fan-out makes invisible to automation (Section 8), so the investment is justified. But it is not a general resilience program, and it shouldn’t be sold internally as one.
When
What failed
Blast shape
Would an AZ drain help?
What actually worked
12 May 2020
HAProxy slot-sync bug after 75% webapp scale-up [S6]
ingress tier, all AZs
no
an ingress fix, not a zonal one; the HAProxy → Envoy/xDS move followed [S20][S2]
4 Jan 2021
Transit Gateway failed to scale; autoscaling + provisioning thrash [S4]
regional network
no
panic mode, retries, breakers; AWS raised TGW capacity [S4]
30 Jun 2021
one AZ's inter-AZ link flaps; ~1% errors [S2][S3]
one AZ, gray
yes: the design case
none then (AWS fixed it); drain button afterwards [S2]
22 Feb 2022
Consul 25% step → empty caches → scatter queries [S5]
fleet-wide step, not AZ-shaped
no, unless rollouts go AZ-first
boot throttle, missing-keys-only, replicas, re-key the query [S5]
Oct 2022
bulk user removal floods one shard; OOM → promote → replace loop [S17]
one shard (tenant-shaped)
no: the shard's load follows it
disable Thread View; tablet throttler + breakers [S17]
The “would a drain help” column is our judgment from each post's own description of scope. The 22 Feb row is amber because Slack later reworked deploys to fill one AZ first [S3]. If an agent rollout follows the same pattern, a bad step becomes AZ-shaped and drainable. That is the second-order payoff of cells: they turn some fleet-wide changes into zonal ones.
Copy it when
Do not when
Sharding key
Your largest tenant is growing faster than your mean tenant and is heading for one host
Tenants are small and bounded; tenant sharding still gives you free isolation
Datastore
You want to keep MySQL semantics, tooling and people, and can own a proxy tier
You are willing to change engines; a managed distributed store fits your query shapes
Migration style
Hot tables need new keys (re-architect them); the long tail only needs moving (move whole shards)
You cannot put a proxy in front of legacy nodes, or legacy topology can't feed replication
Cells = AZs
Most traffic is in one region, request fan-out is deep, and AZ-scoped gray failures are your recurring pain
Your outages are regional or dependency-wide (control planes, transit, quotas); drains won't touch them
Drain at the edge
You already have (or can get) one programmable L7 edge with weighted routing
Traffic enters through many independent paths or DNS-only discovery
Capacity
You can hold N/(N−1) peak headroom per AZ and prove it with routine drains
Your AZs run hot at peak and nobody will pay for the headroom
Three decision rules come out of this case that we have not seen stated elsewhere. First, watch your largest tenant, not your growth rate. When the largest tenant’s load, as a share of your largest host, has a projected date, you are already late to re-key. Slack reached that date with the tail growing 12.5x while the mean doubled[S8]. Second, split a migration by cost curve. Re-architect the few units that need a new key, and move the rest as whole units behind a proxy, validating with consistent-cut diffs[S10]. Third, isolate at the level where you can act. Slack could not act on per-RPC signals, but it could act on an AZ in seconds. The unit of isolation should be the unit your operators can see on a graph and your edge can shift weight away from, and every service below the edge should be classified by whether it can be siloed, failed over, or must be made eventually consistent[S3].
Our analysis — questions to ask in the design review
On a re-shard. What is the largest tenant’s peak load as a share of the largest host, and what does that line do after a 1.5x step (Section 2)? For each hot-path query, which keyspace’s key is in its predicate, and which queries only survive because a cache absorbs their scatter (Section 7)? What happens to query count per user when keys become finer (Section 6)?
On a proxy migration. Which units need a new key and which only need moving? Price them separately. How do you diff two live stores at one logical instant? What stops the tool from cutting over a shard other than the one you validated (Section 5)?
On cells. Can the drain be issued without reaching the sick zone? Does it express intent as weight, or as fake unhealthiness that panic routing will override (Section 9)? At today’s peak, what utilisation do the survivors reach, counting cold caches and moved primaries? How many semi-sync ack candidates does each shard keep during a drain? Which of your last five incidents would a drain have shortened? If the answer is one, is that the class that costs you most?
Primary sources are S1–S10, S12, S17–S20 and S23–S28: Slack engineers’ posts, slides and talks, Slack’s status page and the Vitess documentation. S11 is a vendor case study with first-person quotes. S13 was consulted only for its table of contents. S14–S16 are third-party summaries of podcast interviews, and S21–S22 are Hacker News discussions; they are cited only for what they say and flagged in the text. S7 is a Percona Live 2018 deck with no stable public URL. Where sources disagree (the Vitess start date, the June 2021 end time, drain times, COVID growth), the text gives each figure with its own context.
Scaling Datastores at Slack with Vitess — Arka Ganguli, Guido Iaquinti, Maggie Zhou, Rafael Chacón, Slack Engineering, 1 Dec 2020
Slack’s Migration to a Cellular Architecture — Cooper Bethea, Slack Engineering, 22 Aug 2023
Slack’s Migration to a Cellular Architecture (talk + transcript) — Cooper Bethea, QCon San Francisco 2024 / InfoQ, published 2 May 2025
Slack’s Outage on January 4th 2021 — Laura Nolan, Slack Engineering, 1 Feb 2021
Slack’s Incident on 2-22-22 — Laura Nolan et al., Slack Engineering, 26 Apr 2022
A Terrible, Horrible, No-Good, Very Bad Day at Slack — Laura Nolan, Slack Engineering, 29 Jun 2020
Migrating to Vitess at (Slack) Scale (slides; no stable public URL recorded) — Michael Demmer, Percona Live 2018, Apr 2018
Scaling Slack – The Good, the Unexpected, and the Road Ahead (talk + transcript) — Mike Demmer, QCon SF 2018 / InfoQ, 18 Dec 2018
Scaling Resilient Systems: A Journey into Slack’s Database Service (slides) — Rafael Chacón, Guido Iaquinti, KubeCon + CloudNativeCon NA 2019, Nov 2019
Codename VIFL: How to Migrate MySQL Database Clusters to Vitess (slides) — Rafael Chacón, Guido Iaquinti, KubeCon + CloudNativeCon NA 2020, Nov 2020
How Slack leverages Vitess to keep pace with ever-growing storage needs — CNCF case study (quotes Michael Demmer), 25 Nov 2019
How Big Technical Changes Happen at Slack — Keith Adams, Johnny Rodgers, Slack Engineering, 13 Feb 2020
Refactoring at Scale, ch. 11 “Case Study: Migrating to a New Database” (table of contents only) — Maude Lemaire, O’Reilly, Oct 2020
They Migrated Every Slack Message, Right Before COVID Traffic Tripled — TR Jordan, Tern blog (podcast summary), 14 Jul 2025
Slack’s 6am Database Club — TR Jordan, Tern blog (podcast summary), 3 Jun 2025
“We Should Be Able to Drain an AZ”: Slack’s Cellular Architecture — TR Jordan, Tern blog (podcast summary), 10 Jun 2025
The Query Strikes Again — Emad Mokhtar, Eduardo Ortega, Kevin Van, Slack Engineering, 15 Nov 2023
Unified Grid: How We Re-Architected Slack for Our Largest Customers — Ian Hoffman, Mike Demmer, Slack Engineering, 31 Jul 2024
Traffic 101: Packets Mostly Flow — Pramila Singh, Radha Kumari, Rafael Elvira, Slack Engineering, 28 Aug 2023
Migrating Millions of Concurrent Websockets to Envoy — Ariane van der Steldt, Radha Kumari, Slack Engineering, 15 Mar 2021
Hacker News discussion of S2 — Hacker News, 26 Aug 2023
Hacker News discussion of S1 — Hacker News, 6 Dec 2020
Slack status: incident of 30 June 2021 — Slack status page, 30 Jun 2021
Concepts: Cell — Vitess documentation, last updated 29 Oct 2025
Continuous Load Testing: Building a culture of performance with Koi Pond — Shreya Ramesh, Melissa Khuat, Slack Engineering, 29 Apr 2022
Disasterpiece Theater: Slack’s process for approachable Chaos Engineering — Richard Crowley, Slack Engineering, 1 Aug 2019
Building the Next Evolution of Cloud Networks at Slack — Archie Gunasekara, Slack Engineering, 21 Sep 2020
Slack Engineering #networking tag listing (checked for the promised part 2 of S2) — slack.engineering, checked 27 Sep 2026