At 22:52 UTC on 21 October 2018, routine maintenance to replace failing 100G optical equipment cut connectivity between GitHub’s US East Coast network hub and its primary East Coast data center. The link came back 43 seconds later. GitHub.com was degraded for the next 24 hours and 11 minutes. Nothing broke in the usual sense. The Raft quorum did what it was built to do, and GitHub’s own engineering blog had described this exact failure mode four months earlier. For an architect, this is the cleanest public example of a pattern that keeps getting built into systems: an automated action that takes seconds to perform and whose undo time scales with the size of your data. Our thesis: the failover was not the incident. The asymmetry between doing it and undoing it was.
43 s
network partition
24h 11m
degradation
~2,000×
outage ÷ trigger
5M+
webhook events queued
The first thing to get right is what failed. GitHub runs two very different storage systems. Git repositories live in Spokes, which keeps at least three replicas and commits writes synchronously to a strict majority. Its 2016 design post says that Spokes “puts the highest priority on consistency and partition tolerance.” Everything else, including pull requests, issues, authentication and the coordination of background processing, lives in MySQL. The interim report was explicit that the incident “only impacted website metadata stored in our MySQL databases” and that Git data “remains unaffected and has been available throughout the incident.” So the company already ran one strongly consistent, quorum-committed store, and the incident happened in the other one. That store had chosen availability through asynchronous cross-site replication plus automated promotion.
In 2018 the MySQL estate was multiple functionally-sharded clusters, each from “hundreds of gigabytes to nearly five terabytes,” each with “up to dozens of read replicas.” Each cluster had one primary for writes. Reads went to “a subset of replica servers in the vast majority of cases.” The read skew was extreme. By 2019 the main cluster, mysql1, which held a “large portion” of core data (users, repositories, issues, PRs), averaged 950,000 QPS, 900,000 of them on replicas and 50,000 on the primary. That ratio matters later. When a cross-country failover happens, replica locality is at least as important as primary locality, because about 95% of the load ran through replicas (by the 2019 figures).
Physically, GitHub ran user-facing services mostly “within our own data center facilities”: two physical data centers (US East, the primary, and US West), POPs and hubs on both coasts and in Amsterdam, and public cloud capacity in several regions. The postmortem’s diagrams label the East–West path 60 ms. The failed link was not the backbone. It was the connection between the East Coast hub and the East Coast data center. Judging by which orchestrator nodes went on to form the quorum, losing that one link cut the primary data center off from the rest of the network, including the East Coast public cloud (our inference from the postmortem’s account).
Sites, POPs, cloud regions and the 60 ms label are from the postmortem's network diagram. Its diagrams show orchestrator only in the two data centers; its text says the quorum that failed over was formed by US West DC and US East public cloud nodes, so we draw three voting sites and follow the text. Which hub each cloud region hangs off is our simplification. Our inference from the vote: the East cloud sat on the hub side of the failed link, so a single hub-to-DC link separated the primary site from both of the sites that outvoted it.
The last constraint is organisational. The postmortem says that “in the weeks prior to this incident,” GitHub had started a company-wide initiative to serve from multiple data centers in an active/active/active design. The system that failed was the one they were already planning to replace: one primary site, with a second site mostly holding read replicas.
Our analysis
mysql1 is the number most retellings leave out, and it changes how the incident should be read. Commentary focuses on writes crossing the country. But once the West became primary, the East replicas stopped receiving new data, because West-to-East replication was broken (Section 4). East apps then had two bad options for every read: read stale data locally, or pay 60 ms to the West. At a 95% read share, the read side of the invalid topology probably did more damage to latency and correctness than the write side. GitHub’s alt text points the same way: apps “are unable to read from current replicas.”GitHub had already paid for automated MySQL failover once. In September 2012, an earlier cluster built on Pacemaker, Heartbeat and the Percona Replication Manager failed over when it should not have. A Pacemaker segfault then caused a “cluster state partition” that elected an out-of-date node at 8:19 AM PDT, and the node was powered off at 8:26. For those seven minutes, 16 private repositories were exposed. Jesse Newland’s write-up drew a policy conclusion: primary failover would happen “only … when initiated by a member of our operations team,” because automated failover is fine in many places, but “ensuring the availability of our primary production database is not one of these situations.”
Four years later GitHub automated it again, deliberately and with better tooling. In December 2016 it announced Orchestrator, with master failures “automatically detected and recovered within 30 seconds or less.” Two properties were designed to avoid a repeat of 2012. Detection was holistic: the primary counts as dead only when orchestrator cannot reach it and its replicas agree that replication is broken. And there was anti-flapping: one automated failover per cluster per block period (the 2016 example was “say, 30 minutes”) until a human acknowledges. “Flapping is strictly unacceptable.” In July 2017 GitHub described the safety net underneath: full binary backups with Percona XtraBackup plus logical backups several times a day, a dedicated host per cluster that restores the latest backup on a schedule and must rejoin replication, a replica per production cluster delayed by 4 hours, and continuous failover testing on a production-like cluster spread across three data centers.
The June 2018 post by Shlomi Noach, “MySQL High Availability at GitHub,” is the key pre-incident document, and every senior engineer should read it next to the postmortem. It describes a four-part stack:
Every component and parameter is from Shlomi Noach's June 2018 post. The layout and the cross-DC arrows are our drawing. The October postmortem never mentions Consul, GLB or anycast, so how this stack behaved during the 43 seconds is reconstruction (see Section 3).
orchestrator/raft ran “one or two orchestrator nodes per data center,” arranged so that “no single data center makes a majority, and any n-1 DCs do.” Only the raft leader runs failovers. In a network isolation, nodes inside the isolated site cannot lead, and a leader inside it steps down. Consul ran independently in each DC, not replicated between them, as a KV store for the primary’s identity. Each orchestrator node wrote to its local Consul. GLB/HAProxy in every DC held one writer pool per cluster with exactly one backend. consul-template rewrote the config and reloaded it, with a very short hard-stop-after so old connections died on reload. If Consul was down, GLB kept its last known values. Anycast gave one writer name, such as mysql-writer-1.github.net, one IP everywhere, routed to the local GLB, so applications never knew which host was primary. Semi-synchronous replication ran “on local DC replicas” with a 500 ms timeout before falling back to async. Measured failover outages were “between 10 and 13 seconds … in most cases,” occasionally up to 20, and 25 at the extreme.
Then comes the part that reads differently after October. The post goes through isolation scenarios and states plainly: “Should there be a primary in an isolated DC, orchestrator will initiate the failover to replace it with a server in one of the available DCs.” It then names the leftover risk: “apps in that DC are still able to write to the primary. This may result in state inconsistency once network is brought back up.” It says a STONITH fix was being worked on. It also states the trade-off directly: “The operational cost of avoiding split-brains altogether is very high,” and “Lossless failover on a complete DC failure is costly and we do not expect it.”
Failure
Majority side
Orchestrator action
Consequence
East DC isolated, apps alive (Oct 21)
West DC + East cloud
Promote a West replica; West Consul updated
East apps keep writing to the old primary until reconnect: divergence
East DC dead (power, fire)
West DC + East cloud
Same promotion
No East writers left, so no divergence. Needs the whole site served from West
West DC isolated
East DC + East cloud
Nothing: primary is reachable
West replicas stale; West apps cannot write
East cloud isolated
East DC + West DC
Nothing
Invisible to users
Rows 1 and 2 look identical from the majority side: a primary it cannot reach, replicas reporting broken replication. The only thing that separates them is whether clients in the minority can still reach the old primary, and by construction the majority cannot observe that. That is what fencing (STONITH) is for, and the June 2018 post says GitHub was "working to mitigate this split-brain by implementing a reliable STONITH".
The quorum matrix shows the structural problem. From the majority side, a dead East DC and an isolated East DC look the same: a primary that cannot be reached and West replicas reporting broken replication. The right response is completely different. If the site is dead, promoting in the West is correct and there are no clients left to fork history. If the site is isolated, there are live clients on the minority side still writing to the old primary, and promotion creates two writers. Consensus answers “who may decide.” It cannot answer “is the thing I am replacing actually gone,” because the majority cannot observe the minority. That second question needs fencing, meaning some way to make the old primary stop accepting writes without its cooperation, and the June post says that was not finished.
Our analysis
Here is what GitHub reported. At 22:52, during the optical maintenance, connectivity between the East hub and the primary East DC dropped. During the partition, orchestrator in the primary DC, where the raft leader was, began “leadership deselection, according to Raft consensus.” Orchestrator nodes in the US West DC and the US East public cloud formed a quorum and started failing clusters over so that writes went to the West. Connectivity came back after 43 seconds. Then: “When connectivity was restored, our application tier immediately began directing write traffic to the new primaries in the West Coast site.” GitHub’s verdict: “Orchestrator’s actions behaved as configured, despite our application tier being unable to support this topology change.”
Reported in the postmortem: the partition, the leader stepping down, the West-plus-cloud quorum, the promotion, and the app tier "immediately" writing West on reconnection. Dashed bars are our inference from the June 2018 design (per-DC Consul, GLB keeps its last known backend) and from its own warning that apps in an isolated DC "are still able to write to the primary". GitHub never published when inside the 43 s each promotion completed, or how many clusters moved.
The postmortem leaves several things out, and they matter for anyone trying to copy the lesson. It never mentions Consul, GLB, anycast, semi-sync or STONITH. It does not say how many clusters failed over, whether every cluster moved or only some, or when inside the 43 seconds each promotion finished. The swimlanes above fill those gaps with the June design and say so. With per-DC Consul, the isolated East Consul could not be updated by the majority side, and GLB “keeps its last known values.” For as long as the partition lasted, East applications would have kept writing through the East GLB to the old East primary, which is exactly the case the June post warned about. When the link came back, the majority’s view reached the East, and East writes switched to the West “immediately.” That fits every reported fact, but it is our reconstruction, not GitHub’s account.
The timing is the part that should worry anyone running a similar design. GitHub’s normal failover took 10–13 seconds, and its 2016 goal was 30. The partition lasted 43. Detection plus promotion fit inside the blip with room left over.
The ordering that matters: the failover budget (10–13 s, 2016 goal 30 s) sits below the partition (43 s). Anything that fires faster than your longest plausible transient partition will act on transients. The recovery timescales sit three orders of magnitude above, and are set by data volume, not by any timer.
The first half hour, from the postmortem and the status page:
Twenty-one minutes from the partition to red is not slow by industry standards. What stands out is how long it took, 10 minutes, to understand the topology. At 23:02 the team was reading orchestrator’s API to find out where their primaries were. Our reading: that is what happens when a system is designed so that “apps are never aware of the primary’s identity,” as the June post put it. The abstraction that made failover invisible to applications also made it invisible to on-call engineers until someone went looking. The 23:07 deploy lock is the kind of decision that rarely gets credit: stop adding variables before you understand the ones already in play.
Our analysis — the detection budget
The phrase “split-brain” suggests two halves writing equally for a long time. The reported numbers do not look like that. On the East side there were “several seconds” (elsewhere “a few seconds” and “a brief period”) of writes that never reached the West. For scale, GitHub gave one number: “one of our busiest clusters had 954 writes in the affected window.” On the West side, the new primaries took application writes for “30+ minutes” (the decision paragraph) or “nearly 40 minutes” (the paragraph headed 23:13) before anyone could act.
Quotes, times and the 954 figure are from the postmortem; branch lengths are schematic except the inset. The shape is the argument: the West branch never had to be undone, it simply became canonical, and restored East servers could follow it because a backup is a prefix of it. The East branch, a few seconds wide, is what made in-place re-pointing impossible and turned the rest of the night into a rebuild plus a reconciliation project.
A note on the timestamps: 22:52 to 23:13 is 21 minutes, so “nearly 40 minutes” in the 23:13 paragraph must describe a later moment, probably around 23:30. We use the two figures as GitHub wrote them and do not compute one of our own. The where of the East writes is also not stated. The postmortem does not say whether they were writes committed on the isolated side during the 43 s, or transactions committed just before the partition that had not yet been applied on the West replica when it was promoted. Both are possible, and the June design allows both: semi-sync acknowledgement came from local East replicas with a 500 ms timeout, so an East commit could be acknowledged to a client without any West server having it. Hacker News commenters drew that inference. GitHub never discussed semi-sync’s role.
Our analysis — what 954 says about the window
mysql1’s 50,000 primary QPS (2019) includes reads. But the number suggests the East branch was closer to “seconds of replication lag plus whatever happened before the old primary stopped taking writes” than to “43 seconds of full traffic.” Either way the East branch is two to three orders of magnitude smaller than the West branch. HN readers pointed out that “one of our busiest clusters” gives no fleet total, and no later post provided one.Here is the mechanism that made those few seconds decisive. The West primaries now had writes the East lacked, and the East servers had writes the West lacked. In GitHub’s words: “Because the database clusters in both data centers now contained writes that were not present in the other data center, we were unable to fail the primary back over to the US East Coast data center safely.” The unreplicated East writes also “prevented replication of new writes back to the East Coast.” An East server holding transactions the West primary never saw cannot just be pointed at the West primary as a replica. Its data is a fork, not a prefix. To make East servers follow the West, you need an East copy that is a clean prefix of West history, which means rebuilding them from something consistent with the West. That is exactly what the next 17 hours were.
Counterfactual
read_only before the West promotion (orchestrator sets the old primary read-only “if possible,” and during a partition it is not possible). Then the East servers would be strict prefixes of West history. They could have been re-pointed at the West primaries in minutes, caught up over the WAN, and a planned failover back East (10–13 s) run once they were current. The West’s 30+ minutes would not have been a problem. It would simply have been the newest history. On this reading, the few seconds of East writes, not the 30+ minutes of West writes, turned a cleanup measured in hours into a rebuild that took a day. That is our inference from GitHub’s mechanism statement, not a claim GitHub made.Row A positions inside the 43 s are not reported; the 10–13 s band is GitHub's typical failover from June 2018, used as a stand-in. Row B, the lease length and the 2 s margin are ours. The reason fencing has to run on the minority side: majority-side STONITH needs a path into the isolated site, and a hub-to-DC partition removes exactly that path unless there is an out-of-band management network.
Our analysis — why STONITH was the wrong shape of fix for this partition
MySQL made this harder than it had to be. MySQL has no built-in equivalent of Postgres’s pg_rewind for rolling a diverged server back to a common ancestor, so “rebuild from a clean image” was the conservative path, not an over-reaction. The 2016 orchestrator post describes GitHub using Pseudo-GTID for topology refactoring, on MySQL 5.6 builds with replication then migrating from STATEMENT to ROW format. That tooling is excellent for moving replicas around a single history. It does nothing for two histories.
At about 23:15 the database tier was, in narrow terms, fine. Every cluster had one writable primary and some replicas. The problem was where those primaries were relative to the code that used them. GitHub: “Leader-election within a region is generally safe, but the sudden introduction of cross-country latency was a major contributing factor.” And: “It’s possible for Orchestrator to implement topologies that applications are unable to support.”
Before 22:52 · all reads local
US East
writes: local. reads: local replicas
US West
writes: cross-country to East
After the failover · the invalid topology
US East
writes: 60 ms away. reads: stale locally or 60 ms away
US West
West → East replication broken
App placement follows the postmortem's high-level diagrams, which show the github.com and jobs tiers only in the East. GitHub's alt text: in the invalid topology apps "are unable to read from current replicas as they depend on low latency to maintain transaction performance."
The diagrams show the web tier (github.com) and jobs only in the East; the West has notifications and Pages. These are high-level diagrams, so we do not rely on the exact placement. The direction is still clear: most of the application tier that generated database traffic sat in the East, and after the failover its primaries were 60 ms away. GitHub’s summary of the East application tier is that it was “currently unable to cope with the additional latency introduced by a cross-country round trip for the majority of their database calls.”
Illustrative, not measured: GitHub published no per-request query counts. The shape is the argument. A page that was never latency-bound becomes linearly latency-bound in its query count, and inside a transaction the same arithmetic stretches lock hold times: five statements at 60 ms hold row locks for at least 300 ms, capping a hot row at about 3 transactions per second.
The latency chart turns “unable to cope” into numbers. Illustratively: a request that makes 30 serial database round trips (plausible for an ORM-heavy page rendering a pull request with its reviews, checks and permissions; GitHub published no counts) gains 1.8 seconds. A background job looping over a hundred records gains six. Inside a transaction it is worse: every round trip adds to lock hold time. A hot row updated by five-statement transactions can now take about three commits per second, and every other transaction waiting on that row queues behind it. Connection pools sized for sub-millisecond queries fill up at 60 ms, because connections are held about a hundred times longer at the same arrival rate. By Little’s law, in-flight connections scale with latency (our arithmetic throughout this paragraph). None of this breaks the database. It breaks everything around it.
Our analysis — the invariant nobody wrote down
It is worth being clear about what this does not say. It does not say cross-region failover is always wrong. If the unit of failover had been the whole serving stack (apps, jobs and primaries moving together, as in a cell or region evacuation), a West promotion would have been a working West site rather than a stranded database. An HN commenter argued that intra-DC failover has the same data-loss mode and only a smaller window. That is fair about loss, but it misses GitHub’s stated reason, which was latency the application cannot tolerate, not loss as such.
By 00:05 on 22 October the team had a plan: “restore from backups, synchronize the replicas in both sites, fall back to a stable serving topology, and then resume processing queued jobs.” The postmortem’s recovery diagram breaks it into six steps: fail forward; restore from backups; synchronize across all nodes; fail back (a manual failover); “churn through backlogs,” labelled “A race against incoming traffic, serving capacity, and not DoSing partners”; and manual follow-up to restore missing data, “still in progress” when the postmortem was published.
Option
Time to a sane topology
Data risk
User pain meanwhile
Status in the sources
Fail back to East now, set West writes aside
Minutes (planned failover: 10–13 s)
30+ min of user writes orphaned; already served to users
Short
Rejected, explicitly
Stay on West primary with East apps
Zero
None
“Unusable for many users”
Rejected as an end state; lived with as the interim
Move all serving to the West
Unknown
None
Unknown
Not discussed in any source
Rewind East servers, re-point them at West
Hours
High: surgery on divergent data files
Medium
Not discussed
Merge both histories automatically
Days to weeks
Highest: bespoke conflict rules on live data
Long
Not discussed
Fail forward, restore East from backup, resync, fail back, drain jobs
~24 h
Lowest: West kept; East binlogs kept for reconciliation
Long, degraded
Chosen at 00:05
Rows 1, 2 and 6 and their stated reasons are from the postmortem. Rows 3–5 are ours; the last two echo options a Hacker News commenter listed, which GitHub never addressed. Time and risk scores for rows 3–5 are our estimates.
Two alternatives were rejected explicitly, with reasons. Failing back to the East immediately was ruled out because “the 30+ minutes of data written to the US West Coast data center prevented us from considering options other than failing-forward in order to keep user data safe.” Staying on the West with East apps was ruled out as an end state because the latency “would result in our service being unusable for many users.” The team chose longer degradation over either: “We believe that the extended degradation of service was worth ensuring the consistency of our users’ data.”
Consider the failback option in detail, because it is the one an on-call engineer at another company is most likely to reach for at 23:30. Failing back discards (or rather sets aside for later re-application) half an hour of user-visible writes. Those writes had already been observed: users saw their comments, merged their PRs, closed their issues. Some, we infer, may have triggered external side effects in the 27 minutes before webhooks were paused at 23:19. Rolling them back is not like rolling back an uncommitted transaction. It contradicts what the world already saw, and each re-application has to be checked against East state that has moved on since. The chosen path was slow, but it only had to reconcile the small branch, the seconds of East writes held in captured binlogs. Failback would have meant reconciling the large one. That asymmetry, not a general preference for correctness, is the strongest argument that GitHub chose well.
Counterfactual — serve everything from the West
The underrated decision is 23:19: pausing webhooks and Pages builds. GitHub called it an “explicit choice to partially degrade site usability … instead of jeopardizing data we had already received” and said their strategy was “to prioritize data integrity over site usability and time to recovery.” Pausing the writers that run in the background is the database equivalent of the deploy lock. It stops the fork from getting bigger in ways nobody is watching, such as push-processing jobs writing metadata against whichever primary they reach. It also stopped outbound effects based on state that might later be reconciled. The cost was a queue that grew for 17 hours, and that queue produced the incident’s last surprise (Section 8).
The last decision was about communication. After the 16:24 failback, when the data tier was healthy again, GitHub kept status red and “remained in degraded status” until the backlog had drained, “to avoid further eroding the reliability of our status updates.” By then the status page had missed five ETAs in a row. Choosing to under-promise after that was the correct response to that record.
The recovery timeline, from the postmortem and the archived status page:
Where the 1,451 minutes went
All timestamps from the postmortem and archived status page; the durations and the split are our arithmetic (22:52 → 00:41; 00:41 → 11:12; 11:12 → 16:24; 16:24 → 23:03). The postmortem text says status stayed red after 16:24; its recovery graphic shows a yellow band. We mark it disputed.
We will not re-narrate the status page; the chart carries it. Four facts from the postmortem matter for what follows. The restore path was remote blob storage → decompress, checksum, prepare, load onto newly provisioned servers, from backups taken every four hours; it was tested “daily at minimum,” yet “until this incident we have never needed to fully rebuild an entire cluster from backup.” In the postmortem’s 06:51 entry the team has found a way to restore “directly from the West Coast,” bypassing the off-site download limit: roughly six hours into a procedure whose bottleneck had never been measured at fleet scale. At 11:12 every primary was back in the East, but “dozens” of replicas were “multiple hours delayed” and catch-up followed “a power decay function instead of a linear trajectory,” worsened by European and US morning load until extra cloud read replicas came online at 13:15. And the 07:46 interim post needed extra effort because GitHub’s blog publishing depended on the Pages builds the incident had paused.
Quoted estimates from the archived status page. The 06:51 figure was, per the postmortem, "linearly interpolated from the replication telemetry". Misses against 16:24: 7 h 33 m, 7 h 19 m, 7 h 05 m, 5 h 07 m, 3 h 28 m. Each new estimate kept the same one-to-two-hour horizon; that is the signature of a linear model applied to a non-linear process (Section 8).
The time split is the most useful thing in this section. About 7.5% of the outage went to detecting the problem and deciding what to do. The other 92.5% was spent moving and replaying data: restore (43.5%), replica catch-up (21.5%), and fail-back plus backlog drain (27.5%). Better incident response would have shaved minutes off a 24-hour outage. The postmortem says so itself: “tighter operational controls or improved response times are insufficient safeguards for site reliability.” Recovery time was set by bytes and replication throughput, and none of that could be sped up at 23:13 because the capacity for it was never provisioned.
Our analysis — the critical path was serial by construction
Our analysis — the status page was also a system under test
The multiplier. 24 h 11 m is 87,060 seconds. Divided by 43, that is about 2,025×. An HN commenter called it a “2000x multiplier between connectivity loss and service degradation.” The multiplier is not a random property of a bad night. It is the ratio between the cost of a reversible network event and the cost of an irreversible data event. Any design that can turn the first into the second will have a multiplier like this, and its size depends on how much data has to move.
Link
What amplified it
Reported cost
Cut point (ours)
1. 43 s partition
reversible network event
0 data effect on its own
Region guard: promotion declines across the boundary (GitHub’s chosen fix)
2. Cross-region promotion
seconds to perform
primary 60 ms from most writers
Promotion scope = writer placement; failover unit is the stack, not the DB
3. Seconds of East writes
fork, not prefix
no in-place re-point possible
Self-fencing lease on the old primary (diagram above); a rewind tool
4. Rebuild East from backup
bytes over the slowest link
00:41 → 11:12 · 10 h 31 m
Restore from a live in-sync source; smaller largest cluster
5. Replica catch-up
queue under rising load
11:12 → 16:24 · 5 h 12 m
Spare apply headroom held in reserve; load-aware ETA
6. Paused queues
age = pause length
~200k webhooks dropped at TTL
Runbook raises TTLs to ≥ pause + drain before pausing
7. Comms path
shares the paused pipeline
07:46 post needed extra effort
Status and blog on a path no runbook pauses
Costs are from the postmortem and status page; durations are our arithmetic. The chain is ordered by cost of the cut: links 1–3 are policy and configuration, links 4–5 are provisioned capacity, and 6–7 are runbook hygiene. Cutting either of the first three means the rest never happens. Cutting only the later ones shortens the night but still leaves the fork.
Read the chain for leverage, not blame. Links 1–3 decide whether there is a fork, and each can be cut with configuration or a lease. Links 4–7 decide how long the fork costs, and they are cut with capacity and runbooks that must be in place before the night. GitHub’s announced fix cuts link 1 only. That is sufficient for this trigger, but links 3–5 remain the worst-case recovery path for any other route to divergence, including the intra-region one in Section 11.
Restore bandwidth. The largest cluster was “nearly five terabytes” and the restores were “multiple terabytes” in total. Our calculation, since GitHub gave no bandwidth figures: bringing 5 TB from remote blob storage in 6 hours (00:41 to 06:51, when the first clusters finished) needs about 230 MB/s, roughly 1.9 Gbps sustained, just for the transfer. Compression reduces the bytes moved, but decompression, checksumming and XtraBackup’s prepare phase all run after it, the postmortem lists that prepare step, and the 2017 post names Percona XtraBackup as the backup tool. At 1 GB/s end to end, 5 TB takes about 83 minutes. At 100 MB/s it takes about 14 hours. All primaries were re-established in the East about 10.5 hours after restores began. If the largest cluster took roughly that long, that puts the effective pipeline throughput in the low hundreds of MB/s for the biggest cluster. That fits an off-site download limit, and it explains why switching to “restore directly from the West Coast” mattered. The postmortem does not say whether that meant copying backup files from the West or cloning live West servers. Cloning would also have given a fresher starting point, and less binlog to replay, than a backup up to four hours old.
Our analysis — the restore test that tests the wrong thing
Catch-up. The postmortem’s best technical sentence is the one about catch-up following “a power decay function instead of a linear trajectory.” A restored server starts from a backup that is hours old, so it begins with a large replication backlog and has to replay it while new writes keep arriving. That is a queue, and a queue drains in time proportional to backlog × r/(1−r), where r is incoming load divided by apply capacity.
Model (ours): a replica applies work at A while the primary produces it at W; a backlog of B seconds-of-primary-work drains in B·r/(1−r) with r = W/A. It is a queue, so drain time is hyperbolic in load. GitHub reported the empirical shape ("a power decay function instead of a linear trajectory", made worse by morning load) and the fix (more replicas to spread read load); the formula and the r bands are illustrative.
The model explains three reported facts at once. First, why the 06:51 linear ETA failed: an early slope measured overnight, with low r, predicts a finish time that is completely wrong once r rises. Second, why European and US morning traffic made it worse: going from r ≈ 0.5 to r ≈ 0.8 quadruples the remaining drain time for the same backlog. Third, why adding read replicas at 13:15 helped even though it adds no write capacity: replicas that also serve reads split their I/O and CPU between queries and the replication applier, so spreading the reads raises each replica’s effective apply capacity and lowers r. In MySQL of that era the applier was often limited to a single thread in practice unless parallel replication was tuned, which makes apply capacity especially sensitive to contention from reads. That last point is general MySQL knowledge. GitHub did not say how its appliers were configured.
Our analysis — two things “power decay” hides
1. At constant load, catch-up is linear. A replica applying at A while the primary produces at W advances 1/r primary-seconds per wall-second, so lag falls at a constant (1−r)/r seconds per second. A lag curve that flattens as it approaches zero is therefore the signature of r rising during the drain, not of replication being intrinsically non-linear. The 06:51 interpolation was a reasonable method in a constant world; it failed because the drain ran straight into the daily traffic ramp. Any ETA for a multi-hour catch-up should be computed against the load forecast for the hours it spans, not the slope of the last hour.
2. Restore time is multiplied, not added. A server restored from a backup of age a, after a restore taking T, must replay a + T of primary history before it is current. Time to a current replica is T + (a + T)·r/(1−r) = (T + a·r)/(1−r). With illustrative a = 2 h and T = 8 h: r = 0.3 gives about 12 h, r = 0.5 gives 18 h, r = 0.8 gives 48 h. Every hour shaved off the restore saves 1/(1−r) hours overall, and a source with a ≈ 0 removes the second term entirely. That is the quantitative case for the 06:51 pivot to restoring from the live West side, and the reason a fork’s RTO must be computed with this formula rather than read off a restore benchmark.
Illustrative: time = (S/B + a·r)/(1−r), the Section 8 formula, with a = 2 h. Throughputs and r are assumptions, not GitHub figures; the 5 TB line is reported, and the 15 h 43 m fleet span (00:41 restores start → 16:24 replicas in sync) is our arithmetic on reported times. Reported cluster sizes and fleet timing sit between the 100 and 250 MB/s lines, consistent with Section 8's estimate. Two readings. The RTO for a fork is a slope in terabytes, so splitting the largest cluster moves the worst case left along the x axis. And the r = 0.8 line shows that write load during the rebuild can cost more than a 2.5× slower pipe.
The curve gives decision thresholds you can check against your own fleet (illustrative, our model). If time to an in-sync rebuild of the largest cluster exceeds your tolerable outage, then no automated action that can fork that cluster is acceptable without fencing, whatever its success rate on crashes. The two levers are not equal. Doubling restore throughput halves only the S/B term. Keeping r at 0.5 rather than 0.8 during the rebuild cuts the whole expression by at least 2.5×, because 1/(1−r) multiplies the restore term as well as the replay. Reserved replica headroom, the lever GitHub reached for at 13:15, therefore keeps its value at every cluster size. Throughput’s value shrinks as backup age and write load become the larger terms.
The webhook backlog. More than 5 million hook events and 80,000 Pages builds were queued. Webhook delivery ran from 16:45 to a 22:23 “caught up” status message, minus a pause from 17:32 to 18:26, which the status page attributes to “an issue” and which almost certainly corresponds to the TTL episode, though no source links the two explicitly. That leaves roughly 4 h 44 m of effective drain, or about 290 events per second on top of live traffic, paced so as not to overwhelm partners. The drops are the interesting number. About 200,000 of more than 5 million is roughly 4%. The queue started filling at 23:19 and draining at 16:45, so the oldest events were about 17 h 26 m old when delivery resumed. Any TTL shorter than that was always going to drop events. The pause was the aging. A queue built for normal delivery delays had never been held for 17 hours.
Our analysis — pausing a queue is a schema change
“Raft / Orchestrator malfunctioned.” No. GitHub says it “behaved as configured.” The isolated side lost leadership, the majority side acted, and the June 2018 post had documented this behaviour as intended. The failure was a policy mismatch, cross-region promotion allowed for an application that could not use a cross-region primary, not a consensus bug. The postmortem calls it “emergent behavior of the system given that we hadn’t previously seen an internal network partition of this magnitude.”
“It was a storage hardware failure.” The early press said “dead data storage gear,” taken from status-page wording about repairing “a data storage system.” The trigger was network maintenance on 100G optical equipment. The storage was fine. It just held two histories.
“Git repositories were affected / stale code was served.” The Register wrote about “serving out-of-date code repos.” The interim report says Git data “remains unaffected and has been available throughout the incident.” What users saw was stale metadata such as PR and issue state, all drawn from MySQL. Git objects themselves were not stale. Spokes, the Git store, is the CP half of GitHub’s architecture and it behaved like one.
“No data was lost.” Both true and incomplete. The postmortem says no user data was lost, and also that reconciling “a few seconds of database writes is still in progress,” with some writes possibly requiring “outreach to users.” One category it had already found: writes “repeated by the user and successfully persisted,” where the user retried after the site misbehaved. No follow-up post on the final reconciliation was found. Separately, about 200,000 webhook payloads were dropped. The accurate statement is: stored user data preserved, a small tail reconciled by hand, some outbound events lost.
“Both sides wrote for 40 minutes.” The East branch was seconds, the West branch 30+ minutes (Section 4). The outcome was a split-brain, divergent histories, but not two equal halves writing for a long time. That matters for the lesson. Tiny divergence has the same structural consequence as large divergence, which is why fencing is binary: “we only lose a few seconds” is not a smaller version of the problem, it is the whole problem.
“The lesson is: never automate database failover.” That was the 2012 lesson, and GitHub deliberately moved away from it in 2016. The likely reasons (our reading, not GitHub’s stated ones): human-initiated failover is slow at 3 a.m., and most primary failures are local crashes where automation is simply better. The 2018 lesson is narrower: automation must not cross a boundary the application cannot follow. A May 2020 incident shows the trade-off still exists after the fix. A promoted primary crashed after serving “approximately six seconds of write traffic,” and rebuilding replicas took about 4 hours plus an hour of reconfiguration. In June 2020, orchestrator’s anti-flapping block stopped a follow-up automatic failover during cascading crashes. Automation within a region brought its own problems, but the May 2020 rebuild was measured in about five hours, not 24.
“Microsoft / Azure caused it.” The acquisition closed five days later. The postmortem does not name the blob store holding the backups. An HN commenter guessed S3, which is unverified. Neither fact has anything to do with the cause.
The postmortem committed to four technical changes: configure Orchestrator to “prevent the promotion of database primaries across regional boundaries”; accelerate the move to component-level status; build active/active/active with N+1 facility-level redundancy, “to tolerate the full failure of a single data center failure without user impact”; and take “a more proactive stance in testing our assumptions,” including a “systemic practice of validating failure scenarios” and “investment in fault injection and chaos engineering tooling.” On communication, it admitted the estimates “did not factor in all variables” and committed to being able to publish updates even when Pages builds are paused.
Sep 2012
Pacemaker/PRM cluster fails over when it should not, then partitions and elects a stale node; 16 private repos exposed for 7 min. Primary failover becomes human-initiated only.
Dec 2016
Orchestrator adopted: automated failover, “30 seconds or less”, holistic detection, anti-flapping block until a human acks.
Jul 2017
Testing automation: backups restored and verified continuously; a 4-hour delayed replica per cluster; failover drills on a 3-DC test cluster.
Jun 2018
orchestrator/raft + per-DC Consul + GLB/HAProxy + anycast. Isolated-DC primary will be failed away; the resulting inconsistency is named; STONITH “in the works”.
21–22 Oct 2018
43 s partition → cross-region promotion → 24 h 11 m degradation.
26 Oct 2018
Microsoft completes the GitHub acquisition (five days after).
30 Oct 2018
Postmortem: prevent cross-region promotion; component-level status; active/active/active; fault injection.
Jan 2019
orchestrator PR #766 merged: PreventCrossDataCenterMasterFailover (default false).
May 2019
orchestrator PR #884 merged: Region concept + PreventCrossRegionMasterFailover.
2019 → 2021
mysql1 partitioned: 950k QPS in 2019 → 1.2M QPS in 2021 with host load halved; 130 busiest tables moved.
22 May 2020
Promoted primary crashes after ~6 s of writes; replicas rebuilt from the new primary: ~4 h + 1 h.
Dec 2023
MySQL 8.0 upgrade post: 1,200+ hosts, 300+ TB, 5.5M QPS, 50+ clusters.
Apr 2026
Availability update: moving from smaller custom data centers into public cloud.
Neither orchestrator PR names the October incident. Timing, author and the cc to @github/database-infrastructure make the link a strong inference; "subsequently added" is the accurate phrasing.
The follow-through that can be verified is in the orchestrator source. In January 2019, PR #766 added PreventCrossDataCenterMasterFailover, with @github/database-infrastructure cc’d. Its description is our incident in abstract form: “Say all masters are in dc1 and dc1 gets network isolated … it cannot find a server to promote … all other servers are disqualified.” In May 2019, PR #884 added a Region concept and PreventCrossRegionMasterFailover. Both default to false. With either enabled, a failover that finds no same-DC or same-region candidate aborts. Neither PR mentions October, so “subsequently added” is the accurate phrasing. The design choice is worth noting: the guard does not wait longer or ask a human. It declines. Across the boundary, the answer is to do nothing, and a person decides.
Counterfactual — the same night with the guard enabled
PreventCrossRegionMasterFailover on. The majority still forms and still sees unreachable primaries, finds no same-region candidate, and aborts. For 43 seconds, writers that could not reach the East primaries (the West-side tiers) fail; East apps keep writing locally, exactly as they did. At reconnection there is one history, nothing to reconcile, no restore, no backlog. The cost of the guard on this night is on the order of 43 s of partial write errors against 87,060 s of degradation. The case against the guard is a permanent East loss, and here the postmortem’s own diagrams bite: if the web and jobs tiers ran only in the East (inferred from the diagrams, not stated), a promoted West primary in that scenario would have had no main application tier to serve. On that reading, automated cross-region promotion was insurance that paid out only in a scenario where it could not help, and was the direct cause of damage in the one scenario it actually met. That is an argument about GitHub’s 2018 placement, not about cross-region automation in general; Section 11 gives the conditions under which the calculation flips.The larger change was structural. In 2019 GitHub started partitioning mysql1. The 2021 write-up reports 1.2 million QPS, host load halved, 130 of the busiest tables moved out, and a reduction in “the number of database-related incidents.” By the MySQL 8.0 upgrade post in December 2023, the fleet was 1,200+ hosts, 300+ TB, 5.5 million QPS across 50+ clusters, compared with 5 clusters at the previous major upgrade. Partitioning is not presented as a response to October 2018, and we do not claim it was one. But its effect on this failure mode is direct: rebuild time scales with the largest cluster, and splitting the largest cluster is the only thing that reduces the worst case.
Our analysis — what we could not verify
GitHub’s own fix was to stop automated promotion at regional boundaries. Whether you should do the same depends on answers only you have. The tree below is how we would decide. It applies to any single-writer store with async cross-site replicas: MySQL, Postgres with Patroni or similar, MongoDB replica sets across regions, Redis with Sentinel.
Can every writer of this database (web, jobs, queues) run in the target site at acceptable latency, today, with capacity?
No → the failover unit is not the database. Forbid automated promotion across this boundary.
Yes → continue
Can the promoter fence the old primary before the new one takes writes, from the majority side?
No → an isolated minority with live clients will fork your history. Promotion must wait for fencing or a human.
Yes → continue
Is cross-site replication synchronous at commit (RPO = 0)?
No → every automated promotion is a potential fork. You need a divergence runbook and a measured full-rebuild time.
Yes → continue
Is detection + promotion slower than the longest transient partition you have seen or can induce?
No → the automation acts on blips. Lengthen it, or accept that it will.
Yes → continue
Have you rebuilt your largest cluster from backup, at production size, under load, recently?
No → your recovery time is a guess. GitHub’s was, until this night.
Yes → automation across the boundary is defensible
Our synthesis. In October 2018, GitHub's MySQL tier answered question 1 "no" (per its own postmortem), question 2 "not yet" (per its June post), question 3 "no" (async cross-DC, semi-sync only locally), question 4 "no" (10–13 s vs 43 s), and question 5 "never at full scale".
Copy GitHub’s fix (forbid cross-region automated promotion) when your application tier is not deployed at full capacity next to every candidate primary; your cross-site replication is asynchronous; you cannot fence the old primary from the majority side; or you have never measured a full rebuild. That describes most companies with a “DR region.” For them, a cross-region move should be a human decision taken as a site evacuation, not a database event. The cost is honest: a real regional loss now waits for a person. Put that in your RTO rather than pretending automation covers it.
Do the opposite (automate across regions) only when the unit of failover is a full stack, meaning cells or regions that carry their own writers, jobs and caches; when promotion is gated on fencing (STONITH, storage-level fencing, or a lease the old primary must hold to accept writes); and ideally when commit-time replication is synchronous to a quorum across sites. That last condition is the CP choice GitHub already made for Git data in Spokes. It costs write latency at every commit. GitHub’s June 2018 post said “Lossless failover on a complete DC failure is costly and we do not expect it,” and for a 95%-read workload that trade may well be right.
Pitfalls in copying the fix. First, “prevent cross-region promotion” turns an isolated-primary-site event into write unavailability for its duration. That is fine for 43 seconds and not fine for a real site loss, so the human runbook for the real loss must be rehearsed, not just written. Second, a guard at the promotion layer does nothing about the other path to divergence: clients on the minority side writing to a primary that was fenced too late. Leases and fencing tokens are still needed for intra-region failover, where the window is smaller but the mechanism is the same. The Distributed Systems Deep Dive track covers fencing tokens and lease safety. Third, the fix treats “region” as a static label. When your topology changes, such as a new cloud region, a migration, or a replica placed somewhere for capacity, the label has to change with it, or the guard silently stops protecting you.
Checklist — what to measure this quarter
1. For every automated promoter: the fastest time from partition to promotion, compared with the longest transient partition in your network history. If promotion is faster, it will act on blips.
2. For every cluster: time from “decide to rebuild” to “in-sync replica under production write load,” measured at full size. Largest cluster first. That is your worst-case RTO for a fork.
3. For every queue fed by writers you might pause: the TTL and retention, compared with a realistic incident pause plus drain time.
4. For your status and comms channels: which of them depend on the systems your runbooks tell you to pause.
5. For every service that calls the database: its tolerance to a primary 60 ms (or your inter-region RTT) away, measured and not assumed. If the answer is “unusable,” your failover policy must know it.
Questions to ask in a design review
1. “Draw me the partition that isolates your primary site from both of the other voters. Which single link or device is it?” (On Oct 21 it was one hub-to-DC link, per our reading of the vote.)
2. “After the majority decides, what is the longest the old primary keeps accepting writes, and what enforces that bound without the majority’s help?”
3. “If the promoted primary has to become canonical, how do servers holding the orphaned tail rejoin? Rewind, clone from the new primary, or restore from backup, and how long does each take for the largest cluster?”
4. “Which callers of this database cannot follow the primary, and does the promoter know their names?”
5. “Which of your runbook’s ‘pause’ steps change the semantics of something downstream: TTLs, retries, partner rate limits, your own status page?”
The SRE / Reliability track’s chapters on blast radius and cells are the positive version of this case: design the failover unit so that it contains its writers. The Databases from the Inside track covers why a diverged replica cannot simply be re-pointed. The Architecture Judgement essays on reversibility are the general form of our thesis. When an automated action is cheap to take and expensive to undo, the policy governing it should be set by the cost of undoing it, not the cost of taking it.
The one-sentence version
Primary sources first. Every number and quote above comes from one of these. Analysis, models and counterfactuals are marked as ours in the callouts and diagram captions.