On 4 October 2021, at about 15:39 UTC, Facebook, Instagram, WhatsApp, Messenger and Oculus disappeared from the internet together and stayed gone for roughly six hours. It was retold as “a DNS outage”, “a BGP outage”, or “the day engineers were locked out of their own data centres”. None of those is the mechanism. An operator ran a command meant to assess backbone capacity. A safety check written for one plane at a time let an eight-plane argument through, and every backbone router drained within seconds. The rest followed from designs that were reasonable on their own. Every defence in the chain was built for partial failure, and each one turned into an amplifier when the failure was total. The DNS health check assumed another site would be healthy. Out-of-band access assumed some region would be reachable. Physical security assumed the network it checked against would be up. The restart had to be managed because the fleet had gone quiet all at once.
8 / 8
backbone planes drained by one command
~6 h
Meta's own duration figure
30×
query load on public resolvers
10.6 M
Downdetector reports, a record
3–4 h
saved by the operator speaking up (Richard)
tens of MW
power dip per DC during the outage
172,800 s
NS TTL at .com: no quick escape
~1 month
network-org code freeze afterwards
This study puts Meta's two blog posts and Francois Richard's SREcon22 EMEA talk (the fullest first-hand account, and rarely cited) next to what outside observers measured: Cloudflare, Kentik, ThousandEyes, Catchpoint, RIPE RIS data via MANRS and Catchpoint, and Packet Clearing House. It reconstructs the command path, the health-check logic and the dependency graph of the recovery, and analyses the durations, prefix counts, TTLs and load figures. It then scores the alternatives people proposed afterwards against each phase of the outage. That scoring gives a result the usual retelling misses: none of the DNS-side fixes most often recommended would have kept Facebook up. Analysis is in labelled callouts. Where Meta has published nothing (the exact command time, when anyone reached a router, the tool's code), we say so.
Meta's basic unit of compute is the region: a group of nearby data centres. Richard counted “around 15 to 17” of them, with “up to a million server or more on each region”. Users do not connect to regions. A request goes to the nearest small facility, a POP, or further out to an FNA (Facebook Network Appliance) cache inside an ISP, and then crosses Meta's backbone to a region. Janardhan's post describes that backbone as “tens of thousands of miles of fiber-optic cables crossing the globe”.
There are really two backbones. The Classic Backbone (CBB) carries DC-to-POP traffic on conventional IP/MPLS-TE. The Express Backbone (EBB), serving traffic since around 2015, carries DC-to-DC traffic on a custom stack: Open/R for routing and messaging, a central traffic-engineering controller, on-box LSP agents and, per the 2017 design post, a “centralized (and highly redundant) ensemble of BGP-based route injectors to move traffic on/off the network”. EBB exists because, as Richard put it, traffic between regions (integrity work, ML processing) was growing faster than traffic from users.
Components from Janardhan's 5 Oct 2021 post (smaller facilities connect the backbone to the internet and answer DNS), Richard's SREcon22 talk (regions, POPs, FNAs, EBB/ABB/CBB, DNS served from POPs and health-checking “the region behind me”) and Meta's 2017 and 2025 backbone posts (CBB for DC↔POP, EBB for DC↔DC). Meta never published a single diagram with all of these on it; the layout is our reconstruction. The feature that matters: the component that tells the internet where Facebook is (DNS) lives at the edge, but its health is defined by reachability of the core.
EBB's defining feature is the plane. Meta's 2017 design post describes it split into four parallel topologies, copying the four-plane fabric inside its data centres, and notes that A/B testing between planes, each carrying an equal share, made fast iteration safe. By 2021 there were eight. Richard called them “eight independent data planes … like eight different backbones”, and Meta's 2025 post confirms that going from four to eight planes doubled global capacity.
The planes set the operating model. Take one plane out, do the migration, firmware or config change, send packets back gradually, then validate or roll back. Richard: “it's an operation that we do all the time”, and it does not “risk the entire capacity of the backbone”. That operation is run through tooling Richard calls “backbone managers”, including a command-line tool that takes a data-plane argument. Meta's post adds that such commands are audited: “Our systems are designed to audit commands like these to prevent mistakes like this.”
Plane count and the operating model are from Richard's SREcon22 talk (“eight independent data planes”; take one plane out, change it, send packets back gradually, validate or roll back) and Meta's 2025 post (four to eight planes doubled capacity). The argument syntax is our rendering of Richard's spoken, self-described “simplified version” (“data plane one comma data plane two comma … up to eight”). The 87.5% figure assumes equal shares per plane, which is how Meta's 2017 EBB post describes the planes (“each plane receives equal proportions of traffic”); it is our arithmetic, not a published number.
Our analysis — a blast-radius design is only as good as the one check that enforces it
Two more pieces of context matter later. First, Meta ran storm drills: regions isolated without notice by disabling routers and firewall rules, monthly or every ~45 days by 2022 (Richard), quarterly in the 2018 Maelstrom paper. Second, it had Defcon, a system of feature-degradation knobs built during the early-COVID demand spike and tested monthly, with target savings of 5% (L3), 10% (L2) and 20% (L1, for “unforeseen global system outages”). Neither had ever been used for a total loss of the backbone. Both mattered once the first plane came back, in the restoration phase.
The written record has three versions of the trigger, and they differ in ways that matter. Meta's first post, published on the night of the outage and updated the next day, blamed “configuration changes on the backbone routers” and “a faulty configuration change on our end”. That wording started the “bad BGP config push” story. The next day's post changed the description: during routine maintenance “a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network”, and “a bug in that audit tool prevented it from properly stopping the command.” A year later Richard gave the most specific account.
In his telling (which he calls “a simplified version”), the operator wanted to “check some level of capacity”. Instead of passing the tool one plane, as in --data-plane 1, the command listed every plane from one to eight. The safety check “was just expecting one data plane at a given point in time”. It did not find a plane matching the argument, and fell back to treating it as safe: “this is probably like a new data plane that never existed before”. The command went on. The operator “was expecting the command to … get some result back and to be able to interrupt it”. Instead, “in the space of a few seconds everything went dark.”
The fallback behaviour is Richard's description (“expecting one data plane”; “this is safe because this is probably like a new data plane that never existed before”), which he labels a simplified version. The flowchart shape is our reconstruction. Meta's written post says only that “a bug in that audit tool prevented it from properly stopping the command”. Neither source shows the code, names the tool, or says whether the check evaluated capacity at all on its normal path.
Our analysis — the check validated the input's shape, not the action's consequence
Our analysis — ‘assess’ and ‘act’ in one tool
The simultaneity is the point. Richard's phrase, “a few seconds”, means nothing else could intervene: no human, no alert, no rate limiter. The drain went through route injection designed to move traffic on and off planes quickly, because routine maintenance needs that speed. The mechanism that made one-plane operations cheap made an eight-plane operation instantaneous.
With the backbone drained, regions could not reach each other or the POPs. That was a total outage for anything needing a region, which is everything. But on its own it would have looked, from outside, like a very large number of HTTP errors. What made Facebook vanish from the internet was a second, deliberate design.
Facebook's authoritative DNS ran at the POPs, on well-known addresses announced to the internet by BGP. Janardhan explains the rule: “our DNS servers disable those BGP advertisements if they themselves can not speak to our data centers, since this is an indication of an unhealthy network connection.” Richard describes it the same way: each DNS server is “health checking … can I connect to the region behind me”, and if not, it stops advertising BGP for the DNS addresses. On 4 October every POP lost the region behind it at the same moment, so every POP withdrew. “The end result was that our DNS servers became unreachable even though they were still operational.”
The rule is quoted in Meta's 5 Oct post: DNS servers “disable those BGP advertisements if they themselves can not speak to our data centers”, and on 4 October “the entire backbone was removed from operation, making these locations declare themselves unhealthy”. The result, in Meta's words: “our DNS servers became unreachable even though they were still operational.” The fleet rows are our illustration of why a rule that is correct per site becomes an off-switch when the failure is shared.
Our analysis — a health check without a denominator
The pink line is the rule Meta described: each site withdraws on its own failed probe, so announcements fall one-for-one with failures. The gold and green lines are our alternatives, and the site count, failure rate and threshold are illustrative. Real site failures are positively correlated (shared transit, shared deploys), which fattens the tail, so set T from your own history of simultaneous failures rather than from a binomial. The trade-off is in the gold region between the two. A partial core failure that takes out more than T of sites now leaves some users routed to sites that cannot serve them.
Our analysis — what to ask of any health-gated withdrawal in a design review
The outside view shows how targeted this was. It was not “Facebook withdrew all its routes”. AS32934 normally originated 300-plus IPv4 and IPv6 prefixes. Cogent's table went from 309 at 12:00 to 259 at 16:00. RIPE's rrc10 collector in Milan saw 8 of 133 IPv4 and 14 of 216 IPv6 prefixes withdrawn. ThousandEyes, which listed the missing prefixes from Cogent's table, found that all but one covered nameservers. The service prefixes that carried users' actual traffic mostly stayed announced. Cloudflare noted other Facebook IPs “remained routed” but were useless without DNS.
Nameserver
Address
Specific routes
Covering route
Where the query died
a.ns.facebook.com
129.134.30.12
129.134.30.0/24, /23 · withdrawn
129.134.0.0/17 · still announced
Reached Facebook's edge; dropped there
b.ns.facebook.com
not given in sources
inside 129.134.30.0/23 and 129.134.31.0/24 · withdrawn
129.134.0.0/17 · still announced
Reached Facebook's edge; dropped there
c.ns.facebook.com
not given in sources
185.89.218.0/23 group · withdrawn
none
No route; dropped at the first ISP hop
d.ns.facebook.com
185.89.219.12 · 2a03:2880:f1fd:c:face:b00c:0:35
185.89.219.0/24 · withdrawn
none
No route; dropped at the first ISP hop
Seven IPv4 prefixes
covered DNS before the outage (ThousandEyes): the /17, two /23s and four /24s. All vanished at about 15:39 except the /17.
Cogent: 309 → 259
Facebook prefixes in Cogent's table at 12:00 vs 16:00 UTC. ThousandEyes lists 47 missing (26 IPv4, 21 IPv6) and says all but one covered nameservers.
rrc10 (Milan): 8 v4 + 14 v6
withdrawn around 15:40, out of 133 IPv4 and 216 IPv6 originated (Catchpoint). Roughly 6% of each family.
Addresses from Kentik (a) and MANRS (d); routes and drop points from ThousandEyes' traceroutes; counts from ThousandEyes and Catchpoint. The b and c addresses are not in any source we used, so we give only their route groups (c's group is our inference from the ThousandEyes list). The percentages are ours.
Our analysis — the withdrawal pattern is independent evidence for Meta's account
One detail in the ThousandEyes data is worth dwelling on. The covering prefix 129.134.0.0/17 stayed announced throughout, and it covered the addresses of nameservers a and b. So queries to those two did reach Facebook: traceroutes show them dropped at Facebook's edge routers, where there was no internal path to deliver them. Queries to c and d had no route at all and died at the first ISP hop. ThousandEyes offered two explanations: the /17 was never meant to carry DNS, or something inside the network was broken. Meta has never said.
Reconstruction (inferred) — why the /17 did not save anything
Meta has not published when the command ran. Richard says it was “early in the morning Pacific time”, which fits the external evidence: Kentik saw Facebook's traffic “virtually disappear” and a burst of BGP withdrawals at about 15:39 UTC, 08:39 in California. Everything else in the first twenty minutes comes from outside observers, and put together they give a fairly precise sequence.
Sources: Kentik (15:39 traffic and withdrawals), ThousandEyes (hundreds of withdrawals “almost all immediately reversed”, nameserver prefixes not re-advertised; app unreachable via DNS at 15:40), Cloudflare (15:40 peak, 15:50, 15:51, 15:58), Catchpoint (15:40 HTTP 503 alarms and their headers, rrc10 counts, TTL expiry “five minutes later”). Meta's time for the command is not published. The 15:58 marker is when Cloudflare identified the missing routes, not when they went: they had been gone since about 15:39. The 15:40–15:45 bar marks the window before record TTLs lapsed; 503s did not stop there. The 19-minute bracket is our reading of the diagnosis latency for a well-instrumented outside observer with every signal available to it.
At 15:40 Cloudflare saw a peak of BGP UPDATEs from AS32934, Catchpoint's HTTP checks against Facebook, WhatsApp, Instagram and Oculus began returning 503, and ThousandEyes saw the apps become unreachable because of DNS failure. The New York Times put it simply: “Within five minutes, Facebook had disappeared from the internet.” At about 15:45, according to Catchpoint, the TTLs on Facebook's cached records expired and every query for www.facebook.com returned SERVFAIL. By 15:50, 1.1.1.1 could no longer resolve facebook.com. At 15:51 Cloudflare opened an incident titled “Facebook DNS lookup returning SERVFAIL”, suspecting its own resolver at first. By 15:58 it had worked out that Facebook had stopped announcing routes to its DNS prefixes.
Our analysis — the 503 window is the most informative five minutes of the day
proxy-status: no_server_available; “Facebook's edge was unable to find an upstream proxy server”), and one sample is stamped 16:48 UTC, so clients still holding cached answers kept seeing 503s well after 15:45. That is the backbone outage by itself, before DNS masked it. Two things follow. First, the user-visible outage did not depend on DNS: when the cache ran out, the error just changed from 503 to SERVFAIL. Second, the diagnosis was available from outside by about 15:41 (edge up, core unreachable, DNS about to follow), yet Cloudflare, which had logged the 15:40 UPDATE peak and ran the resolver that was failing, needed until 15:58 to name the missing routes, after first suspecting its own resolver. Nineteen minutes is not slow; it is what happens when the loudest alert (SERVFAIL on your own product) points at the wrong layer. Cloudflare's summary says the same: “This wasn't a DNS issue itself, but failing DNS was the first symptom.” The lesson for monitoring design is to alert on the layer beneath the symptom. External checks that stop at “DNS fails” tell you less than an HTTP check that records who answered and with what.The load did not stay inside Facebook. Cloudflare reported DNS resolvers worldwide handling 30 times their usual query volume, as apps retried aggressively and users reloaded. Cloudflare's follow-up post put 1.1.1.1's SERVFAIL responses for facebook.com and whatsapp.com at about 60 times a normal day, and SERVFAILs over DoT and DoH at more than 300 times. Packet Clearing House saw a 40% global rise in “wayward” DNS traffic. Most 1.1.1.1 queries still resolved in under 10 ms; only the p95 and p99 tails grew, as resolvers waited on the well-known 10-second timeout for nameservers that could not be reached. Around 400 sites that embed Facebook scripts loaded 1.5 times slower, some more than twice as slow, and Catchpoint measured p95 document-complete time on its IR Top 100 sites rising by more than 20 seconds.
Record
TTL
Source
What it governed on 4 October
NS for facebook.com at the .com parent
172,800 s (48 h)
ThousandEyes
Resolvers always knew which four servers to ask. It also made re-delegating to another provider mid-incident impractical.
A / AAAA for d.ns.facebook.com
21,600 s (6 h)
MANRS dig output (AAAA 21,600; A 21,378 remaining when captured)
Resolvers kept the nameserver addresses. Those addresses had no route.
Records for www.facebook.com and friends
“short”; lapsed ≈5 min after first failures
Huston; Catchpoint (a 5-minute claim in a Krebs comment is unverified)
Set how fast users switched from HTTP 503 to SERVFAIL.
×4 upstream
A resolver that gets no answer tries every nameserver in the delegation. ThousandEyes: a single query “would have been quadrupled”.
30× at resolvers
Cloudflare: resolvers worldwide handled 30x more queries than usual, driven by apps retrying and users reloading.
≈60× SERVFAIL
1.1.1.1 served SERVFAIL for facebook.com and whatsapp.com at about 60x a typical day; DoT + DoH SERVFAIL grew more than 300x.
+40% globally
Packet Clearing House saw a 40% global increase in “wayward” DNS traffic (Bill Woodcock via Krebs).
Values as attributed. The point of putting the three TTLs side by side is ours: the only short TTL in the chain decided which error users saw and when, not whether they were served. The long TTLs guaranteed that every resolver on earth kept asking the right, unreachable, servers for six hours.
Our analysis — where the 30× comes from, and why TTLs don't explain it
Inside Meta, the first signals were IRC chatter and frozen terminals (Richard first suspected his own Wi-Fi); responders self-organised on a Zoom call that happened to be open. The first problem was seeing anything at all. Monitoring consoles, dev servers and internal tools were reached by names served by the same DNS that had just withdrawn itself. Engineers fell back on whatever they had cached locally (Richard mentions IP addresses and .ssh files) and on external signals such as Downdetector to judge the state of their own network. Reporters described the same thing from the other side: Workplace, internal calendars, security tooling and external email were down, and an internal memo quoted by the NYT said the global security team had been notified of an outage “affecting all Facebook internal systems and tools”.
Dependencies are from Richard (DNS used “to do out-of-band access” and “VPN to corporate”; OOB “never tested” with all regions down, dependencies had crept in; DC badges rely on the corporate network) and Meta's 5 Oct post (“the total loss of DNS broke many of the internal tools”). The right-hand column is from Richard's account of the response. The graph layout is our reconstruction. Note what the right column has in common: none of it runs on Meta's production network.
The second problem was the out-of-band network. Meta had one: a separate path to each region for exactly this situation. “But we never tested this in a way that all our region were down at the same time,” Richard said, “and over the years some dependencies had crept in and DNS was an issue.” Janardhan's post is blunter: “Our primary and out-of-band network access was down.”
Our analysis — an untested escape hatch accumulates dependencies at the rate of normal engineering
That left physical access. At some point in this phase, and Meta gives no time, the operator came forward: “it's probably me.” Richard credits Meta's blameless culture, and estimates the confession saved “probably three or four hours” of searching and competing hypotheses. The team now knew what to do (undrain the backbone) and roughly where (on routers in data centres that no one could reach remotely).
Getting there was slow for reasons that were each deliberate. Data centres are not near offices, and COVID had reduced on-site staff. People drove to their nearest sites while others built the plan remotely: which router, probably where, without the internal tools that normally show rack layouts. The DC badge system validated against the corporate network, which in each isolated data centre was cut off from the world, so Meta's security team ran a manual process to admit the right people. Once inside, as Janardhan wrote, “the hardware and routers are designed to be difficult to modify even when you have physical access”. Press reports of a team sent to the Santa Clara data centre for a “manual reset” came from an internal memo quoted by the NYT; Richard's account implies several sites, and Meta has never named one.
Bars and markers are from the cited observers; the 20:37 marker is Krebs's 4:37 pm ET update relaying Sheera Frenkel. The blue bar is deliberately vague: Meta has published no time for people reaching a DC, the first undrain, or per-region restoration, so its start and end are placeholders, bounded only by the fact that routes returned around 21:00.
Our analysis — the confession was worth more than any tool
Nobody outside Meta saw any of this, of course. From outside the day looked like this: at 17:02 ThousandEyes still recorded the outage as ongoing; at 4:37 pm ET (20:37 UTC) Krebs relayed Sheera Frenkel's report that employees' badges no longer worked; Facebook's CTO tweeted an apology for “networking issues”. At around 21:00 the DNS prefixes returned (Kentik), with BGP activity from AS32934 peaking at 21:17 (Cloudflare). facebook.com resolved again on 1.1.1.1 at 21:20, and at 21:28 Cloudflare reported Facebook “reconnected to the global Internet”.
The headline number, “about six hours”, hides more than it shows. Each observer measured a different thing, and the differences are informative.
Sources as labelled. The shaded band, from routes returning to broad reachability, is our construction: about 1h44m, a quarter of the ThousandEyes duration, spent entirely in the phase that Meta's storm drills had rehearsed.
Read the bars as a sequence rather than as disagreement. The health-gated DNS prefixes were withdrawn for about 5h18m to 5h21m (MANRS's BGPlay trace of 185.89.218.0/24; Kentik's view of the a.ns /23). Resolution on 1.1.1.1 was out for 5h30m, starting a few minutes later because of cached answers and ending a few minutes later because resolvers had to retry. Meta's own figure is “approximately 6 hours”. ThousandEyes, measuring whether most users could actually use Facebook, got 15:40 to 22:45, just over seven hours. The first ThousandEyes live update said DNS was fully restored around 21:30, while the post's body says DNS “started to come back online around 22:20”. We treat the live update, which agrees with Cloudflare and Kentik, as the routing fact, and the later time as reflecting reachability from ThousandEyes' own agents.
Our analysis — the restoration tail was a quarter of the outage
Scale. Kentik multiplied 3.5 billion users by at least 5.5 hours to get more than 1.2 trillion person-minutes of unavailability, a “1.2 tera-lapse”. (Our check: 3.5×10⁹ × 330 min ≈ 1.16×10¹², so reaching 1.2 trillion needs a little over 5.7 hours.) That is an upper bound on exposure, not a measure of harm: it counts every user as wanting Facebook for the whole outage. Downdetector logged 10.6 million problem reports, the largest it had ever seen. Facebook's share price fell 5.3% that afternoon, but on a day that also had the whistleblower testimony and a tech sell-off, so the outage's share of that fall cannot be separated. Meta's Q3 earnings call three weeks later did not mention the outage.
Planes. With eight equal planes, one drain removes 12.5% of EBB capacity, so routine maintenance implies peak DC-to-DC demand held at or below about 87.5% of capacity. Our arithmetic from there: a second plane out leaves 75%, and any peak above that means a two-plane drain already sheds traffic. A post-condition check would therefore have refused somewhere between plane 2 and plane 8, and on most days early. The 4→8 plane change adds a twist. Because it doubled capacity (Meta, 2025), each plane kept its size, and the routine unit of work fell from 25% to 12.5% of the network. The routine blast radius halved. The blast radius of the tool, one argument away from all planes, stayed at 100%. Growth in redundancy made the everyday operation safer and left the worst case untouched, which is why the fix in section 2's callout (evaluate remaining capacity, not argument syntax) is the cheap one: the capacity model it needs already existed implicitly, because the operating model depended on it.
Facts: eight planes in 2021 (Richard), equal traffic shares per plane (Meta's 2017 EBB post), one plane out as the routine operation. The demand lines are illustrative: Meta has published no utilisation figure. The point is structural. The whole guard is one comparison of two numbers the operating model already depends on, and under any plausible demand it refuses long before k = 8. It needs no knowledge of how the argument was spelled.
Our analysis — the check has to use the hottest link, not the average
Load-shedding capacity. The Defcon paper sizes failure domains at “around 5–12%” of overall capacity and describes 5–20% savings as the sweet spot. By our arithmetic, L1's 20% target covers losing roughly two to four such failure domains. The recovery on 4 October was a different shape: supply returned while demand spiked from reconnecting clients, with Kentik estimating “upwards of a billion” people likely reconnecting to WhatsApp at once. Meta used Defcon plus traffic dropping, meaning straightforward load shedding on top of graceful degradation. Our reading: Defcon's sizing was not enough by itself for a global cold start, which is why the second lever was needed. Meta's 2026 PowerLoss Storm post makes a related point about scale: recovery abilities hardened on single fault domains inside one DC showed new vulnerabilities at region scale (a typical region is 50–60 times a typical fault domain), including circular startup dependencies and orchestration signals that shut down the orchestrator's own control plane.
Facts: dips of “tens of megawatts” and the risk to “everything from electrical systems to caches” (Janardhan); fear of a power spike that could trip breakers, and Defcon plus traffic dropping applied as soon as one plane was up, with load spiking as clients retried (Richard); Defcon L1 targets 20% savings for “unforeseen global system outages” (Defcon paper). The curves are a schematic.
Our analysis — ‘tens of megawatts’ is a control-systems number, not a trivia item
Meta's posts defend two choices explicitly. The first is the security trade-off. Janardhan: “it was interesting to see how that hardening slowed us down”, and he still judged it worth it: “greatly increased day-to-day security vs. a slower recovery from a hopefully rare event like this.” The second is the staged restart: turning everything on at once “could potentially cause a new round of crashes due to a surge in traffic”, and storm experience gave “the confidence and experience to bring things back online and carefully manage the increasing loads.” Both hold up. The first deserves more scrutiny than it usually gets, in both directions.
Our analysis — the security trade-off is real, but it was not the one that cost the most time
Outsiders proposed several alternatives, some within hours. Geoff Huston (APNIC): don't put all authoritative DNS in one AS; beware short TTLs; and “separate infrastructure, separate routes, and separate DNS domains” for command and control. ThousandEyes pointed to Amazon, which hosts amazon.com's authoritative DNS at two outside providers, Dyn and UltraDNS. It also noted that a switch mid-incident was impossible in practice, because the NS records at .com had a 48-hour TTL. Catchpoint: badge systems “should never be dependent on the thing you're trying to fix.” Hacker News commenters suggested a “panic room” with a secure out-of-band connection. The useful exercise is to score each one against the phase of the outage it would actually have changed.
Alternative
User-visible outage
Collateral (resolvers, web)
Time to recover
Cost it introduces
Check the invariant: refuse any drain that leaves remaining capacity below demand
Prevented
n/a
n/a
Low: one more read of capacity state
Drain in stages: one plane at a time, pause, verify (make the command interruptible for real)
Cut to 1/8 of capacity before a human saw it
n/a
n/a
Slower routine ops
DNS health check with a fleet quorum (don't withdraw if most sites fail the same probe)
Still down: edge serves 503
Names still resolve; no resolver storm
Internal DNS and OOB names resolve
A partial core failure now blackholes some users
Secondary authoritative DNS at an outside provider
Still down: answers point at an edge with no core
Much less resolver load
Only helps if the tools it names are reachable
Two providers to keep consistent
Long TTLs on service records (Huston)
Still down: 503 instead of SERVFAIL
Lower
No change
Slower routine failover
Out-of-band network with no DNS or corp dependency, tested with every region down
No change
No change
Remote undrain; likely hours saved
Separate network to run and drill
DC badge systems that validate offline
No change
No change
Faster physical access
Security trade-off Meta chose not to make
Every cell is our judgement, derived from the mechanism in sections 3–5, not a Meta statement. The pattern it shows: the only rows that change the user-visible column are the two that stop the drain. Every DNS-side fix, including the ones most often recommended after the outage, changes collateral damage or recovery time, and none of them keeps Facebook up.
Counterfactual — keep DNS up and Facebook is still down
503. User-visible availability barely moves. What does move: the internet-wide 30× resolver storm mostly disappears, third-party pages embedding Facebook scripts fail fast instead of hanging, and (the one that matters most) Meta's internal names keep resolving. If the OOB path and corporate VPN had depended only on DNS, and not also on the backbone, operators could have reached routers remotely and cut hours from the recovery. So the DNS fixes are worth doing. Their value was operator time and collateral damage, not user uptime. Arguments that “Facebook should have used a second DNS provider” usually claim the latter, and that claim does not survive the mechanism.Counterfactual — the only rows that save the users
Huston's command-and-control point is the strongest of the outside suggestions, and the one Meta's later actions support most. “Separate infrastructure, separate routes, and separate DNS domains” is a description of what the OOB network was meant to be and had drifted from. Meta's response (always test OOB during storms, BellJar-style isolation tests) addresses the drift rather than building a second OOB network. That is probably the right economics at Meta's scale. The separation already existed on paper; what failed was keeping it true.
Our analysis — ‘don't put all your DNS in one AS’ protects against a different failure
Within hours the outage had a folklore. Checked against the sources:
“It was a DNS outage.”
DNS was the first external symptom. The DNS servers were “still operational” (Meta), and withdrew themselves because the backbone was gone. Cloudflare: “This wasn’t a DNS issue itself.” MANRS: “DNS and BGP behaved exactly as designed.”
“A bad BGP update took Facebook off the internet.”
The trigger was a backbone capacity command plus an audit-tool bug (Meta, Richard). The BGP withdrawals were health-check reactions. The “routine BGP update gone wrong” line came from a source relayed by Krebs and from deleted Reddit posts. Meta’s own first-night wording (“configuration changes on the backbone routers”) helped it spread.
“Facebook withdrew all its routes.”
It withdrew a targeted subset: dozens of 300-plus (Kentik), 8 v4 + 14 v6 at rrc10, 309 → 259 in Cogent’s table. What withdrew was mainly the nameserver prefixes. Internally the backbone was completely drained; externally most service prefixes stayed announced and were useless without DNS.
“Short TTLs caused it.”
Short record TTLs made the failure visible within minutes, and did not decide whether users were served (section 7). The long TTLs (48h NS at .com, about 6h on the nameserver address records) are what made an emergency re-delegation impossible.
“A second DNS provider would have prevented it.”
It would have cut resolver load and possibly helped operators reach internal systems. Users would have got HTTP 503 from an edge with no core behind it instead of SERVFAIL.
“Engineers had to use an angle grinder to get into the server room.”
No source we collected contains it; treat it as folklore. What is documented: office badges failed (NYT), DC badges failed because they validate against the corporate network, and the security team ran a manual admission process (Richard); “secure access protocols” took time (Meta).
“It was a hack, or a cover for the whistleblower news.”
Meta: “no malicious activity”. Cloudflare’s firewall saw “nothing out of the ordinary”. MANRS: “This Was Not An Attack.” The timing overlap with Frances Haugen’s interview and testimony is a coincidence, and the mechanism is fully explained without it.
“They should just have rolled back.”
The NYT noted companies “frequently roll back”. There was nothing to roll back through: the tools, the network paths and the out-of-band access needed to issue any command were the things that were down. The undrain happened on site.
“The fix was in Santa Clara.”
An internal memo quoted by the NYT sent a team to Santa Clara for a “manual reset”. Richard describes people driving to their nearest data centres, plural. Meta has never named a site.
Two of these do real damage. “It was DNS” sends teams to buy DNS redundancy for a failure that DNS redundancy doesn't cover. “A bad BGP push” points them at router config review, when the real lesson is about the tool that sits above the routers and the check inside it. Neither myth would have led anyone to ask the question that matters: what is the widest thing one command can do here?
It also matters what the public record looked like on the night. Meta's first post was four short paragraphs, which Huston called “a state-of-the-art example of a vacuous statement”. The detailed post came the next day, and the most technically specific account (the plane list, the fail-open check, the OOB dependency, the badge problem, the 3–4 hours the confession saved) came only a year later, in a conference talk. Most of the myths above grew in that gap.
Richard described the aftermath in unusual detail. The incident was reviewed within the week: “by Friday we had the first review”. Monitoring data was snapshotted before retention expired. Critical follow-ups were done “right now”, starting with the safety check that had failed on a list of planes, plus further defence in depth. Meta then froze “the entire network organization like for almost a month”, reviewed other automation that could cause a similar mass action (his examples: disabling all the switches, or mass power actions), and changed the storm process so that out-of-band access is always tested when a region is isolated.
Layer
State on 4 Oct 2021
After
Assumed one plane per call; unknown input treated as safe
Fixed as a critical follow-up, with more defence in depth (Richard)
Normal: “changing … 24 hours a day”
Network organisation code freeze for almost a month (Richard)
Not inventoried
Review of other automation that could disable switches or power en masse (Richard)
Region isolation; never a global backbone loss
OOB access always tested during storms; Meta to look for ways to simulate a global backbone outage (Meta, Richard)
Integration and unit tests
Services tested coming up under overload and missing dependencies in CI/CD; BellJar (2022) for recoverability
Roles being codified
Monthly “red siren” drills; formal incident- and crisis-manager roles (Richard)
Config updates the top trending cause of SEVs
“Safe change” initiative: targeted canaries plus regional health checks (Richard)
Not described
Instantaneous PowerLoss Storm (2026): de-energise whole regions, bootstrap them (Meta; does not cite 2021)
Not confirmed by any source we found: whether a storm simulating the loss of the whole backbone was ever run; whether the DNS health-check rule changed; whether OOB access lost its DNS dependency (as opposed to being tested for it); whether DC badge validation changed.
Longer-term changes widened the scope. Services are tested coming up under overload and with missing dependencies as part of CI/CD. Incident response was formalised, with defined incident-manager and crisis-manager roles and monthly “red siren” drills. Richard's six-monthly analysis of production issues found configuration updates to be the leading trending cause, followed by overload, unexpected dependencies, and complexity from overlapping migrations. That led to a “safe change” initiative: canaries targeted at the systems that consume a given config, plus regional health checks. In 2022 Meta published BellJar, a framework for testing recoverability by bringing systems up with only their declared dependencies. In 2026 it described the Instantaneous PowerLoss Storm, which de-energises entire regions and has them bootstrap themselves. Neither post names October 2021.
Our analysis — the code freeze is the least discussed and most revealing action
Our analysis — the drill that was promised, and what the record shows
What the record still does not tell us. The exact time of the command. When anyone reached a router, how many sites were involved, and when the first plane was undrained. The audit tool's code or name, and whether the operator meant to read or to drain (Richard says only that they expected results back and to be able to interrupt). Why the covering /17 stayed announced. Whether the DNS health-check rule was changed. Any cost figure from Meta. The SREcon transcript is auto-generated (it renders “edge” as “Hajj” and “crept in” as “Krypton”), so quotations from it here are checked against sense and kept short.
Almost nobody runs an eight-plane private backbone. Many teams run something with the same structure: N redundant domains, one control surface that can address all N, health logic that assumes some domain is always fine, and recovery tooling that runs on the thing it recovers. Examples include multi-AZ Kubernetes fleets with one cluster API, sharded databases with a fleet-wide admin tool, multi-region SaaS on one identity provider, CDN configs pushed through one pipeline, and anycast services with health-gated announcements. Each lesson below says where it stops applying. First, the pattern in one table: every defence on 4 October did its job at one-eighth and inverted at eight-eighths.
Defence
Assumed
At 1/8 (routine)
At 8/8 (4 Oct)
What holds at 8/8
Command safety check
one plane per invocation
blocks a bad single-plane request
unknown plane list treated as a “new plane”; allowed
post-condition on remaining capacity
Route-injection drain path
speed is always wanted
makes maintenance cheap
all eight planes dark in “a few seconds”
stage multi-plane actions; pause between
DNS health-gated withdrawal
some other site is healthy
users move to the next POP
every POP withdraws; names vanish
fleet denominator or fail-static route
Out-of-band network
DNS and some region reachable
exercised in regional storms
DNS dependency had crept in; down
isolation test with DNS removed
DC badge access
corporate network up
invisible
manual security process to admit staff
offline validation (no security cost)
Storm drills
one region at a time
rehearsed monthly or ~45 days
global backbone loss never simulated
regional drills that remove global dependencies
Defcon degradation
a few domains lost
L1 targets 20% savings
needed traffic dropping as a second lever
held, with the second lever
Blameless culture
people will say what they changed
practised on small incidents
“it's probably me”: 3–4 h saved (Richard)
held
Behaviour at 8/8 is sourced (Richard, Janardhan, the 2017 EBB post, the Defcon paper); the “assumed” and “what holds” columns are our analysis. The two rows that held are the two that were not software keyed to a failure domain: a load-shedding lever sized in percent, and a human habit. Every control that encoded “which domain” as an input failed once the answer became “all of them”.
1 · Guard destructive actions with post-conditions, not input validation
Applies when
Applies to every tool that can remove capacity from a redundant system: drains, cordons, scale-downs, failovers, deletes. The last check evaluates what will remain against what is needed, from live state, and refuses otherwise, whatever the arguments looked like.
Weaker / not when
Weaker where live capacity or demand cannot be computed reliably. Then fall back to hard caps (never more than one domain per invocation) enforced in the tool, not in documentation.
Pitfall in copying
Adding more argument parsing checks. Each one is another proxy with a corner case. The failure on 4 October was a proxy being wrong.
2 · Make multi-domain actions sequential and genuinely interruptible
Applies when
Applies when an action can target more than one failure domain. Execute one domain, pause, show the effect, require continuation. This caps the blast radius of any bug in any check to one domain.
Weaker / not when
Weaker in emergencies that genuinely need all domains at once (evacuating a compromised system). Give that a separate, loudly named, two-person path.
Pitfall in copying
A “--yes” or batch flag that becomes the normal way to run it within a month.
3 · Give health-gated withdrawal a fleet denominator
Applies when
Applies to anycast announcements, load-balancer pools, service discovery and any system where members remove themselves on a failed probe. If a large share fail the same probe at once, stop withdrawing: the probe is measuring a shared dependency.
Weaker / not when
Weaker when serving from an unhealthy member is actively harmful (it corrupts data, or serves wrong answers). Then withdrawing everything is correct, and you need the guarantee from lesson 4 instead.
Pitfall in copying
Setting the threshold without testing it. It is another rarely used code path (see Cloudflare 2025 in this series).
4 · Enforce the escape hatch's dependency set by isolation testing
Applies when
Applies to OOB networks, break-glass accounts, bastions, emergency runbooks and badge systems. Regularly bring them up with everything else removed: no production DNS, no SSO, no corporate network. What fails is a dependency that crept in.
Weaker / not when
Weaker only if the hatch is physically separate and its dependencies are simple enough to audit by reading. Rare, and it rarely stays that way.
Pitfall in copying
Testing the hatch in a scenario where the thing it must survive is still up. Meta’s regional storms did exactly this with DNS.
5 · Plan the restart as its own incident
Applies when
Applies to anything that can fail by going quiet at scale: caches, connection pools, power draw, client retry storms. Have load-shedding and degradation controls you can turn down before the network comes back, and rehearse ramping.
Weaker / not when
Weaker for small or stateless systems where a cold start is cheap. Measure that rather than assume it.
Pitfall in copying
Degradation knobs that are never exercised. Richard: if you don’t validate them repeatedly, “the day you need them” they won’t give the savings you expect.
6 · Treat the people channel as a recovery control
Applies when
Applies everywhere. When tooling is down, the fastest path to the root cause is whoever just changed something. Blameless review is what makes them say so in minutes rather than after the postmortem.
Weaker / not when
Not weaker anywhere we can think of. It needs practice on small incidents to be available in a large one.
Pitfall in copying
Announcing blamelessness while punishing the operator of the last big incident. Everyone reads the second signal.
A decision procedure for your own architecture
What not to copy
Related material: the SRE / Reliability track (failure domains, change safety, incident command), the Distributed Systems track (metastable failure and retry storms), and the Cloudflare 2019, CrowdStrike 2024 and Roblox 2021 studies in this series. Cloudflare and CrowdStrike are also cases of a fast global path with no floor, and Roblox is another case of recovery tooling that depended on the failing system.
One sentence to carry out of this case: redundancy protects you from independent failures, and every tool, health check and escape hatch you build on top of it quietly assumes the failures stay independent. Test what happens when they don't.
Meta's primary accounts first, then Meta background papers and posts, then outside measurement, then reporting. The SREcon22 talk is quoted from an auto-generated transcript; quotations are short and checked for sense. All arithmetic, reconstructions and counterfactuals are ours and are labelled as such in the text.