At 13:42 UTC on 2 July 2019, Cloudflare pushed one new Web Application Firewall rule to its edge. The rule was set to “simulate”, so it would log matches and block nothing. It travelled through Quicksilver, Cloudflare's config store, which was replicated to more than 180 cities and delivered changes to every machine with a p99 of 2.29 seconds. Within three minutes the first page fired, and CPU on every core that served HTTP was pinned at around 100%. A Cloudflare leader told the CTO the company had “lost 80% of our traffic”. The usual retelling is “a bad regex caused a ReDoS”. That is true, and it misses most of what an architect should take from the incident. The rule was harmless in every respect the pipeline checked, and ruinous in the one respect it didn't: cost. Every safeguard that existed was built to catch wrong answers. None of the four that could have bounded an expensive answer was in place that afternoon.
27 min
impact, 13:42–14:09 UTC
~80–82%
traffic lost at peak
p99 2.29 s
rule change to every machine
3,868
managed rules, all re-inspected by hand
9 / 9
published step counts our model reproduces
L³
growth of the failing search (ours)
476
rule changes in the prior 60 days
45 min
managed WAF off worldwide, 14:07–14:52
What this adds to Cloudflare's own account: a cost model that reproduces all nine published step counts, timings of the rule as printed on backtracking and linear-time engines, an exposure analysis of why every gate passed, the attribution and saturation arithmetic, and the 2025 recurrence that shows what the 2019 fix did not cover. Analysis sits in labelled callouts. Where the record is silent (which fields the rule inspected, what the removed guard was, whether PCRE's JIT was on) we say so.
By 2019, Cloudflare's WAF had been running in the request path for years. The 2013 design post describes the pipeline. ModSecurity rules and Cloudflare's own rule syntax were compiled to a common JSON intermediate form and then to Lua. That Lua ran inside nginx through access_by_lua, calling helpers such as waf_regex. The 2021 post that replaced it calls the old WAF LuaJIT code implemented as an NGINX module. The 2019 postmortem adds the property that matters: “The Lua WAF uses PCRE internally, and it uses backtracking for matching and has no mechanism to protect against a runaway expression.”
It also says where the WAF ran: on “every CPU core that handles HTTP/HTTPS traffic on the Cloudflare network worldwide”. There was no separate pool of cores for rule evaluation and no sidecar. The code that decided whether to block a request ran on the same worker, and the same core, as the code that proxied it.
Components are from Cloudflare's 2013 WAF post (rules compiled to Lua, run in nginx via access_by_lua) and the 2019 postmortem (the WAF runs on “every CPU core that handles HTTP/HTTPS traffic”; rules delivered by Quicksilver). The per-core layout is our reconstruction. The point it makes is sourced: there was no separate pool of cores for rule evaluation, so the WAF's worst case was also the proxy's worst case.
The 2013 post also gives a performance baseline that is rarely quoted. In production the WAF took 380–480 µs per request, against a goal of under 1 ms, and regex processing and string matching were the most expensive operations. So regex CPU was already the largest line item in a per-request budget of well under a millisecond. The 2019 postmortem explains why the CPU guard was missing: it was removed “by mistake” during a refactor “that was part of making the WAF use less CPU”.
Our analysis — guards are invisible to the benchmark that removes them
Two more facts: the managed rule set held 3,868 rules and saw 476 change requests in the previous 60 days (“one every 3 hours”), and Cloudflare had not had a global outage in six years. It also fronts its internal tools with its own Access product, which matters in section 6.
At 13:31 an engineer merged a new WAF Managed Rule aimed at XSS (inline JavaScript, per the same-day post), to ship in “simulate” mode, “where real customer traffic passes through the rule but nothing is blocked”. CI passed at 13:37; at 13:42 Quicksilver took it to the whole edge. By every gate the pipeline had, this was as low-risk as a WAF change gets: new rather than modified, so nothing could regress; non-blocking, so no false positive could hurt a user; green in CI.
Layer
What it defends against
State on 2 July
Obvious logic errors
Passed. The PR was approved and merged at 13:31
Rules that fail to compile or mis-match
Passed at 13:37. The suite had no way to detect excessive CPU use
False positives blocking real users
On. Stops the block, not the evaluation: the regex still ran on live traffic
A runaway expression
Absent. “Removed by mistake” in a refactor weeks earlier
Super-linear matching
None. PCRE backtracks; “no mechanism to protect against a runaway expression”
Anything visible on real traffic
Not used. WAF rules go straight to Quicksilver, globally
Fast detection
Fired, but too slowly
Undoing the change
Needed two full WAF builds. Too slow to use
Removing a component everywhere
Worked: run at 14:07, traffic normal at 14:09
States from Cloudflare's 12 July postmortem. Colour key (ours): pink = absent; gold = present but aimed at a different failure; grey = present and passed; green = worked. Only one of the four layers that could bound CPU cost (the protection, the engine, a staged rollout, CI profiling) would have had to exist. None did. Every layer that did exist was built to catch wrong answers, not expensive answers.
Our analysis — the exposure vector nobody wrote down
Our analysis — shadow modes protect decisions, not resources
Of the postmortem's 11 “what went wrong” items, only the first is about the regex author; the other ten are about the system around them (section 9). It was “not an attack (as some have speculated)”, though the team's own first hypothesis was one, and it was only discarded at 14:00 (section 6).
The full rule, as printed in the postmortem:
(?:(?:\"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|\`|\-|\+)+[)]*;?((?:\s|-|~|!|{}|\|\||\+)*.*(?:.*=.*)))
(?:(?:\"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|\`|\-|\+)+
① a run of “JavaScript-ish” tokens: quotes, ], }, \, digits, literals, backtick, -, +. One or more.
[)]*;?
② optional closing parens and one optional semicolon
((?:\s|-|~|!|{}|\|\||\+)*
③ optional whitespace / operators (-, ~, !, {}, ||, +). Zero or more.
.*
④ anything
(?:.*=.*)))
⑤ anything, an '=', anything
Search, not match
No ^ anchor. The engine retries the whole pattern at every start offset, so any per-attempt cost is multiplied by input length.
④ + ⑤ = .*.*=.*
Two unbounded wildcards before one required literal. Every way of dividing the text between them is a separate attempt.
① is quantified too
On a run of digits or quotes, ① can end at any of those characters, and each ending re-runs ③–⑤. That adds one more factor of n.
Cloudflare singled out .*(?:.*=.*) as the dangerous part. The two lower-left cards follow from standard backtracking semantics. The third card is our analysis, and our timings in section 4 bear it out: on digit-heavy input the full rule grows faster than the reduced pattern Cloudflare used to explain it.
Cloudflare reduced the dangerous part, .*(?:.*=.*), to .*.*=.* and published step counts. x=x takes 23 steps, x=xx 33, x=xxx 45, and x= followed by 20 x's 555. With the x= removed, so the input is just 20 x's, the engine takes 4,067 steps to conclude there is no match. Adding a trailing ; to the pattern (so it fails on these inputs) gives 90 steps for x=x and 5,353 for x= plus 20 x's. The lazy form .*?.*?=.*? matches both x=x and x= plus 20 x's in 11 steps.
We went further than the postmortem's hand walk-through: a naive backtracking matcher (greedy or lazy .*, literals, unanchored search), with small integer cost weights searched until its step counter matched. One assignment matches all nine published numbers exactly:
Pattern
Input
Outcome
Cloudflare (published)
Our model
.*.*=.*
x=x
match
23
23
.*.*=.*
x=xx
match
33
33
.*.*=.*
x=xxx
match
45
45
.*.*=.*
x= + 20 x
match
555
555
.*.*=.*
20 x (no =)
no match
4,067
4,067
.*.*=.*;
x=x
no match
90
90
.*.*=.*;
x= + 20 x
no match
5,353
5,353
.*?.*?=.*?
x=x
match
11
11
.*?.*?=.*?
x= + 20 x
match
11
11
Cost model (ours)
1 step per start offset tried, plus 2 steps each time a token is attempted (entering a .* counts as one attempt, and so does each try of the literal). Backtracking itself is free: it shows up as more attempts.
Closed form, match
“x=” + n x’s against .*.*=.*: n² + 7n + 15. n=20 → 555. Quadratic.
Closed form, no match
L x’s against .*.*=.*: (L³ + 9L² + 29L + 21) / 3. L=20 → 4,067. Cubic.
Published counts from Cloudflare's postmortem appendix. Model and closed forms are our reconstruction. A few integer weights fitting nine numbers across greedy/lazy and match/fail inputs is strong evidence the model captures what the counter measured.
With a model that reproduces the published data, the closed forms follow directly, and they settle the question most retellings get wrong. For a successful match on x= plus n x's, cost is n² + 7n + 15: quadratic. You can check it against Cloudflare's own numbers: n = 1, 2, 3 and 20 give 23, 33, 45 and 555. For a failed search on L characters with no =, cost is (L³ + 9L² + 29L + 21) / 3: cubic, with leading term L³/3.
Each pink cell is one attempt at the literal = after the wildcards split the text at (i, j). With no =, every cell fails. Greedy visits from the bottom-right, lazy from the top-left. Both visit every cell when the answer is “no match”. The count is our derivation; section 3 shows it reproduces Cloudflare's published step counts exactly.
The lattice shows why. At one start offset, the first .* can stop at any of L+1 positions i, the second at any position j ≥ i, and at each (i, j) the engine tries the literal =. That gives (L+1)(L+2)/2 attempts, a triangle. The search is unanchored, so when the triangle fails the engine moves the start offset forward by one and does a slightly smaller triangle, all the way to the end. Summing triangles gives a tetrahedral number, C(L+3, 3) ≈ L³/6 attempts at the literal. At 2 steps per attempt that is the L³/3 leading term. For L = 20 the sum is 1,771 attempts, which is 3,542 steps, plus 525 steps for entering the wildcards and advancing the start. Total: 4,067.
Reconstruction (inferred) — why a WAF lives on the failing path
Two more observations fall out of the model.
Lazy quantifiers do not help on the hot path. The postmortem notes that “Using lazy rather than greedy matches helps control the amount of backtracking that occurs in this case”, and the 11-step figures show it. Our model shows the limit of that advice. The lazy simulator gives exactly the same count as greedy on every failing input: 4,067 for 20 x's and 5,353 for the ; variant. That follows from the lattice. Laziness changes the order in which cells are visited, not how many there are, and a failing search visits them all. Guidance to authors to “prefer lazy” is fine for match latency and useless for WAF CPU.
The full rule is worse than the reduced one. Segment ① in the anatomy diagram is (?: … |\d| … )+, a quantified alternation of single characters that includes digits, quotes and hyphens. On a run of such characters, ① can stop after any of them, and each stopping point re-runs everything to its right. Our timings in section 4 show the full rule on pure digits growing at close to L⁴, one degree steeper than the reduced pattern. The postmortem does not say this. It follows from the pattern's structure, and our measurements support it.
Our analysis — ‘catastrophic’ does not mean ‘exponential’
(a+)+, or Russ Cox's a?ⁿaⁿ, where Perl takes over 60 seconds at n = 29 and a Thompson NFA takes 20 µs. .*.*=.* is polynomial: quadratic on success and cubic on failure. The full rule reaches roughly quartic on digit runs. This matters for two reasons. First, the usual ReDoS linters and code-review heuristics look for nested quantifiers, and this pattern has none, so a reviewer hunting for exponential shapes would have passed it. Second, polynomial is enough. Degree 3 in a single rule, evaluated on every request at global volume, was fatal. The safe property is not “avoid exponential patterns”. It is linear in the input, guaranteed by the engine.Step counts are abstract. To turn them into CPU time we ran the full rule, as printed, and the reduced patterns through V8's backtracking engine (irregexp, Node 24.13) on inputs that never match. We then ran the full rule through V8's experimental linear-time engine for comparison. These are our measurements, taken once per point on a laptop, and V8 is not PCRE. Treat the absolute numbers as illustrative. What transfers is the slopes.
Illustrative, our measurement: Node 24.13 (V8's backtracking irregexp; the green line uses V8's experimental linear-time engine), single run per point on a laptop, inputs that never match. V8 is not PCRE. Cloudflare has not published PCRE settings, whether JIT was on, or which request fields the rule saw. Absolute times will differ; the slopes won't, because they come from the search space and not from the engine's constant factor. Budget lines are from Cloudflare's 2013 WAF post.
Four readings from the chart:
The model's constant is stable. Dividing our cubic step formula into the measured .*.*=.* times gives about 0.44 ns per model step at both L = 1,000 (≈336M steps, 150 ms) and L = 8,000 (≈171B steps, 74.5 s). A constant cost across a 500× range is independent evidence that the cubic model describes the real search.
The per-request budget is crossed at ordinary lengths. Using the 2013 WAF's 1 ms per-request goal as a yardstick (it covered all rules, not one), the reduced pattern crosses 1 ms at about 190 characters of =-free input. The full rule with a hyphen every eight characters crosses it at about 370. On a run of digits it crosses at about 57. A string of that length is unremarkable in a query string, a cookie or a header. We don't know which fields the rule inspected, and Cloudflare didn't say. The chart doesn't need that: in any field of a few hundred characters, one rule could consume the whole budget of the entire rule set.
The worst inputs are boring. The full rule barely runs on plain x's, because segment ① needs a quote, digit, bracket or operator. It is worst on digits, quotes and hyphens: timestamps, IDs, JSON, URL-encoded data. The expensive inputs are dull legitimate ones, not exotic payloads.
A linear engine changes the order, not just the constant. On 300 digits the backtracking engine took 443 ms and the linear engine 0.10 ms, a ratio of about 4,400. At 64,000 digits the linear engine takes 18 ms. We didn't run the backtracking engine there. Extrapolating its measured slope (our arithmetic), it would take months at L³ and years at the steeper ~L⁴ we measured on digits. The linear engine is not free either: 18 ms per 64 KB field is a lot for one rule, so a per-request CPU budget is still needed alongside it.
Our analysis — what the test suite would have needed
Software went through DOG (employees), PIG (a slice of free traffic) and three Canary PoPs, a process that “can take hours or days”. WAF rules went from CI straight into Quicksilver, because they “are required to address emergent threats”, and Cloudflare is blunt about the consequence: “That speed meant that a change to the rules went global in seconds.” The SOP let non-emergency rule changes use the same lane; 2 July was not an emergency.
Software release · “hours or days”
Git (BitBucket)
TeamCity build
PR review
Jira change request
manager / tech lead
DOG
employees only
PIG
some free traffic
Canary
3 PoPs, paid + free
Global
WAF managed rule · seconds
PR merged
13:31
TeamCity build + tests
13:37 · passed
Quicksilver write
13:42
no DOG · no PIG · no Canary
Every machine
p99 2.29 s
Why the fast lane exists
“react as quickly as possible to new attacks to give our customers a chance to patch their software.”
What the SOP permitted
Non-emergency rule changes could go global without staged rollout. 2 July was not an emergency.
What “simulate” changed
Whether a match blocked. Not whether the regex ran.
Both pipelines are described in Cloudflare's 12 July postmortem. The asymmetry is the same one that recurs in the CrowdStrike case study in this series: the release process was attached to the artifact type (code vs rules), while the risk was attached to the consumer (the CPU on every HTTP core).
Quicksilver (described in 2020) replaced Kyoto Tycoon, whose write locks blocked reads (read P99 from 9 ms to 154 ms under writes), whose corruption needed SRE hand-repair and whose timestamp replication dropped entries. Every one of those complaints is about correctness and latency of delivery; nothing in the design brief is about what is delivered. LMDB, a fan-out tree, sequence numbers and a 500 ms batch window got it to ~350 changes/s at a p99 of 2.29 s to every machine.
Tree structure and scale from “Introducing Quicksilver” (2020); p99 2.29 s and ~350 changes/s from the 2019 postmortem; event times from the postmortem timeline. The per-day comparison, the 1-in-3.8M ratio and the ~80× gap between propagation (2.29 s) and first page (~180 s) are our arithmetic. The shaded band is the argument for staging: when the channel is faster than detection, containment has to happen before detection.
Our analysis — the pipe was designed around volume, which rules out staging per write
The same speed ended the incident: global terminate restored traffic within two minutes. How terminate is implemented isn't published; if it rides the same distribution system (our inference), then any proposal to slow config distribution must keep a fast path for switching things off.
Times are from Cloudflare's 12 July postmortem except 13:52 (The Register's same-day report). The same-day Cloudflare post gives 14:02 for root cause and 14:09 for termination; we use the later, more detailed account. The Register says services were restored by 14:57; that looks like a status-page time (our inference) and does not match the postmortem. Phase labels and the two durations in the footer are our arithmetic.
The first page (13:45) was a failing synthetic WAF test, followed by end-to-end failures, a traffic drop and 502s everywhere. The team first suspected an attack; at 14:00 CPU data pointed to the WAF and strace confirmed it.
The five minutes between the proposal and the execution are the incident's most instructive detail. Cloudflare staff reach internal tools through Cloudflare Access, and Access runs on the edge that was down. The postmortem: “We use our own products and with our Access service down we couldn't authenticate to our internal control panel.” Some SREs' credentials had also timed out for security reasons, and the team fell back on a bypass procedure it rarely used.
Each dependency is stated in the postmortem: “with our Access service down we couldn't authenticate to our internal control panel”; some SRE credentials had timed out for security reasons; the bypass procedure was rarely used; customer dashboard and API traffic went through the edge. The drawing, and the assumption that these obstacles fell inside 14:02–14:07, are ours. The postmortem does not break the five minutes down.
Our analysis — the decomposition that matters
Our analysis — the attribution arithmetic that was available at 13:45
Communication lagged too. The Register reports the status page acknowledged issues at 13:52; Cloudflare's own list says it was not updated quickly enough, and customers could not reach the dashboard or API, because both run through the edge. Prince tweeted that a “Massive spike in CPU usage caused primary and backup systems to fall over” (as quoted by The Register). The detailed postmortem followed ten days later; it traces its candour to a 2010 exchange in which Graham-Cumming, then a customer, asked Prince for openness about an outage.
Reconstruction (inferred) — why terminate beat rollback
How much traffic. The postmortem quotes “we had lost 80% of our traffic”; the same-day post gives 82% at peak. Neither says why a fifth survived, and we won't guess. It is not evidence of selectivity: cost depended on input shape and saturation on each core's headroom, neither published.
How long. Impact ran 27 minutes by the postmortem's timeline, from 13:42 to 14:09. The same-day post says “about 30 minutes” and The Register “almost half an hour”. The managed WAF then stayed off worldwide for 45 minutes, from 14:07 to 14:52. The Register's 14:57 “restored” matches neither Cloudflare time; it most likely reflects the status page (our inference).
Illustrative numbers, not Cloudflare's. Model: a core serving r requests/s with spare CPU fraction H saturates when p · r · c > H. The chart fixes r = 1,000 req/s per core (an assumption), so the boundary is p = H / c with c in ms. The orange points place two of our V8 timings from section 4 on it. The shape is what transfers. Under a super-linear matcher, the slow path doesn't need to be common. It only needs to be common relative to how expensive it is, and on long inputs its cost grows much faster than its frequency falls.
How rare the slow input can be. The saturation chart is a model with an assumed request rate, not Cloudflare data. Its shape shows why a rule does not need to be slow on most requests to take a core down. Saturation depends on the product of frequency and cost. Under a super-linear matcher, cost grows so steeply with length that a small share of long requests is enough. In our V8 numbers, a 300-digit field costs 443 ms, and at 50% headroom on the assumed load about one request in a thousand would be enough. On shared-worker proxies there is a second effect the chart leaves out, which is our inference from the nginx/Lua architecture. A request stuck in a long Lua regex call holds its worker's event loop, so every other connection multiplexed on that worker waits too. The damage per slow request is its CPU time plus head-of-line blocking for everything that shares the worker.
How fast, compared. Propagation p99 was 2.29 s; the first page came about 3 minutes after deploy, roughly 80× the propagation time (our arithmetic). The blast radius was complete long before anyone knew. Where p99 propagation is shorter than detection, containment has to happen before detection, by staging.
How much staging costs. The postmortem's change rate, 476 in 60 days, is about 8 rule changes a day, one every 3 hours. If every rule change took the software path (“hours or days”), then by Little's Law and an illustrative 24-hour rollout, about 8 rule changes would be in flight at any moment. That means several overlapping changes in the three canary PoPs, and difficulty telling which one caused a signal. That arithmetic, more than any institutional reflex, is the honest argument against putting rules on the software train. The honest counter-argument is that the failure here was visible within seconds on any real traffic. A single-PoP stage with a one-minute bake, at 8 changes a day, costs about 8 minutes of bake time a day and would have kept the incident to one PoP.
Our analysis — why ‘primary and backup systems fell over’, and why the loss was a cliff
Counterfactual — the smallest change that turns a global outage into a local one
Actual curve from the postmortem timeline (global at p99 2.29 s, terminate 14:07, normal 14:09). Policies A and B are our counterfactuals with illustrative stage sizes. “Fleet-%·min” is our unit: share of HTTP cores running the rule × minutes. It is a proxy for damage, since on 2 July a core running the rule was a saturated core.
Our analysis — canary sizing: catch the saturated core, not the rare input
Our analysis — the kill switch has its own blast radius
Scope
Earliest usable on 2 July
Collateral (security gap)
Prerequisite
Cause narrowed to “the WAF”: 14:00
All 3,868 managed rules, every customer, 45 min
Existed and worked
Needs only the change log: available at 13:45
Likely one rule: at ~8 changes/day a short window rarely holds two (our arithmetic)
Per-rule off switch + change-indexed selector. Unpublished whether it existed
Needs attribution to the rule: after 14:00
One rule
Per-rule switch; profiling data to name the rule
Immediate
Nothing useful: every PoP was failing
Only helps when the failure is local
Immediate
Nothing useful: the rule ran for all zones
Only helps when the trigger is one tenant's traffic
Times from the postmortem; the ladder and its assessments are ours. The useful property of the “recent changes” scope is that it is selected by time, not by diagnosis, so it can be pulled at first page. Its failure mode is that it is wrong when the cause is not a recent change (an attack, a traffic shift), which makes it a cheap first move, not a conclusion. The two pink rows are the switches most fleets build first, and they do nothing against a fault delivered to every node at once.
The postmortem's rationale for the fast lane is compact and hard to argue with: “it's imperative that Cloudflare are able to react as quickly as possible to new attacks to give our customers a chance to patch their software.” A WAF rule that ships a day after a mass-exploitation campaign starts has missed its purpose. So the question was never “fast or slow”. It was which properties of a rule change had to be checked before it was allowed to be fast. In 2019 the pipeline checked correctness (CI) and blocking risk (simulate). It did not check cost.
Option
Would it have stopped 2 July?
What class does it cover?
Cost
Yes, if set below the per-request budget
All future runaway patterns in this engine
Near zero. The guard existed before the refactor
Yes: the blow-up cannot happen
The whole class, by construction
No backreferences or look-around; 3,868 rules to check and port
Probably, if the corpus had long, digit- or quote-heavy fields
Patterns the corpus can trigger
Corpus design; CI minutes
Yes: saturation is visible in seconds on real traffic
Anything visible on live traffic
Seconds to minutes per rule; ~8 rules/day
No, for the failing inputs that dominate WAF traffic
Some successful matches
Free, and misleading
Would have bounded proxy impact
Any WAF cost blow-up
Capacity, cross-core latency; a redesign
No. Shortens recovery only
Every bad rule push
Artifact retention; tested restore path
Options are drawn from Cloudflare's remediation list, plus two alternatives it did not list (a separate CPU pool, a prebuilt last-known-good). Assessments are ours. Green = would have prevented; gold = would have bounded or shortened; pink = would not have helped. Cloudflare chose all three green rows plus CI profiling at once: the CPU guard restored, the engine replaced, staging for rules, profiling added. That is defence in depth done properly: each layer gets its truth from a different place.
Cloudflare's remediation took the three green rows and one gold at once: guard restored and all 3,868 rules hand-inspected (done at publication), CI profiling (ETA 19 July 2019), “re2 or Rust regex” (ETA 31 July 2019), staged rollout for rules with an emergency override; plus undated items (dashboard/API off the edge in an emergency, an automated status page) and a long-term move off Lua.
Our analysis — why the engine swap is the load-bearing fix
regex crate documents worst-case O(m · n) for pattern size m and input size n. Both leave out backreferences and look-around, and that is the real cost of the migration. Every one of 3,868 rules had to either compile under the new engine or be rewritten. The postmortem's appendix frames the choice historically: “The solution to this problem has been known since 1968”, citing Ken Thompson's NFA construction. The restored CPU guard bounds each evaluation. Profiling catches patterns the corpus can trigger. Staging catches what live traffic can trigger. The engine removes the super-linear class altogether. These are independent layers because each gets its truth from a different source: runtime state, a test corpus, production traffic, and a theorem.Counterfactual — if only the guard had been there
Rule type
Cost of failing open
Cost of failing closed
Default (ours)
Simulate / log-only rule
Record “budget exceeded”, return no-match
Nothing: the rule blocks nothing either way
Fail open. Alert on fire rate; auto-demote the rule above a threshold
Blocking signature, broad attack class (XSS, SQLi)
Other rules and origin defences still apply
A benign request that happens to be long gets blocked: a self-inflicted false positive at scale
Fail open + counter. An attacker who can trigger the budget can bypass this one rule, so track fire rate per rule
Virtual patch for an actively exploited CVE
The exploit may pass: the reason the rule exists
Legitimate long requests blocked while the patch is live
Fail closed, time-boxed, with fire rate as a promotion blocker. The only row where closed is defensible
Cloudflare did not publish how its restored guard behaves when it fires. This matrix is our analysis. The point that is easy to miss: a guard turns a CPU problem into a correctness problem. The fire rate per rule is therefore a first-class health signal, both as an early warning that a pattern is super-linear on live inputs and as the size of the bypass an attacker gets by padding requests.
Our analysis — why not isolate rule evaluation from the proxy?
The incident was retold so often that several claims became received wisdom. Checked against the sources and our model:
“It was an exponential ReDoS.”
The published counts fit a quadratic exactly on success and a cubic exactly on failure. The pattern has no nested quantifier. Polynomial growth at global scale was enough, and ReDoS linters that look for exponential shapes would not have flagged it.
“Lazy quantifiers would have saved them.”
Lazy cut the successful match to 11 steps. On failing inputs, which are nearly all WAF traffic, our model gives identical counts for lazy and greedy (4,067 and 5,353). Laziness changes the search order, not the size of the search.
“Simulate mode made the rule harmless.”
Simulate mode stops blocking, not execution. Real traffic passed through the rule and the regex ran in full. It protected against false positives, the one failure that didn’t happen.
“One engineer made a mistake.”
The regex is item 1 of 11 in Cloudflare’s own list. The others are a removed guard, an engine without guarantees, CPU-blind tests, a permissive SOP, a slow rollback, slow alerting, a slow status page, internal tools behind the failing edge, expired credentials, and a dashboard and API behind the edge.
“They rolled back the bad deploy.”
Early messaging said a “bad software deploy” was rolled back. It was a rule pushed through Quicksilver, and the postmortem shows the WAF was switched off globally at 14:07, because rollback needed two full builds. The specific rules were rolled back afterwards, and the WAF came back at 14:52.
“It was an attack.”
Cloudflare: “not an attack (as some have speculated)”. Its own team initially suspected one, because global CPU saturation looks like one.
“All of Cloudflare was down, DNS included.”
Prince’s tweet said “Impacted all services”, and The Register listed DNS as affected. The postmortem names core proxying, CDN and WAF, and doesn’t name DNS, 1.1.1.1, Workers or Spectrum. Treat DNS impact as not confirmed by Cloudflare.
“Switching to Rust regex closed the book.”
It closed the regex class: no repeat has been publicly reported. The delivery class (instant, unstaged global config) caused outages in November and December 2025, and Cloudflare’s December 2025 post says its global config system still “does not perform gradual rollouts”.
“Code Orange 2023 was a repeat of 2019.”
November 2023 was a data-centre power failure that took down the control plane and analytics. The edge was unaffected. The “Code Orange once before” in the 2025 plan refers to 2023, not 2019.
Two of these myths do real damage. The exponential one produced linters for the wrong shape. The “one engineer” one points remediation at authors, yet none of Cloudflare's remediations is “write better regexes”: apart from a one-off audit, every item changes the system so the next bad regex costs nothing. The Hacker News thread (698 points, 149 comments) got closer than most retellings, arguing the canary bypass was the real failure and noting the missing per-regex watchdog.
The postmortem's short-term items were done or dated at publication. The CPU guard was back, and all 3,868 rules had been inspected. Profiling was due 19 July 2019 and the engine switch 31 July 2019. No follow-up post confirms either date. The long-term item arrived on 29 March 2021: a new WAF whose engine is “implemented in Rust”, uses wirefilter syntax, and “uses the same underlying Rust library” as the custom Firewall Rules Cloudflare described in March 2019. The 2021 post, as we read it, does not name the regex crate inside that engine. wirefilter's current Cargo.toml pulls in the Rust regex-automata crate as an optional dependency that is on by default, and Wikipedia's ReDoS article (a secondary source) says the WAF was rewritten on the non-backtracking Rust regex library. The careful statement is: a Rust engine built on wirefilter, whose current regex dependency is from the linear-time Rust regex family.
The Lua path did not disappear. Cloudflare's December 2025 postmortem refers to the old proxy, FL1, as still Lua-based, alongside the newer FL2 in Rust. Quicksilver kept evolving too: a July 2025 post describes its move toward v2, starting with a split into replica and proxy roles, with 10 Quicksilver instances on every server and a store that serves over 3 billion keys per second.
2 Jul 2019
WAF rule with backtracking regex, pushed globally via Quicksilver; ~27 min
12 Jul 2019
Postmortem: restore CPU guard, re2/Rust regex, profiling, staged rollout for rules
30 Mar 2020
Quicksilver architecture published (fan-out tree, LMDB, 500 ms batching)
29 Mar 2021
New WAF: engine “implemented in Rust”, wirefilter syntax, same library as Firewall Rules
2–4 Nov 2023
Data-centre power failure hits control plane and analytics; edge traffic unaffected; first Code Orange
18 Nov 2025
Bot Management feature file exceeds a hard limit of 200 features (normal ~60); Rust unwrap() panics in FL2; “worst outage since 2019”
5 Dec 2025
WAF-related change via the global config system: killswitch on an “execute” rule hits a nil-index bug in Lua FL1; ~28% of HTTP traffic, ~25 min
19 Dec 2025
Code Orange: Fail Small: health-mediated config deploys; controlled deployments inside Quicksilver
All entries from Cloudflare's own posts (2019 postmortem; Quicksilver 2020; new WAF 2021; Nov 2023, Nov 2025, Dec 2025 postmortems; Code Orange plan). Classification tags are ours. The December 2025 post says the global configuration system “does not perform gradual rollouts”. That is the 2019 property, stated as current, six years later.
Then the delivery pattern came back twice. 18 November 2025: a ClickHouse permissions change (11:05) pushed the network-wide Bot Management feature file, regenerated about every five minutes, past a hard limit of 200 features (normal ~60); a Rust unwrap() in FL2 panicked; impact from 11:20 to full restoration at 17:06; “worst outage since 2019”. Fixes: treat internal config like user input, add more global kill switches.
5 December 2025: during work tied to CVE-2025-55182 (the proxy's body buffer going from 128 KB to 1 MB), a WAF-related change went through the global configuration system, which “does not perform gradual rollouts, but rather propagates changes within seconds to the entire fleet”. A killswitch applied to a rule with an “execute” action hit a nil-index bug in the Lua FL1 proxy: ~28% of HTTP traffic for ~25 minutes (08:47–09:12). The November fixes were “not finished deploying”. Two weeks later “Code Orange: Fail Small” committed to health-mediated config deployments and controlled deployments inside Quicksilver.
Our analysis — what the 2019 fix actually covered
Our analysis — kill switches are code paths too
Most readers don't run a global WAF. Many run something with the same structure: a fast-changing, human-authored artifact, evaluated in the hot path of a shared resource, distributed by a system built for speed. Examples include API-gateway rules, service-mesh routing and authorisation policies, feature-flag evaluators, fraud and abuse rules, log-parsing and alerting rules, search-query rewriters and content-moderation filters. The lessons below are written for that shape, and each says where it stops applying.
1 · Guarantee complexity at the engine, not at the author
Applies when
Applies whenever patterns written by one team are evaluated against inputs controlled by someone else: WAFs, log pipelines, input validators, search. Use a linear-time engine (RE2, Rust regex or equivalent) and treat backreferences and look-around as a feature request that needs a separate, sandboxed path.
Weaker / not when
Weaker when inputs are bounded and trusted (a config parser reading your own 200-byte files). There a backtracking engine’s worst case is small enough not to matter. Write the bound down.
Pitfall in copying
Swapping the engine and keeping a backtracking fallback “for rules that need it”. The fallback becomes the path for exactly the complex rules most likely to blow up.
2 · Shadow and simulate modes are full deployments for resource failures
Applies when
Applies whenever a shadow evaluation shares CPU, memory, a thread pool or an event loop with production. Give it the same staging and budget as a live change.
Weaker / not when
Weaker when the shadow runs out of band: a tee to a separate fleet, offline replay. Then it is a genuinely safe test of decisions and a genuinely useless test of production cost.
Pitfall in copying
Reading “no false positives in simulate” as “safe to promote”. It says nothing about cost at full volume.
3 · Give every fast lane a floor
Applies when
Applies when failures are visible on real traffic within seconds to minutes: CPU, error rate, latency. One small real-traffic stage with a bake of one to a few minutes, gated on those signals, costs minutes and turns global incidents into local ones.
Weaker / not when
Weaker for failures that are rare and input-dependent (a pattern that only blows up on an unusual header). Canary exposure needs to be long enough, or the canary traffic varied enough, for the input to show up. Add scaling tests (lesson 4).
Pitfall in copying
Letting the emergency override become the default. On 2 July the SOP let a non-emergency change use the emergency path.
4 · Test scaling, not speed, and test the guard itself
Applies when
Applies to any rule set with super-linear evaluation risk. Time each rule at n and 2n on synthetic inputs built from its own character classes and flag ratios well above 2. Separately, assert that a known-pathological rule is stopped by the runtime guard.
Weaker / not when
Weaker when the engine is linear by construction. The scaling test then becomes a regression check on constants.
Pitfall in copying
Absolute-time thresholds tuned on one CI machine; they flake, get raised, and stop detecting anything.
5 · Stage by meaning, in the distribution layer
Applies when
Applies to any shared config store (Quicksilver, etcd, Consul, a flag service) carrying many key classes. Put staging in the store, or in one mandatory client, keyed by class. Otherwise every team re-implements it, and the teams that don’t become the next incident.
Weaker / not when
Weaker for single-purpose stores whose only consumer already stages.
Pitfall in copying
Fixing the class that failed and assuming the channel is fixed. Cloudflare’s own 2019→2025 arc is the example.
6 · Keep break-glass independent of what it fixes
Applies when
Applies to any company that dogfoods its own access, identity or networking products. The recovery path (auth to the control panel, the kill switch, the status page, customer-facing controls) must work when the product is down. Rehearse it with the product down.
Weaker / not when
Weaker only if you can show control and data planes share no failure domain.
Pitfall in copying
A bypass that exists on paper, with expired credentials and nobody who has used it this quarter.
Our analysis — questions to ask in a design review of any hot-path rule or config channel
What not to copy
Related material: the SRE / Reliability track (failure domains, telemetry-gated rollouts), Performance Engineering (saturation arithmetic), and the CrowdStrike Channel File 291 study in this series, whose fast content lane likewise had no floor.
One sentence to carry out of this case: a rule that is evaluated on every request is a deployment of its worst case, so bound the worst case in the engine, not in the review.
Primary documents first, then technical references, then secondary reporting. Our measurements (section 4) used Node 24.13 on a laptop; the step-count model (section 3) was re-run during fact-checking and reproduces all nine published counts. They are illustrative and are not from Cloudflare.