From Thursday 28 October to Sunday 31 October 2021, Roblox was down for 73 hours. It was not a regional brownout. Fifty million daily players were locked out of the whole platform over Halloween weekend. The published postmortem is candid and unusually detailed, and most engineers have read it as a story about two obscure bugs: a lock-contention defect in a new Consul feature and a pathological freelist in BoltDB. That reading is correct, and it is the least useful thing to take from the incident. The bugs explain why Consul got slow. They do not explain why a slow Consul took everything down, why finding the cause took about 50 hours, or why bringing the platform back after the fix took another 21. Our thesis: Roblox had put its discovery, scheduling, secrets, locking, general-purpose data and telemetry into one strongly consistent failure domain, and the one failure mode that domain handles worst is the gray failure, a node that is alive but slow. What follows reconstructs the system and the incident from every public source (the postmortem, the CEO’s update, two SEC filings, HashiCorp’s pull requests and docs, the BoltDB author’s own comments, and Roblox’s 2023 and 2025 infrastructure posts), does the arithmetic the postmortem leaves undone, and ends with the precise conditions under which you are exposed to the same shape of failure.
73 h
outage, per the postmortem
~50 h
detection → first fix
$25M
bookings Roblox says it lost
490×
freelist bytes vs payload per append
The facts, compressed. Roblox ran its latency-critical core in its own data centers (“over 18,000 servers and 170,000 containers”, “hundreds of internal online services”), with AWS for some databases, object storage, queuing and burst compute. The 2023 follow-up supplies the fact the postmortem underplays: “At the time of our outage, we had one active data center”. On top sat the HashiStack. Nomad scheduled containers and fed an IP:Port registry; Consul provided discovery, health checks, session locking “for HA systems built on-top”, and KV; Vault held the production secrets used for authentication, and itself depended on Consul. Consul was one cluster of five voters and five non-voters on 64-core machines, serving all backend services. Scattered through the postmortem are four usage patterns that turn out to be half the story: core services using KV “as a convenient place to store data”; the cache fleet keeping its scheduling state in KV; “a large amount of obsolete KV data”; and telemetry with a circular dependency on Consul.
Every box is named in Roblox's postmortem. What it never draws is the shape: one Raft group in one data center, carrying discovery, locking, secrets, scheduling, general-purpose data and the telemetry you would use to debug it. Note the one arrow that points the other way in spirit: traffic routing was not a victim but the load source added the day before. Stall the group's write path and every other box fails in the same minute.
The problem is visible in that graph before a single bug enters the story. A Raft group is a serialization point: a health flip, a session renewal, a Vault operation, a cache-scheduling record and some team’s config blob all queue for one leader, one log, one fsync path and one apply loop. The five non-voters make reads cheaper and do nothing for that queue. Two properties of a shared log are rarely stated. It has no QoS: Vault’s authentication writes cannot overtake a burst of health updates, because the log is FIFO. And its failure is correlated by construction: every tenant sees the same latency at the same moment, so the platform does not degrade by feature, it degrades all at once.
Our analysis — why a shared Raft log has no graceful degradation
W = S ÷ (1 − λS), with S anchored on the 20 ms FSM-apply figure from PR #12080 and the two dots placed on the latencies Roblox reported. Constants are illustrative; Raft batching moves the curves, not their shape. The point for a design review: at a healthy ρ of ~0.35 the cluster has ~3× headroom against either failure direction, and both of Roblox's defects attacked the cost axis (S), not the demand axis. That is why load-shedding (moving left) could not rescue the 3× curve: it has no stable operating point above ρ ≈ 0.33 of the old capacity, and the churn loop kept pushing load back up. A practical threshold follows (the multiple is our illustration, not a vendor default): alert on service time at low load drifting above ~1.5× baseline, long before latency, which only moves near the asymptote.
When the postmortem summarises the incident as “the system failed because Consul was a single point of failure, and Consul was not healthy”, it is describing a design property that existed long before October 27.
Our analysis — the scale numbers do not reconcile, and that is fine
The trigger was a well-motivated performance feature. HashiCorp’s issue #6366 (August 2019) proposed replacing blocking-query long polls, which re-send a service’s whole result to every watcher on every change, with an event stream. Its benchmark, from “an extreme but real customer workload”, showed server CPU 60% → under 1%, memory 420MB → 20MB, bandwidth 2.2 Gbps → 0.5 Mbps. Consul 1.9.0 (November 2020) added the streaming RPC endpoint; 1.10.0 (June 2021) made use_streaming_backend default to true for service health (servers also need rpc.enable_streaming). Roblox upgraded 1.9 → 1.10 “to take advantage of a new streaming feature”, enabled it on “a subset of our services”, saw it work “as expected”, and widened it over months. On 27 October at 14:00 it enabled streaming on “a backend service that is responsible for traffic routing” and, in the same rollout, grew that tier’s node count by 50% for end-of-year traffic. It “worked well with streaming at this level for a day.”
Our analysis — what streaming actually traded, and why the benchmark could not see it
Write path · leader only
KV / health / registration writes
from client agents
Raft log append
raft.db (BoltDB)
#2 freelist rewrite per append
Replicate
to 4 other voters
Commit
quorum 3 of 5
FSM apply
state store update
Streaming read path · coupled to FSM apply
FSM apply
emits change events
Event publisher
one eventBuffer per topic
#1 wake-all contention
Subscriber goroutines
every one woken per event
gRPC streams
to client agents
5 voters
hold authoritative state; one is leader; leader changes several times a day even when healthy
5 non-voters
read-only replicas that scale reads; they do not help the write path at all
Snapshots
periodically write state to disk, then delete the oldest log entries from raft.db
Two defects, two different layers, one symptom. Pathology #1 slowed FSM apply from the read side (HashiCorp's later PR says slow event publishing slows applying changes to the state machine). Pathology #2 slowed the log append on specific leaders. Both surface to operators as the same number: KV write latency.
Two details in that rollout matter later. First, two variables moved at once: a new client population on streaming, and 50% more nodes in it. Neither the service’s name nor its node count is published, so the size of the increment is unknown. Second, the change sat on the read path. Streaming delivers state changes to watchers; nothing about it looks like it should affect how fast a KV write commits. That intuition is exactly what delayed the diagnosis, and it is wrong for a reason shown in the diagram above: in Consul the event publisher is fed by the state-machine apply loop, so anything that slows publishing slows applying, and anything that slows applying slows every write.
Reconstruction (inferred) — what was actually new on October 27
All times below are Pacific local time. (The postmortem’s footnote says “PST”; US daylight saving time ran until 7 November 2021, so the clocks were actually on PDT. This does not change any interval.)
Phase boundaries are the postmortem's own. The green line splits the story: ~50 hours to find the mechanism that mattered, then ~25 more hours from that breakthrough to 100% of players (about 21 of them after Consul was declared stable at 20:00 Saturday). The postmortem's elapsed-hour markers are not all measured from the same zero (its "14h" and "54h" fit 13:37; its "61h" and "73h" fit the 16:35 player drop), so read hour counts as ±3h.
If you have read the postmortem, skim this paragraph; the chart and ladder carry it. Thu 13:37: Vault degraded, one Consul server hot, players fine; KV write p50 2s against a norm under 300ms; a suspect node is replaced; HashiCorp joins. 16:35: players fall to 50% of normal and the platform follows, because with Consul unhealthy “the system cannot schedule new containers or retrieve production secrets used for authentication.” Every Consul server is replaced with 128-core, NVMe machines; at 19:00 p50 is still ~2s and the leader is “regularly out of sync with the other voters”. Fri 02:00–04:00: the cluster is restored from a snapshot from the outage’s start (accepting “a small amount of system config data loss”), fenced with iptables, looks healthy, and relapses to 2s as soon as internal traffic returns: “over 14 hours into the incident, we still didn’t know what it was.” Fri 04:00 → Sat 02:00: all non-essential Consul use disabled (“several hours” of config pushes alone), services cut from hundreds of instances to single digits, health checks 60s → 10 min; relapse “with significantly less load.” Sat 02:00–12:00: attention moves to Consul internals; back to 64-core servers; no help. Sat 12:00–20:00: flame graphs show “kernel spin locks via the Streaming subscription code path”; streaming off everywhere, propagated by 15:51, p50 → 300ms. Some newly elected leaders are still slow, so the team keeps them from staying elected; BoltDB is identified only after the outage. Sat 20:00 → Sun 16:45: the cold start (section 6).
Hypothesis
Action taken
Why it was reasonable
What refuted it
Slow (not failed) hardware
Replace one Consul node
Slow disks do hurt Raft; common at this fleet size
p50 still 2s; players fall to 50% at 16:35
Load tipping point
All nodes → 128-core, faster NVMe
Increased traffic might have pushed Consul past what its servers could handle
19:00: majority of nodes can't keep up; p50 ~2s
Bad cluster state
Restore snapshot; fence with iptables
Clean state from before the outage
Healthy until the fence drops, then 2s again
Reconnect storm from thousands of containers
Disable non-essential use; hundreds → single-digit instances; checks 60s → 10m
Internal traffic alone was measurably heavy
Unhealthy again by 02:00 Sat at far lower load
128 cores amplified contention
Move back to 64-core servers
htop + perf showed contention
Did not help
Streaming subscription contention
Disable streaming everywhere
perf / flame graphs: kernel spin locks in streaming path
15:51 Sat: p50 → 300ms
Numbering follows the postmortem ("our first … fourth attempt at diagnosing"). Note the pattern: hypotheses #1–#3 are all about the environment around Consul (hardware, load, clients). Only after 36 hours did the team pivot to Consul's internals, and that is where both causes were.
Read as a sequence of experiments, the ladder has a clear shape. Hypotheses #1 to #3 are about the environment around Consul: hardware, aggregate load, client behaviour. Each produced a large, slow, multi-variable intervention (replace a node; replace every node; rebuild the cluster from a snapshot; reconfigure hundreds of services). Each took hours to execute and hours more to refute. The team only turned to Consul’s own internals around hour 36. We do not know how early the team had perf data from the servers; the postmortem’s statement that the monitoring which would have helped “relied on affected systems, such as Consul” is the most likely reason the profile came late.
Our analysis — the time went into refutation, not repair
Our analysis — demand or capacity? The question each experiment was really asking
Note on the duration
The postmortem’s explanation of the streaming defect is short. Streaming, HashiCorp told Roblox, “used fewer concurrency control elements (Go channels) in its implementation than long polling”; under very high read and write load it “exacerbates the amount of contention on a single Go channel, which causes blocking during writes.” On the 128-core, dual-socket NUMA servers, contention on shared resources got worse. That is accurate but compressed, and it leaves open a question raised on Hacker News: why would a read feature slow down writes?
The fuller explanation is in the fix HashiCorp shipped three months later. PR #12080 (“streaming: split event buffer by key”, merged 28 January 2022, released in Consul 1.11.3 on 11 February) does not name Roblox, but its description matches the postmortem’s symptoms point for point. Events are published by appending to an eventBuffer, “a single-writer, multiple-reader, linked list in which each element contains a channel” that the publisher closes to wake subscribers waiting for the next element. There was a single buffer for the entire topic, so every subscriber is woken on any health event, “regardless of whether or not it pertains to a service they’re watching”, and then goes back to sleep. Go’s channel locks are backed by Linux futexes, and the kernel’s futex hash buckets are protected by spinlocks the PR calls “semi-global”, which is why the obvious fix of sharding the channel did not help: “it isn’t contention on the individual channel’s lock, but rather the kernel’s own internal (semi-global) lock!” The CPU time showed up in native_queued_spin_lock_slowpath, consistent with the “kernel spin locks via the Streaming subscription code path” in Roblox’s perf report.
The coupling to writes is the key point. Per PR #12080, slow event publishing slows the application of changes to the Raft state machine (the FSM), and the PR adds the number that turns a slowdown into an outage: “sustained FSM Apply latency of just 20ms, when Raft’s internal queues/buffers are full, can result in several minutes of RPC latency.” The fix splits the buffer by key (for example, service name), so an event wakes only the goroutines that have real work. The PR is candid that the pathological case, every subscriber watching one service, is not improved, but in testing it did not show the problem because every woken subscriber had real work to do.
Our analysis — the wake-up arithmetic
The boxes and the interventions are sourced; the loop is our inference. Read it as a control problem. Wake-ups scale with (subscriptions × event rate), and an unhealthy cluster raises the event rate by itself, because failing checks and reconnecting clients are events. Three of the four interventions acted on the input or the state; only the one that removed an edge (streaming off) broke the loop. That is why the system could run for a day at the new level, fall off a cliff, and relapse after every reset.
The loop in the diagram above is our inference, but each edge is sourced, and it explains the three facts that were most confusing during the incident. It explains why the system ran fine for a day at the new level: wake-up cost is proportional to the event rate, and the event rate in a healthy cluster is low. It explains why the cluster kept relapsing after every reset: once writes slow down, health checks and sessions start failing, services mark themselves unhealthy and reconnect, and every one of those state changes is itself a health event that wakes every subscriber. And it explains why load-shedding did not work: cutting instance counts and slowing checks reduces the input to the loop but does nothing to the amplifier. Roblox observed exactly this: the second restart failed “with significantly less load.”
The postmortem also explains the counter-intuitive hardware result (PR #12080 does not discuss NUMA). It says the 128-core machines were “dual socket architectures with a NUMA memory model” and that “the additional contention on shared resources thus got worse under this architecture.” A spinlock on a shared cache line is the textbook case of a workload that scales retrograde: every additional contender makes every existing contender slower, and cross-socket cache-line transfers make each handoff more expensive. Our reading: more cores meant more concurrently running subscriber goroutines contending for the same kernel locks.
C(N) = N / (1 + α(N−1) + βN(N−1)), with α = 0.02 and β = 0.0004 / 0.0008 picked only to draw the shape. When the β term dominates (every added contender makes every other contender slower, which is what a shared spinlock does), more cores sit on the retrograde side of the peak. Roblox's own conclusion was that faster hardware "potentially hurt stability", and HashiCorp attributed that to dual-socket NUMA.
Our analysis — why moving back to 64 cores did not help either
With streaming off, most of the cluster recovered, but some leaders still showed the old write latency when elected. That second defect lived one layer down, in the Raft log store. Consul stored its Raft log in BoltDB, a copy-on-write B+tree in a single file. The log is not Consul’s state; it is “a rolling log of the operations being applied.” Consul periodically snapshots its state and deletes the oldest log entries. BoltDB never shrinks its file: pages freed by the deletion (4kB each) are marked free, tracked in a freelist, and reused. In the BoltDB design Consul used, the freelist is written to disk on every commit; Consul’s telemetry docs say so directly for consul.raft.boltdb.freelistBytes: unless no_freelist_sync is set, “these metadata bytes must also be written to disk for each committed log.”
The postmortem published the BoltDB statistics used in the analysis. On the affected leader, the 4.2GB log store held only 489MB of actual data; 3.8GB was free space. The freelist was 7.8MB because it held “nearly a million free page ids.” And so, for every log append (each Raft write, after some batching), “a new 7.8MB freelist was also being written out to disk even though the actual raw data being appended was 16kB or less.” Back pressure filled TCP buffers (the team saw TCP zero windows) and contributed to 2–3 second writes on unhealthy leaders.
Why this is a “bad design” is not our judgement; it is the author’s. Ben Johnson, who wrote BoltDB, replied on the Hacker News thread: “BoltDB author here. Yes, it is a bad design.” He explained that Bolt started as a port of LMDB to learn its internals, that LMDB stores the freelist in a regular bucket while Bolt “simply saved the list as an array”, and that the simplification rarely mattered: “It only became an issue when someone wrote a ton of data and then deleted it and never used it again.” That sentence describes a Raft log after a write storm exactly.
Healthy file
16 kB data + ~32 kB freelist (assumed)
Mild bloat
16 kB data + 1 MB freelist
Roblox slow leader
16 kB data + 7.8 MB freelist (reported)
appends/s ≈ bandwidth ÷ (payload + freelist). The 1 GB/s figure and the healthy-file freelist size are assumptions for illustration; only the 16 kB and 7.8 MB are reported. The model is generous: it ignores the CPU to encode a million-entry freelist and the two fsyncs per append BoltDB performs (raft-wal README). Even so, a bloated leader loses about two orders of magnitude of append headroom, and Raft batching only partly hides it.
The back-of-envelope above makes one point that the postmortem’s figures imply but do not state. Writing 7.8MB to a modern NVMe drive takes milliseconds, not seconds. The 2-second writes are not the cost of one append; they are what happens when the leader’s append capacity drops by two orders of magnitude while the arrival rate of writes does not. Once arrivals exceed capacity, the queue grows without bound, TCP windows close, followers fall behind, and the leader looks “out of sync with the other voters”, the symptom the team had seen before the first reset. The latency is queueing; the freelist only moved the knee of the curve.
Reconstruction (inferred) — how the file got to 4.2GB, and why only some leaders
The in-incident workaround deserves attention because it is a pattern worth having in your runbook. Raft’s failure model is crash-stop: a leader that keeps sending heartbeats is, as far as the protocol is concerned, healthy. The post-incident dashboard in the postmortem shows the kind of liveness signal Raft sees, “contact with other servers … in the last 32ms”, and a slow leader can pass that test indefinitely while committing writes at a tiny fraction of normal speed. Raft has no concept of “alive but too slow to lead.” The team supplied one by hand: keep elections away from the bad nodes. The postmortem does not say how, and we will not invent a mechanism, but the principle is operator-enforced leadership eligibility, and you want the tooling for it built and tested before you need it.
Signal
Healthy leader
Bloated-freelist leader
Catches it?
Leader contact age
~32 ms (post-incident dashboard)
still fresh: heartbeats are small and cheap
No
Leader elected, quorum present
yes
yes
No
Cluster-wide KV write latency
30.6 ms (< 300 ms = normal)
2 s p50; 2–3 s on unhealthy leaders
Detects, cannot say which node
Write latency keyed by current leader
flat across elections
steps up when certain servers win
Detects and attributes. This is how the team found it
consul.raft.boltdb.freelistBytes / writeCapacity
small / low use
7.8 MB freelist; alert when writes > 75% of capacity
Leading indicator (shipped after the outage)
The fourth row is sourced: the team saw that the cluster was healthy "as long as certain servers were not elected". Our suggestion is to make that an alert rather than an observation: tag every commit-latency sample with the leader's ID, and page when latency under the current leader exceeds (illustratively) 3× the median under the other voters for 10 minutes. Pair the alert with a tested runbook action (transfer leadership away, then drain the node), because a protocol that elects by liveness will keep choosing the gray node.
After the incident, HashiCorp and Roblox used existing BoltDB tooling to “compact” the database, which resolved the performance problem. The longer-term fix is where the popular account gets imprecise, and we return to it in section 8.
At 20:00 Saturday, “54 hours since the start of the outage”, Consul was stable. Players got full access at 16:45 Sunday. That is almost 21 hours, longer than most organisations’ worst outage in total, spent on a system that no longer had a known defect. The postmortem lists why, and every item is a design decision that assumed the platform would never be fully cold.
Consul stable
20:00 Sat · +54h
streaming off; slow leaders kept out
Cache fleet redeploy
→ 05:00 Sun · 9h
stale KV scheduling data; a sick node seen as 'open'; tool not built to bootstrap
Services at capacity
→ 10:00 Sun · 5h
many had been scaled to single digits
Players via DNS steering
→ 16:45 Sun · 6.75h
~10% steps; check DB load, cache, stability
The cache fleet. Roblox’s caching system “regularly handles 1B requests-per-second across its multiple layers.” The databases underneath were unaffected, and caches hold data that can be rebuilt, so the decision was to redeploy the fleet rather than repair it. Three problems followed. The cache system stored internal scheduling data in Consul KV, and that data was wrong, “likely due to the Consul cluster snapshot reset” of Friday morning, so a decision made at hour 12 to save time cost time at hour 55. An unhealthy node was seen by the job scheduler as “completely open” rather than unhealthy, so the scheduler aggressively placed cache jobs on it; small caches deployed slowly and large ones never finished. And the cache deployment tool was built for “incremental adjustments to large scale deployments that were already handling traffic at scale, not iterative attempts to bootstrap a large cluster from scratch.” The team worked through the night; caches were healthy at 05:00 Sunday.
Our analysis — why “unhealthy” read as “completely open”, and why cold starts invite it
Reconstruction (inferred) — the recovery was a serial critical path, and the snapshot reset sat on it
The services. Much of the service fleet had been shut down or scaled to single digits during troubleshooting, so every service had to be restarted at the right capacity and verified. That went smoothly and took five hours.
The players. With cold caches and “a system we were still uncertain about”, Roblox used DNS steering to admit a random percentage of players while sending the rest to a static maintenance page, raising the percentage in roughly 10% increments and checking database load, cache performance and overall stability at each step. That took from 10:00 to 16:45, about 40 minutes per step on average. Some players reverse-engineered the steering scheme and shared it on Twitter to get in early, which the postmortem notes with some affection.
Our analysis — why the ramp had to be slow
The deeper lesson is that the recovery path is its own system with its own dependencies, and Roblox’s recovery path depended on the thing that had failed. The cache scheduler read its state from Consul KV. The telemetry the team needed to watch the ramp had depended on Consul. The deployment tooling assumed an already-running fleet. Recovery time is set by the cold-start path, and most organisations never test it because a full cold start is exactly the event they hope never happens.
Roblox disclosed the financial impact in its Q3 2021 shareholder letter (8 November 2021), with unusual precision, because it had to estimate what October would have looked like to compensate its creators. Through the first 27 days of October, average DAUs were 50.5 million (up 43% year on year); for the full month they came in at 47.1 million. Engagement hours were 3.2 billion through 27 days and 3.4 billion for the month. Bookings were $177–179 million through 27 days and $189–192 million for the month. Roblox estimated $25 million in lost bookings, plus $6 million of bookings received during the outage (mostly prepaid cards) for which developers had not been paid, $31 million in total. At its 22% payout ratio it credited developers with $6.8 million.
Average DAU, Oct 28–31
Oct 1–27 run-rate
Oct 28–31 implied
Engagement hours, Oct 28–31
Run-rate for 4 days
Actual (3.4B − 3.2B)
Bookings, Oct 28–31
Average-day run-rate
Actual
Implied expectation
The last bar is the interesting one. Roblox's $25M loss estimate implies it expected those four days to book $8.75–10M/day, against a ~$6.6M/day average for October 1–27: 1.3–1.5× a normal day. That is consistent with the letter's note that its estimates drew on historical data about weekend and Halloween spending. The outage landed on the most valuable weekend of the month.
The arithmetic in the chart uses only those figures. Average DAU over 28–31 October comes out at about 24 million, half the month’s run-rate, which is roughly what you expect from two days fully down (Friday and Saturday) and a partial day on either side. Engagement over the four days was about 0.2 billion hours against a run-rate of about 0.47 billion. Bookings over the four days were $10–15 million; the letter separately says $6 million of bookings were received during the outage, “primarily from prepaid cards”, for which developers had not been compensated.
The most interesting inference is the last one. Adding the $25 million loss back to actual bookings gives what Roblox expected those four days to earn: $35–40 million, or $8.75–10 million a day, against a ~$6.6 million daily average for October 1–27. The letter says its estimates drew on “a lot of historical data around engagement and spending on weekends and specifically around Halloween.” By Roblox’s own estimate, then, the outage fell on a weekend that should have been worth 1.3–1.5 times an ordinary day. Measured against Roblox’s own estimate, the direct cost was about $350,000 per hour of outage ($25M over ~70 hours), before compensation, reputation, or the engineering programme that followed.
~24M
implied avg DAU, Oct 28–31
~0.27B
engagement hours lost (est.)
$8.75–10M
expected bookings per day
~$350k
lost bookings per outage hour
Our analysis — the outage in the currency of Roblox’s later SLO
The 10-K puts the outage in historical context: “since our inception we have had approximately one outage each year.” The difference in 2021 was duration, not frequency. And duration is the variable the architecture controls least in a single failure domain, because recovery requires both a diagnosis and a cold start of everything.
The postmortem narrates decisions in sequence; it does not score them. The matrix below does, on the two properties an incident commander should care about most in the first hours of a gray failure: whether a move can be undone cheaply, and how much it will tell you whether it works or not.
Option
Time to execute
Reversibility
Information yield
What happened
Revert Oct 27 change (streaming on routing + 50% nodes)
Config push; the eventual streaming-off push finished propagating at 15:51, under 4h into the phase that found it
High: cheap to undo
High: removes the newest variable
Postmortem never says it was considered; streaming-off came at hour ~46-50
Replace a suspect node
~1–3h
High
Low: one data point
Tried first
Replace every Consul server (64 → 128 cores)
~2.5h + membership churn
Medium
Low, and it moved two variables
Made it worse
Snapshot reset + iptables fence
2h
Low: config data loss accepted
Medium: separated state from load
Resurfaced 40h later as stale cache data
Shed all non-essential Consul load
Several hours of config pushes
Medium
High: proved it wasn't just load
Took 22h to reach the conclusion
Profile the servers first (perf, flame graphs)
Minutes to collect
Total: read-only
Highest: pointed at streaming
What actually found it, at hour ~46
The outcome column comes from the postmortem; the time estimates, reversibility and information scores are ours, inferred from the phase durations it reports. The two cheapest, highest-yield moves (revert the newest change to the failing component; profile the failing process) were the last two to pay off.
The rollback that was not tried. A prominent critique on Hacker News was simple: Consul was identified as the failing component at the start of triage, the most recent significant change to Consul’s workload had shipped the day before, and the postmortem never says reverting it was considered. One commenter summarised the standard playbook as “Which system is broken? … Are there any recent changes to this system? If so, can we try reverting them?” Others pushed back that rolling back in stateful systems is not always safe. Both sides are partly right. The streaming change turned out to be a config push that had finished propagating by 15:51, less than four hours into the phase in which it was identified, and it restored write latency at once; it was as rollback-safe as a change gets. But the change had been in production for nearly 24 hours without problems, streaming had been rolling out for months, and the co-shipped change (50% more routing nodes, for end-of-year capacity) was not something you would want to reverse right before a peak.
Counterfactual
Replacing every Consul server with bigger machines. This is the decision that looks worst in hindsight, and the postmortem says so plainly: “Faster hardware hadn’t helped and, as we learned later, potentially hurt stability.” The deeper problem is not that the hypothesis was wrong; it is that the intervention was expensive, changed several variables at once (CPU count, NUMA topology, disk, Raft membership) and could not falsify anything cleanly. A scale-up that fails leaves you not knowing whether load was the cause. A controlled experiment would have taken one voter, or the non-voters, to the new hardware and compared. That is easy to say from outside an outage and hard to do inside one, which is the argument for writing those experiments into the runbook in advance.
The snapshot reset. Restoring the cluster from a snapshot taken before the incident, fenced by iptables, was a good experiment: it separated “bad state” from “bad load” in two hours and ruled out state. The trade-off was explicit (a small, accepted loss of system configuration), and the cost arrived 40 hours later as stale cache-scheduling data in KV. The lesson is not that the reset was wrong; it is that application data (the cache fleet’s scheduling state) was sitting in what the operators thought of as control-plane config. Accepting config loss is only a bounded decision if you know everything that lives in the store.
Shedding load for 22 hours. Attempt #2 was the most thorough experiment of the incident and the most expensive. It proved conclusively that the problem was not proportional to load, which was the insight that sent the team into Consul’s internals. In the churn-loop model it was always going to fail, because it throttled the loop’s input rather than its amplifier. Its real cost was not the 22 hours; it was what it left behind: hundreds of services at single-digit instance counts and a cache fleet that had to be rebuilt, which shows up again in the 21-hour recovery.
Working around the slow leaders. Choosing to keep bad nodes from leading rather than root-causing BoltDB during the incident was plainly right. The root cause took HashiCorp days after the outage to find; Roblox spent those days with players on the platform.
The slow, stepped reopening. Also right, for the cache arithmetic in section 6. The alternative, opening fully and relying on the databases to survive the miss storm, risked a second collapse with no remaining margin of trust.
Staying on-prem. The postmortem includes “A Note on Public Cloud”, arguing that owning data centers “significantly control[s] costs compared to public cloud”, that savings go to creator payouts, and that owned hardware and edge let Roblox manage latency for players “not necessarily located near the data centers of public cloud providers.” It adds: “we are not ideologically wedded to any particular approach.” Note what this does and does not argue. Nothing in the incident was caused by owning hardware: the defects were in software Roblox would have run the same way on EC2, and the single Consul cluster and single active region are choices you can make on any provider. What owning hardware does change is how fast you can create a second failure domain; Roblox had to build a data center to get one, and its 2023 post describes rebuilding cells by wiping and reprovisioning machines as they were freed.
Counterfactual — a stale-tolerant discovery path
“The Chipotle promotion crashed it.” False. A Chipotle promotion on the platform was widely blamed, and the rumour spread fast. Roblox said on 29 October the outage was “not related to any specific experiences or partnerships”, and the CEO’s update said it was “not due to any peak in external traffic or any particular experience.” The postmortem’s timeline settles it: the first symptoms at 13:37 were in Vault and a Consul server, with players unaffected, and the cluster later relapsed with no external traffic at all.
“It was a hack.” False. Roblox said on 30 October it had found “an underlying internal cause of the outage with no evidence of an external intrusion,” and the postmortem states there was “no user data loss or access by unauthorized parties.”
“A cloud-native catastrophe.” InfoWorld’s headline framing is misleading if read as implying public cloud (the article itself quotes Roblox on building on-prem). The platform ran in Roblox’s own data centers, and Roblox argued explicitly for staying there. The accurate framing is narrower: a self-hosted, container-orchestrated control plane whose consensus layer was a shared dependency for everything.
“They turned streaming on the day before.” Half true. Streaming had been expanding service by service for months without problems. The change the day before put it on one large, newly grown client population. That distinction matters for the lesson: gradual rollout was not the missing practice. What was missing was a rollout gate on the metric that actually failed, the product of subscriber count and event rate, rather than on overall health.
“BoltDB caused the outage.” Oversimplified. Roblox names two root causes, and the breakthrough came from disabling streaming; the BoltDB slow-leader problem was worked around during the incident and root-caused only afterwards. Both contributed to the 2-second writes, and no source quantifies how much each did. Roblox describes the two as “primarily unrelated”; if our coupling reading in section 5 is right, the BoltDB bloat was nonetheless partly a consequence of the streaming-induced storm.
“Moving to bbolt fixed the freelist problem.” Only partly. The postmortem says bbolt “does not have the same issue with unbounded freelist growth.” What Consul shipped in 1.11.0 (14 December 2021, PR #11720) was bbolt, an option to disable freelist sync (no_freelist_sync) and new BoltDB metrics. But HashiCorp’s later documentation still describes large BoltDB freelists as a live risk and says Consul’s snapshot and truncation defaults are set “aggressively” to keep BoltDB small; the raft-wal README names truncation after large writes as the “major performance issue in raft-boltdb”; and a November 2023 raft-boltdb PR proposed switching to a bbolt fork “to improve performance with large freelist.” HN commenters also noted that bbolt’s hashmap freelist (contributed by Alibaba for etcd in 2019) did not appear to be exposed by Consul; we have not verified that in the source. The durable fix for the class of problem is the segmented WAL log store: experimental in Consul 1.15 (February 2023), and listed as the default Raft log store in HashiCorp’s current configuration reference. It deletes old segments instead of tracking free pages, and does one fsync per append instead of BoltDB’s two.
“HashiCorp fixed Roblox’s bug.” Probably, but not documented. PRs #11720 and #12080 match the postmortem’s symptoms closely and were merged five weeks and three months after the outage respectively, but neither names Roblox. We call them “HashiCorp’s subsequent fixes.”
“The lesson is: don’t use Consul.” This is the take that transfers worst. The self-inflicted parts of the outage (one cluster for everything, general data in the KV store, obsolete data piling up, telemetry dependent on the monitored system, recovery tooling that assumed a warm fleet) would have produced the same shape of incident on etcd, ZooKeeper or any strongly consistent store. The specific bugs were Consul’s. The blast radius was Roblox’s.
A note on sources we did not use
The postmortem was published on 20 January 2022, 2.5 months after the outage, and Roblox explained the delay: it “owed it” to the community to make significant progress on reliability before publishing. The listed changes fall into four groups. Telemetry: the circular dependency was removed (“Our telemetry systems no longer depend on the systems that they are configured to monitor”), with new targeted alerts on Consul and BoltDB and visibility into service-to-Consul traffic. Consul load and sharding: some critical services split into their own dedicated Consul clusters, KV data migrating to proper stores, obsolete KV deleted, and a bbolt-based Consul upgrade deliberately postponed past peak end-of-year traffic, “will complete in Q1.” Bootstrapping: the cache deployment redesigned to come up “from a standing start”, Nomad enhancements for turning up large jobs after long unavailability, and faster machine configuration changes. Failure domains: an additional, geographically distinct data center built out, with multiple availability zones planned. Streaming would return only once a new implementation “has been tested at our scale with our workload.”
Issue #6366: streaming designed to replace blocking queries; benchmark shows CPU 60% → <1%, bandwidth 2.2 Gbps → 0.5 Mbps
Community issue #8442 asks Consul to move from archived BoltDB to bbolt
Consul 1.9.0: new RPC endpoint for streaming cluster state events
Consul 1.10.0: use_streaming_backend defaults to true
Upgrade 1.9 → 1.10 'to take advantage of' streaming; enabled service by service over months
Streaming on the traffic-routing service + 50% more routing nodes
73-hour outage
Consul 1.11.0: bbolt, option to disable freelist sync, BoltDB metrics (PR #11720)
Year-end surge: 'not a single significant production incident'
Postmortem published; bbolt rollout 'will complete in Q1'
Consul 1.11.3: streaming buffer split by key (PR #12080)
consul.raft.boltdb.writeCapacity metric added
Consul 1.15: experimental WAL log store
Active/active experiments; cells carry >70% of backend traffic
Consul configuration reference lists wal as the default log store
HashiCorp / Consul
Roblox
Neither PR #11720 nor PR #12080 names Roblox. The match with the postmortem's symptoms (128 cores, many subscriptions, high churn, kernel spin locks, freelist writes) and the timing make the link a strong inference, not a documented fact.
The early results were good: Roblox reported “not a single significant production incident” during the December 2021 traffic surge, with excellent Consul and Nomad stability. The longer arc is in the December 2023 post by the CTO, VP of Engineering and a technical director, which opens by recalling the outage: “It started small, with an issue in one component in one data center.” The first priority had been a backup deployment in a second region, so the platform now ran active/passive, with active/active as the long-term goal and a load balancer steering by latency, capacity and health to fail over “nearly instantaneously rather than over several hours.” Active/active experiments started in September 2023 and were left running for a limited set of users. Inside each data center, Roblox built cells of ~1,400 machines, with requirements that make them interchangeable: services must be containerised, each cell is defined as code and can be rebuilt from scratch, and a new deployment tool “stripes” services across cells. By December 2023, close to 30,000 machines were managed by cells (roughly 21 cells’ worth at ~1,400 each, by our arithmetic) and more than 70% of backend service traffic was served from cells at peak.
Oct 2021 · one blast radius
▸ One active data center (components inside acting as backup)
▸ One Consul cluster for all backend services
▸ Vault and Nomad both depend on that cluster
▸ Telemetry depends on the system it monitors
▸ Consul KV used as a general datastore; obsolete KV piling up
▸ Cache deploy tooling built for incremental change only
▸ Streaming on across a growing share of services
Failure domain = the whole platform. Recovery = manual cold start.
2022 → 2025 · walls inside walls
▸ Second, geographically distinct DC: active/passive; active/active experiments from Sep 2023
▸ Some critical services split onto their own Consul clusters
▸ Telemetry independent of what it monitors; Consul + BoltDB alerts
▸ Obsolete KV deleted; KV data migrating to proper stores
▸ Cache bootstrap 'from a standing start'; Nomad cold-start enhancements
▸ Cells of ~1,400 machines; ~30,000 machines and >70% of backend traffic in cells (Dec 2023)
▸ Streaming off until re-tested 'at our scale with our workload'
Failure domain target = one cell. Stated goal: 99.99% monthly user uptime.
Two details in that post deserve more attention than they got. First, Roblox names the unsolved problem of cellular designs directly: how to handle a “query of death”, since redirecting a poisonous request to a healthy cell spreads the failure it was meant to contain. Its stated plans include a “next-generation service discovery process” to be leveraged by a service mesh, targeted for 2024, enabling policies that allow cross-cell communication only when it would not harm the failover cells, plus a separate 2024 method for directing dependent requests to a service version in the same cell. That is a direct descendant of October 2021: discovery was the dependency that turned a slow Consul into a global outage, and the fix redesigns discovery as a blast-containment mechanism. Second, Roblox reports the savings in its own currency: in the first half of 2023 it “saved 125 million engagement hours per month” compared with a year earlier. The machine count went from about 36,000 in mid-2021 to nearly 145,000.
By June 2025 Roblox described two core data centers, 24 edge data centers, and more than 1,600 microservices. What we could not find publicly: whether Roblox ever re-enabled Consul streaming, whether it moved to Consul’s WAL log store, and whether Consul remains its discovery layer after the next-generation system. We leave those as open questions rather than guess.
The Roblox outage transfers to your organisation as a shape, not as a Consul bug. You are exposed to it if all of the following are true, and most organisations at scale have at least three.
A shared consensus store is on the request path
If discovery, secrets, scheduling or locking need a quorum write (or a fresh read) to serve live traffic, the Raft group’s write latency is your platform’s availability. Measure it as an SLO, not as a component metric.
More than one tenant writes to it
Every team that finds the KV store “convenient” adds write volume and a new failure mode to your most critical dependency. Put a written allow-list on who may write, and treat everything else as a migration backlog.
Your observability depends on it
If the metrics, logs or dashboards you debug with resolve endpoints or read config through the system under test, you will be blind in exactly the incident where you most need to see. Test by taking the dependency down in a game day and checking what still works.
Recovery tooling assumes a warm fleet
Cache schedulers, deploy tools and autoscalers built for incremental change often cannot bootstrap from zero. If you have never cold-started a tier, assume it will take many hours longer than the outage that caused it.
1. Treat gray failure as the primary failure mode of your consensus layer. Raft, Paxos and ZooKeeper’s ZAB handle crashes well and slow nodes badly. A leader that stays in contact with its followers every few milliseconds but takes 2 seconds to commit is “healthy” to the protocol. Before you need it, build and test tooling to (a) mark specific servers ineligible to lead, (b) transfer leadership on demand, and (c) alert on per-node commit and apply latency, not only cluster-level health. If you run Consul, the consul.raft.boltdb.* metrics (freelist bytes, free-page bytes, and the writeCapacity metric added in 2022, with a suggested alert when actual writes exceed 75% of it) exist because of this class of incident.
2. Gate rollouts on the cost driver, not on health. Roblox rolled out streaming gradually, and each step looked healthy. The cost that broke the system was (subscribers × event rate), and it only appears under churn, which a healthy rollout never exercises. For any fan-out mechanism (pub/sub, watches, streaming discovery, websocket broadcast), identify the multiplicative term, instrument it, and include a churn test in the rollout: deliberately flap a sample of instances and measure the effect on the write path.
3. Make “revert the most recent change to the failing component” the default, not a judgement call. Inside the first hour of an incident, the value of a cheap, reversible, informative experiment beats the value of the most plausible theory. Write it into incident command so that nobody has to argue for it at 2am, and keep co-shipped changes separately revertible, as the extra routing nodes and the streaming flag could have been.
4. Scale-up is not a diagnostic. If adding hardware is your response to an unexplained slowdown, do it on one node and compare. For contention-bound code, more cores (and a second socket) can make things worse, which is what Roblox saw.
5. Know what lives in your control-plane store before you restore it. A snapshot restore that accepts “a small amount of config data loss” is only a bounded decision if there is a current inventory of every consumer and every keyspace. Roblox built that inventory mid-incident, for the load-shedding attempt, after the snapshot reset had already been done, and the cache scheduler’s state in KV was the cost that surfaced later.
Tenant (all named in the postmortem)
Write profile (inferred)
Can it run on stale data?
Placement (ours)
Discovery + health checks
Highest churn: every flap and registration is a write
Often minutes to hours on a last-known view, if clients are built for it
Own cluster; clients designed to run stale
Vault (secrets for authn)
Moderate, steady
Cached credentials yes; new authentication no
Isolated from churny tenants: it gates every restart
Session locks for HA systems
Low, fixed-cadence renewals
None: a missed renewal is a failover
Small, quiet cluster; never next to a fan-out feature
Core services' data in KV
Application-driven, unbounded
Depends on the app
Out of Consul (Roblox: migrating)
Cache fleet scheduling state
Bursty at redeploy
None: stale state stalled recovery for hours
Out of Consul, or at least out of any snapshot-reset scope
Telemetry
Discovery/config lookups
Must work when Consul does not
No dependency at all (Roblox: done)
Obsolete KV
None
n/a
Delete (Roblox: done; it improved performance)
Roblox says only that "some critical services" got dedicated Consul clusters; which ones is not public, so the placements above are ours. The decision rule behind them has three tests. (1) Is it coordination at all? If not (app data, cache state, telemetry), it leaves the consensus store. (2) Is it on the cold-start critical path? If so (secrets, locks, the registry the scheduler writes), it must not share a log with a tenant whose write rate rises during failures. (3) Does its write rate correlate with platform distress? Health and discovery writes spike exactly when things go wrong, which makes them the worst possible neighbour for anything in (2).
6. Put a number on your blast radius and a clock on your cold start. Roblox’s eventual answer, a second data center plus cells of ~1,400 machines, cost two-plus years of migration work, a hardware reshuffle and new deployment tooling. It is the right answer when one incident costs what October 2021 cost (roughly $25M of bookings and ~700 months of a 99.99% error budget). It is over-investment for an internal platform with a generous SLO. The test is simple: estimate your worst plausible full-platform outage in hours, multiply by your cost per hour, and compare it with the multi-year cost of a second failure domain. The SRE / Reliability track covers cell sizing and blast-radius design in depth; the Distributed Systems Deep Dive covers Raft’s liveness assumptions.
When not to copy Roblox’s remediation
Our analysis — questions to ask in a design review
The one-sentence version for your architecture review: a strongly consistent store is a shared fate for everything that writes to it, and the incident you should design for is not the one where it dies but the one where it is alive, slow, and serving your debugger.
Primary sources first. Every factual claim above traces to one of these; analysis, reconstructions and counterfactuals are labelled as ours.