Vendor gear is built for thousands of machines. AWS runs millions.
SREs, platform engineers, architects — anyone building systems that have to survive their own growth.
Each one trades a little waste, or a little hardware, for a failure that stays small.
One-in-a-million per machine per day, times millions of machines, fires every single day.
You cannot engineer failures away at this size. You can only stop them becoming the customer's problem.
At normal size, paying this overhead is obviously right. At AWS's size it grew larger than the cost of building the hardware.
A few percent on one server is nothing. A few percent across millions is buildings, power and cooling.
A $350M acquisition became a one-time engineering investment. At millions of servers, removing a few percent of overhead is worth far more than designing the silicon costs.
What's left on the CPU is a sliver that only partitions the machine. The rest is yours.
If every incident needs an SSH session, the ops team grows with the fleet. A server nobody can log into is one nobody can fat-finger.
Bare metal is the same box with the thin hypervisor switched off. EC2 Mac is a Mac Mini on a Nitro card. Then Graviton.
AWS didn't make humans better at managing servers. It removed the need for humans to touch them.
At ten routers the chassis wins. At tens of thousands, diagnosis time becomes the limit on growth.
A sophisticated router fails in sophisticated ways. Simple failure is the thing that scales.
AWS traffic between regions rides fiber AWS controls, including ownership stakes in the cables under the oceans. Between regions, AWS is not on the public internet at all.
A congestion event out on the public internet mostly cannot touch traffic that never goes there.
All three assumptions are right for the open internet. None hold inside a datacenter AWS owns end to end.
TCP is doing its job perfectly. The world it was designed for isn't the world inside a datacenter.
SRD only has to survive Amazon's own network — never the open internet — so it doesn't pay for that.
The flow was never on one path to begin with, so losing one path is barely an event.
Load one full-size cell to its cap in test. That is the largest unit that will ever exist in production.
The unit never gets bigger. There are just more units.
Sharding turns total failure into partial failure. Better, and still not good.
75,287,520 distinct combinations from a hundred machines. In practice you have more fault domains than customers.
The only AWS service with a 100% availability SLA for answering queries — no outage in almost two decades. The technique is open-sourced.
Hardware grows linearly. Isolation grows combinatorially.
Change-based work peaks precisely when you can least afford it — and drifts silently the rest of the time.
Most downloads carry almost no new information. That is the trade: a small predictable cost every few seconds, in exchange for behaviour that does not change under stress.
Waste a little, predictably, so that stress changes nothing.
Every volume in the region shared one point of collapse — and every year adds volumes, which means a bigger storm aimed at the same database on the next bad day.
The postmortem lesson was structural, not procedural. One database, one region, one point of collapse.
Consensus needs a majority reachable, so Physalia puts each volume's seven nodes physically and topologically next to that volume's client and replicas.
On a partition it chooses consistency — a configuration master has to — then wins it back through smallness and locality.
When a cell fails it takes out one volume's configuration. Not a region's.
A model checker walks every reachable state. In DynamoDB it found a bug whose shortest failing trace was 35 steps.
The policy becomes a maths problem, and a theorem prover is asked whether any request under any conditions could make the bucket public — the engine behind the "Public" badge in the console.
Every time you change a bucket policy, a theorem prover decides whether it can expose your data.
Disks failed, machines rebooted, a rack may have lost power. Your application never noticed.
Not hardware that never fails. Systems designed so those failures rarely become your problem.
Multiply one machine's cost by the fleet you will eventually have.
If every incident needs an SSH session, the ops team becomes the ceiling.
Easier to test, easier to reason about, and failures stay put.
TCP was right for the internet and wrong for the datacenter.
Shuffle sharding bought 75 million fault domains from 100 machines.
Waste a little every second so the worst day looks like every other.
Then place the pieces next to what they manage.
At scale the one-in-a-billion path runs every day.
AWS's advantage isn't scale itself. It's having been forced to take these decisions seriously first.