CS·00THE HOOKCROWDSTRIKE — THE 78-MINUTE OUTAGE

The 78 Minutes

friday · july 19, 2024 — a security company pushes a small rules file to every machine it protects

04:09 · PUSH
05:27 · REVERT
78 MINUTES LIVE
CROWDSTRIKE CLOUD
CHANNEL FILE 291
THE FLEET
THE BOOT LOOP
04:09 UTC — the push begins
~40 KB · configuration — not code
8.5M
machines — every online falcon endpoint
crash → reboot → crash → …
…and at 05:27 they pulled it back — but the machines could no longer stay alive to hear it
CS·01THE DAMAGECROWDSTRIKE — THE 78-MINUTE OUTAGE

Planes, Hospitals, 911

aviation, healthcare, banking, broadcast and emergency lines — all in the same morning

one file — 04:09 UTC · everywhere at once
OFF AIR
flights canceled
surgeries postponed
banks froze
911 lines down
sky news off-air
$5.4B
fortune 500 losses
a sliver of it insured
DELTA
five straight days · ~$500M claimed
handwritten boarding passes at the gates — for days
CS·02THE HOOKCROWDSTRIKE — THE 78-MINUTE OUTAGE

Not a Cyberattack · One Number Wrong

through the kernel, through the driver, through the file — down to a 21 where the code delivered a 20

not a cyberattack — the file came from the company these machines were paying to protect them
THE KERNEL — one trust domain
THE DRIVER — csagent.sys
CHANNEL FILE 291 — the rules
THE FAULT — MAGNIFIED
21
20
EXPECTED
DELIVERED
an out-of-bounds read — coming up
the biggest outage in history is an out-of-bounds array read
CS·03THE TARGETCROWDSTRIKE — THE 78-MINUTE OUTAGE

The Most Important 1%

falcon doesn’t sit on home laptops — it sits on the machines companies install when they’re big enough to be a target

ALL THE WINDOWS MACHINES ON EARTH
<1%
of windows machines — microsoft’s own number
THE 1% THE WORLD RUNS ON
~¼ of the fortune 500 took a direct hit
airline gates
hospital records
bank workstations
trading floors
the kiosk · the departure board · the POS terminal · the MRI screen — underneath, windows with falcon on it
CS·04THE SOFTWARECROWDSTRIKE — THE 78-MINUTE OUTAGE

What Falcon Actually Is

it watches behavior live — every process, file, connection and message — and blocks attacks as they happen

FALCON — EDR · antivirus that grew up
process starts
file writes
network connections
IPC messages
blocked — mid-attack, in real time
it must see everything
it must act first
it must be un-killable
ALL THREE POINT AT THE KERNEL
CS·05THE KERNELCROWDSTRIKE — THE 78-MINUTE OUTAGE

Two Worlds · User Mode and Kernel Mode

any bug, in any kernel driver, from any vendor — by design — is fatal to the entire machine

USER MODE — walls everywhere
chrome
your code — dies alone
excel
touch bad memory → windows kills that process · everything else keeps running
KERNEL MODE — no walls
scheduler
memory mgr
drivers
one shared memory space · one trust domain — kernel corrupt → HALT
fail-stop: the blue screen is a design decision — not a failure
CS·06THE COLLISIONCROWDSTRIKE — THE 78-MINUTE OUTAGE

Frozen Code vs. Hourly Threats

kernel code is supposed to be rare, boring and certified — detection logic changes every single day

WHQL — MICROSOFT’S KERNEL GATE
weeks
tested · signed · certified — then FROZEN
ATTACKER TRADECRAFT — THE STORM
new techniques — hourly
customers expect the detection today — speed is the product
?
daily logic · frozen ground
how do you ship hourly logic into a place where code cannot change?
CS·07THE TRICKCROWDSTRIKE — THE 78-MINUTE OUTAGE

An Interpreter in the Kernel

split the engine from the fuel — every serious EDR on windows works this way

csagent.sys — THE ENGINE · certified once
an interpreter inside the driver — reads data, changes behavior, zero bytes of signed code change
CHANNEL FILES — data, not code
TEMPLATE TYPES — code
one per behavior category · lives in the sensor · ships the slow, certified way
TEMPLATE INSTANCES — data
the specific rules · ship as channel files · constantly — the fast path
FILE 291 — named pipes · IPC
attackers love named pipes · it changes often — remember the number
microsoft certified the engine — the fuel is on crowdstrike’s honor
CS·08THE PIPELINECROWDSTRIKE — THE 78-MINUTE OUTAGE

Two Lanes · And One Has No Brakes

code travels the guarded causeway — content travels the ground, straight through, past everything

CROWDSTRIKE — BUILD & CONTENT
THE SLOW LANE — code
full QA
dogfood first
staged rollout
CUSTOMER CONTROL
pinned at N−1 · N−2 ✓
THE FAST LANE — content
content validator — one automated check
the ENTIRE fleet
every customer · instant
ignores customer staging — no brakes
CS·09THE BUGCROWDSTRIKE — THE 78-MINUTE OUTAGE

February 2024 · An Off-By-One Is Born

two components, built to interlock — misaligned by exactly one

SENSOR 7.11 · FEB 2024
THE DEFINITION — promises 21
THE CODE — delivers 20
21
20
the 21st — promised, never received
FEB
MAR
APR
MAY
JUN
ARMED
five months · on every machine
CS·10THE CAMOUFLAGECROWDSTRIKE — THE 78-MINUTE OUTAGE

The Wildcard That Hid It

the gaze never falls on slot 21 — so nothing ever goes wrong

THE INTERPRETER — reads what it must
FIELDS 1–20
✱
slot 21 — the wildcard
match anything → nothing to compare → never read
VALIDATOR · 21 = 21 ✓
✓
THE CONTRACT
SENSOR CODE · consistent ✓
✓
checked by nobody
CS·11THE TESTSCROWDSTRIKE — THE 78-MINUTE OUTAGE

Why Every Test Passed

five months of green — every orbit that existed was safe

FEB → JUN — every test green · all wildcards
FEB ✓
MAR ✓
APR ✓
MAY ✓
JUN ✓
JUL — the input that didn’t exist
THE SENSOR — deployed · broken · untouched
your test suite samples reality — it cannot sample the config file somebody writes in july
CS·12THE NIGHTCROWDSTRIKE — THE 78-MINUTE OUTAGE

July 19 · 04:09 UTC

a real response to a real attack — one gate, no brakes, straight to the world

named-pipe attacks — in the wild
a new instance — field 21: a REAL value
✓
validator: 21 = 21 · PASS
04:09:00 UTC — THE PUSH
every online falcon machine on earth begins downloading
CS·13THE CRASHCROWDSTRIKE — THE 78-MINUTE OUTAGE

The Read Past the End

field 21: go fetch input 21 — the array has twenty entries

file lands → sensor loads → a named-pipe event fires — constantly
1
5
10
15
20
compare · compare · compare …
21
fetch input 21 — past the end
memory it does not own
page fault — unsatisfiable
PAGE_FAULT_IN_NONPAGED_AREA
blamed on csagent.sys
user mode: the process dies, life goes on · kernel mode: fail-stop — the machine executed its final responsibility perfectly
sydney — watched it live
europe — waking up to it
america — asleep
CS·14THE LOOPCROWDSTRIKE — THE 78-MINUTE OUTAGE

Locked Out of the Fix

reboot · load driver · parse file · crash — forever

① BOOT
② LOAD DRIVER — boot-start · early, by design
③ PARSE 291 — the poison on disk
④ CRASH — before the network exists
sealed
05:27 — THE FIX, LIVE ON THEIR SERVERS
80 ms away
the network never comes up
the outage sealed itself in
CS·15THE RACECROWDSTRIKE — THE 78-MINUTE OUTAGE

Reboot — Up to 15 Times

the parser starts earlier and has less to do — but boot isn’t deterministic

POWER ON — the race starts
THE PARSER
driver loads early
parse file 291
THE UPDATER — long track
network stack rises — slow
updater fetches the fix
channel file 291 — whoever touches it first, decides
fixed — done · never crashes again
15×
reboot — race again
official microsoft guidance — in writing
“reboot the machine —
up to 15 times”
re-run the race · the network might win
some machines actually won ✓
CS·16THE RECOVERYCROWDSTRIKE — THE 78-MINUTE OUTAGE

8.5 Million Keyboards

the fix is thirty seconds — multiplied by every machine, by hand

boot into safe mode / recovery
system32 → drivers → crowdstrike
delete the C-00000291 files
reboot — clean
C:\\Windows\\System32\\drivers\\CrowdStrike\\C-00000291*.sys
30 s × 8,500,000
remote tools need a booted OS — the one thing these machines will not do
one human · one machine at a time
remote management
CS·17THE TRAPCROWDSTRIKE — THE 78-MINUTE OUTAGE

BitLocker · The Circular Lock

encrypted disks demand the recovery key — and the key servers were blue too

the machine — needs its key to enter safe mode
48-digit bitlocker recovery key
active directory — where the keys are escrowed
the key you need to unlock the fix — locked inside the thing that needs fixing
USB sticks · desk to desk
ladders in data centers
recovery keys, printed in bulk
zzz
the survivors: machines that were off — asleep during the 78 minutes
CS·18THE BILLCROWDSTRIKE — THE 78-MINUTE OUTAGE

What It Cost

the damage — and the strange epilogue

WHAT 78 MINUTES COST
8.5M
devices · an undercount
$5.4B
fortune 500 · a sliver insured
5 DAYS
delta — $500M · lawsuits
−⅓
stock, in the weeks after
and yet — most stayed
switching costs · the monoculture holds
CS·19THE PLATFORMSCROWDSTRIKE — THE 78-MINUTE OUTAGE

Why Macs Didn’t Die · Why Linux Didn’t

the blast radius of july 19 was shaped, in part, by a legal settlement from fifteen years earlier

macOS — 2019: everyone out of the kernel
falcon dies → the mac keeps running ✓
endpoint security framework · user space
LINUX — eBPF · a verifier at the door
proves all paths — out-of-bounds cannot load
zelkova’s idea, aimed at kernel code
WINDOWS — vendors inside ring zero
a parser bug = everyone’s friday
2009 EU commitment: vendors get defender-level kernel access
same company · same product · same day — zero crashed macs, zero crashed linux boxes
CS·20TRADE-OFFSCROWDSTRIKE — THE 78-MINUTE OUTAGE

None of Them Comfortable

the fast lane exists for a reason · the kernel exists for a reason · the monoculture happened for a reason

① SPEED IS THE PRODUCT
staged rings mean someone sits unprotected against a live campaign — for a day
② KERNEL POWER vs FRAGILITY
inside: un-killable and fatal · outside: survivable and weaker — pick
③ THE MONOCULTURE
best product → biggest install base → one shared fault domain
your vendor choices are fault-domain choices — nobody draws them on the diagram
CS·21THE FIXESCROWDSTRIKE — THE 78-MINUTE OUTAGE

Read Them Backwards

three layers — and they get progressively more interesting

read crowdstrike’s fix list backwards…
SHALLOW — bolt the brakes on
bounds checks · staged rings · customer control — the slow lane, finally, for content
DEEPER — leave the kernel
windows resiliency initiative: a crashing agent = a dead process, not a dead machine
DEEPEST — prove, don’t test
a verifier in front of the interpreter — the poisoned rule cannot load
…it’s a confession — the slow lane had all of it, for code
CS·22LESSONSCROWDSTRIKE — THE 78-MINUTE OUTAGE

Five Lessons

what to take straight into your own systems

1
CONFIG
IS CODE
if a file changes behavior, it deserves review, tests, canary, rollback — everything code gets
2
BLAST RADIUS =
DEPLOYMENT RADIUS
anything reaching 100% of your fleet in under an hour is your biggest single point of failure
3
VALIDATE CONTRACTS,
NOT ARTIFACTS
the instances were valid · the code was consistent · the boundary was checked by nobody
4
TESTS ONLY KNOW
INPUTS THAT EXIST
the fatal input hadn’t been written — fuzz your content pipeline, test the schema’s edges
5
PLAN FOR THE MACHINE
THAT CAN’T HELP
no network · no agent · encrypted disk — which runbook steps assume a booting OS?
five checks — before your next fleet-wide anything
CS·23THE CLOSERCROWDSTRIKE — THE 78-MINUTE OUTAGE

One File · 78 Minutes · 8.5 Million Machines

your infrastructure is full of fast lanes

your EDR
observability
MDM
container runtime
past your change management
YOUR FLEET
the region map —
the diagram we draw
the update channels —
the diagram nobody draws
the next global outage won’t come from a region — it’ll ship through one of these pipes, at four in the morning, on a friday
−
100%
+
✎
▮
●
↶
⌫