BT·00HOOKBITTORRENT — PROTOCOL TEARDOWN

One Third of the Internet

2004 · a single protocol with no company, no datacenters, no servers behind it

35%
of ALL internet traffic
CACHELOGIC MEASUREMENT · 2004
BITTORRENT
ALL OTHER P2P
WEB TRAFFIC
share of internet traffic
more than the web ·
more than every other P2P net combined
one protocol
zero servers
zero companies
BT·01WHY IT MATTERSBITTORRENT — PROTOCOL TEARDOWN

The Spike That Makes You Faster

capacity fixed vs. capacity that grows with demand — the inversion, still in production today

CLIENT-SERVER
PEGGED
the spike saturates the only uplink
BITTORRENT SWARM
the same spike adds uplinks — the swarm gets faster
same crowd — opposite outcome
BT·02ANATOMYBITTORRENT — PROTOCOL TEARDOWN

File, Pieces, .torrent

the three artifacts every other mechanism builds on

THE FILE · e.g. 4 GB
SPLIT
PIECES — equal slices, 256 KB – 4 MB each · a big file ⇒ thousands of pieces
file.torrent — the metadata ─────────────────────────────── pieces : 0001 … N (ordered) sha1[0001] : a94f6c…03be sha1[0002] : 77c1e2…9d2a sha1[ ··· ] : one hash per piece tracker : http://tr.example/announce
download this first — everything starts here
BT·03ANATOMYBITTORRENT — PROTOCOL TEARDOWN

Peers, Tracker, Swarm

download anything and you become a peer — an equal, by design

TRACKER
keeps one list: who is active on this file ·
hands each newcomer ~40 addresses
never touches the file bytes
coordination only
THE SWARM — everyone active on this one file
SEEDER · has 100%
PEER · has 72%
LEECHER · has 35%
NEW PEER
“announce me”
peer list · ~40 addresses
joins the swarm
BT·04ROADMAPBITTORRENT — PROTOCOL TEARDOWN

Seven Things We’ll Unpack

each one is a technique you can lift into your own systems

1
THE CEILING
2
UNTRUSTED INTEGRITY
3
PIECE PICKING
4
INCENTIVES · FREE-RIDING
5
DHT · KADEMLIA
6
UTP · LEDBAT
7
PRODUCTION + LESSONS
BT·05THE PROBLEMBITTORRENT — PROTOCOL TEARDOWN

The Client-Server Ceiling

to distribute a file in 2001: host it, and pray — cost scales with users, capacity doesn’t

ONE SERVER · fixed uplink
EVERY BYTE × EVERY USER — from one place
SATURATED
capacity (provisioned in advance)
OUTAGE
wasted $
demand is sudden — capacity was decided months ago
a link from somewhere popular ⇒ the Slashdot effect — saturation at the exact moment everyone wants the file
BT·06THE INVERSIONBITTORRENT — PROTOCOL TEARDOWN

Every Downloader Is a Server

not a setting — a property of the protocol · downloading makes you a distributor

BEFORE
1 uplink ÷ 1000 clients
INTEGRITY
SCHEDULING
INCENTIVES
the inversion creates 3 new problems — each solved inside the protocol
AFTER — THE SWARM
the origin uploads each byte once — the swarm replicates the rest
capacity = Σ every peer’s uplink — it scales with participants
BT·07MECHANISM 1 · PIECESBITTORRENT — PROTOCOL TEARDOWN

Pieces

slice the file → thousands of concurrent sources · a download becomes an upload in seconds

① PARALLELISM
② INSTANT RE-SERVE
THE FILE
DOWNLOADER
piece #12
piece #87
piece #403
16 KB block requests — pipelined, never waiting on ACKs
already re-serving piece #87
⏱ a verified piece is served to others immediately
you’re a source within seconds of joining
BT·08MECHANISM 2 · INTEGRITYBITTORRENT — PROTOCOL TEARDOWN

Verify Content, Not the Sender

you can’t authenticate anonymous peers — so the data has to prove itself

ANY PEER · anonymous, interchangeable
a piece arrives
hash it locally — SHA-1
computed a94f6c…03be
expected a94f6c…03be
from the .torrent — written by the file’s creator
MATCH ✓ store it — and re-serve it
MISMATCH ✗ discard the piece
re-request it from another peer
optionally drop the sender
BT·09MECHANISM 3 · SCHEDULINGBITTORRENT — PROTOCOL TEARDOWN

Why Not Sequential?

piece 1, 2, 3… feels natural — and kills the swarm in three separate ways

① NOTHING TO TRADE
in-order ⇒ identical inventories — peers have nothing to swap
② SEEDER HOTSPOT
SEEDER
late pieces exist in one place — every request converges there
③ STUCK AT 99%
seeder leaves
pieces 1–3: thousands of copies
9–10: ZERO
the file is now permanently incomplete
BT·10MECHANISM 3 · SCHEDULINGBITTORRENT — PROTOCOL TEARDOWN

Rarest First

every client optimizes swarm health — which turns out to equal its own speed

COPIES PER PIECE — BEFORE
YOUR CLIENT
fetch the RAREST next
every peer broadcasts which pieces it has — your client counts copies
equalizes
AFTER — AVAILABILITY LEVELS OUT
the seeder uploads each piece ≈ once — the swarm replicates it forever after
🎲 exception: the very FIRST piece is random — you need something to trade, fast
BT·11MECHANISM 3 · SCHEDULINGBITTORRENT — PROTOCOL TEARDOWN

Endgame Mode

the last blocks get requested from everyone — first arrival wins, the rest are cancelled

99%
the last block — #1042 — is behind ONE slow peer
YOUR CLIENT
the same block → EVERYONE
first copy back wins
CANCEL
CANCEL
not even asked
WITHOUT ENDGAME
minutes at 99%
WITH ENDGAME
seconds
BT·12MECHANISM 4 · INCENTIVESBITTORRENT — PROTOCOL TEARDOWN

Free Riding

uploading costs money — so the rational move is: download everything, upload nothing

GNUTELLA · 2000 MEASUREMENT
70%
shared nothing
top 1% of hosts
~50% of responses
carry
no accounts · no admins · no control over the client binary — nothing to enforce with
WHAT THE NETWORK DEGENERATED INTO
a de-facto client-server topology — minus every guarantee of one
BT·13MECHANISM 4 · INCENTIVESBITTORRENT — PROTOCOL TEARDOWN

Tit for Tat

choking: upload fast and neighbors reciprocate — upload nothing and the swarm freezes you out

950 KB/s
720 KB/s
540 KB/s
310 KB/s
120 KB/s
60 KB/s
0 KB/s
YOU · 4 UPLOAD SLOTS
UNCHOKED — top 4 by recent rate
CHOKED
connection stays open · piece announcements still flow · file data doesn’t
⏱ every 10 s: re-rank all peers by recent download rate · unchoke top 4 · choke the rest
BT·14MECHANISM 4 · INCENTIVESBITTORRENT — PROTOCOL TEARDOWN

The Optimistic Unchoke

pure tit-for-tat can’t admit newcomers — one random slot every 30 s fixes the bootstrap

NEW PEER
0 pieces → 0 upload → choked by everyone → never a first piece
940 KB/s
720 KB/s
540 KB/s
THE OPTIMISTIC SLOT
one slot re-rolled every 30 s — rates ignored
first pieces
reciprocation follows
3 slots by rate · 1 by chance
THE OPTIMISTIC UNCHOKE
EXPLORE — sample a random stranger
EXPLOIT — keep the fastest four
sometimes the random stranger is faster than your slot #4 — it earns a regular slot on merit. exploration keeps the leaderboard honest.
BT·15DISCOVERYBITTORRENT — PROTOCOL TEARDOWN

The Last Central Component

as of 2005: a legal single point of failure as much as a technical one — trackers kept vanishing

SOLVED SO FAR ───────────────────────────────────── integrity a SHA-1 per piece scheduling rarest-first + endgame incentives tit-for-tat + optimistic unchoke discovery still needs the tracker
the last central piece
a peer list is trivial — until the machine keeping it disappears
WHILE IT’S UP
has an operator
has an address
has a hosting bill
one list — everyone finds everyone
THE DAY IT’S NOT
new peers
locked out
no new members can be found — the swarm slowly starves
BT·16MECHANISM 5 · DHTBITTORRENT — PROTOCOL TEARDOWN

The DHT · Kademlia

2005: Mainline DHT — a hash table with no table server · nodes and torrents share one 160-bit ruler

THE 160-BIT ID SPACE
0x3f…
0xb2…
0x71…
0xe9…
node IDs — picked at random on join
TORRENT INFOHASH — SHA-1 · also 160-bit
the nodes whose IDs sit closest to the infohash store this torrent’s peer list
lookup = hop to nodes ever-closer in ID space — a handful of hops across tens of millions of machines
key : infohash (SHA-1 of metadata) value : live peer list for that torrent table : spread over millions of nodes
BT·17MECHANISM 6 · MAGNETBITTORRENT — PROTOCOL TEARDOWN

Magnet Links

content addressing all the way down — no site, no file, just the fingerprint

① the takedown
② the magnet
③ peers rebuild the .torrent
.torrent on a website
404 — the last centralized artifact
collapses to
magnet:?xt=urn:btih:a94f6c…03be
the 40-char infohash — all you need
ask the DHT:
who has this hash?
each chunk verified against the hash
the .torrent —
rebuilt from the swarm
the file that describes the file is now also distributed
BT·18TIMELINEBITTORRENT — PROTOCOL TEARDOWN

2001 → a 40-Character String

the architecture underneath flipped: from addressing a place to addressing the data itself

2001
2005 · DHT
LATER · METADATA EXCHANGE
NOW · THE MAGNET LINK
site
.torrent
tracker
DHT
metadata → peers
a 40-char hash
WATCH EVERY DEPENDENCY DISAPPEAR
a URL addresses a location — a server · servers go down
a magnet addresses the content — anyone holding it can serve it
BT·19MECHANISM 7 · TRANSPORTBITTORRENT — PROTOCOL TEARDOWN

Bufferbloat

why torrents used to wreck your video calls — and it wasn’t about raw bandwidth

VIDEO CALL
GAME
BROWSER
TORRENT CLIENT
dozens of TCP connections at once
HOME ROUTER
THE BUFFER — packets waiting in line
your call — position 9 of 11 in the queue
drains ONE packet at a time
≈50 ms
≈400 ms
≈2 s
queue depth → delay
the link is not out of bandwidth — it’s full of queued packets · delay grows to seconds
QUEUEING DELAY
TCP: send faster until packets DROP
…but routers buffer first — queues fill before any loss
BT·20MECHANISM 7 · TRANSPORTBITTORRENT — PROTOCOL TEARDOWN

uTP · LEDBAT

≈2009: stop waiting for packet loss — measure queueing delay and yield like a background task

≈100 ms one-way delay — the target
backs off BEFORE crossing
t₁
t₂
t₂ − t₁ = queueing delay
every packet carries a timestamp — measure delay, not loss
rate += gain × (100 ms − delay)
delay rises toward 100 ms ⇒ the sender slows down — buffers never fill
link capacity
■ TCP · foreground
■ uTP · background
burst
burst
TCP bursts — take what they want
uTP fills the leftover
TCP is more aggressive — so uTP loses on purpose and yields the instant real traffic appears
BT·21IN PRODUCTIONBITTORRENT — PROTOCOL TEARDOWN

The Same Architecture, Everywhere

when demand is the load, demand is also the fleet

TWITTER · 2010
deploys to thousands of servers — 40 min → 12 s
FACEBOOK
large binaries to tens of thousands of machines — prefer same-rack peers
BLIZZARD · WOW
the patcher IS a torrent — patch day = the perfect flash crowd
UBER · KRAKEN 2018
Docker images across clusters of thousands of hosts, peer-to-peer
MICROSOFT
Delivery Optimization — Windows updates across 1B+ devices
MISTRAL · OPEN WEIGHTS
model weights: 10s–100s of GB, demand all at once — shipped as magnet links
BT·22TRADE-OFFSBITTORRENT — PROTOCOL TEARDOWN

What the Design Gives Up

every consequence has a shadow — know which ones you inherit before you borrow the pattern

① NO PRIVACY
14:02:11  83.44.19.7
14:02:12  121.8.33.90
14:02:13  190.2.77.4
+1,204 more
to download you must announce — any participant can log every IP it sees
② THROUGHPUT ≫ LATENCY
player needs
04
05
06
07
08
09
swarm fetches
61
07
33
90
12
58
rarest-first is the opposite of sequential — poorly suited to streaming
③ AVAILABILITY FOLLOWS DEMAND
flash crowd
last volunteer
if they leave, the file dies
flash crowd over → only volunteers keep it alive
④ OPEN MEMBERSHIP CUTS BOTH WAYS
one attacker
1 machine → 10,000 IDs
lookup deflected to a fake
thousands of fake IDs — Sybil attacks bias lookups, poison records
BT·23LESSONSBITTORRENT — PROTOCOL TEARDOWN

Five Transplants for Your Systems

none of these require a P2P network — they’re design moves

I
MAKE DEMAND CARRY SUPPLY
every new peer starts seeding within seconds of joining
II
INCENTIVES ARE ARCHITECTURE
choking + the optimistic unchoke — cooperation made rational
III
VERIFY CONTENT, NOT THE CHANNEL
self-verifying data can travel ANY path: mirrors, caches, peers
IV
OPTIMIZE THE SYSTEM, NOT THE NODE
rarest-first: worse for the node, best for the network — then for the node too
V
LET BACKGROUND TRAFFIC YIELD
use leftover bandwidth · never fight the traffic people care about
BT·24RECAPBITTORRENT — PROTOCOL TEARDOWN

The Whole Story

six mechanisms, one property — a protocol with no company carried a third of the internet

verify
schedule
trade
discover
transport
deliver
PIECES
contribute within seconds
HASHES
safe trade with strangers
RAREST-FIRST
every piece stays replicated
TIT-FOR-TAT
contributing is the rational move
DHT
no one left to shut down
uTP
coexists with everything else
2004 · 35% of the internet
zero servers · zero companies
EVERY DOWNLOADER IS ALSO AN UPLOADER
great architectures don’t maximize features — they maximize consequences
100%
+