“Instagram runs on sharded Postgres” is one of the most repeated sentences in system design, and most of the retellings stop at the 2011 blog post: thousands of logical shards, a clever 64-bit ID, done. Read the thirty-one sources together, from Mike Krieger’s 2011 post and his Hacker News replies through the 2015 multi-data-centre write-up to the 2017 and 2018 talks, and a more useful story appears. Instagram did not build a sharding system. It built three small primitives: IDs that carry their own location, a logical-to-physical map owned by the application, and physical streaming replication as the only way data ever moved. It then reused those primitives for the hard problems of the next four years: resharding, leaving AWS, and going multi-region. The thesis of this case (our reading of the record): Instagram’s Postgres tier scaled because its designers refused to add moving parts. The same refusal left it with manual failover, a single write region and no way to split a hot user, and none of those limits was fixed inside Postgres. By 2017 the core data had moved to Facebook’s TAO, for reasons Instagram gave as developer velocity and self-serve data modelling for a much larger team.
> 60 GB
photosdb vs 68 GB max EC2 RAM, 2011 [S6]
41 · 13 · 10
bits: time · shard · sequence [S1]
< 2 h
whole PG fleet cut to VPC, two engineers [S15]
10 → 60 ms
inter-DC latency, 2015 [S19]
Read it beside the Figma sharding and Uber Postgres-to-MySQL case studies: Figma sharded Postgres a decade later with a proxy and a large team, Uber left Postgres over replication and write amplification. Instagram is the earliest and smallest of the three. The WAL, replication and index internals referred to below are covered in the Databases from the Inside track.
Instagram launched in 2010 on one machine in Los Angeles that Krieger’s 2012 slides describe as “less powerful than my MacBook Pro”[S6]. It took 25,000 signups on day one[S6], and the host quoted four days (48 hours rushed) for a second server, so the company moved to AWS[S18]. The storage choice was made for a reason that had nothing to do with scale. Krieger’s 2012 slide on it reads, in full, “why pg? postgis”[S6]. Nobody chose Postgres as a sharding substrate; it became one because it was already there and the team knew it.
By December 2011 the company had 14 million users, three engineers and more than 100 EC2 instances[S4]. The stack, reconstructed below from the “What Powers Instagram” post, Krieger’s 2012 slides and his Hacker News replies, is ordinary for its era, and deliberately so (“optimize for minimal operational burden; instrument everything”[S6]).
2010
Launch: 25,000 signups on day one, one LA server, then a move to AWS
S6 S18
2010–11
Django ORM on Postgres (“why pg? postgis”); photos app split to its own photodb
S6
2011
photosdb > 60 GB against 68 GB of RAM on the biggest EC2 box
S6
~30 Sep 2011
“Sharding & IDs”: thousands of logical shards as schemas, 64-bit IDs from PL/pgSQL
S1 S2
Dec 2011
14M+ users, 3 engineers, 12 Postgres masters + 12 replicas in another zone
S4
Apr 2012
Android: >1M users in 12 h. Facebook acquisition announced 9 Apr
S5 S29
29 Jun 2012
us-east-1 power event: “almost half” of instances down, 36 h to rebuild
S8 S18
Late 2012
Chef, WAL-E instead of EBS snapshots; Cassandra takes a Redis audit log
S9 S11 S18
Jan 2013
>10,000 likes/s at peak, “fundamental storage technology hasn’t changed”
S9
Apr 2013–Apr 2014
EC2 → VPC → Facebook DCs. Whole Postgres fleet cut to VPC in < 2 h
S13 S15
2015
1 → 3 data centres; regional replicas, cross-DC writes, PgQ invalidation. 400M MAU
S19 S20
2017
QCon Q&A: “when we were still managing Postgres … we don’t anymore”; TAO
S25
Oct 2018
Stateful-service examples given: Cassandra and TAO; 3 → 5 DCs “trivial”
S28
Three constraints shaped everything that follows, and none of them was throughput.
The disk. Postgres ran on EBS volumes striped with mdadm software RAID, formatted XFS so the arrays could be frozen for consistent snapshots[S4]. The post is blunt about the limitation: EBS “doesn’t support enough disk seeks per second, so having all of our working set in memory is extremely important”[S4]. That sentence defines the capacity model. On a 2011 EC2 database the unit of capacity was not CPU or IOPS but RAM that could hold the hot set. A query that missed the page cache did not just get slower; it competed for a seek budget shared with every other miss on the volume. The team even used vmtouch to manage what was in memory when failing over from one machine to another, copying the old machine’s page-cache profile onto its replacement so it would not start cold[S4]. That trick exists only when a cold cache is itself an outage; Section 2 puts numbers on why it was one.
The machine ceiling. The largest EC2 instance had 68 GB of RAM[S6]. There was no bigger box to buy, no local NVMe, and no managed database that would take the problem away. Vertical scaling had a hard, published limit.
The team. Two engineers in 2010, three in 2011, five in 2012[S6]. Krieger’s own framing of the sharding decision is that they had assumed “surely we’ll have hired someone experienced before we actually need to shard,” and then learned that “you don’t get to choose when scaling challenges come up”[S6]. Any design that needed a new always-on service (a coordinator, an ID server, a proxy) was a design that needed someone to be woken up for it.
Edge
Route53 DNS
Amazon ELB
SSL terminates here
3 × nginx
was 2 behind DNS round-robin
App tier
Django on Gunicorn
> 25 High-CPU XL · CPU-bound
Upload + resize
own servers, then S3 / CloudFront
Gearman
~200 Python workers: fan-out, shares, notifications
Memory stores
Redis (master/replica)
main feed · activity · sessions
Redis hash buckets
300M legacy media → user IDs
memcached × 6
pylibmc / libmemcached
Relational
PgBouncer
“huge impact on performance”
12 Postgres masters
Quadruple XL memory · thousands of schemas
12 streaming replicas
different AZ · repmgr
EBS in mdadm RAID + XFS
frozen for frequent snapshots
Search / ops
Solr
geo-search, after media sharded
Munin · Sentry · statsd/Graphite
Pingdom · PagerDuty · PGFouine
Everything here ran on 100+ EC2 instances operated by three engineers[S4]. The Postgres tier is the only one whose data placement the sources describe as owned by the application. Redis ran as master/replica pairs; memcached was reached through pylibmc/libmemcached, which (our inference) left key placement to the client library.
Our analysis
The sequence is documented in the 2012 slides. First, the database moved to its own machine[S6]. Then the team vertically partitioned it, using Django database routers to send the photos app to its own photodb, which works “once you untangle all your foreign key relationships”[S6]. Then, “a few months later,” photosdb > 60GB, against “only 68GB of RAM on biggest machine in EC2”[S6]. Horizontal partitioning was now unavoidable.
The 68 GB and 60 GB figures are reported[S6]; the usable-cache bar and the whole lower panel are our illustrative model, with uniform access over the hot set. Skewed access moves the cliff to the right but does not remove it. Throughput at the time (25 photos and 90 likes per second[S1]) plays no part.
It is worth being precise about what was not the problem. When the sharding post went out in late September 2011, Instagram was handling “more than 25 photos and 90 likes every second”[S1]. A single 2011 Postgres host handles 90 single-row inserts per second without noticing. Even allowing for a peak-to-average ratio of several times, write throughput was one to two orders of magnitude below anything that would force a split (our estimate). The trigger was capacity in the storage sense: the hot set was about to exceed the only fast tier the platform offered.
Our analysis
shared_buffers and in the kernel page cache) all come out of the same 68 GB. Second, disk reads scale with the miss fraction, and the miss fraction rises from zero. With uniform access, disk reads ≈ R × (1 − cache/hot set). If the host sees an illustrative 20,000 page reads per second and the striped EBS volumes sustain an illustrative 1,000 random reads per second, the disk saturates when the hot set exceeds the cache by 1/(1 − 1,000/20,000) ≈ 5.3%. Past that, queueing theory takes over: at utilisation ρ the wait grows like 1/(1 − ρ), so every query that misses waits behind every other miss, and p99 latency goes vertical within a few percent of growth. On a platform growing weekly, “a few percent” is days. The same arithmetic explains vmtouch: a failover target with a cold cache has a hit ratio near zero, which in this model is twenty times the disk budget. Pre-warming it was not an optimisation; without it a failover was an outage with extra steps.Our analysis
Why did read replicas not buy time? Instagram ran streaming replicas in another availability zone by December 2011[S4]. But (our analysis) a streaming replica is a byte-for-byte copy of the primary, so it needs the same RAM to hold the same hot set. Replicas multiply read capacity; they do not divide the working set. On a platform where the working set was the constraint, adding replicas moved the wall nowhere. That is plausibly why vertical partitioning came first. It is the only pre-sharding move that actually shrinks what each host must keep hot, and it ran out as soon as a single application’s tables (photos) were bigger than a host.
NoSQL was evaluated and rejected as the primary store[S1][S6]; Section 9 examines whether that was right.
The published design has four parts, spread across one blog post, eleven Hacker News comments and a slide deck. No single source puts them together.
Many logical shards, few physical hosts. Data is pre-split into “many many many (thousands) of logical shards that map to fewer physical ones”[S6]. The post says “several thousand”[S1]. Its worked example uses 2,000, Krieger’s HN comment uses 1,000 and the slides use 8[S1][S2][S6]. The real production count was never published. The 13-bit shard field caps it at 8,192. Range-based splitting in the MongoDB style (A–H, I–Z) was rejected because it moves data on every split, which on EC2 meant disk I/O the platform could not spare[S6]. Pre-splitting meant they could “start with just a few database servers” and later move whole logical shards “without having to re-bucket any of our data”[S1].
Each logical shard is a Postgres schema. Every sharded table (the example is likes) exists once in every schema. Table names only need to be unique per schema[S1]. Krieger called the schema feature “one huge benefit to PostgreSQL,” because “all our logical shards all live inside one database,” where MySQL would have needed separate databases or table-name prefixes[S2].
Routing is two dictionary lookups in the application. “We hash on user ID (in most cases) and then look the shard # in the dict, then look up the shard # in a logical-to-physical dict”[S2]. In HN shorthand: user_id % 1000 → schema ID; schema ID → database ID; then select FROM schemaID.tablename[S2]. The map was “a lookup dict in our Django app--which involves brief downtime just to update the shard map when moving the data”[S2].
IDs carry the logical shard. Every new primary key embeds 13 bits of logical shard ID, so an object ID alone is enough to route (Section 4).
By owner · most queries
user_id
request context
logical shard #
e.g. user_id % 1000 [S2]
logical → physical map
a Python dict in Django [S2]
PgBouncer → host D
one Postgres DB
SELECT … FROM insta1234.likes
schema = logical shard
By object · IDs minted after sharding
media_id (64-bit)
minted by next_id()
logical shard #
“contain the shard ID” [S1]
same dict → host → schema
no lookup service needed
By object · IDs minted before sharding (legacy clients)
old media_id
no shard bits
Redis hash buckets
~300M media → user mappings [S3]
user_id
then the owner path above
The bit arithmetic in the middle row is our reconstruction from the published layout (41 time, 13 shard, 10 sequence bits[S1]). The bottom row is the reported cost of IDs that did not carry their location: a separate, permanently maintained mapping store.
The bottom row of that diagram is the part retellings drop. Objects created before the ID scheme existed had IDs with no shard bits, and old clients still held them. Instagram had to keep roughly 300 million photo-to-user mappings “in order to know which shard to query,” and it stored them in Redis hashes because a SQL database “seemed like overkill” for insert-only, non-transactional data with no relations[S3]. The Redis post is famous as a memory-optimisation trick: bucketing keys into hashes of 1,000 entries took memory from about 70 MB to about 16 MB per million keys, so 300 million keys fit in “just under 5GB” on an instance about a third of the cost of the one otherwise needed[S3]. Read in context, it is the maintenance bill for the IDs that did not carry their location.
Reconstruction (inferred)
wait_callback hook to issue queries on several connections at once, “useful for fan-out queries that hit multiple nodes”[S9]. Nothing in the sources describes a global secondary index. For access by any key other than the owner (likes by media rather than by user, for example) the sources are silent. The embedded shard ID in media IDs is the likely answer for anything keyed by a media ID, since routing by media ID then needs no index at all.Our analysis
wait_callback[S9] fixes the latency sum, not this maximum. The practical threshold: any access path that fans out on a hot request should be denormalised onto the owner’s shard or served from a cache before the host count reaches the tens, because past that point no amount of per-host tuning brings its p99 back.The second-order consequence of schemas as shards is density. “Several thousand” schemas on a dozen masters[S4] is a few hundred per host. With an illustrative 20 sharded tables and three indexes each, that is around 16,000 relations per host: real catalog, relcache and dump-time overhead. The benefit is easy to miss. Every table and B-tree is small, vacuum and pg_reorg run on small objects, and the right-edge insert hot spot of a time-ordered index is spread across hundreds of indexes per host. The design traded catalog overhead for thousands of small independent units, the right trade for a three-person team.
The ID requirements were stated plainly: sortable by time, ideally 64 bits “for smaller indexes, and better storage in systems like Redis,” and “as few new ‘moving parts’ as possible”[S1]. The solution was to mint IDs inside Postgres itself: 41 bits of milliseconds since a custom epoch, 13 bits of logical shard ID, and 10 bits of a per-table sequence taken modulo 1024[S1]. The function is the column default, and the ID comes back through RETURNING on the insert[S1]. There is no extra round trip, no extra service, and no way for an ID to be minted on the wrong shard, because the shard ID is a constant compiled into each schema’s copy of the function.
-- As published in "Sharding & IDs at Instagram" (2011) [S1], with one change:
-- the original shows "%%" (almost certainly a string-escaping artefact);
-- PostgreSQL's modulo operator is "%".
CREATE OR REPLACE FUNCTION insta5.next_id(OUT result bigint) AS $$
DECLARE
our_epoch bigint := 1314220021721;
seq_id bigint;
now_millis bigint;
shard_id int := 5;
BEGIN
SELECT nextval('insta5.table_id_seq') % 1024 INTO seq_id;
SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
result := (now_millis - our_epoch) << 23;
result := result | (shard_id << 10);
result := result | (seq_id);
END;
$$ LANGUAGE PLPGSQL;
CREATE TABLE insta5.our_table (
"id" bigint NOT NULL DEFAULT insta5.next_id(),
-- ...rest of table schema...
);Four properties of this function are routinely misstated, including in the original post.
The lifetime is neither 41 years nor 69. The post says 41 bits “gives us 41 years of IDs”[S1]. As arithmetic, 2⁴¹ milliseconds is about 69.7 years, which with the published epoch runs to about April 2081. But the result is a Postgres bigint, which is signed, and the time field occupies the top 41 bits including bit 63.
Our analysis
our_epoch = 1314220021721 (24 August 2011, 21:07:01.721 UTC) that happens around 27 June 2046, about 34.8 years after the epoch. PostgreSQL’s bigint shift operator does not raise on overflow, so the function would keep returning values, now negative. Every ID minted after that date would sort before every ID minted before it, which silently breaks any “ORDER BY id DESC” used as a proxy for recency, and any keyset pagination built on it. The fix is trivial if made early (shift one bit less, or treat the value as unsigned in every consumer) and very expensive if made after twenty years of IDs have been handed to clients. Anyone copying this layout today should decide the sign-bit question explicitly, because the original did not.The throughput ceiling is per table, per logical shard. The post says the layout gives “1024 IDs, per shard, per millisecond”[S1]. The sequence is “unique to each table in each schema”[S1], so the true limit is 1,024 inserts per millisecond into one table in one logical shard, about a million per second. No single Postgres host of the era could approach that for one table, so the ceiling never binds. The real failure mode is different. Past 1,024 inserts in a millisecond, or after a clock step backwards, nextval() % 1024 can repeat a value already used for the same millisecond. Krieger’s answer was that “at worst... you’d have a duplicate key when trying to insert, and can re-try without the risk of having a duplicate ID floating around your system”[S2]. That is correct, and it is correct only because the ID is that table’s primary key, so the unique index is the collision detector. The same scheme used for an ID that is not a unique key somewhere would produce silent duplicates.
Uniqueness is per table, not global. This follows from the same fact and is almost never mentioned.
Our analysis
Ordering is by clock, and only roughly. Krieger: “We don’t need the IDs to be exactly sortable, only roughly sortable within a second or so. As long as the clock doesn’t move backwards on any given machine (we use ntpd in its gradual-adjustment mode), the IDs are unique”[S2]. IDs minted on different hosts are ordered by different clocks, so a merge of IDs from many shards (a feed, for instance) is correct to within the clock skew between database hosts. For a photo feed that is irrelevant. For anything that uses ID order as a causality signal (a message thread, a ledger) it is a bug waiting for a bad NTP day. The Distributed Systems Deep Dive track covers why physical clocks cannot give you causality; this is a concrete instance of that.
Two smaller errors matter if you work from the post. Its worked example assumes a 1 January 2011 epoch and a 9 September offset of 1,387,263,000 ms, which is about 16 days; the code’s epoch is actually 24 August 2011[S1]. And the published PL/pgSQL shows modulo as %%, which is not valid Postgres; the listing above uses %.
Size
Time-sortable
New moving parts
Carries location
Reported verdict
UUID / ObjectId
96–128 bits
some are random
none
no
too big for Redis/indexes [S1]
Ticket DB (Flickr)
64 bits
not across several DBs
ticket DB hosts
no
SPOF with one, unsortable with many [S1]
Redis INCR
64 bits
hard across instances
Redis as critical path
no
SPOF, durability window [S2]
Composite (shard, seq)
2 columns
per shard only
none
yes
less portable, bigger in Redis [S2]
Snowflake
64 bits
yes
ZooKeeper + ID servers
worker, not data
“came the closest” [S1]
PL/pgSQL next_id()
64 bits
~1 s
none
yes (13 bits)
chosen [S1]
The criteria are Instagram’s own: time-sortable, 64 bits, “as few new ‘moving parts’ as possible”[S1]. The “carries location” column is ours. It is the property that paid off most in later years, and the only one on which the chosen design beats Snowflake outright.
The options and verdicts in the table are Instagram’s own[S1][S2]. Snowflake “came the closest, but the additional complexity required to run an ID service was a point against it”[S1].
Counterfactual
The shard-move procedure is described in one HN comment: “use PostgreSQL’s built-in streaming replication to create an exact, in-sync copy of a set of tablespaces, then ‘fail over’ to a new machine and start reading/writing to a subset of those tablespaces (this is similar to how Facebook describes their shard-moving process)”[S2]. The 2012 slides give the example: eight logical shards on machines A and B become four machines by replicating A to a new machine and B to another, then editing the dict[S6].
Before · user_id % 8
A
B
{0:A,1:A,2:A,3:A,4:B,5:B,6:B,7:B}
Copy · streaming replication
A → C (replica of A)
exact, in-sync physical copy of all of A
B → D (replica of B)
no row is re-bucketed or rewritten
“brief downtime” to update the shard map [S2]; promote C and D, repoint (order inferred)
After · same hash, new map
A
C
B
D
{0:A,1:A,2:C,3:C,4:B,5:B,6:D,7:D}
The dictionaries are verbatim from the slides[S6]. What happens to the now-unserved schemas (2 and 3 still on A, 0 and 1 on C) is not described in any source; presumably they were dropped later to reclaim space. Until then each new host carries a full copy of its parent, which is the cost of using physical replication as the move primitive.
The consequences of choosing physical replication as the move primitive are worth spelling out, because they explain both the speed of later migrations and the limits the design never escaped.
Moves are host splits. A streaming replica copies the whole cluster. You cannot replicate “schemas 2 and 3” to a new host; you replicate everything and then stop using the half you do not want. The natural resharding step is therefore a split: one host becomes two, each carrying a full copy of the parent and serving half the schemas. To relocate 50% of a host’s data you copy 100% of it. The unserved half stays on disk until someone drops it, and no source says when or how that happened.
Moves carry no correctness risk in the data. Nothing is re-bucketed or re-inserted; indexes, sequences and constraints arrive intact. No dual writes, no backfill, no reconciliation. The only new code path is the dict change.
The fence is the downtime. Krieger said moving data “involves brief downtime just to update the shard map”[S2].
Reconstruction (inferred)
Reported: moves use streaming replication then a “fail over” to a subset of schemas[S2], with “brief downtime just to update the shard map”[S2]. The phase order, the three-term pause and panel B are our reconstruction; widths are not to scale.
Our analysis
The granularity is fixed forever. The modulus never changes, so one logical shard is the largest unit that can ever be placed, and it cannot be divided. Krieger named the consequence in 2013: “it’s harder to split off a single user if you need to special case”[S10].
The rest is craft aimed at the same constraint, the hot set. The January 2013 “5 tips” post reported over 10,000 likes per second at peak, and that “our fundamental storage technology hasn’t changed”[S9].
Partial indexes. Tag prefix search (LIKE 'snow%' ORDER BY media_count DESC LIMIT 10) went from 215.275 ms to 3.137 ms, about 69 times faster, with an index restricted to tags with media_count >= 100. The planner’s row estimate for the sort dropped from 15,455 to 169[S9]. The deeper point is that the partial index excludes the long tail of rare tags, so the index that has to stay in RAM is much smaller than the table. On a working-set-bound fleet, index size is capacity.
Prefix (functional) indexes on long tokens such as 64-character base64 strings produced an index “1/10th the size”[S9]. (The published statement, substr(token), 0, 8, is malformed; the intent is substr(token, 0, 8).) Same logic: an index a tenth the size is a tenth of the RAM.
pg_reorg rewrote tables in index order, for example so all of one user’s likes are stored together, and the team reported “dozens of reorgs across hundreds of machines without issues”[S9]. Physical clustering by owner means a user’s rows touch a handful of pages instead of hundreds, so the pages that must be hot for active users shrink. (The post also says the tool takes an exclusive lock at the start, so it is not fully lock-free[S9].)
Our analysis
Autocommit for reads. Using psycopg2’s autocommit for read-only queries “lowered chatter between our application servers and DBs significantly, and lowered system CPU,” and with PgBouncer it let connections return to the pool sooner[S9]. Rick Branson added that autocommit was used only for read-only queries, and that the read/write skew was “50:1”[S10]. The likely mechanism (our analysis): the driver otherwise opens a transaction on the first statement, and with a transaction-level pooler the server connection stays pinned until it ends, including the Python work between queries. At 50 reads per write, unpinning reads cuts the backend connections a given request rate needs by a large factor. (Instagram never published its PgBouncer mode or pool sizes.)
Backups and replica bootstrap. In April 2012 new read replicas came from EBS snapshots and were caught up “in around 20 minutes”[S5]. By 2013 WAL was archived to S3 with WAL-E, giving point-in-time recovery and fast replica bootstrap[S9]. Section 6 explains why.
April 2012: Android. The Android launch brought more than a million new users in twelve hours[S5][S18]. The write-up is mostly about the tiers around Postgres. Memcached at 50,000 requests per second was slowing app servers, found with Dogslow; a Redis database hit 40,000 requests per second and got a read replica in under 20 minutes[S5]. For Postgres the response was more read replicas from EBS snapshots, about 20 minutes each, and a query-analysis loop: log every query for 30 seconds, run PGFouine, deploy fixes, repeat[S5]. Facebook announced the acquisition days later, on 9 April 2012[S29][S14]; at Krieger’s 12 April talk the team was four or five engineers serving about 30 million monthly actives[S6].
Our analysis
29 June 2012: the storm. An electrical storm in Northern Virginia took power from one AWS datacenter in us-east-1. By AWS’s account, generators failed to stabilise, the UPS ran down, servers lost power at 8:04 pm PDT, power began returning from 8:14 pm and the full facility had it by 8:24 pm, and recovery of EBS and the control plane was slow[S8]. AWS put the affected share of regional resources in the “single-digit percentage”[S8]. At Instagram, “almost half of our instances had lost power,” and three backend engineers spent 36 hours rebuilding[S18][S19]. No user data was lost[S18].
Our analysis
2012–2013: Cassandra takes the Redis workloads, not the Postgres ones. From September 2012 (“Redis fillin’ up”) the first workload moved: an audit and anti-spam log with a 1,000:1 write-to-read ratio that only grew, where keeping it in RAM was “Dumb”[S11]. In 2013 the activity inbox followed, off a 32-instance sharded Redis (16 masters, 16 replicas, 68 GB each) where “Resharding sucks” and “Failover is manual, wakes us up at night,” onto 12 hi1.4xlarge nodes, RF 3, writes at TWO and reads at ONE, via double writes, shadow reads, then real reads[S11]. Branson later put the Redis-to-Cassandra saving at paying “around a quarter of what we were paying before”[S16].
Reconstruction (inferred)
2011
2013
2015
2017–18
Users, media metadata, likes, comments, tags
Sharded Postgres [S4]
Sharded Postgres [S9]
Postgres, regional replicas [S19]
TAO, per Guo / Xiao [S25][S28]
Follow graph
PG + parallel Redis copy [S6]
PG + light memcached (by Apr 2012) [S6]
not documented
not documented
Main feed
Redis lists of media IDs [S4][S6]
not documented
not documented
Cassandra (“Feed”) [S27]
Activity inbox
Redis [S4]
Cassandra, from 32-node Redis [S11]
Cassandra
Cassandra (Direct inbox too) [S27]
Audit / spam log
Redis master/slave [S11]
Cassandra since Sep 2012 [S11]
Cassandra
Cassandra (fraud detection) [S27]
Legacy media → user map
Redis hash buckets [S3]
not documented
not documented
not documented
Geo / search
PostGIS, then Solr [S4][S6]
Solr / ElasticSearch [S4][S9]
FB Unicorn [S17][S20]
not documented
Cells marked “not documented” are left blank on purpose. Every source that describes a Cassandra migration has it replacing Redis workloads; none describes it replacing a Postgres one[S11][S16][S27].
The deck also records a vnodes switch attempted on 31 January 2013 and reverted on 10 February (“We aren’t ready for vnodes yet guys”), plus heap, key-cache and compaction trouble[S11]. Instagram adopted Cassandra carefully, for workloads shaped the way Cassandra likes, and still hit operational problems. That is the best evidence available for the 2011 claim that no NoSQL store was ready to be the primary database for a team that size.
Earlier, the 2012 slides record the opposite lesson. The follow graph was a Postgres table, and because the “DB was busy” the team kept a parallel copy in Redis. That brought “inconsistency, extra logic, so much extra logic” and meant “exposing your support team to the idea of cache invalidation.” They moved it back, since “PG can handle tens of thousands of requests,” with light memcached caching: “try not to have two tools trying to do the same job”[S6].
April 2013 – April 2014: leaving AWS. After the acquisition, Instagram moved into Facebook’s data centres to integrate with Facebook’s site-integrity, monitoring and search systems. The stated urgency was that “the longer we waited the harder it would be to migrate our ever-growing (and ever-pricier) infrastructure”[S18][S13]. The blocker was networking. Facebook’s private address space overlapped EC2’s, and Instagram had no control over EC2 addressing[S13][S15]. So the move had two hops: EC2 Classic to VPC, which Instagram controlled, and then VPC to Facebook over Direct Connect with “multiple redundant 10Gbps links”[S15].
EC2 Classic
“thousands” of instances [S14]
EC2 VPC
Neti overlay during transition
Facebook data centres
Direct Connect, redundant 10 Gbps
Postgres
replica sets in the target, then an application-level cut
Cassandra
target hosts join as another “datacenter”
Redis
new replicas sync from the production slave, never the master
Django
stateless: build new, shift traffic
For the transition Instagram built Neti, a ZooKeeper-backed Python daemon giving each instance an overlay IP through iptables DNAT[S15]. Its failures were kernel-level: 8,000 linear iptables rules cut memcached throughput “by an order of magnitude” (fixed with ipset), and the newly enabled conntrack table’s 65,536 entries filled in seconds[S15].
Postgres was the least eventful tier. “Using the streaming replication built into version 9.2, we were able to bring up master/slave replica sets in VPC, and cut to them with application-level controls… two of us cut over our entire DB fleet to VPC in less than 2 hours once the tiers were online and replicating”[S15]. Thousands of instances moved from EC2 to VPC in under three weeks[S13][S15], and the VPC-to-Facebook step took two weeks, with the last data arriving by the end of April 2014[S13]. The team’s rule was to “change just the bare minimum needed to support the new environment, and avoid the temptation of ‘while we’re here’”[S13]. A small team of eight Instagram and Facebook engineers ran it[S18]. A global outage in mid-April 2014 was said to be unrelated to the migration[S14], and no postmortem was published.
Our analysis
Reconstruction (inferred)
Afterwards Wired relayed Facebook’s claim of one server in its data centres for every three on AWS and about 80% less time fetching data[S14]; no method was given, so treat these as directional.
The 2012 storm started a reliability programme that became, in 2015, an expansion from one data centre to three[S18][S19][S20]. The motivations given were regional failures, flexible capacity[S19], and Facebook’s “storm” drills, which deliberately take a data centre offline[S25]. By then Instagram had 400 million monthly actives, 40 billion photos and videos, and served more than a million requests per second[S19]. The inter-data-centre latency went “From 10ms Latency to 60ms”[S19].
The design is conservative. Data is split into global and local; global data lives in Postgres and Cassandra, with the goal of “eventual consistency of these data across data centers”[S19]. Each region gets Postgres read replicas. “Writing to PostgreSQL, however, still goes across data centers because they always write to the primary”[S19]. Compute is deployed in per-region pods of Django, RabbitMQ and Celery, and asynchronous jobs are produced and consumed in the same region[S19][S25]. Memcache is regional; there is no global cache[S19].
Lisa Guo’s 2017 account adds that requests were batched where possible to cut cross-region write round trips, and that “the increased latency between the master and replicas turns out to not have been a big problem for us”[S25]. Both statements hide more mechanism than they show.
Our analysis
Every number here is illustrative: 60 ms is the reported inter-DC latency[S19], the rest are assumptions. Two different fixes, two different curves. Reordering statements rescues the hot row (4 → 16/s) but not the connection budget, because the transaction still spans four round trips. Only collapsing the transaction into one round trip fixes both, which is why “batched where possible”[S25] is a capacity decision, not a latency one. The “≥ 120” assumes the batched transaction still costs one 60 ms round trip of connection time.
Cache invalidation through the database. Regional caches create the classic stale-read race: a write lands on the primary, a remote region’s cache still holds the old value, and something must evict it. Instagram’s answer used PgQ, which the post says they adopted “and enhance[d]”[S19]. On the primary, the web server inserts the data (a comment, in the example) and then a cache-invalidation entry into the same database. The invalidation row replicates with the data, and a process on each region’s replica side reads it and invalidates that region’s memcache[S19][S25].
The mechanism is reported: the web tier writes the data and an invalidation row in the same database, the row replicates, and a per-region process invalidates the regional memcache[S19][S25]. The source says the two rows go to the same database, not whether they share a transaction. The ordering argument in steps 2 and 3 is ours: because the event is written after the data and replays in WAL order, a region can never invalidate before its own replica holds the write. Invalidating from the primary’s side can let a region re-fill its cache from a replica that has not yet applied it.
In our analysis, the choice of channel is the clever part. An invalidation sent over a separate bus can reach a region before that region’s replica has applied the write; the region evicts, re-fills from the stale replica, and stays stale with no invalidation left to fix it. Riding the WAL behind the data, the invalidation is applied on each replica only after the data it refers to, so a region cannot evict before its database holds the new data. It is the transactional outbox pattern (see the API & Integration Design track) with the replication stream as the relay.
The cost: more database reads. Invalidation by deletion turns every write into a future cache miss, and misses on popular objects arrive in herds. Instagram cited count queries on hot posts (the example given is a Justin Bieber post) and said hot counters falling out of cache used to need “hurried work from an engineer to manually fix the cache”[S19][S25]. Two mitigations followed[S19][S25]:
Denormalised counters. A like-count table updated in the same transaction as the like row. Guo put the difference as a count query of “hundreds of milliseconds” (select count(*)) against “tens of microseconds”[S25], a gap of three to four orders of magnitude, though this comes from an auto-transcript and the exact figures should be read loosely. The counter turns an O(likes) read into an O(1) read. The cost moves to the write side, where every like now updates one shared row per post, which is exactly the hot row the analysis above says cross-region transactions make expensive.
Memcache leases. On a miss, one “first miss” is sent to the database, and other requesters get a “hot miss,” served a stale value if one exists[S19][S25]. It turns a herd of N identical queries into one. Instagram’s later “Thundering Herds & Promises” post covers the same problem from the caching side[S30]. The 2015 post’s summary of the result: “those incidents have become memories for old-timer engineers”[S19].
Building replicas across 60 ms. The new regions needed replicas, and the first attempt failed in a way worth studying. A replica built from a base snapshot and then fed WAL fetched from archive storage could fall behind the primary’s write rate: “The replica will fall further and further behind and never catch up!”[S19]. The fix was to start a second WAL streamer as soon as the base snapshot transfer began, and keep the WAL on local disk. That “cut down the time it took to build a new replica by half”[S19].
Illustrative rates, not reported ones. The reported facts are the failure (“fall further and further behind and never catch up”), the fix (a second streamer started when the base transfer begins, WAL kept on local disk) and the result (replica build time cut “by half”)[S19]. The model shows why the fix is structural rather than a tuning change: catch-up needs consume rate > produce rate, and per-segment fetches over a long link lower the consume rate, while a streamer running from t=0 means the backlog never has to cross the link again.
The principle: catch-up needs the consume rate to exceed the produce rate by enough to drain the backlog built up during the base copy. Per-segment archive fetches over a long link can push the consume rate below the produce rate, and then no amount of waiting helps. Streaming from time zero keeps the backlog local, so replay runs at disk speed. This is a queueing argument, and it applies to any log-shipped replica, CDC consumer or search indexer bootstrapped across a slow link.
Failover stayed manual. Guo said in 2017 that “we do not do automatic master slave promotion,” and that before storm drills servers are promoted ahead of the cut-off rather than failing over under load[S25] (auto-transcript, paraphrased). The drills themselves were real tests: one hit the new data centre “right as we had finished migrated our data,” and users did not notice[S19]. Separately, a rack of memcache and async servers in Oregon lost power, “which caused large exceptions to user requests,” and traffic was moved away from the region[S19].
Our analysis
Instagram published a lot of numbers and almost none of them about Postgres directly. There are no QPS, latency, replication-lag or data-volume figures for the database tier after 2011. What follows is what can honestly be derived from what was published, with the arithmetic shown.
Write growth, 2011–2017. September 2011: “more than 25 photos and 90 likes every second”[S1]. January 2013: “over 10,000 likes per second at peak”[S9]. The 2013 post frames this as 90 to 10,000 in “less than a year”[S9], a factor of 111. The interval was actually about fifteen and a half months, and the first figure reads as an average while the second is a peak. With an illustrative peak-to-average ratio of 3, like-for-like growth is closer to 37 times: still a doubling roughly every three months, on an unchanged architecture. In 2017 Guo reported 4 billion likes per day[S25] (auto-transcript), which averages to about 46,000 per second. On average rates alone, likes therefore grew roughly 500-fold (90 to about 46,000 per second, our arithmetic) from the time of the sharding post to the time Postgres was reported gone.
How oversized the ID sequence field was. Using the 2017 average and an illustrative 2,000 logical shards, a single shard receives about 23 likes per second, or 0.023 per millisecond, against a capacity of 1,024 per millisecond per table per shard. Even a shard with a thousand times the average load would use a few percent of the field. Four or five bits of sequence would have covered every realistic load; the other five or six bits could have extended the time field and pushed the sign-bit problem from 2046 past the end of the century. This is hindsight, and in 2011 buying headroom against unknown growth was reasonable. But it is a useful calibration for anyone designing a bit layout today. The sequence field protects against bursts on one table on one shard. Size it from the per-shard burst rate, not the global rate.
Working-set arithmetic. Before sharding: one database of more than 60 GB against a 68 GB ceiling[S6]. December 2011: 12 masters on “Quadruple Extra-Large memory” instances[S4]; taking these to be the m2.4xlarge that the Cassandra deck lists at 68 GB[S11] (our identification), that is about 816 GB of RAM on the masters, and the same again on the cross-zone replicas. Sharding raised the effective memory ceiling of the database by roughly 12 times inside a few months, and the pre-split let it keep rising by adding hosts. That, not query throughput, is the quantity the architecture was built to scale.
The price of not carrying location. Three hundred million legacy media-to-user mappings took just under 5 GB of Redis after bucketing, against about 21 GB at the unoptimised 70 MB per million keys[S3]. Extrapolated to 2015’s 40 billion objects[S19], the optimised figure is about 640 GB, and the naive one about 2.8 TB (our arithmetic). The legacy map was finite because new IDs carried their shard. Without that, it would have grown with every upload, forever.
Redis to Cassandra, by resources. The inbox moved from 32 m2.4xlarge instances at 68 GB each, about 2.2 TB of RAM with half of it on replicas, to 12 hi1.4xlarge nodes with 60 GB of RAM and 2 TB of SSD each: about 720 GB of RAM and 24 TB of SSD[S11]. The data set was about 1.2 TB[S16]. The saving Branson reported for moving off Redis (paying “around a quarter”[S16]; the interview does not tie it to one cluster) comes, in our reading, from replacing RAM as the storage medium with SSD as the storage medium and RAM as a cache, which is exactly the trade the Postgres tier could not make on 2011 EBS.
Headcount. Three engineers for 14 million users in December 2011[S4]; eight on infrastructure for the 2013 migration[S14]; 95 engineers in June 2015[S17]; about 250 in 2017, 30% of whom had joined in the previous six months[S25]. Users per engineer fell from about 4.7 million to about 1.6 million (400 million daily actives over 250 engineers, with the caveat that the user definitions differ). This is the number that explains Section 10. The 2011 design was optimised for an organisation where the people who designed the shard map were the people who used it. By 2017 most engineers writing data-access code had not been there for the design.
~37–111x
like-rate growth, Sep 2011 → Jan 2013 (see text)
~46k/s
average likes, 2017 (4B/day) [S25]
~640 GB
Redis map if no ID carried location (ours)
~¼
cost after Redis → Cassandra [S16]
What the numbers do not show. No figure anywhere gives Postgres replication lag across regions (only that it was “not a big problem”[S25]), primary failover duration, per-host QPS, total data volume after 2011, PgBouncer configuration, or the host count after the “hundreds of machines” reference in 2013. Anyone who quotes a specific number for these is quoting something Instagram did not publish.
The design is a chain of five decisions. Each was defensible in 2011, and each left a specific bill that came due at a specific scale. Judging them requires keeping both halves in view.
Decision 1: stay relational rather than adopt a NoSQL primary. The stated reasons were familiarity, operational burden, and the fact that CouchDB would still have needed custom sharding middleware[S2][S6]. The strongest evidence that this was right is what happened when Instagram did adopt Cassandra a year later. It picked the workload shapes Cassandra is good at (append-heavy, read-light, time-ordered per key), started with a three-node cluster, rolled out through double writes and shadow reads, and still recorded a reverted vnodes migration, heap tuning and compaction trouble in its first year[S11]. Branson’s rule, “Adopt a technology by understanding what it’s best at and letting it do that first, then expand”[S11], is the opposite of putting it under the whole product in 2011.
Counterfactual
Decision 2: pre-split into thousands of logical shards, hashed by owner. The alternative named was range partitioning[S6]. Unnamed but obvious are consistent hashing and a directory (per-user placement in a lookup table). Modulo-plus-map is a directory with a very coarse key: it gives the operational benefit of a directory (any logical shard can go on any host) without a per-user lookup on the request path.
The price is that the modulus is permanent. The logical shard count fixes the largest unit that can ever be placed, and the finest granularity of rebalancing. Two criteria follow, and teams copying the design usually get the second wrong. The logical count must be large enough that the largest logical shard at the end of the planning horizon fits comfortably on one host. It must also be large relative to the maximum number of hosts, because balance comes from averaging many units per host: with k logical shards per host, the load imbalance from uneven shards shrinks roughly like 1/√k. Several thousand shards on tens to hundreds of hosts meets both. A scheme with 64 logical shards meets neither once you pass a few dozen hosts. The price at the top end is catalog overhead (Section 3), which is modest.
Hashing by owner was the right key for a product in which almost every hot read is “this user’s things.” It makes every per-user query single-shard. It also makes every per-user hot spot unsplittable, which is the “single user” weakness Krieger named[S10]. For a photo-sharing app in 2011 that was a distant risk. By 2017 the top account had more than 110 million followers[S25].
Decision 3: IDs minted in the database, carrying the logical shard. Covered in Section 4. It aged best of the five; its costs (sign-bit date, per-type uniqueness, clock-dependent ordering) were all avoidable at design time.
Decision 4: the map as a dict in code, moves by physical replication, failover by hand. The alternative is a control plane: a metadata service holding the map, a proxy or client library that watches it, and automated promotion. Instagram had none of these, and said so: moves were “adhoc” and “semi-automated”[S10], failover was not automatic even in 2017[S25].
Our analysis
Failure rates and the trigger line are illustrative; the only reported facts are the 2011 master count[S4], the 2013 “hundreds of machines” wording[S9] and that promotion was still manual in 2017[S25]. At 12 masters every rate gives well under one event a year, so manual promotion was nearly free. The design never changed; the line it sits on did.
Decision 5: multi-region with a single write primary and invalidation through the WAL. The alternatives in 2015 were multi-master Postgres (no mature option), moving global data to Cassandra wholesale, or keeping one region. Instagram kept writes on one primary, accepted cross-region write latency, and used the replication stream for cache invalidation[S19]. This preserved every invariant the application relied on (one writer per row, serialisable local transactions, counters updated in the same transaction as the event) at the cost of write latency and write availability during a regional failure. Given a 50:1 read skew[S10], that was the correct trade. It is also a trade that only works because the earlier decisions kept all of a user’s data on one primary. A design that had spread one user’s writes across several shards would have needed cross-region distributed transactions here.
The meta-decision. Krieger’s 2015 rule, “do the simple thing first ... until your {scale, team, product} changes”[S17], is usually quoted for its first half. The move to TAO, described next, was justified almost entirely by the team clause.
“Instagram runs on sharded Postgres.” As a present-tense claim, this is not supported by any Instagram source after 2015. Postgres was central through at least November 2015[S19]. In the 2017 QCon Q&A, Lisa Guo spoke of the time “when we were still managing Postgres,” said “we don’t anymore,” and said Instagram uses TAO[S25]; this is an auto-transcript and somewhat garbled, but the meaning is clear. In October 2018 Sherry Xiao gave Cassandra and TAO as the examples of Instagram’s stateful services and did not mention Postgres[S28]. No Instagram source describes the Postgres-to-TAO migration itself: not its date, method, duration, or whether any Postgres remained. The accurate statement is that by 2017 Instagram engineers described their core relational data as living on Facebook’s TAO.
“Instagram moved from Postgres to Cassandra.” A common conflation. Every source that describes a Cassandra migration describes it replacing Redis: the audit log, the activity inbox, and later feed and Direct inbox workloads[S11][S16][S27]. In 2015 Postgres and Cassandra were the two main back ends side by side[S19][S25]. What left Postgres went to TAO, which by Guo’s own description “still uses a relational database, MySQL, at the back end”[S25] (auto-transcript, lightly cleaned). Instagram’s relational data ended up on a relational database behind a graph API and a write-through cache.
Three smaller myths. “Thousands of shards” were thousands of schemas, a few hundred per host on 12 masters in 2011[S1][S4]. “41 years of IDs, 1,024 per shard per millisecond” is imprecise on both counts (Section 4). And the sharding was forced by RAM against EBS seek limits, not by write volume, which was 90 likes per second at the time[S1][S4][S6].
“A sophisticated, automated sharding platform.” The record shows a Django dict, manual resharding, brief downtime per move and no automatic failover[S2][S10][S25]. The sophistication was in what they left out.
“Three engineers ran Instagram.” True of December 2011[S4]. By 2017 there were about 250 engineers[S25]. The three-engineer design did not survive the 250-engineer organisation unchanged, and that is the point of the story, not a footnote to it.
What happened next. Three threads run from 2015 to 2018.
TAO. Guo’s reasons for the move were developer velocity, automatic caching, and self-serve data modelling for product engineers, with infrastructure engineers responsible only for scaling the service[S25]. TAO exposes a simplified model of nodes and edges over MySQL with a write-through cache[S25]. The TAO paper describes a geographically distributed store serving “a billion reads and millions of writes each second”[S12]. In 2018 Xiao described Instagram’s use of it: “We run TAO as a single master per shard,” and it “forwards all writes to the shard’s master region”[S28], which was in the United States. Scaling “from three to five data centers was trivial”[S28].
Our analysis
Limit
Sharded Postgres, 2015
On TAO, 2017–18
Net
One write region per shard
writes always go to the primary, across DCs [S19]
“single master per shard”, writes forwarded to the master region [S28]
unchanged
Primary failover
“we do not do automatic master slave promotion” [S25]
not documented
unknown
Cache + invalidation
hand-built: regional memcache, PgQ, leases, counters [S19][S25]
“data base plus write-through cache” [S25]
moved to platform
Data modelling
ORM on schema-qualified tables, routing in Django [S2]
nodes and edges, “self-serve by product engineers” [S25]
moved to product teams
Shard map and moves
dict in code; moves ad hoc, semi-automated [S2][S10]
not documented
owned by platform
Splitting one hot user
“harder to split off a single user” [S10]
not documented
unknown
The “Net” column is our reading. What moved is ownership of the machinery around the data; the write topology, the one property users feel during a regional failure, is the same on both sides of the table.
Counterfactual
Cassandra. By 2018 Instagram ran Cassandra for “fraud detection, Feed, and the Direct inbox,” and had built Rocksandra, a RocksDB-based storage engine for it, reporting a P99 read latency drop from 60 ms to 20 ms and GC stall time from 2.5% to 0.3%[S27]. The 2018 Europe write-up notes that simply extending the Cassandra clusters to Europe would mean a replication factor of eight (five US plus three EU replicas)[S28], which is the same “copy everything everywhere” cost that physical replication imposed on Postgres shard moves, now multiplied by region count.
The application tier. Django stayed: “the world’s largest deployment of the Django web framework” in 2016[S22], moved to Python 3 by early 2017 after a ten-month migration[S26]. The data layer changed underneath an application tier that did not.
The Instagram design is widely copied in the wrong way: people copy the shard count and the ID function and skip the conditions that made them work. The matrix below states those conditions. After it are the lessons we think transfer, each with the condition that has to hold.
Copy it when
Do not when
Access pattern
Almost every hot query is scoped to one owner (a user, tenant or account)
Many-to-many hot paths: the celebrity, the shared document, the global counter
Forcing constraint
A per-node ceiling you cannot buy past (RAM, IOPS, instance size)
You can still buy a bigger box, or add replicas, for a year or more
Topology change rate
Resharding a few times a year, with seconds of write pause acceptable
Continuous rebalancing, or no write pause is tolerable
ID ownership
You can mint all new IDs yourself, and old ones are a finite legacy
External systems assign IDs, or IDs must be opaque / unguessable
Team
A small team that already knows the engine well
No one who can run physical replication, PITR and manual promotion
Multi-region
Reads dominate (Instagram cited 50:1) and a primary region is acceptable
Writes must succeed locally in every region during a partition
1. Put the location in the identifier, and make it logical. This is the lesson with the widest transfer. An ID that embeds a logical partition removes a lookup service from every read path and survives every physical move. Copy it when you mint your own IDs and can afford some bits. Do not copy it when IDs must be opaque to clients (the partition leaks into every URL), or when entities genuinely change owners, because the embedded partition then points to the wrong place forever and you rebuild the lookup store you were avoiding. Decide the sign bit, the epoch and per-type versus global uniqueness on day one (Section 4).
2. Pre-split far past your host count. Choose the logical count so that the largest logical shard at the end of your horizon fits comfortably on one host, and so that each host carries tens to hundreds of logical shards for balance. Copy it when your partition key is stable and your hottest single key fits on one machine. Do not copy it when single keys can outgrow a machine (one tenant, one celebrity, one global counter); pre-splitting does nothing for a key that cannot be split. The Multi-Tenant SaaS Architecture track covers the tenant-placement version of this problem, where the largest tenant is usually the design constraint.
3. Have exactly one way to move data, and rehearse it constantly. Instagram’s physical-replication move primitive handled resharding, a platform migration and multi-region replica builds. Its virtue was not efficiency (it copies whole hosts) but that it was the same operation every time. Copy the principle whenever you run a stateful fleet. The specific primitive (physical replication plus a map flip with a brief write pause) transfers when a few seconds of write unavailability per move is acceptable and moves are rare. When moves must be online or frequent, you need a real fence (privilege revocation or fencing tokens, as Figma used) and probably logical replication, and at that point you are building a control plane.
4. Treat the replication stream as your ordered bus. Instagram’s PgQ invalidation made the cache correct by construction, because the invalidation cannot overtake the data it invalidates. Copy it when caches sit next to replicas and staleness after a write is user-visible. The cost is that every write becomes a future miss, so budget for leases and denormalised counters at the same time (Section 7). And remember the hot-row cost of the counter when the writer is far from the primary.
5. Partition for the binding constraint. Instagram’s was RAM against EBS seek rates, and all its tuning (partial indexes, prefix indexes, reordering by owner) shrank the hot set. If yours is table size or IOPS on a managed database, Figma is the closer case. If it is write throughput on a few keys, neither design helps; change the data model.
6. Attach a revisit trigger to every “simple thing first.” Manual failover, a dict in code and ad hoc moves were right at 12 masters and three engineers, and grew expensive as N grew while the design stood still. Write down the scale, team size or event rate at which each shortcut must go. Instagram’s answer, a shared in-house platform, exists only inside an organisation that has one; for everyone else it is a managed service, or a control plane built before manual events become the on-call load. The Technical Leadership essays on platform ownership cover that call.
7. Do not keep two authoritative copies of the same fact. The follow-graph episode[S6] and the Cassandra anti-columns[S11] are the same lesson from opposite directions. The first shows that a parallel copy for performance becomes a consistency problem for people. The second shows that read-before-write on a replicated store is a correctness bug under replica divergence. If a derived copy is needed, make it derived: rebuilt or invalidated from one source of truth through one ordered channel.
Our analysis — questions for a design review of a design like this
Bottom line
Primary sources are Instagram Engineering posts, Instagram engineers’ talks and slides, the authors’ own Hacker News comments, AWS’s event summary and the TAO paper. S14, S16, S26 and S29 are secondary: journalism, a vendor-published interview and a summary blog, cited only for what they report. S25 is a machine transcript of a recorded talk, so quotations from it are short and treated with caution. Medium shows the wrong date (December 2012) for S1; the Hacker News thread and an Archive.org snapshot place it at about 30 September 2011. The Cassandra Summit 2014 slides and transcripts of the PyCon 2017 keynote and @Scale 2016 talk could not be retrieved, and nothing here depends on them.
Sharding & IDs at Instagram — Instagram Engineering (Mike Krieger), c. 30 Sep 2011 (Medium shows Dec 2012)
HN discussion of S1, with author comments by mikeyk — Hacker News, 30 Sep – 1 Oct 2011
Storing hundreds of millions of simple key-value pairs in Redis — Instagram Engineering (Mike Krieger), 1 Nov 2011
What Powers Instagram: Hundreds of Instances, Dozens of Technologies — Instagram Engineering, 2 Dec 2011
Keeping Instagram up with over a million new users in twelve hours — Instagram Engineering, Apr 2012
Scaling Instagram (AirBnB Tech Talk), slides — Mike Krieger, Speaker Deck, 12 Apr 2012
redis-faina: a query analysis tool for Redis — Instagram Engineering, 15 May 2012
Summary of the AWS Service Event in the US East Region — Amazon Web Services, Jul 2012
Handling Growth with Postgres: 5 Tips From Instagram — Instagram Engineering (Mike Krieger), 17 Jan 2013
HN discussion of S9, with comments by rbranson and mikeyk — Hacker News, 17–18 Jan 2013
Cassandra at Instagram (August 2013), meetup slides — Rick Branson, SlideShare, 29 Aug 2013
TAO: Facebook’s Distributed Data Store for the Social Graph — Bronson et al., USENIX ATC ’13, Jun 2013
Migrating From AWS to FB — Instagram Engineering, 25 Jun 2014
How Facebook Moved 20 Billion Instagram Photos Without You Noticing — WIRED (Cade Metz), 26 Jun 2014
Migrating from AWS to AWS — Instagram Engineering (Nick Shortway), Oct 2014
Facebook’s Instagram: Making the Switch to Cassandra from Redis, a 75% ‘Insta’ Savings — DataStax blog (interview with Rick Branson), 15 Oct 2014
Scaling Instagram at AirBnB OpenAir 2015, slides — Mike Krieger, Speaker Deck, 4 Jun 2015
Five Years of Building Instagram (War Stories) — Mike Krieger, Backchannel / Medium, 7 Oct 2015
Instagration Pt. 2: Scaling our infrastructure to multiple data centers — Instagram Engineering, 11 Nov 2015
Under the Hood: Instagram in 2015 — Instagram Engineering, 29 Dec 2015
Continuous Deployment at Instagram — Instagram Engineering, 12 Apr 2016
Web Service Efficiency at Instagram with Python — Instagram Engineering, 21 Jun 2016
Video: Running Instagram at @Scale 2016 — Instagram Engineering (Lisa Guo), 10 Oct 2016
Dismissing Python Garbage Collection at Instagram — Instagram Engineering, 17 Jan 2017
Scaling Instagram Infrastructure (QCon London talk and Q&A) — Lisa Guo, InfoQ, 2017 (page dated 19 May 2017)
Instagram Makes a Smooth Move to Python 3 — The New Stack (Hui Ding, Lisa Guo interview), 14 Jun 2017
Open-sourcing a 10x reduction in Apache Cassandra tail latency — Instagram Engineering (Dikang Gu), 5 Mar 2018
How Instagram is scaling its infrastructure across the ocean — Opensource.com (Sherry Xiao), Oct 2018
The Instagram Architecture Facebook Bought for a Cool Billion Dollars — HighScalability (Todd Hoff), 9 Apr 2012
Thundering Herds & Promises — Instagram Engineering (Nick Cooper), 16 Apr 2019
HN discussion of S4, with author comments — Hacker News, 3 Dec 2011