In March 2023 a Prime Video engineer published a short post on moving one internal monitoring tool off Step Functions and Lambda. Six weeks later it was being read as Amazon’s verdict on microservices. Both sides of that argument skipped the part worth studying. The post is a clean natural experiment in metered coordination: a workload whose unit of work is a second of video, running on services that bill and throttle per event, at a rate the design never priced. The thesis of this case: Prime Video did not trade microservices for a monolith. It moved the network boundary from the per-frame, per-second tier of its workload to the per-request tier, which removed every metered hop on the hot path and kept a distributed shape where the event rate is one per stream. The “over 90%” is what you would expect from that move, and it tells you less about architectural style than about where the old bill was going.
~5%
of expected load at the scaling wall [S1]
> 90%
infrastructure cost reduction [S1]
“thousands”
of concurrent streams, the target [S1]
0
absolute costs or baselines published [S1]
Source tags like [S1] refer to the list in Section 11. S1 is the only first-hand account of the system; everything else is context, commentary, or AWS reference material. The AWS pricing and quota figures used here (S9, S14) are 2026 values, not the ones Prime Video faced in 2023, and are used only to reason about shape and scale. Anything labelled Our analysis, Reconstruction (inferred) or Counterfactual is ours, not Amazon’s.
The system is the Video Quality Analysis (VQA) team’s audio/video monitoring service. Prime Video offers “thousands of live streams”, and the tool exists to watch every stream viewed by customers, find perceptual defects such as block corruption or audio/video sync problems, and “trigger a process to fix them”[S1]. It has three components, and they were the same three before and after the redesign: a media converter that turns the input stream into frames and decrypted audio buffers; defect detectors that analyse those frames and buffers in real time and send notifications when something is wrong; and an orchestration component that controls the flow[S1].
The detectors are ML models, and their cost profile is what matters here. By March 2022 VQA had detectors for 18 defect types[S2]; block corruption is a residual network producing a per-pixel probability map, and LipSync runs shot detection, face detection and tracking over a four-second clip before a SyncNet-based model scores it[S2]. Two consequences follow for the architecture. Per-detector cost is heterogeneous (a per-frame CNN and a multi-stage four-second-window pipeline are not the same size), and some detectors need temporal context, a window of consecutive frames, not isolated images. The sources do not confirm that all 18 run in the monitoring service, and S1 gives no detector count.
Two properties of this workload matter more than anything else in the story, and neither is stated as such by the post. First, the unit of work is time, not requests. A live stream produces frames for as long as it runs, and a monitor for that stream produces work every second, whether or not anything interesting happens. The work rate is streams × seconds × (frames or ticks per second), and it is flat and predictable once streams are onboarded. Second, the payload is large and short-lived. A decoded frame exists only to be looked at by a handful of detectors and then discarded. Nothing downstream needs to read it later, and the detection result is tiny by comparison.
Those two properties describe a data-plane workload: high, steady event rate, large ephemeral payloads, small outputs. The services it was first built on, Step Functions for orchestration and S3 as the hand-off medium between components[S1], are excellent control-plane tools: durable, auditable, per-event pricing, zero idle cost. The mismatch between those two sentences is the entire case. Everything else, including the 90%, falls out of it.
Our analysis — why this is the right frame
The tool was not new when the problem surfaced. VQA “already owned a tool for audio/video quality inspection”, and Kolny is direct about its pedigree: “we never intended nor designed it to run at high scale”[S1]. The target was “to monitor thousands of concurrent streams and grow that number over time”[S1]. As more streams were onboarded, two things showed up together. Running the infrastructure at high scale “was very expensive”, and there were “scaling bottlenecks that prevented us from monitoring thousands of streams”[S1].
The scaling failure has a number attached: “the way we used some components caused us to hit a hard scaling limit at around 5% of the expected load”[S1]. The cost failure does not. The post says only that “the overall cost of all the building blocks was too high to accept the solution at a large scale”[S1]. Note the conditional: at a large scale. The v1 bill at its actual 5% operating point may have been perfectly tolerable. What was unacceptable was the projection, which for a linear-cost system is just the current bill times twenty.
The post names two culprits, “the orchestration workflow and when data passed between distributed components”[S1]: “multiple state transitions for every second of the stream”, billed per transition and exhausting account limits, and “the high number of Tier-1 calls to the S3 bucket” used to hand frames to detectors[S1]. So one mechanism, per-second state transitions, produced both the scaling wall and a cost line; a second, frames as S3 objects, produced the other cost line. Compute, meaning the actual running of ML models on frames, is not named as a cost problem at all. Hold on to that absence; Section 6 shows it is the most informative fact in the post.
Our analysis — reading the outcome sentence backwards
The post describes the first response as incremental: “we initially considered fixing problems separately to reduce cost and increase scaling capabilities. We experimented and took a bold decision: we decided to rearchitect”[S1]. What those experiments were is not described. The New Stack’s version, that optimising individual components “did not bring about significant improvements”[S8], is a stronger claim than the primary text makes. The post does not say the fixes failed, only that the team experimented and then chose a rewrite.
S1’s diagram of the initial design is redrawn below: a Lambda entry point inside a Step Functions workflow starts a separate Media Conversion Service (drawn outside the workflow, with a generic compute icon) that writes buffers to S3; a Parallel state then runs one nested state machine per detector; results go to the customer’s SNS topic and, via an aggregation Lambda, to S3[S1].
1 · frame write
Converter uploads frames to S3 as objects: f write requests per stream-second, independent of detector count
2 · frame read
Every detector branch downloads the frames it analyses, concurrently, on Lambda: up to N·f reads per stream-second
3 · state machine
“Multiple state transitions for every second of the stream” [S1], each billed and each drawn from one shared bucket
Boxes, labels and the two-detector example follow S1’s diagram. The rate labels and the k formula are ours. Grey arrows fire once per analysis or only on a defect; pink arrows fire every stream-second, for every stream, for as long as it runs.
The diagram and the text disagree on one detail, and it matters for the cost model. The text says detectors “download images and processed it concurrently using AWS Lambda” and that each detector “runs as a separate microservice”[S1]. The diagram’s compute units carry a generic compute icon, not Lambda’s. The most consistent reading is that each detector was a nested state machine whose tasks invoked Lambda functions, but the post does not settle it, and we mark the detector compute as Lambda only because the text says so.
What the diagram cannot show is the time dimension, and that is where the design lives or dies. The only temporal fact in S1 is “multiple state transitions for every second of the stream”[S1]. Combined with the parallel detector branches, it pins down the rough shape of the control loop.
Reconstruction (inferred) — the v1 control loop
k = k₀ + N·kᵦ, where kᵦ is the number of states per detector branch. Adding a detector raised the transition rate of every stream. Nothing in S1 gives k₀ or kᵦ; the form follows from how Parallel states are metered (S9 bills “the total number of state transitions across all your state machines”) combined with the diagram.That last point is worth pausing on, because it inverts the usual intuition about microservices. The textbook benefit of one-detector-per-service is that adding a detector is additive and isolated: “adding a new detector required creating a new microservice and plug it in to the orchestration”[S1]. Operationally that is true. Economically, in this design, each new detector multiplied into the stream-second rate on three meters at once: transitions, S3 reads and Lambda invocations. The team “regularly adds more detectors”[S1], so the v1 cost curve was not linear in streams. It was linear in streams × detectors, and both factors were growing.
Hop
Metered unit
Events per stream-second
Per hour (S streams)
Scaling behaviour
Frame write
S3 PUT (Tier-1)
f
f × S × 3,600
Grows with sample rate; independent of detector count
Frame read
S3 GET
N × f (if every detector reads every frame)
N·f × S × 3,600
Grows with detectors; S1 names only “Tier-1” calls, and GETs are priced separately [S15]
Orchestration
Step Functions transition
k = k₀ + N·kᵦ
k × S × 3,600
“Multiple” per second; Parallel branches each transition
Detection
Lambda invocation + duration
≥ N per batch
scales with N × S
Per-request fee on top of the compute actually used
Notification
SNS publish
only on defect
defect-rate × S
Rare; not a cost driver
f = frames sampled per second, N = detectors, k₀ = fixed transitions per cycle, kᵦ = transitions per detector branch, S = concurrent streams. None of f, N, k₀, kᵦ or S is published; the post gives only “multiple state transitions for every second” and “thousands” of streams [S1]. The point of the table is the shape: every metered line is linear in stream-seconds, and two of them are also linear in N.
The ledger above is the whole v1 economic model in one table. Every metered line is proportional to stream-seconds. Two of them, frame reads and branch transitions, are also proportional to N. None of them is proportional to the thing the service exists to do, which is floating-point work inside a detector. That is the defining property of an overhead-dominated design: the metered cost tracks the number of hand-offs, not the amount of useful computation between them.
S1 names two bottlenecks. Looking at how the underlying services meter and limit work turns them into three distinct mechanisms, each with its own failure mode. The numbers below come from today’s AWS documentation[S9][S14]. They are not the 2023 values Prime Video hit, and the post does not say which limit it reached. We use them because the kind of limit has not changed, and the kind is what explains the design’s behaviour.
Meter 1: the StateTransition token bucket. Standard Workflows throttle state transitions per account per region with a token bucket: today 5,000 tokens with a 5,000/s refill in us-east-1, us-west-2 and eu-west-1, and 800/800 in all other regions; the quota is soft and can be raised, and Express Workflows show “Unlimited”[S14]. Throttling surfaces as ExecutionThrottled in CloudWatch[S14]. The important property is that this bucket is account-wide and shared. Every stream’s workflow and every detector branch draws from the same bucket. With k transitions per stream-second, the account can sustain at most quota / k concurrent streams, however many Lambdas or ECS tasks sit behind the states. “We quickly reached account limits”[S1] is what that ceiling looks like from the inside.
streams at wall
k = 1
k = 3
k = 5
k = 10
5,000/s quota
5,000
1,667
1,000
500
800/s quota
800
267
160
80
Quota values are the 2026 defaults from S14 (soft; raisable). The 2023 values, the region, and which limit Prime Video actually hit are not published. The gold band is an inference: if the wall was 5% of expected load and expected load was 2,000–5,000 streams, the wall sat at 100–250 streams.
The chart makes one inference possible that the post does not state. The wall was at about 5% of expected load, so the expected load was about twenty times the wall. If “expected” means a few thousand streams (S1’s own word is “thousands”), the wall sat somewhere around 100 to 250 concurrent streams. Under today’s 800/s default that corresponds to roughly 3 to 8 transitions per stream-second, a range that fits the word “multiple” well. Under today’s 5,000/s default it would take 20 to 50 transitions per stream-second, which is plausible only if the per-branch count across many detectors was large. We cannot tell which regime Prime Video was in; the region and the 2023 quota values are unknown. Either way, the post’s two facts are consistent with a Standard-workflow throttle and not obviously with anything else.
Our analysis — why ‘just raise the quota’ misses the point
How the wall shows up in operation matters as much as where it sits, and here the load shape makes things worse. Throttling works as a pressure valve when the client is a request source that backs off: demand falls, and the bucket refills. A live-stream monitor cannot back off, because the stream keeps producing seconds whether or not the previous ones were processed. Retries make it worse again, since each retry is billed as another transition[S9] and draws on the same bucket it is waiting for.
The chain is our reconstruction; S1 reports only that the team “quickly reached account limits”. ExecutionThrottled and the shared bucket are from S14; retries billed as extra transitions is from S9. The lag curves are illustrative queueing arithmetic, assuming the shortfall is spread evenly across streams and nothing is dropped. If throttled work is dropped instead, lag stays flat and coverage silently falls, which is arguably worse for a monitor.
Our analysis — a real-time monitor has no graceful overload mode
1 − C/D seconds per second. At an illustrative 10% over the quota, every stream is about 5.5 minutes behind after an hour and 16 minutes behind after three, which for “real-time notifications”[S1] means an alert about a defect the viewers have already seen. If throttled work is dropped, lag stays flat and coverage falls silently. Neither is a failure an SLO on error rate would catch. The monitor to add is lag per stream against wall clock, and the operational response S1’s wording suggests (our inference, Section 2), monitoring the highest-viewership streams first[S1], is exactly what a team does when it chooses admission control over unbounded lag. That was the right choice, and it is also why the wall showed up as a coverage policy rather than an outage.Meter 2: execution lifetime. A live stream is long-lived, and neither workflow type is built to hold a long, dense loop in one execution. A Standard execution may run for up to a year, but its history is capped at 25,000 events, a hard quota; when it is reached, “the execution will fail”[S14]. Express executions have no history cap but a hard five-minute maximum duration[S14]. A per-stream execution with k transitions per second exhausts a Standard history in at most 25,000/k seconds, even under the generous assumption of one event per transition.
Standard, k = 1
≈ 6.9 h
Standard, k = 3
≈ 2.3 h
Standard, k = 5
≈ 83 min
Standard, k = 10
≈ 42 min
Express, any k
5 min (hard)
Standard: 25,000 history events per execution (hard), lower-bounded here at one event per transition, so lifetime ≤ 25,000 / k seconds. Express: 5-minute maximum execution time (hard). Both are current values from S14. A stream that outlives these bounds needs a chain of executions, and each new execution is another metered, throttled StartExecution.
So a long stream on either workflow type has to be cut into a chain of executions, with each link a separate StartExecution. That API is itself throttled, today at 1,300 burst and 300/s refill for Standard in the large regions[S14], and it adds orchestration logic of its own: continuation, deduplication, and hand-over of in-flight state between links. S1 does not describe how v1 handled stream length, so we cannot say whether this bit Prime Video. We raise it because it is the next limit a team would meet after fixing the transition throttle, and because it shows the structural issue: a workflow engine models a process with a beginning and an end, while a live-stream monitor is a process without an end.
Meter 3: the claim-check tax. Step Functions passes state between tasks as JSON with a 256 KiB payload limit per task[S14]. A decoded video frame does not fit, and neither does a second of them. Any design that orchestrates frame processing through a workflow engine therefore has to pass frames by reference: write the frame somewhere, pass a key through the state machine, and have each consumer fetch it. This is the claim-check pattern, and it is correct for control-plane payloads such as documents, images uploaded by users, or batch inputs. On a data-plane hot loop it turns every frame into at least one write request plus one read per consumer. Windowed detectors make it worse. A LipSync-style model that scores a four-second clip[S2] has to reassemble its window from many single-frame objects, so its read count scales with window length, not with the number of verdicts it produces (our inference; S1 does not say whether LipSync runs in this service).
Topology per S1’s v1 diagram (media conversion outside the workflow, per-detector nested state machines). The 256 KiB task payload limit is the current value in S14; that it is why frames went by reference is our inference. S1 gives a different reason for the split: converting once “to reduce computationally expensive video conversion jobs”. f, N and the window read are illustrative; S1 states none of them, and does not say whether a clip-based detector such as LipSync[S2] runs here.
S1 gives a different reason for the S3 hop, and it is worth taking seriously. The converter was split out “to reduce computationally expensive video conversion jobs”[S1]: decode once, let N detectors share the result. That is a real saving in compute. But the saving is paid for with f PUTs and up to N·f GETs per stream-second, each a metered request, and the post says “the high number of Tier-1 calls to the S3 bucket was expensive”[S1]. S1 does not define Tier-1. S3’s request pricing puts PUT, COPY, POST and LIST requests in one price group, separate from GET[S15], and that grouping, which would make Tier-1 the write side, is the likely meaning here (our interpretation). The design traded CPU for requests at a per-frame granularity, and at thousands of streams that trade went the wrong way.
Our analysis — the three meters compound
The rebuild kept the components and changed the boundaries: all of them “packed … into a single process” in one ECS task on EC2, frames handed over in memory, orchestration as in-process code[S1]. Both metered hops disappeared in one move, not two.
Green dashed: the hand-off that used to be an S3 PUT plus N GETs is now a pointer into the same heap. Remaining network egress per stream: the input stream in, rare SNS publishes, and aggregated results to S3.
The phrase “Conceptually, the high-level architecture remained the same”[S1] is doing more work than it appears to. The team kept “exactly the same components … media conversion, detectors, or orchestration”, which “allowed us to reuse a lot of code and quickly migrate”[S1]. This is the detail Yan Cui singled out as the under-read part of the story: code written for Lambda moved into a long-running container without a rewrite[S11]. For architects, the lesson is more specific than “Lambda lock-in is overstated”. The components’ interfaces were clean enough that the transport between them could be swapped, from S3 objects and state-machine JSON to in-process calls, without touching their logic. The expensive part of a microservice decomposition is usually drawing the boundaries; v1 had drawn them well and simply put the wrong transport across them.
v2 then ran into the obvious limit. In v1, detectors scaled horizontally because each was its own service; in v2 “the number of detectors only scale vertically because they all run within the same instance”[S1]. The team “regularly adds more detectors” and had “already exceeded the capacity of a single instance”[S1]. The fix was not to split detectors back into services. Instead they “cloned the service multiple times, parametrizing each copy with a different subset of detectors”, and added “a lightweight orchestration layer to distribute customer requests”[S1]. S1’s third diagram shows that layer as a Lambda forwarding each analysis request to separate ECS services, one for detectors 1 and 2 and one for detectors 3, 4 and 5, with per-detector aggregated results in S3 and real-time results on SNS[S1].
S1’s diagram shows two services (detectors 1–2 and 3–5) in separate ECS clusters, a Lambda forwarding each request to both, per-detector results to S3 and real-time results to SNS. The per-service task stacks are our rendering of “cloned the service multiple times”; S1 does not show how streams are assigned to tasks. The m-way fan-out happens once per analysis request, not once per frame or per second, and that difference in event rate is the whole redesign.
The final shape deserves a precise description, because the popular one (“a monolith”) is wrong in an instructive way. What shipped is m horizontally scaled services, each a single process, partitioned by detector subset, invoked by a serverless front door, emitting to managed pub/sub and object storage. Adrian Cockcroft’s description, a “single long running microservice that is horizontally scaled using ECS, and which is invoked via a lambda function”[S10], is accurate for one clone. The post’s word “monolith” is accurate relative to v1’s per-component split. Both are true, and neither is the interesting fact. The interesting fact is which hops stayed distributed: exactly the ones that fire once per analysis request.
~2019
VQA group starts using ML to find defects in video captured from consoles, TVs and set-top boxes (“three years ago” as of Mar 2022)
S2
Mar 2022
Amazon Science: detectors for 18 defect types; now being applied to real-time monitoring of “thousands of channels and live events”
S2
Undated
Monitoring tool built as Step Functions + Lambda + S3, “never intended nor designed” for high scale
S1
Undated
Onboarding more streams: hard scaling limit at ~5% of expected load; cost “too high to accept”
S1
Undated
Rearchitected to one process in one ECS task; later cloned per detector subset behind a Lambda
S1
15 Mar 2023
Same blog: live-sports publishing scheduler on SQS FIFO + Lambda, running “for the past five years”
S12
22 Mar 2023
Kolny publishes the post; first Wayback capture the next day
S1 S13
3–4 May 2023
InfoQ, Hacker News (989 points, 507 comments), DHH, The New Stack
S3 S7 S4 S8
5–8 May 2023
Vogels, DevClass quoting Sam Newman, Cockcroft, Yan Cui
S5 S6 S10 S11
2024
primevideotech.com starts answering 301 in Wayback captures; Vogels’ post now points readers to the Archive
S13 S5
Only the context (2019 to 2022) and the publication and reaction (March to May 2023) are dated[S2][S1][S13]. The v1 build, the wall, the experiments, the rewrite and the clone rollout are undated, as is team size, so any account giving a migration duration is inventing it. The record supports only the order: v1 built quickly, onboarding exposes the wall, incremental fixes tried, v2 built on reused code, v2 outgrows one instance, v3 clones it and is handling “thousands of streams” by publication[S1].
The post gives four quantities, and only one of them is a hard number. “Around 5% of the expected load” locates the wall. “Thousands” describes the stream count before and after. “Multiple state transitions for every second of the stream” describes the control loop. “Over 90%” describes the saving; the title rounds it to “by 90%”[S1]. There is no baseline, no absolute figure, no time window, and no statement of whether the saving is per stream or total[S1]. Most commentary treated the 90% as the finding. It is better treated as a constraint: a fact that any reconstruction of the old bill has to satisfy.
Start with scale. At today’s US-East list price of $0.000025 per Standard state transition[S9], a control loop that makes one transition per second for one stream costs 2,592,000 × $0.000025 = $64.80 per 30-day month. That figure is linear in everything: multiply by k for “multiple” transitions, and by the number of concurrent streams.
per month
k = 1
k = 3
k = 5
k = 10
per stream
$64.80
$194.40
$324.00
$648.00
1,000 streams
$64,800
$194,400
$324,000
$648,000
3,000 streams
$194,400
$583,200
$972,000
$1,944,000
$0.000025 per transition [S9, 2026 page] × 2,592,000 seconds in a 30-day month = $64.80 per stream per unit of k. Excludes the free tier (4,000 transitions a month, negligible here), retries (billed as transitions [S9]), Lambda, and S3 requests. Prime Video’s actual k, stream count, region, discounts and 2023 prices are unknown; this is a scale check, not a reconstruction of their bill.
At a thousand streams and five transitions per stream-second, the transition line alone is about $324,000 a month at list price. We do not claim this was Prime Video’s bill. Their k, their stream count, their 2023 price and any internal pricing are all unknown, and internal teams may not pay list price (DevClass noted that the mention of Savings Plans suggests internal customers are “billed according to a similar model”[S6], but that is an inference too). The table is a scale check. It shows that the orchestration line of a per-second loop over “thousands” of streams lands in the range where a team would call it “too high to accept”, and it would do so before a single S3 request or Lambda millisecond is counted.
Our analysis — at a throttle-bound wall, the bill stops depending on k
quota × price, independent of the design details we do not know. At today’s defaults[S14][S9]: 800/s × $0.000025 = $0.02 a second, or $51,840 per 30-day month; 5,000/s gives $0.125 a second, or $324,000. S1’s 5% supplies the other factor. Reaching expected load on a linear meter needs twenty times the rate, a quota of 16,000/s or 100,000/s, and a transition line of roughly $1.04M or $6.48M a month at list price, before any S3 request or Lambda millisecond. The team did not need to know k to know the projection was unacceptable; the quota and the 5% fixed it. Caveats: 2026 list prices and quotas; assumes the wall was sustained saturation of the transition bucket, not bursts, and a region in one of the two quota tiers. Internal pricing may differ.The frame path is harder to price, because we did not extract S3 per-request prices, but the counts are simple. At an illustrative one sampled frame per second, a thousand streams produce 1,000 PUTs a second, which is about 2.6 billion object writes per 30-day month, before any detector reads anything. At ten frames per second the figure is 26 billion. If each of N detectors reads each frame, reads are N times that. S1 does not give the sample rate, so these are shapes, not estimates. But they explain why a Senior SDE would single out “Tier-1 calls” as a line item. An object store’s request fee is designed for objects, and here each object was a single decoded frame with a lifetime of seconds.
Our analysis — the 90% is a statement about overhead share
α·P, where α captures how efficiently containers run the same work and how much v3 duplicates (it converts the stream once per clone). “Over 90%” then means α·P ≤ 0.1·(E + P), or E / (E + P) ≥ 1 − 1/(10α). Even if containers were twice as efficient as Lambda for the same work (α = 0.5), at least 80% of the old bill was per-event overhead. If the clones doubled compute spend through duplicated conversion (α = 2), at least 95% was.α = 0.5
v2 does the same work on half the compute spend
≥ 80.0%
α = 1
same compute spend before and after
≥ 90.0%
α = 2
clones duplicate conversion; compute spend doubles
≥ 95.0%
α = 3
more duplication, more headroom
≥ 96.7%
Old bill = E + P. New bill = α·P (containers doing the same detection). “Over 90%” means α·P ≤ 0.1·(E + P), so E / (E + P) ≥ 1 − 1/(10α). Savings Plans are excluded because S1 describes them as further, future savings.
This is the most useful thing that can be extracted from the 90%, and it agrees with the post’s own diagnosis. S1 names the orchestration workflow and inter-component data passing as “the two most expensive operations”, and never mentions compute as a cost problem[S1]. A system whose bill is 80% to 95% coordination is not suffering from its architectural style. It is suffering from a pricing mismatch between the granularity of the work and the granularity of the meter. Sam Newman made the same point in one line: the article “is really speaking more about pricing models of functions vs long-running VMs than anything”[S6].
Three more readings of the numbers are worth making explicit.
The 5% and the 90% measure different things. The wall is a throughput limit on one meter (transitions per second, account-wide). The saving is a cost ratio across all meters. They share a cause but not a unit. A team that had negotiated the quota up twentyfold would have reached expected load and paid roughly twenty times the v1 bill at the wall, since every metered line in the ledger is linear in stream-seconds. The saving is best compared not with the bill v1 actually ran at, which was capped by the wall, but with the bill v1 would have run at to reach expected load. The post does not say which comparison it makes.
Savings Plans are not in the 90%. S1 says moving to EC2 and ECS “also allowed us to use the Amazon EC2 compute saving plans that will help drive costs down even further”[S1]. The future tense matters. The 90% is before the commitment discount. A flat, predictable, always-on load, which is exactly what continuous monitoring of live streams is, is the best possible candidate for a compute commitment. This is a structural advantage the per-event design could not use in the same way, and a second source of saving the headlines left out.
Coverage changed at the same time. v1 ran at a fraction of the target. v3 monitors “all streams viewed by our customers”[S1]. So the 90% may compare a bill for partial coverage with a bill for full coverage, or two bills at the same coverage. The post does not say. If it is the former, the unit-cost improvement per monitored stream is larger than 90%, not smaller. Either way, anyone quoting the figure as “monoliths cost 10% of microservices” is quoting a ratio whose numerator and denominator are both undefined.
S1 describes four decisions: rewrite instead of fixing problems separately; pack all components into one process; scale by cloning the process with detector subsets; and replicate media conversion in each clone instead of converting once and caching the result[S1]. It names no alternatives, and it gives reasons for only the first two. The matrix below scores the options that commentators raised, plus the obvious ones, against both cost drivers. It is our assessment; nothing in the record says the team evaluated any of them.
Option
Throttle
Transition $
Frame path
Per-detector scaling
Note
Decision 1: rearchitect rather than tune. The post’s wording is “we initially considered fixing problems separately … We experimented and took a bold decision”[S1]. Section 4 gives the reason we think tuning could not work: the three meters trade off against one another, so every local fix moves cost from one meter to another. There is also a simpler argument. Each tuning option in the matrix keeps a metered event on the per-second or per-frame tier. The best it can do is cut the constant in front of a linear function of stream-seconds. Only moving the boundary changes the function. When the target is twenty times the current load and the overhead share is above 80%, cutting a constant by half still leaves you an order of magnitude out.
Counterfactual — Express Workflows
Decision 2: one process. The stated reason is that the distributed approach “wasn’t bringing a lot of benefits in our specific use case”[S1]. The benefits a per-detector split normally buys are independent scaling, independent deployment, fault isolation and technology freedom. It is worth asking which of those the workload actually needed. Independent scaling is the interesting one. Every detector processes the same stream at the same rate, so their load moves together. The only thing that varies between detectors is cost per unit of load, and that is a sizing problem, not a scaling problem. Independent deployment has value, but the post notes that adding a detector in v1 meant “creating a new microservice and plug it in to the orchestration”[S1], which is more work, not less, than adding a module to a process. Fault isolation per frame (a failed Lambda affects one frame) is real, but it is only valuable if someone acts on it, and for a monitor a lost frame is a lost sample, not a lost transaction.
The cost side of the ledger was concrete. Every per-frame hand-off became a pointer. Every per-second transition became a loop iteration. Orchestration logic that had been spread across state-machine definitions became code in one place; S1 says the single process “also simplified the orchestration logic”[S1].
Our analysis — Vogels’ four criteria, applied to the detectors
Decision 3: clone by detector subset. This is the most underrated decision in the post. Faced with a process that no longer fit one instance, the familiar move is to split detectors back into services, which would have rebuilt v1’s network hops. The team instead “cloned the service multiple times, parametrizing each copy with a different subset of detectors”, with “a lightweight orchestration layer to distribute customer requests”[S1]. Each clone contains the full pipeline, orchestration, converter and aggregation, and differs only in configuration. Partitioning happens at deploy time; nothing crosses a network on the hot path.
The trade-offs the post does not discuss follow directly from the shape:
Scaling becomes a bin-packing problem. A subset’s instance size is set by the most demanding detectors in it. Placing a new detector means finding a subset with headroom or creating a new clone, and rebalancing subsets is a redeployment. This is manual capacity planning, done at the rate the team adds detectors, which S1 says is “regularly”[S1].
The input is pulled m times. Each clone ingests the stream and decodes it on its own (Decision 4). Ingest bandwidth and decode CPU scale with m, the number of subsets. Keeping m small is therefore a cost objective, and it pushes toward larger instances and larger subsets, which pushes against the vertical limit that caused the cloning in the first place. The design has a natural optimum where decoding cost per clone balances instance-size limits, and it moves every time a detector is added.
Blast radius is a rectangle. Reconstruction (inferred): in v1 a failure was scoped to one invocation, roughly one frame for one detector on one stream. In v3 a crashed process loses every detector in its subset for every stream it was handling until it restarts, and one detector’s memory leak or pathological input can take down its subset neighbours. The post does not describe how streams are assigned to tasks or how failures are handled. But for a monitoring service the right comparison is with the alternative, which was not monitoring 95% of the target load at all.
Grid size, subset split and stream-to-task assignment are illustrative; S1 does not publish any of them. The account-wide token bucket is from S14. The middle panel is the one the “fault isolation” argument for v1 leaves out.
The “fault isolation” argument for v1 counts only crash domains. v1’s overload domain was the shared transition bucket: past the wall, every detector on every stream slowed together, along with any other Standard workflow in the same account and region[S14]. v3 has no shared per-event resource on the hot path, so overload stays in the clone that is short of capacity. Reconstruction (inferred) of what v3 then needs, none of it described in S1: crash detection and stream reassignment fast enough that the gap in coverage is shorter than the defects worth catching; a memory ceiling per detector, so one leaking model cannot evict its subset neighbours; and placement that spreads a single popular stream’s subsets across hosts, so one instance failure does not blind every detector on it.
The serverless front door stays. The Lambda in front of the clones fires once per analysis request[S1]. At that rate, a per-invocation fee is negligible and zero idle cost is a real advantage. This is the part of v1 that was right all along, and it survived because it sat on the right tier.
Decision 4: replicate conversion, do not cache it. S1 calls this one “not obvious”: “we replicated a computationally expensive media conversion process and placed it closer to the detectors”, because converting once and caching the outcome, “might be considered to be a cheaper option”, turned out not to be cost-effective[S1]. No reason is given. The reason is easy to reconstruct, and it is the same lesson as the whole case in miniature.
S1 says caching was “not … a cost-effective approach” and gives no reason. B wins when m × conversion cost is less than conversion cost + the cost of moving and storing decoded output m times. Decoded frames are much larger than the compressed stream they came from (raw 1080p in 4:2:0 is 1920 × 1080 × 1.5 ≈ 3.1 MB per frame by definition), so the inequality usually tips toward B.
Caching the converter’s output means moving decoded frames, which are far larger than the compressed stream they came from, across a network or through a store to every consumer. That is v1’s S3 hop again, with a different product name. Replicating the converter moves only the compressed stream and spends CPU m times. For small m and large decoded frames, CPU is the cheaper resource. This is the rare case where duplicating work is the optimisation, and it is the right call whenever the output of a stage is much larger than its input and the stage is cheaper to redo than to ship. The decision also implies m is small; if it grew large, the inequality would flip back toward a shared conversion tier.
Counterfactual — what if v1 had been built on containers from day one?
The post was published on 22 March 2023 and captured by the Wayback Machine the next day[S13]. Our sources show no coverage for six weeks. Then on 3 and 4 May it was covered by InfoQ, reached Hacker News (989 points and 507 comments at capture), drew a post from David Heinemeier Hansson and was covered by The New Stack[S3][S7][S4][S8]. Within four more days Werner Vogels, Adrian Cockcroft and Yan Cui had each responded, and DevClass had quoted Sam Newman[S5][S10][S11][S6]. A short post about one internal tool became a proxy for a decade-old argument.
Claim
What the record supports
Sources
“Amazon abandoned microservices”
One service in one org was consolidated; the same blog had just described an SQS + Lambda scheduler
S1 S12 S5
“It became a monolith”
Internally, yes. Externally: m cloned services behind a Lambda, emitting to SNS and S3
S1 S10
“Serverless doesn’t scale”
S1 says “the way we used some components” hit the limit; Standard quotas are soft today
S1 S14
“Monoliths are 90% cheaper”
No baseline, no absolute numbers, no window; the saving is removal of per-event pricing on a hot loop
S1 S6 S11
“They designed it wrong”
Built quickly for a tool not designed for scale; refactored once load was known
S1 S10 S11
“The video streaming moved to a monolith”
It was the quality-monitoring service, not delivery
S8 S1
“The post was pulled out of embarrassment”
The whole primevideotech.com site redirects from 2024; nothing ties that to this post
S13 S5
“Amazon abandoned microservices.” One service inside one organisation was consolidated. The same blog, one week earlier, had described a live-sports publishing scheduler built on SQS FIFO queues and a periodically invoked Lambda implementing a centralised leaky bucket, which had been running “for the past five years”[S12]. Vogels pointed to both posts together to argue “there is no one-size-fits-all” and that “no particular architectural style is mandated”[S5]. The contrast is worth more than a rebuttal. The scheduler is a control-plane system. Its work items are discrete, bursty, and weighted by an abstract cost[S12]; rate-shaping them is the whole point, and per-event services are a natural fit. The monitoring service is a data-plane system with a flat, time-driven load. The same organisation, in the same month, made opposite platform choices for the same underlying reason.
“It became a monolith.” Inside each clone, yes; the post uses the word itself, and the diagram shows every component inside one ECS task[S1]. From outside, v3 is a Lambda fanning requests out to several independently deployed ECS services, which publish to SNS and write to S3. Cockcroft’s verdict, “The result isn’t a monolith” and “it’s clearly a microservice refactoring step”[S10], is as defensible as the post’s own framing. The argument over the label is sterile. What matters is the granularity change, which neither label describes.
“Serverless doesn’t scale.” The post’s own sentence is narrower: “the way we used some components caused us to hit a hard scaling limit”[S1]. The limit was an account-level throttle on a soft quota[S14], reached because of a per-second usage pattern. It says nothing about Lambda’s ability to fan out, or Step Functions’ ability to run large numbers of short workflows. Yan Cui’s rough threshold, that above about 1,000 requests per second API Gateway and Lambda “can be much more expensive to operate than running a bunch of containers”[S11], is the more accurate summary. It is a claim about cost at sustained throughput, not about scalability.
“Monoliths are 90% cheaper.” Section 6 covers this: no baseline, and a saving that measures the removal of per-event pricing from a hot loop[S1][S6]. The same monolith with an S3 hop per frame and a metered orchestration tick would have had the same bill.
“They designed it wrong.” The strongest form of this came from Hacker News. One commenter wrote that it was “not a discussion of monolith vs serverless. This is some terrible engineering all over that was ‘fixed’”, and the top comment said “I’m sort of gob smacked this article exists”[S7]. The analyst Donnie Berkholz, quoted by The New Stack, called the post “an absolute embarrassment for Amazon as a company”[S8]. The record is more forgiving and more interesting. S1 says the tool was built quickly, for a purpose that was “never intended nor designed” for high scale[S1]. Cockcroft calls it a rapid prototype turned into a long-running container, exactly what his “Serverless First” advice recommends, and Cui describes the same “serverless-first” path: start serverless to go to market quickly, then move to containers when scale makes them cheaper[S10][S11]. Newman: “Changing your mind when the situation changes is totally sane”[S6]. Our position is in between. Passing video frames through an object store is a design that an engineer who had written down the event rate would not have chosen. But the event rate is the thing a prototype is supposed to discover, and v1 discovered it, in a form (an account limit) that was loud and cheap to diagnose.
“The video streaming moved to a monolith.” The New Stack’s article described Prime Video “moving the video streaming to a monolithic architecture”[S8]. It was the quality-monitoring service, a side-channel observer of streams, not the delivery path[S1]. The distinction matters for anyone citing the case. The delivery path has different availability requirements, different load shape and different owners, and the post says nothing about it.
“AWS forced Lambda on its teams” and “the post was pulled.” Cui reports the forced-Lambda theory was “debunked by people who were closer to the leadership”, without naming evidence[S11]. Hacker News commenters predicted a takedown[S7]. What actually happened is that the entire primevideotech.com site began redirecting in 2024; Wayback captures of the post’s URL first show a 301 on 25 March 2024 and consistently from 5 August 2024[S13]. Vogels’ post now carries a note that the site “is no longer available” and points readers to the Archive[S5]. No source connects the site’s retirement to this post, and treating it as a retraction goes beyond the evidence.
Our analysis — why this post, and why this reaction
The engineering record ends with the post. It reports v3 handling “thousands of streams” with capacity to scale further, and Savings Plans to come[S1]. No follow-up from Kolny or VQA, no talk and no revised figures appear in our sources. We found no later public account of how the clone layout evolved, how many subsets exist, what the detector count became, or whether the Savings Plans saving was realised. Anyone writing that part of the story is guessing.
The Amazon Science article offers the only forward-looking context. As of March 2022 VQA was using “custom AWS cloud-native applications and SageMaker implementations to scale our defect detectors, to monitor all live events and video channels”[S2]. That sentence predates S1, and its phrase “to monitor all live events” matches the post’s final result: monitoring “all streams viewed by our customers”[S1]. Read together, the two documents describe an objective set in 2022 and reported as met in March 2023. Whether the 2022 phrase referred to the v1 design, v3, or an interim state is not stated.
What did continue was the post’s career as a citation. Vogels’ reply, still live, frames it with the line most often quoted from the episode: “Building evolvable software systems is a strategy, not a religion”[S5]. His rule of thumb, “with every order of magnitude of growth you should revisit your architecture”[S5], fits the case almost exactly. The wall sat at about 5% of the target, a factor of twenty, and the team revisited the architecture when onboarding made that factor visible. The post itself now lives only in the Wayback Machine[S13].
Our analysis — the durable lesson is about when you find out
The redesign is best summarised as moving the network boundary from the busiest tier of the workload’s event hierarchy to the quietest. Every workload has such a hierarchy. For this one, the tiers are per analysis request, per stream-second and per frame. The question for your own system is which tier each metered, throttled or networked boundary sits on.
Event tier
What it is
Events per stream
v1 crossed the network via
v3 handles it with
Per analysis request
once per stream
1
Step Functions start + Lambda
Lambda fan-out to m clones
Per stream-second
orchestration tick
k × T
Step Functions transitions
a loop inside the process
Per frame
data hand-off
f × T (× N reads)
S3 PUT + GETs
a pointer in memory
T = stream duration in seconds. For an illustrative three-hour event, T = 10,800: the per-second tier is four orders of magnitude busier than the per-request tier, and the per-frame tier is busier again by the sample rate. v3 keeps a managed, metered, distributed hop only where the rate is 1.
The transferable principle is not “prefer monoliths” and not “avoid Step Functions”. It is: keep a metered or throttled hop only where its firing rate is proportional to business requests, not to time or to data volume. v3 keeps exactly one managed hop per analysis request (the Lambda front door) and one per defect (SNS)[S1]. Both are low-rate. Everything proportional to time or to frames happens inside a process.
Decision criteria — our synthesis
1 · Clock-driven load. Does any boundary fire per unit of time per entity (per stream-second, per device-minute, per session-tick) rather than per business request? If so, its monthly cost per entity is fixed by the tick rate, not by usage, and pay-per-event pricing has lost its main advantage. At today’s Standard rate, a one-per-second tick is $64.80 per entity-month[S9]; compare that with the entity’s share of a long-running process.
2 · Payload size vs orchestration payload limits. If the data between stages is larger than the orchestrator’s payload limit (256 KiB for Step Functions today[S14]), you are forced into claim-check hand-offs, and every hand-off becomes at least one write and one read per consumer. Acceptable for per-request documents; ruinous for per-frame or per-record data.
3 · Process lifetime vs execution lifetime. If the logical process runs as long as a live event or longer, compare that with the engine’s execution bounds: 25,000 history events for Standard, five minutes for Express[S14]. A mismatch means you will build continuation logic, and continuation is itself metered and throttled.
4 · Account-wide shared quotas. Is the limit per workflow, or per account and region? A shared token bucket turns entity count into a hard concurrency ceiling (quota / events-per-entity-second) that no amount of downstream capacity can lift.
5 · Overhead share. Estimate E / (E + P). If per-event fees are more than about half the bill and the load is flat, a move to long-running compute will likely cut cost by at least half, whatever the topology. If E is small, the style change buys you nothing on cost and you should decide on other grounds.
6 · Correlated component load. Do the components you split into services scale together because they see the same input? If yes, “independent scaling” is illusory, and the only real variable is per-component cost, which is a partitioning-by-size problem (v3’s detector subsets), not a service-boundary problem.
7 · Output much larger than input. For a stage whose output is much bigger than its input (decoding, decompression, feature extraction), replicating the stage next to each consumer usually beats sharing its output over a network, while the number of consumers stays small.
Questions to ask in the design review — our synthesis
Rate. For each arrow on the diagram: how many times does it fire per entity per second at target load, and what is rate × unit price × seconds per month? If nobody can answer for the busiest arrow, the review is not finished.
Headroom. Which quota is shared across the account or region, what is quota / per-entity rate, and what fraction of target load is that? Prime Video’s answer turned out to be about 5%[S1].
Overload. When that quota saturates, does work queue (lag grows) or drop (coverage falls), and which metric pages someone? Error rate will not.
Lifetime. How long does one logical process live, against the engine’s execution bounds, and who owns the continuation logic?
Payload. What crosses each hop by value, and what by reference? For every by-reference hop: requests per unit of work, times consumers.
Overhead share. What is E / (E + P) at target load? Above one half on a flat load, the compute platform is the wrong question and the boundary is the right one.
Correlation. Do the separately scaled components see the same input? If yes, what are you actually scaling independently?
Next wall. After the fix, which limit arrives next, and at what multiple of today’s load? For v3 it is the size of one instance[S1] and, our inference, rebalancing subsets every time a detector is added.
When the lesson does not transfer. Invert each criterion and the serverless design wins. Cockcroft notes that “many enterprise workloads are intermittent and small scale and very good candidates for a serverless first approach”[S10]. Cui, citing Vlad Ionescu, points out that Lambda scales much faster than container services under unpredictable spikes, and that over-provisioning containers for peaks wastes EC2 spend on idle CPU[S11]. Prime Video’s own scheduler post is an example: bursty, cost-weighted work items, where the right design shapes the rate of discrete events and pays per event[S12]. If your load is spiky, low-volume, request-driven, or made of small payloads, v1’s shape may be correct, and copying v3 would give you idle capacity and a capacity-planning job for no benefit.
Where the monolith lesson itself runs out. v3 moved to vertical scaling per clone, and the team had already exceeded one instance[S1]. The subset design holds while subsets can be balanced by hand and m stays small enough that duplicate ingest and decoding are cheap. If the detector count grew by an order of magnitude, or detectors needed different hardware, the natural next step is not back to per-detector services on the frame path. It is a small number of heterogeneous clone types, each placed on suitable hardware, still sharing nothing on the hot path. Vogels’ rule, revisit at every order of magnitude[S5], applies to v3 as well.
Bottom line
S1 is the only first-hand engineering account and is quoted from the Internet Archive copy, since primevideotech.com is offline. S2 and S12 are Amazon primary sources on related systems. S5 is an Amazon executive response. S3, S4 and S6–S8, S10 and S11 are secondary commentary, cited only for what they claim. S9, S14 and S15 are current (2026) AWS pricing and quota pages, used only to reason about the kind and scale of limits; the 2023 values Prime Video faced are not published. Every figure labelled illustrative or model is our arithmetic on those pages, not a reported Prime Video number.
Scaling up the Prime Video audio/video monitoring service and reducing costs by 90% (Wayback capture; original site offline) — Marcin Kolny, Prime Video Tech, 22 Mar 2023
How Prime Video uses machine learning to ensure video quality — Amazon Science, 4 Mar 2022
Prime Video Switched from Serverless to EC2 and ECS to Save Costs — Rafał Gancarz, InfoQ, 3 May 2023
Even Amazon can’t make sense of serverless or microservices — David Heinemeier Hansson, HEY World, 4 May 2023
Monoliths are not dinosaurs — Werner Vogels, All Things Distributed, 5 May 2023
Reduce costs by 90% by moving from microservices to monolith: Amazon internal case study raises eyebrows — Tim Anderson, DevClass, 5 May 2023
Hacker News discussion of S1 (989 points, 507 comments at capture) — Hacker News, 4 May 2023
Return of the Monolith: Amazon Dumps Microservices for Video Monitoring — Joab Jackson, The New Stack, 4 May 2023
AWS Step Functions pricing — Amazon Web Services, retrieved Sep 2026
So many bad takes — What is there to learn from the Prime Video microservices to monolith story — Adrian Cockcroft, Medium, 6 May 2023
Is serverless overpriced? What can we learn from the PrimeVideo team? — Yan Cui, theburningmonk.com, 8 May 2023
Shaping live sports publishing traffic through a distributed scheduling system (Wayback) — Paul Martin & Vishal Anand, Prime Video Tech, 15 Mar 2023
Wayback Machine CDX capture history for S1 — Internet Archive, captures Mar 2023 – Sep 2026
Step Functions service quotas — AWS Documentation, retrieved Sep 2026
Amazon S3 pricing (request price groups) — Amazon Web Services, retrieved Sep 2026