A model that works in a notebook and a model that serves ten thousand requests per second at a 99.9% availability target are separated by an entire discipline. That discipline is LLM Ops: the practice of running a probabilistic, expensive, GPU-bound service in production — reliably, affordably, and safely — long after the demo has impressed everyone. This chapter defines the field and draws the map for the rest of the track.
It is deceptively easy to make a language model do something impressive once. Paste a prompt, get a brilliant answer, ship a screenshot. The distance between that moment and a dependable product is where LLM Ops lives. The instant a model sits on a request path that real users depend on, you inherit a service with three uncomfortable properties at once: it is probabilistic (the same input can yield different output), it is expensive (metered per token, on the hot path, forever), and it is GPU-bound (its performance is governed by scarce, memory-limited hardware you may not even own).
LLM Ops is the engineering discipline of holding that service to production standards regardless. It means an explicit SLO — say, a p99 time-to-first-token under a second, a p99 end-to-end budget for non-streamed calls, and 99.9% availability (an end-to-end p99 on a streamed response mostly measures how long users asked the model to talk) — and the machinery to defend it: batching and caching for throughput, routing and failover for resilience, guardrails for safety, evals for quality, and cost attribution so the bill doesn't quietly triple. Crucially, a live LLM service is never "done." It sits inside a continuous operating loop — you deploy, serve, observe, evaluate, improve, and roll out again — because the model, the traffic, the prompts, and even the provider underneath you all change over time.
Deploy
ship a model + config behind the gateway
Serve
answer live traffic at an SLO
Observe
trace latency, cost, errors per request
Evaluate
score quality on real + golden data
Improve
tune prompt, model, routing, params
Roll out
canary → shadow → full, or roll back
The mental shift is from "it works" to "it keeps working, for everyone, within budget, under load, when the provider has a bad day." Everything in this track is a tool for closing some part of that loop.
What makes this genuinely different from operating any other service is the failure that defines the field: the 200-OK regression. A prompt edit, a retrieval tweak, or a provider updating the weights behind an unchanged model alias shifts output quality, and every signal your existing stack emits stays green. Status codes are fine, latency is fine, error budgets are untouched. Traditional monitoring was built on the assumption that a wrong answer and a failed request are the same event; for an LLM they are orthogonal. The incident below is illustrative, but its shape is the common one: detection is gated on humans noticing, so time-to-detect is measured in days, not minutes.
Two consequences follow. First, a prompt is a deploy: anything that changes model input or model identity (prompt templates, retrieval config, tool schemas, the pinned model version, sampling params) needs the same change management as a binary, including a rollback path. Teams that store prompts in a CMS "so PMs can iterate" have built an unaudited production deploy channel. Second, the SLO needs a quality indicator alongside availability and latency, even a crude one: sampled online evals, a structured-output parse rate, or a refusal rate. Without one, your error budget is a statement about the transport, not about the product. Chapter 6 — Observability & Evaluation Ops builds that instrumentation.
The part that surprises teams is that a quality SLI does not behave like an availability SLI. Availability is counted on every request and costs nothing to measure. Quality is estimated from a judged sample, so it is a statistical test with a detection floor and a price per data point. Suppose task success sits at 80% and you want to detect a 5-point drop with the usual 5% false-alarm rate and 80% power. A two-proportion test needs roughly 1,100 judged requests in the window, per variant. A 2-point drop needs about 6,500, and a 1-point drop about 25,000. The window has to contain that many judged requests, so the SLI's alerting latency is set by judged volume, not by how often you evaluate the metric. A feature doing 20k requests a day that judges 5% of them gets 1,000 judgments a day. It can see a 5-point drop within about a day but needs close to a week of traffic to see a 2-point drop, unless it judges more or accepts more false pages. Nine days of silent regression in the timeline above is what "no quality SLI" looks like; six days is what an underpowered one looks like.
80% → 70% (−10 pts)
~290 / arm · $3 judged
80% → 75% (−5 pts)
~1,090 / arm · $11 judged
80% → 78% (−2 pts)
~6,500 / arm · $65 judged
80% → 79% (−1 pt)
~25,550 / arm · $256 judged
Two-proportion test, α = 0.05 two-sided, power 0.8, independent samples. Judge cost at an illustrative $0.01 per judgment. Halving the detectable drop roughly quadruples the sample count, and the window has to hold that many judged requests: a 2-point alert on an hourly window needs ~6,500 judged requests per hour, per variant, per suite.
Two further effects make this harder than the formula suggests. First, the judge is noisy. A judge that disagrees with human labels on a meaningful share of items adds variance, so you need more samples than the textbook figure. It can also be biased in a direction that correlates with the change you just made, for example favouring longer answers right after a prompt edit made answers longer. Second, temperature 0 is not deterministic in production. Inference servers batch requests together, and the floating-point reduction order in batched kernels depends on what else is in the batch, so the same prompt can yield different tokens depending on concurrent traffic. Once one token differs, the rest of the generation diverges. The operational consequence is that golden-output tests which diff exact strings raise false alarms on unchanged systems. Regression checks have to compare distributions of scores, with enough repeats per item to separate a real shift from sampling noise, not compare one output with another.
LLM Ops grew out of MLOps, but its center of gravity moved. MLOps was organized around models you trained: feature pipelines, training runs, experiment tracking, model registries, and the eternal fight against training/serving skew and data drift. The hard, spiky work happened offline, in batch, before anything reached a user. Serving was often the easy last mile — wrap the model in an endpoint and move on.
LLM Ops inverts that. You usually don't train the model — you rent or download one someone else spent millions training — so the entire discipline shifts onto inference. The work that used to be an afterthought becomes the whole job: serving throughput, token economics, prompt and context construction, model routing, guardrails, and GPU capacity. The pipeline is no longer "data to model"; it's "prompt to answer," and it runs live on every single request.
MLOps
LLM Ops
A model you trained on your data
A model someone else trained — you operate it
Training, feature pipelines, versioning
Inference: serving, routing, context, guardrails
Data → features → train → validate → deploy
Prompt → retrieve → call → validate → stream
Training compute (spiky, offline)
Per-token inference (continuous, on the request path)
Held-out metrics: accuracy, AUC
Evals: LLM-as-judge, groundedness, task success
Data drift, stale features, skew
Hallucination, injection, provider drift, cost blowout
Batch GPUs for training jobs
GPUs on the hot path — memory-bound, always on
This is why an experienced MLOps team can still be surprised by their first LLM system. The instincts about versioning and reproducibility transfer; the instincts about where the cost and risk live do not. In classical ML, the expensive, risky moment was training. In LLM Ops, it is every inference — continuously, on the request path, priced by the token.
The second-order effect is that cost becomes a property of code review, not of capacity planning. In a classic service, a 2× cost change needs a 2× capacity change, which someone has to approve. Here, cost per request is a product of input tokens, output tokens and calls per request, and all three are set in config. The worked example below uses illustrative prices, but the ratios are what matter.
Prefix caching on
2.5k of 3k input is a stable prefix, billed at an illustrative 10%
$9.8k/day · 0.6×
Baseline
3k in · 500 out per request
$16.5k/day · 1.0×
RAG top-k 5 → 20
12k in · 500 out — one config line
$43.5k/day · 2.6×
Agent loop, 4 calls
4 × (6k in · 400 out) — context re-sent every step
$96.0k/day · 5.8×
Assumptions (illustrative, not any provider's list price): 1M requests/day, $3 / M input tokens, $15 / M output tokens. Baseline = 3B input ($9.0k) + 0.5B output ($7.5k). The spread from cheapest to most expensive row is ~10× — and every row is a prompt or config change, not a capacity change.
Read the multipliers, not the dollars. Raising retrieval top-k is a one-line change that makes the feature 2.6× more expensive, and nobody filed a capacity request. Turning a single call into a four-step agent loop is 5.8×, because each step re-sends an ever-growing context. Output tokens cost 5× input here, so a prompt that asks for "detailed reasoning" moves the bill more than a prompt twice as long. Going the other way, putting the stable part of the prompt first so prefix caching can hit is the cheapest optimization available, and it only works if the system prompt stays byte-identical across requests. A timestamp or user name in line one silently defeats it. The operational rule that follows: track tokens per request as a first-class metric and diff it in CI, the same way you would guard a latency or bundle-size regression.
Tokens per request is necessary, but it is still the wrong denominator. The unit that matches the business is cost per successful task, and it can move the opposite way from cost per call. Take an illustrative cascade: a small model costs 1 unit per call and succeeds 70% of the time, and a large model costs 10 units and succeeds 95%. Large-only costs 10 ÷ 0.95 ≈ 10.5 units per success. Small-first with escalation on failure costs 1 + 0.3 × 10 = 4 units per attempt and succeeds 0.7 + 0.3 × 0.95 ≈ 98.5% of the time, about 4.1 units per success. That is a 2.6× saving, and it rests entirely on an assumption that is usually left unstated: that you can detect the small model's failures. If your failure detector (schema validation, a verifier, a confidence heuristic) catches only 60% of them, 12% of all tasks ship a wrong answer that nobody escalated. Your cost chart improves, and the quality loss is spread across enough users that no single complaint looks like an incident. So the recall of the failure detector is a production dependency of the cost model, and it needs its own measurement.
LLM Ops is often confused with AI-native design, but they answer different questions. AI-native architecture asks what to build: should this be retrieval-augmented generation or an agent, how does context flow, which model handles which task, what are the prompts and schemas. It is the blueprint. LLM Ops asks how to run it: how to keep it up, make it fast and cheap, keep it safe, and keep it honest as traffic and models change. It is the operations manual for that blueprint once it is load-bearing.
AI-native design
what to build
Choose the pattern
RAG vs agent vs fine-tune
Shape the data flow
retrieval, memory, tools
Pick the models
capability per task
Define the contract
prompts, schemas, interfaces
LLM Ops
how to run it
Keep it up
SLOs, uptime, failover
Make it fast & cheap
batching, caching, routing
Keep it safe
guardrails, rate limits, abuse
Keep it honest
evals in CI, drift alerts, canaries
A brilliant design that falls over at 10k rps is a bad product; a flawlessly operated system built on the wrong pattern is an expensive one. You need both — this track owns the right column.
The two are not sequential phases but two lenses on one running system. A design decision — say, stuffing the full document into every prompt — is also an ops decision, because it sets your per-request cost and your latency tail. Conversely, an ops constraint — a p99 budget, a GPU quota — feeds back into the design. This track deliberately takes the operator's seat. We assume something has been designed and now must survive contact with real users, real load, and a real invoice.
The coupling is easiest to see through maximum context length, which usually gets decided in a design doc as a capability question ("can it read the whole contract?"). On a self-hosted fleet it is a capacity question. Each token held in context occupies KV cache for the whole life of the request. For a 70B-class model with grouped-query attention (80 layers, 8 KV heads, head dimension 128, FP16), that is about 0.33 MB per token. On an illustrative 8×80 GB node, after weights and runtime reserve, roughly 440 GB is left for KV. That holds about 330 concurrent sequences at 4k tokens and about 40 at 32k.
4k context
1.3 GB KV / seq
~328 sequences
8k context
2.7 GB KV / seq
~164 sequences
32k context
10.7 GB KV / seq
~41 sequences
128k context
42.9 GB KV / seq
~10 sequences
KV bytes/token = 2 (K and V) × 80 layers × 8 KV heads × 128 head dim × 2 bytes ≈ 0.33 MB. With ~440 GB left for KV after weights and runtime reserve, a sequence that actually fills 32k tokens holds ~10.7 GB. Paged attention allocates only what a sequence uses, so this is the ceiling for the requests that do use the window. They are the ones that set your tail.
Paged KV allocation means you only pay for the tokens a request actually uses, so a 32k limit costs nothing by itself. What costs is the distribution. If 10% of requests carry 30k tokens and the rest 3k, the long tail holds about half the KV token-slots while being a tenth of the traffic, and more than half once you weight by how much longer those requests live. Those requests also take the longest to prefill, and when the cache fills the scheduler preempts or queues them, so they drive p99 for everyone. On a hosted API the same coupling shows up as price and quota instead of memory: long-context requests are billed per token and draw down tokens-per-minute quota in proportion to their size. The design review question is therefore not "what is the max context?" but "what is the p95 context, and who pays for it?" The usual answers are a separate pool or route for long-context traffic, a hard cap per tenant, or retrieval that trims context before it reaches the model. Each of those is a design change made for an operational reason.
When people say "LLM platform," they mean a stack of operational layers, each with its own failure modes and dials. At the bottom is compute — the GPU fleet, its scheduling and quotas. On top sits the serving runtime (vLLM, TGI, SGLang, or a hosted API) that turns those GPUs into throughput via batching and the KV cache. Above that, a gateway and control plane handles auth, rate limiting, key management, model routing between cheap and strong models, guardrails, and caching. Then observability — tracing, metrics, SLOs, and online evals. And wrapping everything, cost governance: budgets, quality gates in CI, access control, and audit.
Governance
Cost budgets
per team / per feature
Evals in CI
quality gate
Access & audit
who called what
Observability
Tracing
per-span cost / latency
Metrics & SLOs
p50 / p99 / error rate
Online evals
score live traffic
Gateway & control
LLM gateway
auth · rate limit · keys
Model router
cheap ↔ strong · failover
Guardrails
in / out filtering
Caching
exact + semantic
Serving runtime
vLLM / TGI / SGLang
batching · KV cache
Hosted API
rent the fleet
Autoscaler
replicas ↔ load
Compute
GPU fleet
H100 / A100 / L4
Scheduling
bin-pack · spot
Capacity & quota
reservations
On a hosted API you rent the bottom two lanes and own the top three. Self-hosting means you own all five — which is why the build-vs-buy line is really a question of how many of these layers you want to operate.
The reason this layering matters is that build-vs-buy is a layer-by-layer decision, not an all-or-nothing one. Rent a hosted API and you outsource the bottom two lanes but still own the top three — a gateway, observability, and governance are yours no matter who runs the GPUs. Self-host and you inherit all five, including the hardest, most operationally demanding ones. Most of this track walks up this stack lane by lane, starting from the runtime in Chapter 2.
The gateway lane is where teams first discover that LLM traffic breaks assumptions baked into their existing infrastructure. Those assumptions are about request duration. A typical API call finishes in tens of milliseconds, while a streamed LLM response lasts seconds. By Little's law, concurrency = arrival rate × time in system. Take an illustrative feature at 200 rps with 0.6 s to first token and 400 output tokens streamed at 50 tok/s: each request lives about 8.6 s, so the gateway holds about 1,700 open streams. A conventional 50 ms service at the same rate holds 10. Everything sized for the second number misbehaves at the first. That includes per-host connection pools, file-descriptor limits, load-balancer idle timeouts (commonly on the order of 60 s by default, which silently cuts off non-streamed generations or queued requests that send no bytes before the limit), deploys whose connection draining now takes tens of seconds, and autoscalers keyed on CPU, which barely moves because the gateway is mostly waiting.
Hosted providers add a second mismatch: quotas are usually expressed in tokens per minute as well as requests per minute. At an illustrative 2M TPM quota and 4k tokens per request, the ceiling is 500 requests a minute, about 8 rps, for the whole organisation's key. A retrieval change that doubles context halves that ceiling without touching anything anyone would recognise as the capacity path. Quota is a shared, finite resource sitting behind a single key. That is why it belongs to the gateway, with per-feature sub-budgets, rather than being discovered as 429s by whichever team's traffic happened to arrive last.
~8.6 s
life of one streamed request
~1,700
open streams at 200 rps
~8 rps
2M TPM ÷ 4k tok/req
60 s
a common LB idle default
The usual framing of self-hosting ("GPUs are cheaper than API calls at scale") is wrong in a specific way: the GPU price barely matters, utilization does. A self-hosted node costs the same per hour whether it is saturated or idle, so its cost per token is hourly cost ÷ tokens actually served. Take an illustrative 8-GPU node at $30/hr that sustains 3,000 output tok/s when saturated, about 10.8M tokens an hour. That is roughly $2.8/M at 100% utilization, $5.6/M at 50% and $11/M at 25%. Against an illustrative hosted price of $6/M, the break-even is near 46% average utilization. Add an amortized share of the platform team and on-call rotation you now need, and it moves to roughly 69%.
Now do the capacity arithmetic most business cases skip. Traffic with an illustrative 3:1 peak-to-trough diurnal swing averages about two-thirds of peak. Provision for peak with 30% headroom to protect p99 (queueing delay rises steeply as utilization approaches saturation, which Chapter 2 covers), and average utilization comes out around 0.67 × 0.7 ≈ 47%. That puts you at break-even on raw GPU cost and underwater once people are counted. So the decision criteria are sharper than "volume". Self-hosting pays when you have flat or batchable load (offline jobs back-filling the troughs), a smaller model you can saturate, or a hard constraint such as data residency or a fine-tuned model no provider will serve. It rarely pays for spiky, interactive traffic on a frontier-class model. Chapter 3 — GPU Infrastructure & Scheduling works through the full cost-per-token model.
Nearly every LLM Ops decision is a negotiation between three forces that pull against each other: reliability (uptime, SLOs, failover), cost ($/token, GPU utilization, cache hit rate), and quality and latency (eval score, p99, time-to-first-token). You can push any two hard, but rarely all three, and never for free. This is the operator's version of the design-time cost/latency/quality triangle — here it governs infrastructure choices, not just model choices.
Reliability
uptime, SLOs, failover, graceful degradation
Cost
$/token, GPU utilization, cache hit rate
Quality / latency
eval score, p99, time-to-first-token
Add a second replica for reliability and cost rises. Batch harder for cost and tail latency rises. Route to a bigger model for quality and both cost and latency rise. There is no setting that maximizes all three — ops is choosing where to stand.
Watch how tightly coupled they are. Add a replica in a second region and reliability improves, but cost rises. Turn the batch size up to cut cost per token and tail latency stretches. Route to a larger model to lift quality and you pay more and wait longer. A shared endpoint is cheap but couples your tail latency to other tenants' traffic; dedicated capacity fixes that and raises your bill. The job is not to eliminate these tensions — it's to choose, deliberately and per feature, where to stand, and to make that choice visible with numbers rather than vibes.
Reliability
Uptime · SLO
Cost
$ / token
Latency
p99 · TTFT
Quality
Eval score
The coupling is sharpest during incidents, when the reliability mechanisms you added attack the other two dials. Follow a typical cascade. A provider starts returning 429s or timing out at the 60-second mark. Clients retry three times with no shared retry budget, so offered load goes up to 4×, the provider throttles harder, and your own gateway's concurrency limit fills with requests parked on long timeouts. Each retry of a long-context request re-pays the full input-token cost (and on a timeout you may already have been billed for tokens generated before the cut). So the incident bill scales with the retry multiplier while success rate falls. Then the automatic failover to a second provider kicks in, and it quietly swaps a model your prompts were never evaluated against. Structured-output parse failures climb, and downstream code retries those too.
The conventional advice ("add retries, add a fallback provider") is not wrong, but it is incomplete in ways that matter. Retries need a budget (for example, retries capped at a small percentage of successful calls across the fleet, not a per-request count). Timeouts should track the SLO, not the provider default: a request that cannot meet its p99 is better failed fast than completed late. And a fallback model is a second production model that must pass the same eval gate as the primary, with its own prompt variants if needed. Otherwise failover converts an availability incident into a quality incident, which, per 1.1, is the kind you won't see. Chapter 7 — Reliability, Cost & Rollouts designs these controls.
Expected attempts = (1 − p⁴) ÷ (1 − p) with three retries. Every attempt re-sends, and re-bills, the full input context. The amplification is worst exactly when the provider is least able to absorb it.
The arithmetic behind the cascade is worth doing once. With three retries and an upstream failure probability p, each logical request makes (1 − p⁴) ÷ (1 − p) attempts on average: 1.1× at 10% failing, 1.9× at 50% and 3.4× at 90%. Now price it with the illustrative baseline from 1.2, about $16.5 per thousand requests, most of it re-sent input context. At 50% failure, each logical request re-sends its context 1.9 times on average and about 94% eventually succeed, so the cost per successful request is up about 2× on the part of the bill that is being retried, assuming failed attempts are billed. The static figure understates the damage, because p is not independent of the retries. The extra load is what the provider is throttling, so p rises towards the right-hand side of the curve, and the system settles near 3–4× offered load with a falling success rate. That is a metastable failure: it continues after the original trigger has cleared. A fleet-wide budget that caps retries at 10% of calls holds amplification at about 1.1× whatever p does. It trades a lower success rate during the incident for a provider that can recover and a bill that stays flat. That is the right trade for almost every interactive feature, because a user who waited 3 × 60 s for a timeout has already left. The exception is idempotent batch work with no user waiting, where retries belong in a queue with backoff measured in minutes, not in the request path.
At any real scale, no single person operates an LLM system, so LLM Ops is also an organizational discipline — a shared-responsibility model. Typically a platform (or ML-platform) team owns the shared substrate: the serving runtime, GPU capacity, the gateway, routing, guardrail policy, and cost attribution, so that every product team doesn't reinvent them. Product engineers own what's specific to their feature: the prompts, the context and retrieval logic, and the evals that define "good" for their use case. And SRE / on-call owns the operational front line: SLOs, alerting, and incident response.
Platform / ML-platform
Product engineers
SRE / on-call
The platform team runs the shared substrate so product teams don't each reinvent serving and gateways; product owns the parts specific to their feature; SRE owns the pager. Blurring these lines is how a "someone should have caught that" incident happens.
These lines can be drawn differently — a small startup may collapse all three roles into one team — but they must be drawn explicitly. The classic LLM incident is a quality regression or a cost spike that everyone assumed someone else was watching: product thought the platform's guardrails would catch it, the platform assumed product's evals would, and SRE had no signal for it at all. A clear ownership map, backed by shared observability, is what turns "someone should have caught that" into "the eval gate caught it before rollout."
The seam that most often goes unowned is the model version itself. Product owns the evals, but the eval suite is only run when product changes something. The platform owns the gateway, but a hosted alias like "latest" can move underneath it without a deploy. The fix is structural: pin explicit model versions at the gateway, treat a version bump as a platform-initiated rollout that must re-run every consuming team's eval suite, and alert on the provider's deprecation schedule the way you would on a certificate expiry. A related trap: an LLM-as-judge eval is itself a model dependency. If the judge's version drifts, scores move with no change in the system under test, so the judge gets pinned and versioned too.
Cost ownership has a mechanical prerequisite that org charts skip. A provider invoice is itemised per API key or project, not per feature. If five teams share one key, the platform team owns a bill it cannot divide, and "chargeback" becomes a spreadsheet allocation that nobody trusts or acts on. Attribution has to be stamped on at the one point every call passes through, the gateway. That means a feature and tenant tag on every request, token counts recorded from the response rather than estimated from the prompt, and untagged traffic rejected, not just logged. Once that exists, the useful budget control is a per-feature soft limit that alerts the owning team, plus a hard limit that degrades gracefully (a cheaper model or a shorter context), rather than a global cap that fails every feature at once when one runaway agent loop spends the month's money in an afternoon.
The ownership map also has to cover the incident path, and quality incidents do not fit an SRE runbook written for outages. The first action on a quality page should be the same as for a bad binary: roll back the last change to model input or identity, whether that was a prompt version, a retrieval config or a model pin. That is only possible if those changes are versioned and deployable independently of application code. Diagnosis comes after mitigation. The second trap is who can declare the incident. If only SRE can page and SRE has no quality signal, quality incidents are declared by customer support, days late. The online eval alert should page the product team that owns the eval, with SRE as a responder for the rollback mechanics.
Put it all together and LLM Ops is a maturity ladder. The first rung is "call an API and hope" — a single hardcoded model, no evals, no cost visibility, latency you notice only when a user complains. It ships fast and it is fragile. Each rung up adds a control from this chapter: an SLO and the observability to measure it; evals in CI so no prompt or model change ships without a quality check; canaries and shadow traffic so rollouts are reversible; cost budgets attributed per team so the bill is a dashboard, not a surprise; a gateway and router so failover and model choice are policy, not code changes.
Notice that the ladder is exactly the operating loop from 1.1, hardened one control at a time. A governed LLM platform is not a different thing from a hacked-together one — it is the same loop with instrumentation, gates, and clear ownership bolted onto every step. The goal of this track is to walk you up that ladder deliberately, so that by the end you can take a probabilistic, expensive, GPU-bound service and operate it like the dependable production system your users assume it already is.
Call an API and hope
climb when: prototype, internal users, no SLO
Gateway · pinned versions · token metrics
climb when: second team, second model, or first invoice someone asks about
Prompts as deploys · offline eval gate in CI
climb when: more than one person can change a prompt, or the first shipped regression
Online quality SLI · canary · shadow
climb when: volume where ~1k judged samples per window is a small slice of traffic
Budgets · chargeback · retry budgets
climb when: shared quota across teams; spend is a line item finance tracks
Multi-provider routing · self-hosted fleet
climb when: outage cost > cost of an eval'd second model; or utilization clears break-even
Skipping rungs is the common failure: rung-5 multi-provider failover without rung-2 eval gates turns every outage into an unmeasured model swap.
The ladder is worth reading as a decision tool, because the expensive mistakes are climbing the wrong rung first. Multi-provider failover (rung 5) is the control most often built early, because an outage is visible and embarrassing. Without the eval gate from rung 2, though, it is an untested model swap wired to trigger at the worst possible moment. Online quality SLIs (rung 3) are frequently attempted at volumes where the statistics from 1.1 cannot support them. Below roughly a thousand judged samples per alerting window, a daily offline eval over a fixed, versioned set gives a cleaner signal for less money, and the honest SLO says so. Self-hosting belongs at the top of the ladder, not the bottom. It adds two lanes of the stack to operate, and its economics depend on the utilization discipline that the lower rungs' metrics make visible.
Where the conventional advice is wrong
Questions to ask in a design review
Where next