Every latency problem you have debugged was either work taking too long or work waiting its turn, and in most systems under load it is the second. You do not need the derivations. You need a handful of short results, because they answer questions engineers usually settle by argument: how busy a service can safely run, why the p99 fell apart at 75% CPU, whether to buy bigger machines or more of them, and where a system will break before it does. This chapter covers the parts that change decisions and skips the rest.
L = λW. Everyone has seen it; fewer people use it as a measurement tool, which is what its lack of assumptions makes it. It does not care about arrival distribution, service distribution, scheduling discipline or the number of servers. The only requirement is that the system is stable over the window you measure, meaning what goes in comes out. That makes it the one queueing result you can apply to a real production system without apologising for the model, and it has three uses that pay off regularly.
Sizing concurrency. A service taking 2,000 rps with 50 ms mean latency has 100 requests in flight on average. If each holds a database connection for 8 ms of that, the pool is 16 connections busy on average. The mistake is sizing to the average. Occupancy fluctuates around the mean with a standard deviation on the order of its square root for Poisson-ish traffic, so a pool of 20 against a mean of 16 is waiting often. The same arithmetic sets thread pools, HTTP/2 stream limits, Kafka consumer concurrency (lag in messages ÷ ingest rate = lag in seconds), and in-flight limits for rate limiting. It also runs in reverse: if a vendor gives you 50 concurrent connections and their mean latency is 200 ms, your ceiling is 250 rps (use the mean, not the p50: Little's law is about averages, and a heavy tail drags the mean well above the median) no matter how many machines you add on your side.
Finding queues you did not instrument. This is the use that most teams miss. Little's law applies to any boundary you draw, so measure L and λ at two nested boundaries and the difference in W is time spent between them. The load balancer knows how many requests it has in flight to a pool; your service knows how many handlers are running. Divide both by throughput. If the LB implies 130 ms and your handler timer says 50 ms, 80 ms per request is being spent in places your service metrics cannot see: the kernel accept queue, the socket receive buffer, the executor's task queue before the timer starts, a sidecar. Services that start their latency timer in the handler are instrumented to be blind to exactly the queueing that grows under load.
load balancer
L = 260
W = 130 ms
measured at LB
kernel accept queue
L ≈ 60
W ≈ 30 ms
invisible · somaxconn
worker-pool queue
L ≈ 100
W ≈ 50 ms
invisible · executor queue
handler (timer on)
L = 100
W = 50 ms
what the service reports
Little's law holds for any stable boundary you draw, whatever the arrival or service distribution. Compute L / λ at two nested boundaries and the difference is the time spent between them — even where nothing is instrumented.
The accept queue deserves its own warning because its failure mode is not gradual. It is bounded by the listen() backlog, capped by net.core.somaxconn (128 on older Linux kernels, 4096 since 5.4). While it has room, it is an unmeasured queue that adds milliseconds. When it overflows, Linux by default (tcp_abort_on_overflow = 0) silently drops the handshake: either the SYN, so the client retransmits it after its initial retransmission timeout, typically one second, then doubling; or the final ACK, so the client believes it is connected and has sent its request into a socket the server has not accepted, and the request waits on the server's SYN-ACK retransmission instead. Latency does not rise smoothly; it grows a second mode at roughly +1 s and +3 s, which looks like a network problem and gets routed to the wrong team. The kernel counts it (ListenOverflows / ListenDrops in nstat), and almost nobody alerts on it. Any latency histogram with a bump just above one second is worth checking against that counter before anything else.
Checking whether the numbers are lying. If your reported concurrency, throughput and latency do not satisfy L = λW over a window where the system is stable, one of them is wrong. The usual culprits are latency averaged over a different population than throughput (successes only, or one endpoint), a timer that excludes queueing, or clients that time out and disappear, so the server keeps working on requests nobody is waiting for. The measurement problems themselves belong to chapter 2; Little's law is the cheap consistency check that tells you to go and look.
Architect's note
For a single server with random arrivals and exponential service (M/M/1), mean response time is R = S / (1 − ρ). At 50% utilisation requests take 2× their service time; at 80%, 5×; at 90%, 10×; at 95%, 20×. The intuition people bring from CPU dashboards, that a server at 80% is somewhat slower than one at 40%, is wrong in a specific way: latency is not proportional to load, it is proportional to 1 / idle fraction. Going from 80% to 90% busy halves the idle time and so doubles response time.
The tail is worse than the mean. In M/M/1 the response time is itself exponentially distributed with rate μ − λ, so the p99 is ln(100) ≈ 4.6 times the mean: 9× service time at 50% busy, 23× at 80%, 46× at 90%. A service with a 4 ms handler has a p99 around 46 ms at 60% busy and around 120 ms at 85%, without any change in code, and the flame graph will show nothing because nothing got slower. It all got longer in the queue.
At 80% busy the mean is 3–9× the service time depending on variability, and the M/M/1 p99 is 23×. The slope is the real problem: d(R)/dρ = S / (1 − ρ)², so the last 10 points of utilisation cost more latency than the first 80.
The shape matters more than the constants, and the shape is set by the derivative. dR/dρ = S / (1 − ρ)². At 50% utilisation, one extra point of load adds 4% of a service time to the mean; at 90%, it adds a full service time; at 95%, four. This is why capacity incidents feel sudden. A system that absorbed six months of 5% monthly growth without visible change crosses from the flat part of the curve to the steep part in a few weeks, and the team that owns it experiences it as a cliff even though the arithmetic was never hidden.
Two practical corrections to how utilisation is usually read. First, the averaging window lies. Queues form and drain on the timescale of the service time, which is milliseconds; CPU is reported per minute. A service at 55% averaged over 60 s may spend ten one-second intervals at 100% during a traffic burst or a cron-aligned batch, and its p99 is set by those seconds. If your metrics pipeline can record per-second or per-100 ms utilisation for the bottleneck resource, even sampled, that is the number to plot against latency. Second, the bottleneck is rarely the CPU you are graphing. It is whichever resource has the highest ρ weighted by how few servers it has, a point 1.7 makes concrete. In containers, CPU quota throttling adds another queue with its own period; chapter 5 covers why a pod at 40% of its limit can still be throttled. And the CPU percentage itself is not ρ: with SMT enabled, the OS counts two hardware threads per core, but the second thread adds only a fraction of a core's throughput, so a host reporting 50–60% can already have its physical cores near saturation. The queueing curve bends at the real capacity, not the one the gauge is scaled to; calibrate by finding the reported utilisation at which throughput stops rising in a load test, and treat that as ρ = 1.
The useful move is to run the formula backwards. Given a service time S and a p99 budget B for the queueing resource, the M/M/1 tail gives ρmax = 1 − 4.6·S / B. A 4 ms handler with a 50 ms p99 budget can run at 63%; the same handler with a 20 ms budget, at 8%, which in practice means it cannot be a single queue at all and needs the pooling in 1.4. This is the number that should replace the organisation-wide “scale out at 70% CPU” rule, which is too conservative for a 64-core pool of short requests and dangerously loose for a single-threaded path with multi-millisecond work. The target utilisation is a derived quantity: it falls out of S, the tail budget, the variability and the server count, and it differs by an order of magnitude across the resources in one request path.
5×
mean R at ρ = 0.8, M/M/1
23×
p99 R at ρ = 0.8
S/(1−ρ)²
marginal cost of load
4.6×
p99 / mean, M/M/1
M/M/1 assumes exponential service times, which have a coefficient of variation of exactly 1. Real systems are rarely that tidy, and the general result is where the useful engineering lives. For one server with Poisson arrivals and any service distribution, the Pollaczek–Khinchine formula gives mean queueing delay exactly: Wq = ρ/(1−ρ) · (1 + c²ₛ)/2 · E[S], where c²ₛ is the squared coefficient of variation of service time. Kingman's approximation extends this to non-Poisson arrivals by replacing (1 + c²ₛ) with (c²ₐ + c²ₛ). Either way, the structure is the lesson: utilisation term × variability term × service time. They multiply. Most teams manage only the first.
Consider an API where 99% of requests take 1 ms and 1% take 100 ms: an export endpoint, a cold cache path, a report query sharing the same worker pool. Mean service time is 1.99 ms, which looks fine. But E[S²] is dominated by the rare slow requests, and c²ₛ comes out at about 24.5. At 70% utilisation the mean wait is ~59 ms, 12.75 times what an exponential service with the same mean would produce. The 1 ms requests, which are 99% of traffic and probably the ones your SLO is about, spend 98% of their latency waiting behind jobs they have nothing to do with. The profile shows 1 ms. The dashboard shows 60.
constant service
c²ₛ = 0
exponential
c²ₛ = 1
log-normal-ish API mix
c²ₛ ≈ 4
99% at 1 ms · 1% at 100 ms
c²ₛ ≈ 24.5
Bottom row: a request needing 1 ms of work waits ~59 ms on average, because the 1% of 100 ms jobs set everybody's queue. Its own work is 2% of its latency.
Wq = ρ/(1−ρ) · (1 + c²ₛ)/2 · E[S]. Utilisation and variance multiply. Halving c²ₛ buys as much as a large drop in load — and is usually cheaper.
This gives you a family of fixes that do not require more hardware, and a way to rank them. Separating classes (a dedicated pool or queue for the slow endpoint) removes the heavy tail from the fast queue. c²ₛ in the fast pool collapses towards the variance of the fast requests alone, and the fast path's waiting time drops by an order of magnitude at the same utilisation. The price is lost pooling, covered in 1.4, and it is almost always worth paying when the classes differ by more than 10× in service time. Chunking long work (paginating the export, yielding in a batch loop) turns one long job into many short ones and moves an FCFS queue closer to processor sharing. Under processor sharing, mean response time is S/(1 − ρ) regardless of the service distribution, which makes it insensitive to variance. That is also the queueing argument for preemptive threads over a non-preemptive event loop on mixed workloads: an event loop is FCFS with no preemption, so one 200 ms synchronous handler stalls every request on that loop.
Arrival variability matters just as much, and your own architecture creates most of it. Clients retrying on a fixed interval, cron jobs firing on the minute, cache TTLs that all expire together after a deploy, a GC pause upstream that holds requests and then releases them in a burst, batching producers: each raises c²ₐ above 1. Because of the multiplication, a service that is fine at 70% with smooth arrivals can have twice the queueing at the same utilisation when a dependency starts sending bursts. Jitter on timers, TTLs and retry schedules is a variance-reduction technique, not a style preference.
Pitfall · optimising the mean service time
The second big lever is how many servers share a queue. With c servers drawing from one queue (M/M/c, the Erlang C model), the probability that an arriving request waits at all falls sharply as c grows at a fixed per-server utilisation. At ρ = 0.8, a single server makes 80% of arrivals wait, averaging 4 service times. Ten servers sharing a queue make 41% wait, and the mean wait is 0.21 service times, twenty times lower at the same utilisation. The practical form of this is the pooling table below: for a fixed tail promise (p99 queue wait no more than one service time), a lone worker must sit nearly idle, four workers can run at about 42%, sixteen at 79%, sixty-four at 94%. Large pools are more efficient per unit of latency, and the effect is large.
max ρ with p99 queue wait ≤ 1 service time
Same latency promise, same per-server speed: a single worker must idle at 3%; a pool of 64 can run at 94%.
same total capacity, ρ = 0.8
10 servers, 10 queues
random / round-robin, no work stealing
Wq 4.0 S · R 5.0 S
10 servers, 1 queue
M/M/10 · shared queue or JSQ
Wq 0.21 S · R 1.2 S
1 server, 10× faster
M/M/1 at the same ρ
Wq 0.4 S · R 0.5 S
The fast server wins on mean with exponential service — and loses badly once service times are heavy-tailed, because one long job blocks the only lane.
Four architecture consequences follow. First, how you distribute work across servers decides whether you get the pooling benefit. Ten instances behind random or round-robin balancing, each with its own local queue, behave like ten independent M/M/1 queues, not like M/M/10. Work cannot move from a busy instance to an idle one. That is the 4.0 S versus 0.21 S gap in the diagram. Least-outstanding-requests, join-the-shortest-queue and power-of-two-choices balancing recover most of the pooling benefit by approximating a shared queue. They are queueing-theory decisions presented as LB configuration, and on a fleet running near 70% they are often worth more than a 20% capacity increase.
Second, isolation has a measurable price. Splitting one pool of 64 into four bulkheads of 16 moves you from 94% to 79% usable utilisation at the same tail budget. That is 15 points, or roughly 19% more hardware for the same latency. Per-tenant pools, per-endpoint thread pools, cells and shards all pay this tax. Sometimes it is exactly right: 1.3 showed that separating classes with very different service times reduces variance by more than it loses in pooling, and the SRE track's Blast Radius chapter makes the case for containment. But the price should be a stated cost in the design, not a surprise on the capacity bill. The rule of thumb: split by service-time class, pool within a class.
Third, one fast server versus many slow ones. With exponential service, a single server ten times faster has a lower mean response time than ten slow servers at the same load. At ρ = 0.8 it is 0.5 S versus 1.2 S, because at low load a request on the fast server finishes in S/10, while on a slow server it still takes S. This is the queueing argument for vertical scaling and for a faster single-threaded path. It reverses as soon as service times are heavy-tailed. The single fast lane has head-of-line blocking: one long job holds up everything behind it, and there is no second lane to go around it. Many slower servers let short work flow past long work. For workloads with c²ₛ well above 1 (most real APIs), more parallel lanes win on the tail even when a single fast server wins on the mean. The same reasoning explains why a single-threaded datastore is excellent with microsecond commands and disastrous with one O(N) command in the mix.
Fourth, shared-queue approximations fail in a specific way. Least-outstanding-requests works because it routes on a live signal, and that signal is dangerous when it is wrong. An instance that fails fast (returns 503 in 1 ms because its database pool is broken, or a dependency is down) has the fewest requests outstanding, so the balancer sends it more traffic, and it fails that faster still. One bad host can absorb a large share of the fleet's requests and turn a 1-in-N hardware fault into a fleet-wide error rate. The same happens after a deploy when a fresh instance with cold caches reports zero in-flight work, and with many independent balancer instances that each see an idle host at the same moment and all pile onto it. The mitigations are queueing-aware: power-of-two-choices rather than global least-loaded (randomisation breaks the herd), outlier ejection keyed on error rate as well as latency, and slow-start ramps for new instances. Any balancing policy that rewards low latency must also treat errors as load, or it rewards failing quickly.
The asynchronous side of the system has the same maths and fewer guard rails. A key-partitioned log is the anti-pooled case taken to its limit: each partition is a lane with exactly one consumer, and ordering guarantees forbid work from ever moving between lanes. So the queueing behaviour of the topic is the behaviour of its hottest partition, and key skew plays the role that c²ₛ played in 1.3. In the illustrative topic below the group runs at 67% on paper while one partition is at 200% and its backlog grows without bound. Two second-order effects make this worse in practice. Consumer autoscalers usually key on summed lag, so they add consumers that cannot receive a partition, and each membership change triggers a rebalance, which is a stall of every lane: from 1.6, a lane at ρ = 0.8 needs four times the rebalance duration to drain. An autoscaler reacting to lag by adding consumers can therefore generate the lag it is reacting to. And the partition count is a pooling decision you make once, early, and pay for later: it fixes c for every consumer group on the topic, and the max-ρ-per-lane, not the total, is what you size it against.
ingest per partition vs consumer capacity
- - - consumer capacity 1,000 msg/s per partition
What the dashboard says
group capacity 12,000 · load 8,000 · ρ = 0.67 · CPU on 11 of 12 consumers idle-ish
What p0 is doing
ρ = 2.0 · backlog +1,000 msg/s · after 10 min: 600k messages, newest waits ~600 s
Why scaling out fails
a 13th consumer gets no partition; more partitions do not split one key; the hot key is a c = 1 resource
What works
cut per-message S on the hot path (batching), split the key's work (sub-keys + a reorder step), or relax ordering for that key
Hash partitioning by key is the per-instance-queue case from this section, with the added constraint that work can never move between lanes. Alert on per-partition message age (max over partitions), never on summed lag, which hides a single saturated lane behind eleven healthy ones.
Architect's note
Queueing models describe what happens at a fixed capacity. The Universal Scalability Law (Neil Gunther) describes how capacity changes as you add parallelism: threads, connections, nodes, concurrent clients. X(N) = N / (1 + σ(N−1) + κN(N−1)). The σ term is contention: the fraction of work that serialises on something shared. With κ = 0 this is Amdahl's law, and throughput approaches the ceiling 1/σ. The κ term is coherency: the cost of N parties keeping each other up to date. It grows with N², so any non-zero κ means there is a peak at N* = √((1−σ)/κ), after which adding parallelism reduces throughput. That is retrograde scaling, and it happens in production more often than capacity plans admit.
The coherency term has concrete sources, and naming them is most of the diagnostic value. Inside a process: cache lines bouncing between cores on a contended counter or lock (chapter 5 covers false sharing), spin-lock collapse, GC with many allocating threads. In a database: more connections than cores, where lock-manager and buffer-pool contention grows faster than useful work. The usual advice to cap Postgres connections near a small multiple of core count and put a pooler in front is USL advice, whether or not it is described that way. Across a cluster: cache invalidation broadcast to every node, gossip, all-to-all heartbeats, scatter-gather queries whose fan-out grows with cluster size, and consensus groups that get slower as members are added. The Distributed Systems Deep Dive covers the protocols. The USL gives you the shape of the cost they impose.
X(N) = N / (1 + σ(N−1) + κN(N−1)). σ is serialisation (a lock, a single leader, a shared log); κ is the cost of keeping N parties coherent (cache-line bouncing, cross-node invalidation, gossip). κ grows with N², so any non-zero κ guarantees a peak.
The practical value is that σ and κ can be fitted from a handful of load-test points, which turns a vague sense that the system does not scale into a prediction. Measure throughput at six to eight levels of N, normalise by X(1), and fit; a linearising transform makes it an ordinary quadratic regression, and a spreadsheet solver is enough. With the illustrative fit above (σ = 0.05, κ = 0.0005), the peak is at N ≈ 44, delivering 10.7× single-node throughput. Two numbers on that curve are more useful than the peak. At N = 16 you already have 8.6×, 80% of the achievable maximum. Going from 16 to 44 buys the last 25% at almost three times the resources. And at 128 you are down to 8.3×, paying for eight times the parallelism of N = 16 to get less.
Fits come with honest caveats. The parameters describe the workload you tested, so a different read/write mix or key distribution gives a different κ. Extrapolating far past the largest measured N is guesswork, and a fit to points far below the peak cannot distinguish a small κ from zero. Use it to rank designs and to find which term to attack. High σ means removing a serialisation point: a single writer, a global lock, a sequence generator. High κ means reducing crosstalk: partition so nodes stop talking to each other, batch invalidations, replace all-to-all with hierarchical. Adding nodes does not fix either.
Pitfall · the thread-pool reflex
Steady-state formulas describe a system that is stable. Incidents are about the transient: what happens after something stalls and the queue has to recover. The arithmetic is simple and not widely known. During a stall of length T, backlog grows at λ. Afterwards it drains at μ − λ, the spare capacity, not at μ. So the time to recover is T · ρ / (1 − ρ). At 60% utilisation a 30-second stall (a long GC pause, a leader failover, a dependency hanging until timeout) clears in 45 seconds. At 90% it takes 270 seconds, and every request that arrives in that window pays the backlog as extra latency. A team that sees a four-minute latency spike from a 30-second failover is usually looking at this effect, not a slow failover.
Backlog grows at λ during the stall and drains at only μ − λ afterwards, so recovery time = stall × ρ / (1 − ρ). At 90% busy a 30-second hiccup costs four and a half minutes of elevated latency. Add retries that push effective load past μ and the queue has no downward slope at all.
Now add the behaviour of real clients. When queueing delay exceeds the client timeout, three things happen together. Callers give up, but the server still does the work for requests nobody is waiting for, so goodput falls while throughput looks normal. Callers retry, so offered load becomes λ(1 + r) for a retry fraction r. And the retries join the back of the same queue, where they will also time out. The recovery condition is simply ρ(1 + r) < 1. At 70% utilisation, one retry for every timed-out request (r = 1 once most requests are timing out) gives an effective load of 1.4. The queue has no downward slope, and the system stays down after the trigger has gone. That is a metastable failure. The SRE track's How Large Systems Actually Fail chapter covers its incident history, and Resilience Patterns & Their Failure Modes covers retry budgets and breakers. The queueing view adds a sizing rule: your retry policy needs headroom of at least r·ρ, so a service run at 85% cannot afford a policy that retries even 18% of requests under stress.
The same analysis explains why queue discipline matters in overload even though it barely matters in steady state. FIFO under overload serves the oldest requests first, and those are the ones most likely to have already been abandoned. So the server spends its capacity on dead work while fresh requests age into the same fate. Serving newest-first during overload, and dropping requests that have waited longer than a target, means capacity goes to requests that can still succeed. Facebook described this combination (adaptive LIFO with a CoDel-style queue timeout) in “Fail at Scale” (ACM Queue, 2015). The general principle is to bound queues by time, not length, because a queue's length says nothing about whether its contents are still wanted, while its sojourn time does.
Finally, whether your load is open or closed changes the shape of the failure. In a closed system (a fixed pool of workers or users who wait for a response before sending the next request) queueing is self-limiting. As latency rises, arrivals slow down, and response time grows linearly with the number of clients past saturation. In an open system (independent users on the internet, upstream services with their own traffic) arrivals do not wait for you, and response time follows the hyperbola in 1.2. Most load-testing tools are closed by default, so they show gentle linear degradation exactly where production will hit a cliff. Chapter 2 deals with the measurement side of this, coordinated omission. The architectural point is to decide which model your real traffic follows before trusting any capacity number.
The stall has a common cousin that teams do not recognise as one: a surge against a reactive autoscaler. From the queue's point of view, the scale-out lag (metric window, evaluation period, provisioning, boot, runtime warm-up, passing health checks) is a period in which λ can exceed μ and nothing can change it. Backlog is the integral of the excess over that lag, and it drains at the new spare capacity afterwards. In the illustrative case below, a service targeting ρ = 0.7 meets a 70% surge within a minute and an autoscaler that adds capacity three minutes later. It accumulates about 26 seconds of queued work, which in practice means minutes of timeouts, while every autoscaling dashboard shows the policy working as designed.
The autoscaler is not wrong; it is late. For the 144 s that load sits above capacity (shaded), every second adds 0.2 s of work to the queue, and the new capacity then drains it at only 1.5 − 1.2 = 0.3. With a typical client timeout of a few seconds, nearly every request in that window fails, and the retries from 1.6 are waiting to make it worse.
The design rule falls out directly. Let g be the largest fractional growth in offered load you must absorb within one scale-out lag. The utilisation target has to satisfy ρtarget ≤ 1 / (1 + g) just to keep λ below μ, before any tail budget from 1.2 is spent: absorbing a 40% jump needs ρ ≤ 0.71, a 2× jump needs ρ ≤ 0.5. That makes the target utilisation a function of the traffic's shape and the platform's lag, not a house number, and it shows which lever is cheaper. Halving the lag (pre-pulled images, warm pools, snapshot restore, smaller runtime warm-up) often buys more than ten points of standing headroom. Three second-order effects bite during the scale-out itself. New instances run with cold caches and unwarmed runtimes, so their S is temporarily higher and their effective capacity lower than the plan assumed. Their arrival triggers a burst of connection establishment and cache fills against shared dependencies, raising c²ₐ downstream exactly when those dependencies are also hot. And a policy keyed on CPU or served request rate saturates at 100% of μ: it cannot see how much demand is queued or being rejected behind it, so a step policy under-sizes the correction and needs several lag cycles to converge. Scale on a signal that queues cannot hide, such as in-flight concurrency or queue sojourn time, and make scale-in slower than scale-out.
T·ρ/(1−ρ)
drain time after stall T
270 s
30 s stall at ρ = 0.9
ρ(1+r) < 1
retry recovery condition
time, not length
how to bound a queue
All of the above becomes useful when it fits on a page and can be updated in an afternoon. The tool is operational analysis (Denning and Buzen), which uses only measurable quantities and needs no distribution assumptions. For each resource a request touches, write down its service demand D: total busy time on that resource per request, including every visit. The utilisation law U = X · D gives each resource's utilisation at any throughput X. The bottleneck law says X can never exceed c/D for any resource, so the resource with the smallest c/D sets the ceiling. Everything else in the model is detail.
The step most capacity models skip is to list the single-server resources alongside the machines. A row lock, a partition leader, a single-threaded cache shard, a sequence generator, a Kafka partition with one consumer, a leader-only metadata service, a vendor's per-account concurrency cap: each is a resource with c = 1 or close to it. From 1.4, a resource with c = 1 has almost no usable utilisation for a tail target. Its demand D is rarely on a dashboard because it is not a machine. In the illustrative place-order model below, the app tier and database CPUs look like the constraint because they are what gets graphed. The actual ceiling is a hot-SKU inventory row lock held for 3 ms by 10% of orders. It is already at 60% utilisation with a 31 ms p99 wait at today's load, 83 ms at +40%, and 342 ms at +60%, while the app tier at +60% is still comfortable at 80% on 48 cores.
Utilisation law
U = X · D. Every row is one multiplication; no load test needed to fill it in.
Bottleneck law
X ≤ min(c/D). The lock caps orders at 3,333 rps — below the CPU everyone watches.
Where it falls over
Not at the highest ρ but where c is small and S is large: a single lane with multi-ms holds.
The Redis row shows the other half of the rule. A single-threaded shard at 50% busy sounds alarming, but its service time is 50 microseconds, so even the M/M/1 p99 wait per command is under half a millisecond, and about a millisecond at +60%. What falls over first is small c combined with large S. A useful ranking across the table is the queueing term evaluated for each resource at the projected load, in absolute milliseconds: ρ/(1−ρ) × (1 + c²ₛ)/2 × S, with the Erlang-C correction for c > 1. The row with the largest number is where the next incident will start.
A model that earns its page has five columns filled from production data and a few decisions attached. Demands come from per-request resource accounting: CPU time per request from the profiler (chapter 4), lock hold times from database statistics, per-command counts from client instrumentation. Server counts include logical servers (locks, leaders, partitions), not just cores. Variability is measured c²ₛ per resource, which you can compute from the same histograms you already keep. Projected load is tied to a business driver, which chapter 7 builds out. Finally, a validation point: one real peak where the model's predicted utilisations and latencies were checked against what happened. Without that last column the model is a hypothesis. Its value is that it is cheap to be wrong on paper, and each disagreement with production points to a queue or a demand you had not measured.
The decisions it supports are the ones this chapter has been building towards. Run large pooled resources hot and keep single-lane resources cold. Buy variance reduction (class separation, chunking, jitter) before buying hardware when c²ₛ is high. Keep retry headroom proportional to utilisation. Stop adding parallelism at the fitted N*. Every one of these can be checked against a number rather than argued from experience, which also makes them easier to defend in a design review.
Read top to bottom: the cheaper levers (variance, balancing, lag) usually come before the expensive one (standing headroom), and the most dangerous resources are the ones with the fewest servers, not the highest CPU.
The model also fails in recognisable ways, and a reviewer should know them. It is usually built from averages over a quiet hour, so D is understated for the peak mix, where expensive endpoints are over-represented. It counts each resource once per request when retries, N+1 query patterns and cache misses make visit counts load-dependent: D itself rises with load, which is how a model that predicted, say, 3,300 rps meets its knee at 2,600 (illustrative numbers). And it treats resources as independent when they are coupled; a connection held across a slow downstream call turns that call's latency into demand on the pool, so the pool's effective S is the downstream's response time, queueing included. That coupling is how a queue one hop away becomes your bottleneck without any change to your own code.
Questions to ask in a design review
U = X·D
utilisation law
X ≤ c/D
bottleneck law
c = 1, big S
what breaks first
1 real peak
to validate the model
Where next