Messaging decisions are usually made on familiarity, a feature matrix, or throughput numbers nobody will reach. None of those predicts the failure. What predicts it is a mismatch between the shape of the stream — how many readers it will have, whether anyone needs it after it is processed, what ordering it really requires — and where the broker keeps its state. The mismatch never shows at launch. It shows 12 to 24 months later, as a second consumer, a replay request, a hot key or a backlog, and by then the broker's semantics have leaked into every service that touches it.
A checklist comparison is now useless, because every product ticks every box. SQS has FIFO queues with deduplication. RabbitMQ has had streams — offset-addressed, replayable — since 3.9. Kafka is growing queue semantics through share groups (KIP-932). Pub/Sub has ordering keys, seek-to-timestamp and an exactly-once delivery mode.
The useful observation is that each of those features lives at the edge of its product's design and inherits the constraints of the centre. SQS FIFO gives you ordering, but ordering in a queue means serialising a message group — so your parallelism is now your group cardinality, and dedup is a five-minute producer-side window rather than a property of the consumer. RabbitMQ streams give you replay, but not the ecosystem that makes replay useful: connectors, schema registries, stream processors, CDC tooling. Pub/Sub seek gives you history, bounded by topic retention you pay for per byte. The feature works; it just works the way a queue, or a log, or a managed subscription works.
So the first question is not which product but which model: does the broker track state per message (queue), does the consumer track a position in an immutable sequence (log), or does a managed service track per-message state per subscription (managed fan-out)? Almost every downstream property — cost of a new consumer, replay, blast radius of a poison message, what a backlog does to the broker — follows mechanically from that one answer.
Queue
SQS · RabbitMQ quorum/classic
Log
Kafka · Kinesis · RabbitMQ streams
Managed subscription
Pub/Sub · SNS→SQS · EventBridge
broker: in-flight, receive count, per message
consumer: one integer per partition
broker: ack state per message × subscription
producer or exchange must copy; history before binding is gone
zero: new group starts at any retained offset
new subscription; sees history only via seek within topic retention
none once acked
any offset in retention; compaction for bootstrap
seek to time/snapshot, bounded by retention
isolated: redrive to DLQ after N receives
blocks its whole partition until skipped
isolated; blocks one ordering key if keys are on
message (add workers freely)
partition (can only grow, and growing remaps keys)
message; ordering key serialises
SQS: fine until retention (default 4 days, max 14) silently deletes it · RabbitMQ: resource alarm, publishers blocked
just lag — until it outlives topic retention
fine until subscription retention expires it
Every row follows from the first. Feature checklists hide this: each product now has a checkbox for the other columns, but those features are built at the edge of its design and inherit the constraints of its centre.
Architect's note · the centre-of-design rule
If you can answer only one question, answer this one: how many independent consumers will this exact fact have in two years? Not today — today it is always one. The discriminator is whether the message is a command or a fact. "Render invoice 42 as PDF" has one owner for life; nobody else will ever want it. "Order 42 was paid" starts with billing and within a year has email, fraud, search, the warehouse loader and a loyalty service. Commands stay put; facts accumulate readers. Classify every stream before you pick anything — most organisations genuinely need both a queue and a log, and the costly mistake is forcing both kinds of stream onto whichever broker already exists.
The second question is history, and "we might need replay" hides three different requirements with three different horizons. Reprocessing — a consumer had a bug for two days, fix it and re-run — needs days of retention, which a log gives you cheaply. Bootstrapping — a new service needs the current state of every customer — needs all of history or a snapshot plus the tail; seven days of retention does nothing for it. That requires keyed, compacted topics or a snapshot-and-tail mechanism (the CDC and outbox patterns covered in the Data Engineering track). Audit and analytics — what did we know on 3 March two years ago — belongs in object storage or a lakehouse, not in a broker; tiered storage blurs this, but the broker should not become the system of record for history by accident.
The x-axis is a forecast, not today's count. Commands ("render this PDF") stay on the left for life; domain facts ("order paid") drift right as teams discover them. Most companies have streams in all four quadrants — the mistake is forcing them onto one broker.
Two things make the log's replay value smaller than it looks on a whiteboard. First, replay is a property of consumers, not brokers. A consumer that sends emails, charges cards or calls a partner API cannot be rewound without a replay guard — a stored high-water mark past which side effects are suppressed. If most of your consumers are not replay-safe, you are paying for a capability you cannot use. Second, replay is only as good as the event payload. Thin notification events ("customer 7 changed") force every consumer to call back for current state, so replaying last Tuesday's events fetches today's data — the replay is meaningless. If you want the log's history to be worth anything, events must carry state (event-carried state transfer), which in turn makes schema evolution a first-order concern from day one.
Pitfall · the SNS fan-out that ends as a log
Ordering is the requirement most often over-bought. When a team says "events for an account must be processed in order," ask what goes wrong if they are not. Usually the answer is "an older update overwrites a newer one." That is not an ordering requirement; it is a staleness requirement, and it is solved at the sink with a version check — apply the update only if its version is greater than the stored one. The version must come from the source of truth — a row version, an LSN, a per-aggregate sequence assigned in the producing transaction — never a producer wall-clock timestamp, which turns clock skew between two app servers into silently dropped updates. Once the sink rejects stale writes, the transport can be unordered, and you get unbounded parallelism and per-message failure isolation for free.
True ordering requirements exist where operations do not commute and intermediate states have consequences: ledger postings where a debit's acceptance depends on the balance produced by the previous credit, state machines whose transitions trigger side effects, order-book matching. For these, pay for ordering — but understand what you are paying. Ordering means no concurrency within a key, so the per-key throughput ceiling is simply one over the handler's latency. A 40 ms handler (a database write plus one downstream call) caps any single key at 25 messages per second, regardless of how many consumers or partitions you add. Key by tenant, and if one tenant produces 30% of traffic, 25 messages per second is the ceiling for 30% of the business — which is why the ordering key should be the narrowest entity whose operations actually fail to commute (the account, not the customer; the order, not the tenant).
200/s
per-key ceiling · 5 ms handler
25/s
per-key ceiling · 40 ms
1/12
keys stalled · 12 partitions
≈0
stalled · unordered + DLQ
The second cost is subtler, and it is where Kafka surprises people. Kafka does not order by key; it orders by partition, and many keys share a partition. A poison message or a slow downstream call for one key blocks every key hashed to the same partition. With 12 partitions, one bad order stalls a twelfth of all orders. SQS FIFO and Pub/Sub ordering keys scope the stall to the message group or key — narrower blast radius, lower ceiling on aggregate throughput. RabbitMQ gets ordering by pinning a queue to a single active consumer, which makes the blast radius the whole queue.
Partition count is therefore a blast-radius decision made on day one, not a throughput knob you tune later. Adding partitions changes the key-to-partition mapping for existing keys, so for a window after the change a key's old events sit in one partition and its new events in another, and two consumers can process them concurrently — a real ordering violation for exactly the streams that asked for ordering. The workable options are to over-provision partitions up front (the per-partition cost is metadata, file handles, smaller producer batches and slower leader failover when a broker dies — real, but it grows gently until partitions per cluster reach the thousands) or to migrate to a new topic with a controlled cut-over.
The number that sizes partitions is almost never broker throughput; it is Little's law on the consumer. A classic Kafka consumer processes each partition serially, so the number of partitions is the number of messages that can be in flight at once, and required slots equal arrival rate times handler latency. Take an illustrative stream at 3,000 messages per second with the same 40 ms handler: 3,000 × 0.04 = 120 slots at steady state. The team that picked 48 partitions "for headroom" — because each broker can take far more than 3,000 small messages per second — has a hard ceiling of 48 × 25 = 1,200 per second and is in permanent lag on day one. The two cases that matter are worse: when a dependency's p99 goes from 40 ms to 120 ms, required slots triple; when you need to drain a backlog at twice the arrival rate, they double again. Size for degraded latency × drain rate, or move the concurrency into the consumer (key-level parallelism within a partition), which decouples slots from partitions but makes offset commits track the lowest incomplete record per partition.
Broker throughput is irrelevant here; the ceiling is the consumer's serial slots. Either partition for the degraded-drain case, or add key-level concurrency inside the consumer so slots scale with keys in flight rather than with partitions.
Kafka orders by partition, not by key: you asked for per-order ordering and received per-1/N-of-all-orders ordering. Key-level concurrency inside a partition (e.g. Confluent's Parallel Consumer) narrows the radius back to the key, at the cost of more complex offset tracking.
Architect's note · where the line is
Exactly-once is a scoped claim; the useful knowledge is the mechanism by which each broker produces a duplicate, because it decides when duplicates arrive — and they do not arrive uniformly. Every broker has a redelivery timer: SQS's visibility timeout, Kafka's max.poll.interval.ms, Pub/Sub's ack deadline, RabbitMQ's connection and consumer timeouts. If the handler outlives the timer, the message is handed to someone else while the first handler is still running.
That makes the timer an undeclared SLO on the handler's tail latency, and it creates a positive feedback loop under stress. A downstream dependency slows; handler p99 crosses the timer; messages are redelivered; now two workers hit the slow dependency for the same unit of work; it slows further. In Kafka the same shape appears as a rebalance storm: a consumer exceeds the poll interval, is evicted, its partitions move, the new owner reprocesses the whole uncommitted batch against the same slow dependency and is itself evicted. The broker converts latency into load, precisely during the incident.
Redelivery trigger
What happens when the handler is slow
Built-in dedup & its boundary
SQS
visibility timeout (default 30 s)
message reappears; a 2nd worker runs it concurrently with the 1st
FIFO dedup: 5-min window, producer side only
Kafka
max.poll.interval.ms (default 5 min)
member evicted → rebalance → whole uncommitted batch re-run elsewhere
idempotent producer + txns: Kafka→Kafka only
RabbitMQ
channel/connection loss; consumer_timeout (default 30 min) closes the channel
every unacked message in the prefetch window requeued, redelivered=true
none in core
Pub/Sub
ack deadline (10 s default, ≤ 600 s)
redelivered, often to another subscriber client
exactly-once delivery: acks, not side effects
The redelivery timer is an undeclared SLO on your handler. Breach it during an incident and the broker converts latency into load — a positive feedback loop that starts exactly when you can least afford it.
The design consequence is concrete. Every outbound call inside a handler must have a timeout, and the sum of those timeouts plus local work must be shorter than the redelivery timer, so the handler fails and releases the message rather than being duplicated. Kafka hides a multiplier here: the timer bounds the whole poll batch, not one record. With the default max.poll.records of 500 and a 5-minute poll interval, the budget is 600 ms per record at p99 of the batch — a handler that normally takes 50 ms but waits 2 s on a degraded dependency blows it with a full batch, and the consumer is evicted mid-batch. Size max.poll.records from the degraded latency, not the healthy one. Long jobs must extend their lease explicitly (SQS ChangeMessageVisibility, Pub/Sub lease management) or not live on a broker at all. And the only exactly-once guarantee that survives contact with a real side effect is effect-level idempotency at the sink — a unique constraint on the event ID in the same transaction as the effect, or an idempotency key passed to the external API. Kafka's transactions cover Kafka-to-Kafka; Pub/Sub's exactly-once delivery covers acknowledgements; SQS FIFO's dedup covers producer retries for five minutes. None of them covers your database or a payment provider.
Pitfall · the loss is usually on the producer side
The operational tax is where Kafka decisions go wrong most often, in both directions. Self-run Kafka is a product you operate, and the parts that hurt are not the ones on the runbook: quotas so one team's backfill does not starve another's real-time path, and a schema registry whose compatibility rules are a governance process in disguise. A useful test: if you cannot name two people who could safely reassign partitions under pressure at 3 a.m., you are not running Kafka, you are hosting it until the first bad day.
The costs that surprise are second-order. On AWS, inter-AZ traffic is billed; a three-AZ cluster with replication factor 3 ships every produced byte across AZ boundaries twice for replication, plus whatever producer and consumer traffic crosses zones. At an illustrative 100 MB/s sustained ingest and roughly $0.02/GB for a round inter-AZ crossing, replication alone is on the order of $10k per month before a single consumer reads anything. Add producers: with leaders spread evenly across three zones, about two-thirds of produced bytes also cross a zone to reach their leader. Rack-aware follower fetching (KIP-392) removes most consumer-side crossings; on self-run Kafka nothing removes the replication term except fewer zones. That term is also where the self-run-versus-managed comparison quietly flips: MSK does not bill broker-to-broker replication traffic inside the cluster, so the line item that dominates a self-run bill at high ingest is absent from the managed one. RabbitMQ's surprise is different: it is designed for queues that are usually empty. Recent versions page classic and quorum queue bodies to disk far more aggressively than older ones, but a backlog of tens of millions still degrades throughput, and once the broker crosses its memory or disk watermark it blocks every publishing connection, not just the ones feeding the deep queue — backpressure propagates synchronously into whichever API is publishing, and an outage of a background worker becomes an outage of the user-facing path.
Circles mark the crossovers: ≈500/s, unbatched SQS overtakes managed Kafka; ≈7.5k/s, batched SQS does; ≈70k/s, batched SQS overtakes even self-run Kafka with its people cost. Assumptions (illustrative, not a quote): SQS standard at $0.40 per million requests with send + receive + delete per message; managed Kafka as a small provisioned cluster growing with inter-AZ replication traffic; self-run Kafka carrying half an engineer's loaded cost. Move the people term and every crossover moves with it.
The curves carry the point: below a few thousand messages per second, managed queues are cheaper than any Kafka, and the people term keeps self-run Kafka more expensive than managed until volume is large. Throughput is rarely the reason to pick Kafka — any of these handle 10,000 small messages per second — and cost is almost never the reason at modest scale. The reason is capability: many independent readers, replay, bootstrap, per-key order at volume. If none of those apply, Kafka is an expensive queue with worse per-message semantics.
The capacity question nobody asks at selection time is how long recovery takes. A broker's job during an incident is to hold the backlog; the consumer fleet's job afterwards is to drain it, and drain time is λT / (μ − λ): backlog divided by headroom, not by capacity. At an illustrative 2,000 messages per second, a four-hour downstream outage leaves 28.8 million messages queued. A fleet with 25% headroom — 2,500 per second, which looks comfortable on a steady-state dashboard — needs 16 more hours to catch up, so a four-hour outage becomes twenty hours of stale data. Three constraints then decide whether you can buy that time back. Retention: SQS's four-day default survives a Friday-evening outage plus a slow drain only if someone noticed on Saturday; a Kafka topic with 24-hour retention does not survive the example at all, because the oldest records expire while you are still draining. Parallelism: an unordered queue drains faster the moment you add workers; an ordered log drains no faster than its partition count allows, which is the case for sizing partitions from the previous section's drain row. And the downstream: the system that just recovered is now asked to take twice its normal load, so the drain rate is often capped by the thing that failed, not by the consumer.
Recovery time = λT / (μ − λ). A consumer fleet with 25% headroom — which looks generous on a steady-state dashboard — takes 16 hours to recover from 4 hours of downtime, and every message in that window is served stale. Doubling capacity cuts it to 4 hours; it does not help if the ordered stream has only enough partitions for 1.25×.
Managed is not the same as zero blast radius. On 25 November 2020 an operational event in Kinesis Data Streams in us-east-1 — triggered by adding capacity to its front-end fleet, which pushed servers past an operating-system thread limit — took down or degraded a string of AWS services that depended on it internally, including CloudWatch and Cognito. The lesson is not "self-host"; it is that your messaging layer's dependencies are part of your failure domain, and you should know which of your other managed services share it.
Architect's note · team-size thresholds (a heuristic, not a law)
Wrong messaging choices rarely produce a dramatic outage; they produce compensating machinery, each piece reasonable in isolation. The three composite timelines below are the common shapes.
None of these is visible at month 0, and none is a single bad day — so nobody can point at the moment to revisit the decision.
A queue where a log was needed decays into an ad hoc log built from fan-out topics, archive copies and backfill scripts. The terminal symptom is an ordering or consistency bug in a derived view — a projection that shows a refund before its payment because the backfill and the live feed interleaved — that nobody can fix locally, because there is no single ordered source to replay from.
A log where a queue was needed decays the other way. Job-shaped work (variable duration, per-item retries, delays, priorities) fights the partition model at every step. Long jobs exceed the poll interval and trigger rebalances; per-message retry with backoff has to be rebuilt as a tier of retry topics, which destroys per-key ordering anyway; one poison job stalls unrelated tenants. The terminal symptom is organisational: a platform burden carried by a team that chose Kafka because it was the standard, for a workload that SQS would have handled with no one on call.
A broker used as a buffer fails the most abruptly. A queue sized for steady-state emptiness absorbs a downstream outage until it cannot, and the backpressure lands on the producer — typically a synchronous API. The fix is rarely a bigger broker; it is deciding explicitly where backlog is allowed to accumulate (a log, object storage, or a shed decision at the edge) and making that part of the capacity plan rather than an emergent property.
Because none of these shows up as an incident, detect them with structural signals reviewed quarterly rather than with alerts. A queue-shaped stream with a third subscription, or any backfill script that reads from a database to "catch up" a consumer, means a log is being assembled by hand. A Kafka topic family with -retry and -dlq suffixes, or a consumer group whose rebalance rate tracks downstream latency, means job-shaped work is on a log. A broker queue whose depth has a non-zero p99 in normal operation means it has become a buffer nobody sized.
The exits are asymmetric, and knowing that changes how hard you argue at the start. Log-to-queue is cheap: bridge the topic into SQS with a connector, move the job workers, delete the retry topics — a one-team change. Queue-to-log is expensive, because the history you now need was never kept. The least painful path is to put the log behind the source of truth rather than in front of the queues: an outbox or CDC feed publishes to the log, a snapshot of the table seeds compacted topics, and existing queues are re-fed from the log so consumers migrate one at a time. Budget it as a multi-quarter programme, because it touches every consumer of the fact.
Pitfall · the event backbone as a database
The decision tree below asks the questions in the order that actually discriminates. Fan-out and history first, because they separate queues from logs, which is the expensive distinction. Then routing and deployment constraints, which separate queue products from each other — a much cheaper distinction, because a work queue has a small API surface and can be swapped in a sprint. Ownership last, because it chooses between managed and self-run versions of a model you have already settled.
Context moves the answer in predictable directions. Stage: a seed-stage company with one Postgres database should use a jobs table with SELECT … FOR UPDATE SKIP LOCKED for work and a transactional outbox for facts until either volume or a third consumer argues otherwise — Chapter 07, When Postgres Is Enough, makes that argument in full. Regulation: an erasure obligation makes retention a liability rather than a feature; either keep personal data out of events (reference IDs, not payloads), use short retention on topics that must carry it, or plan crypto-shredding with per-subject keys before the first event is written, not after the first request arrives. Data residency may rule out globally-routed managed services or force per-region topics, which changes the fan-out topology. Organisation: the more teams consume a stream, the more the schema and the retention policy become a contract, and the more the broker becomes an organisational interface rather than a technical component.
Commands / jobs
Facts · 1–2 consumers
Facts · 3+ consumers or bootstrap
jobs table, SKIP LOCKED
outbox table, polled
outbox → managed log only when the 3rd consumer is real
SQS / Pub/Sub
SNS→SQS or Pub/Sub, DLQ per sub
managed Kafka / Kinesis + schema registry
cloud queue; not the platform's problem
managed fan-out
Kafka as an internal product with quotas and SLOs
queue; short retention is a feature
reference IDs, not PII payloads
per-region topics; crypto-shredding designed before first event
Read down a column, not across a row: the command column barely changes with scale, which is why deliberation belongs in the right-hand column.
A worked composite shows how the framework plays out. A 60-engineer marketplace on one cloud (composite, illustrative numbers) proposes "move everything to Kafka" after a quarter of SQS pain. The inventory finds eleven streams. Seven are commands — image resizing, email sends, webhook deliveries, search reindex jobs — each with one consumer, a peak under 300 messages per second, and handler latencies from 200 ms to 90 s. Four are facts: OrderPlaced, OrderPaid, ListingChanged and UserProfileChanged, with three to seven consumers each and a new analytics team asking for twelve months of history. The pain turns out to live entirely in the facts: two hand-built backfills and an SNS fan-out whose newest subscriber missed six weeks of events. The review outcome is a split, not a migration. The seven command queues stay on SQS; moving them would put 90-second jobs onto a partition model and buy nothing. The four facts move to a managed log fed by a transactional outbox, keyed by order and listing ID, with state-carrying payloads and a schema registry. History older than the topic's retention goes to object storage via a sink connector rather than living in the broker. At an illustrative combined 1,500 events per second, the managed cluster is a four-figure monthly line; the self-run alternative would have cost more in people than in infrastructure. The expensive part is the consumer migration, run one team at a time across two quarters.
Where the conventional advice is wrong
"Pick one broker and standardise." Standardise on one of each model. A single broker for commands and facts guarantees one of them runs on the wrong semantics, and the operational saving of one fewer system is smaller than the cost of rebuilding queue semantics on a log or log semantics on a queue.
"Kafka is for high throughput." Throughput is the weakest reason; every option here clears a few thousand messages per second. Kafka is for multiple independent readers, replay and bootstrap. Below roughly 1,000 messages per second with those needs, a managed log is still right; above 50,000 without them, a batched queue still is.
"More partitions cost throughput, so start small." Partition count is a consumer-concurrency and blast-radius decision fixed on day one; under-partitioning is the expensive error because growing it breaks per-key order, while over-partitioning costs metadata and slightly longer failovers.
Finally, apply the reversibility lens from Chapter 01. For commands, the broker is a two-way door: the producer and one consumer are usually owned by the same team, the API surface is small, and swapping SQS for RabbitMQ is a contained migration. For facts, it is a one-way door: event schemas, ordering assumptions, offsets and retention semantics leak into every consuming team, and migrating a backbone means coordinating all of them. Spend your deliberation there, and spend almost none on the command queues.
2
stream kinds · commands vs facts
3
replay horizons · reprocess, bootstrap, audit
1/latency
per-key ceiling when ordered
timer > p99.9
handler budget vs redelivery
Questions to ask in a design review
1. Is this stream a command or a fact, and how many independent consumers do you expect in two years — not today?
2. Which replay horizon do you need — reprocess, bootstrap or audit — and where does the history for the last two live if not in the broker?
3. What breaks if two events for the same key arrive out of order? Could a version check at the sink replace transport ordering?
4. What is the ordering key, what share of traffic does the largest key carry, and what is its per-key ceiling at degraded handler latency?
5. How were partitions sized — from arrival rate × degraded latency × drain factor, or from broker throughput?
6. What is the redelivery timer, the handler's p99.9, and the sum of its outbound timeouts? Which is larger?
7. Where is the effect-level idempotency key enforced, and how is the event published — outbox, CDC, or a dual write?
8. For a four-hour downstream outage, what is the backlog, the drain time at current headroom, and does retention outlast both?
9. What does a poison message stall, and who can skip it at 3 a.m. without a deploy?
10. Who is on call for the broker, and does any personal data in the payload outlive an erasure request?
Where next