Choosing Between Kafka, RabbitMQ, and NATS: A Decision Framework

Message brokers are one of those decisions that's easy to overthink. Teams spend weeks evaluating and still pick based on which one they've used before. Here's a practical framework that cuts through the noise.

The Core Difference

These three systems solve different problems despite overlapping in marketing:

• Kafka is a distributed log. Messages are persisted, ordered within partitions, and replayable. It's infrastructure for event streaming.
• RabbitMQ is a message broker. Messages are routed, queued, and delivered. It's infrastructure for task distribution and request-reply.
• NATS is a messaging system. Messages are delivered with minimal overhead. It's infrastructure for lightweight pub/sub and request-reply.

Feature Comparison

Throughput and Latency

Real-world numbers from a 3-node cluster, 1KB messages, replication factor 2:

Kafka wins on throughput by a wide margin because it batches writes and does sequential I/O. RabbitMQ and NATS win on latency because they have simpler protocols and less batching overhead.

Important caveat: RabbitMQ's throughput varies dramatically with configuration. Durable queues with publisher confirms: ~15K msg/s. Transient queues with no confirms: ~80K msg/s. Kafka's throughput is more consistent because durability is always on.

Operational Complexity

Kafka

Dependencies: ZooKeeper (legacy) or KRaft (modern). Minimum 3 brokers for production.

Operational concerns: partition rebalancing, leader elections, consumer group management, log compaction, disk sizing. Kafka requires dedicated operations knowledge.

RabbitMQ

Dependencies: Erlang runtime. Minimum 3 nodes for quorum queues.

Operational concerns: cluster partition handling (split-brain), memory alarms, queue mirroring (classic) vs quorum queues (modern), Erlang VM tuning. Simpler than Kafka but Erlang-specific issues can be cryptic.

NATS

Dependencies: None. Single binary.

Operational concerns: JetStream storage sizing, cluster route configuration. NATS is operationally the simplest — a single Go binary with no external dependencies.

Use Case Mapping

When to Use Kafka

• Event sourcing and event streaming. Kafka's immutable, replayable log is purpose-built for this. You can rebuild state by replaying events from any point.
• High-throughput data pipelines. Ingesting clickstream data, IoT telemetry, or log aggregation at 100K+ events/sec.
• Multiple consumers per event. Consumer groups allow different services to independently process the same event stream.
• Long-term event retention. Keep events for days, weeks, or indefinitely for audit, replay, or analytics.

When to Use RabbitMQ

• Task queues / work distribution. Distributing jobs across workers with acknowledgment, retry, and dead-lettering.
• Complex routing. RabbitMQ's exchange types (direct, topic, fanout, headers) enable sophisticated message routing without consumer-side filtering.
• Request-reply patterns. Built-in support for correlation IDs and reply-to queues.
• Priority queues. RabbitMQ supports message priorities natively.

When to Use NATS

• Microservice communication. Lightweight request-reply and pub/sub between services. NATS adds minimal latency and operational overhead.
• IoT and edge computing. Small binary, low resource usage, works well in constrained environments.
• Replacing HTTP for inter-service calls. NATS request-reply is often faster than HTTP with connection pooling overhead.
• Real-time notifications. Fire-and-forget pub/sub for events that don't need durability.

Decision Matrix

Score each criterion 1-5 for your specific use case, then tally:

Migration Paths

If you start with one and need to switch:

RabbitMQ → Kafka: Common migration. Usually driven by needing event replay or higher throughput. Bridge with a consumer that reads from RabbitMQ and produces to Kafka. Gradually move producers to Kafka directly.

Kafka → NATS JetStream: Less common but growing. Motivated by operational simplicity. NATS JetStream covers many Kafka use cases with less infrastructure. Use NATS's Kafka bridge connector for gradual migration.

NATS → Kafka: Common when outgrowing NATS's throughput or needing Kafka Connect's connector ecosystem for data integration.

The Pragmatic Choice

If you're starting fresh and unsure:

1. Default to NATS with JetStream if you need lightweight messaging between services and your throughput is under 200K msg/s. Simplest to operate, lowest latency.
2. Choose Kafka if you need event streaming, replay, or your throughput exceeds 200K msg/s. Accept the operational complexity.
3. Choose RabbitMQ if your primary pattern is task distribution with complex routing, retries, and dead-lettering. It's the best job queue.

Don't combine multiple brokers unless you have distinct use cases that genuinely require different systems. Running Kafka for event streaming AND RabbitMQ for task queues is valid. Running Kafka for everything when you only need a task queue is wasteful.