Kafka vs MQTT

Apache Kafka and MQTT solve different parts of the messaging pipeline. MQTT is a lightweight protocol for device-to-cloud communication. Kafka is a streaming platform for server-side data processing. They are complementary, not competing.

Protocol Design

MQTT was designed for constrained environments — low bandwidth, unreliable networks, devices with limited CPU and memory. Its minimum packet size is 2 bytes. It supports three QoS levels letting devices balance reliability against resource usage.

Kafka's protocol is designed for high-throughput server-to-server communication. Its clients require significant memory for batching, compression, and partition management. It is not suitable for microcontrollers or constrained devices.

Connection Patterns

MQTT brokers (like Mosquitto, EMQX, or HiveMQ) are designed to handle millions of concurrent connections from IoT devices. Each device maintains a persistent TCP connection with keepalive heartbeats.

Kafka is designed for fewer, higher-throughput connections. A consumer group might have dozens of consumers, not millions. Running a million Kafka clients is neither designed for nor practical.

The Combined Architecture

The standard IoT architecture uses both:

1. MQTT for device-to-cloud ingestion — millions of devices publish telemetry to an MQTT broker
2. MQTT-Kafka bridge forwards messages from the MQTT broker to Kafka topics
3. Kafka for processing, aggregation, and routing telemetry data to storage, analytics, and alerting systems

This separation lets each technology do what it does best. The MQTT broker handles massive device connectivity. Kafka handles high-throughput stream processing and persistence.

For system design interviews involving IoT or real-time telemetry, demonstrating this two-tier architecture shows strong understanding. See our interview questions for IoT design patterns.