Message Queues: Kafka, RabbitMQ & SQS
Master Apache Kafka architecture (topics, partitions, consumer groups, offsets), RabbitMQ exchanges and routing, AWS SQS, at-least-once vs exactly-once delivery semantics, and Dead Letter Queues for production async messaging systems.
Master Apache Kafka architecture (topics, partitions, consumer groups, offsets), RabbitMQ exchanges and routing, AWS SQS, at-least-once vs exactly-once delivery semantics, and Dead Letter Queues for production async messaging systems. This hands-on tutorial focuses on practical implementation of message queues: kafka, rabbitmq & sqs concepts.
Message Queues: Kafka, RabbitMQ & SQS
Message queues decouple producers (who create work) from consumers (who process work). Instead of Service A calling Service B directly and waiting for a response, A puts a message in a queue and B processes it when ready. This enables async communication, load leveling, and resilience.
[!IMPORTANT] Message queues solve the "slow consumer" problem: if your payment processing service takes 5 seconds per transaction but you're receiving 10,000 transactions/second, you need a queue to buffer the load. Without it, your upstream API would either timeout or need 50,000 payment processing servers.
Message Queue vs Event Streaming
| Message Queue | Event Streaming | |
|---|---|---|
| Consumption | Message deleted after consumed | Retained (replayed by multiple consumers) |
| Consumers | Compete for messages (one winner) | Independent consumer groups |
| Retention | Until consumed | Days to forever |
| Use case | Task queue, work distribution | Event log, analytics, ML pipelines |
| Examples | RabbitMQ, AWS SQS | Apache Kafka, AWS Kinesis |
Apache Kafka Deep Dive
Delivery Semantics
Quiz
Quiz
Question 1 of 2You use Kafka to process payment events. User 123 has 3 events in order: 'payment_initiated', 'payment_processing', 'payment_completed'. What must be true to guarantee these 3 events are processed in order?
Key Takeaways
✅ Message queues (RabbitMQ, SQS) are for task distribution; event streaming (Kafka) is for event logs readable by multiple independent consumers.
✅ Kafka partition key determines which partition a message lands on — use a key that guarantees ordering for related events (e.g., user_id).
✅ At-least-once is the default — make consumers idempotent (safe to process the same message twice).
✅ Dead Letter Queues prevent poison messages from blocking entire queues — always configure DLQs in production.
✅ Multiple consumer groups can read the same Kafka topic independently — analytics, ML training, and search indexing can all consume the same event stream.
Official Resources
- 📖 Apache Kafka Documentation — Topics, partitions, consumer groups
- 📖 AWS SQS Developer Guide — SQS, DLQ, FIFO queues
- 📖 RabbitMQ Tutorials — Work queues, pub/sub, routing
- 📖 Confluent Kafka Python — Python Kafka client
What's Next?
Message queues decouple services — now let's explore the full architectural pattern that leverages them.
Next: Event-Driven Architecture & CQRS — Event sourcing, CQRS pattern, and Saga for distributed transactions.