Module 08 of 10

Streaming & Messaging

Lessons

About This Module

So far your pipelines have worked on data that has already arrived and been stored. But plenty of data never stops arriving: payments, clicks, sensor readings, app logs. Waiting for tonight's batch job is too slow if you want to catch fraud as it happens or update a dashboard within seconds. Streaming is how you handle that.

You'll start with the difference between batch and stream processing, and when each one is the right choice. Then you'll meet Apache Kafka, the most widely used platform for event streaming. You'll learn its core ideas, including topics, producers, and consumers, and set it up hands-on.

Kafka is one tool in a larger family of messaging systems. These sit between the parts of a system so that one service can hand work or events to another without waiting for it. You'll compare Kafka with RabbitMQ and other message queues, and learn how to pick the right pattern for a given problem: a task queue, a topic, or a durable stream of events.

Once you can move data in real time, the next question is where it all lives and who runs the infrastructure. That comes next, in Cloud & Lakehouse.

Lessons

5 videos
01

Streaming vs Batch Processing Explained with Examples

What stream processing and batch processing are, why and when to use streaming over batch, and a real-world example that shows the difference.

02

Apache Kafka in 5 Minutes

A short introduction to the principles of Apache Kafka and how it works, using easy examples and diagrams.

03

Apache Kafka Tutorial for Beginners: Setup, Producer, Consumer and Topics

A complete hands-on beginner tutorial that walks through core concepts, setting up Kafka, and working with producers, consumers, and topics.

04

When to Use Kafka or RabbitMQ

Why Kafka and RabbitMQ are not interchangeable, with the fundamental differences in consumer patterns, message routing, acknowledgements, and use cases.

05

How to Choose a Message Queue: RabbitMQ, Kafka, SQS or Redis Streams

A framework for choosing the right message queue for your own problem, comparing RabbitMQ, Kafka, SQS, and Redis Streams.