So far your pipelines have worked on data that has already arrived and been stored. But plenty of data never stops arriving: payments, clicks, sensor readings, app logs. Waiting for tonight's batch job is too slow if you want to catch fraud as it happens or update a dashboard within seconds. Streaming is how you handle that.
You'll start with the difference between batch and stream processing, and when each one is the right choice. Then you'll meet Apache Kafka, the most widely used platform for event streaming. You'll learn its core ideas, including topics, producers, and consumers, and set it up hands-on.
Kafka is one tool in a larger family of messaging systems. These sit between the parts of a system so that one service can hand work or events to another without waiting for it. You'll compare Kafka with RabbitMQ and other message queues, and learn how to pick the right pattern for a given problem: a task queue, a topic, or a durable stream of events.
Once you can move data in real time, the next question is where it all lives and who runs the infrastructure. That comes next, in Cloud & Lakehouse.
What stream processing and batch processing are, why and when to use streaming over batch, and a real-world example that shows the difference.
A short introduction to the principles of Apache Kafka and how it works, using easy examples and diagrams.
A complete hands-on beginner tutorial that walks through core concepts, setting up Kafka, and working with producers, consumers, and topics.
Why Kafka and RabbitMQ are not interchangeable, with the fundamental differences in consumer patterns, message routing, acknowledgements, and use cases.
A framework for choosing the right message queue for your own problem, comparing RabbitMQ, Kafka, SQS, and Redis Streams.