Data has to get into your system before anything else can happen to it. This module is about ingestion, the job of collecting data from apps, databases, sensors, and logs, and about stream processing, where events are handled within moments of arriving instead of waiting for the next batch.
You'll start by comparing batch and streaming ingestion. Then you'll meet Apache Kafka, the platform most companies use to move events between systems, and learn its building blocks: topics, partitions, producers, and consumers. Finally you'll connect Kafka to Spark Streaming to process events as they flow in.
Where Module 4 handled data at rest, this one handles data in motion. Once you've finished it, move on to NoSQL & Distributed Databases.
The difference between batch and streaming ingestion, with AWS Glue for batch loads, Amazon Kinesis for real-time streams, and IoT data as an example.
A short introduction to what Kafka offers, how a Kafka cluster works, and the fundamentals of its design.
A longer walkthrough of Kafka architecture, topics and partitions, the publish and subscribe flow, command-line tools, and cluster setup.
How Kafka and Spark fit together, ending with an example of Spark Streaming reading events from Kafka.