The best way to make the modules stick is to build something. These five projects take you from batch processing with Spark to real-time streaming with Kafka, a full pipeline with Airflow and Cassandra, and finally two cloud lakehouse builds on Azure and Databricks.
Each one is a code-along video, so you can pause, type it out yourself and adapt it to your own ideas. Pick the project that matches the module you've just finished and come back for the rest.
Upload data to Amazon S3, write Spark code in Databricks to transform it, then run SQL analysis and build visualizations.
Stream data over a TCP socket into Apache Spark, publish it to Kafka, and index the results in Elasticsearch, all running in Docker.
Fetch data from an API with Airflow, stream it through Kafka, process it with Spark, and store it in Cassandra and PostgreSQL.
Start an end-to-end big data project from scratch on Azure, using a real e-commerce dataset.
Build bronze, silver and gold layers with incremental loading, Unity Catalog, Delta Live Tables and Databricks Workflows.