The best way to make the modules stick is to build something. These five projects take you from an end-to-end pipeline with Docker, SQL and dbt, through Python ETL into PostgreSQL and Airflow orchestration, to real-time streaming with Kafka and Spark and Spark Structured Streaming on Databricks.
Each one is a code-along video, so you can pause, type it out yourself and adapt it to your own ideas. Pick the project that matches the module you've just finished and come back for the rest.
Build a full data pipeline step by step, starting with Docker and SQL, then adding dbt transformations and Airflow scheduling.
Generate sample data with Faker, clean it with Pandas, load it into PostgreSQL running in Docker, and schedule it with Airflow.
Pull data from an API with Python, process it, and automate the whole run as an Apache Airflow workflow.
Ingest data from an API with Airflow, stream it through Kafka, process it with Spark, and store it in Cassandra and PostgreSQL, all in Docker.
Build a real-time pipeline with Spark Structured Streaming on Databricks, writing to PostgreSQL and finishing with a live dashboard.