Hands-on builds

Projects

Projects

About These Projects

The best way to make the modules stick is to build something. These five projects take you from an end-to-end pipeline with Docker, SQL and dbt, through Python ETL into PostgreSQL and Airflow orchestration, to real-time streaming with Kafka and Spark and Spark Structured Streaming on Databricks.

Each one is a code-along video, so you can pause, type it out yourself and adapt it to your own ideas. Pick the project that matches the module you've just finished and come back for the rest.

Projects

5 builds
01

Build a Data Pipeline from Scratch with Docker, SQL, dbt & Airflow

Build a full data pipeline step by step, starting with Docker and SQL, then adding dbt transformations and Airflow scheduling.

02

Build a Python ETL Pipeline with PostgreSQL & Airflow

Generate sample data with Faker, clean it with Pandas, load it into PostgreSQL running in Docker, and schedule it with Airflow.

03

Create an API Data Pipeline with Airflow

Pull data from an API with Python, process it, and automate the whole run as an Apache Airflow workflow.

04

Stream Real-Time Data with Kafka, Spark & Airflow

Ingest data from an API with Airflow, stream it through Kafka, process it with Spark, and store it in Cassandra and PostgreSQL, all in Docker.

05

Spark Structured Streaming with Databricks, Kafka & PostgreSQL

Build a real-time pipeline with Spark Structured Streaming on Databricks, writing to PostgreSQL and finishing with a live dashboard.