Now that you know where data is going, it's time to build the pipelines that get it there. A pipeline pulls data out of a source, reshapes it, and loads it into a destination such as the warehouse you designed in the last module. Almost everything a data engineer builds is a variation on this pattern.
You'll start with the two classic approaches. In ETL (extract, transform, load) data is cleaned and shaped before it reaches the destination. In ELT (extract, load, transform) raw data lands in the warehouse first and is transformed there. You'll learn why modern cloud warehouses made ELT so popular and when each approach is the better fit.
From there you'll build and learn the tools. You'll write a working ETL pipeline in Python, then meet dbt, the tool most teams use to write, test, and document the transformation step inside the warehouse. The module closes with change data capture (CDC), a technique for copying only what changed in a source database instead of reloading everything.
Once you can build a pipeline, you'll need a way to run it reliably on a schedule. That comes next, in Workflow Orchestration.
The fundamentals of ETL and ELT, and why both approaches matter for getting data from your sources into the systems that use it.
A hands-on walkthrough of what ETL is and how to build a data pipeline in Python, moving data from SQL Server into PostgreSQL.
A beginner-friendly look at what dbt does in a modern data pipeline: turning raw warehouse data into tested, documented, ready-to-use tables.
A longer, hands-on dbt course covering core concepts and project structure, building and running models, macros, generic and singular tests, and snapshots, using BigQuery and GitHub.
How change data capture streams every change in a Postgres database to systems like Kafka or Kinesis, so analysts never have to query the production database directly.