A model is only as good as the data feeding it. A data pipeline is the automated workflow that collects raw data, cleans and transforms it, and delivers it to wherever training and inference need it, on a schedule and without anyone running scripts by hand.
You'll start with what a pipeline is and how data moves through ingestion, transformation, and storage. From there the module covers the tools used to build them: Apache Airflow for scheduling and orchestrating workflows, Apache Spark for processing data that is too big for a single machine, and Great Expectations for checking that the data arriving in your pipeline is actually what you expect.
Reliable, validated data is what makes everything downstream trustworthy. Next up is Experiment Tracking & Registry, where you'll record how that data is turned into models.
A short explainer that uses three simple examples to show what a data pipeline is and why teams build them.
A beginner overview of what Airflow is and is not, its benefits, and its core concepts for orchestrating workflows.
A visual walkthrough of how Apache Spark works, including its architecture, drivers, executors, partitions, and the wider ecosystem.
An introduction to using the Great Expectations library for data quality testing, as an alternative to writing checks by hand with pytest.