You can now build a pipeline, but a pipeline that only runs when you remember to start it isn't much use. Real pipelines have to run every hour or every night, in the right order, and keep going when something breaks. That job belongs to workflow orchestration.
An orchestrator lets you describe a workflow as a set of tasks and the dependencies between them, usually as a DAG (directed acyclic graph). It then handles scheduling, retries, alerting, and monitoring for you. Before orchestrators existed, many teams relied on cron jobs, which can start a script on a timer but know nothing about whether the previous step finished or failed.
The module centres on Apache Airflow, the most widely used orchestrator in data engineering. You'll learn what it is, its core concepts (DAGs, tasks, operators, task instances, and the scheduler), and how to install it and run your first workflow. A longer full course then walks through building and scheduling real pipelines.
You'll finish by comparing Airflow with newer alternatives, Dagster and Prefect, so you can see how different tools approach the same problem. Once your pipelines run reliably on a schedule, the next challenge is data that is too large for one machine. That comes next, in Big Data Processing.
A beginner-friendly overview of what Apache Airflow is and is not, covering its benefits and main concepts so you know where it fits in a data stack.
A quick explanation of the building blocks you will see in every Airflow project, including DAGs, tasks, operators, task instances, and execution dates.
Get Airflow running on your own machine: install it, initialise the database, start the web server and scheduler, create a user, and switch on an example DAG.
A longer course that combines theory with hands-on demonstrations, covering the essential concepts and building working workflows in Apache Airflow 2.0.