Module 06 of 10

Workflow Orchestration

Lessons

About This Module

You can now build a pipeline, but a pipeline that only runs when you remember to start it isn't much use. Real pipelines have to run every hour or every night, in the right order, and keep going when something breaks. That job belongs to workflow orchestration.

An orchestrator lets you describe a workflow as a set of tasks and the dependencies between them, usually as a DAG (directed acyclic graph). It then handles scheduling, retries, alerting, and monitoring for you. Before orchestrators existed, many teams relied on cron jobs, which can start a script on a timer but know nothing about whether the previous step finished or failed.

The module centres on Apache Airflow, the most widely used orchestrator in data engineering. You'll learn what it is, its core concepts (DAGs, tasks, operators, task instances, and the scheduler), and how to install it and run your first workflow. A longer full course then walks through building and scheduling real pipelines.

You'll finish by comparing Airflow with newer alternatives, Dagster and Prefect, so you can see how different tools approach the same problem. Once your pipelines run reliably on a schedule, the next challenge is data that is too large for one machine. That comes next, in Big Data Processing.

Lessons

4 videos
01

What Is Apache Airflow? For Beginners

A beginner-friendly overview of what Apache Airflow is and is not, covering its benefits and main concepts so you know where it fits in a data stack.

02

Airflow Core Concepts in 5 Minutes

A quick explanation of the building blocks you will see in every Airflow project, including DAGs, tasks, operators, task instances, and execution dates.

03

Airflow Introduction and Installation

Get Airflow running on your own machine: install it, initialise the database, start the web server and scheduler, create a user, and switch on an example DAG.

04

Airflow Tutorial for Beginners: Full Course

A longer course that combines theory with hands-on demonstrations, covering the essential concepts and building working workflows in Apache Airflow 2.0.