Data Engineering

Overview

About Data Engineering

Data engineering is the craft of building the pipelines, warehouses, and platforms that move raw data to the people and models that need it, and this course walks through it in the order most engineers actually learn it. It starts with Data Engineering Foundations (the data lifecycle, batch vs streaming, and OLTP vs OLAP), then Python & SQL for Data and Linux, Git & the Command Line so you can write, version, and run real code.

Next comes Data Modeling & Warehousing: star schemas, dimensional modeling, and analytical stores like Snowflake, BigQuery, and Redshift. From there, ETL & ELT Pipelines covers ingestion, transformation with dbt, and change data capture, while Workflow Orchestration covers scheduling, dependencies, retries, and backfills with Airflow, Dagster, and Prefect.

Big Data Processing covers Spark, partitioning, and distributed execution, and Streaming & Messaging covers Kafka, Flink, and event-driven pipelines that process data as it arrives.

Finally, Cloud & Lakehouse covers object storage, Delta Lake and Iceberg, containers, and infrastructure as code, and Data Quality & Observability covers testing, monitoring, lineage, and governance: the skills that separate a pipeline that runs once from a data platform people can trust.

Content

10 modules
01

Data Engineering Foundations

What data engineers do, how data flows from source to consumer, and the core trade-offs between batch and streaming, OLTP and OLAP.

Data LifecycleBatch vs StreamingOLTP vs OLAP
02

Python & SQL for Data

Writing clean Python for data work and mastering SQL beyond the basics: joins, CTEs, and window functions.

PythonpandasWindow Functions
03

Linux, Git & the Command Line

Working comfortably in the shell, scripting repetitive tasks, and tracking changes with Git and GitHub or GitLab.

BashGitcron
04

Data Modeling & Warehousing

Designing schemas for analytics with dimensional modeling, and choosing a cloud data warehouse that fits the workload.

Star SchemaSnowflakeBigQuery
05

ETL & ELT Pipelines

Ingesting data from databases, APIs, and files, then transforming it in the warehouse with tested, version-controlled models.

dbtCDCIncremental Loads
06

Workflow Orchestration

Scheduling pipelines as dependency graphs, handling retries and failures, and running backfills safely.

AirflowDagsterIdempotency
07

Big Data Processing

Processing datasets too large for one machine with distributed engines, and tuning partitioning, shuffles, and file formats.

Apache SparkParquetPartitioning
08

Streaming & Messaging

Building event-driven pipelines with message brokers and stream processors, including delivery guarantees and windowing.

KafkaFlinkExactly-Once
09

Cloud & Lakehouse

Storing data in object storage with open table formats, and deploying reproducibly with containers and infrastructure as code.

S3Delta Lake / IcebergTerraform
10

Data Quality & Observability

Validating data, monitoring pipelines in production, tracing lineage, and applying governance so downstream users can trust the data.

Great ExpectationsLineageData Contracts

What You Can Do After This Course

By the end of the curriculum you'll have worked through 5 hands-on projects and covered the core of what a junior to mid-level data engineering role expects. Concretely, you'll be able to:

Explain how data platforms work: how data moves from source systems through storage and processing to analysts, dashboards, and models.

Write and version real code: build data jobs in Python and SQL, tracked with Git and run from the command line.

Model data for analytics: design dimensional schemas and load them into a cloud data warehouse.

Build reliable pipelines: ship ETL and ELT jobs with dbt, orchestrate them with Airflow, and make them idempotent and safe to rerun.

Process data at scale: use Spark for large batch workloads and Kafka for real-time streams.

Run it in the cloud: store data in a lakehouse, containerize your jobs, and provision infrastructure with code.

Keep data trustworthy: add tests and quality checks, monitor pipelines in production, and trace lineage when something breaks.