About Data Engineering
Data engineering is the craft of building the pipelines, warehouses, and platforms that
move raw data to the people and models that need it, and this course walks through it in
the order most engineers actually learn it. It starts with
Data Engineering Foundations (the data lifecycle, batch vs streaming,
and OLTP vs OLAP), then Python & SQL for Data and
Linux, Git & the Command Line so you can write, version, and run
real code.
Next comes Data Modeling & Warehousing: star schemas, dimensional
modeling, and analytical stores like Snowflake, BigQuery, and Redshift. From there,
ETL & ELT Pipelines covers ingestion, transformation with dbt, and
change data capture, while Workflow Orchestration covers scheduling,
dependencies, retries, and backfills with Airflow, Dagster, and Prefect.
Big Data Processing covers Spark, partitioning, and distributed
execution, and Streaming & Messaging covers Kafka, Flink, and
event-driven pipelines that process data as it arrives.
Finally, Cloud & Lakehouse covers object storage, Delta Lake and
Iceberg, containers, and infrastructure as code, and
Data Quality & Observability covers testing, monitoring, lineage,
and governance: the skills that separate a pipeline that runs once from a data platform
people can trust.
Content
10 modules
01
Data Engineering Foundations
What data engineers do, how data flows from source to consumer, and the core trade-offs between batch and streaming, OLTP and OLAP.
Data LifecycleBatch vs StreamingOLTP vs OLAP
02
Python & SQL for Data
Writing clean Python for data work and mastering SQL beyond the basics: joins, CTEs, and window functions.
PythonpandasWindow Functions
03
Linux, Git & the Command Line
Working comfortably in the shell, scripting repetitive tasks, and tracking changes with Git and GitHub or GitLab.
BashGitcron
04
Data Modeling & Warehousing
Designing schemas for analytics with dimensional modeling, and choosing a cloud data warehouse that fits the workload.
Star SchemaSnowflakeBigQuery
05
ETL & ELT Pipelines
Ingesting data from databases, APIs, and files, then transforming it in the warehouse with tested, version-controlled models.
dbtCDCIncremental Loads
06
Workflow Orchestration
Scheduling pipelines as dependency graphs, handling retries and failures, and running backfills safely.
AirflowDagsterIdempotency
07
Big Data Processing
Processing datasets too large for one machine with distributed engines, and tuning partitioning, shuffles, and file formats.
Apache SparkParquetPartitioning
08
Streaming & Messaging
Building event-driven pipelines with message brokers and stream processors, including delivery guarantees and windowing.
KafkaFlinkExactly-Once
09
Cloud & Lakehouse
Storing data in object storage with open table formats, and deploying reproducibly with containers and infrastructure as code.
S3Delta Lake / IcebergTerraform
10
Data Quality & Observability
Validating data, monitoring pipelines in production, tracing lineage, and applying governance so downstream users can trust the data.
Great ExpectationsLineageData Contracts
What You Can Do After This Course
By the end of the curriculum you'll have worked through 5 hands-on projects and
covered the core of what a junior to mid-level data engineering role expects.
Concretely, you'll be able to:
Explain how data platforms work: how data moves from source systems through storage and processing to analysts, dashboards, and models.
Write and version real code: build data jobs in Python and SQL, tracked with Git and run from the command line.
Model data for analytics: design dimensional schemas and load them into a cloud data warehouse.
Build reliable pipelines: ship ETL and ELT jobs with dbt, orchestrate them with Airflow, and make them idempotent and safe to rerun.
Process data at scale: use Spark for large batch workloads and Kafka for real-time streams.
Run it in the cloud: store data in a lakehouse, containerize your jobs, and provision infrastructure with code.
Keep data trustworthy: add tests and quality checks, monitor pipelines in production, and trace lineage when something breaks.