About Big Data
Big data is the discipline of collecting, storing, and processing datasets too large
or too fast for a single machine, and this course follows the order most data
engineers learn it. It starts with Big Data Fundamentals (the five
Vs, batch versus streaming), then Programming for Data and
Distributed Storage & File Formats so you can work with real
data on real clusters.
Next comes Batch Processing with Spark: DataFrames, partitioning,
and tuning jobs that handle billions of rows. Ingestion & Stream
Processing then covers Kafka, Spark Streaming, and Flink for data that
arrives continuously, while NoSQL & Distributed Databases
explains Cassandra, MongoDB, and the CAP theorem.
Data Warehousing & Modelling covers star schemas, cloud
warehouses such as BigQuery and Snowflake, and transformations with dbt, and
Pipelines & Orchestration covers ETL and ELT design, Airflow,
and data quality checks.
Finally, Cloud Platforms & the Lakehouse covers AWS, Azure, GCP,
Databricks, Delta Lake, and Iceberg, and Governance, Security &
Analytics covers lineage, privacy rules like GDPR, dashboards, and ML
basics: the skills that turn a pile of raw data into something a business can trust.
Content
10 modules
01
Big Data Fundamentals
What makes data "big", the five Vs, batch versus streaming, and how data moves from source to insight.
5 VsBatch vs StreamingData Lifecycle
02
Programming for Data
The everyday toolkit: Python, advanced SQL, Scala basics, and the Linux command line for working on clusters.
PythonSQLLinux / Bash
03
Distributed Storage & File Formats
How data is split and replicated across machines, and why columnar formats make analytics faster and cheaper.
HDFSAmazon S3Parquet / Avro
04
Batch Processing with Spark
From MapReduce ideas to Apache Spark: transformations, DataFrames, partitioning, and tuning jobs that run on large datasets.
MapReduceApache SparkPySpark
05
Ingestion & Stream Processing
Moving data in real time with message brokers, then processing events as they arrive.
Apache KafkaSpark StreamingApache Flink
06
NoSQL & Distributed Databases
Choosing the right store for scale: wide-column, document, and key-value databases, plus the CAP theorem trade-offs.
CassandraMongoDBCAP Theorem
07
Data Warehousing & Modelling
Designing analytical schemas and querying them at scale in modern cloud warehouses.
Star SchemaBigQuery / Snowflakedbt
08
Pipelines & Orchestration
Building reliable ETL and ELT pipelines, scheduling them, and catching bad data before it reaches users.
Apache AirflowETL vs ELTData Quality
09
Cloud Platforms & the Lakehouse
Running big data on AWS, Azure, or GCP, and combining lake and warehouse ideas with open table formats.
DatabricksDelta LakeApache Iceberg
10
Governance, Security & Analytics
Keeping data trustworthy and compliant, then turning it into dashboards and machine learning features.
Data LineageGDPRBI & ML Basics
What You Can Do After This Course
By the end of the curriculum you'll have worked through 5 hands-on projects and
covered the core of what a junior to mid-level data engineering role expects. Concretely,
you'll be able to:
Explain the big data landscape: what makes a problem a big data problem, and when batch, streaming, or both are the right answer.
Work with data at scale: use Python, SQL, and the Linux shell, and choose storage formats like Parquet that keep queries fast and cheap.
Process large datasets: write Spark jobs that transform billions of rows, with sensible partitioning and tuning.
Handle real-time data: ingest events with Kafka and process them as they arrive with Spark Streaming or Flink.
Pick the right database: choose between NoSQL stores and warehouses, and reason about consistency and availability trade-offs.
Build dependable pipelines: design ETL and ELT workflows, schedule them with Airflow, and add data quality checks.
Deliver trusted insight: run it all on a cloud lakehouse, govern and secure the data, and serve it to dashboards and ML models.