Module 4 of 10

Batch Processing with Spark

Lessons

About This Module

Batch processing means working through a large, fixed pile of data in one go: a day of sales, a month of logs, a full history of events. It is the workhorse of big data, and Apache Spark is the tool most teams reach for to do it.

You'll start with MapReduce, the original idea of splitting work into small pieces that run on many machines. Then you'll see how Spark builds on it, how its architecture spreads a job across a cluster, and how to write real jobs with PySpark using DataFrames, aggregations, and more.

Spark also handles streaming, which comes up in the next module. Once you've finished this one, move on to Ingestion & Stream Processing.

Lessons

4 videos
01

Map Reduce Tutorial | Hadoop Introduction (Intellipaat)

The MapReduce programming model, what the map step does, and the basics of Hadoop architecture that Spark later improved on.

02

What Is Apache Spark? | Apache Spark vs Hadoop (Intellipaat)

Where Spark came from, how it distributes work across a cluster, and how it compares to Hadoop.

03

Spark Tutorial for Beginners | Spark Architecture & Components (Intellipaat)

What Spark is made of, how a job is laid out across a cluster, and a hands-on demo.

04

PySpark Tutorial (freeCodeCamp)

Writing Spark jobs in Python: DataFrames, handling missing values, grouping and aggregating, and a machine learning example on Databricks.