Module 07 of 10

Big Data Processing

Lessons

About This Module

Your pipelines now run on a schedule, but what happens when the data grows past what a single machine can handle? A script that works on a million rows can crawl or crash on a billion. Big data processing solves this by splitting the work across many machines that run in parallel.

You'll start with batch processing, where large volumes of data are collected and processed together on a schedule. The first tool to learn about is Hadoop MapReduce, which breaks a job into small pieces, processes them in parallel, and combines the results. It made large-scale processing possible, but it writes intermediate results to disk at every step.

That limitation led to Apache Spark, which keeps data in memory wherever it can and is now the standard engine for large-scale processing. You'll see how Spark differs from MapReduce, then work with it hands-on using PySpark, the Python interface. You'll meet the core ideas of SparkSession, RDDs, DataFrames, and Spark SQL.

The module closes with an overview of the wider big data landscape, so you can see where Hadoop, Spark, and managed services fit together. Batch processing works on data that has already arrived. Next you'll look at data that arrives continuously, in Streaming & Messaging.

Lessons

5 videos
01

Batch Processing Explained: MapReduce and Apache Spark for Beginners

An introduction to batch processing and how large datasets are processed efficiently, with a first look at MapReduce and Apache Spark.

02

Spark vs Hadoop MapReduce: Performance and Usability

A breakdown of the key differences between Spark and Hadoop MapReduce, to help you understand why Spark largely replaced it for modern big data work.

03

How Apache Spark Redefines Big Data Processing at Massive Scale

An explanation of Apache Spark as a multi-language analytics engine for scalable computing, and the main parts that make it up.

04

PySpark Tutorial for Beginners

A hands-on PySpark course covering Spark installation, SparkContext and SparkSession, RDD transformations and actions, DataFrames, and Spark SQL.

05

Comparing Big Data Processing: Hadoop, Spark, EMR, and Hudi

An overview of popular distributed big data processing frameworks, including Hadoop, Spark, Amazon EMR, and Apache Hudi, and how their capabilities compare.