Your pipelines now run on a schedule, but what happens when the data grows past what a single machine can handle? A script that works on a million rows can crawl or crash on a billion. Big data processing solves this by splitting the work across many machines that run in parallel.
You'll start with batch processing, where large volumes of data are collected and processed together on a schedule. The first tool to learn about is Hadoop MapReduce, which breaks a job into small pieces, processes them in parallel, and combines the results. It made large-scale processing possible, but it writes intermediate results to disk at every step.
That limitation led to Apache Spark, which keeps data in memory wherever it can and is now the standard engine for large-scale processing. You'll see how Spark differs from MapReduce, then work with it hands-on using PySpark, the Python interface. You'll meet the core ideas of SparkSession, RDDs, DataFrames, and Spark SQL.
The module closes with an overview of the wider big data landscape, so you can see where Hadoop, Spark, and managed services fit together. Batch processing works on data that has already arrived. Next you'll look at data that arrives continuously, in Streaming & Messaging.
An introduction to batch processing and how large datasets are processed efficiently, with a first look at MapReduce and Apache Spark.
A breakdown of the key differences between Spark and Hadoop MapReduce, to help you understand why Spark largely replaced it for modern big data work.
An explanation of Apache Spark as a multi-language analytics engine for scalable computing, and the main parts that make it up.
A hands-on PySpark course covering Spark installation, SparkContext and SparkSession, RDD transformations and actions, DataFrames, and Spark SQL.
An overview of popular distributed big data processing frameworks, including Hadoop, Spark, Amazon EMR, and Apache Hudi, and how their capabilities compare.