When a dataset no longer fits on one machine, you have to decide where it lives and how it is written down. This module covers both halves of that question: how data is spread across many computers, and the file formats that make it fast to read back.
You'll start with HDFS, the Hadoop Distributed File System, and see how it splits files into blocks and keeps copies on several machines so that a single failure doesn't lose anything. Then you'll compare big data file formats such as Parquet and Avro, and finish with cloud object storage like Amazon S3, which has become the default home for data in modern platforms.
These choices affect cost and speed in every later module. Once you've finished it, move on to Batch Processing with Spark.
Why a distributed file system exists, the ideas behind HDFS, and how its pieces work together to store very large files across a cluster.
The problems big data creates for storage, how HDFS solves them with blocks and replication, and a hands-on demo to finish.
Why big data systems use formats like Avro and Parquet, where each one is strong or weak, and live Python examples of working with them.
An introduction to Amazon S3 object storage: buckets, objects, storage classes, versioning, replication, and encryption.