A model that fits on one GPU is the easy case. Large models do not fit, and their training data takes too long to process on a single device. Distributed training spreads the work across many GPUs: with data parallelism each GPU holds a copy of the model and sees different batches, and larger models also need the model itself split across devices. Mixed precision and gradient accumulation help squeeze more out of the memory and compute you have.
Deployment brings its own constraints: cost, speed and memory at inference time. Quantization stores weights in fewer bits to shrink a model, and parameter-efficient methods such as LoRA adapt a large model by training only a small number of extra weights. Together these are what make large models practical outside big labs.
Watch the lessons in order, then put it all to work in the Projects.
Umar Jamil explains how distributed data parallel training works and walks through running it across multiple GPUs in PyTorch.
Video 1h 12minUmar Jamil shows how LoRA adapts a large model by training small low-rank matrices instead of all the weights, then implements it in PyTorch.
Video 26minUmar Jamil explains how quantization shrinks models by using fewer bits per weight, and covers both post-training and quantization-aware approaches.
Video 50minA long, hands-on Andrej Karpathy session that trains GPT-2 from scratch and speeds it up with mixed precision, multi-GPU training and other practical tricks. Best watched after the videos above.
Video 4h 1min