RNNs read text one word at a time, which makes them slow to train and forgetful over long sentences. Attention removes that bottleneck by letting a model look back at every word in the input and decide how much each one matters for the word it is producing now. It first appeared as an add-on to sequence-to-sequence translation models, then turned out to be powerful enough to stand on its own.
The lessons start with the encoder-decoder idea and the problem attention solves. You then see self-attention built up through queries, keys and values, and how the transformer combines it with positional encoding and parallel computation. A look at decoder-only transformers explains the design behind ChatGPT-style models, and the module ends by building a small GPT in code.
This is the architecture behind nearly every modern language model, so it is worth taking your time. Watch the lessons in order, then continue on to Pretrained Models & Fine-Tuning.
Introduces the basics of how neural networks translate one language into another by converting one sequence into another.
Video 17minStatQuest explains attention, one of the most important concepts behind transformers and large language models, and shows it is not that complicated.
Video 16minA step-by-step walkthrough of the transformer, from word embedding and positional encoding to self-attention and encoder-decoder attention.
Video 36min3Blue1Brown visualizes self-attention, multiple heads and cross-attention, building the mechanism up from queries, keys and values.
Video 26minShows how the decoder-only transformer behind ChatGPT works, including masked self-attention, and how it differs from a normal transformer.
Video 36minAndrej Karpathy builds a small GPT from scratch in code. A long, hands-on watch that puts the whole module together.
Video 1h 56min