Earlier modules treated vision as a closed set of labels. This module connects images with language, which lets a model recognize things it was never explicitly trained on and even create new images from a text description.
CLIP learns a shared embedding space for images and text by training on image and caption pairs, which makes zero-shot classification and image search possible. Multimodal language models such as LLaVA attach a vision encoder to a large language model so you can ask questions about an image.
On the generative side, diffusion models learn to turn random noise into an image step by step, and Stable Diffusion adds text guidance so a prompt steers the result. Watch the lessons in order, then continue on to Video Understanding.
How CLIP trains vision models by pairing images with text, and why that works well for zero-shot tasks and contrastive learning.
An introduction to LLaVA, a multimodal model that combines a CLIP vision encoder with a large language model, tested on a range of images.
A short tutorial on diffusion models: the forward noising process, the reverse process that generates samples, and conditional generation with text.
A visual walkthrough of how Stable Diffusion turns a text prompt into an image, covering the text encoder, refinement over timesteps, and the guidance scale.