Information theory gives a precise way to measure uncertainty and surprise. It explains why the loss functions used to train models are built the way they are, and it ties together the probability ideas from the earlier modules.
The lessons start with entropy, the average surprise in a distribution, and mutual information, how much one variable tells you about another. Then they cover KL divergence, a measure of how far apart two distributions are, and finish with softmax and cross entropy loss, the standard way to train classifiers.
Watch the lessons in order, then put it all to work in the Projects.
Explains entropy as a measure of the average surprise in a probability distribution.
Video 16minShows how mutual information measures what one variable tells you about another.
Video 16minBuilds an intuition for KL divergence as a measure of distance between distributions.
Video 5minExplains cross entropy loss through a simple classification setup and links it to KL divergence.
Video 5minCovers softmax, which turns a model's raw outputs into probabilities.
Video 14minShows how cross entropy is used as the loss when training a neural network classifier.
Video 9min