A network with enough weights can memorize its training data instead of learning the pattern behind it. It then scores very well on examples it has seen and poorly on new ones. This is overfitting, and it is the main reason a model that looks great in training disappoints in practice.
The first defence is a proper train / validation / test split, so you measure performance on data the model has never trained on. The second is regularization: weight decay penalizes large weights, dropout randomly switches neurons off during training, and normalization keeps activations well behaved. Each one makes it harder for the network to simply memorize.
Watch the lessons in order, then continue to PyTorch Essentials, where these ideas turn into working code.
StatQuest uses a simple example, predicting height from weight, to show how a model that is too simple misses the pattern (bias) while one that is too flexible fits the training data but does poorly on new data (variance).
Video 6minStatQuest on ridge regression, a way to stop a model overfitting by making it less sensitive to the training data. The same idea of penalizing large weights is what weight decay does in neural networks.
Video 20minAndrew Ng explains dropout: randomly switching off neurons during training so that the network cannot lean on any single one, which makes it generalize better.
Video 7min