Training only works if the network is told clearly what "good" means. That is the job of the loss function: a single number that measures how far the predictions are from the targets. Mean squared error is the usual choice for predicting numbers, and cross entropy is the usual choice for classification. Picking the wrong loss means optimizing the wrong thing.
The optimizer decides how to use the gradients to update the weights. Plain gradient descent is the starting point, SGD works on small batches to make each step cheap, and momentum and Adam add smoothing and per-parameter step sizes. The learning rate, and how it changes over time in a schedule, is often the single setting that matters most.
Watch the lessons in order, then continue to Overfitting & Regularization.
StatQuest walks through gradient descent with a worked example: a loss function, its slope, the learning rate and step size, and how the parameters settle at the minimum. It also touches on stochastic gradient descent.
Video 23minStatQuest on cross entropy, the loss usually used to judge how well a classification network fits its data, and why it works better than squared error for that job.
Video 9minWhy training on one example or a small batch at a time is so much cheaper than using the whole dataset for every step, and what that does to the path toward the minimum.
Video 10minA tour of the optimizers used in practice: how momentum smooths and speeds up updates, how RMSprop and AdaGrad adapt the step for each parameter, and how Adam combines the ideas.
Video 15min