Optimization Algorithms in Deep Learning: Mathematically Explained

Hello from the AI, but simple team! If you enjoy our content, consider supporting us so we can keep doing what we do.

Our newsletter is no longer sustainable to run at no cost, so we’re relying on different measures to cover operational expenses. Thanks again for reading!

Optimization Algorithms in Deep Learning: Mathematically Explained

AI, But Simple Issue #24

An optimization algorithm in deep learning is a method used to adjust the parameters (weights and biases) of a neural network to improve its predictions.

These algorithms are based on loss functions, functions used to measure how good or bad a given prediction is.

• It calculates the difference between the network's prediction and the actual result.

• If the output is 5, and the real value is 3, the loss is going to be greater than if the output was 4.

The goal of optimization is to reduce this error (or loss) as much as possible. Reducing this loss happens through shifting the numerical values of the parameters, and it makes the model’s predictions more accurate.

The optimization process stops when the algorithm reaches a point where the error can’t get any lower (or only gets lower very slowly). At this point, we say the model has converged. Ideally, this means the network is now as accurate as it can be on the given data.