Gradient Descent

The optimisation method that trains most models: repeatedly step every parameter a little way downhill on the error surface.

Deep Learning Intermediate 1 min read

Definition

Gradient descent minimises a loss function by repeatedly moving parameters in the direction that reduces it fastest — the negative gradient. How far to move each step is the learning rate, and it is the hyperparameter most likely to ruin a training run in either direction.

In practice nobody computes the gradient over the whole dataset. Stochastic gradient descent estimates it from a small batch, which is noisier per step but vastly cheaper, and the noise turns out to help escape poor minima.

How it relates to other terms

See also Backpropagation The algorithm that works out how much each weight in a network contributed to the error, by applying the chain... See also Overfitting When a model learns the training data so closely — including its noise — that it performs worse on anything ne...
Referenced by
Neural Network See also Backpropagation See also Overfitting See also

Related terms

Transformer

The neural network architecture behind almost all modern language models, built around attention instead of recurrence.

Intermediate Deep Learning

Attention

A mechanism that lets a model weigh how much each part of the input should influence each output, learned rather than fixed.

Intermediate Deep Learning

Backpropagation

The algorithm that works out how much each weight in a network contributed to the error, by applying the chain rule backwards through the layers.

Advanced Deep Learning

Recurrent Neural Network

RNN

A network that processes sequences one element at a time, carrying a hidden state forward — the standard approach before transformers.

Advanced Deep Learning

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction