Backpropagation

The algorithm that works out how much each weight in a network contributed to the error, by applying the chain rule backwards through the layers.

Deep Learning Advanced 1 min read

Also known as: Backprop

Definition

Backpropagation computes the gradient of the loss with respect to every weight in a network. It works backwards from the output: given how wrong the final answer was, the chain rule gives how wrong each preceding layer's contribution was, all the way to the input.

Its importance is one of efficiency. A naive approach would perturb each weight separately and re-run the network — hopeless for millions of parameters. Backpropagation gets all the gradients in roughly the cost of one extra forward pass.

How it relates to other terms

See also Gradient Descent The optimisation method that trains most models: repeatedly step every parameter a little way downhill on the... Part of Neural Network A model built from layers of simple weighted units, trained by adjusting those weights until the output matche...
Referenced by
Neural Network See also Gradient Descent See also

Related terms

Transformer

The neural network architecture behind almost all modern language models, built around attention instead of recurrence.

Intermediate Deep Learning

Attention

A mechanism that lets a model weigh how much each part of the input should influence each output, learned rather than fixed.

Intermediate Deep Learning

Gradient Descent

The optimisation method that trains most models: repeatedly step every parameter a little way downhill on the error surface.

Intermediate Deep Learning

Recurrent Neural Network

RNN

A network that processes sequences one element at a time, carrying a hidden state forward — the standard approach before transformers.

Advanced Deep Learning

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction