Attention

A mechanism that lets a model weigh how much each part of the input should influence each output, learned rather than fixed.

Deep Learning Intermediate 1 min read Start here

Also known as: Self-Attention

Definition

Attention computes, for every position in a sequence, a set of weights over all the other positions — how much each one should contribute. Those weights are computed from the data itself rather than fixed in advance, so the model learns which relationships matter.

In self-attention the sequence attends to itself, which is how a language model resolves what "it" refers to three sentences earlier. The standard formulation projects each token into a query, a key and a value; the query is compared against every key to produce the weights, and the weighted sum of values becomes the output.

Running several attention operations in parallel with different projections — multi-head attention — lets one head track syntax while another tracks long-range reference.

Key points

Weights are computed from the data, not fixed
Query/key/value is the standard formulation
Multiple heads let a model track several relationship types at once

How it relates to other terms

Broader term Transformer The neural network architecture behind almost all modern language models, built around attention instead of re... See also Embedding A list of numbers representing a piece of data, arranged so that similar things end up close together.
Referenced by
Transformer Builds on this Context Window See also

Related terms

Transformer

The neural network architecture behind almost all modern language models, built around attention instead of recurrence.

Intermediate Deep Learning

Backpropagation

The algorithm that works out how much each weight in a network contributed to the error, by applying the chain rule backwards through the layers.

Advanced Deep Learning

Gradient Descent

The optimisation method that trains most models: repeatedly step every parameter a little way downhill on the error surface.

Intermediate Deep Learning

Recurrent Neural Network

RNN

A network that processes sequences one element at a time, carrying a hidden state forward — the standard approach before transformers.

Advanced Deep Learning

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction