RLHF

Tuning a model using human preference comparisons, so it produces the kind of answer people actually rate highly.

Ethics & Safety Advanced 1 min read

Also known as: Reinforcement Learning from Human Feedback

Definition

A pre-trained language model predicts plausible text, which is not the same as being helpful, honest or safe. RLHF closes that gap by collecting human comparisons — shown two responses, which is better? — training a reward model to predict those preferences, and then optimising the language model against that reward model.

It is the step that turns a raw next-token predictor into something you would ship. It also inherits the taste and blind spots of whoever provided the comparisons, and over-optimising against a reward model produces answers that score well while getting worse — reward hacking.

How it relates to other terms

Broader term Reinforcement Learning Learning by acting in an environment and adjusting behaviour based on rewards, rather than from labelled examp... Part of Alignment The problem of getting a system to pursue what we actually intend, rather than a proxy that merely correlates... See also Large Language Model A neural network trained on very large amounts of text to predict what comes next, which turns out to be enoug...
Referenced by
Reinforcement Learning Broader term Alignment See also

Related terms

Alignment

The problem of getting a system to pursue what we actually intend, rather than a proxy that merely correlates with it.

Intermediate Ethics & Safety

Hallucination

When a model states something false with the same fluency and confidence it uses for things that are true.

Beginner Ethics & Safety

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction