Model Distillation

Training a small model to imitate a large one, keeping most of the capability at a fraction of the cost.

Infrastructure Advanced 1 min read

Also known as: Knowledge Distillation

Definition

Distillation trains a small "student" model on the outputs of a large "teacher". The student learns from the teacher's full probability distribution rather than just the correct label, which carries more information than the label alone — the teacher's uncertainty is itself a useful signal.

The result is a model that is often far smaller and faster while retaining most of the teacher's performance on the distribution it was distilled over. It generalises less well outside that distribution, which is the trade being made.

How it relates to other terms

Contrast with Quantization Storing model weights at lower numerical precision to cut memory and speed up inference, trading a little accu... See also Inference Running a trained model to get an output — as opposed to training, which is producing the model in the first p... See also Fine-Tuning Continuing to train an existing model on a smaller, specific dataset so it adapts to a particular task, domain...
Referenced by
Quantization Contrast with

Related terms

Quantization

Storing model weights at lower numerical precision to cut memory and speed up inference, trading a little accuracy for a lot of resource.

Advanced Infrastructure

Vector Database

A database built to store embeddings and find the nearest ones fast, which is what makes semantic search practical at scale.

Intermediate Infrastructure

Inference

Running a trained model to get an output — as opposed to training, which is producing the model in the first place.

Beginner Infrastructure

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction