Quantization

Storing model weights at lower numerical precision to cut memory and speed up inference, trading a little accuracy for a lot of resource.

Infrastructure Advanced 1 min read

Also known as: Quantisation

Definition

Models are usually trained in 16- or 32-bit floating point. Quantisation converts the weights to a smaller representation — 8-bit or 4-bit integers are common — which shrinks the model proportionally and speeds up inference, because the bottleneck is moving weights rather than multiplying them.

Quality degrades gracefully rather than falling off a cliff: 8-bit is usually indistinguishable, and 4-bit is often acceptable. It is what makes running a capable model on a laptop possible at all.

How it relates to other terms

Part of Inference Running a trained model to get an output — as opposed to training, which is producing the model in the first p... Contrast with Model Distillation Training a small model to imitate a large one, keeping most of the capability at a fraction of the cost.
Referenced by
Inference See also Model Distillation Contrast with

Related terms

Vector Database

A database built to store embeddings and find the nearest ones fast, which is what makes semantic search practical at scale.

Intermediate Infrastructure

Inference

Running a trained model to get an output — as opposed to training, which is producing the model in the first place.

Beginner Infrastructure

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction