Inference

Running a trained model to get an output — as opposed to training, which is producing the model in the first place.

Infrastructure Beginner 1 min read

Definition

Inference is using a model rather than creating one. Training happens once and costs a great deal; inference happens on every request and is where nearly all cumulative cost ends up for a deployed system.

Inference is dominated by memory bandwidth rather than raw compute — the weights have to be read for every token generated — which is why quantisation, batching and caching matter so much more in production than they do in a notebook.

How it relates to other terms

See also Quantization Storing model weights at lower numerical precision to cut memory and speed up inference, trading a little accu... See also Temperature A setting that controls how much randomness goes into choosing each token — low is predictable, high is varied...
Referenced by
Temperature Includes Quantization Includes Model Distillation See also

Related terms

Quantization

Storing model weights at lower numerical precision to cut memory and speed up inference, trading a little accuracy for a lot of resource.

Advanced Infrastructure

Vector Database

A database built to store embeddings and find the nearest ones fast, which is what makes semantic search practical at scale.

Intermediate Infrastructure

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction