Token

The unit a language model actually reads and writes — usually a word fragment rather than a word or a character.

Natural Language Beginner 1 min read Start here

Also known as: Tokenization, Tokenisation

Definition

Models do not read characters or words; they read tokens. A tokeniser splits text into pieces drawn from a fixed vocabulary, typically using an algorithm like byte-pair encoding that keeps common words whole and breaks rare ones into fragments.

As a rough guide, one token is about four characters of English, and 100 tokens is about 75 words. Code, non-Latin scripts and unusual names all tokenise less efficiently, which is why the same paragraph can cost noticeably more in Japanese than in English.

Tokenisation explains several otherwise-baffling model behaviours: difficulty counting letters in a word, inconsistent arithmetic on long numbers, and sensitivity to whitespace all trace back to the fact that the model never saw the characters.

Key points

~4 characters of English per token
Rare words split into several tokens
Explains why models struggle to count letters

Why it matters

Pricing, rate limits and context windows are all measured in tokens, so it is the unit you are actually billed in.

How it relates to other terms

See also Context Window The maximum number of tokens a model can consider at once — everything it is allowed to "see" for a single res... See also Embedding A list of numbers representing a piece of data, arranged so that similar things end up close together. Part of Large Language Model A neural network trained on very large amounts of text to predict what comes next, which turns out to be enoug...
Referenced by
Large Language Model Builds on this Embedding See also Context Window Builds on this

Related terms

Large Language Model

LLM

A neural network trained on very large amounts of text to predict what comes next, which turns out to be enough to answer questions, write code and ho...

Beginner Natural Language

Context Window

The maximum number of tokens a model can consider at once — everything it is allowed to "see" for a single response.

Beginner Natural Language

Chain-of-Thought

CoT

Prompting a model to work through its reasoning step by step before answering, which measurably improves multi-step problems.

Intermediate Natural Language

Was this definition helpful?

Definitions are reviewed by our editorial team. Spotted a problem? Tell us.

Keep reading

Browse the glossary Suggest a correction