How to Read an AI Benchmark Without Being Fooled

Benchmarks are useful and routinely over-read. A score is a measurement of a specific capability, on a specific dataset, under a specific harness, at a specific moment. Strip away any one of those qualifiers and the number stops meaning what it appears to mean.

1. What exactly was measured?

Two benchmarks with similar names frequently test different skills. Read the task description and look at three sample items before you look at any score. If the samples do not resemble your work, the ranking does not transfer to your work.

2. How was it prompted?

Prompt format, system instructions, temperature, whether the model was allowed to think before answering, and how many attempts it got can move a result by more than the gap between two models. A comparison is only a comparison if the harness was identical.

3. How was it scored?

Exact-match scoring punishes a correct answer in an unexpected format. Model-graded scoring inherits the grader's biases and tends to favour outputs that resemble the grader's own style. Human scoring is the most trustworthy and the least reproducible. None of these is wrong; all of them change the number.

4. Could the answers already be in the training data?

Contamination is the hardest problem in the field and the least visible in a results table. A public benchmark that has existed for a few years has almost certainly been scraped. Treat a suspiciously strong result on an old, popular dataset as a question rather than an answer, and give more weight to held-out or freshly constructed sets.

5. Where is the error bar?

A two-point gap on a five-hundred-item benchmark is frequently noise. Look for repeated runs and a stated variance. If neither is offered, treat small differences between adjacent entries as a tie — because that is usually what they are.

6. Who ran it, and what were they hoping to show?

First-party results are not automatically dishonest, but they are selected. The interesting question is never whether a vendor's own numbers look good; it is which comparisons are missing, and whether an independent run reproduces them.

What to do with the score afterwards

Use benchmarks to build a shortlist and never to make the decision. Two or three candidates that all plausibly clear the bar, evaluated on fifty real examples from your own workload, will tell you more in an afternoon than a leaderboard will in a month.

The score answers "is this model in the right class". Your own examples answer "does this model do my job". Only the second one is the question you started with.