Frontier Model or Small Model? Match the Job, Not the Hype

The reflex to reach for the largest available model is understandable and usually wrong. Model capability is only one of four axes that decide whether a deployment works, and it is the one that matters least for the majority of tasks people actually run.

The four axes

• Capability — can it do the task at all, reliably, on your hardest inputs?
• Latency — how long does the user wait, and does that wait sit inside an interaction or behind a queue?
• Cost — per call, multiplied by realistic volume rather than pilot volume.
• Predictability — does the same input produce a usably similar output tomorrow?

A frontier model wins the first axis by definition. It rarely wins the other three, and most production problems are lost on the other three.

Where a small model is the better answer

Classification, extraction, routing, tagging, short rewrites, format conversion and structured output all tend to be solved comfortably by a small model — often at a fraction of the latency and cost, and with more consistent formatting because there is less room for the model to be creative.

These also happen to be the bulk of the calls in most real systems. The impressive reasoning task is usually a small share of traffic and a large share of the conversation.

Where the large model earns its price

Multi-step reasoning, long-document synthesis, code that has to hold several constraints at once, ambiguous instructions, and anything where the failure mode is subtle rather than obvious. If you cannot easily tell whether an answer is wrong, spend the money on the model that is wrong less often.

Route, do not choose

The framing that resolves most of this is that you are not picking a model, you are picking a policy. A cheap model handles the common case; a check — a confidence signal, a schema validation, a simple heuristic — escalates the hard case to the expensive one.

Two things make this work in practice. The escalation rule has to be measurable, so you can see what share of traffic escalates and why. And the fallback has to be tested, because a routing layer that has never been exercised under failure is a single point of failure with extra steps.

Measure on your traffic, not on a leaderboard

Public benchmarks tell you how models compare on someone else's distribution of problems. Build a set of fifty to a hundred real inputs from your own system, including the awkward ones, and score candidates against that. It is a day of work and it will outlast several model generations.

The practical default

Start small, measure the failures, and escalate only the categories that genuinely fail. Starting large and optimising downward sounds equivalent but is not: once a system is built around a model's tolerance for vague instructions, moving to a smaller one means rewriting the prompts, the schema and often the product.