Small Language Models Are Quietly Winning

Every few months a new frontier model arrives and the discussion is about scale: parameters, context length, benchmark scores.

Meanwhile, a quieter thing has been happening. A large and growing share of AI that actually ships - in phones, in keyboards, in editors, in the boring internals of apps you use daily - runs on models that are small enough to be unremarkable.

This is not a compromise story. It is a story about matching the tool to the job.

Capability per parameter went up, fast

The naive assumption was that capability scales with size, so smaller means worse. That was directionally right and is now much too crude.

What actually happened is that the capability you get per parameter improved dramatically. Better training data beat more training data. Distillation - training a small model on a large one's outputs - transferred a surprising amount. Quantisation cut memory requirements with modest quality loss.

The result: a model small enough to run on a phone now handles tasks that needed a data centre not long ago.

Most tasks are not frontier tasks

Here is the part that gets underweighted. The hardest problems need the best models. But most product problems are not the hardest problems.

Classify this message. Extract the date. Rewrite this sentence more formally. Shorten this to one line. Suggest three replies to a short conversation.

These are bounded. The input is small, the output is small, the success criteria are clear. A small model does them well, and a frontier model does them well too - just slower and at meaningfully higher cost, for output the user cannot distinguish.

Paying frontier prices for a classification task is not thoroughness. It is a rounding error you multiply by every request.

The three things small models buy you

Latency. Local inference has no network round trip. The difference between 60ms and 600ms is the difference between a feature that feels like part of the interface and one that feels like a request.

Cost. Per-request economics decide which features can exist. A feature that fires on every keystroke is impossible at frontier prices and trivial at small-model prices.

Privacy. A model on the device means data that never leaves it. Not "leaves it and is handled responsibly" - does not leave.

Routing is the actual architecture

The pattern in well-built products is not "pick a model." It is routing: send each request to the smallest thing that can handle it.

Simple, high-frequency, latency-sensitive work goes to a small local model. Open-ended, low-frequency, quality-sensitive work goes to a large hosted one. A classifier in the middle decides.

Done well, the user notices nothing except that the product is fast and the hard things still work. Done badly - everything to the big model - the product is slow and expensive. Done badly the other way, it is fast and disappointing.

What this means going forward

Scale is not finished; frontier models keep getting better and keep unlocking genuinely new things.

But the axis of progress in applied AI has shifted. The interesting engineering is no longer "how large a model can we use?" It is "how small a model can we get away with, and how do we know when we cannot?"

That is a healthier question. It produces products that are fast, affordable and private by default, and it reserves the expensive machinery for the moments that actually need it.