Home / Work / Choosing the model that earns its cost

Choosing the model that earns its cost

Benchmarking open-weight LLMs on real data, not only on test data.

Models compared
Four open-weight LLMs, from 24B to 397B parameters
Result
16.5× smaller and 91% cheaper than the first pick
Hosting
France

The trap

The model that scores best on a test set is not always the one to ship. Test data is cleaner than real data, and a larger model multiplies hosting costs.

What I did

Same rules and prompts for every model, simulated and real inputs side by side, and a fixed test set. GPU capacity was planned from load simulations.

Result

The model that shipped is 16.5× smaller and 91% cheaper than the one that scored best at first, and it is hosted in France.

Facing a similar problem? Email me.