Home / Work / Choosing the model that earns its cost
Choosing the model that earns its cost
Benchmarking open-weight LLMs on real data, not only on test data.
- Models compared
- Four open-weight LLMs, from 24B to 397B parameters
- Result
- 16.5× smaller and 91% cheaper than the first pick
- Hosting
- France
The trap
The model that scores best on a test set is not always the one to ship. Test data is cleaner than real data, and a larger model multiplies hosting costs.
What I did
Same rules and prompts for every model, simulated and real inputs side by side, and a fixed test set. GPU capacity was planned from load simulations.
Result
The model that shipped is 16.5× smaller and 91% cheaper than the one that scored best at first, and it is hosted in France.
Facing a similar problem? Email me.