Research
New open-weight releases bring fresh benchmarks and claims of matching leading proprietary models. For teams building an open-weight model strategy, it can be hard to know where to start. The practical question is how these models perform on the tasks a product needs to handle, and at what cost and latency.
We hosted Pulling Their Weight: Shipping Open Models to Production at Fastino Labs’ new San Francisco office to hear how practitioners choose and customize open-weight models for production.
The panel brought together NVIDIA research scientist Venkat Srinivasan, OpenAI product staff member Kathy Koverec, Abridge senior ML scientist Man Luo, and Alex Marquez, global head of Experian Ventures. Our co-founder, George Hurn-Maloney, moderated the discussion. [1]
We asked the panel: if you were starting a company today, which models would you build with? Their answers offer three lessons on choosing a starting model, deciding when to customize it, and putting smaller models to work.
1. Build for quality before optimizing cost
Srinivasan recommended comparing models on a representative benchmark, then building the prototype with the strongest performer. Starting with a capable model helps a team judge the product idea without first having to work around a weaker model’s limitations.
Once the prototype works, test lower-cost options, including open-weight models, against the prototype’s results one task at a time. Compare quality, inference cost, and response time, as Koverec emphasized, to identify where switching makes sense.
2. Use post-training to adapt open-weight models to your product
A model that performs well in general can still fall short on your product’s terminology, labels, or output requirements. Post-training lets teams target those gaps by training an open-weight model on task-specific examples.
Luo described startups starting with a strong model and collecting data for later distillation into an in-house open-weight model. One way to do this is to use the stronger model’s responses as training examples for the new model.
Our Fastino Training API provides managed infrastructure for supervised fine-tuning and reinforcement learning on supported open-weight models.
3. Consider using small models as specialists within the product
Small language models are worth considering when a team wants to lower the cost or latency of a well-defined task while keeping larger models for broader work. Open-weight options also give teams a way to customize the specialist for their own data and deployment needs. In a support application, for example, a small model could extract request details or select a queue while a larger model drafts the reply.
Srinivasan’s case for smaller models depended on evaluations that closely reflect customer queries. In the support example, the routing evaluation should include clear messages, ambiguous requests, and messages that do not belong in any queue. The team should also define a fallback, such as human review, for requests the model cannot route reliably.
Our GLiNER2.5 models (74M to 300M parameters) and support entity and relation extraction, classification, and structured output. The 340M-parameter GLiNER2.5-Decide supports structured decisions, including routing requests and choosing among permitted answers, with constraints across related decisions. Both have downloadable weights under Apache 2.0 and support fine-tuning, so teams can deploy them in their own environments and customize them with task data.
The same product can still use larger models for broader work. Srinivasan would lean toward a larger model when requests were hard to predict. Another panelist would consider starting larger when immediate revenue was the priority, because a larger model might perform better. The choice of model size can vary across tasks, whether the weights are open or proprietary.
Conclusion
A team can introduce open-weight models one component at a time. A working prototype gives the team a quality baseline, and task data creates an opportunity to train a model around the product’s needs.
For your next experiment, choose a task with measurable quality, cost, and latency, then compare an open-weight model with the one you use today. Use the results to decide whether to deploy it as is, fine-tune it, or keep the current model.
For deployment options and implementation details, see our guide to using open-weight models.
[1]: Panelists participated in their individual capacities, not as official representatives of their employers. Affiliations are provided for context.
