Ars previewed Mozilla's report on how cheap open models caught up on capability.
Ars previewed Mozilla's report on how cheap open models caught up on capability.
The performance gap between frontier AI models from US tech companies and the best open-weights models from Chinese companies has closed to just 4.4 months, according to a Mozilla report. That explains why many companies are shifting to the significantly cheaper open models for routine work - and helps reveal a narrow band of workloads where frontier models are worth the cost.
Most organizations should ideally be using open models as the default for the majority of their work, according to the latest State of Open Source AI report from Mozilla, published on September 15 and shared with Ars prior to publication. The report highlights how a leading open model, Moonshot AI's Kimi K3, achieves a composite AI performance score on the Artificial Analysis Intelligence Index that is just three points behind Anthropic's Fable 5 closed frontier model, all while costing just 30 percent of the latter.
"[A Closed model] earns its premium in a few places: expert professional work, high-intensity retrieval, and long context," Raffi Krikorian, chief technology officer at Mozilla, said in an email to Ars. "We see the decision to pay for closed [models] as workload-specific rather than organization-specific."
The open-weights AI models allow anyone to download the main model components and run the models on their own computers, but developers still typically withhold vital information, such as training data, the data pipeline, and training code. By comparison, US tech companies, like Anthropic and OpenAI, mostly offer closed frontier models that keep everything proprietary, requiring customers to pay more for access.
Organizations still pay for closed frontier models because they work out of the box and come bundled with "compliance packaging, support, and accountability," whereas many organizations lack the staff to run open-weights models well, Krikorian explained.
But the steadily narrowing performance gap between open and closed frontier models, coupled with the cost-effectiveness of open models, is now widely recognized by many companies. For example, delivery company DoorDash has been using Kimi for routine work while reserving Fable for more difficult tasks that would normally take human experts longer to complete.
Unsurprisingly, this has accelerated the use of open models since Mozilla's inaugural State of Open Source AI report was published on July 14.
The narrowing performance gap is being measured in several ways. For example, the research nonprofit METR has defined an AI model's time horizon as the length of tasks - measured by how long human experts require to complete them - that can be handled by AI models with a "reliable" 50 percent success rate. That time horizon has been doubling on a set cadence that has accelerated over time.
The best closed model can currently do a job that is 1.7 times as long as the longest job that the best open model can reliably finish.
"If the open frontier can handle a seven-hour job, the closed frontier can handle a 12-hour one," Krikorian told Ars. "In four months, the open model handles the 12-hour job, and the closed one handles something around 20."
Tasks requiring between eight and 12 hours are typically the ones that a closed frontier model can do and the open models cannot handle yet, Krikorian said. No models are generally capable of reliably doing tasks longer than 12 hours, whereas tasks taking less than eight hours can be done by either type of model and could be handed off to the cheaper open models.
There are many nuances and caveats for such comparisons. For example, closed frontier models often come with their own harness - the software layer that helps the models access various tools and memory to perform agent-like actions. Harnesses custom-built by AI labs for their models can sometimes boost performance on various tasks, whereas the models may perform less well with third-party harnesses.
To level the playing field, benchmarking company Vals AI has been evaluating different open and closed models by using its own neutral harness. When every model ran on the same harness in the Terminal-Bench 2.1 evaluation , the open-weights model GLM 5.2 from Chinese company Z.ai (Zhipu AI) scored within a point of Anthropic's Claude Opus 4.7 and 4.8 while costing about five times less per completed task.
In other words, paying for closed frontier models buys about a four-month head start at about five times the per-task cost - but only when currently looking at tasks taking between eight and 12 hours.
Reporting synthesized and verified under Nexvoro.tech editorial guidelines. Full primary records referenced via Ars Technica.
OpenAI has formed a new independent mathematics advisory group hosted at the Institute for Advanced Study in Princeton following a surge in automated problem-solving. The move arrives amid mounting academic scrutiny regarding the rapid pace of artificial intelligence in advanced mathematical research.
As debate intensifies over the rapid acceleration of artificial intelligence, OpenAI has proposed a comprehensive framework for international safety standards, focusing heavily on alignment research and recursive self-improvement. The move follows recent high-profile departures and escalating concerns from industry insiders regarding humanity's long-term control over advanced frontier models.
As Washington fiercely debates artificial intelligence oversight, Nvidia CEO Jensen Huang has emerged as President Donald Trump's top confidant, successfully pushing back against growing regulatory pressures. While rival tech executives advocate for government slowdowns, the head of the world's most valuable chipmaker is charting a rapid course for American tech supremacy.
