Introducing Convergent 3.0

Our benchmark suite has a new version: Convergent 3.0. Every model we publish is measured with it – our own models and the open models we compare against.

What's in Convergent 3.0

• Coding – programming tasks checked by automated tests.
• HTML Coding – web pages the model builds, checked automatically in a browser.
• Knowledge – reading longer documents and answering questions about them.
• QA – factual questions, plus control questions with made-up words that must be answered with "I don't know".
• Humanity's Current Exam – our own set of hard questions with verified answers.
• Validation loss – how well a model predicts unseen text.
• Parameters – model size, read directly from the model file.

For comparison with the wider field we also run the original Terminal-Bench 4.0.

What changes

Benchmarks were updated, so scores measured with earlier versions are not directly comparable with new ones. We will re-measure our models with Convergent 3.0 and update the tables on our benchmark pages.

As always: made-up answers count as wrong.