Held-out benchmark
Twenty tasks taken from merged pull requests in a repository we don't publish, across three complexity levels and six categories. Every model and harness attempts the same twenty in the same sealed container, graded by the repository's own tests.
The repository stays private on purpose · no model has trained on these answers
Placeholder figures. This page is waiting on its first published run — the numbers below are shaped like real output but are not a result.
Standings
Best value marks the lowest cost per solved task, not the lowest price per token — a setup that loops and retries pays for every attempt and only banks the ones that landed. Model prices are maintained by us rather than scraped.
Where the tasks came from
A benchmark made only of easy bug fixes flatters everything. These twenty are spread across three complexity levels and six categories so a setup that only handles small, local changes can't hide behind an average.
The gap between Low and High is the number worth watching. It is where setups separate.
Categories and complexity levels are the same vocabulary Teev applies to your own pull requests.
Method
Every public benchmark is built from open-source repositories, and those repositories sit in the training data of the models being ranked. A score earned on code a model has already read is a memory test. Teev-SWE is drawn from a repository we have never published, so nothing here has been seen before.
That also means you can't audit it, which is the honest trade. Teev-SWE exists to show the shape of the answer. The number that should decide anything is the one you get from your own repositories.
Each setup gets the repository at the commit before the fix, the task description, and the test suite. It passes when the repository's own tests pass, and fails otherwise. There is no partial credit and no judge model.
Tokens in and out are counted from the provider's own accounting. Cost divides total spend by the tasks a setup actually resolved, so retries and loops are paid for but never credited.
Teev builds the same twenty-task shape from your own merged pull requests, and no model has trained on those either.
