Free during early access
Teev builds a private benchmark from your team's merged pull requests and tests every model and coding agent against it. Evidence from your own code, not a public leaderboard.
Runs in your own account · your code and keys never reach Teev
Codex leads — 7 of 10
70% pass rate, 2 tasks ahead of Claude Sonnet 4.5 (5 of 10).
Every setup attempted the same ten tasks in the same sealed container. Cost is per task solved, not per token, so a model that loops and retries doesn't look cheap. Model prices are maintained by us, not scraped.
Not a public leaderboard the models trained on. The migrations, edge-case bugs and multi-service fixes your team actually merges. You'll see where each setup holds and where it falls over.
You leave with: the setup to default to and the ones to stop using.
Once you know what solves, you know what you can stop paying for. Measured per solved task and per token spent getting there.
You leave with: a defensible number for what your current defaults cost, and what they'd cost done differently.
Try it
Teev ships with a reviewed catalog of 158 open-source repositories, already screened and packaged. Sign in and run a comparison on one in minutes, with no GitHub connection and no repo of your own.
These are public repos, so the models may have seen this code. The catalog shows you how Teev works. Point it at your own repositories for numbers you can act on.
Not a public leaderboard
Every public benchmark is built from open-source repositories, and those repositories are in the training data of the models being ranked. Enterprise code doesn't look like that. Your services, your conventions, your legacy, your test suite, your review standards. A score earned on a public repo tells you almost nothing about a migration across four of your internal services.
Your merged pull requests fix both problems at once. No model has trained on them, and they are the work you actually do.
How it works
A setup is a model, the coding agent it runs in, and an effort level — for example, a Claude model in Claude Code on high effort.
Connect a repo. Teev rebuilds it exactly as it stood before each merged fix, with the fix hidden as the answer key. Screening is deterministic, and every task is verified in your own environment before it counts. Tasks that can't grade themselves reliably are thrown out.
Queue as many model, coding agent and effort combinations as you like. Every one gets the same task, the same environment and the same validation policy. No open internet, so nothing can look up the fix. The console only offers combinations your runner has already proven it can run.
An overall verdict, standings per setup, and a task-by-task matrix. Every result links to the patch, logs and diffs behind it, pulled from your own artifact storage, so you can see why a setup won or lost.
Token efficiency
Cost per token is the number everyone quotes and the one that misleads. Cheap tokens are no bargain if the setup burns ten times as many of them.
The same model in two coding agents can differ several-fold, usually because one re-feeds context on every step. Teev measures the tokens each setup spends to solve the same task.
We hold vetted prices for every model, refreshed every five minutes. Multiply that by the tokens a setup actually burns and you get the number your bill reflects.
Input, output and reasoning tokens, per setup, per task, with the logs underneath.
Security
Teev runs on your own machine or in your own AWS account. Your runner does the work and sends us scores and pull request facts, never code or credentials.
Your cloud account
Nothing in this box reaches Teev
Teev
No route into your network · no credential of yours
GitHub access is issued straight to your runner by device flow, so no token is ever pasted into a hosted page. Model keys are configured locally, from your own secret store, or replaced entirely by a workload identity such as a Bedrock IAM role. We refuse credential material outright.
Your runner calls out to ask for work. There's no inbound connection, and it only talks to a fixed list of addresses you can inspect.
Packaging and benchmark work run inside a disposable microVM, not on your host Docker daemon. If that isolation isn't available, the run fails rather than quietly falling back.
Screening can send linked issue text to a third-party model to help rank candidate pull requests. You choose whether that happens, and the decision is honoured end to end. During a run, agents send code to your model provider on your key, exactly as they do today. None of it touches Teev.
Results
Nobody doubts frontier models are the best available. The question is whether you need the best for a dependency bump, a null check, or a migration shaped like the last eleven. Teev answers it with your own data.
The metric your bill reflects, measured on work your team actually merged.
Tasks are frozen with a pinned environment. When a new model ships, re-run the same benchmark and the numbers are directly comparable.
Pre-signed links into your own artifact storage for any result: patches, logs and diffs.
We have no horse in the raceWe don't resell models and we take no referral fees. If the setup you already pay for wins, that's what the report says.
Early access
We're taking on a small number of teams, giving them the whole product, and keeping it free for them.
Everything. Private and public repositories, every model and coding agent, unlimited runs, team access with owner, admin and member roles. No tiers and nothing to upgrade to.
Honest feedback, and a short call after your first run.
We'll set it up alongside your team and turn the results into defaults your developers actually follow. Everything we produce lives in your Teev workspace.
Talk to us →Questions
A tool. It builds a coding benchmark from your own merged pull requests and tests models and coding agents against it. If you want help running it or acting on the results, we offer that too, but the tool works on its own.
Yes. Sign in and run a comparison against the 158-repo public catalog straight away. It's packaged and ready.
They show you how Teev works and what a comparison looks like. They're public repos, so the models may have trained on that code and it won't resemble your codebase. Your own repositories are where the numbers get trustworthy.
A GitHub repo with tests, Docker Engine or any Docker-compatible runtime (OrbStack, Colima, Podman, Docker Desktop), and API keys for the models you want to test. Or deploy the runner into your own AWS account.
Install and connect a repo in minutes. Packaging tasks and running a first comparison usually takes a few hours, depending on your repo's build.
Fewer than you'd expect. Teev tells you how many tasks your history can build before you run anything.
Any repo whose tests run in a container.
GitHub today.
Yes. Owners and admins invite teammates from the console, with owner, admin and member roles.
Public benchmarks are built from open-source repositories that the models have usually trained on, and open-source code is not what most enterprise teams ship. Teev uses your private work, so no model has seen the answers and the tasks look like your actual codebase. It also measures cost and tokens, which no leaderboard does.
Each task's hidden test must fail before the fix and pass after it, verified in your own environment. Tasks that can't grade themselves reliably are thrown out, as are ones where the linked issue already spells out the fix.
Cost per token only tells you the rate. Tokens per solved task tells you how much of it a setup actually burns, and that's where setups differ most.
Yes. Every result links to the patch, logs and diffs in your own artifact storage. Nothing is a bare score.
Model prices are vetted and entered by us rather than scraped or guessed, and refresh every five minutes.
Re-run the same benchmark. Tasks are frozen, so results are directly comparable, and a re-run costs very little.
Models from OpenAI, Anthropic and Google directly; open-weight models like Qwen, Kimi, GLM and DeepSeek through OpenRouter; and anything in Amazon Bedrock or Azure AI Foundry. Any coding agent your runner can run, and the console only offers combinations it has proven.
Yes. The benchmark library lists everything you've packaged, and deletion is a real, reconciled request. If a delete fails, it says so rather than hiding the row.
No. Your runner does the work on your infrastructure and sends us scores and pull request facts only.
Nowhere near us. They stay on your runner, come from your own secret store, or are replaced by a workload identity. We refuse credentials outright.
Device flow, issued directly to your runner. No token is ever pasted into a hosted page.
Yes, and it's the strongest setup: no model key enters the container at all.
Only during screening, when linked issue text can be sent to help rank candidate pull requests. You choose whether that happens, and your decision is honoured end to end.
Yes. Read the install script before running it. The runner only talks to a fixed list of addresses, and work executes in a disposable microVM rather than on your host Docker daemon.
Benchmark storage is tied to the enrolled runner identity, not a directory on disk, so one account's data never mixes into another's.
Yes. Early access teams get the whole product free, and we intend to keep it that way for them.
You do, directly to your model provider on your own keys. We never mark it up.
Yes. The coding agent and effort level change token spend as much as the model does, and you'll know what each fix actually costs.
You could. The hard part is making tasks trustworthy: rebuilding old environments, proving each task grades itself, filtering out bad tasks, and keeping it reproducible as models change.
No. No reselling, no referral fees, no partnerships.
Run a comparison on the public catalog in minutes, then point Teev at your own repositories.
