Terminal-Bench 2.1: The Benchmark That Changed Agent Evals
Official TB2.1 rows for DeepSeek V4-Flash-0731, Kimi K3, GLM-5.2 and the rest — every score with its harness, effort, and checkpoint caveats attached.
Every “who’s the best coding model” headline this month reduces to one acronym: TB2.1. Terminal-Bench 2.1 is the agent benchmark that finally made terminal scores believable — by doing the boring things right. This explainer is part of the open-weight coding hub and is written to the same rule as its siblings: every number below is official and every harness is named.
What Terminal-Bench 2.1 actually is
Terminal-Bench is “a collection of benchmarks for measuring agents’ abilities to complete valuable and complex tasks in container environments” — per the official repo. Not repo-patch puzzles: real tasks like assembling proteins for synthesis, debugging async code, and resolving security vulnerabilities, run end-to-end in a sandboxed terminal that the agent actually operates.
2.1 is the verified hardening of 2.0, per the official release notes: 26 tasks were modified to fix bugs, adjust timeouts or resources, or improve robustness to reward hacking, with changes adopted from Z.ai’s Terminal-Bench 2.0 Verified set. Two structural choices make it trustworthy where earlier agent benchmarks drifted: a minimum of 5 trials per task, and — since 2.1 — community submissions are closed; only maintainer-run rows enter the leaderboard.
The official numbers
DeepSeek’s release changelog (fetched August 12) lists the 0731 build’s card: Terminal-Bench 2.1 at 82.7, NL2Repo 54.2, Cybergym 76.7, DeepSWE 54.4, Toolathlon-Verified 70.3, Agents’ Last Exam 25.2, Automation Bench 25.1, DSBench-FullStack 68.7, DSBench-Hard 59.6. The cross-lab context comes from Moonshot’s official card (fetched August 12), which reports Terminal-Bench 2.1 rows for the whole frontier:
| Model | TB2.1 | Harness (per official footnote) |
|---|---|---|
| GPT-5.6 Sol | 88.8 | Codex (OpenAI) |
| Kimi K3 | 88.3 | Kimi Code harness |
| Claude Fable 5 | 88.0 | Terminus 2 (Artificial Analysis) |
| Claude Opus 4.8 | 84.6 | Terminus 2 (Artificial Analysis) |
| GPT-5.5 | 83.4 | Codex (OpenAI) |
| DeepSeek V4-Flash-0731 | 82.7 | DeepSeek Harness (own card) |
| GLM-5.2 | 82.7 | Claude Code (GLM-5.2 blog) |
Moonshot’s card also covers DeepSWE: Kimi K3 67.5 (Kimi Code harness; 67.3 with the official leaderboard’s mini-SWE-agent), GLM-5.2 46.2 — versus DeepSeek’s own card at 54.4. Qwen3.8-Max’s launch-reported 79.7 remains vendor-reported (no official TB2.1 row at publish) — see the Qwen vs Kimi numbers beat for why that matters.
Harness and effort caveats (the part most coverage skips)
Every one of the rows above runs on a different instrument, and the official footnotes say so explicitly:
- DeepSeek tested with the DeepSeek Harness minimal mode — “to be released soon” — at max effort, topp 0.95, temperature 1.0 (own changelog). Its 82.7 is measured on a harness nobody outside DeepSeek can run yet.
- Kimi K3 is evaluated with the Kimi Code harness at max effort, temperature 1.0 (own card). Same model, different harness: DeepSWE 67.5 (Kimi Code) vs 67.3 (mini-SWE-agent) — a 0.2-point reminder that harnesses move scores.
- GLM-5.2’s 82.7 uses Claude Code, per the GLM-5.2 release blog as footnoted on Moonshot’s card.
- The official DeepSWE board (mini-swe-agent harness, maintainer-run) carries its own rows: kimi-k3 68.5 (max), qwen3-8-max 57.5 ± 3 (xhigh), deepseek-v4-flash 53.3 (max), glm-5-2 43.8 (max) — effort configs differ per row.
Rule of thumb: a TB2.1 delta between models on different harnesses is a claim about the harnesses, not just the models. When one side’s harness is unreleased, treat the delta as directional at best.
Why it changed agent evals
Three structural shifts. Verification instead of self-report: 2.1’s task fixes directly target reward hacking, and maintainer-only rows remove the “everyone cherry-picks their best run” dynamic. Terminal realism: tasks are end-to-end container operations (build, debug, exploit, assemble), so a score predicts work an agent actually does, not patch-matching. Harness accountability: the leading labs now publish their harness configuration with the score — the footnote is now part of the benchmark. That is why the open-weight cluster (hub, 0731 vs Max, pricing reconciliation) cites TB2.1 rows with harness attached, and why “82.7” means more on the DeepSeek card than on any aggregator.
How to read a TB2.1 number
- Harness first. Unreleased harness > measured on it — read any delta as provisional.
- Effort second. xhigh and max are not the same setting; the DeepSWE board’s own rows mix them.
- Checkpoint third. “V4-Flash” TB2.1 rows without the 0731 date refer to a different build (the disambiguation covers the full mechanics).
- Trials last. 5+ trials per task is the floor the maintainers enforce; anything below is marketing.
Official source
- Terminal-Bench 2.1 official repo: harbor-framework/terminal-bench-2-1 (fetched 2026-08-12)
- DeepSeek 0731 release card: api-docs.deepseek.com/updates (fetched 2026-08-12)
- Moonshot model card with cross-lab rows and harness footnotes: moonshotai/Kimi-K3 (fetched 2026-08-12)
- Official DeepSWE board: deepswe.datacurve.ai (fetched 2026-08-12)
First published August 12, 2026. All scores and harness notes trace to the official pages above, fetched in this session; vendor-reported figures are labeled as such.