# EvalMapBench 6.7 · private coding benchmark: every unstated detail, checked

Source page: https://evalmap.ai/evalmapbench/

[Skip to benchmark content](https://evalmap.ai/evalmapbench/#bench-main)

EvalMapBench 6.7 / measured here

# Terse tickets. Every detail checked.

43 maintenance tickets on two small private services, each written the way a colleague writes it: the feature, not the checklist. The repository states the rest — in CONTRIBUTING, in the docs, in the code, or only as a pattern — and a hidden verifier checks whether the model found it. A view of attention to detail on this specific body of work, not a statistical ranking of models across all software engineering.

- Models: 49

- Tasks: 43 on 2 private repositories

- Detail checks: 284 counted of 506; 222 met by the baseline

- Core · regression: 97 · 240

- Attempts: 3 × 20 min

- Harness: Harbor 0.24 terminus-2

- Measured: 2026-11-10

[Results](https://evalmap.ai/evalmapbench/#results)[Where the detail was stated](https://evalmap.ai/evalmapbench/#tiers)[Method](https://evalmap.ai/evalmapbench/#method)[Limitations](https://evalmap.ai/evalmapbench/#limitations)

01 / measured results

## Details met, against price and output tokens.

Up means a higher share of the detail checks met. The floor carries the list price per million output tokens and the median output tokens a trial spent, both on log scales: the corner nearest the left is cheaper and thriftier. Drag the chart, or focus it and use the arrow keys, to turn it; Price and Tokens face it square on. Every model ran the same tasks the same number of times through the same harness.

Drag to turn the chart; swipe or use the arrow keys when the frame is focused to see the full range.44 models with a quoted price; numbered keys identify the points.

### Point key · detail checks met

- [1Claude Fable 5.2 (high with fallback)92.6%](https://evalmap.ai/evalmapbench/#row-claude-fable-5-2-high)
- [2Claude Mythos 5.192.1%](https://evalmap.ai/evalmapbench/#row-claude-mythos-5-1)
- [3Claude Fable 5.1 (high with fallback)91.3%](https://evalmap.ai/evalmapbench/#row-claude-fable-5-1-high)
- [4Devin SWE-2 (max) (no per-token price, so no point on the chart)89.0%](https://evalmap.ai/evalmapbench/#row-devin-swe-2-max)
- [5Claude Opus 5.5 (medium)86.2%](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-medium)
- [6Claude Opus 5.5 (high)85.6%](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-high)
- [7Muse Wonder Full (max) (no per-token price, so no point on the chart)85.4%](https://evalmap.ai/evalmapbench/#row-muse-wonder-full-1-0-max)
- [8Claude Sonnet 5.5 (xhigh)84.4%](https://evalmap.ai/evalmapbench/#row-claude-sonnet-5-5-xhigh)
- [9Claude Sonnet 5.5 (high)84.1%](https://evalmap.ai/evalmapbench/#row-claude-sonnet-5-5-high)
- [10Hy484.1%](https://evalmap.ai/evalmapbench/#row-hy4)
- [11GPT-6.1 Terra (Max)82.7%](https://evalmap.ai/evalmapbench/#row-gpt-6-1-terra)
- [12Muse Wonder (max)82.3%](https://evalmap.ai/evalmapbench/#row-muse-wonder-1-0-max)
- [13Grok 4.7 (xhigh)81.8%](https://evalmap.ai/evalmapbench/#row-grok-4-7)
- [14Kimi K3.1 Pro81.5%](https://evalmap.ai/evalmapbench/#row-kimi-k3-1-pro)
- [15MiniMax M3.1 Flash (no per-token price, so no point on the chart)81.3%](https://evalmap.ai/evalmapbench/#row-minimax-m31-flash)
- [16GPT-6 Astra (xhigh)81.2%](https://evalmap.ai/evalmapbench/#row-gpt-6-astra-xhigh)
- [17Grok 4.7 (high)81.2%](https://evalmap.ai/evalmapbench/#row-grok-4-7-high)
- [18GPT-6.1 Terra (High)★81.0%](https://evalmap.ai/evalmapbench/#row-gpt-6-1-terra-high)
- [19Devin SWE-2 Flash (max) (no per-token price, so no point on the chart)80.9%](https://evalmap.ai/evalmapbench/#row-devin-swe-2-flash-max)
- [20GPT-6.1 Sol (xhigh, ultrafast)80.9%](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-xhigh-ultrafast)
- [21Muse Ember (Max)80.4%](https://evalmap.ai/evalmapbench/#row-muse-ember-1-0-max)
- [22Pareto 26.1080.4%](https://evalmap.ai/evalmapbench/#row-pareto-26-10)
- [23DeepSeek V4.1 Flash (high, 500k context)80.1%](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high-ctx500k)
- [24GPT-6.1 Sol (high)80.1%](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-high)
- [25GPT-6 Astra (high)80.0%](https://evalmap.ai/evalmapbench/#row-gpt-6-astra-high)
- [26GPT-6.1 Sol (high, ultrafast)79.6%](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-high-ultrafast)
- [27LongCat 2.5 (max)79.6%](https://evalmap.ai/evalmapbench/#row-longcat-2-5)
- [28DeepSeek V4.1 Flash (max)79.4%](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash)
- [29Claude Opus 5.5 (xhigh)79.3%](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-xhigh)
- [30DeepSeek V4.1 Flash (high, 1M context)79.3%](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high)
- [31Gemini 3.8 Flash (high)79.0%](https://evalmap.ai/evalmapbench/#row-gemini-3-8-flash)
- [32Gemini 4 Argon (max)78.9%](https://evalmap.ai/evalmapbench/#row-gemini-4-argon-max)
- [33Kimi K3 (max)78.6%](https://evalmap.ai/evalmapbench/#row-kimi-k3)
- [34GLM-5.3-FlashX78.5%](https://evalmap.ai/evalmapbench/#row-glm-5-3-flashx)
- [35GLM-5.3-Prime78.4%](https://evalmap.ai/evalmapbench/#row-glm-5-3-prime)
- [36Muse Spark 1.3 Contributor (max)77.9%](https://evalmap.ai/evalmapbench/#row-muse-spark-1-3-contributor-max)
- [37Ultramarine 1 Flash (max)77.8%](https://evalmap.ai/evalmapbench/#row-ultramarine-1-flash-max)
- [38Qwen3.8 27B (high, Cerebras)77.7%](https://evalmap.ai/evalmapbench/#row-qwen3-8-27b-cerebras-high)
- [39DeepSeek V4.1 Flash 1017 (Max)77.4%](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-1017)
- [40DeepSeek V4.1 Flash (high, 200k context)77.1%](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high-ctx200k)
- [41Mercury 3 (high)77.0%](https://evalmap.ai/evalmapbench/#row-mercury-3-high)
- [42Claude Haiku 5.5 (Xhigh)75.9%](https://evalmap.ai/evalmapbench/#row-claude-haiku-5-5-xhigh)
- [43Step 5 Preview73.0%](https://evalmap.ai/evalmapbench/#row-step-5)
- [44Beam 1.071.1%](https://evalmap.ai/evalmapbench/#row-reflection-beam)
- [45Nemotron 3.5 Super VL54.7%](https://evalmap.ai/evalmapbench/#row-nemotron-3-5-super-vl)
- [46Qwen3.8 Max Prime (high)46.6%](https://evalmap.ai/evalmapbench/#row-qwen3-8-max-prime-high)
- [47Qwen3.8 Max Prime (xhigh)44.3%](https://evalmap.ai/evalmapbench/#row-qwen3-8-max-prime-xhigh)
- [48Ornith 1.5 122B (high) (no per-token price, so no point on the chart)0.8%](https://evalmap.ai/evalmapbench/#row-ornith-1-5-122b-high)
- [49gpt-oss-120b (High)0.0%](https://evalmap.ai/evalmapbench/#row-gpt-oss-120b)

- Anthropic
- Other
- OpenAI
- xAI
- Moonshot
- DeepSeek
- Google
- Zhipu
- Alibaba
- StepFun
- Reflection

- Point: share of detail checks met beyond the gpt-oss-120b (high) baseline, mean over all trials
- Whisker: 95% interval over the trials, run-to-run spread included
- Dashed, on the back walls: point-estimate Pareto frontier against price and against output tokens, not significance proof
- ★Best on price, output tokens and detail share at once: of the models on the three-axis Pareto frontier, the nearest the cheap, thrifty, highest-share corner
- Hollow: run through the model maker’s own CLI, not the shared harness; see the model cell
- Mixed harness: some of the row’s trials ran through Claude Code or Codex rather than the shared harness; the model cell says which

What is measured. A terse ticket, a repository, twenty minutes. A hidden verifier then probes the changed repository as a black box: core checks for the feature itself, detail checks for what the ticket left unsaid, regression checks for behaviour nothing mentioned. Strict counts the trials where every check passed.

[Open in the 3D atlas →](https://evalmap.ai/?x=evalmapbench&z=tb4&p=100&go=3d#claude-fable-5-2-high)

49 published 6.7 rows · 43 tasks × 3 attempts, every row through Harbor 0.24 terminus-2 at the effort named under the model, except where the model cell names another harness. 8 rows marked mixed harness ran some trials through Claude Code or Codex. Spend is the measured cost per trial; for a flat-rate run, the list-price equivalent its CLI reported; the list price is USD per 1M output tokens. Tokens are the median trial’s input (the context every step re-sends, cached or not) and output (reasoning included). At 45 minutes re-runs only the trials that hit the 20-minute budget with a 45-minute one and recomputes the row; these re-runs covered only the 26 tasks of 6.4, and the ranking stays at 20 minutes. Baseline: the 222 detail checks gpt-oss-120b (high) met in any of its attempts are set aside and the score counts the remaining 284; the baseline row itself reads 0% by construction, and the score over every check is in the public JSON as detailAll.
| Model | Detail checks met | 95% interval | Strict trials | Core met | Regressions kept | Spend per trial | Tokens per trial | Median minutes | At 45 minutes | Output list price |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 [Claude Fable 5.2 (high with fallback)](https://evalmap.ai/#claude-fable-5-2-high)claude-fable-5-2 · high effort · via Claude Code 2.1.321, Anthropic's own CLI, inside Harbor 0.24 · Claude Code's switch to another model on a safety flag was off: every reply came from Fable 5.2, none was refused | 92.6% | 89.8–95.4% | 96 of 129 | 126 of 129 | 129 of 129 | $2.63 list-price equivalent $338.77 in all | 781k in 22k out | 4.8 3 timed out | Pending | $40 AA |
| 2 Claude Mythos 5.1claude-mythos-5-1 · high effort · via Claude Code 2.1.294, Anthropic's own CLI, inside Harbor 0.24 · not yet in the atlas | 92.1% | 89.5–94.7% | 92 of 129 | 127 of 129 | 129 of 129 | $1.81 list-price equivalent $233.11 in all | 526k in 15k out | 3.5 | 92.1% no trial needed more time | $50 LiteLLM list |
| 3 [Claude Fable 5.1 (high with fallback)](https://evalmap.ai/#claude-fable-5-1-high)claude-fable-5-1 · high effort · via Claude Code 2.1.294, Anthropic's own CLI, inside Harbor 0.24 | 91.3% | 88.4–94.1% | 89 of 129 | 128 of 129 | 129 of 129 | $1.67 list-price equivalent $215.67 in all | 504k in 15k out | 3.4 | 91.3% no trial needed more time | $50 AA |
| 4 Devin SWE-2 (max)devin/swe-2-max · max effort · via Devin CLI 3000.11.3, Cognition's own CLI, on the host (no container) · not yet in the atlas | 89.0% | 85.5–92.4% | 78 of 129 | 126 of 129 | 129 of 129 | Not billed list-price equivalent | Not reported | 8.2 5 timed out | Pending | $0 Devin CLI, free |
| 5 [Claude Opus 5.5 (medium)](https://evalmap.ai/#claude-opus-5-5-medium)openrouter/anthropic/claude-opus-5.5 · medium effort · mixed harness: some trials via Claude Code | 86.2% | 82.6–89.9% | 72 of 129 | 124 of 129 | 129 of 129 | $0.65 $83.26 in all | 400k in 15k out | 4.1 1 timed out | Pending | $20 AA |
| 6 [Claude Opus 5.5 (high)](https://evalmap.ai/#claude-opus-5-5-high)openrouter/anthropic/claude-opus-5.5 · high effort | 85.6% | 81.8–89.4% | 66 of 129 | 125 of 129 | 129 of 129 | $0.82 $105.76 in all | 433k in 20k out | 5.1 2 timed out | Pending | $20 AA |
| 7 Muse Wonder Full (max)meta/muse-wonder-full-1.0 · max effort · via Muse Code 1.4.6, Meta's own CLI, on the host (no container) · not yet in the atlas | 85.4% | 81.8–89.0% | 63 of 129 | 126 of 129 | 129 of 129 | Not billed list-price equivalent | Not reported | 10.2 5 timed out | Pending | Not quoted subscription only, no API price |
| 8 [Claude Sonnet 5.5 (xhigh)](https://evalmap.ai/#claude-sonnet-5-5-xhigh)openrouter/anthropic/claude-sonnet-5.5 · xhigh effort · mixed harness: some trials via Claude Code | 84.4% | 80.2–88.6% | 73 of 129 | 120 of 129 | 129 of 129 | $1.23 $158.91 in all | 1.2M in 78k out | 15.0 26 timed out | 87.7% strict 78 of 129; 8 trials still timed out | $10 AA |
| 9 [Claude Sonnet 5.5 (high)](https://evalmap.ai/#claude-sonnet-5-5-high)openrouter/anthropic/claude-sonnet-5.5 · high effort · mixed harness: some trials via Claude Code | 84.1% | 80.4–87.7% | 63 of 129 | 126 of 129 | 129 of 129 | $0.55 $71.49 in all | 671k in 20k out | 5.6 1 timed out | 84.5% strict 64 of 129; none timed out | $10 AA |
| 10 Hy4openrouter/tencent/hy4 · high effort · OpenRouter, Tencent provider only without fallbacks, max_tokens 65536 · not yet in the atlas | 84.1% | 80.1–88.1% | 61 of 129 | 123 of 129 | 128 of 129 | $0.21 $26.77 in all | 2.3M in 85k out | 19.1 63 timed out | Pending | $1.2505 Tencent Cloud TokenHub pricing (https://www.tencentcloud.com/document/product/1300/78937; 50% limited-time discount until 31 Dec 2026, list $2.501) |
| 11 [GPT-6.1 Terra (Max)](https://evalmap.ai/#gpt-6-1-terra)openrouter/openai/gpt-6.1-terra · max effort | 82.7% | 79.5–85.8% | 49 of 129 | 126 of 129 | 129 of 129 | $0.50 $65.04 in all | 1.0M in 85k out | 18.5 37 timed out | 82.7% strict 50 of 129; 20 trials still timed out | $5 AA |
| 12 Muse Wonder (max)meta/muse-wonder-1.0 · max effort · via Muse Code 1.4.6, Meta's own CLI, inside Harbor 0.24 · not yet in the atlas | 82.3% | 78.2–86.4% | 63 of 129 | 124 of 129 | 129 of 129 | $1.52 list-price equivalent $196.59 in all | 2.9M in 23k out | 4.7 | 82.3% no trial needed more time | $20 Meta API list |
| 13 [Grok 4.7 (xhigh)](https://evalmap.ai/#grok-4-7)openrouter/x-ai/grok-4.7 · xhigh effort | 81.8% | 78.2–85.3% | 49 of 129 | 125 of 129 | 128 of 129 | $0.65 $83.84 in all | 630k in 16k out | 5.4 | 81.8% no trial needed more time | $6 AA |
| 14 [Kimi K3.1 Pro](https://evalmap.ai/#kimi-k3-1-pro)openai/k3.1-pro · max effort · via Terminus-2 with each turn's reasoning passed back, inside Harbor 0.24 · max was set explicitly: Kimi Code's model page gave max as K3.1 Pro's default on 23 October 2026, during the run, and high by 24 October | 81.5% | 78.1–85.0% | 44 of 129 | 127 of 129 | 129 of 129 | Not billed list-price equivalent | 509k in 22k out | 8.2 | 81.5% no trial needed more time | $10 AA |
| 15 MiniMax M3.1 Flashopenai/MiniMax-M3.1-Flash · the API ignores reasoning_effort · not yet in the atlas | 81.3% | 77.4–85.2% | 49 of 129 | 126 of 129 | 129 of 129 | Not billed list-price equivalent | 1.1M in 43k out | 8.7 7 timed out | Pending | Not quoted no pay-as-you-go price published yet; run on api.minimax.io with the owner's key |
| 16 [GPT-6 Astra (xhigh)](https://evalmap.ai/#gpt-6-astra-xhigh)openrouter/openai/gpt-6-astra · xhigh effort · mixed harness: some trials via Codex | 81.2% | 77.0–85.3% | 52 of 129 | 126 of 129 | 129 of 129 | $1.57 $202.73 in all | 377k in 15k out | 6.6 | 81.2% no trial needed more time | $50 AA |
| 17 [Grok 4.7 (high)](https://evalmap.ai/#grok-4-7-high)openrouter/x-ai/grok-4.7 · high effort | 81.2% | 77.7–84.8% | 50 of 129 | 126 of 129 | 129 of 129 | $0.58 $74.27 in all | 505k in 15k out | 5.3 | 81.2% no trial needed more time | $6 AA |
| 18 [GPT-6.1 Terra (High)](https://evalmap.ai/#gpt-6-1-terra-high) ★openrouter/openai/gpt-6.1-terra · high effort | 81.0% | 77.6–84.4% | 47 of 129 | 129 of 129 | 129 of 129 | $0.10 $12.8 in all | 322k in 9k out | 2.8 | 81.0% no trial needed more time | $5 AA |
| 19 Devin SWE-2 Flash (max)devin/swe-2-flash-max · max effort · via Devin CLI 3000.11.234, Cognition's own CLI, on the host in a bubblewrap sandbox (no container) · each trial ran in a bubblewrap sandbox that showed Devin only its own task copy, with an empty home and no MCP servers; the network stayed open: 2 trials ran Devin's web_search (3 queries), which found only unrelated projects and the public EvalMapBench page, no task text, tests or solutions; 14 of 129 trials hit the 20-minute budget, none after finishing its work (7 were waiting on a model reply, 7 still working) · not yet in the atlas | 80.9% | 76.2–85.5% | 57 of 129 | 123 of 129 | 129 of 129 | Not billed list-price equivalent | Not reported | 5.4 14 timed out | 83.7% strict 60 of 129; 5 trials still timed out | $0 Devin CLI, free |
| 20 GPT-6.1 Sol (xhigh, ultrafast)openai/gpt-6.1-sol · xhigh effort · via Codex 0.161.0, OpenAI's own CLI, ultrafast service tier, inside Harbor 0.24 · not yet in the atlas | 80.9% | 76.7–85.2% | 55 of 129 | 125 of 129 | 129 of 129 | $0.28 list-price equivalent $35.51 in all | 500k in 11k out | 1.3 | 80.9% no trial needed more time | $10 OpenAI list, standard tier |
| 21 [Muse Ember (Max)](https://evalmap.ai/#muse-ember)meta/muse-ember-1.0 · max effort · via Muse Code 1.4.14, Meta's own CLI, inside Harbor 0.24 | 80.4% | 76.4–84.3% | 48 of 129 | 126 of 129 | 129 of 129 | $0.05 list-price equivalent $6 in all | 3.4M in 27k out | 3.3 | 80.4% no trial needed more time | $0.5 AA |
| 22 Pareto 26.10unbiased/pareto-26.10 · the API exposes no effort levels · not yet in the atlas | 80.4% | 76.6–84.3% | 40 of 129 | 129 of 129 | 129 of 129 | $0.12 list-price equivalent $16.06 in all | 316k in 13k out | 4.0 | 80.4% no trial needed more time | $3.2 Unbiased API list |
| 23 DeepSeek V4.1 Flash (high, 500k context)deepseek/deepseek-flash · high effort · not yet in the atlas | 80.1% | 76.1–84.1% | 44 of 129 | 128 of 129 | 128 of 129 | $0.09 $10.96 in all | 936k in 112k out | 10.0 17 timed out | Pending | $1.2 AA |
| 24 [GPT-6.1 Sol (high)](https://evalmap.ai/#gpt-6-1-sol-high)openrouter/openai/gpt-6.1-sol · high effort · mixed harness: some trials via Codex | 80.1% | 76.1–84.1% | 49 of 129 | 126 of 129 | 129 of 129 | $0.21 $26.86 in all | 291k in 9k out | 3.9 | 80.1% no trial needed more time | $10 AA |
| 25 [GPT-6 Astra (high)](https://evalmap.ai/#gpt-6-astra-high)openrouter/openai/gpt-6-astra · high effort · mixed harness: some trials via Codex | 80.0% | 76.0–84.0% | 48 of 129 | 126 of 129 | 129 of 129 | $1.20 $154.72 in all | 305k in 9k out | 4.3 | 80.0% no trial needed more time | $50 AA |
| 26 GPT-6.1 Sol (high, ultrafast)openai/gpt-6.1-sol · high effort · via Codex 0.161.0, OpenAI's own CLI, ultrafast service tier, inside Harbor 0.24 · not yet in the atlas | 79.6% | 75.2–83.9% | 51 of 129 | 125 of 129 | 128 of 129 | $0.19 list-price equivalent $24.07 in all | 321k in 6k out | 0.8 | 79.6% no trial needed more time | $10 OpenAI list, standard tier |
| 27 LongCat 2.5 (max)openai/LongCat-2.5 · max effort · via Terminus-2 with each turn's reasoning passed back, inside Harbor 0.24 · LongCat API with thinking enabled and a 262,144-token thinking budget and output limit · not yet in the atlas | 79.6% | 75.1–84.2% | 51 of 129 | 124 of 129 | 125 of 129 | $0.10 $12.57 in all | 2.0M in 48k out | 18.1 41 timed out | Pending | $1.2 LongCat API pricing (https://longcat.chat/platform/docs/pricing/longcat-2.5; limited-time discount, 2026-11-02) |
| 28 DeepSeek V4.1 Flash (max)deepseek/deepseek-flash · max effort · not yet in the atlas | 79.4% | 75.2–83.5% | 45 of 129 | 126 of 129 | 128 of 129 | $0.10 $13.08 in all | 995k in 146k out | 13.2 28 timed out | 79.5% strict 46 of 129; 22 trials still timed out | $1.2 AA, 20 Oct 2026 snapshot (no Intelligence Index v5 row yet) |
| 29 [Claude Opus 5.5 (xhigh)](https://evalmap.ai/#claude-opus-5-5-xhigh)openrouter/anthropic/claude-opus-5.5 · xhigh effort · mixed harness: some trials via Claude Code | 79.3% | 73.8–84.8% | 73 of 129 | 117 of 129 | 129 of 129 | $1.99 $256.55 in all | 877k in 63k out | 12.9 30 timed out | 86.3% strict 81 of 129; 13 trials still timed out | $20 AA |
| 30 DeepSeek V4.1 Flash (high, 1M context)deepseek/deepseek-flash · high effort · not yet in the atlas | 79.3% | 75.1–83.4% | 46 of 129 | 127 of 129 | 128 of 129 | $0.10 $13.43 in all | 801k in 117k out | 10.5 17 timed out | Pending | $1.2 AA |
| 31 [Gemini 3.8 Flash (high)](https://evalmap.ai/#gemini-3-8-flash)antigravity/gemini-3.8-flash · high effort · via Antigravity CLI 1.3.2, Google's own CLI, inside Harbor 0.24 | 79.0% | 75.1–82.9% | 44 of 129 | 127 of 129 | 128 of 129 | $1.08 $139.86 in all | 6.5M in 66k out | 12.4 | 79.0% no trial needed more time | $3.75 AA |
| 32 Gemini 4 Argon (max)google/gemini-4-argon · max effort · via cli2027 0.2.4, the owner's own CLI, inside Harbor 0.24 · the 1M output budget was barely used: 4 of 3,375 requests wrote more than 32k tokens, the longest 77k · not yet in the atlas | 78.9% | 74.2–83.6% | 53 of 129 | 126 of 129 | 129 of 129 | $0.77 list-price equivalent $99.29 in all | 1.2M in 32k out | 8.9 6 timed out | Pending | $10 Google API list (introductory) |
| 33 Kimi K3 (max)openrouter/moonshotai/kimi-k3 · max effort · not yet in the atlas | 78.6% | 74.5–82.7% | 39 of 129 | 120 of 129 | 123 of 129 | $0.48 $62.03 in all | 278k in 19k out | 11.0 9 timed out | 78.6% strict 39 of 129; 7 trials still timed out | $15 AA, 20 Oct 2026 snapshot (no Intelligence Index v5 row yet) |
| 34 GLM-5.3-FlashXopenrouter/z-ai/glm-5.3-flashx · thinking on at the API default; no effort requested · not yet in the atlas | 78.5% | 74.3–82.7% | 40 of 129 | 122 of 129 | 127 of 129 | $0.16 $20.62 in all | 716k in 51k out | 10.6 17 timed out | 78.0% strict 39 of 129; 12 trials still timed out | $1.25 OpenRouter list |
| 35 GLM-5.3-Primeopenrouter/z-ai/glm-5.3-prime · thinking on at the API default; no effort requested · not yet in the atlas | 78.4% | 74.2–82.6% | 40 of 129 | 126 of 129 | 127 of 129 | $1.12 $144.49 in all | 784k in 48k out | 12.9 38 timed out | 78.3% strict 39 of 129; 23 trials still timed out | $8.8 OpenRouter list |
| 36 Muse Spark 1.3 Contributor (max)meta/muse-spark-1.3-contributor · max effort · via Muse Code 1.4.4, Meta's own CLI, inside Harbor 0.24 · not yet in the atlas | 77.9% | 73.7–82.1% | 40 of 129 | 128 of 129 | 129 of 129 | $0.02 list-price equivalent $2.95 in all | 2.7M in 29k out | 5.6 | 77.9% no trial needed more time | $0.2 LiteLLM list |
| 37 Ultramarine 1 Flash (max)paperinstruments/ultramarine-1-flash · max effort · requests went through a local proxy pacing them under the API's 60-a-minute limit · not yet in the atlas | 77.8% | 73.6–82.0% | 38 of 129 | 124 of 129 | 128 of 129 | $0.10 list-price equivalent $12.2 in all | 1.2M in 66k out | 10.3 19 timed out | Pending | $0.5 Paper Instruments API list |
| 38 Qwen3.8 27B (high, Cerebras)cerebras/qwen-3.8-27b · high effort · not yet in the atlas | 77.7% | 73.8–81.7% | 39 of 129 | 124 of 129 | 124 of 129 | $1.56 $201.47 in all | 1.2M in 91k out | 10.4 16 timed out | Pending | $1.49 OpenRouter list (Cerebras endpoint) |
| 39 [DeepSeek V4.1 Flash 1017 (Max)](https://evalmap.ai/#deepseek-v4-1-flash-1017)deepseek/deepseek-flash · max effort | 77.4% | 72.7–82.1% | 52 of 129 | 123 of 129 | 126 of 129 | $0.09 $11.58 in all | 927k in 126k out | 10.7 11 timed out | Pending | $1.2 AA |
| 40 DeepSeek V4.1 Flash (high, 200k context)deepseek/deepseek-flash · high effort · not yet in the atlas | 77.1% | 72.8–81.4% | 36 of 129 | 126 of 129 | 127 of 129 | $0.09 $11.25 in all | 933k in 116k out | 10.5 21 timed out | Pending | $1.2 AA |
| 41 Mercury 3 (high)openrouter/inception/mercury-3 · high effort · OpenRouter, Inception provider only without fallbacks, max_tokens 131072 · not yet in the atlas | 77.0% | 72.6–81.3% | 41 of 129 | 122 of 129 | 128 of 129 | $0.04 $4.93 in all | 608k in 70k out | 4.3 | 77.0% no trial needed more time | $0.35 OpenRouter list for the Inception endpoint (50% launch discount, 2026-11-06; Inception list $0.70) |
| 42 [Claude Haiku 5.5 (Xhigh)](https://evalmap.ai/#claude-haiku-5-5-xhigh)openrouter/anthropic/claude-haiku-5.5 · xhigh effort · mixed harness: some trials via Claude Code | 75.9% | 71.0–80.9% | 48 of 129 | 118 of 129 | 129 of 129 | $0.12 $15.12 in all | 701k in 111k out | 12.2 18 timed out | 78.5% strict 50 of 129; 7 trials still timed out | $0.5 AA |
| 43 [Step 5 Preview](https://evalmap.ai/#step-5)openrouter/stepfun/step-5-preview · max effort | 73.0% | 68.4–77.6% | 34 of 129 | 120 of 129 | 125 of 129 | $0.38 $48.91 in all | 1.3M in 71k out | 17.1 23 timed out | 75.1% strict 33 of 129; 5 trials still timed out | $2.7 AA |
| 44 [Beam 1.0](https://evalmap.ai/#reflection-beam)reflection/Beam-1.0-501B-A23B · max effort · via Terminus-2 with each turn's reasoning passed back, inside Harbor 0.24 | 71.1% | 67.0–75.3% | 30 of 129 | 122 of 129 | 125 of 129 | $2.06 $266.3 in all | 1.5M in 36k out | 4.4 7 timed out | Pending | $1.2 AA |
| 45 [Nemotron 3.5 Super VL](https://evalmap.ai/#nemotron-3-5-super-vl)openrouter/nvidia/nemotron-3.5-super-vl | 54.7% | 49.6–59.8% | 9 of 129 | 109 of 129 | 117 of 129 | $0.08 $10.13 in all | 546k in 28k out | 3.0 2 timed out | Pending | $0.5 AA |
| 46 Qwen3.8 Max Prime (high)openrouter/qwen/qwen3.8-max-prime · high effort · not yet in the atlas | 46.6% | 39.6–53.6% | 26 of 129 | 86 of 129 | 125 of 129 | $1.64 $211.38 in all | 798k in 86k out | 20.7 125 timed out | 69.0% strict 59 of 129; 73 trials still timed out | $12 OpenRouter list |
| 47 Qwen3.8 Max Prime (xhigh)openrouter/qwen/qwen3.8-max-prime · xhigh effort · not yet in the atlas | 44.3% | 36.9–51.8% | 29 of 129 | 79 of 129 | 125 of 129 | $1.61 $208.06 in all | 799k in 83k out | 20.8 126 timed out | 65.6% strict 56 of 129; 79 trials still timed out | $12 OpenRouter list |
| 48 Ornith 1.5 122B (high)openai/ornith-ai/Ornith-1.5-122B · high effort · every one of the 129 trials hit the 20-minute limit; replies from RunInfra often took minutes, so the score reflects the provider's speed as much as the model · not yet in the atlas | 0.8% | 0.0–2.3% | 0 of 129 | 5 of 129 | 129 of 129 | Not billed list-price equivalent | 376k in 15k out | 14.7 129 timed out | Pending | Not quoted no per-token price found for RunInfra; run with the owner's key |
| 49 [gpt-oss-120b (High)](https://evalmap.ai/#gpt-oss-120b)openrouter/openai/gpt-oss-120b · high effort · the baseline: its checks are set aside | 0.0% | 0.0–0.0% | 1 of 129 | 52 of 129 | 109 of 129 | $0.22 $28.15 in all | 261k in 26k out | 4.2 14 timed out | 0.4% strict 1 of 129; 10 trials still timed out | $0.6 AA |

Controls. The reference solution passes every check on 43 of 43 tasks in the same containers; the untouched repository meets 0% of the detail checks and keeps 43 of 43 regression sets. The scale has a top and a bottom: doing nothing scores nothing.

02 / where the detail was stated

## The further from the ticket, the more is missed.

Every detail check is tagged with where a careful engineer could have learnt the obligation. Ticket: it is in the ticket. Rule: a sentence of CONTRIBUTING that applies. Code: a docstring, a document or an ADR elsewhere in the repository. Pattern: written nowhere, but every neighbour does it.

Detail checks met by tier beyond the baseline, every trial counted · 27 ticket, 125 rule, 81 code and 51 pattern checks per attempt.
| Model | Ticket 27 checks · stated in the ticket | Rule 125 checks · a rule in CONTRIBUTING | Code 81 checks · written elsewhere in the repository | Pattern 51 checks · only a pattern the code follows |
| --- | --- | --- | --- | --- |
| [Claude Fable 5.2 (high with fallback)](https://evalmap.ai/evalmapbench/#row-claude-fable-5-2-high) | 96% 78 of 81 | 97% 362 of 375 | 91% 221 of 243 | 82% 125 of 153 |
| [Claude Mythos 5.1](https://evalmap.ai/evalmapbench/#row-claude-mythos-5-1) | 96% 78 of 81 | 95% 357 of 375 | 93% 226 of 243 | 79% 121 of 153 |
| [Claude Fable 5.1 (high with fallback)](https://evalmap.ai/evalmapbench/#row-claude-fable-5-1-high) | 98% 79 of 81 | 96% 359 of 375 | 93% 226 of 243 | 75% 114 of 153 |
| [Devin SWE-2 (max)](https://evalmap.ai/evalmapbench/#row-devin-swe-2-max) | 94% 76 of 81 | 91% 343 of 375 | 87% 212 of 243 | 76% 117 of 153 |
| [Claude Opus 5.5 (medium)](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-medium) | 96% 78 of 81 | 94% 354 of 375 | 84% 205 of 243 | 64% 98 of 153 |
| [Claude Opus 5.5 (high)](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-high) | 95% 77 of 81 | 94% 352 of 375 | 85% 206 of 243 | 62% 95 of 153 |
| [Muse Wonder Full (max)](https://evalmap.ai/evalmapbench/#row-muse-wonder-full-1-0-max) | 98% 79 of 81 | 88% 331 of 375 | 88% 214 of 243 | 71% 108 of 153 |
| [Claude Sonnet 5.5 (xhigh)](https://evalmap.ai/evalmapbench/#row-claude-sonnet-5-5-xhigh) | 89% 72 of 81 | 87% 325 of 375 | 84% 204 of 243 | 65% 100 of 153 |
| [Claude Sonnet 5.5 (high)](https://evalmap.ai/evalmapbench/#row-claude-sonnet-5-5-high) | 96% 78 of 81 | 92% 344 of 375 | 82% 200 of 243 | 61% 93 of 153 |
| [Hy4](https://evalmap.ai/evalmapbench/#row-hy4) | 98% 79 of 81 | 89% 332 of 375 | 84% 204 of 243 | 70% 107 of 153 |
| [GPT-6.1 Terra (Max)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-terra) | 96% 78 of 81 | 90% 338 of 375 | 80% 195 of 243 | 59% 91 of 153 |
| [Muse Wonder (max)](https://evalmap.ai/evalmapbench/#row-muse-wonder-1-0-max) | 84% 68 of 81 | 90% 338 of 375 | 80% 194 of 243 | 63% 97 of 153 |
| [Grok 4.7 (xhigh)](https://evalmap.ai/evalmapbench/#row-grok-4-7) | 100% 81 of 81 | 92% 346 of 375 | 79% 191 of 243 | 53% 81 of 153 |
| [Kimi K3.1 Pro](https://evalmap.ai/evalmapbench/#row-kimi-k3-1-pro) | 94% 76 of 81 | 92% 345 of 375 | 80% 195 of 243 | 56% 86 of 153 |
| [MiniMax M3.1 Flash](https://evalmap.ai/evalmapbench/#row-minimax-m31-flash) | 91% 74 of 81 | 92% 344 of 375 | 77% 187 of 243 | 58% 88 of 153 |
| [GPT-6 Astra (xhigh)](https://evalmap.ai/evalmapbench/#row-gpt-6-astra-xhigh) | 96% 78 of 81 | 91% 342 of 375 | 82% 199 of 243 | 52% 79 of 153 |
| [Grok 4.7 (high)](https://evalmap.ai/evalmapbench/#row-grok-4-7-high) | 100% 81 of 81 | 91% 340 of 375 | 80% 195 of 243 | 51% 78 of 153 |
| [GPT-6.1 Terra (High)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-terra-high) | 100% 81 of 81 | 90% 339 of 375 | 81% 197 of 243 | 48% 73 of 153 |
| [Devin SWE-2 Flash (max)](https://evalmap.ai/evalmapbench/#row-devin-swe-2-flash-max) | 96% 78 of 81 | 87% 326 of 375 | 81% 198 of 243 | 64% 98 of 153 |
| [GPT-6.1 Sol (xhigh, ultrafast)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-xhigh-ultrafast) | 95% 77 of 81 | 91% 341 of 375 | 81% 197 of 243 | 56% 85 of 153 |
| [Muse Ember (Max)](https://evalmap.ai/evalmapbench/#row-muse-ember-1-0-max) | 95% 77 of 81 | 87% 326 of 375 | 81% 197 of 243 | 59% 91 of 153 |
| [Pareto 26.10](https://evalmap.ai/evalmapbench/#row-pareto-26-10) | 100% 81 of 81 | 91% 342 of 375 | 83% 201 of 243 | 46% 71 of 153 |
| [DeepSeek V4.1 Flash (high, 500k context)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high-ctx500k) | 100% 81 of 81 | 88% 330 of 375 | 78% 190 of 243 | 55% 84 of 153 |
| [GPT-6.1 Sol (high)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-high) | 96% 78 of 81 | 91% 342 of 375 | 80% 195 of 243 | 48% 73 of 153 |
| [GPT-6 Astra (high)](https://evalmap.ai/evalmapbench/#row-gpt-6-astra-high) | 96% 78 of 81 | 90% 337 of 375 | 82% 199 of 243 | 47% 72 of 153 |
| [GPT-6.1 Sol (high, ultrafast)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-high-ultrafast) | 95% 77 of 81 | 89% 335 of 375 | 81% 197 of 243 | 52% 80 of 153 |
| [LongCat 2.5 (max)](https://evalmap.ai/evalmapbench/#row-longcat-2-5) | 86% 70 of 81 | 87% 325 of 375 | 78% 189 of 243 | 59% 91 of 153 |
| [DeepSeek V4.1 Flash (max)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash) | 91% 74 of 81 | 87% 327 of 375 | 77% 188 of 243 | 61% 93 of 153 |
| [Claude Opus 5.5 (xhigh)](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-xhigh) | 90% 73 of 81 | 79% 298 of 375 | 77% 186 of 243 | 65% 99 of 153 |
| [DeepSeek V4.1 Flash (high, 1M context)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high) | 100% 81 of 81 | 88% 329 of 375 | 76% 184 of 243 | 57% 87 of 153 |
| [Gemini 3.8 Flash (high)](https://evalmap.ai/evalmapbench/#row-gemini-3-8-flash) | 99% 80 of 81 | 86% 324 of 375 | 80% 194 of 243 | 60% 92 of 153 |
| [Gemini 4 Argon (max)](https://evalmap.ai/evalmapbench/#row-gemini-4-argon-max) | 100% 81 of 81 | 82% 307 of 375 | 83% 202 of 243 | 63% 97 of 153 |
| [Kimi K3 (max)](https://evalmap.ai/evalmapbench/#row-kimi-k3) | 90% 73 of 81 | 88% 330 of 375 | 72% 175 of 243 | 52% 80 of 153 |
| [GLM-5.3-FlashX](https://evalmap.ai/evalmapbench/#row-glm-5-3-flashx) | 81% 66 of 81 | 89% 333 of 375 | 79% 191 of 243 | 56% 85 of 153 |
| [GLM-5.3-Prime](https://evalmap.ai/evalmapbench/#row-glm-5-3-prime) | 81% 66 of 81 | 87% 327 of 375 | 76% 185 of 243 | 57% 87 of 153 |
| [Muse Spark 1.3 Contributor (max)](https://evalmap.ai/evalmapbench/#row-muse-spark-1-3-contributor-max) | 90% 73 of 81 | 84% 314 of 375 | 81% 197 of 243 | 59% 90 of 153 |
| [Ultramarine 1 Flash (max)](https://evalmap.ai/evalmapbench/#row-ultramarine-1-flash-max) | 79% 64 of 81 | 89% 334 of 375 | 75% 183 of 243 | 55% 84 of 153 |
| [Qwen3.8 27B (high, Cerebras)](https://evalmap.ai/evalmapbench/#row-qwen3-8-27b-cerebras-high) | 79% 64 of 81 | 87% 327 of 375 | 77% 187 of 243 | 51% 78 of 153 |
| [DeepSeek V4.1 Flash 1017 (Max)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-1017) | 100% 81 of 81 | 88% 330 of 375 | 72% 175 of 243 | 53% 81 of 153 |
| [DeepSeek V4.1 Flash (high, 200k context)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high-ctx200k) | 100% 81 of 81 | 87% 325 of 375 | 76% 184 of 243 | 52% 79 of 153 |
| [Mercury 3 (high)](https://evalmap.ai/evalmapbench/#row-mercury-3-high) | 84% 68 of 81 | 90% 338 of 375 | 69% 167 of 243 | 52% 79 of 153 |
| [Claude Haiku 5.5 (Xhigh)](https://evalmap.ai/evalmapbench/#row-claude-haiku-5-5-xhigh) | 91% 74 of 81 | 83% 313 of 375 | 76% 184 of 243 | 47% 72 of 153 |
| [Step 5 Preview](https://evalmap.ai/evalmapbench/#row-step-5) | 86% 70 of 81 | 82% 307 of 375 | 69% 167 of 243 | 46% 70 of 153 |
| [Beam 1.0](https://evalmap.ai/evalmapbench/#row-reflection-beam) | 80% 65 of 81 | 81% 304 of 375 | 66% 161 of 243 | 46% 70 of 153 |
| [Nemotron 3.5 Super VL](https://evalmap.ai/evalmapbench/#row-nemotron-3-5-super-vl) | 72% 58 of 81 | 63% 238 of 375 | 53% 128 of 243 | 34% 52 of 153 |
| [Qwen3.8 Max Prime (high)](https://evalmap.ai/evalmapbench/#row-qwen3-8-max-prime-high) | 43% 35 of 81 | 38% 144 of 375 | 47% 114 of 243 | 39% 59 of 153 |
| [Qwen3.8 Max Prime (xhigh)](https://evalmap.ai/evalmapbench/#row-qwen3-8-max-prime-xhigh) | 36% 29 of 81 | 36% 135 of 375 | 40% 97 of 243 | 37% 57 of 153 |
| [Ornith 1.5 122B (high)](https://evalmap.ai/evalmapbench/#row-ornith-1-5-122b-high) | 0% 0 of 81 | 1% 2 of 375 | 0% 0 of 243 | 0% 0 of 153 |
| [gpt-oss-120b (High)](https://evalmap.ai/evalmapbench/#row-gpt-oss-120b) | 0% 0 of 81 | 0% 0 of 375 | 0% 0 of 243 | 0% 0 of 153 |

02b / task by task

## Every model on every task.

One square per model and task. Hover a square for the attempts behind it. All 43 tasks of EvalMapBench 6.7 are ranked. A dot marks a cell run through the maker’s own CLI (Claude Code or Codex, on a subscription) rather than the row’s own harness.

solved: every check passed in at least two of three attemptsrun through Claude Code or Codexnot solvednot run

|  | solved | gateway-stats-percentiles | gateway-frame-limit | fairqueue-requeue | fairqueue-export-json | gateway-channel-limit | gateway-close-reason | gateway-damaged-fixture | fairqueue-enqueue-delay | fairqueue-sim-p95 | fairqueue-admin-purge | gateway-delete-key-limit | gateway-capture-bytes | gateway-verify-frozen-tool | gateway-dump-capture | gateway-conflict-equal-value | fairqueue-finished-at | fairqueue-sim-crash | fairqueue-lease-default | fairqueue-fail-retry | fairqueue-release | gateway-fuzz-kit | fairqueue-priority-aging | fairqueue-sim-replay | gateway-verify-archive | gateway-unfilled-settlement | fairqueue-lease-kit | gateway-metrics-series | gateway-rename-compat | gateway-session-setting | gateway-compat-export | gateway-tool-option | fairqueue-inspect-tenant | fairqueue-tenant-pause | fairqueue-signature-field | fairqueue-adr-supersede | fairqueue-sim-tenants | gateway-canonical-capture | fairqueue-already-settled | fairqueue-ready-forecast | fairqueue-claim-receipt | fairqueue-history-check | gateway-divergence-witness | fairqueue-migrate-chain |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [Claude Fable 5.2 (high with fallback)](https://evalmap.ai/evalmapbench/#row-claude-fable-5-2-high) | 34 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Mythos 5.1](https://evalmap.ai/evalmapbench/#row-claude-mythos-5-1) | 31 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Fable 5.1 (high with fallback)](https://evalmap.ai/evalmapbench/#row-claude-fable-5-1-high) | 31 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Devin SWE-2 (max)](https://evalmap.ai/evalmapbench/#row-devin-swe-2-max) | 27 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Sonnet 5.5 (xhigh)](https://evalmap.ai/evalmapbench/#row-claude-sonnet-5-5-xhigh) | 26 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Opus 5.5 (medium)](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-medium) | 24 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Opus 5.5 (high)](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-high) | 23 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Opus 5.5 (xhigh)](https://evalmap.ai/evalmapbench/#row-claude-opus-5-5-xhigh) | 23 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Muse Wonder Full (max)](https://evalmap.ai/evalmapbench/#row-muse-wonder-full-1-0-max) | 21 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Sonnet 5.5 (high)](https://evalmap.ai/evalmapbench/#row-claude-sonnet-5-5-high) | 21 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Muse Wonder (max)](https://evalmap.ai/evalmapbench/#row-muse-wonder-1-0-max) | 21 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Devin SWE-2 Flash (max)](https://evalmap.ai/evalmapbench/#row-devin-swe-2-flash-max) | 20 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Hy4](https://evalmap.ai/evalmapbench/#row-hy4) | 19 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [MiniMax M3.1 Flash](https://evalmap.ai/evalmapbench/#row-minimax-m31-flash) | 18 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GPT-6.1 Sol (xhigh, ultrafast)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-xhigh-ultrafast) | 18 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [DeepSeek V4.1 Flash 1017 (Max)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-1017) | 18 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GPT-6 Astra (xhigh)](https://evalmap.ai/evalmapbench/#row-gpt-6-astra-xhigh) | 17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GPT-6.1 Sol (high)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-high) | 17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GPT-6.1 Sol (high, ultrafast)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-sol-high-ultrafast) | 17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [LongCat 2.5 (max)](https://evalmap.ai/evalmapbench/#row-longcat-2-5) | 17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Gemini 4 Argon (max)](https://evalmap.ai/evalmapbench/#row-gemini-4-argon-max) | 17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Grok 4.7 (xhigh)](https://evalmap.ai/evalmapbench/#row-grok-4-7) | 16 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Grok 4.7 (high)](https://evalmap.ai/evalmapbench/#row-grok-4-7-high) | 16 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Muse Ember (Max)](https://evalmap.ai/evalmapbench/#row-muse-ember-1-0-max) | 16 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GPT-6 Astra (high)](https://evalmap.ai/evalmapbench/#row-gpt-6-astra-high) | 16 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [DeepSeek V4.1 Flash (max)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash) | 16 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Claude Haiku 5.5 (Xhigh)](https://evalmap.ai/evalmapbench/#row-claude-haiku-5-5-xhigh) | 16 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GPT-6.1 Terra (Max)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-terra) | 15 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GPT-6.1 Terra (High)](https://evalmap.ai/evalmapbench/#row-gpt-6-1-terra-high) | 15 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [DeepSeek V4.1 Flash (high, 500k context)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high-ctx500k) | 15 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Mercury 3 (high)](https://evalmap.ai/evalmapbench/#row-mercury-3-high) | 15 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Kimi K3.1 Pro](https://evalmap.ai/evalmapbench/#row-kimi-k3-1-pro) | 14 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [DeepSeek V4.1 Flash (high, 1M context)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high) | 14 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Gemini 3.8 Flash (high)](https://evalmap.ai/evalmapbench/#row-gemini-3-8-flash) | 14 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GLM-5.3-FlashX](https://evalmap.ai/evalmapbench/#row-glm-5-3-flashx) | 13 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Pareto 26.10](https://evalmap.ai/evalmapbench/#row-pareto-26-10) | 12 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Kimi K3 (max)](https://evalmap.ai/evalmapbench/#row-kimi-k3) | 12 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [GLM-5.3-Prime](https://evalmap.ai/evalmapbench/#row-glm-5-3-prime) | 12 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Muse Spark 1.3 Contributor (max)](https://evalmap.ai/evalmapbench/#row-muse-spark-1-3-contributor-max) | 12 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Ultramarine 1 Flash (max)](https://evalmap.ai/evalmapbench/#row-ultramarine-1-flash-max) | 12 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Qwen3.8 27B (high, Cerebras)](https://evalmap.ai/evalmapbench/#row-qwen3-8-27b-cerebras-high) | 12 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Step 5 Preview](https://evalmap.ai/evalmapbench/#row-step-5) | 12 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [DeepSeek V4.1 Flash (high, 200k context)](https://evalmap.ai/evalmapbench/#row-deepseek-v4-1-flash-high-ctx200k) | 11 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Beam 1.0](https://evalmap.ai/evalmapbench/#row-reflection-beam) | 10 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Qwen3.8 Max Prime (high)](https://evalmap.ai/evalmapbench/#row-qwen3-8-max-prime-high) | 7 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Qwen3.8 Max Prime (xhigh)](https://evalmap.ai/evalmapbench/#row-qwen3-8-max-prime-xhigh) | 7 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Nemotron 3.5 Super VL](https://evalmap.ai/evalmapbench/#row-nemotron-3-5-super-vl) | 3 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [Ornith 1.5 122B (high)](https://evalmap.ai/evalmapbench/#row-ornith-1-5-122b-high) | 0 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| [gpt-oss-120b (High)](https://evalmap.ai/evalmapbench/#row-gpt-oss-120b) | 0 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

03 / from ticket to score

## How a result gets here.

-

### Start with real work

Two small private services of one developer, with their docs, decision records, frozen release files and conventions; the tickets are maintenance he would file himself.

-

### Write the ticket tersely

One sentence of what the feature does. Everything else the change must respect is already in the repository, and every check records where.

-

### Run 3 attempts

Every model runs through the same open harness (Harbor 0.24 terminus-2), 20 minutes per attempt, in a throwaway container, so the spread between rows is the model.

-

### Verify as a black box

A hidden verifier probes the changed repository and never reads the model’s reasoning. The reference solution passes every check; the untouched repository fails every core and detail check. Only these aggregates are published.

04 / read with care

## What these numbers cannot tell you.

-

### One developer, two codebases

The tasks reflect one person’s services and conventions, not all maintenance work.

-

### Private is not independently reproducible

The tickets, repositories and checks are not public, so outsiders cannot reproduce the benchmark from these aggregates.

-

### Few rows so far

49 configurations measured. Each row is 43 tasks × 3 attempts; the interval is over those trials and nothing else.

-

### The middle is a tie

Rows between the top and the bottom sit within each other’s intervals. Measured task by task, the tasks added since 6.4 separate two mid-table models no better than the earlier tasks did. Telling such a pair apart would take around sixty tasks. Read the order inside the middle as a tie.

-

### One harness is not every harness

Every row ran through the same generic agent, not the maker’s own CLI. A model may do better or worse in its own tooling.

-

### Detail is not difficulty

The tickets are small. A high score means the unsaid obligations were found and kept, not that hard problems were solved.

-

### List price is not actual spend

The chart’s price axis is the atlas’s output-token list price; the spend column is what the trials actually cost through the API used.

EvalMapBench 6.7 · measured here · 2026-11-10[Back to the atlas →](https://evalmap.ai/)[Sources & licences ↗](https://evalmap.ai/licenses/)