e·EvalMapAI CAPABILITY ATLAS
AtlasSources & licences ↗

EvalMapBench 6.7 / measured here

Terse tickets. Every detail checked.

43 maintenance tickets on two small private services, each written the way a colleague writes it: the feature, not the checklist. The repository states the rest — in CONTRIBUTING, in the docs, in the code, or only as a pattern — and a hidden verifier checks whether the model found it. A view of attention to detail on this specific body of work, not a statistical ranking of models across all software engineering.

Models
49
Tasks
43 on 2 private repositories
Detail checks
284 counted of 506; 222 met by the baseline
Core · regression
97 · 240
Attempts
3 × 20 min
Harness
Harbor 0.24 terminus-2
Measured
2026-11-10

01 / measured results

Details met, against price and output tokens.

Up means a higher share of the detail checks met. The floor carries the list price per million output tokens and the median output tokens a trial spent, both on log scales: the corner nearest the left is cheaper and thriftier. Drag the chart, or focus it and use the arrow keys, to turn it; Price and Tokens face it square on. Every model ran the same tasks the same number of times through the same harness.

$0.3$1$3$10$305k10k20k50k100k30%40%50%60%70%80%90%100%0%Claude Haiku 5.5 (Xhigh): 75.9% of detail checks met over 129 trials (48 met every check; 95% 71.0–80.9%), $0.5 per 1M output tokens (AA), median 111k output tokens and 12.2 min per trial, run through Harbor 0.24 terminus-238DeepSeek V4.1 Flash (max): 79.4% of detail checks met over 129 trials (45 met every check; 95% 75.2–83.5%), $1.2 per 1M output tokens (AA, 20 Oct 2026 snapshot (no Intelligence Index v5 row yet)), median 146k output tokens and 13.2 min per trial, run through Harbor 0.24 terminus-224Mercury 3 (high): 77.0% of detail checks met over 129 trials (41 met every check; 95% 72.6–81.3%), $0.35 per 1M output tokens (OpenRouter list for the Inception endpoint (50% launch discount, 2026-11-06; Inception list $0.70)), median 70k output tokens and 4.3 min per trial, run through Harbor 0.24 terminus-237DeepSeek V4.1 Flash 1017 (Max): 77.4% of detail checks met over 129 trials (52 met every check; 95% 72.7–82.1%), $1.2 per 1M output tokens (AA), median 126k output tokens and 10.7 min per trial, run through Harbor 0.24 terminus-235DeepSeek V4.1 Flash (high, 200k context): 77.1% of detail checks met over 129 trials (36 met every check; 95% 72.8–81.4%), $1.2 per 1M output tokens (AA), median 116k output tokens and 10.5 min per trial, run through Harbor 0.24 terminus-236DeepSeek V4.1 Flash (high, 1M context): 79.3% of detail checks met over 129 trials (46 met every check; 95% 75.1–83.4%), $1.2 per 1M output tokens (AA), median 117k output tokens and 10.5 min per trial, run through Harbor 0.24 terminus-226DeepSeek V4.1 Flash (high, 500k context): 80.1% of detail checks met over 129 trials (44 met every check; 95% 76.1–84.1%), $1.2 per 1M output tokens (AA), median 112k output tokens and 10 min per trial, run through Harbor 0.24 terminus-219Ultramarine 1 Flash (max): 77.8% of detail checks met over 129 trials (38 met every check; 95% 73.6–82.0%), $0.5 per 1M output tokens (Paper Instruments API list), median 66k output tokens and 10.3 min per trial, run through Harbor 0.24 terminus-233gpt-oss-120b (High): 0.0% of detail checks met over 129 trials (1 met every check; 95% 0.0–0.0%), $0.6 per 1M output tokens (AA), median 26k output tokens and 4.2 min per trial, run through Harbor 0.24 terminus-244Qwen3.8 27B (high, Cerebras): 77.7% of detail checks met over 129 trials (39 met every check; 95% 73.8–81.7%), $1.49 per 1M output tokens (OpenRouter list (Cerebras endpoint)), median 91k output tokens and 10.4 min per trial, run through Harbor 0.24 terminus-234Muse Spark 1.3 Contributor (max): 77.9% of detail checks met over 129 trials (40 met every check; 95% 73.7–82.1%), $0.2 per 1M output tokens (LiteLLM list), median 29k output tokens and 5.6 min per trial, run through Muse Code 1.4.4, Meta's own CLI, inside Harbor 0.2432Hy4: 84.1% of detail checks met over 129 trials (61 met every check; 95% 80.1–88.1%), $1.2505 per 1M output tokens (Tencent Cloud TokenHub pricing (https://www.tencentcloud.com/document/product/1300/78937; 50% limited-time discount until 31 Dec 2026, list $2.501)), median 85k output tokens and 19.1 min per trial, run through Harbor 0.24 terminus-28Nemotron 3.5 Super VL: 54.7% of detail checks met over 129 trials (9 met every check; 95% 49.6–59.8%), $0.5 per 1M output tokens (AA), median 28k output tokens and 3 min per trial, run through Harbor 0.24 terminus-241GLM-5.3-FlashX: 78.5% of detail checks met over 129 trials (40 met every check; 95% 74.3–82.7%), $1.25 per 1M output tokens (OpenRouter list), median 51k output tokens and 10.6 min per trial, run through Harbor 0.24 terminus-230Step 5 Preview: 73.0% of detail checks met over 129 trials (34 met every check; 95% 68.4–77.6%), $2.7 per 1M output tokens (AA), median 71k output tokens and 17.1 min per trial, run through Harbor 0.24 terminus-239LongCat 2.5 (max): 79.6% of detail checks met over 129 trials (51 met every check; 95% 75.1–84.2%), $1.2 per 1M output tokens (LongCat API pricing (https://longcat.chat/platform/docs/pricing/longcat-2.5; limited-time discount, 2026-11-02)), median 48k output tokens and 18.1 min per trial, run through Terminus-2 with each turn's reasoning passed back, inside Harbor 0.2423Muse Ember (Max): 80.4% of detail checks met over 129 trials (48 met every check; 95% 76.4–84.3%), $0.5 per 1M output tokens (AA), median 27k output tokens and 3.3 min per trial, run through Muse Code 1.4.14, Meta's own CLI, inside Harbor 0.2417Qwen3.8 Max Prime (high): 46.6% of detail checks met over 129 trials (26 met every check; 95% 39.6–53.6%), $12 per 1M output tokens (OpenRouter list), median 86k output tokens and 20.7 min per trial, run through Harbor 0.24 terminus-242Beam 1.0: 71.1% of detail checks met over 129 trials (30 met every check; 95% 67.0–75.3%), $1.2 per 1M output tokens (AA), median 36k output tokens and 4.4 min per trial, run through Terminus-2 with each turn's reasoning passed back, inside Harbor 0.2440Qwen3.8 Max Prime (xhigh): 44.3% of detail checks met over 129 trials (29 met every check; 95% 36.9–51.8%), $12 per 1M output tokens (OpenRouter list), median 83k output tokens and 20.8 min per trial, run through Harbor 0.24 terminus-243GPT-6.1 Terra (Max): 82.7% of detail checks met over 129 trials (49 met every check; 95% 79.5–85.8%), $5 per 1M output tokens (AA), median 85k output tokens and 18.5 min per trial, run through Harbor 0.24 terminus-29Gemini 3.8 Flash (high): 79.0% of detail checks met over 129 trials (44 met every check; 95% 75.1–82.9%), $3.75 per 1M output tokens (AA), median 66k output tokens and 12.4 min per trial, run through Antigravity CLI 1.3.2, Google's own CLI, inside Harbor 0.2427Claude Sonnet 5.5 (xhigh): 84.4% of detail checks met over 129 trials (73 met every check; 95% 80.2–88.6%), $10 per 1M output tokens (AA), median 78k output tokens and 15 min per trial, run through Harbor 0.24 terminus-26GLM-5.3-Prime: 78.4% of detail checks met over 129 trials (40 met every check; 95% 74.2–82.6%), $8.8 per 1M output tokens (OpenRouter list), median 48k output tokens and 12.9 min per trial, run through Harbor 0.24 terminus-231Claude Opus 5.5 (xhigh): 79.3% of detail checks met over 129 trials (73 met every check; 95% 73.8–84.8%), $20 per 1M output tokens (AA), median 63k output tokens and 12.9 min per trial, run through Harbor 0.24 terminus-225Gemini 4 Argon (max): 78.9% of detail checks met over 129 trials (53 met every check; 95% 74.2–83.6%), $10 per 1M output tokens (Google API list (introductory)), median 32k output tokens and 8.9 min per trial, run through cli2027 0.2.4, the owner's own CLI, inside Harbor 0.2428Pareto 26.10: 80.4% of detail checks met over 129 trials (40 met every check; 95% 76.6–84.3%), $3.2 per 1M output tokens (Unbiased API list), median 13k output tokens and 4 min per trial, run through Harbor 0.24 terminus-218Kimi K3.1 Pro: 81.5% of detail checks met over 129 trials (44 met every check; 95% 78.1–85.0%), $10 per 1M output tokens (AA), median 22k output tokens and 8.2 min per trial, run through Terminus-2 with each turn's reasoning passed back, inside Harbor 0.2412Grok 4.7 (xhigh): 81.8% of detail checks met over 129 trials (49 met every check; 95% 78.2–85.3%), $6 per 1M output tokens (AA), median 16k output tokens and 5.4 min per trial, run through Harbor 0.24 terminus-211Grok 4.7 (high): 81.2% of detail checks met over 129 trials (50 met every check; 95% 77.7–84.8%), $6 per 1M output tokens (AA), median 15k output tokens and 5.3 min per trial, run through Harbor 0.24 terminus-214Claude Sonnet 5.5 (high): 84.1% of detail checks met over 129 trials (63 met every check; 95% 80.4–87.7%), $10 per 1M output tokens (AA), median 20k output tokens and 5.6 min per trial, run through Harbor 0.24 terminus-27Kimi K3 (max): 78.6% of detail checks met over 129 trials (39 met every check; 95% 74.5–82.7%), $15 per 1M output tokens (AA, 20 Oct 2026 snapshot (no Intelligence Index v5 row yet)), median 19k output tokens and 11 min per trial, run through Harbor 0.24 terminus-229Muse Wonder (max): 82.3% of detail checks met over 129 trials (63 met every check; 95% 78.2–86.4%), $20 per 1M output tokens (Meta API list), median 23k output tokens and 4.7 min per trial, run through Muse Code 1.4.6, Meta's own CLI, inside Harbor 0.2410GPT-6.1 Terra (High): 81.0% of detail checks met over 129 trials (47 met every check; 95% 77.6–84.4%), $5 per 1M output tokens (AA), median 9k output tokens and 2.8 min per trial, run through Harbor 0.24 terminus-215Claude Opus 5.5 (high): 85.6% of detail checks met over 129 trials (66 met every check; 95% 81.8–89.4%), $20 per 1M output tokens (AA), median 20k output tokens and 5.1 min per trial, run through Harbor 0.24 terminus-25GPT-6.1 Sol (xhigh, ultrafast): 80.9% of detail checks met over 129 trials (55 met every check; 95% 76.7–85.2%), $10 per 1M output tokens (OpenAI list, standard tier), median 11k output tokens and 1.3 min per trial, run through Codex 0.161.0, OpenAI's own CLI, ultrafast service tier, inside Harbor 0.2416Claude Opus 5.5 (medium): 86.2% of detail checks met over 129 trials (72 met every check; 95% 82.6–89.9%), $20 per 1M output tokens (AA), median 15k output tokens and 4.1 min per trial, run through Harbor 0.24 terminus-24GPT-6.1 Sol (high): 80.1% of detail checks met over 129 trials (49 met every check; 95% 76.1–84.1%), $10 per 1M output tokens (AA), median 9k output tokens and 3.9 min per trial, run through Harbor 0.24 terminus-220Claude Fable 5.2 (high with fallback): 92.6% of detail checks met over 129 trials (96 met every check; 95% 89.8–95.4%), $40 per 1M output tokens (AA), median 22k output tokens and 4.8 min per trial, run through Claude Code 2.1.321, Anthropic's own CLI, inside Harbor 0.241GPT-6.1 Sol (high, ultrafast): 79.6% of detail checks met over 129 trials (51 met every check; 95% 75.2–83.9%), $10 per 1M output tokens (OpenAI list, standard tier), median 6k output tokens and 0.8 min per trial, run through Codex 0.161.0, OpenAI's own CLI, ultrafast service tier, inside Harbor 0.2422GPT-6 Astra (xhigh): 81.2% of detail checks met over 129 trials (52 met every check; 95% 77.0–85.3%), $50 per 1M output tokens (AA), median 15k output tokens and 6.6 min per trial, run through Harbor 0.24 terminus-213Claude Fable 5.1 (high with fallback): 91.3% of detail checks met over 129 trials (89 met every check; 95% 88.4–94.1%), $50 per 1M output tokens (AA), median 15k output tokens and 3.4 min per trial, run through Claude Code 2.1.294, Anthropic's own CLI, inside Harbor 0.243Claude Mythos 5.1: 92.1% of detail checks met over 129 trials (92 met every check; 95% 89.5–94.7%), $50 per 1M output tokens (LiteLLM list), median 15k output tokens and 3.5 min per trial, run through Claude Code 2.1.294, Anthropic's own CLI, inside Harbor 0.242GPT-6 Astra (high): 80.0% of detail checks met over 129 trials (48 met every check; 95% 76.0–84.0%), $50 per 1M output tokens (AA), median 9k output tokens and 4.3 min per trial, run through Harbor 0.24 terminus-221price per 1M output tokens, logmedian output tokens per trial, logdetail checks met, %
Drag to turn the chart; swipe or use the arrow keys when the frame is focused to see the full range.44 models with a quoted price; numbered keys identify the points.
49 published 6.7 rows · 43 tasks × 3 attempts, every row through Harbor 0.24 terminus-2 at the effort named under the model, except where the model cell names another harness. 8 rows marked mixed harness ran some trials through Claude Code or Codex. Spend is the measured cost per trial; for a flat-rate run, the list-price equivalent its CLI reported; the list price is USD per 1M output tokens. Tokens are the median trial’s input (the context every step re-sends, cached or not) and output (reasoning included). At 45 minutes re-runs only the trials that hit the 20-minute budget with a 45-minute one and recomputes the row; these re-runs covered only the 26 tasks of 6.4, and the ranking stays at 20 minutes. Baseline: the 222 detail checks gpt-oss-120b (high) met in any of its attempts are set aside and the score counts the remaining 284; the baseline row itself reads 0% by construction, and the score over every check is in the public JSON as detailAll.
ModelDetail checks met95% intervalStrict trialsCore metRegressions keptSpend per trialTokens per trialMedian minutesAt 45 minutesOutput list price
1 Claude Fable 5.2 (high with fallback)claude-fable-5-2 · high effort · via Claude Code 2.1.321, Anthropic's own CLI, inside Harbor 0.24 · Claude Code's switch to another model on a safety flag was off: every reply came from Fable 5.2, none was refused92.6%89.8–95.4%96 of 129126 of 129129 of 129$2.63 list-price equivalent $338.77 in all781k in 22k out4.8 3 timed outPending$40 AA
2 Claude Mythos 5.1claude-mythos-5-1 · high effort · via Claude Code 2.1.294, Anthropic's own CLI, inside Harbor 0.24 · not yet in the atlas92.1%89.5–94.7%92 of 129127 of 129129 of 129$1.81 list-price equivalent $233.11 in all526k in 15k out3.592.1% no trial needed more time$50 LiteLLM list
3 Claude Fable 5.1 (high with fallback)claude-fable-5-1 · high effort · via Claude Code 2.1.294, Anthropic's own CLI, inside Harbor 0.2491.3%88.4–94.1%89 of 129128 of 129129 of 129$1.67 list-price equivalent $215.67 in all504k in 15k out3.491.3% no trial needed more time$50 AA
4 Devin SWE-2 (max)devin/swe-2-max · max effort · via Devin CLI 3000.11.3, Cognition's own CLI, on the host (no container) · not yet in the atlas89.0%85.5–92.4%78 of 129126 of 129129 of 129Not billed list-price equivalentNot reported8.2 5 timed outPending$0 Devin CLI, free
5 Claude Opus 5.5 (medium)openrouter/anthropic/claude-opus-5.5 · medium effort · mixed harness: some trials via Claude Code86.2%82.6–89.9%72 of 129124 of 129129 of 129$0.65 $83.26 in all400k in 15k out4.1 1 timed outPending$20 AA
6 Claude Opus 5.5 (high)openrouter/anthropic/claude-opus-5.5 · high effort85.6%81.8–89.4%66 of 129125 of 129129 of 129$0.82 $105.76 in all433k in 20k out5.1 2 timed outPending$20 AA
7 Muse Wonder Full (max)meta/muse-wonder-full-1.0 · max effort · via Muse Code 1.4.6, Meta's own CLI, on the host (no container) · not yet in the atlas85.4%81.8–89.0%63 of 129126 of 129129 of 129Not billed list-price equivalentNot reported10.2 5 timed outPendingNot quoted subscription only, no API price
8 Claude Sonnet 5.5 (xhigh)openrouter/anthropic/claude-sonnet-5.5 · xhigh effort · mixed harness: some trials via Claude Code84.4%80.2–88.6%73 of 129120 of 129129 of 129$1.23 $158.91 in all1.2M in 78k out15.0 26 timed out87.7% strict 78 of 129; 8 trials still timed out$10 AA
9 Claude Sonnet 5.5 (high)openrouter/anthropic/claude-sonnet-5.5 · high effort · mixed harness: some trials via Claude Code84.1%80.4–87.7%63 of 129126 of 129129 of 129$0.55 $71.49 in all671k in 20k out5.6 1 timed out84.5% strict 64 of 129; none timed out$10 AA
10 Hy4openrouter/tencent/hy4 · high effort · OpenRouter, Tencent provider only without fallbacks, max_tokens 65536 · not yet in the atlas84.1%80.1–88.1%61 of 129123 of 129128 of 129$0.21 $26.77 in all2.3M in 85k out19.1 63 timed outPending$1.2505 Tencent Cloud TokenHub pricing (https://www.tencentcloud.com/document/product/1300/78937; 50% limited-time discount until 31 Dec 2026, list $2.501)
11 GPT-6.1 Terra (Max)openrouter/openai/gpt-6.1-terra · max effort82.7%79.5–85.8%49 of 129126 of 129129 of 129$0.50 $65.04 in all1.0M in 85k out18.5 37 timed out82.7% strict 50 of 129; 20 trials still timed out$5 AA
12 Muse Wonder (max)meta/muse-wonder-1.0 · max effort · via Muse Code 1.4.6, Meta's own CLI, inside Harbor 0.24 · not yet in the atlas82.3%78.2–86.4%63 of 129124 of 129129 of 129$1.52 list-price equivalent $196.59 in all2.9M in 23k out4.782.3% no trial needed more time$20 Meta API list
13 Grok 4.7 (xhigh)openrouter/x-ai/grok-4.7 · xhigh effort81.8%78.2–85.3%49 of 129125 of 129128 of 129$0.65 $83.84 in all630k in 16k out5.481.8% no trial needed more time$6 AA
14 Kimi K3.1 Proopenai/k3.1-pro · max effort · via Terminus-2 with each turn's reasoning passed back, inside Harbor 0.24 · max was set explicitly: Kimi Code's model page gave max as K3.1 Pro's default on 23 October 2026, during the run, and high by 24 October81.5%78.1–85.0%44 of 129127 of 129129 of 129Not billed list-price equivalent509k in 22k out8.281.5% no trial needed more time$10 AA
15 MiniMax M3.1 Flashopenai/MiniMax-M3.1-Flash · the API ignores reasoning_effort · not yet in the atlas81.3%77.4–85.2%49 of 129126 of 129129 of 129Not billed list-price equivalent1.1M in 43k out8.7 7 timed outPendingNot quoted no pay-as-you-go price published yet; run on api.minimax.io with the owner's key
16 GPT-6 Astra (xhigh)openrouter/openai/gpt-6-astra · xhigh effort · mixed harness: some trials via Codex81.2%77.0–85.3%52 of 129126 of 129129 of 129$1.57 $202.73 in all377k in 15k out6.681.2% no trial needed more time$50 AA
17 Grok 4.7 (high)openrouter/x-ai/grok-4.7 · high effort81.2%77.7–84.8%50 of 129126 of 129129 of 129$0.58 $74.27 in all505k in 15k out5.381.2% no trial needed more time$6 AA
18 GPT-6.1 Terra (High) ★openrouter/openai/gpt-6.1-terra · high effort81.0%77.6–84.4%47 of 129129 of 129129 of 129$0.10 $12.8 in all322k in 9k out2.881.0% no trial needed more time$5 AA
19 Devin SWE-2 Flash (max)devin/swe-2-flash-max · max effort · via Devin CLI 3000.11.234, Cognition's own CLI, on the host in a bubblewrap sandbox (no container) · each trial ran in a bubblewrap sandbox that showed Devin only its own task copy, with an empty home and no MCP servers; the network stayed open: 2 trials ran Devin's web_search (3 queries), which found only unrelated projects and the public EvalMapBench page, no task text, tests or solutions; 14 of 129 trials hit the 20-minute budget, none after finishing its work (7 were waiting on a model reply, 7 still working) · not yet in the atlas80.9%76.2–85.5%57 of 129123 of 129129 of 129Not billed list-price equivalentNot reported5.4 14 timed out83.7% strict 60 of 129; 5 trials still timed out$0 Devin CLI, free
20 GPT-6.1 Sol (xhigh, ultrafast)openai/gpt-6.1-sol · xhigh effort · via Codex 0.161.0, OpenAI's own CLI, ultrafast service tier, inside Harbor 0.24 · not yet in the atlas80.9%76.7–85.2%55 of 129125 of 129129 of 129$0.28 list-price equivalent $35.51 in all500k in 11k out1.380.9% no trial needed more time$10 OpenAI list, standard tier
21 Muse Ember (Max)meta/muse-ember-1.0 · max effort · via Muse Code 1.4.14, Meta's own CLI, inside Harbor 0.2480.4%76.4–84.3%48 of 129126 of 129129 of 129$0.05 list-price equivalent $6 in all3.4M in 27k out3.380.4% no trial needed more time$0.5 AA
22 Pareto 26.10unbiased/pareto-26.10 · the API exposes no effort levels · not yet in the atlas80.4%76.6–84.3%40 of 129129 of 129129 of 129$0.12 list-price equivalent $16.06 in all316k in 13k out4.080.4% no trial needed more time$3.2 Unbiased API list
23 DeepSeek V4.1 Flash (high, 500k context)deepseek/deepseek-flash · high effort · not yet in the atlas80.1%76.1–84.1%44 of 129128 of 129128 of 129$0.09 $10.96 in all936k in 112k out10.0 17 timed outPending$1.2 AA
24 GPT-6.1 Sol (high)openrouter/openai/gpt-6.1-sol · high effort · mixed harness: some trials via Codex80.1%76.1–84.1%49 of 129126 of 129129 of 129$0.21 $26.86 in all291k in 9k out3.980.1% no trial needed more time$10 AA
25 GPT-6 Astra (high)openrouter/openai/gpt-6-astra · high effort · mixed harness: some trials via Codex80.0%76.0–84.0%48 of 129126 of 129129 of 129$1.20 $154.72 in all305k in 9k out4.380.0% no trial needed more time$50 AA
26 GPT-6.1 Sol (high, ultrafast)openai/gpt-6.1-sol · high effort · via Codex 0.161.0, OpenAI's own CLI, ultrafast service tier, inside Harbor 0.24 · not yet in the atlas79.6%75.2–83.9%51 of 129125 of 129128 of 129$0.19 list-price equivalent $24.07 in all321k in 6k out0.879.6% no trial needed more time$10 OpenAI list, standard tier
27 LongCat 2.5 (max)openai/LongCat-2.5 · max effort · via Terminus-2 with each turn's reasoning passed back, inside Harbor 0.24 · LongCat API with thinking enabled and a 262,144-token thinking budget and output limit · not yet in the atlas79.6%75.1–84.2%51 of 129124 of 129125 of 129$0.10 $12.57 in all2.0M in 48k out18.1 41 timed outPending$1.2 LongCat API pricing (https://longcat.chat/platform/docs/pricing/longcat-2.5; limited-time discount, 2026-11-02)
28 DeepSeek V4.1 Flash (max)deepseek/deepseek-flash · max effort · not yet in the atlas79.4%75.2–83.5%45 of 129126 of 129128 of 129$0.10 $13.08 in all995k in 146k out13.2 28 timed out79.5% strict 46 of 129; 22 trials still timed out$1.2 AA, 20 Oct 2026 snapshot (no Intelligence Index v5 row yet)
29 Claude Opus 5.5 (xhigh)openrouter/anthropic/claude-opus-5.5 · xhigh effort · mixed harness: some trials via Claude Code79.3%73.8–84.8%73 of 129117 of 129129 of 129$1.99 $256.55 in all877k in 63k out12.9 30 timed out86.3% strict 81 of 129; 13 trials still timed out$20 AA
30 DeepSeek V4.1 Flash (high, 1M context)deepseek/deepseek-flash · high effort · not yet in the atlas79.3%75.1–83.4%46 of 129127 of 129128 of 129$0.10 $13.43 in all801k in 117k out10.5 17 timed outPending$1.2 AA
31 Gemini 3.8 Flash (high)antigravity/gemini-3.8-flash · high effort · via Antigravity CLI 1.3.2, Google's own CLI, inside Harbor 0.2479.0%75.1–82.9%44 of 129127 of 129128 of 129$1.08 $139.86 in all6.5M in 66k out12.479.0% no trial needed more time$3.75 AA
32 Gemini 4 Argon (max)google/gemini-4-argon · max effort · via cli2027 0.2.4, the owner's own CLI, inside Harbor 0.24 · the 1M output budget was barely used: 4 of 3,375 requests wrote more than 32k tokens, the longest 77k · not yet in the atlas78.9%74.2–83.6%53 of 129126 of 129129 of 129$0.77 list-price equivalent $99.29 in all1.2M in 32k out8.9 6 timed outPending$10 Google API list (introductory)
33 Kimi K3 (max)openrouter/moonshotai/kimi-k3 · max effort · not yet in the atlas78.6%74.5–82.7%39 of 129120 of 129123 of 129$0.48 $62.03 in all278k in 19k out11.0 9 timed out78.6% strict 39 of 129; 7 trials still timed out$15 AA, 20 Oct 2026 snapshot (no Intelligence Index v5 row yet)
34 GLM-5.3-FlashXopenrouter/z-ai/glm-5.3-flashx · thinking on at the API default; no effort requested · not yet in the atlas78.5%74.3–82.7%40 of 129122 of 129127 of 129$0.16 $20.62 in all716k in 51k out10.6 17 timed out78.0% strict 39 of 129; 12 trials still timed out$1.25 OpenRouter list
35 GLM-5.3-Primeopenrouter/z-ai/glm-5.3-prime · thinking on at the API default; no effort requested · not yet in the atlas78.4%74.2–82.6%40 of 129126 of 129127 of 129$1.12 $144.49 in all784k in 48k out12.9 38 timed out78.3% strict 39 of 129; 23 trials still timed out$8.8 OpenRouter list
36 Muse Spark 1.3 Contributor (max)meta/muse-spark-1.3-contributor · max effort · via Muse Code 1.4.4, Meta's own CLI, inside Harbor 0.24 · not yet in the atlas77.9%73.7–82.1%40 of 129128 of 129129 of 129$0.02 list-price equivalent $2.95 in all2.7M in 29k out5.677.9% no trial needed more time$0.2 LiteLLM list
37 Ultramarine 1 Flash (max)paperinstruments/ultramarine-1-flash · max effort · requests went through a local proxy pacing them under the API's 60-a-minute limit · not yet in the atlas77.8%73.6–82.0%38 of 129124 of 129128 of 129$0.10 list-price equivalent $12.2 in all1.2M in 66k out10.3 19 timed outPending$0.5 Paper Instruments API list
38 Qwen3.8 27B (high, Cerebras)cerebras/qwen-3.8-27b · high effort · not yet in the atlas77.7%73.8–81.7%39 of 129124 of 129124 of 129$1.56 $201.47 in all1.2M in 91k out10.4 16 timed outPending$1.49 OpenRouter list (Cerebras endpoint)
39 DeepSeek V4.1 Flash 1017 (Max)deepseek/deepseek-flash · max effort77.4%72.7–82.1%52 of 129123 of 129126 of 129$0.09 $11.58 in all927k in 126k out10.7 11 timed outPending$1.2 AA
40 DeepSeek V4.1 Flash (high, 200k context)deepseek/deepseek-flash · high effort · not yet in the atlas77.1%72.8–81.4%36 of 129126 of 129127 of 129$0.09 $11.25 in all933k in 116k out10.5 21 timed outPending$1.2 AA
41 Mercury 3 (high)openrouter/inception/mercury-3 · high effort · OpenRouter, Inception provider only without fallbacks, max_tokens 131072 · not yet in the atlas77.0%72.6–81.3%41 of 129122 of 129128 of 129$0.04 $4.93 in all608k in 70k out4.377.0% no trial needed more time$0.35 OpenRouter list for the Inception endpoint (50% launch discount, 2026-11-06; Inception list $0.70)
42 Claude Haiku 5.5 (Xhigh)openrouter/anthropic/claude-haiku-5.5 · xhigh effort · mixed harness: some trials via Claude Code75.9%71.0–80.9%48 of 129118 of 129129 of 129$0.12 $15.12 in all701k in 111k out12.2 18 timed out78.5% strict 50 of 129; 7 trials still timed out$0.5 AA
43 Step 5 Previewopenrouter/stepfun/step-5-preview · max effort73.0%68.4–77.6%34 of 129120 of 129125 of 129$0.38 $48.91 in all1.3M in 71k out17.1 23 timed out75.1% strict 33 of 129; 5 trials still timed out$2.7 AA
44 Beam 1.0reflection/Beam-1.0-501B-A23B · max effort · via Terminus-2 with each turn's reasoning passed back, inside Harbor 0.2471.1%67.0–75.3%30 of 129122 of 129125 of 129$2.06 $266.3 in all1.5M in 36k out4.4 7 timed outPending$1.2 AA
45 Nemotron 3.5 Super VLopenrouter/nvidia/nemotron-3.5-super-vl54.7%49.6–59.8%9 of 129109 of 129117 of 129$0.08 $10.13 in all546k in 28k out3.0 2 timed outPending$0.5 AA
46 Qwen3.8 Max Prime (high)openrouter/qwen/qwen3.8-max-prime · high effort · not yet in the atlas46.6%39.6–53.6%26 of 12986 of 129125 of 129$1.64 $211.38 in all798k in 86k out20.7 125 timed out69.0% strict 59 of 129; 73 trials still timed out$12 OpenRouter list
47 Qwen3.8 Max Prime (xhigh)openrouter/qwen/qwen3.8-max-prime · xhigh effort · not yet in the atlas44.3%36.9–51.8%29 of 12979 of 129125 of 129$1.61 $208.06 in all799k in 83k out20.8 126 timed out65.6% strict 56 of 129; 79 trials still timed out$12 OpenRouter list
48 Ornith 1.5 122B (high)openai/ornith-ai/Ornith-1.5-122B · high effort · every one of the 129 trials hit the 20-minute limit; replies from RunInfra often took minutes, so the score reflects the provider's speed as much as the model · not yet in the atlas0.8%0.0–2.3%0 of 1295 of 129129 of 129Not billed list-price equivalent376k in 15k out14.7 129 timed outPendingNot quoted no per-token price found for RunInfra; run with the owner's key
49 gpt-oss-120b (High)openrouter/openai/gpt-oss-120b · high effort · the baseline: its checks are set aside0.0%0.0–0.0%1 of 12952 of 129109 of 129$0.22 $28.15 in all261k in 26k out4.2 14 timed out0.4% strict 1 of 129; 10 trials still timed out$0.6 AA

Controls. The reference solution passes every check on 43 of 43 tasks in the same containers; the untouched repository meets 0% of the detail checks and keeps 43 of 43 regression sets. The scale has a top and a bottom: doing nothing scores nothing.

02 / where the detail was stated

The further from the ticket, the more is missed.

Every detail check is tagged with where a careful engineer could have learnt the obligation. Ticket: it is in the ticket. Rule: a sentence of CONTRIBUTING that applies. Code: a docstring, a document or an ADR elsewhere in the repository. Pattern: written nowhere, but every neighbour does it.

Detail checks met by tier beyond the baseline, every trial counted · 27 ticket, 125 rule, 81 code and 51 pattern checks per attempt.
ModelTicket 27 checks · stated in the ticketRule 125 checks · a rule in CONTRIBUTINGCode 81 checks · written elsewhere in the repositoryPattern 51 checks · only a pattern the code follows
Claude Fable 5.2 (high with fallback)96% 78 of 8197% 362 of 37591% 221 of 24382% 125 of 153
Claude Mythos 5.196% 78 of 8195% 357 of 37593% 226 of 24379% 121 of 153
Claude Fable 5.1 (high with fallback)98% 79 of 8196% 359 of 37593% 226 of 24375% 114 of 153
Devin SWE-2 (max)94% 76 of 8191% 343 of 37587% 212 of 24376% 117 of 153
Claude Opus 5.5 (medium)96% 78 of 8194% 354 of 37584% 205 of 24364% 98 of 153
Claude Opus 5.5 (high)95% 77 of 8194% 352 of 37585% 206 of 24362% 95 of 153
Muse Wonder Full (max)98% 79 of 8188% 331 of 37588% 214 of 24371% 108 of 153
Claude Sonnet 5.5 (xhigh)89% 72 of 8187% 325 of 37584% 204 of 24365% 100 of 153
Claude Sonnet 5.5 (high)96% 78 of 8192% 344 of 37582% 200 of 24361% 93 of 153
Hy498% 79 of 8189% 332 of 37584% 204 of 24370% 107 of 153
GPT-6.1 Terra (Max)96% 78 of 8190% 338 of 37580% 195 of 24359% 91 of 153
Muse Wonder (max)84% 68 of 8190% 338 of 37580% 194 of 24363% 97 of 153
Grok 4.7 (xhigh)100% 81 of 8192% 346 of 37579% 191 of 24353% 81 of 153
Kimi K3.1 Pro94% 76 of 8192% 345 of 37580% 195 of 24356% 86 of 153
MiniMax M3.1 Flash91% 74 of 8192% 344 of 37577% 187 of 24358% 88 of 153
GPT-6 Astra (xhigh)96% 78 of 8191% 342 of 37582% 199 of 24352% 79 of 153
Grok 4.7 (high)100% 81 of 8191% 340 of 37580% 195 of 24351% 78 of 153
GPT-6.1 Terra (High)100% 81 of 8190% 339 of 37581% 197 of 24348% 73 of 153
Devin SWE-2 Flash (max)96% 78 of 8187% 326 of 37581% 198 of 24364% 98 of 153
GPT-6.1 Sol (xhigh, ultrafast)95% 77 of 8191% 341 of 37581% 197 of 24356% 85 of 153
Muse Ember (Max)95% 77 of 8187% 326 of 37581% 197 of 24359% 91 of 153
Pareto 26.10100% 81 of 8191% 342 of 37583% 201 of 24346% 71 of 153
DeepSeek V4.1 Flash (high, 500k context)100% 81 of 8188% 330 of 37578% 190 of 24355% 84 of 153
GPT-6.1 Sol (high)96% 78 of 8191% 342 of 37580% 195 of 24348% 73 of 153
GPT-6 Astra (high)96% 78 of 8190% 337 of 37582% 199 of 24347% 72 of 153
GPT-6.1 Sol (high, ultrafast)95% 77 of 8189% 335 of 37581% 197 of 24352% 80 of 153
LongCat 2.5 (max)86% 70 of 8187% 325 of 37578% 189 of 24359% 91 of 153
DeepSeek V4.1 Flash (max)91% 74 of 8187% 327 of 37577% 188 of 24361% 93 of 153
Claude Opus 5.5 (xhigh)90% 73 of 8179% 298 of 37577% 186 of 24365% 99 of 153
DeepSeek V4.1 Flash (high, 1M context)100% 81 of 8188% 329 of 37576% 184 of 24357% 87 of 153
Gemini 3.8 Flash (high)99% 80 of 8186% 324 of 37580% 194 of 24360% 92 of 153
Gemini 4 Argon (max)100% 81 of 8182% 307 of 37583% 202 of 24363% 97 of 153
Kimi K3 (max)90% 73 of 8188% 330 of 37572% 175 of 24352% 80 of 153
GLM-5.3-FlashX81% 66 of 8189% 333 of 37579% 191 of 24356% 85 of 153
GLM-5.3-Prime81% 66 of 8187% 327 of 37576% 185 of 24357% 87 of 153
Muse Spark 1.3 Contributor (max)90% 73 of 8184% 314 of 37581% 197 of 24359% 90 of 153
Ultramarine 1 Flash (max)79% 64 of 8189% 334 of 37575% 183 of 24355% 84 of 153
Qwen3.8 27B (high, Cerebras)79% 64 of 8187% 327 of 37577% 187 of 24351% 78 of 153
DeepSeek V4.1 Flash 1017 (Max)100% 81 of 8188% 330 of 37572% 175 of 24353% 81 of 153
DeepSeek V4.1 Flash (high, 200k context)100% 81 of 8187% 325 of 37576% 184 of 24352% 79 of 153
Mercury 3 (high)84% 68 of 8190% 338 of 37569% 167 of 24352% 79 of 153
Claude Haiku 5.5 (Xhigh)91% 74 of 8183% 313 of 37576% 184 of 24347% 72 of 153
Step 5 Preview86% 70 of 8182% 307 of 37569% 167 of 24346% 70 of 153
Beam 1.080% 65 of 8181% 304 of 37566% 161 of 24346% 70 of 153
Nemotron 3.5 Super VL72% 58 of 8163% 238 of 37553% 128 of 24334% 52 of 153
Qwen3.8 Max Prime (high)43% 35 of 8138% 144 of 37547% 114 of 24339% 59 of 153
Qwen3.8 Max Prime (xhigh)36% 29 of 8136% 135 of 37540% 97 of 24337% 57 of 153
Ornith 1.5 122B (high)0% 0 of 811% 2 of 3750% 0 of 2430% 0 of 153
gpt-oss-120b (High)0% 0 of 810% 0 of 3750% 0 of 2430% 0 of 153

02b / task by task

Every model on every task.

One square per model and task. Hover a square for the attempts behind it. All 43 tasks of EvalMapBench 6.7 are ranked. A dot marks a cell run through the maker’s own CLI (Claude Code or Codex, on a subscription) rather than the row’s own harness.

solved: every check passed in at least two of three attemptsrun through Claude Code or Codexnot solvednot run

solvedgateway-stats-percentilesgateway-frame-limitfairqueue-requeuefairqueue-export-jsongateway-channel-limitgateway-close-reasongateway-damaged-fixturefairqueue-enqueue-delayfairqueue-sim-p95fairqueue-admin-purgegateway-delete-key-limitgateway-capture-bytesgateway-verify-frozen-toolgateway-dump-capturegateway-conflict-equal-valuefairqueue-finished-atfairqueue-sim-crashfairqueue-lease-defaultfairqueue-fail-retryfairqueue-releasegateway-fuzz-kitfairqueue-priority-agingfairqueue-sim-replaygateway-verify-archivegateway-unfilled-settlementfairqueue-lease-kitgateway-metrics-seriesgateway-rename-compatgateway-session-settinggateway-compat-exportgateway-tool-optionfairqueue-inspect-tenantfairqueue-tenant-pausefairqueue-signature-fieldfairqueue-adr-supersedefairqueue-sim-tenantsgateway-canonical-capturefairqueue-already-settledfairqueue-ready-forecastfairqueue-claim-receiptfairqueue-history-checkgateway-divergence-witnessfairqueue-migrate-chain
Claude Fable 5.2 (high with fallback)34
Claude Mythos 5.131
Claude Fable 5.1 (high with fallback)31
Devin SWE-2 (max)27
Claude Sonnet 5.5 (xhigh)26
Claude Opus 5.5 (medium)24
Claude Opus 5.5 (high)23
Claude Opus 5.5 (xhigh)23
Muse Wonder Full (max)21
Claude Sonnet 5.5 (high)21
Muse Wonder (max)21
Devin SWE-2 Flash (max)20
Hy419
MiniMax M3.1 Flash18
GPT-6.1 Sol (xhigh, ultrafast)18
DeepSeek V4.1 Flash 1017 (Max)18
GPT-6 Astra (xhigh)17
GPT-6.1 Sol (high)17
GPT-6.1 Sol (high, ultrafast)17
LongCat 2.5 (max)17
Gemini 4 Argon (max)17
Grok 4.7 (xhigh)16
Grok 4.7 (high)16
Muse Ember (Max)16
GPT-6 Astra (high)16
DeepSeek V4.1 Flash (max)16
Claude Haiku 5.5 (Xhigh)16
GPT-6.1 Terra (Max)15
GPT-6.1 Terra (High)15
DeepSeek V4.1 Flash (high, 500k context)15
Mercury 3 (high)15
Kimi K3.1 Pro14
DeepSeek V4.1 Flash (high, 1M context)14
Gemini 3.8 Flash (high)14
GLM-5.3-FlashX13
Pareto 26.1012
Kimi K3 (max)12
GLM-5.3-Prime12
Muse Spark 1.3 Contributor (max)12
Ultramarine 1 Flash (max)12
Qwen3.8 27B (high, Cerebras)12
Step 5 Preview12
DeepSeek V4.1 Flash (high, 200k context)11
Beam 1.010
Qwen3.8 Max Prime (high)7
Qwen3.8 Max Prime (xhigh)7
Nemotron 3.5 Super VL3
Ornith 1.5 122B (high)0
gpt-oss-120b (High)0

03 / from ticket to score

How a result gets here.

  1. Start with real work

    Two small private services of one developer, with their docs, decision records, frozen release files and conventions; the tickets are maintenance he would file himself.

  2. Write the ticket tersely

    One sentence of what the feature does. Everything else the change must respect is already in the repository, and every check records where.

  3. Run 3 attempts

    Every model runs through the same open harness (Harbor 0.24 terminus-2), 20 minutes per attempt, in a throwaway container, so the spread between rows is the model.

  4. Verify as a black box

    A hidden verifier probes the changed repository and never reads the model’s reasoning. The reference solution passes every check; the untouched repository fails every core and detail check. Only these aggregates are published.

04 / read with care

What these numbers cannot tell you.