Small tests, run here.

Aion 3.5 against the GLM it is built on: a long role-play

Aion Labs sells Aion 3.5 as a role-play system built on GLM, so the question is whether it keeps a long game straighter than GLM-5.3 does by itself.

The same six games in Russian

Played first, before the tests moved to English: the same events and the same questions, in Russian. Fourteen memory questions a model were voided there because the scenario generator, not the model, made their answer ambiguous — the same fourteen for every model — so a model answers 170 memory questions and 73 hard ones.

What it says

• Keeping track of a game is solved at this length (about 25,000 tokens by the last turn): every model answers nearly every memory question right, in both languages. Aion's wrapper around several models does not cost it consistency, and does not buy any either.
• In English the hard questions at the end do not separate them (+0.95 to +0.97). In Russian Aion 3.5 came last on them (+0.86), so that gap was about the language, not about keeping a game. Every miss was read by hand; none is the grader's.
• Rules are where English differs: every model writes longer in English, and the 250-word limit is broken on 15–41% of turns, most by GLM-5.3. Aion 3.5 Mini keeps the rules best in both languages; GLM-5.3-Flash narrates the player's own actions (“you take the key”) on 12% of its English turns.
• Aion 3.5 Mini repeats itself most: 6.9% of its four-word runs come back from earlier turns, against about 2% for the others. It is also the slowest.
• GLM-5.3-Flash is as good as any of them on memory at a tenth or less of Aion 3.5's price.
• Six games a language are a small sample: differences of 0.02–0.03 on the indices are noise.

How it was run

• Six generated scenarios of 36 turns each, in English. The player's turns are written in advance, so every model plays the same game, and every change to the world — who holds what, who is hurt, what was paid — is made by the player, so the true state of the world is known at every turn before any model plays it. The same six games were also played in Russian, from the same seeds: the same events and questions in the other language.
• Memory: after turns 12, 24 and 36 an out-of-game question is asked on a copy of the conversation, never in the game itself, about the state of the world and the town the system prompt describes. +1 for a right answer, −1 for a wrong one, 0 for “I don’t know”, as in AA-Omniscience. The first value an answer names is the one scored.
• Hard questions: once, after the last turn, questions about the whole game rather than its present — who held an item before its last hand-over, how many times it changed hands, totals paid and earned, who was hurt first — and look-ups across the town description. Scored the same way.
• Rules of the table: eight rules the system prompt sets — format, length, language and the speech habits of named characters — each checked by a script on every turn. A clean turn breaks none of them.
• Prose: distinct word pairs within windows of the narration, and how often a four-word sequence comes back from an earlier turn.
• Every model through OpenRouter with prompt logging refused (data_collection: deny); Aion offers no zero-retention route. Up to 8,000 output tokens a turn.
• The whole test cost $13.93 in API calls.

Recraft V4.1 Flash against Recraft V4.1: exact text and composition

Flash is sold as the fast, cheap V4.1, so the question is what the fifth of the price gives up.

What it says

• V4.1 writes long strings and Cyrillic better and places objects more reliably. On one or two words, on a single colour and on two objects side by side the two do not differ.
• Flash gives up little for a fifth of the price and a third of the time: a passing image costs $0.0085 against $0.040.
• Counting is not a difference between them: both fail it as measured for the detector's sake, and on audit both count nearly every image right.
• 32 prompts per part: a difference of one or two prompts is inside the noise.

How it was run

• 64 generated prompts, four images each at 1024×1024 through the Vercel AI Gateway, 256 scored images a model.
• Text: 32 prompts ask for a sign, label, cover or poster carrying an exact string of one to eleven words, half in Latin and half in Cyrillic script, some with a number. An image passes when the string is on it exactly, ignoring case. Read by two OCR engines (EasyOCR, PaddleOCR) and a vision model asked to transcribe without seeing the target. OCR misses decorative lettering and the vision model corrects misspellings toward real words — of nine passes only it saw, seven were right on inspection — so the share is given as a range: passed by a classical OCR (floor), by any reader (ceiling), and the estimate that counts the vision model's passes at seven in nine. The same engines read the same strings drawn in a plain font 32 times out of 32.
• Composition: 32 prompts in the manner of GenEval — counts, two objects, a colour, two colours bound to two objects, and position — checked with a Mask2Former detector; a colour passes when either the pixel colour of the object's mask or CLIP on the masked object names it.
• The detector undercounts: looked at one by one, about 13 of the 16 failed counting images show the right number of objects — cats, stone benches and vases the detector did not see. Counting is given both as measured and as audited.
• The whole test cost $10.92 in API calls.

Ten image models: text in three scripts, clocks, hands, a full glass

Arena ratings say which image a voter liked. This asks which models draw what was asked: an exact string, a stated time, a count of fingers, a colour.

What it says

• GPT Image 2 and GPT Image 2.5 Sunburst are the only models right on nearly everything: exact text in all three scripts, every clock, every hand, every glass, and the truest colour. Sunburst does it at a quarter of GPT Image 2's price and in less than half the time.
• Muse Image is the cheapest here at $0.01 an image and writes text as well as the OpenAI pair, but misses one clock in six and refuses more than any model but MAI.
• Clocks split the field in two: seven models set the stated time on 83–100% of clocks; MAI-Image-2.6, FLUX.2 Max and Recraft V4.1 never did, drawing 10:10 or hands at random.
• A glass full to the brim is still a trap: Grok Imagine 2.0 never drew one, Recraft once in twelve, Gemini and FLUX three times.
• Grok Imagine 2.0 has almost no variety: asked eight times for the same open prompt, it draws nearly the same picture every time.
• The scene checklist barely separates them (89–99%); the hard block, text outside Latin script and colour do.
• Five models refused the same one harmless scene; MAI-Image-2.6 and GPT Image 2 refused a second. Seedream, Gemini, Qwen, Grok and Recraft refused nothing.
• 12 to 32 images a part: differences of five to ten points are inside the noise.

How it was run

• 110 prompts built from specifications, so what a right image holds is known before any model draws it; English prompts, four images each (eight for the variety prompts), 456 images a model and 4,560 in all, at 1:1 and medium quality where the provider has a setting.
• Text: 28 prompts ask for a sign, label or cover carrying an exact string — Latin (1–12 words, some with prices and numbers), Cyrillic, and Chinese. An image passes when the string is on it exactly, ignoring case and spacing. Read by two OCR engines (EasyOCR, PaddleOCR) and by a vision model asked to transcribe without seeing the target. Given as three figures: read by a classical OCR, read by any reader, and the final share after every pass only the vision model saw was checked by hand (126 of 137 were right). Layout: 8 posters with a title, subtitle and date in a stated order, read the same way.
• Scene: 40 prompts with a checklist of yes/no questions in the manner of DSG — exact counts, three objects with bound colours, spatial relations, an object the prompt forbids, and scenes against common sense. Answered by two vision models of different families, Gemini 3.8 Flash and Claude Sonnet 5.
• Hard: analog clocks at a stated time (right within three minutes), a hand with one to five fingers raised and five in all, a wine glass filled to the brim. The judges misread these badly — on clocks one judge failed 73 of the images a person reads as right — so every clock image, every image the two judges disagreed on and every image both failed was checked by hand, blind to the model, with a sample of the images both passed (all of them right). The table gives the audited shares.
• Colour: a flat background in an exact hex colour; the median CIEDE2000 distance of the border pixels from it (under 2 is hard to see). Variety: four open prompts sampled eight times each; the mean pairwise CLIP similarity of the eight (lower is more varied).
• Refusals: 32 images of ordinary, harmless scenes of the kind a cautious filter may stop. A refusal is a result, not a retry. The refusal column counts every refusal across the whole test.
• Every model through OpenRouter with provider data collection refused, except Muse Image, which went through the Vercel AI Gateway. Seconds are the median per image as the route returned it; dollars are what the route billed.
• The whole test cost $209.17 in API calls.