Six models, fifty problems, four clouds.
The most expensive model in the panel costs 3.9× as much per correct answer — that is cost divided by accuracy, not the per-token price — as one that scores within two points of it. The cheapest is 8× cheaper again and gets a third of the questions right. This is the teardown we run on a client’s own prompts, shown on fifty problems written for this page.
Measured 5 September 2026 · 300 live API calls per run · 50 original problems · 6 models · 4 clouds · no cached or fixture responses · machine-readable version · run receipts
What each model got right, and what a right answer costs
Fifty grade-school multi-step word problems, each with a single unambiguous numeric answer, written for this page rather than pulled from a public benchmark — every public set is in the training data of every model here and would measure memory as much as arithmetic. Costs are estimates from published per-token prices; see method.
| Model | Cloud | Accuracy | Correct | $ / 1M answers | $ / 1M correct | Median |
|---|---|---|---|---|---|---|
| GPT-5.5 | Azure | 88% | 44 / 50 | 45.63 | 51.85 | 1.5s |
| Gemini 3.8 Flash | Google Vertex | 86% | 43 / 50 | 11.45 | 13.32 | 3.7s |
| Qwen3 Next 80B | Google Vertex | 56% | 28 / 50 | 4.74 | 6.77 | 0.5s |
| DeepSeek V3.2 | Azure | 50% | 25 / 50 | 7.27 | 14.54 | 0.6s |
| Llama 3.3 70B | AWS Bedrock | 46% | 23 / 50 | 20.09 | 43.67 | 0.3s |
| Nova Lite | AWS Bedrock | 32% | 16 / 50 | 1.94 | 6.06 | 0.5s |
Qwen3 Next returned nothing on 10 of its 50 calls; accuracy is scored over all 50, so those count against it, because in production an empty answer is a wrong answer. Cost per correct answer divides cost by accuracy, so a model that is cheap per call but often wrong is not cheap.
The cheap-model advantage reverses when the task gets harder
On an easier twenty-item set run the day before, Nova Lite matched GPT-5.5 on accuracy at a fraction of the price. On these fifty multi-step problems it gets 32% and GPT-5.5 gets 88%. Same models, same week, opposite conclusion.
That is not a defect in either model. It is the reason a benchmark you did not write cannot tell you which model to buy: the answer depends on how hard your own prompts are, and only your own prompts can say. The number a trivial benchmark produces is not wrong, it is about a different job. We run this measurement on the prompts a client actually sends, and the table is different every time.
Our own instrument was wrong twice before these numbers were right
The answer key had two errors
We wrote the fifty problems and their keys by hand. Before scoring any model, we ran the key through our own answer-key audit — two independent programs compute each answer from the question text, and a key is flagged only where both ran, agreed with each other, and disagreed with the key. It found two keys we had got wrong and one question that could be read two ways. All three were fixed before a model saw them; the corrected keys are in the receipts.
The comparator punished rounding
An earlier version of the scorer marked a model wrong for answering 7.17 when the exact value was 7.1666… A number written to two decimals claims accuracy to half a unit in its last place, and the scorer now honours that. The bug had been silently understating every model’s accuracy; we say so here because a measurement that hides its own corrections is not one you should trust.
We ran it twice. Here is the second run beside the first.
Another 300 live calls, same problems, same prompts. The top four finished in the same order both times; the two weakest moved.
| Model | Run A | Run B | Change |
|---|---|---|---|
| GPT-5.5 | 88% | 88% | — |
| Gemini 3.8 Flash | 86% | 86% | — |
| Qwen3 Next 80B | 56% | 56% | — |
| DeepSeek V3.2 | 50% | 50% | — |
| Llama 3.3 70B | 46% | 52% | +6 pts |
| Nova Lite | 32% | 36% | +4 pts |
The table at the top of this page is Run A. Run B is printed here rather than folded into an average so the size of run-to-run movement is visible: two to three problems out of fifty for the weaker models, none for the stronger ones.
How this was measured, and what it does not show
What we did
Problems. 50 original multi-step word problems with a single numeric answer, written for this page and published in the receipts so anyone can re-run them. Future issues use fresh problems.
Calls. Six models on four clouds — Azure, Google Vertex, AWS Bedrock and one more — 300 live calls per run, two runs, identical prompt, no cached answers. Every call records the model the provider reports serving.
Scoring. A numeric answer is correct if it matches the key to the precision it was written to; integers and exact fractions must match exactly. An empty or unparseable answer is wrong.
What it does not show
Prices are estimates. Published per-token prices, tokens estimated at four characters per token, for the prompts and answers actually exchanged. Estimates, not invoices.
The limits. One task family, fifty items, two runs, one day. This measures multi-step arithmetic under our prompt rules. It is not a general capability ranking and we would not defend it as one — the previous section is the reason.
What changes the answer. Your prompts. The same six models on a support-ticket classification task, or a contract-extraction task, produce a different table. That table is the product.
If you think a number here is wrong, the useful thing to send us is which problem and which model — we will rerun it and print what happens.