VERDICT / PROJECT BENCHMARK10-TASK ENGINEERING DIAGNOSTIC SUITE · EMPIRICAL AUDIT
Real Code. Real Measurements.
Comparing Local Apple Silicon Execution with Frontier Cloud Models.
We evaluated 7 local and frontier AI models across 10 deterministic engineering tasks (33 unit test assertions). Every run was executed live on Apple Silicon (M4 Max 128GB) and cloud endpoints with monotonic timers, provider-reported token usage where the provider returned it (GLM mostly did not), and in-process Python test execution.
11-Model Empirical Leaderboard
Measured from live execution receipts on 2026-10-15 across identical unit test assertions.
| Model | Tier | Concurrency | Pass Rate | Single-Stream Speed | Suite Wall Time | Provider Cost | Empirical Observation |
|---|---|---|---|---|---|---|---|
| Verdict Tool Terse v5 (Local, c=1)TARGET LOCAL | Local Workhorse (Serial) | c=1 (serial) | 10/10 (100%) | 90.5 tok/s | 67.5s | $0.0000 | 100% assertions passed; 90.5 tok/s single-stream on local Apple Silicon at zero provider cost. |
| Verdict Tool Terse v5 (Local, c=4) | Local Workhorse (Parallel) | c=4 (parallel) | 10/10 (100%) | 23.6 tok/s | 44.1s | $0.0000 | 100% assertions passed; 44.1s parallel runtime (c=4) on local Apple Silicon at zero provider cost. |
| Claude Haiku 5.5 (Cloud) | Frontier Lightweight Cloud | c=4 (parallel) | 10/10 (100%) | 94.8 tok/s | 7.7s | $0.0029 | 10/10 passed in 7.7s wall time; 3.5x parallel speedup at just 0.29¢ (less than a third of a cent). |
| Claude 4.5 Sonnet (Cloud) | Frontier Cloud Standard | c=4 (parallel) | 10/10 (100%) | 60.5 tok/s | 7.7s | $0.0270 | 10/10 passed; 60.5 tok/s per stream (189.5 tok/s aggregate across 4 streams) at 2.7¢. |
| Gemini 4 Argon (Cloud) | Frontier Multimodal Reasoning | c=4 (parallel) | 10/10 (100%) | 12.7 tok/s | 33.4s | $0.0284 | 10/10 passed in 33.4s wall time; 36.7 aggregate tok/s at 2.8¢ ($2/$10 official Migration 291 catalog rate). |
| Claude Sonnet 5.5 (Cloud) | Frontier Flagship Cloud | c=4 (parallel) | 10/10 (100%) | 91.9 tok/s | 8.7s | $0.0308 | 10/10 passed in 8.7s wall time; 300 aggregate tok/s at 3.1¢ ($2/$10 official catalog rate). |
| GLM-5.3 High Thinking (Cloud) | Cloud Architecture Specialist | c=2 (parallel) | 10/10 (100%) | Unmeasured | 49.3s | Unmeasured (provider omitted token usage) | 10/10 passed; gateway omitted token usage on 7/10 stories (the other 3 hit the 1024-token cap), so speed and cost are unmeasured. |
| Qwen3 Coder 30B (Local) | High-Speed Local MoE | c=4 (parallel) | 9/10 (90%) | 40.0 tok/s | 7.7s | $0.0000 | 7.7s total suite runtime via 4 parallel streams; missed 1 topological sort story. |
| Claude Opus 5.5 (Cloud) | Frontier Flagship Reasoning | c=4 (parallel) | 9/10 (90%) | 81.5 tok/s | 9.3s | ~$0.0886 (est.) | 9/10 passed; 81.5 tok/s per stream; failed story 4 empty content; ~8.9¢ per suite (estimated at Opus 4.5 list rates). |
| GPT-OSS 20B (Local) | Open-Weights BCB Leader | c=4 (parallel) | 8/10 (80%) | 38.4 tok/s | 9.4s | $0.0000 | 9.4s total suite runtime; failed stub regex check & reversed topological order. |
| GLM-5.3-Flash (Cloud) | Inexpensive Cloud Baseline | c=2 (parallel) | 8/10 (80%) | Unmeasured | 52.9s | Unmeasured (provider omitted token usage) | 8/10 passed; syntax formatting errors on stories 7 & 9; gateway omitted token usage on 8/10 (the other 2 hit the 1024-token cap), so speed and cost are unmeasured. |
* Speed & Concurrency Note: Tool Terse ran with concurrency=1 (sequential stream), generating at 90.5 tok/s per stream and completing in 67.5s. Claude, Qwen, and GPT-OSS ran with concurrency=4 (parallel streams), completing the entire suite in 7.7–9.4s of wall time while generating at 38–82 tok/s per stream. Qwen and GPT-OSS shared one local GPU across their 4 streams, so their per-stream speed is not directly comparable to Tool Terse running alone. GLM token telemetry was omitted by the gateway upstream, so GLM speed and cost are marked unmeasured.
Explore the 10 Progressive Stories
Click any story below to view the task, requirements, live scorecard across all 7 models, and the generated Python code.
Format Compact Number Range
4 Automated Test AssertionsAs a developer, I need to format integer sequences into compact bracketed strings like [1,2,3] or [] when empty.
Write a python function `format_range(values: list[int]) -> str`: - Returns numbers formatted in brackets separated by commas without spaces, e.g. [1,2,3]. - Empty list returns '[]'. - Negative numbers must be preserved, e.g. [-5,0,5] -> '[-5,0,5]'. Output only the python function inside a ```python ``` code block.
| Model | Status | Concurrency | Duration | Tokens (In/Out) | Stream Speed | Cost | Diagnostic / Error Details |
|---|---|---|---|---|---|---|---|
| Verdict Tool Terse v5 (Local, c=1)TARGET | ✅ PASS | c=1 | 2.81s | 165/235 | 83.6 tok/s | $0.0000 | 4/4 assertions passed |
| Verdict Tool Terse v5 (Local, c=4) | ✅ PASS | c=4 | 3.23s | 0/61 | 18.9 tok/s | $0.0000 | 4/4 assertions passed |
| Claude Haiku 5.5 (Cloud) | ✅ PASS | c=4 | 1.75s | 1442/48 | 27.4 tok/s | $0.0002 | 4/4 assertions passed |
| Claude Sonnet 5.5 (Cloud) | ✅ PASS | c=4 | 1.59s | 232/48 | 30.1 tok/s | $0.0009 | 4/4 assertions passed |
| Claude 4.5 Sonnet (Cloud) | ✅ PASS | c=4 | 1.49s | 167/57 | 38.2 tok/s | $0.0014 | 4/4 assertions passed |
| Claude Opus 5.5 (Cloud) | ✅ PASS | c=4 | 2.12s | 232/48 | 22.6 tok/s | $0.0024 | 4/4 assertions passed |
| Qwen3 Coder 30B (Local) | ✅ PASS | c=4 | 1.96s | 158/36 | 18.3 tok/s | $0.0000 | 4/4 assertions passed |
| GPT-OSS 20B (Local) | ✅ PASS | c=4 | 3.35s | 219/48 | 14.3 tok/s | $0.0000 | 4/4 assertions passed |
| GLM-5.3 High Thinking (Cloud) | ✅ PASS | c=2 | 2.49s | Unreported | — | — | 4/4 assertions passed |
| GLM-5.3-Flash (Cloud) | ✅ PASS | c=2 | 4.08s | Unreported | — | — | 4/4 assertions passed |
| Gemini 4 Argon (Cloud) | ✅ PASS | c=4 | 10.88s | 809/39 | 3.6 tok/s | $0.0020 | 4/4 assertions passed |
def format_range(values: list[int]) -> str:
return '[' + ','.join(map(str, values)) + ']'Auditable Run Receipts & Ground Truth
All metrics on this page are compiled directly from raw execution JSON receipts generated by scripts/benchmarks/run_ten_stories.py. Pricing is derived from cloud/config/verdict_external_frontier_pricing.json; that file has no Opus 5.5 entry, so the Opus 5.5 cost is an estimate at Opus 4.5 list rates ($5/$25 per million tokens).
artifacts/benchmarks/ten_stories_toolterse_v5_c4.json— Verdict Tool Terse v5 MLX (10/10, c=4 parallel, 44.1s wall time, $0.00)artifacts/benchmarks/ten_stories_claude_haiku_5_5_c4.json— Claude Haiku 5.5 (10/10, c=4 parallel, 7.7s wall time, $0.0029)artifacts/benchmarks/ten_stories_claude_sonnet_5_5_c4.json— Claude Sonnet 5.5 (10/10, c=4 parallel, 8.7s wall time, $0.0308)artifacts/benchmarks/ten_stories_20261015T223540Z.json— Verdict Tool Terse v5 MLX (10/10, c=1 serial, 90.5 tok/s stream, $0.00)artifacts/benchmarks/ten_stories_20261015T224258Z.json— Qwen3 Coder 30B (9/10, c=4, 40.0 tok/s stream, $0.00)artifacts/benchmarks/ten_stories_20261015T224337Z.json— GPT-OSS 20B (8/10, c=4, 38.4 tok/s stream, $0.00)artifacts/benchmarks/ten_stories_20261015T224624Z.json— Claude 4.5 Sonnet (10/10, c=4, 60.5 tok/s stream, $0.027)artifacts/benchmarks/ten_stories_20261015T224646Z.json— Claude Opus 5.5 (9/10, c=4, 81.5 tok/s stream, ~$0.089 est.)artifacts/benchmarks/ten_stories_20261015T224739Z.json— GLM-5.3 High Thinking (10/10, c=2, token telemetry omitted upstream)artifacts/benchmarks/ten_stories_20261015T223731Z.json— GLM-5.3-Flash (8/10, c=2, token telemetry omitted upstream)artifacts/benchmarks/ten_stories_gemini_4_argon_c4.json— Google Gemini 4 Argon (10/10, c=4 parallel, 33.4s wall time, $0.0284)