Public benchmark tasks · measured by Verdict · October 28, 2026

More reasoning.
More tasks solved.

ToolTerse v5 on all 1,140 BigCodeBench-Complete tasks. The reasoning-on configuration solved 91 more tasks directly and 84 more through Verdict Code, with higher token use.

Model & license: ToolTerse v5 · Q4_K_M GGUF
Reported base: Qwen3.6-35B-A3B · Apache-2.0; not rechecked here
Weight provenance: Modification technique, tensor delta and model artifact SHA-256 unverified in this campaign
Evaluation: BigCodeBench-Complete · 1,140 tasks · temperature 1.0 · one replicate · seed not recorded
Host & control: Apple M5 Max · LM Studio · 32,768 loaded context
Earlier direct-off run: 564/1,140; current 570, paired p=0.755

The four-arm scorecard

After infrastructure repair

Direct · reasoning off

50.0%

570 / 1,140 tasks passed

530 recorded tokens / task

Verdict Code · reasoning off

50.1%

571 / 1,140 tasks passed

1,512 recorded tokens / task

Direct · reasoning on

58.0%

661 / 1,140 tasks passed

4,703 recorded tokens / task

Verdict Code · reasoning on

57.5%

655 / 1,140 tasks passed

2,922 recorded tokens / task

One final scored result per task; first-pass scores precede infrastructure repair.
Metric	Direct · reasoning off	Verdict Code · reasoning off	Direct · reasoning on	Verdict Code · reasoning on
Passed / 1,140	570 (50.0%)	571 (50.1%)	661 (58.0%)	655 (57.5%)
First-pass passed	569	568	655	652
Output tokens / summed task second	30.6	26.0	41.2	76.6
Total recorded tokens	604,282	1,723,422	5,361,683	3,331,579
Recorded output tokens	184,443	284,654	4,944,008	1,894,671
Tokens / task: mean · median	530 · 492	1,512 · 1,435	4,703 · 4,373	2,922 · 2,176
First-pass wall time · concurrency	25.8 min · ×4	203.0 min · ×1	503.1 min · ×4	418.7 min · ×1

Token totals omit usage from five resource-limit failures. Route prompts, accounting and effective output caps differ. Wall times exclude repairs and use unequal concurrency; they do not establish a speed advantage. Energy and dollar savings were not measured.

What changes on the same tasks?

Exact two-sided McNemar tests, n=1,140. b = first arm only passed; c = second arm only passed. pp = percentage points.
Comparison	Score change	b / c	p-value
Direct: off → on	+7.98 pp	95 / 186	6.13e-8
Verdict Code: off → on	+7.37 pp	90 / 174	2.60e-7
Reasoning off: direct → Verdict Code	+0.09 pp	118 / 119	1.000
Reasoning on: direct → Verdict Code	-0.53 pp	113 / 107	0.736

Supported within this run

Reasoning-on configurations solve more.

The gain is statistically clear on this task set. Direct-on also doubled the output cap to 16,384, so the result reflects the configuration as a whole; it does not isolate reasoning from a larger budget.

No detected score difference

Harness equivalence is not established.

With reasoning on, Verdict Code passed six fewer tasks. The paired difference is −0.53 pp (approximate 95% interval −3.08 to +2.02 pp). A non-significant result is not proof of equal quality.

Promising, with accounting limits

37.9% fewer recorded tokens.

Reasoning-on Verdict Code recorded 2,922 tokens per task versus 4,703 direct. Different prompts, token accounting and output caps prevent a clean efficiency claim. With reasoning off, the harness recorded 2.85× as many tokens for one extra pass.

A real failure that reasoning on fixed

BigCodeBench/28: encode a payload and send it to an API. Excerpts from the saved direct-arm answers:

Direct · reasoning off · failed

return requests.post(url, data=base64.b64encode(json.dumps(data).encode()).decode())

Direct · reasoning on · passed

response = requests.post(url, json={'payload': b64_str})

Why it fails: test_case_6 asserts a POST with a JSON object containing a payload key. The off answer sends the encoded string as the raw request body; assert_called_once_with reports “expected call not found.” This is one worked example, not a measurement of how often this mechanism occurs.

External context · separate evaluation conditions

Historical competitor references

Selected published Full / Complete results from the official BigCodeBench data, last updated April 16, 2025 and retrieved October 28, 2026. These are historical model checkpoints, not current frontier rankings or results rerun by Verdict.

Context only — not a head-to-head leaderboard. Official results use calibrated pass@1 with greedy decoding. Verdict used one sampled attempt per task at temperature 1.0 plus infrastructure replacements, different prompts and budgets, and a local evaluator. Dataset/evaluator revision parity is unverified. No cross-study significance or win claim is made.

All rows: official Full / Complete, calibrated pass@1. Dates are checkpoint dates recorded by the source, not evaluation dates.
Model checkpoint	Published score	Source checkpoint date
Gemini-Exp-1206	62.4%	2024-12-06
DeepSeek-V3	62.2%	2024-12-26
Llama-4-Maverick	61.4%	2025-04-05
GPT-4o-2024-05-13	61.1%	2024-05-13
Claude-3.5-Sonnet-20240620	58.6%	2024-06-20
Qwen2.5-Coder-32B-Instruct	58.0%	2024-09-19

Pinned official results snapshot ↗ · Official benchmark and decoding notes ↗

What this evidence can support

ToolTerse v5 is a credible local candidate for this public Python task set. Reasoning-on configurations buy a measurable accuracy gain; reasoning off keeps the direct route substantially leaner. This run does not establish general software-engineering ability, superiority to current cloud models, or a product-wide default policy.

Method, repairs and reproducibility

One replicate at temperature 1.0; repeat-run variability and contamination were not assessed. No model-weight hash was captured.
Direct-off cap: 8,192; direct-on: 16,384. Harness receipts record effective max_tokens of 13,107 or 104,857 on most rows, plus three 8,192 rows in the on arm. This is not a uniform-budget factorial experiment.
Final scores replace infrastructure losses. Repaired pass totals differ from reported first-pass totals by +1, +3, +6 and +3 respectively; the archived first-pass reconstruction was not independently revalidated here.
Five 8 GiB memory breaches remain scored failures: task 791 in both off arms; 779, 791 and 1042 in harness-on. Containment was not active uniformly across all original rows.
Harness-on task 812 was replaced after a zero-token Gateway timeout; its final real attempt used 3,166 tokens and failed at runtime.
Reported base/license come from the prior campaign record. The v5 weight-modification method and tensor delta remain unverified.

All four saved final row files were independently checked for the exact task IDs 0–1139, no duplicates, pass totals, tokens and paired tests. This verifies recorded results; generated programs were not executed again.

Reproduction from the retained run files: python3 scripts/benchmarks/verify_tt5_scorecard.py. The hashes below identify the exact final row files used.

Direct · reasoning off · row-file SHA-256: db6a10c6b8b15a2399920907629bf5e7cef6bf55d906a52dffcc83189b654fc1
Verdict Code · reasoning off · row-file SHA-256: 530949280566cfe97517dd32d3c6cde3113f2fbe537f111a5e3eb930fdc622a0
Direct · reasoning on · row-file SHA-256: f50a872e16811d040e0028f5e8726d6f90ac5049799e5ca1164e3f0b7617f314
Verdict Code · reasoning on · row-file SHA-256: 6d65d9bc4b608064a6348f47578d0f75d8369a07526c6df9551ff0b7450f1764