BETTER ANSWERS. DELIBERATE TRADEOFFS.
The right model is only the beginning. Choose how Verdict spends time, tokens and second chances to get your work done.
Find your setupSmart Retry vs its sampled first answer
One local-model research run, October 2, 2026. Smart Retry used benchmark checks to trigger and accept retries; repair saw grading failures. Different baselines. These are component measurements, not promised gains in your projects.
Explore the costs and conditions01 / YOUR PRIORITIES
Start with your goal.
See the feature set that fits.
RECOMMENDED STARTING POINT
Define success, run the check, and let a failed check guide a bounded repair.
In Verdict Server: Chat settings → Test → Auto-Validate. In Verdict Code CLI: add --test auto_validate. Use a supported coding model with meaningful project tests.
Auto-Validate defaults to two repair attempts. This is an existing workflow; the research gains below were measured with benchmark feedback, not this shipped switch.Automatically repeats eligible failed requests where the route supports recovery. It repeats the original prompt; it does not explain the failed code to the model. Quality retries require a verifier and sampling temperature of at least 0.8.
Runs a frozen acceptance check, shares failure output with the coding agent, allows bounded repair and checks again. Choose Test → Auto-Validate in Verdict Server, or use --test auto_validate.
In Desktop, choose VERIFY → Fast; this enables Verdict-Code when needed. Fast can try up to four additional samples behind checks. It is a different strategy from the one-repair research arm.
Controls checked against current source on October 4, 2026; availability depends on your app version and route. Auto-Validate and Smart Harness are alternative settings, not a stack. The Server browser’s Smart Harness control currently reports live streaming retry unavailable; use the supported Desktop coding path for Proof Mode. The manual retry-button visibility setting does not enable automatic recovery.
02 / YOUR COMPUTER
Real weight sizes.
A transparent memory estimate.
Estimated fit: weights + your allowance. We also reserve the greater of 4 GiB or 20% for macOS and other apps. Longer context needs more memory. No CPU offload is assumed.
MLX 4-bit · Apple silicon MLX variant.
Historical Complete-100 direct run · 2026-08-27. This is a shortlist recommendation, not a speed prediction for your computer or a measured minimum RAM requirement.
Quality ranks recorded success rate. Speed ranks recorded time per task. Cost ranks recorded time per solved task, assuming equal machine cost per hour. These historical runs are useful context, not controlled comparisons on identical hardware.
| Model / current variant | Weight size | Estimated fit | Solved | Recorded s/task | Tokens/task |
|---|---|---|---|---|---|
| Qwen2.5 Coder 1.5BGGUF Q8_0 | 1.76 GiB | Fits estimate | 33% | 4.23 | 770 |
| Verdict 4 4B QuillGGUF Q4_K_M | 2.33 GiB | Fits estimate | 45% | 4.16 | 567 |
| Qwen3 4B Instruct 2507MLX 4-bit | 2.11 GiB | Fits estimate | 51% | 6.18 | 785 |
| Qwen3 Coder 30BMLX 4-bit | 16 GiB | Fits estimate | 62% | 8.22 | 707 |
| Verdict 4 35B Toolterse v5MLX 4-bit | 18.17 GiB | Fits estimate | 55% | 4.49 | 542 |
All runs cover BCB Complete tasks 0–99; direct mode had Smart Retry configured. Harness revisions differ; hardware and benchmark quantization are unrecorded. Current weight variants are not cryptographically bound to those runs. MLX is shown only for Apple silicon; GGUF requires a compatible GPU backend. The shortlist is curated from complete records, not the entire catalog. No Bonsai repair uplift is applied to these models.
Download model evidence03 / THE MEASURED TRADEOFF
Fixed model: Ternary Bonsai 2 27B PTQ1_0 on an M5 Max. This experiment stays separate from your model recommendation above.
One feedback repair
Relative to first answer · t=0: 51 → 65 of 148 tasks. Success rate: 34.5% → 43.9%.
One repair saw the failed answer and grading feedback. The same mechanism exists in test-guided workflows, but this is not a measured Auto-Validate or shipped Proof Mode gain.
COMPUTE COST EXPLORER
44.7s model calls + 5.1s checks
1,131 input + 893 output
Includes spend on unsuccessful tasks
A planning scenario, not a bill or measured energy use. Defaults of 100 W, $0.20/kWh and $0 hardware/hour are editable assumptions. Accounted time sums request and check durations, including retries, failed tasks and test-author calls; it is not end-to-end user latency. API pricing is hypothetical for these recorded tokens, not an available Bonsai API offer. Excludes subscriptions, taxes and human review. Per solved task = total estimated compute spend ÷ tasks solved; it does not promise success after spending that amount.
237 unique tasks, 11 shared across the two cohorts, 1,120 model calls and 1,065,124 input + output tokens. One replicate, 4,096-token output cap. First answer and repair used T=0; resampling used T=0.7; active Smart Retry used T=0.8. Comparisons share baselines where shown, but do not isolate feedback from every other setting.
Reference implementations passed 138/148 Hard and 99/100 Complete tasks in our sandbox. Full cohorts retain those environment failures. Repair vs one resample added 14.0% more solved tasks for 34.1% more total tokens (paired p=0.0768). Neither comparison establishes a clear advantage in this single pilot.
Generated checks accepted 96 incorrect first answers on Hard and 43 on Complete. No new repair feature was deployed from this experiment. Energy was not measured. Timings came from a shared inference host and a separate grader, not your device.
Download the measured evidence04 / IN PRACTICE
Two saved benchmark examples.
What changed, and why it matters.
You ask for a chart object you can customize. The first answer returns a Figure where an Axes object is required. The failed check tells the model exactly what is wrong.
ax = voronoi_plot_2d(vor)AFTER · CREATE AND RETURN THE AXES
fig, ax = plt.subplots() voronoi_plot_2d(vor, ax=ax) return vor, ax
4 of 6 → 6 of 6 tests passing
Saved feedback-repair result using benchmark failures. Matches the public return contract. Selected example, not a typical-result estimate.
The first answer returns a file hash without sending it over the secure socket. Repair adds the missing sends and a cleanup block; the mocked checks now pass.
2 of 5 → 5 of 5 tests passing
Benchmark-feedback example, not a live network test. The added socket close matches the grader but conflicts with the prompt’s “does not handle opening or closing” note. Sending is the clear repair; cleanup exposes a contract ambiguity that needs review.
THE PRINCIPLE
Start with a model that fits. Spend extra attempts where correctness matters. Keep the evidence—and the cost—in view. Compare Verdict’s execution modes.