Benchmark methodology

Design a comparison

Freeze the pack version and commit, task inputs, models, execution environment, tool and network policy, budgets, and scoring configuration. Run candidates under the same constraints and preserve the inputs and evidence needed to inspect the results.

Scoring and repetitions

Deterministic checks and judge dimensions answer different questions. Repeat each configuration and report sample sizes and variation before estimating reliability. A single run cannot establish a pass rate. Normalized token efficiency is not money spent; dollar cost requires prices and usage.

What the expression report records

The current report describes one expression evaluator task and one run per model, with shell and filesystem tools, no network, and ten iterations. It reports hidden deterministic tiers, an output contract, and judge dimensions. Its fixture link points to mutable main; no immutable inputs or raw replay artifacts are attached, so the published numbers are not an exact reproduction recipe.

• Expression evaluator report
• Example fixture