Benchmark
How ratel-bench measures Ratel's token savings and accuracy, and where the results live.
Ratel's numbers come from an open harness, ratel-ai/ratel-bench, and the published results live at benchmark.ratel.sh.
Three arms
Every scenario runs three ways on the same model:
- control-baseline — the entire candidate tool pool is exposed to the agent on every turn.
- control-oracle — only the gold tool is exposed; the accuracy upper bound.
- ratel-full — the model sees a handful of targeted tools plus the capability tools, regardless of how large the underlying pool is.
The gap between baseline and ratel-full is the token saving; the gap to oracle is the accuracy headroom.
Two suites
- BFCL — the Berkeley Function-Calling Leaderboard, simple and multiple splits, with pool sizes swept across realistic MCP setups (30 → 600 tools).
- Skills suite — agent tasks over six datasets (bigcodebench, champ, logicbench, medcalcbench, theoremqa, toolqa), measuring retrieval and completion on multi-step work.
What gets measured
Per model, arm, and split: task completion, tool selection accuracy, recall, mean total tokens, and latency. A separate retriever evaluation sweeps the pool sizes (precision, recall, MRR, nDCG).
Where results live
benchmark.ratel.sh renders the reports as interactive charts and tables, methodology included. The site regenerates at build time from ratel-bench's published reports, so it always tracks the source of truth.
For the headline numbers and what they mean in your agent, see Efficiency.