首个同时评估大模型推理准确率与令牌效率的基准测试。
OckBench: Measuring the Efficiency of LLM Reasoning
- 构建多任务评估体系，同步测量推理与编码的准确率和令牌消耗。
- 实测显示同精度下令牌用量差达5倍，存在严重冗余。
- 适合关注模型成本优化与推理效率的研究者使用。
大型语言模型（如 GPT-5 和 Gemini 3）在自动化推理与代码生成方面取得了突破。然而，现有基准主要关注准确率与输出质量，忽视了关键维度——令牌使用效率。实际应用中，相同问题的解决路径在相似准确率下，令牌长度可相差高达5.0×，暴露了显著的冗余问题。这凸显出建立标准化基准以量化令牌效率差距的迫切需求。为此，我们提出 OckBench，首个在推理与编码任务中联合评估准确率与令牌效率的基准。评估发现，当前模型的令牌效率普遍未被优化，导致服务成本与延迟大幅增加。该研究为社区提供了优化隐式推理能力与令牌效率的明确路线图。最终，我们主张评估范式转变：令牌不应无必要地膨胀。相关基准已开源：https://ockbench.github.io/。
原文摘要 · Abstract (English)
Large language models (LLMs) such as GPT-5 and Gemini 3 have pushed the frontier of automated reasoning and code generation. Yet current benchmarks emphasize accuracy and output quality, neglecting a critical dimension: efficiency of token usage. The token efficiency is highly variable in practical. Models solving the same problem with similar accuracy can exhibit up to a \textbf{5.0$\times$} difference in token length, leading to massive gap of model reasoning ability. Such variance exposes significant redundancy, highlighting the critical need for a standardized benchmark to quantify the gap of token efficiency. Thus, we introduce OckBench, the first benchmark that jointly measures accuracy and token efficiency across reasoning and coding tasks. Our evaluation reveals that token efficiency remains largely unoptimized across current models, significantly inflating serving costs and latency. These findings provide a concrete roadmap for the community to optimize the latent reasoning ability, token efficiency. Ultimately, we argue for an evaluation paradigm shift: tokens must not be multiplied beyond necessity. Our benchmarks are available at https://ockbench.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。