拆解大模型推理效率,看懂每一步耗时是否值得。
Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs
- 用完成率、正确率和生成长度三指标分解推理效率。
- 发现不同模型在逻辑、上下文、冗余上的失效模式各异。
- 适合评估推理模型性能,尤其关注效率而非单纯准确率。
随着推理型大模型通过反复思考、搜索与自我修正来提升准确率,单一准确率已无法判断这些消耗的token是否带来有效推理、错误恢复或无谓冗长。本文提出一种无需模型踪迹的评估协议,精确分解令牌效率,仅依赖三个可观测指标:完成率、给定完成情况下的条件正确率、生成长度。当获得实例级工作量元数据时,进一步以任务隐含工作量为基准归一化生成长度,分离平均表述开销与依赖工作量的扩展部分。若元数据缺失,则定义可审计的求解器衍生工作量尺度,并在留自身、留前k个、保留参照池扰动下验证其稳定性。我们在CogniLoad、GSM8K、ProofWriter和ZebraLogic上评估14个公开权重模型,在CogniLoad上额外评估11个模型,实现对推理任务难度因素(任务长度、内在难度、干扰密度)的细粒度分析。效率与开销排名在所有基准对中保持稳定,优于准确率排名;该分解能区分逻辑受限、上下文受限(截断驱动)、冗余受限三类看似相同的失败模式。我们发布评估工具包与报告模板,帮助阐明大模型推理低效的根本原因。
原文摘要 · Abstract (English)
As reasoning LLMs increasingly trade tokens for accuracy through deliberation, search, and self-correction, a single accuracy score can no longer tell whether those tokens buy useful reasoning, recovery from hard instances, or unnecessary verbosity. We introduce a trace-optional evaluation protocol that exactly decomposes token efficiency using three observables available even for closed models: completion rate, conditional correctness given completion, and generated length. When instance-level workload metadata is available, we further normalize generated length by declared task-implied work and separate mean verbalization overhead from workload-dependent scaling. When such metadata is absent, we define an auditable solver-derived workload scale and evaluate its stability under leave-self-out, leave-top-k, and held-out-reference-pool perturbations. We evaluate 14 shared open-weight models on CogniLoad, GSM8K, ProofWriter, and ZebraLogic. We further evaluate 11 additional models on CogniLoad, enabling a fine-grained analysis of reasoning-task difficulty factors: task length, intrinsic difficulty, and distractor density. Efficiency and overhead rankings remain stable across all benchmark pairs, more robustly than accuracy rankings, while the decomposition separates logic-limited, context-limited (truncation-driven), and verbosity-limited failure modes that look identical under accuracy-per-token. We release an evaluation artifact and reporting template, which elaborates on why an LLM is inefficient at reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。