评估开源大模型推理效率与准确率权衡，发现计算量增益有上限。
Compute-Accuracy Pareto Frontiers for Open-Source Reasoning Large Language Models
- 在数学和推理任务上测试多个开源模型，绘制计算-准确率帕累托前沿
- 发现扩展推理步骤虽提升性能，但超过阈值后准确率不再增长
- MoE架构在效率与性能间表现最优，适合资源受限场景
大型语言模型在复杂推理基准测试中持续进步，尤其在允许生成中间推理步骤的情况下。然而，现有研究常忽略生成长推理序列带来的巨大计算负担。工业应用中，模型选择不仅依赖准确率，还需考虑资源约束与推理成本。本文对当前及旧版开源大模型进行推理时长感知的评估，绘制其在数学与高推理强度基准上的帕累托前沿。结果表明，混合专家（MoE）架构在本评估设置中是性能与效率的良好平衡点。此外，我们追踪了帕累托效率随时间的变化趋势，揭示出每单位计算量带来的准确率提升呈现递减趋势。最后，我们证明推理时计算存在饱和点：超过一定阈值后，准确率增益消失，说明尽管延长推理有益，但无法突破模型固有的复杂性限制。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are demonstrating rapid improvements on complex reasoning benchmarks, particularly when allowed to utilize intermediate reasoning steps before converging on a final solution. However, current literature often overlooks the significant computational burden associated with generating long reasoning sequences. For industrial applications, model selection depends not only on raw accuracy but also on resource constraints and inference costs. In this work, we conduct a test-time-compute aware evaluation of both contemporary and older open-source LLMs, mapping their Pareto frontiers across math- and reasoning-intensive benchmarks. Our findings identify the Mixture of Experts (MoE) architecture as a strong candidate to balance performance and efficiency in our evaluation setting. Furthermore, we trace the trajectory of Pareto efficiency over time to derive an emergent trend regarding accuracy gain per unit of compute. Finally, we demonstrate that there is a saturation point for inference-time compute. Beyond a certain threshold, accuracy gains diminish, indicating that while extended reasoning capabilities are beneficial, they cannot overcome intrinsic model limitations regarding specific complexities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。