让大模型对简单题简答、复杂题深思，提升推理效率与准确率。
Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty
- 按问题难度动态调整输出长度惩罚，简单题短答，难题长思。
- 在GSM8K和MATH500上缩短输出同时保持或提升准确率。
- 适合追求高效精准推理的AI系统研发者使用。
大型语言模型在推理能力上取得显著进展，已在多个挑战性基准测试中表现优异。虽然链式思维提示（Chain-of-Thought prompting）等方法进一步提升了推理效果，但通常导致输出过长，增加计算延迟。尽管已有研究采用强化学习来缩短推理路径，但普遍采用统一的长度惩罚，未考虑问题复杂度差异，导致结果不理想。本文提出一种新策略：通过分段奖励函数并引入新型长度惩罚机制，实现对简单问题的简洁回答与复杂问题的充分推理，从而兼顾效率与准确率。在GSM8K、MATH500和AIME2024三个数据集上的实验表明，该方法在相对简单的GSM8K和MATH500上显著缩短输出长度，同时保持或提升准确率；在更具挑战性的AIME2024上实现了更高的准确率。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated significant advancements in reasoning capabilities, performing well on various challenging benchmarks. Techniques like Chain-of-Thought prompting have been introduced to further improve reasoning. However, these approaches frequently generate longer outputs, which in turn increase computational latency. Although some methods use reinforcement learning to shorten reasoning, they often apply uniform penalties without considering the problem's complexity, leading to suboptimal outcomes. In this study, we seek to enhance the efficiency of LLM reasoning by promoting conciseness for simpler problems while preserving sufficient reasoning for more complex ones for accuracy, thus improving the model's overall performance. Specifically, we manage the model's reasoning efficiency by dividing the reward function and including a novel penalty for output length. Our approach has yielded impressive outcomes in benchmark evaluations across three datasets: GSM8K, MATH500, and AIME2024. For the comparatively simpler datasets GSM8K and MATH500, our method has effectively shortened output lengths while preserving or enhancing accuracy. On the more demanding AIME2024 dataset, our approach has resulted in improved accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。