arXiv:2505.21765cs.AI2025-05NeurIPS被引 13

优化大模型推理过程,让思考更高效而非更长。

Don't Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models

  • 将推理路径拆分为不同思维模式,动态筛选优劣
  • 减少47%计算量,错误率降低15.6%,准确率提升12%
  • 适合关注推理效率与成本的AI研究者和开发者

尽管大型推理模型(LRMs)通过强化学习显著提升了答案准确性,但其往往因过度思考导致输出过长,表现为不必要的复杂推理路径,浪费算力且可能降低性能。我们假设此类低效源于模型无法在恰当位置动态选择合适的推理模块化策略。为此,提出一种动态优化框架,将生成的推理路径分割为不同思维模式,系统识别并保留有益模式,剔除有害模式。实证分析表明,优化后的推理路径更简洁且信息充分,注意力计算量减少高达47%,同时保持原有正确答案的准确率;此外,约有非微小比例原错误回答被纠正,准确率提升15.6%。基于优化路径带来的改进,我们采用偏好优化技术,结合成对数据集对比次优与最优推理路径。在多个数学推理基准上的实验显示,该方法显著降低计算开销,同时提升推理准确率,最高实现12%准确率提升,令牌使用量从约5000降至3000。

原文摘要 · Abstract (English)

While recent success of large reasoning models (LRMs) significantly advanced LLMs' reasoning capability by optimizing the final answer accuracy using reinforcement learning, they may also drastically increase the output length due to overthinking, characterized by unnecessarily complex reasoning paths that waste computation and potentially degrade the performance. We hypothesize that such inefficiencies stem from LRMs' limited capability to dynamically select the proper modular reasoning strategies, termed thinking patterns at the right position. To investigate this hypothesis, we propose a dynamic optimization framework that segments model-generated reasoning paths into distinct thinking patterns, systematically identifying and promoting beneficial patterns that improve the answer while removing detrimental ones. Empirical analysis confirms that our optimized thinking paths yield more concise yet sufficiently informative trajectories, enhancing reasoning efficiency by reducing attention FLOPs by up to 47% while maintaining accuracy for originally correct responses. Moreover, a non-trivial portion of originally incorrect responses are transformed into correct ones, achieving a 15.6% accuracy improvement with reduced length. Motivated by the improvement brought by the optimized thinking paths, we apply a preference optimization technique supported by a pairwise dataset contrasting suboptimal and optimal reasoning paths. Experimental evaluations across multiple mathematical reasoning benchmarks reveal that our method notably reduces computational overhead while simultaneously improving reasoning accuracy, achieving up to a 12% accuracy improvement and reducing token usage from approximately 5,000 to 3,000 tokens.

推理优化大模型思维模式效率提升

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。