arXiv:2512.01925cs.CLcs.AI2025-12中稿 · ICLR被引 3

用优化视角改进大模型推理,减少冗长思考

Rectifying LLM Thought from Lens of Optimization

  • 将推理链视为梯度下降过程,设计可量化的优化评估机制
  • 在数学、科学、编程任务上提升性能,缓解过度思考问题
  • 适合想优化推理质量的AI研究者和工程师

近期大语言模型(LLMs)的进展得益于其涌现的推理能力,尤其是通过长链思维(CoT)提示实现全面探索与推敲。然而,长CoT LLM常表现出次优推理行为,如过度思考和过长推理链,影响性能。本文从优化角度分析推理过程,将CoT视为梯度下降过程,每一步推理为向解空间的一次更新。基于此,提出RePro(过程级奖励修正)方法,在后训练阶段优化推理。RePro定义代理目标函数,通过双评分机制量化推理过程的强度与稳定性,生成综合过程级奖励,并无缝集成至可验证奖励强化学习(RLVR)流程中。在多种强化学习算法和不同规模模型上,于数学、科学、编码等多个基准测试中,实验表明RePro持续提升推理性能,有效缓解次优推理行为。

原文摘要 · Abstract (English)

Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thought (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper, we analyze reasoning processes through an optimization lens, framing CoT as a gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce RePro (Rectifying Process-level Reward), a novel approach to refine LLM reasoning during post-training. RePro defines a surrogate objective function to assess the optimization process underlying CoT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into a composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that RePro consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.

大模型推理强化学习优化视角

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。