通过锚点奖励抑制大模型推理中的重复验证，提升效率。
APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards
- 定义推理锚点，识别答案稳定时刻，定位冗余重复阶段。
- 在5个数学推理数据集上，1.5B和7B模型达性能-效率最优平衡。
- 专治推理后期无效重复，适合追求高效推理的部署场景。
测试时扩展（TTS）显著提升了大推理模型（LRMs）的能力，但带来了名为过度思考的副作用。我们从细粒度角度重新审视该现象，发现模型在得出最终答案后仍频繁进行无修改的重复自验证。我们正式定义答案首次稳定的时刻为推理锚点。分析锚点前后行为，揭示出模型中固有的结构性冗余：即答案稳定后的无意义重复验证，称为答案稳定尾（AST）。受此启发，我们提出锚点过程奖励（APR），一种结构感知的奖励塑造方法，能准确定位推理锚点并仅惩罚锚点后的AST。结合适用于长度惩罚的策略优化算法，我们的APR模型在1.5B和7B规模下，于五个数学推理数据集上平均达到性能-效率帕累托前沿，且强化学习训练所需计算资源显著减少。
原文摘要 · Abstract (English)
Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs) but introduces a critical side-effect known as Overthinking. We conduct a preliminary study to rethink this phenomenon from a fine-grained perspective. We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process. We formally define this specific position where the answer first stabilizes as the Reasoning Anchor. By analyzing pre- and post-anchor reasoning behaviors, we uncover the structural redundancy fixed in LRMs: the meaningless repetitive verification after deriving the first complete answer, which we term the Answer-Stable Tail (AST). Motivated by this observation, we propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST. Leveraging the policy optimization algorithm suitable for length penalties, our APR models achieved the performance-efficiency Pareto frontier at 1.5B and 7B scales averaged across five mathematical reasoning datasets while requiring substantially fewer computational resources for RL training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。