用注意力信号精准识别冗余推理步骤，提升效率与准确率
ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning
- 基于模型注意力机制实现每一步的精细化奖励分配
- 在9个基准上减少推理长度同时显著提升性能
- 适合需要高效推理的复杂任务场景
通过强化学习与可验证奖励训练的大规模推理模型在复杂推理任务中表现优异，但常出现过度思考，生成无益冗余推理。现有轨迹级长度惩罚方法因对所有推理步骤一视同仁，难以有效压缩长度且损害准确率。过程监督方法通常资源开销大，且信用分配不准确。为此，我们提出低开销的过程监督强化学习框架 ATTNPO，利用模型内在注意力信号实现步骤级信用分配。首先识别出一组聚焦关键步骤、抑制冗余步骤的特殊注意力头。随后，基于这些头的注意力分数，采用两种子策略：通过抑制冗余步骤缓解过度思考，同时减少对关键步骤的惩罚以保持准确率。实验表明，ATTNPO 在9个基准上显著缩短推理长度并大幅提升性能。
原文摘要 · Abstract (English)
Large reasoning models trained with reinforcement learning and verifiable rewards (RLVR) achieve strong performance on complex reasoning tasks, yet often overthink, generating redundant reasoning without performance gains. Existing trajectory-level length penalties often fail to effectively shorten reasoning length and degrade accuracy, as they uniformly treat all reasoning steps and lack fine-grained signals to distinguish redundancy from necessity. Meanwhile, process-supervised methods are typically resource-intensive and suffer from inaccurate credit assignment. To address these issues, we propose ATTNPO, a low-overhead process-supervised RL framework that leverages the model's intrinsic attention signals for step-level credit assignment. We first identify a set of special attention heads that naturally focus on essential steps while suppressing redundant ones. By leveraging the attention scores of these heads, We then employ two sub-strategies to mitigate overthinking by discouraging redundant steps while preserving accuracy by reducing penalties on essential steps. Experimental results show that ATTNPO substantially reduces reasoning length while significantly improving performance across 9 benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。