用注意力引导探索，提升大模型推理强化学习效率
Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
- 根据注意力分数选择探索分支位置，聚焦关键推理步骤
- 自适应采样策略确保训练批次优势值非零，提升采样效率
- 一阶段离策略训练减少计算开销，适合数学推理任务
强化学习在提升大语言模型推理能力方面表现卓越。相较于基于结果的强化学习，过程监督强化学习（PSRL）展现出更优效果。然而，现有PSRL方法在探索效率上受限，体现在分支位置和采样数量不足。本文提出一种新型PSRL框架AttnRL，实现推理模型的高效探索。受初步观察启发——高注意力得分的步骤常对应推理行为，我们提出从高注意力位置进行分支。此外，设计了一种自适应采样策略，考虑问题难度与历史批量大小，确保整个训练批次的收益值均不为零。为进一步提升采样效率，构建了适用于PSRL的一步离策略训练流程。在多个具有挑战性的数学推理基准上的实验表明，该方法在性能、采样效率和训练效率上均优于先前方法。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL (PSRL) has emerged as a more effective paradigm compared to outcome-based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttnRL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, we design a one-step off-policy training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that our method consistently outperforms prior approaches in terms of performance and sampling and training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。