通过参数空间噪声提升大样本下的强化学习探索能力
Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards
- 在策略参数层面加噪声，实现更连贯的长序列推理探索
- 在数学推理任务中，大样本下通过率显著高于现有方法
- 适合追求高精度推理的大型语言模型研究者使用
基于可验证奖励的强化学习（RLVR）虽能提升大模型推理能力，但存在探索天花板：常仅重加权已有解题路径，而非发现新策略，导致在大规模采样（如pass-at-256）下收益受限。本文提出PSN-RLVR，通过在轨迹生成前扰动策略参数，实现时间一致、层级化的轨迹级探索，相比动作空间噪声更能保持长程思维链连贯性。为缓解采样与更新间的不匹配问题，引入截断重要性采样（TIS）。为避免昂贵的KL自适应噪声控制，设计轻量级实时自适应噪声调度器，结合语义多样性与归一化自置信度。以广泛使用的GRPO为基础实例化为PSN-GRPO，该方法在多个数学推理基准与模型家族上持续扩展有效推理边界，在大样本预算下获得更高pass-at-k性能，优于先前探索导向的RLVR方法（如pass-at-k训练），且具备正交性，可进一步组合增益。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning, yet growing evidence indicates an exploration ceiling: it often reweights existing solution traces rather than discovering new strategies, limiting gains under large sampling budgets (e.g., pass-at-256). We address this limitation with PSN-RLVR, which perturbs policy parameters before rollout generation to induce temporally consistent, trajectory-level exploration that better preserves long-horizon chain-of-thought coherence than action-space noise. To mitigate the resulting sampling-update mismatch, we incorporate truncated importance sampling (TIS). To avoid expensive KL-based adaptive noise control, we propose a computationally efficient real-time adaptive noise scheduler driven by a lightweight surrogate that combines semantic diversity with normalized self-certainty. Instantiated on GRPO, a widely used RLVR method, PSN-GRPO consistently expands the effective reasoning capability boundary across multiple mathematical reasoning benchmarks and model families, yielding higher pass-at-k under large sampling budgets and outperforming prior exploration-oriented RLVR methods (e.g., Pass-at-k-style training) while remaining orthogonal and thus composable for additional gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。