arXiv:2604.04894cs.CLcs.AI2026-04被引 1

通过分离正负反馈通道,精准调节大模型推理中的熵动态。

Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR

论文配图:Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
图 1 · 摘自论文原文
  • 将优势估计分为正负通道,分别调控有效探索与无效噪声
  • 在5个数学推理数据集上显著优于现有强化学习方法
  • 适合需要精细控制推理过程的大型语言模型研究者

基于可验证奖励的强化学习(RLVR)显著提升了大语言模型(LLM)的推理能力,但常面临探索受限问题,即策略过快集中于少数解法。常用熵正则化虽试图维持探索性,但对系数敏感,易引入语义弱不确定性,且增益有限。为此,本文将组相对策略优化(GRPO)中的优势估计分解为正负结果条件通道,分析其熵动态。结果显示,正通道增强与成功推理路径相关的‘有效熵’,而负通道消除失败轨迹带来的‘噪声熵’,减少对正确路径的干扰。基于此,提出AsymGRPO,解耦正负优势的调制强度,实现对不同提示难度下模型更新的灵活控制:可在难提示中强化罕见成功,在简单提示中抑制残余失败,无需共享相同调制强度。在五个数学推理基准上的实验表明,AsymGRPO持续优于强基线,跨多种模型架构表现一致提升。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of large language models (LLMs), but it often suffers from \textit{restricted exploration}, where the policy rapidly concentrates on a narrow set of solutions. A common remedy is entropy regularization, which attempts to preserve exploration by increasing policy entropy. However, for LLM-RL, this intervention is highly sensitive to its coefficient, can introduce semantically weak uncertainty, and often yields limited accuracy gains. This motivates a more precise question: which entropy helps reasoning, and which entropy should be reduced? To study this, we parameterize the advantage estimator in Group Relative Policy Optimization (GRPO) into positive and negative outcome-conditioned channels and analyze their entropy dynamics. Our results show that positive-channel modulation raises \textit{productive entropy} associated with successful reasoning trajectories, while negative-channel modulation removes \textit{noisy entropy} associated with failed rollouts and reduces interference with correct paths. Guided by this channel-wise view, we propose \textbf{AsymGRPO}, which decouples the modulation strengths of positive and negative advantages. This enables flexible control over how the model updates across prompt difficulty levels, allowing stronger reinforcement of rare successes on harder prompts or stronger suppression of residual failures on easier prompts without forcing the two channels to share the same modulation strength. Experiments on five mathematical reasoning benchmarks show that AsymGRPO outperforms strong RLVR baselines, with consistent gains across model backbones.

强化学习大模型推理熵调控策略优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。