提出MASPO框架，让大模型推理更高效可靠
MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning
- 用可微软门控提升梯度利用效率
- 在多种评测中显著优于基线方法
- 适合追求高效可靠的LLM推理研究者
现有基于可验证奖励的强化学习（RLVR）算法如GRPO，依赖僵化、均匀且对称的信任区域机制，与大语言模型（LLMs）复杂的优化动态本质不符。本文指出三方面关键问题：(1) 硬截断导致梯度利用效率低下；(2) 均匀比率约束忽略词元分布，引发概率质量不敏感；(3) 正负样本信用分配模糊性造成信号可靠性不对称。为此，我们提出质量自适应软策略优化（MASPO），统一整合三个维度：采用可微软高斯门控最大化梯度效用，质量自适应限幅器平衡概率谱上的探索，以及非对称风险控制器使更新幅度与信号置信度对齐。大量实验表明，MASPO作为一体化的鲁棒RLVR方案，显著超越基线。代码已开源：https://github.com/FlyTune/MASPO-RL。
原文摘要 · Abstract (English)
Existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, such as GRPO, rely on rigid, uniform, and symmetric trust region mechanisms that are fundamentally misaligned with the complex optimization dynamics of Large Language Models (LLMs). In this paper, we identify three critical challenges in these methods: (1) inefficient gradient utilization caused by the binary cutoff of hard clipping, (2) insensitive probability mass arising from uniform ratio constraints that ignore the token distribution, and (3) asymmetric signal reliability stemming from the disparate credit assignment ambiguity between positive and negative samples. To bridge these gaps, we propose Mass-Adaptive Soft Policy Optimization (MASPO), a unified framework designed to harmonize these three dimensions. MASPO integrates a differentiable soft Gaussian gating to maximize gradient utility, a mass-adaptive limiter to balance exploration across the probability spectrum, and an asymmetric risk controller to align update magnitudes with signal confidence. Extensive evaluations demonstrate that MASPO serves as a robust, all-in-one RLVR solution, significantly outperforming baselines. Our code is available at: https://github.com/FlyTune/MASPO-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。