通过控制策略比率方差，提升大模型强化学习微调的稳定性和数据效率。
Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
- 引入比率方差正则化，软化传统裁剪机制，保留高回报动作梯度信号。
- 在数学推理任务上平均性能提升17%，收敛所需采样次数减少50%。
- 适合追求高效、稳定微调大模型的研究者和工程团队使用。
基于策略的强化学习（如PPO和GRPO）已成为大语言模型（LLM）微调的主流范式。尽管策略比率裁剪能稳定训练，但其硬约束会无差别截断高回报但高偏差的动作梯度，抑制复杂推理中的‘顿悟时刻’。此外，一旦数据稍显陈旧，硬裁剪使其无法再用，导致严重样本低效。本文重新审视策略优化中的信任区域目标，提出显式约束策略比率的方差（二阶中心矩），提供一种合理且平滑的硬裁剪替代方案。基于此，我们提出R²VPO（Ratio-Variance Regularized Policy Optimization），一个新型的原对偶框架，支持稳定在线学习，并通过动态重加权而非丢弃的方式实现陈旧数据的合理复用。我们在DeepSeek-Distill-Qwen-1.5B及openPangu-Embedded系列（1B和7B）上广泛评估R²VPO，在多个数学推理基准上表现优异：相比强裁剪基线，平均相对性能提升达17%，收敛所需滚子次数约减少50%。结果表明，比率方差控制是提升基于强化学习的LLM对齐稳定性与数据效率的有前景方向。
原文摘要 · Abstract (English)
On-policy reinforcement learning (RL), particularly Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), has become the dominant paradigm for fine-tuning large language models (LLMs). While policy ratio clipping stabilizes training, this heuristic hard constraint incurs a fundamental cost: it indiscriminately truncates gradients from high-return yet high-divergence actions, suppressing rare but highly informative "eureka moments" in complex reasoning. Moreover, once data becomes slightly stale, hard clipping renders it unusable, leading to severe sample inefficiency. In this work, we revisit the trust-region objective in policy optimization and show that explicitly constraining the \emph{variance (second central moment) of the policy ratio} provides a principled and smooth relaxation of hard clipping. This distributional constraint stabilizes policy updates while preserving gradient signals from valuable trajectories. Building on this insight, we propose $R^2VPO$ (Ratio-Variance Regularized Policy Optimization), a novel primal-dual framework that supports stable on-policy learning and enables principled off-policy data reuse by dynamically reweighting stale samples rather than discarding them. We extensively evaluate $R^2VPO$ on fine-tuning state-of-the-art LLMs, including DeepSeek-Distill-Qwen-1.5B and the openPangu-Embedded series (1B and 7B), across challenging mathematical reasoning benchmarks. Experimental results show that $R^2VPO$ consistently achieves superior asymptotic performance, with average relative gains of up to 17% over strong clipping-based baselines, while requiring approximately 50% fewer rollouts to reach convergence. These findings establish ratio-variance control as a promising direction for improving both stability and data efficiency in RL-based LLM alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。