PROMA通过投影去除梯度方差，实现无参考的稳定强化学习训练。
PROMA: Projected Microbatch Accumulation for Reference-Free Proximal Policy Updates
- 用正交投影移除高方差梯度分量，控制策略更新的KL散度。
- 累积变体比GRPO+PPO裁剪更严格地控制每步KL值。
- 片内变体兼容数据并行，验证性能最佳，适合大规模训练。
本文提出无参考近端策略更新方法PROMA，通过将策略梯度中高方差成分正交投影移除，实现对KL散度的控制。提出两种变体：累积型在运行梯度上沿每个微批次的序列级对数概率梯度方向正交投影；片内型则在每个微批次内独立使用激活与梯度输出的主要子空间进行分解投影，兼容标准数据并行训练。实验表明，累积型在每步KL控制上优于使用PPO裁剪的GRPO，而片内型在验证集上取得最佳性能。
原文摘要 · Abstract (English)
This note introduces Projected Microbatch Accumulation (PROMA), a reference-free proximal policy method that controls KL divergence by projecting away high-variance components of the policy gradient. Two variants are presented. In the accumulation-based variant, the running gradient is projected orthogonal to the sequence-wise log-probability gradients of each microbatch. In the intra-microbatch variant, a factored projection using dominant subspaces of activations and gradient outputs is applied independently within each microbatch, making it compatible with standard data-parallel training. Empirically, the accumulation variant achieves tighter per-step KL control than GRPO with PPO clipping, while the intra-microbatch variant achieves the best validation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。