增加采样轮次未必提升强化学习效果，SALT通过几何重加权让多轮次真正发挥作用。
SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter
- 利用样本梯度几何构建共享子空间，分离并增强关键更新信号
- 在多个推理型任务上提升学习效果，模型规模从70M到700B均有效
- 无需修改奖励模型或采样策略，可作为插件直接集成
基于可验证奖励的强化学习（RLVR）常采用GRPO类组相对更新机制，每提示采样多轮以构建归一化学习信号。然而单纯增加轮次并不保证学习增强：在组归一化下，每轮策略梯度特征会聚集于低秩有符号几何，导致聚合时严重抵消，削弱有效更新。我们提出SALT——一种子空间自适应的几何插件组件，通过样本级梯度几何重加权组相对更新系数。SALT从小批量的格拉姆几何中估计主导共享子空间，将组相对系数分解为共享与残差通道，并在有符号抵消严重时自适应放大残差通道。在多种面向推理的RLVR基准及不同模型规模（70M–700B）上，SALT改善了有效更新几何并提升了性能，且无需修改奖励模型或采样过程。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) often adopts GRPO-style group-relative updates, sampling multiple rollouts per prompt to construct normalized learning signals. However, merely increasing the number of rollouts does not reliably strengthen learning: under GRPO-style group normalization, per-rollout policy-gradient features can concentrate into a low-rank, signed geometry, causing substantial cancellation during aggregation and weakening the effective update. We address this failure mode with SALT, a Subspace-Adaptive geometry pLug-in componenT that uses sample-wise gradient geometry to reweight the coefficients of group-relative updates. SALT estimates a dominant shared subspace from the mini-batch Gram geometry, decomposes group-relative coefficients into shared and residual channels, and adaptively amplifies the residual channel when signed cancellation is severe. Across diverse reasoning-oriented RLVR benchmarks and model scales, SALT improves effective update geometry and performance without modifying the reward model or the rollout sampling procedure
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。