用批量数据自适应调整强化学习更新,提升大模型训练稳定性。
Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
- 根据当前批次的策略比率分布动态调整优化目标
- 无需人工调参,在不同任务和模型规模下表现稳定
- 适合需要高鲁棒性的大规模强化学习场景
强化学习因策略改变其学习数据分布而结构上比监督学习更复杂。在大模型训练中,训练与回放系统在数值精度、采样方式等实现细节上的差异加剧了这种脆弱性。现有方法通过在目标函数中引入超参数来缓解,但导致算法对配置敏感,且任务、模型规模或分布不匹配变化时需重新调参。这种脆弱性源于两个被固定超参数纠缠的问题:信任域问题(更新不应使策略偏离过远)与离策略问题(旧行为策略的数据仅在可靠时才影响更新)。二者并非先验常数,其严重程度体现在当前批次的策略比率分布中。本文提出一种简单的批处理自适应目标,以策略比率的归一化有效样本量替代固定截断。该统计量同时限制得分函数权重并调节离策略正则化强度,当比率接近均匀时接近标准的在线策略更新;当过时或不匹配数据导致比率集中时自动收紧,同时保留高比率令牌的非零学习信号。跨多种设置的实验表明,该方法性能达到或超过调优基线,无需新增目标超参数,并可移除多个已有超参数。代码已公开于 https://github.com/FeynRL-project/FeynRL。
原文摘要 · Abstract (English)
Reinforcement learning is structurally harder than supervised learning because the policy changes the data distribution it learns from. The resulting fragility is especially visible in large-model training, where the training and rollout systems differ in numerical precision, sampling, and other implementation details. Existing methods manage this fragility by adding hyper-parameters to the training objective, which makes the algorithm more sensitive to its configuration and requires retuning whenever the task, model scale, or distribution mismatch changes. This fragility traces to two concerns that current objectives entangle through hyper-parameters set before training begins: a trust-region concern, that updates should not move the policy too far from its current value, and an off-policy concern, that data from older or different behavior policies should influence the update only to the extent that it remains reliable. Neither concern is a constant to set in advance, and their severity is reflected in the policy-ratio distribution of the current batch. We present a simple yet effective batch-adaptive objective that replaces fixed clipping with the normalized effective sample size of the policy ratios. The same statistic caps the score-function weight and sets the strength of an off-policy regularizer, so the update stays close to the usual on-policy score-function update when ratios are nearly uniform, and tightens automatically when stale or mismatched data cause ratio concentration, while retaining a nonzero learning signal on high-ratio tokens. Experiments across a wide range of settings show that our method matches or exceeds tuned baselines, introducing no new objective hyper-parameters and removing several existing ones. The code is available at https://github.com/FeynRL-project/FeynRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。