提出新方法稳定训练大模型的离策略强化学习。
VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
- 用变分法设计序列级软策略优化，直接调整生成序列权重。
- 在64倍延迟下仍保持训练稳定，比现有方法更优。
- 适合需要高稳定性的大模型推理与代码生成任务。
由于异步训练导致的推理过时以及训练与推理引擎不匹配，大语言模型的强化学习不可避免地面临离策略更新问题。朴素重要性采样虽无偏但方差极高，尤其在自回归生成和无界比例下被放大。此前方法或依赖特定场景设计，或通过词元级裁剪或序列级归一化以偏换方差，多为启发式手段。本文提出变分序列级软策略优化（VESPO），将方差减小纳入变分框架，推导出可直接作用于序列级重要性权重的闭式重塑核，避免词元级近似和长度归一化，且能显式给出部署核的方差上界。在数学推理和代码生成任务上的实验表明，即使在64倍过时条件下，VESPO仍能维持稳定训练，并在密集模型和专家混合（MoE）模型中持续取得提升，优于近期重塑基线方法。代码已开源。
原文摘要 · Abstract (English)
Off-policy updates are inevitable in reinforcement learning (RL) for large language models (LLMs) due to rollout staleness from asynchronous training and mismatches between training and inference engines. Naive importance sampling gives an unbiased correction but suffers from high variance, which is amplified by unbounded ratios and autoregressive generation. Prior remedies either rely on scenario-specific engineering, or trade bias for variance via token-level clipping or sequence-level normalization, yet these approaches remain largely heuristic. We propose Variational sEquence-level Soft Policy Optimization (VESPO). By explicitly incorporating variance reduction into a variational formulation, we derive a principled closed-form reshaping kernel that operates directly on sequence-level importance weights, avoids token-level approximation and length normalization, and admits an explicit variance bound for the deployed kernel. Experiments on math reasoning and code generation show that VESPO maintains stable training under severe off-policy conditions (staleness up to 64x) and delivers consistent gains across both dense and Mixture-of-Experts (MoE) models, outperforming recent reshaping baselines under matched setup. Code is available at https://github.com/FloyedShen/VESPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。