提出新方法让大模型生成文本长度更公平，提升强化学习稳定性。
Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL
- 用随长度平方根缩放的裁剪带，修正序列级强化学习中的长度偏差。
- 在不同长度文本上实现均匀裁剪率，训练更稳定，尤其对Qwen3-8B效果显著。
- 理论证明小长度重加权误差可保证更新方向准确，适合长文本生成任务。
我们提出FSPO（公平序列策略优化）——一种面向大语言模型的序列级强化学习方法，通过在重要性采样权重上施加长度公平的裁剪机制。研究发现，将传统PPO/GRPO的固定裁剪范围直接应用于序列时，会系统性地改变短与长响应的权重比例，扭曲优化方向。FSPO引入简单修正：以√L为尺度的可变裁剪带裁剪序列对数重要性采样比。理论上，我们通过长度重加权误差（LRE）形式化了长度公平性，并证明小LRE可保证裁剪更新与真实更新方向间的余弦一致性。实验表明，FSPO在不同模型规模和评估数据集上均实现跨长度的均匀裁剪率，显著提升训练稳定性并超越基线，其中对Qwen3-8B-Base模型提升最大。
原文摘要 · Abstract (English)
We propose FSPO (Fair Sequence Policy Optimization), a sequence-level reinforcement learning method for LLMs that enforces length-fair clipping on the importance-sampling (IS) weight. We study RL methods with sequence-level IS and identify a mismatch when PPO/GRPO-style clipping is transplanted to sequences: a fixed clip range systematically reweights short vs. long responses, distorting the optimization direction. FSPO introduces a simple remedy: we clip the sequence log-IS ratio with a band that scales as $\sqrt{L}$. Theoretically, we formalize length fairness via a Length Reweighting Error (LRE) and prove that small LRE yields a cosine directional guarantee between the clipped and true updates. Empirically, FSPO flattens clip rates across length bins, stabilizes training, and outperforms baselines across model sizes and evaluation datasets, with the largest gains on the Qwen3-8B-Base model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。