提出等长配对训练框架，解决序列强化学习中的长度不可比问题
Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Construction
- 通过生成时主动构造等长可比的样本片段，而非事后修正
- 在GRPO/GSPO/RLOO上实现更稳定的训练与性能提升
- 适合需要稳定序列生成的RLHF场景
本文研究序列级相对强化学习中的长度问题。我们发现，尽管现有方法部分缓解了长度相关现象，但根本性问题仍未被充分揭示：训练中使用的比较单元本身缺乏内在可比性。基于此观察，我们提出新视角：长度问题不应仅视为损失缩放或归一化偏差，而应看作‘比较单元构建’问题。为此，我们建立一种基于样本构建的训练框架，不依赖对不等长响应的事后修正，而是主动在生成阶段构建等长、对齐且可比的训练片段。在此框架下，我们提出EqLen方法，适用于GRPO、GSPO和RLOO等组相对比较算法。通过双轨同步生成、前缀继承与片段掩码机制，EqLen高效收集有效等长训练片段，实现稳定训练。
原文摘要 · Abstract (English)
This paper investigates the length problem in sequence-level relative reinforcement learning. We observe that, although existing methods partially alleviate length-related phenomena, a more fundamental issue remains insufficiently characterized: the comparison units used during training lack inherent comparability. Building on this observation, we propose a new perspective: the length problem should not be viewed merely as a loss-scaling or normalization bias, but rather as a \emph{comparison unit construction} problem. We further establish a sample-construction-based training framework that, instead of applying post-hoc corrections to unequal-length responses, proactively constructs equal-length, alignable, and comparable training segments during generation. Within this framework, we propose EqLen, a concrete method applicable to group-relative comparison algorithms such as GRPO, GSPO, and RLOO. Through dual-track synchronous generation, prefix inheritance, and segment masking, EqLen efficiently collects effective equal-length training segments and enables stable
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。