通过控制结束符概率，避免模型因缩短回答而作弊，提升对齐质量。
REFA: Reference Free Alignment for multi-preference optimization
- 引入新正则化机制，直接调控结束符（EOS）概率以防止过早截断。
- 在AlpacaEval2上实现60.29%胜率，长度约束下仍达52.17%胜率。
- 适合需要精准控制生成长度的部署场景，如资源受限设备。
为缓解响应冗长导致的奖励欺骗问题，现代偏好优化方法越来越多地采用长度归一化（如SimPO、ORPO、LN-DPO）。尽管有效抑制该偏差，我们发现长度归一化自身会引入一种失效模式：URSLA捷径——模型学会通过提前截断低质量回答来满足对齐目标，而非学习语义内容。为此，我们提出REFA，一种新的对齐框架，通过概率控制结构化终止标记实现终止决策。其核心创新在于一类作用于结束符（EOS）概率的新正则化器，这是此前未被利用的调控手段。该令牌级干预提供了对URSLA捷径的原理性解决方案，确保真正的质量提升。此外，它还解锁了灵活调节对齐效率权衡的机制，使从业者可微调符合特定令牌预算的模型。实验表明，REFA在AlpacaEval2上使用Llama-3-8B-Instruct实现60.29%胜率，长度控制下仍达52.17%胜率，验证了令牌级控制范式的有效性。
原文摘要 · Abstract (English)
To mitigate reward hacking from response verbosity, modern preference optimization methods are increasingly adopting length normalization (e.g., SimPO, ORPO, LN-DPO). While effective against this bias, we demonstrate that length normalization itself introduces a failure mode: the URSLA shortcut. Here models learn to satisfy the alignment objective by prematurely truncating low-quality responses rather than learning from their semantic content. To address this, we introduce REFA, a new alignment framework that proposes probabilistic control on a structural token that controls termination. Our core innovation is a new class of regularizers that operate directly on the probability of the End-of-Sequence (EOS) token, a previously unexploited control lever. This token-level intervention provides a principled solution to the URSLA shortcut, ensuring genuine quality improvements. Furthermore, it unlocks a versatile mechanism for managing the alignment-efficiency tradeoff, enabling practitioners to fine-tune models that adhere to specific token budgets. Empirically, REFA achieves a 60.29% win rate and a 52.17% length-controlled win rate on AlpacaEval2 with Llama-3-8B-Instruct, demonstrating the power of our token-level control paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。