RePO用单参数优化让大模型更懂人类偏好，比现有方法更稳定高效。
RePO: Understanding Preference Learning Through ReLU-Based Optimization
- 基于ReLU的极大边界损失，自动过滤无意义样本对。
- 理论证明是SimPO在β趋于无穷时的极限情况，仅需调一个超参数。
- 在AlpacaEval 2和Arena-Hard上超越DPO与SimPO，适配多类基础模型。
将大语言模型（LLMs）对齐人类偏好对实际部署至关重要，但现有方法如RLHF存在计算与稳定性挑战。尽管DPO建立了单超参数β的离线范式，后续方法如SimPO又引入双参数（β, γ），增加复杂性。我们提出基于ReLU的偏好优化（RePO），通过两项改进消除β：(1) 保留SimPO的无参考边距，通过梯度分析移除β；(2) 采用基于ReLU的极大边界损失，自然过滤平凡样本对。理论上，RePO是SimPO的极限情形（β→∞），逻辑加权退化为二值阈值，形成0-1损失的凸包络。实证结果表明，在AlpacaEval 2与Arena-Hard数据集上，RePO在多个基模型上均优于DPO与SimPO，且仅需调节一个超参数。
原文摘要 · Abstract (English)
Aligning large language models (LLMs) with human preferences is critical for real-world deployment, yet existing methods like RLHF face computational and stability challenges. While DPO establishes an offline paradigm with single hyperparameter $β$, subsequent methods like SimPO reintroduce complexity through dual parameters ($β$, $γ$). We propose {ReLU-based Preference Optimization (RePO)}, a streamlined algorithm that eliminates $β$ via two advances: (1) retaining SimPO's reference-free margins but removing $β$ through gradient analysis, and (2) adopting a ReLU-based max-margin loss that naturally filters trivial pairs. Theoretically, RePO is characterized as SimPO's limiting case ($β\to \infty$), where the logistic weighting collapses to binary thresholding, forming a convex envelope of the 0-1 loss. Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO and SimPO across multiple base models, requiring only one hyperparameter to tune.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。