arXiv:2502.01237cs.LG2025-02被引 2

直接对齐算法的性能差异主要由排序目标决定,而非优化的分数类型。

The Differences Between Direct Alignment Algorithms are a Blur

  • 统一框架下将单阶段方法转为双阶段,引入β参数统一超参空间
  • 排序目标(成对/点对)是决定对齐质量的核心因素,分数类型影响较小
  • 适用于评估大模型对齐方法的研究者,尤其关注方法本质差异

直接对齐算法(DAAs)通过直接优化策略简化大语言模型对齐过程,跳过奖励建模与强化学习。尽管DAAs在SFT阶段(单阶段与双阶段)及优化的标量分数(似然比或几率比)上存在差异,但关键性能驱动因素仍不明确。本文系统比较并分析了一个被忽视的维度——排序目标(成对对比与点对预测)。为此,我们提出统一训练框架:(i) 将单阶段方法(ORPO、ASFT)转化为含显式SFT阶段的双阶段流程;(ii) 引入β参数使所有方法处于同一超参空间,提升几率比类方法(ORPO、ASFT)的质量。在此设置下,排序目标成为对齐质量的主要决定因素,而具体优化的标量分数(政策-参考比值与几率比)则为次要。我们在指令遵循任务中验证此结论,并在不同模型规模的数学推理基准上进一步确认。证据表明,这一现象源于不同目标与提示特定偏差的交互机制,既来自严格控制实验,也得到真实数据观察支持。研究强调需对DAAs进行更精细评估,避免过度简化优越性宣称。

原文摘要 · Abstract (English)

Direct Alignment Algorithms (DAAs) simplify LLM alignment by directly optimizing policies, bypassing reward modeling and RL. While DAAs differ in their use of SFT (one-stage vs. two-stage) and the scalar score they optimize (likelihood vs. odds ratios), the key performance drivers remain underexplored. We present a systematic comparison and analyze a previously overlooked axis - the ranking objective (pairwise vs. pointwise). To isolate this factor, we propose a unified training framework across DAAs by (i) converting one-stage methods (ORPO, ASFT) into a two-stage pipeline with an explicit SFT phase and (ii) introducing a $β$ parameter that places all methods in the same hyperparameter space and improves the quality of odds-ratio DAAs (ORPO, ASFT). Under this setup, the ranking objective emerges as the primary determinant of alignment quality, whereas the particular scalar score (policy-reference ratio vs. odds ratio) is secondary. We corroborate this on instruction-following tasks and further confirm it on math-reasoning benchmarks across model scales. Evidence suggests that this stems from how these objectives interact with prompt-specific biases, supported both by strictly controlled experiments and by observations on real data. Our findings underscore the need for nuanced evaluations in DAA research to avoid oversimplified claims of superiority.

大模型对齐直接对齐排序目标

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。