让语言模型每步生成都更懂用户偏好，提升对齐效果与多样性。
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
- 基于前缀建模每步选词偏好，从序列比较中推导出逐令牌最优策略。
- 在指令跟随、摘要等任务上，比现有方法对齐更好、训练更稳、输出更丰富。
- 无需额外奖励模型，支持轻量基线或优势归一化，适合实用部署。
直接偏好优化（DPO）是主流的免强化学习对齐方法，但其对全序列偏好建模，而生成实际由逐令牌决策驱动。现有令牌级扩展多将序列级 Bradley-Terry 目标分解到各时间步，隐含了前缀（状态）层面的最优性。本文研究如何仅用标准序列级成对比较恢复令牌级偏好最优性。提出令牌级 Bregman 偏好优化（TBPO），构建条件于前缀的下个词动作的令牌级 Bradley-Terry 模型，并推导出基于 Bregman 散度密度比匹配的目标函数，该函数推广了逻辑回归和 DPO 损失，同时保持令牌级模型诱导的最优策略并维持 DPO 式简洁性。提出两种实现：TBPO-Q 显式学习轻量状态基线；TBPO-A 通过优势归一化移除基线。在指令遵循、有用性/无害性、摘要等基准上，TBPO 提升对齐质量与训练稳定性，且输出多样性优于强序列级与令牌级基线。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley-Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。