让大模型自己判断哪些词更重要，提升对齐效果
Token-weighted Direct Preference Optimization with Attention
- 用注意力机制自动计算响应中每个词的重要程度
- 在多个评测集上超越现有方法，最高提升4.2分
- 无需额外训练，仅多两次前向传播即可实现
直接偏好优化（DPO）可将大语言模型对齐人类偏好，而无需单独的奖励模型。然而，传统DPO对响应中的所有词同等对待，忽略了单个词的重要性差异。现有词级别优化方法依赖基于位置的启发式函数或额外训练的模型来估计权重，缺乏鲁棒性且增加训练成本。为此，我们提出词权重直接偏好优化（TwDPO），一种基于词权重强化学习的新训练目标，并设计AttentionPO作为其实例，利用大模型自身的注意力机制估计词权重。AttentionPO让模型充当成对评判者，通过观察其在比较响应时的关注点来调整权重。该设计使方法具备内容感知能力，能根据响应内容动态调整权重，且高效——每样本仅需两次额外前向传播。实验结果表明，AttentionPO在AlpacaEval、MT-Bench和ArenaHard上显著提升性能，超越现有偏好优化方法。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。