让大模型生成更符合人类偏好的内容，同时精准控制每个词的重要性。
Token-Importance Guided Direct Preference Optimization
- 用梯度+高斯先验结合的方法计算词的重要程度，更准确可靠。
- 在多个数据集上表现优于DPO，生成结果更多样且稳定。
- 适合需要精细控制输出质量的场景，如对话系统、内容创作。
将大型语言模型（LLMs）与人类偏好对齐对于实现安全有效的AI交互至关重要。尽管流行的直接偏好优化（DPO）方法简化了对齐过程，但仍对数据噪声敏感，并忽略单个词元的重要性差异。现有的词元级方法通常依赖于概率预测或简单的加权方案来获取词元重要性，仍无法完全解决这些问题。为此，我们提出了一种基于词元重要性的直接偏好优化框架（TI-DPO），通过两项协同创新实现细粒度语义控制。首先，提出一种新的混合加权机制，结合梯度归因与高斯先验，确保词元重要性评分的准确性与鲁棒性。其次，采用三元组损失提供结构化引导，明确指导模型输出向偏好响应靠近，远离非偏好响应。实验结果表明，TI-DPO在准确性和生成多样性方面均优于DPO及其他基于强化学习的对齐方法，提供了更稳定且计算效率更高的解决方案。
原文摘要 · Abstract (English)
Aligning Large Language Models (LLMs) with human preferences is crucial for safe and effective AI interactions. While popular methods like Direct Preference Optimization (DPO) have simplified alignment, they remain sensitive to data noise and overlook the differential importance of individual tokens. Existing token-level approaches often rely on probability prediction or simplistic weighting schemes to obtain token importance, which still cannot fully address these issues. To solve this problem, we propose the Token-Importance Guided Direct Preference Optimization (TI-DPO), a framework that achieves fine-grained semantic control through two synergistic innovations. First, we propose a novel hybrid weighting mechanism that combines gradient attribution with a Gaussian prior, ensuring both the accuracy and robustness of token importance scores. Second, we employ a triplet loss to provide structured guidance for the optimization, explicitly guiding model outputs to approach preferred responses and diverge from non-preferred ones. Experimental results show that TI-DPO achieves higher accuracy and stronger generative diversity, providing more stable and computationally efficient solutions compared with DPO and other RLHF methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。