用对比偏好优化让大模型学会理解整段文本意图，提升指令遵循能力。
Sequence-level Large Language Model Training with Contrastive Preference Optimization
- 通过对比学习注入序列级偏好信号，无需人工标注数据。
- 在指令遵循和文本生成任务中胜率超越传统预测目标。
- 适合需要高质量生成与指令对齐的场景，如对话系统。
下一个词预测损失是大型语言模型的主要自监督训练目标，在多种下游任务中表现优异。然而，深入分析发现该目标缺乏对序列级信号的理解，导致训练与推理过程不匹配。为此，我们提出一种对比偏好优化（CPO）方法，可在任意训练阶段注入序列级信息，且无需昂贵的人工标注数据。实验表明，该目标在指令遵循和文本生成任务中的胜率优于下一个词预测。
原文摘要 · Abstract (English)
The next token prediction loss is the dominant self-supervised training objective for large language models and has achieved promising results in a variety of downstream tasks. However, upon closer investigation of this objective, we find that it lacks an understanding of sequence-level signals, leading to a mismatch between training and inference processes. To bridge this gap, we introduce a contrastive preference optimization (CPO) procedure that can inject sequence-level information into the language model at any training stage without expensive human labeled data. Our experiments show that the proposed objective surpasses the next token prediction in terms of win rate in the instruction-following and text generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。