让大模型生成更多样且高质量的回答，同时保持性能。
Diverse Preference Optimization
- 从回复池中选稀有但优质样本作为偏好对，提升多样性。
- 生成的个性特征多样性提升45.6%，故事多样性提升74.6%。
- 适合创意写作等需要多样输出的任务场景。
语言模型后训练阶段（如强化学习、偏好优化或监督微调）往往使输出概率分布变尖锐，降低生成结果的多样性，这在需要多样化回复的创造性任务中尤为不利。本文提出一种名为「多样偏好优化」（DivPO）的新方法，通过从一组回复中选取稀有但高质量的样本作为优选项，而将常见但低质量的样本作为拒选项，从而在保持生成质量的同时显著提升多样性。实验表明，该方法使个性化属性多样性提升45.6%，故事多样性提升74.6%，且与标准基线相比胜率相近。在通用指令遵循任务中，多样性提升46.2%，胜率提高2.4%。
原文摘要 · Abstract (English)
Post-training of language models, either through reinforcement learning, preference optimization or supervised finetuning, tends to sharpen the output probability distribution and reduce the diversity of generated responses. This is particularly a problem for creative generative tasks where varied responses are desired. In this work we introduce Diverse Preference Optimization (DivPO), an optimization method which learns to generate much more diverse responses than standard pipelines, while maintaining the quality of the generations. In DivPO, preference pairs are selected by first considering a pool of responses, and a measure of diversity among them, and selecting chosen examples as being more rare but high quality, while rejected examples are more common, but low quality. DivPO results in generating 45.6% more diverse persona attributes, and a 74.6% increase in story diversity, while maintaining similar win rates as standard baselines. On general instruction following, DivPO results in a 46.2% increase in diversity, and a 2.4% winrate improvement compared to DPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。