提出无需单独偏好模型的对齐方法，降低训练成本
Investigating on RLHF methodology
- 用困惑度过滤法高效构建偏好数据集
- 直接优化偏好，避免训练独立偏好模型
- 适用于希望低成本对齐大模型的研究者
本文研究大语言模型与人类偏好对齐的方法。重点讨论了模拟人类偏好的偏好模型训练特征，以及实现最佳效果的关键方法与细节。探讨了使用强化学习微调大模型的策略，分析了遇到的挑战及应对方式。此外，介绍了直接偏好优化（DPO）方法，可在不构建独立偏好模型的情况下实现模型对齐。作为贡献，提出了通过困惑度过滤收集偏好数据集的方法，显著简化了特定语言模型的偏好数据构建流程，提升了效率并降低了成本。
原文摘要 · Abstract (English)
In this article, we investigate the alignment of Large Language Models according to human preferences. We discuss the features of training a Preference Model, which simulates human preferences, and the methods and details we found essential for achieving the best results. We also discuss using Reinforcement Learning to fine-tune Large Language Models and describe the challenges we faced and the ways to overcome them. Additionally, we present our experience with the Direct Preference Optimization method, which enables us to align a Large Language Model with human preferences without creating a separate Preference Model. As our contribution, we introduce the approach for collecting a preference dataset through perplexity filtering, which makes the process of creating such a dataset for a specific Language Model much easier and more cost-effective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。