梳理大模型偏好对齐方法，构建统一分析框架。
Towards a Unified View of Preference Learning for Large Language Models: A Survey
- 将对齐方法拆解为模型、数据、反馈、算法四部分
- 揭示不同方法间的内在联系，促进优势互补
- 适合研究大模型对齐与智能系统设计的读者
大型语言模型（LLMs）展现出强大的能力，其成功关键之一是使输出与人类偏好对齐。这一对齐过程通常只需少量数据即可有效提升模型性能。尽管高效，相关研究分散于多个领域，方法复杂且相互关系未被充分探讨，制约了该方向的发展。为此，本文将现有主流对齐策略分解为四个核心组件：模型、数据、反馈与算法，提出一个统一框架，以厘清各类方法间的关联。该框架不仅深化了对现有对齐算法的理解，还为融合不同策略优势提供可能。此外，文中详述了多种主流算法的运行实例，帮助读者全面掌握。最后，基于统一视角，探讨了大模型与人类偏好对齐所面临的挑战及未来研究方向。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. This alignment process often requires only a small amount of data to efficiently enhance the LLM's performance. While effective, research in this area spans multiple domains, and the methods involved are relatively complex to understand. The relationships between different methods have been under-explored, limiting the development of the preference alignment. In light of this, we break down the existing popular alignment strategies into different components and provide a unified framework to study the current alignment strategies, thereby establishing connections among them. In this survey, we decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies. Furthermore, we present detailed working examples of prevalent existing algorithms to facilitate a comprehensive understanding for the readers. Finally, based on our unified perspective, we explore the challenges and future research directions for aligning large language models with human preferences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。