综述多模态生成模型的人类偏好对齐技术进展。
Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
- 系统梳理语言、语音、视觉任务中的偏好调优方法
- 涵盖强化学习框架与人类反馈集成的完整流程
- 适合研究生成模型对齐与人机交互的学者参考
偏好调优是使深度生成模型与人类偏好对齐的关键过程。本文全面综述了近期在偏好调优及人类反馈融合方面的进展。论文分为三个主要部分：1）引言与基础：介绍强化学习框架、跨模态（语言、语音、视觉）的偏好调优任务、模型与数据集，以及不同策略方法；2）深入分析各类偏好调优方法；3）应用、讨论与未来方向：探讨偏好调优在下游任务中的应用，包括各模态的评估方法，并展望未来研究趋势。本文旨在呈现最新的偏好调优方法与模型对齐技术，提升研究者与实践者对该领域的理解，推动该方向的持续探索与创新。
原文摘要 · Abstract (English)
Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth exploration of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。