提出高效对齐大模型的新方法，用少量反馈实现更好对齐效果。
Sample-Efficient Alignment for LLMs
- 基于上下文竞斗老虎机框架，设计基于汤普森采样的统一算法。
- 在三个模型规模上验证，仅用少量反馈即达到接近最优对齐效果。
- 适合需要低样本成本对齐的大模型研究者，代码已开源。
我们研究在预算内在线反馈条件下，高效对齐大语言模型（LLMs）与人类偏好。首先将对齐问题形式化为上下文竞斗老虎机问题，该框架涵盖在线RLHF和在线DPO等范式，天然要求具备在线主动探索能力的高样本效率算法。基于博弈论洞察，提出一种基于汤普森采样的统一算法，并在两个不同大模型对齐场景中应用。实际部署的代理SEA（Sample-Efficient Alignment）在三个模型规模（1B、2.8B、6.9B）和三种偏好学习算法（DPO、IPO、SLiC）上通过大量实验验证，结果表明其能以极低样本量实现接近理想偏好对齐，优于近期主动探索方法。此外，我们公开了SEA实现代码及高效在线对齐代码库，旨在推动该领域研究进展。
原文摘要 · Abstract (English)
We study methods for efficiently aligning large language models (LLMs) with human preferences given budgeted online feedback. We first formulate the LLM alignment problem in the frame of contextual dueling bandits. This formulation, subsuming recent paradigms such as online RLHF and online DPO, inherently quests for sample-efficient algorithms that incorporate online active exploration. Leveraging insights from bandit theory, we introduce a unified algorithm based on Thompson sampling and highlight its applications in two distinct LLM alignment scenarios. The practical agent that efficiently implements this algorithm, named SEA (Sample-Efficient Alignment), is empirically validated through extensive experiments across three model scales (1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The results demonstrate that SEA achieves highly sample-efficient alignment with oracle's preferences, outperforming recent active exploration methods for LLMs. Additionally, we release the implementation of SEA together with an efficient codebase designed for online alignment of LLMs, aiming to accelerate future research in this field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。