让大模型推理时更懂人话，速度还更快
DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
- 用扩散模型思路做偏好优化，直接在句子级对齐
- 比传统方法快，多轮测试中表现更好且延迟更低
- 可插拔适配各类模型，尤其提升70B大模型表现
推理时对齐为大语言模型与人类对齐提供了一种高效替代方案。然而，现有方法仍面临扩展性受限（因依赖特定策略的价值函数）和推理阶段延迟高等问题。本文提出一种新方法——扩散风格偏好优化（DiffPO），实现高效且策略无关的推理时对齐。通过直接在句子层面进行对齐，避免了传统逐标记生成带来的延迟。作为即插即用模块，DiffPO可无缝集成到多种基础模型中以增强对齐能力。在AlpacaEval 2、MT-bench和HH-RLHF上的大量实验表明，DiffPO在多种设置下均取得优异对齐性能，实现了对齐质量与推理延迟间的良好权衡。此外，其模型无关的可扩展性显著提升了Llama-3-70B等大模型的表现。
原文摘要 · Abstract (English)
Inference-time alignment provides an efficient alternative for aligning LLMs with humans. However, these approaches still face challenges, such as limited scalability due to policy-specific value functions and latency during the inference phase. In this paper, we propose a novel approach, Diffusion-styled Preference Optimization (\model), which provides an efficient and policy-agnostic solution for aligning LLMs with humans. By directly performing alignment at sentence level, \model~avoids the time latency associated with token-level generation. Designed as a plug-and-play module, \model~can be seamlessly integrated with various base models to enhance their alignment. Extensive experiments on AlpacaEval 2, MT-bench, and HH-RLHF demonstrate that \model~achieves superior alignment performance across various settings, achieving a favorable trade-off between alignment quality and inference-time latency. Furthermore, \model~demonstrates model-agnostic scalability, significantly improving the performance of large models such as Llama-3-70B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。