提出一种更高效的推理阶段对齐方法，显著降低大模型生成成本。
Fast Best-of-N Decoding via Speculative Rejection
- 通过推测性拒绝机制，在不修改模型权重的前提下优化生成质量。
- 相比经典Best-of-N方法，计算效率提升16至32倍。
- 适合需要快速部署且对生成安全有要求的应用场景。
大语言模型（LLM）的安全有效部署需经历对齐阶段，以确保输出符合人类偏好。现有对齐方法如DPO、PPO等通过后训练调整模型权重，虽有效但流程复杂。推理阶段对齐方法避免后训练，直接在生成时引导模型产出偏好响应。其中，Best-of-N是当前最有效的推理阶段对齐方法，效果媲美先进后训练方案，但其推理资源消耗巨大，难以实用。本文提出推测性拒绝（Speculative Rejection），一种计算高效的推理阶段对齐算法。该方法能生成高奖励得分的响应，与Best-of-N效果相当，同时在推理效率上提升16至32倍，使大规模部署成为可能。
原文摘要 · Abstract (English)
The safe and effective deployment of Large Language Models (LLMs) involves a critical step called alignment, which ensures that the model's responses are in accordance with human preferences. Prevalent alignment techniques, such as DPO, PPO and their variants, align LLMs by changing the pre-trained model weights during a phase called post-training. While predominant, these post-training methods add substantial complexity before LLMs can be deployed. Inference-time alignment methods avoid the complex post-training step and instead bias the generation towards responses that are aligned with human preferences. The best-known inference-time alignment method, called Best-of-N, is as effective as the state-of-the-art post-training procedures. Unfortunately, Best-of-N requires vastly more resources at inference time than standard decoding strategies, which makes it computationally not viable. In this work, we introduce Speculative Rejection, a computationally-viable inference-time alignment algorithm. It generates high-scoring responses according to a given reward model, like Best-of-N does, while being between 16 to 32 times more computationally efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。