根据提示难易动态分配计算资源，提升大模型对齐效率
AdaBoN: Adaptive Best-of-N Alignment
- 按提示难度分阶段分配计算资源，先小预算探查再精准投放
- 相同预算下性能优于均匀分配，批处理规模越大优势越明显
- 适配任意大模型与评分模型组合，实测50批提示均有效
近期测试时对齐方法（如Best-of-N采样）通过奖励模型简单有效地引导语言模型生成期望行为。但这类方法在所有提示上均匀使用计算资源，未考虑对齐难度差异，导致计算成本偏高。本文提出一种提示自适应的Best-of-N对齐策略，实现推理时计算资源的高效分配。基于延迟敏感性，设计两阶段算法：第一阶段用小预算探索每个提示的奖励分布；第二阶段根据估计结果自适应分配剩余预算。该方法简单实用，兼容任意语言模型与奖励模型组合。在AlpacaEval、HH-RLHF和PKU-SafeRLHF数据集上，针对12组语言模型/奖励模型搭配及50个不同提示批次的实验表明，本方法在相同推理预算下优于均匀分配；且相比预算增加20%的均匀分配仍具竞争力，性能随批处理规模增大而提升。
原文摘要 · Abstract (English)
Recent advances in test-time alignment methods, such as Best-of-N sampling, offer a simple and effective way to steer language models (LMs) toward preferred behaviors using reward models (RM). However, these approaches can be computationally expensive, especially when applied uniformly across prompts without accounting for differences in alignment difficulty. In this work, we propose a prompt-adaptive strategy for Best-of-N alignment that allocates inference-time compute more efficiently. Motivated by latency concerns, we develop a two-stage algorithm: an initial exploratory phase estimates the reward distribution for each prompt using a small exploration budget, and a second stage adaptively allocates the remaining budget using these estimates. Our method is simple, practical, and compatible with any LM-RM combination. Empirical results on prompts from the AlpacaEval, HH-RLHF, and PKU-SafeRLHF datasets for 12 LM/RM pairs and 50 different batches of prompts show that our adaptive strategy outperforms the uniform allocation with the same inference budget. Moreover, we show that our adaptive strategy remains competitive against uniform allocations with 20 percent larger inference budgets and improves in performance as the batch size grows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。