用树搜索优化生成质量，让大模型推理更高效
TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling
- 构建树状搜索结构，动态筛选并剪枝低质量回复路径
- 在相同算力下，TutorEval胜率高达65%，其他数据集超60%
- 适合追求高质量输出且受限于计算资源的部署场景
推理阶段对齐可提升大语言模型性能，无需额外训练但需平衡计算效率与输出质量。最佳N（Best-of-N, BoN）采样通过生成多个回复并选取最优者，虽有效却代价高昂。本文提出TreeBoN框架，将推测性树搜索引入BoN采样中。该方法维护一组父节点，迭代地分支与剪枝低质量路径，从而降低计算开销同时保持高输出质量。此外，利用直接偏好优化（DPO）的分词级别奖励指导树的扩展和剪枝。我们在AlpacaFarm、HH-RLHF、UltraFeedback、GSM8K和TutorEval数据集上进行评估，结果一致显示性能提升。具体而言，TreeBoN在TutorEval上达到65%最高胜率，其余数据集均接近60%胜率，优于同等计算成本下的标准BoN，展现出良好的可扩展性与对齐效果。
原文摘要 · Abstract (English)
Inference-time alignment enhances the performance of large language models without requiring additional training or fine-tuning but presents challenges due to balancing computational efficiency with high-quality output. Best-of-N (BoN) sampling, as a simple yet powerful approach, generates multiple responses and selects the best one, achieving improved performance but with a high computational cost. We propose TreeBoN, a novel framework that integrates a speculative tree-search strategy into Best-of-N (BoN) Sampling. TreeBoN maintains a set of parent nodes, iteratively branching and pruning low-quality responses, thereby reducing computational overhead while maintaining high output quality. Our approach also leverages token-level rewards from Direct Preference Optimization (DPO) to guide tree expansion and prune low-quality paths. We evaluate TreeBoN using AlpacaFarm, HH-RLHF, UltraFeedback, GSM8K, and TutorEval datasets, demonstrating consistent improvements. Specifically, TreeBoN achieves the highest win rate of 65% on TutorEval and around 60% win rates across other different datasets, outperforming standard BoN with the same computational cost and showcasing its scalability and alignment efficacy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。