让大模型在推理时更聪明地选答案，提升准确率
Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
- 训练时模拟推理策略，让模型学会配合最佳选择机制
- 数学题准确率从26.8%提至30.8%，代码测试通过率升至67.1%
- 适合追求推理效率与质量的AI研发者和应用落地团队
近期研究指出，有效利用推理阶段计算资源对提升大语言模型性能至关重要。本文提出一种新的推理感知微调范式，使模型在微调过程中直接优化推理策略的表现。以简单高效的Best-of-N（BoN）推理策略为例，该策略由验证器从多个模型生成的回答中选出最优者。我们首次设计了基于模仿学习和强化学习的BoN感知微调方法，克服了BoN中难以求导的argmax操作难题。实验表明，所提模型隐式学习到一种元策略：在选取最优回答的同时穿插更具多样性的回应，类似强化学习中的探索-利用权衡。在实际测试中，Gemma 2B模型在Hendrycks MATH数据集上的Bo32准确率从26.8%提升至30.8%，pass@32从60.0%增至67.0%；HumanEval上pass@16从61.6%提升至67.1%，显著提升性能并优化推理计算效率。
原文摘要 · Abstract (English)
Recent studies have indicated that effectively utilizing inference-time compute is crucial for attaining better performance from large language models (LLMs). In this work, we propose a novel inference-aware fine-tuning paradigm, in which the model is fine-tuned in a manner that directly optimizes the performance of the inference-time strategy. We study this paradigm using the simple yet effective Best-of-N (BoN) inference strategy, in which a verifier selects the best out of a set of LLM-generated responses. We devise the first imitation learning and reinforcement learning~(RL) methods for BoN-aware fine-tuning, overcoming the challenging, non-differentiable argmax operator within BoN. We empirically demonstrate that our BoN-aware models implicitly learn a meta-strategy that interleaves best responses with more diverse responses that might be better suited to a test-time input -- a process reminiscent of the exploration-exploitation trade-off in RL. Our experiments demonstrate the effectiveness of BoN-aware fine-tuning in terms of improved performance and inference-time compute. In particular, we show that our methods improve the Bo32 performance of Gemma 2B on Hendrycks MATH from 26.8% to 30.8%, and pass@32 from 60.0% to 67.0%, as well as the pass@16 on HumanEval from 61.6% to 67.1%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。