让大模型教小模型如何自我优化推理,提升答案准确率。
Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs
- 用多个推理路径生成候选答案,再综合优化成更优解。
- 在五个数学基准上达到当前最佳,且跨模型规模泛化能力强。
- 适用于希望提升推理能力但算力有限的研究者或应用者。
测试时扩展(TTS)已成为提升大语言模型推理能力的热门方法。现有方法如Best-of-N和多数投票受限于候选答案质量,当所有候选均错误时无法得出正确结果。并行自精炼通过生成多个候选并基于它们合成优化答案,展现出潜力,但其有效机制尚不明确。为此,我们提出新度量指标‘精炼差距’(Refinement Gap),用于量化自精炼相较于多数投票的相对提升。实验表明,该差距随模型规模增长显著,与基础能力关联较弱。基于此,我们提出生成式自精炼(GSR),将大模型的精炼策略迁移至小模型。GSR联合训练单个模型,既能生成高质量候选,又能基于候选生成更优最终答案。实验显示,该方法在五个数学基准上优于其他并行聚合方法,且精炼能力可跨模型规模与家族迁移,对分布外领域也具备强泛化性。
原文摘要 · Abstract (English)
Test-time scaling (TTS) has gained widespread attention for enhancing LLM reasoning. Existing approaches such as Best-of-N and majority voting are limited as their performance depends on the quality of candidate responses, making them unable to produce a correct solution when all candidates are incorrect. Parallel self-refinement, generating multiple candidates and synthesizing a refined answer conditioned on them, offers a promising alternative, but the underlying mechanism driving its effectiveness remains obscure. To bridge this gap in understanding, we introduce a new metric, the Refinement Gap, designed to quantify the relative improvement of self-refinement beyond majority voting. We show that the Refinement Gap exhibits a clear scaling trend with model size and is only weakly correlated with the base capability. Based on this discovery, we propose Generative Self-Refinement (GSR), a parallel test-time scaling framework that transfers the refinement policy from larger teacher models with higher refinement gap into smaller students. Crucially, GSR jointly trains a single model to generate strong candidates and refine a better final answer based on these candidates. Experimental results demonstrate that our method achieves state-of-the-art performance across five mathematical benchmarks over other parallel aggregation methods, while the learned refinement skill transfers across multiple model scales and families and exhibits robust generalization to an out-of-distribution domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。