小模型通过工具辅助验证,大幅提升推理能力。
T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models
- 用外部工具预筛选候选结果,再由小模型做最终验证。
- 在MATH数据集上,1B小模型超越8B大模型性能。
- 适合资源有限但需高可靠推理的场景。
近期研究表明,测试时计算扩展可有效提升小语言模型(sLMs)性能。然而,以往研究多依赖更大模型作为验证器,忽视了sLM自身验证能力。本文发现,即使通过知识蒸馏从大模型学习,sLM在需要记忆的任务(如数值计算、事实核查)中仍表现不佳。为此提出工具集成验证(T1)框架:先用外部工具(如代码解释器)过滤候选结果,再由sLM完成最终验证,将记忆密集型任务移交给工具。理论证明,工具外推能降低sLM的记忆负担,提升测试时扩展效果。在MATH基准测试中,经T1优化的Llama-3.2 1B模型性能超越更大的Llama-3.1 8B模型。同时,T1也提升了过程奖励模型(PRMs)与评阅模型的验证准确率。结果表明,工具集成能显著增强sLM的验证能力。
原文摘要 · Abstract (English)
Recent studies have demonstrated that test-time compute scaling effectively improves the performance of small language models (sLMs). However, prior research has mainly examined test-time compute scaling with an additional larger model as a verifier, leaving verification by sLMs underexplored. In this work, we investigate whether sLMs can reliably verify the output candidates under test-time scaling. We find that even with knowledge distillation from larger verifiers, sLMs struggle with verification tasks requiring memorization, such as numerical calculations and fact-checking. To address this limitation, we propose Tool-integrated verification (T1), a two-stage framework that first filters candidates with external tools and then uses an sLM for final verification, offloading memorization-heavy steps to tools such as a code interpreter. Within T1, we prove that offloading to external tools reduces the memorization burden on sLMs and improves test-time scaling performance. Experiments on the MATH benchmark demonstrate that, with T1, a Llama-3.2 1B model under test-time scaling outperforms the significantly larger Llama-3.1 8B model. Moreover, T1 improves the verification accuracy of both process reward models (PRMs) and critic models. Our findings highlight the potential of tool integration to substantially improve the verification abilities of sLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。