用小模型高效筛选数学推理路径，提升大模型准确率
Learning to Rank Chain-of-Thought: Using a Small Model
- 用能量模型框架基于结果标签排序推理链，无需复杂标注
- 仅55M参数就让Llama 3 8B在GSM8k上达90.7%准确率
- 轻量高效且能泛化到新问题和新模型，适合实际部署
大型语言模型在可靠数学推理方面表现不佳，现有验证方法往往计算成本高。本文提出能量结果奖励模型（EORM），一种高效轻量的后处理验证器，通过基于能量的框架对思维链（CoT）解法进行排序，仅使用简单结果标签即可学习区分正确与错误推理，无需昂贵的人工标注。EORM仅含5500万参数，比典型奖励模型小127倍以上，在Llama 3 8B上使GSM8k准确率提升至90.7%，MATH达63.7%。该模型通过高效选择最优推理路径，性能可媲美甚至超越资源消耗更大的Best-of-N采样。实验表明，EORM对分布外问题和未见模型均有良好泛化能力，说明其学习到了有效推理的本质规律。这种鲁棒性与高效性使其成为部署更可靠大模型于复杂现实应用的实用工具。
原文摘要 · Abstract (English)
Large Language Models (LLMs) struggle with reliable mathematical reasoning, and current verification methods are often computationally expensive. This paper introduces the Energy Outcome Reward Model (EORM), a highly efficient, lightweight post-hoc verifier designed to address this challenge. EORM uses an energy-based framework to rank Chain-of-Thought (CoT) solutions, learning to distinguish correct from incorrect reasoning using only simple outcome labels, thus eliminating the need for expensive annotations. With only 55M parameters, over 127 times smaller than typical reward models, EORM boosts the accuracy of Llama 3 8B to 90.7\% on GSM8k and 63.7\% on MATH. This performance is achieved by efficiently selecting the optimal reasoning path from a pool of candidates, allowing it to match or exceed the accuracy of far more resource-intensive Best-of-N sampling techniques. Crucially, our experiments show that EORM generalizes effectively to out-of-distribution problems and unseen models, indicating it learns fundamental principles of valid reasoning. This robustness, combined with its efficiency, establishes EORM as a practical tool for deploying more dependable LLMs in complex, real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。