arXiv:2410.05318cs.LGcs.AI2024-10被引 44

通过多路径推理与协作验证,提升大模型在数学和代码任务中的推理准确率。

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

  • 生成多条推理路径,用验证器评估并排序正确性。
  • 在GSM8k和MATH上超越现有模型,甚至超过GPT-4o。
  • 融合思维链与程序链优势,兼顾可解释性与精确验证。

尽管大语言模型(LLMs)在通用能力上取得显著进展,但在数学和代码等复杂任务中仍存在推理不一致、不准确的问题。主要原因是模型训练时仅学习正确解法,缺乏对错误的识别与纠正能力。为此,本文在推理阶段大幅增加计算量,生成多条推理路径,并引入验证器评估输出的正确性与排序。为此构建了一个涵盖数学与代码任务的综合性数据集,包含多个大模型生成的正确与错误解法,帮助验证器更有效地区分正误。验证器的训练方法经过对多种现有方法的系统比较选定。为发挥不同推理策略的优势,提出一种新型协同方法,整合思维链(CoT)与程序链(PoT):CoT提供清晰步骤增强可解释性,PoT可执行且敏感于错误,二者互补。所提出的数学验证器Math-Rev与代码验证器Code-Rev,在GSM8k与MATH等基准测试中表现优异,达到当前最优水平,且使用Qwen-72B-Instruct作为推理器时,性能超过GPT-4o。

原文摘要 · Abstract (English)

Despite significant advancements in the general capability of large language models (LLMs), they continue to struggle with consistent and accurate reasoning, especially in complex tasks such as mathematical and code reasoning. One key limitation is that LLMs are trained primarily on correct solutions, reducing their ability to detect and learn from errors, which hampers their ability to reliably verify and rank outputs. To address this, we scale up the inference-time computation by generating multiple reasoning paths and employing verifiers to assess and rank the generated outputs by correctness. To facilitate this, we introduce a comprehensive dataset consisting of correct and incorrect solutions for math and code tasks, generated by multiple LLMs. This diverse set of solutions enables verifiers to more effectively distinguish and rank correct answers from erroneous outputs. The training methods for building verifiers were selected based on an extensive comparison of existing approaches. Moreover, to leverage the unique strengths of different reasoning strategies, we propose a novel collaborative method integrating Chain-of-Thought (CoT) and Program-of-Thought (PoT) solutions for verification. CoT provides a clear, step-by-step reasoning process that enhances interpretability, while PoT, being executable, offers a precise and error-sensitive validation mechanism. By taking both of their strengths, our approach significantly improves the accuracy and reliability of reasoning verification. Our verifiers, Math-Rev and Code-Rev, demonstrate substantial performance gains to existing LLMs, achieving state-of-the-art results on benchmarks such as GSM8k and MATH and even outperforming GPT-4o with Qwen-72B-Instruct as the reasoner.

大模型推理数学推理代码生成验证机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。