让大模型像人一样思考：用双系统验证提升推理准确性
LLM2: Let Large Language Models Harness System 2 Reasoning
- 用一个生成模型+一个过程验证器组成双系统架构
- 在GSM8K上将小模型准确率从50.3提升至57.8（+7.5）
- 适合需要高可靠推理的场景，如数学题求解
大型语言模型（LLMs）在众多任务中表现出色，但有时会产生不良输出。我们认为这些局限源于其基础的自回归架构，缺乏区分优劣结果的能力。受人类认知双重加工理论启发，我们提出LLM2框架，将一个语言模型（系统1）与一个基于过程的验证器（系统2）结合。在该框架中，语言模型负责生成合理候选答案，验证器则通过过程反馈判断输出优劣。验证器使用合成的过程监督数据，基于我们的标记质量探索策略训练，并采用成对比较损失。在数学推理基准上的实证结果表明，LLM2显著有效：在GSM8K上，Llama3-1B的准确率从50.3提升至57.8（+7.5）。进一步结合自一致性方法，major@20准确率从56.2提升至70.2（+14.0）。
原文摘要 · Abstract (English)
Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We posit that these limitations are rooted in the foundational autoregressive architecture of LLMs, which inherently lacks mechanisms for differentiating between desirable and undesirable results. Drawing inspiration from the dual-process theory of human cognition, we introduce LLM2, a novel framework that combines an LLM (System 1) with a process-based verifier (System 2). Within LLM2, the LLM is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs. The verifier is trained with a pairwise comparison loss on synthetic process-supervision data generated through our token quality exploration strategy. Empirical results on mathematical reasoning benchmarks substantiate the efficacy of LLM2, exemplified by an accuracy enhancement from 50.3 to 57.8 (+7.5) for Llama3-1B on GSM8K. Furthermore, when combined with self-consistency, LLM2 achieves additional improvements, boosting major@20 accuracy from 56.2 to 70.2 (+14.0).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。