让大模型学会自我纠错，迭代优化推理结果
Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
- 通过自批判数据微调，使模型在推理中主动反思并改进答案
- 在AIME难题上，正确率从4.4%提升至18.2%
- 适合需要高可靠推理的场景，如数学证明与复杂决策
尽管慢思考大型语言模型具备类似反思的推理能力，常被称为“顿悟时刻”，但其生成有信息量的批判性意见并改进先前解法的能力仍有限。本文提出Double-Checker，一种旨在通过显式自批判和迭代优化前期输出来增强慢思考大模型推理能力的原则性框架。基于我们精心构建的1,730个自批判实例进行微调，Double-Checker使长思维链（long-CoT）模型在推理过程中可迭代地批判并修正自身输出，直至在自生成的批判下判定解法正确。我们在一系列综合性推理基准上验证了该方法的有效性，结果显示，迭代自批判显著提升了长思维链模型的推理能力。尤其在挑战性的AIME基准上，相比原始长思维链模型，pass@1性能从4.4%提升至18.2%。这些结果揭示了发展具备结构化自批判能力的更可信、高效大模型的潜在方向。代码与数据已公开于https://github.com/XinXU-USTC/DoubleChecker。
原文摘要 · Abstract (English)
While slow-thinking large language models (LLMs) exhibit reflection-like reasoning, commonly referred to as the "aha moment:, their ability to generate informative critiques and refine prior solutions remains limited. In this paper, we introduce Double-Checker, a principled framework designed to enhance the reasoning capabilities of slow-thinking LLMs by fostering explicit self-critique and iterative refinement of their previous solutions. By fine-tuning on our curated 1,730 self-critical instances, Double-Checker empowers long-CoT LLMs to iteratively critique and refine their outputs during inference until they evaluate their solutions as correct under self-generated critiques. We validate the efficacy of Double-Checker across a comprehensive suite of reasoning benchmarks, demonstrating that iterative self-critique significantly enhances the reasoning capabilities of long-CoT LLMs. Notably, our Double-Checker increases the pass@1 performance on challenging AIME benchmarks from 4.4% to 18.2% compared to the original long-CoT LLMs. These results highlight a promising direction for developing more trustworthy and effective LLMs capable of structured self-critique. Our codes and data are available at https://github.com/XinXU-USTC/DoubleChecker
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。