让大模型自己学会推理，无需外部反馈就能提升复杂问题解决能力。
Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding
- 通过变分优化框架，让模型在训练中自主学习推理过程。
- 在GSM8K上零样本准确率平均提升12.5%，超过监督微调9.6%。
- 适用于多种模型架构，适合想提升推理能力的研究者使用。
大型语言模型虽具强大能力，但在需要多步推理的复杂任务上仍表现不足。尽管提示工程方法如思维链（CoT）可在推理时提升性能，但训练阶段优化推理能力仍具挑战。本文提出LaTent Reasoning Optimization（LaTRO），将推理建模为从隐式分布中采样，并通过变分方法进行优化。该方法使模型在无需外部反馈或奖励模型的情况下，同时提升推理过程与评估推理质量的能力。我们在GSM8K和ARC-Challenge数据集上验证了该方法，使用Phi-3.5-mini、Mistral-7B和Llama-3.1-8B等多种模型架构。在GSM8K上，LaTRO相较基础模型零样本准确率平均提升12.5%，较监督微调提升9.6%。结果表明，预训练大模型具有可被激活和增强的潜在推理能力，可通过本方法实现自我改进。代码已开源： https://github.com/SalesforceAIResearch/LaTRO。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. While prompt-based methods like Chain-of-Thought (CoT) can improve LLM reasoning at inference time, optimizing reasoning capabilities during training remains challenging. We introduce LaTent Reasoning Optimization (LaTRO), a principled framework that formulates reasoning as sampling from a latent distribution and optimizes it via variational approaches. LaTRO enables LLMs to concurrently improve both their reasoning process and ability to evaluate reasoning quality, without requiring external feedback or reward models. We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures. On GSM8K, LaTRO improves zero-shot accuracy by an average of 12.5% over base models and 9.6% over supervised fine-tuning across Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B. Our findings suggest that pre-trained LLMs possess latent reasoning capabilities that can be unlocked and enhanced through our proposed optimization approach in a self-improvement manner. The code of LaTRO is available at \url{https://github.com/SalesforceAIResearch/LaTRO}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。