用强化学习引导大模型非线性推理，提升复杂问题求解能力
Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoning
- 强化学习代理基于领域信息评估方案质量，弥补大模型训练目标不足
- 在程序等价性任务中准确率达87.3%，优于CoT和ToT方法
- 支持多路径探索与回溯，适合需要长程规划的逻辑推理场景
大语言模型在长期规划方面表现不佳，可能源于其对解空间的探索方式受限。本文提出一种新架构：由强化学习（RL）代理引导大模型的解空间探索。首先，该代理可访问领域特定信息，能基于具体相关指标评估候选方案质量，这些指标未被大模型训练目标显式包含；其次，大模型仅需生成下一步行动，无需承担长期规划负担。通过探索多条路径并支持回溯，实现非线性推理。我们在程序等价性任务上评估该架构，对比链式思维（CoT）和思维树（ToT）。评估涵盖下游任务（二分类）及中间推理步骤。结果表明，该方法在准确率和推理质量上均优于CoT与ToT。
原文摘要 · Abstract (English)
Large Language Models (LLMs) were shown to struggle with long-term planning, which may be caused by the limited way in which they explore the space of possible solutions. We propose an architecture where a Reinforcement Learning (RL) Agent guides an LLM's space exploration: (1) the Agent has access to domain-specific information, and can therefore make decisions about the quality of candidate solutions based on specific and relevant metrics, which were not explicitly considered by the LLM's training objective; (2) the LLM can focus on generating immediate next steps, without the need for long-term planning. We allow non-linear reasoning by exploring alternative paths and backtracking. We evaluate this architecture on the program equivalence task, and compare it against Chain of Thought (CoT) and Tree of Thoughts (ToT). We assess both the downstream task, denoting the binary classification, and the intermediate reasoning steps. Our approach compares positively against CoT and ToT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。