arXiv:2502.13170cs.AIcs.LG2025-02ICLR被引 13

提出代码推理新任务,用迭代修正法提升大模型逻辑推理能力。

Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment

  • 通过假设分解与修正的迭代流程,模拟人类解题思路。
  • 在8个基准任务上实现最高3倍性能提升,缓解多步推理中的遗忘问题。
  • 适用于需要复杂推理与容错的智能体任务,如虚拟家庭场景应用。

推理能力是大语言模型最神秘且引人注目的特性之一。尽管众多研究致力于拓展其推理边界,但兼具推理与记忆特征的任务常被忽视。本文引入一种新任务——代码推理,为大模型推理能力提供新视角。基于经典逻辑推理形式,我们构建了三个元基准,并细化为八个具体任务。测试显示,大模型在识别有效推理路径方面仍表现不佳。为此,我们提出受人类解题启发的反思性假设分解与修正(RHDA)流程:首先基于观察分解假设;其次利用工具验证与反思;最后根据结果修正假设。该方法有效缓解多步推理中的遗忘与幻觉导致的链路崩溃,性能提升达3倍。此外,我们将该流程应用于虚拟家庭环境(VirtualHome)中模拟复杂家务任务,显著增强对失败情况的处理能力。代码与结果已开源。

原文摘要 · Abstract (English)

The reasoning abilities are one of the most enigmatic and captivating aspects of large language models (LLMs). Numerous studies are dedicated to exploring and expanding the boundaries of this reasoning capability. However, tasks that embody both reasoning and recall characteristics are often overlooked. In this paper, we introduce such a novel task, code reasoning, to provide a new perspective for the reasoning abilities of LLMs. We summarize three meta-benchmarks based on established forms of logical reasoning, and instantiate these into eight specific benchmark tasks. Our testing on these benchmarks reveals that LLMs continue to struggle with identifying satisfactory reasoning pathways. Additionally, we present a new pathway exploration pipeline inspired by human intricate problem-solving methods. This Reflective Hypothesis Decomposition and Amendment (RHDA) pipeline consists of the following iterative steps: (1) Proposing potential hypotheses based on observations and decomposing them; (2) Utilizing tools to validate hypotheses and reflection outcomes; (3) Revising hypothesis in light of observations. Our approach effectively mitigates logical chain collapses arising from forgetting or hallucination issues in multi-step reasoning, resulting in performance gains of up to $3\times$. Finally, we expanded this pipeline by applying it to simulate complex household tasks in real-world scenarios, specifically in VirtualHome, enhancing the handling of failure cases. We release our code and all of results at https://github.com/TnTWoW/code_reasoning.

代码推理逻辑推理大模型智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。