让大模型画图思考,显著提升复杂任务规划能力
Visualizing Thought: Conceptual Diagrams Enable Robust Planning in LMMs
- 让大模型自动生成概念图进行多步推理
- 在积木世界等任务上准确率从35.5%提至90.2%
- 适合需要深度规划的复杂决策场景
人类推理依赖于构建和操作心理模型——对情境的简化内部表征,用于理解与解决问题。概念图(如辅助思考的草图)能外部化这些心理模型,抽象无关细节以高效捕捉实体间交互关系。相比之下,大语言模型(LLMs)和大模态模型(LMMs)主要通过文本推理,在复杂多步任务中表现受限。本文提出视觉思维(Visual Thinking)框架,使LMMs可通过自生成的概念图链进行多轮推理,显著增强其组合规划能力。该方法仅需任务的自然语言描述,无需人工干预。它在优化的思维图推理框架中融合文本与图示推理,结合束搜索与逐层回溯。在多个挑战性PDDL规划领域评估中,性能大幅提升(如GPT-4o在积木世界任务中从35.5%提升至90.2%),且持续优于纯文本搜索方法。在解深达40步的更难领域,亦超越o1-preview模型(如地板瓷砖任务提升16个百分点)。结果表明,概念图作为推理媒介在LMMs中具有强大潜力。
原文摘要 · Abstract (English)
Human reasoning relies on constructing and manipulating mental models -- simplified internal representations of situations used to understand and solve problems. Conceptual diagrams (e.g., a sketch drawn to aid reasoning) externalize these mental models, abstracting irrelevant details to efficiently capture how entities interact. In contrast, Large Language Models (LLMs) and Large MultiModal Models (LMMs) predominantly reason through text, limiting their effectiveness on complex multi-step tasks. In this paper, we propose Visual Thinking, a generalizable framework that enables LMMs to reason through multiple chains of self-generated conceptual diagrams, significantly enhancing their combinatorial planning capabilities. Our approach requires no human input beyond the natural language description of the task. It integrates textual and diagrammatic reasoning within an optimized Graph-of-Thought inference framework, enhanced by beam search and depth-wise backtracking. Evaluated on multiple challenging PDDL planning domains, our method substantially improves LMM performance (e.g., GPT-4o: 35.5% -> 90.2% in Blocksworld) and consistently outperforms text-only search-based inference methods. On more difficult domains with solution depths up to 40, it also surpasses the o1-preview reasoning model (e.g., 16 percentage points improvement in Floor Tiles). These results demonstrate the power of conceptual diagrams as a reasoning medium in LMMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。