不依赖训练,自动检测并修正多模态大模型的空间推理错误。
Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs

- 构建空间证据图,关联推理步骤与视觉信息
- 通过可靠性评估发现错误源头,平均提升8.55%准确率
- 适合希望提升推理可信度的研究者与应用开发者
尽管多模态大语言模型(MLLMs)取得显著进展,其空间推理仍可能产生与输入图像不一致的中间判断,导致错误在推理链中传播并影响最终答案。现有方法主要通过训练或增加空间信息改进,未关注推理过程是否忠实于模型输入。本研究发现,不忠实的推理链会显著降低最终答案准确率。为此,我们提出一种模块化、无需训练的空间推理验证与修正框架。该框架构建空间证据图(SEG),将思维链推理中提取的原子空间证据与视觉实体、空间关系、来源步骤及视觉证据相关联。空间证据可靠性评估(SERA)基于物体存在性、定位精度和几何测量评估视觉证据可靠性。框架识别出与可靠视觉证据矛盾的最早证据单元,并引导原始MLLM修正后续推理与最终答案。在15个模型-数据集组合上,该方法平均准确率达68.94%,比基线平均高出8.55个百分点。代码将开源。
原文摘要 · Abstract (English)
Although Multimodal Large Language Models (MLLMs) have made substantial progress, their spatial reasoning may still produce intermediate judgments inconsistent with the input image, allowing errors to propagate through the reasoning chain and affect the final answer. Existing methods mainly improve spatial reasoning through training or additional spatial information, without considering whether the reasoning process itself is faithful to the model input. Our study shows that unfaithful reasoning chains significantly reduce final-answer accuracy. To address this issue, we propose a modular and training-free framework for spatial reasoning verification and correction. The framework constructs a Spatial Evidence Graph (SEG), which associates atomic spatial evidence extracted from Chain-of-Thought reasoning with visual entities, spatial relations, source steps, and visual evidence. Spatial Evidence Reliability Assessment (SERA) evaluates the reliability of visual evidence based on object existence, localization, and geometric measurements. The framework then identifies the earliest spatial evidence unit contradicted by reliable visual evidence and guides the original MLLM to revise the subsequent reasoning and final answer. Across 15 model-dataset settings, our method achieves an average accuracy of 68.94%, outperforming the compared baselines by 8.55 percentage points on average. Our code will be open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。