无需训练即可增强大模型的空间理解能力
CoCoSI: Collaborative Cognitive Map Construction for Spatial Intelligence
- 多智能体协作构建结构化空间记忆图
- 在多个空间任务上显著提升理解性能
- 适用于任意预训练大模型，无需修改架构
空间智能是多模态大语言模型（MLLMs）的重要前沿，使其能够从视觉经验中推理物理世界。受人类空间认知启发，现有方法通过多帧视觉输入构建网格状认知地图，以保持时空一致性。然而，受限的上下文长度仍制约空间理解，而长上下文建模或外部记忆等方法通常需改变架构、引入记忆模块或微调，限制了其在现成预训练模型上的应用。为此，我们提出一种轻量级、模型无关的即插即用多智能体框架，通过协作构建认知地图作为结构化空间记忆，无需任何架构修改或额外训练即可增强任意预训练MLLM的空间理解能力。框架包含局部-全局智能体协同、原子化提交的地图构建以及跨智能体验证机制。大量实验表明，该方法在空间理解任务上表现优异，且完全无需训练。代码将开源。
原文摘要 · Abstract (English)
Spatial intelligence is a key frontier for multimodal large language models (MLLMs), enabling them to reason about the physical world from visual experience. Inspired by human spatial cognition, recent approaches construct grid-based cognitive maps from multi-frame visual inputs to maintain coherent spatial representations over time. However, limited context lengths still challenge spatial understanding, while existing methods, such as long-context modeling and external memory, often require architectural changes, memory modules, or finetuning, limiting their applicability to off-the-shelf pretrained MLLMs. This motivates a lightweight, model-agnostic method for preserving spatial information beyond the native context window. To this end, we propose a plug-and-play multi-agent framework that collaboratively constructs cognitive maps as structured spatial memory, enhancing the spatial understanding of arbitrary pretrained MLLMs without architectural modification or additional training. Our framework features local-global agent coordination, cognitive map construction with atomic commits, and cross-agent verification. Extensive experiments demonstrate that our method achieves superior performance on spatial understanding tasks while remaining fully training-free. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。