让大模型学会空间定位，还不丢掉原有能力。
TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning
- 用双专家渐进式微调，保留原知识并新增视觉定位能力
- 在多个基准上表现优异，零样本定位准确率超基线
- 适合需要精准视觉理解的机器人与交互系统
空间感知是实现具身多模态人工智能系统的关键。然而，在缺乏大量空间标注数据的情况下，当前多模态大语言模型（MLLM）在此任务上表现不佳。本文提出TWIST & SCOUT框架，使预训练的MLLM在不遗忘原有图像与语言理解能力的前提下，获得视觉定位能力。为此，我们设计了TWIST——一种双专家渐进微调模块，通过一个冻结的图像理解模块和一个可学习的视觉定位模块共同作用于语言模型解码器，实现知识保留与新技能获取。为高效微调，我们构建了一个名为SCOUT的高质量合成数据集，模拟人类在视觉定位中的逐步推理过程，提供丰富的多模态推理监督信号。我们在多个标准基准数据集上评估该方法，涵盖基于场景的图像描述、零样本定位及视觉定位任务。实验结果表明，该方法在所有任务中均表现强劲，同时完整保留了预训练模型的图像理解能力。
原文摘要 · Abstract (English)
Spatial awareness is key to enable embodied multimodal AI systems. Yet, without vast amounts of spatial supervision, current Multimodal Large Language Models (MLLMs) struggle at this task. In this paper, we introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability without forgetting their existing image and language understanding skills. To this end, we propose TWIST, a twin-expert stepwise tuning module that modifies the decoder of the language model using one frozen module pre-trained on image understanding tasks and another learnable one for visual grounding tasks. This allows the MLLM to retain previously learned knowledge and skills, while acquiring what is missing. To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT, which mimics human reasoning in visual grounding. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process, thereby simplifying the task of visual grounding. We evaluate our approach on several standard benchmark datasets, encompassing grounded image captioning, zero-shot localization, and visual grounding tasks. Our method consistently delivers strong performance across all tasks, while retaining the pre-trained image understanding capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。