通过拼接图像生成空间感知文本,缓解视觉语言模型的空间幻觉问题。
Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
- 用图像拼接构造空间结构数据,自动生成带位置信息的描述或问答对。
- 在MME_Position上提升5.50%,Spatial-MM提升4.19%,通用任务性能不降反升。
- 无需人工标注或复杂模型,可直接用于主流视觉语言模型训练。
现有视觉语言模型常出现空间幻觉,即错误描述图像中物体的相对位置。我们认为这主要源于图像与文本之间的不对称性。为此,提出一种简单、无需标注、可即插即用的方法Stitch and Tell(SiTe),通过沿空间轴拼接图像并基于拼接后布局生成空间感知的文本描述或问答对,注入结构化空间监督。在LLaVA-v1.5-7B、LLaVA-Qwen2-1.5B和HALVA-7B三种架构上,使用两个训练数据集和八个基准测试进行评估。结果表明,SiTe显著提升空间理解任务表现,如MME_Position提升5.50%、Spatial-MM提升4.19%,同时保持或改善通用任务性能,如COCO-QA提升1.02%、MMBench提升4.76%。研究说明,显式注入空间结构信息能有效缓解空间幻觉并增强空间理解能力,同时保留通用视觉语言能力。
原文摘要 · Abstract (English)
Existing vision-language models often suffer from spatial hallucinations, i.e., generating incorrect descriptions about the relative positions of objects in an image. We argue that this problem mainly stems from the asymmetric properties between images and text. To enrich the spatial understanding ability of vision-language models, we propose a simple, annotation-free, plug-and-play method named $\text{Stitch and Tell}$ (abbreviated as SiTe), which injects structured spatial supervision into data. It constructs stitched image-text pairs by stitching images along a spatial axis and generating spatially-aware captions or question answer pairs based on the layout of stitched image, without relying on costly advanced models or human involvement. We evaluate SiTe across three architectures including LLaVA-v1.5-7B, LLaVA-Qwen2-1.5B and HALVA-7B, two training datasets, and eight benchmarks. Experiments show that SiTe improves spatial understanding tasks such as $\text{MME}_{\text{Position}}$ (+5.50%) and Spatial-MM (+4.19%), while maintaining or improving performance on general vision-language benchmarks including COCO-QA (+1.02%) and MMBench (+4.76%). Our findings suggest that explicitly injecting spatially-aware structure into training data offers an effective way to mitigate spatial hallucinations and improve spatial understanding, while preserving general vision-language capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。