arXiv:2510.14374cs.CV2025-10ICCV被引 5

通过奖励精准定位的描述,提升多模态大模型的空间理解能力。

Spatial Preference Rewarding for MLLMs Spatial Understanding

  • 引入语义与定位评分,评估模型生成描述的质量和定位准确性。
  • 在标准基准上显著提升空间理解性能,训练开销极小。
  • 适合需要精细空间感知的应用,如视觉对话、图像描述生成。

多模态大语言模型(MLLMs)在参考和定位物体描述方面展现出良好的空间理解能力,但在细粒度空间感知方面仍显不足,例如生成详细区域描述或准确定位物体。此外,它们常无法满足用户对精细空间理解的需求。这可能是因为现有方法主要依赖预标注指令数据进行微调以注入空间知识,而缺乏对模型实际输出的直接监督。为此,本文提出空间偏好奖励(SPR)方法,通过奖励具有精确物体定位的详细回答,抑制模糊或错误的回答,从而增强模型的空间能力。SPR利用随机选取的图像区域及其对应描述,结合语义得分与定位得分,全面评估生成文本的质量与定位精度。进一步地,将高分优化后的描述与低分初始描述配对,进行直接偏好优化,实现与视觉输入的精细对齐。大量实验表明,SPR在多个标准指代与定位基准上有效提升了MLLM的空间理解能力,且训练开销极小。数据与代码将在 https://github.com/hanqiu-hq/SPR 公开。

原文摘要 · Abstract (English)

Multimodal large language models~(MLLMs) have demonstrated promising spatial understanding capabilities, such as referencing and grounding object descriptions. Despite their successes, MLLMs still fall short in fine-grained spatial perception abilities, such as generating detailed region descriptions or accurately localizing objects. Additionally, they often fail to respond to the user's requirements for desired fine-grained spatial understanding. This issue might arise because existing approaches primarily focus on tuning MLLMs to model pre-annotated instruction data to inject spatial knowledge, without direct supervision of MLLMs' actual responses. We address this issue by SPR, a Spatial Preference Rewarding~(SPR) approach that enhances MLLMs' spatial capabilities by rewarding MLLMs' detailed responses with precise object localization over vague or inaccurate responses. With randomly selected image regions and region descriptions from MLLMs, SPR introduces semantic and localization scores to comprehensively evaluate the text quality and localization quality in MLLM-generated descriptions. We also refine the MLLM descriptions with better localization accuracy and pair the best-scored refinement with the initial descriptions of the lowest score for direct preference optimization, thereby enhancing fine-grained alignment with visual input. Extensive experiments over standard referring and grounding benchmarks show that SPR improves MLLM spatial understanding capabilities effectively with minimal overhead in training. Data and code will be released at https://github.com/hanqiu-hq/SPR

空间理解多模态偏好优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。