让开源大模型精准定位图像中物体的局部点，提升细粒度视觉理解能力。
Enhancing Part-Level Point Grounding for Any Open-Source MLLMs
- 通过重构中间层查询，利用注意力机制捕捉目标相关特征。
- 在多个数据集上显著提升部件级点定位准确率，效果稳定。
- 无需微调模型参数，适配任意开源多模态大模型使用。
视觉定位旨在将自由形式的文本查询与图像中的特定区域关联。尽管近期多模态大语言模型（MLLMs）在该领域展现出良好性能，但主要擅长对象级定位，对部件级定位表现不佳——而后者是机器人操作等细粒度任务的关键需求。本文提出一种通用方法，可为任意开源MLLM赋予精确的2D部件级点定位能力，提供比传统表示更直接的替代方案。该方法利用MLLM固有的注意力机制，通过提出的Q-Synth模块在中间层合成文本条件化、定位感知的查询，捕获目标相关的注意力模式，并由轻量级Attention-to-Point解码器将其转化为以点为中心的热力图进行最终预测。所有原始MLLM参数均保持冻结，确保预训练能力完整保留。实验表明，该设计在多个数据集上持续提升部件级定位精度，且可无缝集成至任意开源MLLM中。
原文摘要 · Abstract (English)
Visual grounding aims to associate free-form textual queries with specific regions in an image. While recent Multimodal Large Language Models (MLLMs) have demonstrated promising capabilities in this domain, they primarily excel at object-level grounding and often struggle with part-level grounding-an essential requirement for fine-grained tasks such as robotic manipulation. In this work, we introduce a general approach that equips any open-source MLLMs with accurate 2D part-level point grounding, offering a more direct alternative to conventional grounding representations. Our method leverages the attention mechanisms inherently present in MLLMs. By synthesizing text-conditioned, grounding-aware queries within intermediate layers via the proposed Q-Synth Module, we capture target-relevant attention patterns and refine them with a lightweight Attention-to-Point Decoder, which converts these patterns into a point-centric heatmap for final prediction. Notably, all original MLLM parameters are frozen, ensuring full preservation of their pre-trained capabilities. Experiments show that our design consistently improves part-level grounding accuracy across datasets and can be seamlessly integrated into any open-source MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。