arXiv:2409.04999cs.CVcs.MM2024-09中稿 · ACM MM 2024 [Oral]被引 28

让视觉编码器根据语言描述动态调整注意力,提升跨模态定位精度。

Visual Grounding with Multi-modal Conditional Adaptation

  • 用多模态嵌入生成权重系数,动态调节视觉编码器
  • 在四个数据集上达到最新最优性能,提升显著
  • 轻量高效,适合实时多模态定位场景

视觉定位旨在定位由自然语言表达指定的物体。现有方法通常采用独立的视觉与文本编码器分别提取特征,再在多模态解码器中融合。然而,同一图像中存在多种语言描述对应不同目标时,传统方法因视觉编码器输出固定特征而表现受限。尽管近期研究提出语言引导的视觉编码器,但大多仅依赖文本信息且结构复杂。本文提出多模态条件自适应(MMCA),使视觉编码器能基于多模态嵌入生成权重系数,动态调整其参数,聚焦于与语言相关的视觉区域。实验在四个常用数据集上验证了该方法的有效性,均取得显著提升与当前最优结果。消融实验进一步证明其轻量与高效。代码已开源。

原文摘要 · Abstract (English)

Visual grounding is the task of locating objects specified by natural language expressions. Existing methods extend generic object detection frameworks to tackle this task. They typically extract visual and textual features separately using independent visual and textual encoders, then fuse these features in a multi-modal decoder for final prediction. However, visual grounding presents unique challenges. It often involves locating objects with different text descriptions within the same image. Existing methods struggle with this task because the independent visual encoder produces identical visual features for the same image, limiting detection performance. Some recently approaches propose various language-guided visual encoders to address this issue, but they mostly rely solely on textual information and require sophisticated designs. In this paper, we introduce Multi-modal Conditional Adaptation (MMCA), which enables the visual encoder to adaptively update weights, directing its focus towards text-relevant regions. Specifically, we first integrate information from different modalities to obtain multi-modal embeddings. Then we utilize a set of weighting coefficients, which generated from the multimodal embeddings, to reorganize the weight update matrices and apply them to the visual encoder of the visual grounding model. Extensive experiments on four widely used datasets demonstrate that MMCA achieves significant improvements and state-of-the-art results. Ablation experiments further demonstrate the lightweight and efficiency of our method. Our source code is available at: https://github.com/Mr-Bigworth/MMCA.

视觉定位多模态学习自适应编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。