无需训练即可实现遥感图像零样本开放词汇定位,借助大模型直接推理。
RSVG-ZeroOV: Exploring a Training-Free Framework for Zero-Shot Open-Vocabulary Visual Grounding in Remote Sensing Images
- 利用视觉语言模型生成文本与图像区域的语义关联图
- 通过扩散模型补全物体的结构和形状信息,提升定位精度
- 设计注意力演化模块过滤无关激活,输出纯净分割掩码
遥感视觉定位(RSVG)旨在基于自然语言表达定位遥感图像中的目标。现有方法通常局限于封闭词汇集,难以适应开放世界场景。尽管近期研究尝试利用通用基础模型实现开放词汇RSVG,但过度依赖高质量数据集和耗时的微调。为此,本文提出训练免去的框架 RSVG-ZeroOV,探索冻结通用基础模型在零样本开放词汇RSVG中的潜力。该框架包含三个关键阶段:(i) 概览:使用视觉语言模型(VLM)获取跨注意力图,捕捉文本查询与视觉区域间的语义关联;(ii) 聚焦:利用扩散模型(DM)的细粒度建模先验,补全常被VLM忽略的物体结构与形状信息;(iii) 演化:引入简洁有效的注意力演化模块,抑制无关激活,生成针对目标的纯净分割掩码。无需任务特定训练,该方法兼具高效性与可扩展性。大量实验表明,所提框架持续优于现有弱监督与零样本方法。
原文摘要 · Abstract (English)
Remote sensing visual grounding (RSVG) aims to localize objects in remote sensing images based on free-form natural language expressions. Existing approaches are typically constrained to closed-set vocabularies, limiting their applicability in open-world scenarios. While recent attempts to leverage generic foundation models for open-vocabulary RSVG, they overly rely on expensive high-quality datasets and time-consuming fine-tuning. To address these limitations, we propose \textbf{RSVG-ZeroOV}, a training-free framework that aims to explore the potential of frozen generic foundation models for zero-shot open-vocabulary RSVG. Specifically, RSVG-ZeroOV comprises three key stages: (i) Overview: We utilize a vision-language model (VLM) to obtain cross-attention\footnote[1]{In this paper, although decoder-only VLMs use self-attention over all tokens, we refer to the image-text interaction part as cross-attention to distinguish it from pure visual self-attention.}maps that capture semantic correlations between text queries and visual regions. (ii) Focus: By leveraging the fine-grained modeling priors of a diffusion model (DM), we fill in gaps in structural and shape information of objects, which are often overlooked by VLM. (iii) Evolve: A simple yet effective attention evolution module is introduced to suppress irrelevant activations, yielding purified segmentation masks over the referred objects. Without cumbersome task-specific training, RSVG-ZeroOV offers an efficient and scalable solution. Extensive experiments demonstrate that the proposed framework consistently outperforms existing weakly-supervised and zero-shot methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。