统一遥感多模态嵌入,实现高精度检索与区域推理
VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing
- 单编码器融合图像、文本、框坐标与地理坐标,对比学习生成统一向量
- 在遥感基准上实现26.6%的区域图文检索准确率,超基线25个百分点
- 适合需要跨模态检索与空间推理的遥感分析任务,如地理定位与场景理解
卫星图像与自然图像本质不同:其俯视视角、极高分辨率、多尺度变化及大量小目标,要求兼具区域级空间推理与整体场景理解。现有遥感方法在双编码器检索模型(擅长大规模跨模态搜索但无法融合模态)与生成式助手(支持区域解释但缺乏可扩展检索)间割裂。本文提出VLM2GeoVec,一种指令跟随的单编码器视觉语言模型,通过对比学习将图像、文本、边界框和地理坐标联合嵌入统一向量空间。该单编码器将所有输入交织后生成联合嵌入,避免多阶段流水线与任务特定模块。为评估其通用性,我们构建新基准RSMEB,涵盖遥感嵌入关键应用:场景分类、跨模态搜索、组合检索、视觉问答、视觉定位与区域推理、语义地理检索。在RSMEB上,其区域-标题检索达到26.6% P@1(较双编码器基线提升25个百分点),指代表达检索达32.5% P@1(+19个百分点),语义地理定位检索达17.8% P@1(超过之前最佳结果三倍以上),同时在场景分类与跨模态检索等传统任务上与专用模型持平或超越。VLM2GeoVec统一了可扩展检索与区域级空间推理,实现遥感中连贯的多模态分析。代码、模型与数据将在论文录用后公开。
原文摘要 · Abstract (English)
Satellite imagery differs fundamentally from natural images: its aerial viewpoint, very high resolution, diverse scale variations, and abundance of small objects demand both region-level spatial reasoning and holistic scene understanding. Current remote-sensing approaches remain fragmented between dual-encoder retrieval models, which excel at large-scale cross-modal search but cannot interleave modalities, and generative assistants, which support region-level interpretation but lack scalable retrieval capabilities. We propose $\textbf{VLM2GeoVec}$, an instruction-following, single-encoder vision-language model trained contrastively to embed interleaved inputs (images, text, bounding boxes, and geographic coordinates) in a unified vector space. Our single encoder interleaves all inputs into one joint embedding trained with a contrastive loss, eliminating multi-stage pipelines and task-specific modules. To evaluate its versatility, we introduce $\textbf{RSMEB}$, a novel benchmark covering key remote-sensing embedding applications: scene classification; cross-modal search; compositional retrieval; visual-question answering; visual grounding and region-level reasoning; and semantic geospatial retrieval. On RSMEB, it achieves $\textbf{26.6%}$ P@1 on region-caption retrieval (+25 pp vs. dual-encoder baselines), $\textbf{32.5%}$ P@1 on referring-expression retrieval (+19 pp), and $\textbf{17.8%}$ P@1 on semantic geo-localization retrieval (over $3\times$ prior best), while matching or exceeding specialized baselines on conventional tasks such as scene classification and cross-modal retrieval. VLM2GeoVec unifies scalable retrieval with region-level spatial reasoning, enabling cohesive multimodal analysis in remote sensing. We will publicly release the code, checkpoints, and data upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。