用轻量微调让大模型直接做地理定位,效果超群且参数少。
Turning Generators into Retrievers: Unlocking MLLMs for Natural Language-Guided Geo-Localization

- 通过优化大模型内部表征,不改架构实现跨模态对齐。
- 在GeoText-1652上文本到图像召回率提升12.2%,多项任务第一。
- 适合想用大模型做地理检索的开发者,部署成本低。
自然语言引导的跨视角地理定位(NGCG)旨在通过地面场景的文本描述检索带有地理标签的卫星图像。现有方法多采用CLIP式双编码器架构,但普遍存在跨模态泛化能力弱、结构复杂的问题。相比之下,多模态大语言模型(MLLMs)具备强大的语义推理能力,却未针对检索任务优化。本文提出一种简单高效的框架,通过参数高效微调将MLLM适配于NGCG。该方法在保持预训练多模态知识的同时,优化模型内部潜在表示,实现强跨模态对齐,无需重设计模型架构。通过系统分析模型主干、特征聚合等变量,提供了可复用的实践洞见。所提方法在GeoText-1652上达到当前最优,文本到图像Recall@1提升12.2%;在CVG-Text的12个子任务中有5项排名第一,且远少于基线模型的可训练参数。结果表明,MLLMs可作为语义跨视角检索的稳健基础,为大模型驱动的NGCG提供了可扩展、高性能的替代方案。项目页面与代码见:https://yuqichen888.github.io/NGCG-MLLMs-web/
原文摘要 · Abstract (English)
Natural-language Guided Cross-view Geo-localization (NGCG) aims to retrieve geo-tagged satellite imagery using textual descriptions of ground scenes. While recent NGCG methods commonly rely on CLIP-style dual-encoder architectures, they often suffer from weak cross-modal generalization and require complex architectural designs. In contrast, Multimodal Large Language Models (MLLMs) offer powerful semantic reasoning capabilities but are not directly optimized for retrieval tasks. In this work, we present a simple yet effective framework to adapt MLLMs for NGCG via parameter-efficient finetuning. Our approach optimizes latent representations within the MLLM while preserving its pretrained multimodal knowledge, enabling strong cross-modal alignment without redesigning model architectures. Through systematic analysis of diverse variables, from model backbone to feature aggregation, we provide practical and generalizable insights for leveraging MLLMs in NGCG. Our method achieves SOTA on GeoText-1652 with a 12.2% improvement in Text-to-Image Recall@1 and secures top performance in 5 out of 12 subtasks on CVG-Text, all while surpassing baselines with far fewer trainable parameters. These results position MLLMs as a robust foundation for semantic cross-view retrieval and pave the way for MLLM-based NGCG to be adopted as a scalable, powerful alternative to traditional dual-encoder designs. Project page and code are available at https://yuqichen888.github.io/NGCG-MLLMs-web/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。