无需训练即可生成多语言遥感图像描述,靠检索增强提示实现跨语言泛化。
Multilingual Training-Free Remote Sensing Image Captioning

- 用检索增强提示,从数据池中找相关描述和示例,不需训练模型。
- 在10种语言上表现接近全监督英文系统,图谱重排提升35%性能。
- 直接生成目标语言描述优于翻译策略,适合无标注数据场景。
遥感图像描述生成虽借助编码器-解码器模型快速发展,但依赖大量标注数据且仅限英文,限制了全球应用。为此,本文提出首个无需训练的多语言方法,基于检索增强提示。给定航空图像时,使用领域适配的SigLIP2编码器从数据存储中检索相关描述与少样本示例,再提供给语言模型。探索两种变体:仅依赖文本提示的多语言大语言模型(LLM)方案,以及联合处理提示与图像的视觉-语言模型(VLM)方案。为提升检索内容一致性,引入基于图结构的PageRank重排序策略。在四个基准数据集、十种语言上的实验表明,该方法性能可媲美全监督英文系统,并具备良好跨语言泛化能力。结果还显示,使用PageRank重排序可带来最高达35%的指标提升。此外,尽管VLM生成更具视觉关联性但词汇多样性高,而LLM在BLEU和CIDEr指标上表现更优。直接在目标语言生成描述始终优于翻译策略。本工作首次系统评估了遥感图像的多语言、免训练描述生成,推动更具包容性与可扩展性的多模态地球观测体系发展。
原文摘要 · Abstract (English)
Remote sensing image captioning has advanced rapidly through encoder--decoder models, although the reliance on large annotated datasets and the focus on English restricts global applicability. To address these limitations, we propose the first training-free multilingual approach, based on retrieval-augmented prompting. For a given aerial image, we employ a domain-adapted SigLIP2 encoder to retrieve related captions and few-shot examples from a datastore, which are then provided to a language model. We explore two variants: an image-blind setup, where a multilingual Large Language Model (LLM) generates the caption from textual prompts alone, and an image-aware setup, where a Vision--Language Model (VLM) jointly processes the prompt and the input image. To improve the coherence of the retrieved content, we introduce a graph-based re-ranking strategy using PageRank on a graph of images and captions. Experiments on four benchmark datasets across ten languages demonstrate that our approach is competitive with fully supervised English-only systems and generalizes to other languages. Results also highlight the importance of re-ranking with PageRank, yielding up to 35% improvements in performance metrics. Additionally, it was observed that while VLMs tend to generate visually grounded but lexically diverse captions, LLMs can achieve stronger BLEU and CIDEr scores. Lastly, directly generating captions in the target language consistently outperforms other translation-based strategies. Overall, our work delivers one of the first systematic evaluations of multilingual, training-free captioning for remote sensing imagery, advancing toward more inclusive and scalable multimodal Earth observation systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。