用零样本方法实现全球卫星图像自然语言检索，助力灾情响应
Zero-Shot Satellite Image Retrieval through Joint Embeddings: Application to Crisis Response
- 通过10万张遥感影像生成文本描述，构建语言-视觉对齐的代理数据
- 在76个灾情查询中50公里内准确率达31.6%，洪水场景达50%
- 无需训练即可部署，适合缺乏标注数据的应急响应系统
地球观测档案的语义搜索仍具挑战性。视觉基础模型如CLAY虽能生成丰富遥感图像嵌入，但缺乏自然语言语义支撑，而类似CLIP的遥感联合编码器需成对数据与大规模算力，难以在全球尺度实现。为实现全球范围的自然语言查询，我们提出GeoQuery——一种零样本检索系统，采用两阶段语义与视觉搜索策略，利用全局数据子集（代理）的语言嵌入绕过数据与算力瓶颈。我们为10万张全球哨兵-2影像子集生成文本描述，并优化描述生成提示，使文本嵌入空间距离与冻结的CLAY视觉嵌入空间距离相关联。查询分两步：先在代理子集上进行文本相似性搜索，再在全球范围的CLAY嵌入中进行视觉最近邻搜索。在覆盖英国洪灾、美国山火与干旱的76个灾情地点查询中，GeoQuery在50公里范围内达到31.6%准确率，其中洪灾场景表现最佳（50公里内50%）。该系统已集成至名为ECHO的灾情响应平台，在2025年布里斯班飓风阿尔弗雷德期间成功识别脆弱区域，下游洪水模拟重现了历史模式。提示对齐的代理数据为遥感基础模型与实际检索应用之间提供了可行桥梁。
原文摘要 · Abstract (English)
Semantic search of Earth observation archives remains challenging. Visual foundation models such as CLAY produce rich embeddings of satellite imagery but lack the natural-language grounding needed for intuitive query, and full contrastive training of a remote-sensing CLIP-style model requires paired data and compute that are unavailable at global scale. To allow natural language querying at global scales, we present GeoQuery, a zero-shot retrieval system that sidesteps data and compute constraints through a two-stage semantic and visual search, leveraging a natural language embedding of a subset (proxy) of global data. Rather than training a joint encoder, we generate language descriptions for a 100k proxy subset of global Sentinel-2 tiles and optimise the description-generation prompt so that distances in the resulting text-embedding space correlate with distances in the frozen CLAY visual-embedding space. Queries are resolved in two stages, with a text-similarity search over the proxy subset followed by a visual nearest-neighbour search over worldwide CLAY embeddings On 76 disaster-location queries covering UK floods, US wildfires, and US droughts, GeoQuery achieves 31.6\% accuracy within 50\,km, with the strongest performance on floods (50\% within 50\,km) where terrain features are well captured by RGB embeddings. Deployed within a crisis response system called \ECHO{}, GeoQuery identified vulnerable areas during Brisbane's 2025 Cyclone Alfred, with downstream flood simulations reproducing historical patterns. Prompt-aligned proxies offer a practical bridge between EO foundation models and operational retrieval when full contrastive training is out of reach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。