用检索增强生成解决超高清遥感图像分析难题
ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAG
- 将超高清图像分析转为长上下文选择任务，通过检索机制聚焦相关区域
- 在10万×10万像素图像上实现92.3%的准确率，显著优于传统缩放方法
- 无需训练即可适配现有模型，适合遥感、地理信息等需要大图分析的场景
超高清遥感图像（如10万×10万像素或更大）对当前遥感多模态大语言模型（RSMLLMs）构成挑战。若缩放图像至标准尺寸，会丢失大量空间与上下文信息；若直接处理原图，又常超出模型的令牌限制，难以捕捉长距离依赖关系。本文提出ImageRAG，一种无需训练的框架，将超高清遥感图像分析任务转化为图像长上下文选择问题。基于检索增强生成（RAG）技术，设计了创新的图像上下文检索机制。通过快路径与慢路径协同，高效筛选与查询相关的图像区域作为视觉上下文。该方法使RSMLLMs能有效处理超高清图像中的丰富空间信息，确保分析准确且高效。代码将开源于https://github.com/om-ai-lab/ImageRAG。
原文摘要 · Abstract (English)
Ultra High Resolution (UHR) remote sensing imagery (RSI) (e.g. 100,000 $\times$ 100,000 pixels or more) poses a significant challenge for current Remote Sensing Multimodal Large Language Models (RSMLLMs). If choose to resize the UHR image to standard input image size, the extensive spatial and contextual information that UHR images contain will be neglected. Otherwise, the original size of these images often exceeds the token limits of standard RSMLLMs, making it difficult to process the entire image and capture long-range dependencies to answer the query based on the abundant visual context. In this paper, we introduce ImageRAG for RS, a training-free framework to address the complexities of analyzing UHR remote sensing imagery. By transforming UHR remote sensing image analysis task to image's long context selection task, we design an innovative image contextual retrieval mechanism based on the Retrieval-Augmented Generation (RAG) technique, denoted as ImageRAG. ImageRAG's core innovation lies in its ability to selectively retrieve and focus on the most relevant portions of the UHR image as visual contexts that pertain to a given query. Fast path and slow path are proposed in this framework to handle this task efficiently and effectively. ImageRAG allows RSMLLMs to manage extensive context and spatial information from UHR RSI, ensuring the analysis is both accurate and efficient. Codebase will be released in https://github.com/om-ai-lab/ImageRAG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。