用大模型动态优化遥感图像检索的文本嵌入,无需训练即可提升准确率。
Open-SAT: LLM-Guided Query Embedding Refinement for Open-Vocabulary Object Retrieval in Satellite Imagery

- 利用大模型在推理时动态优化查询文本嵌入,增强与遥感图像的语义对齐。
- 在三个公开数据集上最高提升F1分数16.04%,检索结果数量相当。
- 无需额外训练,适合需要快速适应新查询场景的遥感应用开发者。
在遥感应用中,用户查询常以开放式自然语言形式出现,超出预定义类别范围。这种开放词汇特性给图像片元检索带来挑战,因系统需泛化至大量未见物体和概念。尽管视觉-语言模型(如CLIP)广泛用于图文检索,其微调变体仍难以精准对齐此类查询与卫星影像。为此,我们提出Open-SAT,一种无需训练的推理时查询嵌入优化算法,通过大型语言模型(LLMs)结合目标对象及其环境的上下文信息,改进文本嵌入以提升与卫星图像内容的匹配度。该方法将图像片元的嵌入存入向量数据库,实现高效检索,并采用无阈值机制进一步提高准确性和效率。在三个公开基准上的实验表明,Open-SAT在保持相似检索数量的前提下,最高使F1得分提升16.04%。结果证明,Open-SAT在开放词汇遥感图像检索中有效,借助大模型引导而无需额外训练或监督。
原文摘要 · Abstract (English)
In satellite applications, user queries often take the form of open-ended natural language, extending beyond a fixed set of predefined categories. This open-vocabulary nature poses significant challenges for retrieving relevant image tiles, as the retrieval system must generalize to a wide range of unseen objects and concepts. While vision-language models (VLMs) such as CLIP are widely used for text-image retrieval, even fine-tuned variants often struggle to accurately align such queries with satellite imagery. To address this, we propose Open-SAT, a training-free query embedding refinement algorithm that operates at inference time to improve alignment between user queries and satellite image content. Open-SAT uses VLMs to compute embeddings for image tiles, which are stored in a vector database for efficient retrieval. At query time, it leverages Large Language Models (LLMs) to refine the text embeddings by incorporating contextual information about objects of interest and their surroundings. A threshold-free retrieval mechanism further enhances accuracy and efficiency. Experimental results in three public benchmarks demonstrate that Open-SAT improves the F1 score by up to 16.04%, while retrieving a comparable number of image tiles. These results demonstrate the effectiveness of Open-SAT in open-vocabulary satellite image retrieval, leveraging LLM guidance without the need for additional training or supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。