让遥感图像理解到像素级,通过标注掩码提升语言指令精度。
Aquila-plus: Prompt-Driven Visual-Language Models for Pixel-Level Remote Sensing Image Understanding
- 引入掩码文本指令微调,将像素级区域融入语言指令。
- 构建10万样本掩码-文本数据集,支持细粒度对齐。
- 适合需要高精度遥感图像解析的研究者与应用开发。
视觉语言模型(VLMs)通过视觉指令微调在遥感图像理解领域取得显著进展,但现有遥感视觉语言模型(RSVLMs)主要关注图像级或帧级理解,难以实现像素级的视觉-语言对齐。此外,缺乏基于掩码的指令数据限制了其进一步发展。本文提出一种名为Aquila-plus的掩码-文本指令微调方法,通过将细粒度掩码区域注入语言指令,拓展了RSVLMs在像素级视觉理解上的能力。首先,我们精心构建了一个包含10万样本的掩码区域-文本数据集;随后,设计了一种视觉语言模型,将像素级表示注入大语言模型(LLM)。具体而言,Aquila-plus采用卷积CLIP作为视觉编码器,并使用掩码感知的视觉提取器从高分辨率输入中提取精确的视觉掩码特征。实验结果表明,Aquila-plus在多种区域理解任务中优于现有方法,展现出像素级指令微调的全新能力。
原文摘要 · Abstract (English)
The recent development of vision language models (VLMs) has led to significant advances in visual-language integration through visual instruction tuning, and they have rapidly evolved in the field of remote sensing image understanding, demonstrating their powerful capabilities. However, existing RSVLMs mainly focus on image-level or frame-level understanding, making it difficult to achieve fine-grained pixel-level visual-language alignment. Additionally, the lack of mask-based instructional data limits their further development. In this paper, we propose a mask-text instruction tuning method called Aquila-plus, which extends the capabilities of RSVLMs to achieve pixel-level visual understanding by incorporating fine-grained mask regions into language instructions. To achieve this, we first meticulously constructed a mask region-text dataset containing 100K samples, and then designed a visual-language model by injecting pixel-level representations into a large language model (LLM). Specifically, Aquila-plus uses a convolutional CLIP as the visual encoder and employs a mask-aware visual extractor to extract precise visual mask features from high-resolution inputs. Experimental results demonstrate that Aquila-plus outperforms existing methods in various region understanding tasks, showcasing its novel capabilities in pixel-level instruction tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。