用文本引导的分步剪枝,高效处理超大遥感图像。
When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruning
- 通过文本感知定位关键视觉区域,动态选择图像块并剪枝冗余特征。
- 在4个数据集上超越现有高分辨率方法,最大图像达27,328像素。
- 适合需要处理超大遥感图像的科研与工业应用。
高效理解大型遥感图像(RSI)具有重要意义但极具挑战性。当前大型视觉语言模型(LVLM)通常采用预定义的有限网格处理图像,导致处理千兆像素级遥感图像时信息损失。而使用无限网格则显著增加计算成本。为在保留图像细节的同时降低计算复杂度,我们提出一种融合动态图像金字塔(DIP)的文本引导令牌剪枝方法。该方法引入:(i) 区域聚焦模块(RFM),利用文本感知区域定位能力识别关键视觉令牌;(ii) 基于DIP的粗粒度到细粒度图像块选择与视觉令牌剪枝策略,由RFM输出引导,避免直接处理整幅大图像。此外,现有评估LVLM在大尺度遥感图像上感知能力的基准存在问题多样性不足和图像尺寸受限。为此,我们构建新基准LRS-VQA,包含8个类别共7,333个问答对,图像长度最高达27,328像素。我们的方法在相同数据下,优于现有高分辨率策略。相较于现有令牌缩减方法,在高分辨率设置下表现出更高效率。数据集与代码见https://github.com/VisionXLab/LRS-VQA。
原文摘要 · Abstract (English)
Efficient vision-language understanding of large Remote Sensing Images (RSIs) is meaningful but challenging. Current Large Vision-Language Models (LVLMs) typically employ limited pre-defined grids to process images, leading to information loss when handling gigapixel RSIs. Conversely, using unlimited grids significantly increases computational costs. To preserve image details while reducing computational complexity, we propose a text-guided token pruning method with Dynamic Image Pyramid (DIP) integration. Our method introduces: (i) a Region Focus Module (RFM) that leverages text-aware region localization capability to identify critical vision tokens, and (ii) a coarse-to-fine image tile selection and vision token pruning strategy based on DIP, which is guided by RFM outputs and avoids directly processing the entire large imagery. Additionally, existing benchmarks for evaluating LVLMs' perception ability on large RSI suffer from limited question diversity and constrained image sizes. We construct a new benchmark named LRS-VQA, which contains 7,333 QA pairs across 8 categories, with image length up to 27,328 pixels. Our method outperforms existing high-resolution strategies on four datasets using the same data. Moreover, compared to existing token reduction methods, our approach demonstrates higher efficiency under high-resolution settings. Dataset and code are in https://github.com/VisionXLab/LRS-VQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。