用标签引导轻量模型,精准生成航拍图像描述
AeroLite: Tag-Guided Lightweight Generation of Aerial Image Captions
- 用GPT-4o生成多数据集伪标题,结合语义标签增强理解
- 在13B大模型上超越基准,参数仅1-3B且开销更低
- 适合资源受限场景下的遥感图像自动标注
准确自动地为航拍图像生成描述对环境监测、城市规划和灾害管理等应用至关重要。然而,由于空间语义复杂且领域差异大,该任务仍具挑战性。为此,我们提出AeroLite,一种轻量级、标签引导的图像描述框架,专为小型语言模型(1–3B参数)设计,具备鲁棒且可解释的遥感图像描述能力。AeroLite利用GPT-4o整合DLRSD、iSAID、LoveDA、WHU和RSSCN7等多个遥感基准数据集,生成大规模、语义丰富的伪标题数据集。通过自然语言处理提取方向、用地类型等关键语义标签,并由专用多标签CLIP编码器学习这些标签,实现精确语义预测。为有效融合视觉与语义信息,我们提出新型桥接多层感知机(MLP)架构,在保持低计算开销的同时对齐语义标签与视觉嵌入。AeroLite支持与多种预训练大语言模型无缝集成。采用两阶段LoRA训练:第一阶段基于伪标题数据集捕捉广义遥感语义,第二阶段在UCM和Sydney Captions等小规模精选数据集上微调,优化领域适配性。实验表明,AeroLite在标准指标(如BLEU、METEOR)上显著优于13B参数的大模型,同时计算成本大幅降低。
原文摘要 · Abstract (English)
Accurate and automated captioning of aerial imagery is crucial for applications like environmental monitoring, urban planning, and disaster management. However, this task remains challenging due to complex spatial semantics and domain variability. To address these issues, we introduce \textbf{AeroLite}, a lightweight, tag-guided captioning framework designed to equip small-scale language models (1--3B parameters) with robust and interpretable captioning capabilities specifically for remote sensing images. \textbf{AeroLite} leverages GPT-4o to generate a large-scale, semantically rich pseudo-caption dataset by integrating multiple remote sensing benchmarks, including DLRSD, iSAID, LoveDA, WHU, and RSSCN7. To explicitly capture key semantic elements such as orientation and land-use types, AeroLite employs natural language processing techniques to extract relevant semantic tags. These tags are then learned by a dedicated multi-label CLIP encoder, ensuring precise semantic predictions. To effectively fuse visual and semantic information, we propose a novel bridging multilayer perceptron (MLP) architecture, aligning semantic tags with visual embeddings while maintaining minimal computational overhead. AeroLite's flexible design also enables seamless integration with various pretrained large language models. We adopt a two-stage LoRA-based training approach: the initial stage leverages our pseudo-caption dataset to capture broad remote sensing semantics, followed by fine-tuning on smaller, curated datasets like UCM and Sydney Captions to refine domain-specific alignment. Experimental evaluations demonstrate that AeroLite surpasses significantly larger models (e.g., 13B parameters) in standard captioning metrics, including BLEU and METEOR, while maintaining substantially lower computational costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。