arXiv:2502.16680cs.CV2025-02被引 13

针对无人机图像的指代分割,提出新模型与自动标注方法。

AeroReformer: Aerial Referring Transformer for UAV-based Referring Image Segmentation

  • 用大模型自动生成无人机图像的文本描述和像素标注。
  • 新模型在两个新数据集上准确率显著优于现有方法。
  • 适合做低空视觉理解、无人机智能导航的研究者参考。

指代分割将计算机视觉与自然语言处理结合,根据文本描述定位并分割目标物体。尽管自然图像中的指代分割已有广泛研究,但针对无人机(UAV)航拍图像的工作极少。由于复杂空间尺度、遮挡和物体方向多变,现有方法难以适用。主要瓶颈在于缺乏专用于无人机的标注数据集——人工标注像素级掩码并生成文本描述耗时费力。为此,我们设计了一种自动化标注流程,利用已有无人机分割数据集和多模态大模型(MLLM)自动生成文本描述。同时提出Aerial Referring Transformer(AeroReformer)框架,包含视觉-语言交叉注意力模块(VLCAM)实现跨模态理解,以及旋转感知多尺度融合解码器(RAMSF)提升航拍场景分割精度。在两个新构建的数据集上大量实验表明,AeroReformer性能显著优于现有方法,建立了无人机指代分割的新基准。代码与数据集将公开于:https://github.com/lironui/AeroReformer。

原文摘要 · Abstract (English)

As a novel and challenging task, referring segmentation combines computer vision and natural language processing to localize and segment objects based on textual descriptions. While referring image segmentation (RIS) has been extensively studied in natural images, little attention has been given to aerial imagery, particularly from unmanned aerial vehicles (UAVs). The unique challenges of UAV imagery, including complex spatial scales, occlusions, and varying object orientations, render existing RIS approaches ineffective. A key limitation has been the lack of UAV-specific datasets, as manually annotating pixel-level masks and generating textual descriptions is labour-intensive and time-consuming. To address this gap, we design an automatic labelling pipeline that leverages pre-existing UAV segmentation datasets and Multimodal Large Language Models (MLLM) for generating textual descriptions. Furthermore, we propose Aerial Referring Transformer (AeroReformer), a novel framework for UAV referring image segmentation (UAV-RIS), featuring a Vision-Language Cross-Attention Module (VLCAM) for effective cross-modal understanding and a Rotation-Aware Multi-Scale Fusion (RAMSF) decoder to enhance segmentation accuracy in aerial scenes. Extensive experiments on two newly developed datasets demonstrate the superiority of AeroReformer over existing methods, establishing a new benchmark for UAV-RIS. The datasets and code will be publicly available at: https://github.com/lironui/AeroReformer.

无人机指代分割多模态自动标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。