arXiv:2501.09720cs.CVcs.AI2025-01被引 10

首次将多模态大模型用于航拍目标检测,实现与传统检测器相当的性能。

A Simple Aerial Detection Baseline of Multimodal Language Models

  • 将检测框转为文本格式,适配多模态大模型的自回归生成机制。
  • 在多个航拍数据集上达到接近传统检测器的精度,平均mAP达63.2%。
  • 为遥感多模态模型发展提供可复现的基准,适合研究图像理解的新范式。

基于生成式预训练变换器的多模态语言模型(MLMs)被视为统一多领域任务的有力候选。针对遥感(RS)场景的MLMs已在视觉问答和视觉定位等任务中表现优异。除对应指令的特定目标定位外,检测多类别全部目标的航拍目标检测也是遥感基础模型的重要挑战任务。然而,现有遥感MLMs尚未探索此任务,因MLMs的自回归生成机制与检测输出形式差异显著。本文首次提出一个简单基线LMMRotate,将检测输出通过归一化转化为文本,使其兼容MLM框架;并设计公平评估方法,确保与传统检测模型对比的合理性。通过微调开源通用多模态大模型,该基线在多个航拍数据集上达到平均mAP 63.2%,性能媲美传统检测器。代码已公开,旨在为未来遥感多模态模型发展提供参考,推动其对遥感图像的全面理解能力。

原文摘要 · Abstract (English)

The multimodal language models (MLMs) based on generative pre-trained Transformer are considered powerful candidates for unifying various domains and tasks. MLMs developed for remote sensing (RS) have demonstrated outstanding performance in multiple tasks, such as visual question answering and visual grounding. In addition to visual grounding that detects specific objects corresponded to given instruction, aerial detection, which detects all objects of multiple categories, is also a valuable and challenging task for RS foundation models. However, aerial detection has not been explored by existing RS MLMs because the autoregressive prediction mechanism of MLMs differs significantly from the detection outputs. In this paper, we present a simple baseline for applying MLMs to aerial detection for the first time, named LMMRotate. Specifically, we first introduce a normalization method to transform detection outputs into textual outputs to be compatible with the MLM framework. Then, we propose a evaluation method, which ensures a fair comparison between MLMs and conventional object detection models. We construct the baseline by fine-tuning open-source general-purpose MLMs and achieve impressive detection performance comparable to conventional detector. We hope that this baseline will serve as a reference for future MLM development, enabling more comprehensive capabilities for understanding RS images. Code is available at https://github.com/Li-Qingyun/mllm-mmrotate.

遥感检测多模态大模型目标检测文本生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。