arXiv:2411.01445cs.CV2024-11被引 2

无需额外数据与微调,用目标检测+视觉语言模型实现雷达船舰问答。

A Visual Question Answering Method for SAR Ship: Breaking the Requirement for Multimodal Dataset Construction and Model Fine-Tuning

  • 用YOLOv8n检测船舰,结合Qwen2-VL生成多轮对话与场景描述。
  • 在SSDD和HRSID数据集上检测准确率提升,支持复杂问答任务。
  • 适合遥感图像分析、智能航运监控等无需标注数据的场景。

当前视觉问答(VQA)任务通常需要构建多模态数据集并微调视觉语言模型,耗费大量时间和资源,严重限制了其在下游任务中的应用,例如基于合成孔径雷达(SAR)影像的船舰信息分析。为此,本文提出一种新方法,将目标检测网络与视觉语言模型集成,专门用于分析SAR图像中的船舰。首先,在两个代表性SAR船舰检测数据集SSDD和HRSID上对YOLO系列模型进行基线实验,评估检测精度;结果表明,YOLOv8n表现最优,被选为本任务的最佳检测网络。随后,基于视觉语言模型Qwen2-VL,设计并实现针对SAR场景的VQA任务,利用检测网络输出的船舰位置与尺寸信息,生成多轮对话和场景描述。实验表明,该方法不仅无需额外数据集或模型微调即可完成基础的SAR场景问答,还能动态适应复杂的多轮对话需求,展现出强大的语义理解与适应能力。

原文摘要 · Abstract (English)

Current visual question answering (VQA) tasks often require constructing multimodal datasets and fine-tuning visual language models, which demands significant time and resources. This has greatly hindered the application of VQA to downstream tasks, such as ship information analysis based on Synthetic Aperture Radar (SAR) imagery. To address this challenge, this letter proposes a novel VQA approach that integrates object detection networks with visual language models, specifically designed for analyzing ships in SAR images. This integration aims to enhance the capabilities of VQA systems, focusing on aspects such as ship location, density, and size analysis, as well as risk behavior detection. Initially, we conducted baseline experiments using YOLO networks on two representative SAR ship detection datasets, SSDD and HRSID, to assess each model's performance in terms of detection accuracy. Based on these results, we selected the optimal model, YOLOv8n, as the most suitable detection network for this task. Subsequently, leveraging the vision-language model Qwen2-VL, we designed and implemented a VQA task specifically for SAR scenes. This task employs the ship location and size information output by the detection network to generate multi-turn dialogues and scene descriptions for SAR imagery. Experimental results indicate that this method not only enables fundamental SAR scene question-answering without the need for additional datasets or fine-tuning but also dynamically adapts to complex, multi-turn dialogue requirements, demonstrating robust semantic understanding and adaptability.

视觉问答SAR图像目标检测零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。