YOLOv11-RGBT融合多光谱信息,提升复杂环境检测精度。
YOLOv11-RGBT: Towards a Comprehensive Single-Stage Multispectral Object Detection Framework
- 基于YOLOv11设计六种融合模式,实现多光谱信息高效整合。
- 在FLIR数据集上提升mAP达3.41%-5.65%,最高达47.61%。
- 适合需要高鲁棒性的红外可见光检测场景,如安防与自动驾驶。
多光谱目标检测通过融合多波段信息,可提升检测精度与环境适应性,在多个领域具有重要应用潜力。尽管现有方法在跨模态交互、低光照条件和模型轻量化方面取得进展,仍面临缺乏统一的单阶段框架、性能与融合策略难以平衡、模态权重分配不合理等挑战。为此,基于YOLOv11框架,本文提出YOLOv11-RGBT,一种全新的多模态目标检测框架。设计了六种多光谱融合模式,并成功应用于YOLOv3至YOLOv12及RT-DETR模型。重新评估双模态重要性后,提出P3中融合策略与多光谱可控微调(MCF)策略,优化特征融合,减少冗余与错配,提升整体性能。实验表明,该框架在三个主流开源多光谱检测数据集(如LLVIP和FLIR)上表现优异。尤其在FLIR数据集上,MCF策略持续提升YOLOv11模型的mAP 3.41%-5.65%,最高达47.61%,验证了框架与策略的有效性。代码已公开:https://github.com/wandahangFY/YOLOv11-RGBT。
原文摘要 · Abstract (English)
Multispectral object detection, which integrates information from multiple bands, can enhance detection accuracy and environmental adaptability, holding great application potential across various fields. Although existing methods have made progress in cross-modal interaction, low-light conditions, and model lightweight, there are still challenges like the lack of a unified single-stage framework, difficulty in balancing performance and fusion strategy, and unreasonable modality weight allocation. To address these, based on the YOLOv11 framework, we present YOLOv11-RGBT, a new comprehensive multimodal object detection framework. We designed six multispectral fusion modes and successfully applied them to models from YOLOv3 to YOLOv12 and RT-DETR. After reevaluating the importance of the two modalities, we proposed a P3 mid-fusion strategy and multispectral controllable fine-tuning (MCF) strategy for multispectral models. These improvements optimize feature fusion, reduce redundancy and mismatches, and boost overall model performance. Experiments show our framework excels on three major open-source multispectral object detection datasets, like LLVIP and FLIR. Particularly, the multispectral controllable fine-tuning strategy significantly enhanced model adaptability and robustness. On the FLIR dataset, it consistently improved YOLOv11 models' mAP by 3.41%-5.65%, reaching a maximum of 47.61%, verifying the framework and strategies' effectiveness. The code is available at: https://github.com/wandahangFY/YOLOv11-RGBT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。