针对光照变化导致的融合图像失真,提出动态专家网络提升多模态图像融合质量。
MoCTEFuse: Illumination-Gated Mixture of Chiral Transformer Experts for Multi-Level Infrared and Visible Image Fusion
- 设计光照门控的异构专家网络,动态平衡红外与可见光信息。
- 在多个数据集上实现最高70.93%目标检测mAP,显著优于现有方法。
- 适合需要高精度融合的自动驾驶与夜视系统应用。
光照变化不可避免地影响红外与可见光图像融合质量,许多优秀方法忽略此因素,直接融合源图像信息,导致融合结果出现模态偏差。为此,我们提出一种动态多层级图像融合网络MoCTEFuse,采用光照门控的异构变压器专家(MoCTE)自适应地平衡保留纹理细节与物体对比度。MoCTE包含高光照和低光照专家子网络,均由异构变压器融合块(CTFB)构建。受光照门控信号引导,CTFB通过非对称交叉注意力机制动态切换主辅模态并分配相应权重。同时,在多阶段堆叠以逐步聚合和精炼模态特异性及跨模态信息。为促进鲁棒训练,提出结合光照分布的竞赛式损失函数,包含三个层级的子损失项。在DroneVehicle、MSRS、TNO和RoadScene数据集上的大量实验表明,MoCTEFuse具有优越的融合性能。最终在MFNet数据集上达到70.93%的最佳检测平均精度(mAP),在DroneVehicle数据集上达到45.14%。代码与模型已公开于https://github.com/Bitlijinfu/MoCTEFuse。
原文摘要 · Abstract (English)
While illumination changes inevitably affect the quality of infrared and visible image fusion, many outstanding methods still ignore this factor and directly merge the information from source images, leading to modality bias in the fused results. To this end, we propose a dynamic multi-level image fusion network called MoCTEFuse, which applies an illumination-gated Mixture of Chiral Transformer Experts (MoCTE) to adaptively preserve texture details and object contrasts in balance. MoCTE consists of high- and low-illumination expert subnetworks, each built upon the Chiral Transformer Fusion Block (CTFB). Guided by the illumination gating signals, CTFB dynamically switches between the primary and auxiliary modalities as well as assigning them corresponding weights with its asymmetric cross-attention mechanism. Meanwhile, it is stacked at multiple stages to progressively aggregate and refine modality-specific and cross-modality information. To facilitate robust training, we propose a competitive loss function that integrates illumination distributions with three levels of sub-loss terms. Extensive experiments conducted on the DroneVehicle, MSRS, TNO and RoadScene datasets show MoCTEFuse's superior fusion performance. Finally, it achieves the best detection mean Average Precision (mAP) of 70.93% on the MFNet dataset and 45.14% on the DroneVehicle dataset. The code and model are released at https://github.com/Bitlijinfu/MoCTEFuse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。