提出轻量级跨模态注意力机制,提升红外与可见光目标检测精度
Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection

- 基于局部邻域先验,聚焦关键区域进行特征对齐
- 在多个数据集上实现84.2~90.8 [email protected],优于现有方法
- 适合资源受限平台,计算开销更低,可部署于边缘设备
多模态目标检测中,跨模态特征对齐与交互是核心挑战。尽管全局交叉注意力具备强大的长程建模能力,但其随特征尺寸呈二次增长的复杂度限制了在资源受限平台的部署。为此,本文提出渐进式像素邻域可变形交叉注意力(PNAFusion),基于两个观察:可见光与热成像间的错位通常集中于局部邻域;跨模态语义对应常遵循非线性空间映射,固定感受野难以捕捉。PNAFusion引入局部空间先验,将特征交互与对齐聚焦于最相关邻域。具体地,设计像素邻域交叉注意力(PNCA)模块,避免冗余全局匹配并抑制背景噪声;同时,自适应可变形对齐(ADA)模块通过学习像素级偏移捕捉非线性空间对应。二者通过迭代反馈机制集成,逐步优化跨模态特征对齐。在FLIR、M3FD和DroneVehicle数据集上,采用YOLOv5检测器时,PNAFusion分别取得84.2、90.5、85.5 [email protected];迁移至Co-DETR后,分别达到86.8和90.8 [email protected]。效率分析显示,相比ICAFusion,GPU内存降低33.0%,理论浮点运算量从194.8 G降至156.4 G,尽管可变形采样与迭代精化带来额外延迟。
原文摘要 · Abstract (English)
Effective cross-modal feature alignment and interaction are central challenges in multispectral object detection. Although global cross-attention provides strong long-range modeling ability, its quadratic complexity with respect to feature size limits deployment on resource-constrained platforms. We therefore propose Progressive Pixel-Neighborhood Deformable Cross-Attention for multispectral feature fusion, termed PNAFusion. The proposed framework is motivated by two observations: weak misalignment between visible and thermal images is usually concentrated around local neighborhoods, and semantic correspondence across modalities often follows non-linear spatial mappings that fixed receptive fields cannot model well. To address these issues, PNAFusion incorporates local spatial priors into its architectural design to concentrate feature interaction and alignment on the most relevant neighborhoods. Specifically, a Pixel-Neighborhood Cross-Attention (PNCA) module is introduced to avoid redundant global feature matching and suppress background noise. Meanwhile, an Adaptive Deformable Alignment (ADA) module captures non-linear spatial correspondences through learned pixel-wise offsets. These components are further integrated through an iterative feedback mechanism to progressively refine cross-modal feature alignment. Experiments on FLIR, M3FD, and DroneVehicle show that PNAFusion achieves 84.2, 90.5, and 85.5 [email protected], respectively, under the YOLOv5 detector, and further reaches 86.8 [email protected] on FLIR and 90.8 [email protected] on M3FD when transferred to Co-DETR. Efficiency analysis indicates that PNAFusion reduces allocated GPU memory by 33.0\% compared with ICAFusion and reduces theoretical FLOPs from 194.8 G to 156.4 G, although the deformable sampling and iterative refinement introduce additional latency. Our code will be available at https://github.com/DanielQiuTian/PNAFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。