用可变形注意力自适应调整特征点位置,提升图像分割等任务的上采样效果。
LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention
- 基于局部自注意力建立动态上采样机制,实现查询引导的自适应点重排。
- 在目标检测、实例分割等4项任务中分别提升1.7 AP、1.5 AP、2.0 PQ、2.5 mIoU。
- 轻量级设计,可无缝嵌入多种网络结构,适合复杂场景下的高精度上采样需求。
特征上采样是构建深度卷积神经网络的关键操作。现有方法或缺乏特征引导,或依赖高分辨率特征图,导致性能与灵活性受限。本文发现局部自注意力天然具备特征引导能力,其计算范式与特征上采样的本质(邻近点的特征重组)高度契合。因此,将局部自注意力引入上采样任务,并证明多数现有上采样器可视为其特例。针对上采样点与其邻域间潜在语义差异,进一步引入变形机制,提出LDA-AQU。该方法为基于动态核的新型上采样器,利用查询特征自适应调整邻近点的位置与聚合权重,以满足复杂场景下的上采样需求。同时,模型轻量,易于集成至各类架构。在物体检测、实例分割、全景分割和语义分割四项密集预测任务上评估,均显著优于先前最优上采样器,分别取得1.7 AP、1.5 AP、2.0 PQ、2.5 mIoU的性能提升。
原文摘要 · Abstract (English)
Feature upsampling is an essential operation in constructing deep convolutional neural networks. However, existing upsamplers either lack specific feature guidance or necessitate the utilization of high-resolution feature maps, resulting in a loss of performance and flexibility. In this paper, we find that the local self-attention naturally has the feature guidance capability, and its computational paradigm aligns closely with the essence of feature upsampling (\ie feature reassembly of neighboring points). Therefore, we introduce local self-attention into the upsampling task and demonstrate that the majority of existing upsamplers can be regarded as special cases of upsamplers based on local self-attention. Considering the potential semantic gap between upsampled points and their neighboring points, we further introduce the deformation mechanism into the upsampler based on local self-attention, thereby proposing LDA-AQU. As a novel dynamic kernel-based upsampler, LDA-AQU utilizes the feature of queries to guide the model in adaptively adjusting the position and aggregation weight of neighboring points, thereby meeting the upsampling requirements across various complex scenarios. In addition, LDA-AQU is lightweight and can be easily integrated into various model architectures. We evaluate the effectiveness of LDA-AQU across four dense prediction tasks: object detection, instance segmentation, panoptic segmentation, and semantic segmentation. LDA-AQU consistently outperforms previous state-of-the-art upsamplers, achieving performance enhancements of 1.7 AP, 1.5 AP, 2.0 PQ, and 2.5 mIoU compared to the baseline models in the aforementioned four tasks, respectively. Code is available at \url{https://github.com/duzw9311/LDA-AQU}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。