提出可变形双层路由注意力，提升视觉Transformer的语义聚焦能力
DeBiFormer: Vision Transformer with Deformable Agent Bi-level Routing Attention
- 用代理查询优化关键值对选择，增强注意力聚焦
- 在图像分类、检测与分割任务上均取得优异性能
- 适合需要高精度语义理解的视觉任务研究者
具有不同注意力模块的视觉变换器在视觉任务中表现出色。尽管自适应稀疏注意力（如DAT）在图像分类中表现良好，但在细调用于语义分割任务时，可变形点选取的关键值对缺乏语义相关性。BiFormer中的查询感知稀疏注意力虽旨在让每个查询聚焦于前k个路由区域，但注意力计算中被大量无关查询干扰，削弱了重要查询的关注度。为此，我们提出可变形双层路由注意力（DBRA）模块，通过代理查询优化关键值对的选择，并提升注意力图中查询的可解释性。基于此，我们构建了新的通用视觉变换器DeBiFormer。该模型在图像分类、目标检测和语义分割等任务上均经过验证，展现出显著有效性。代码已公开。
原文摘要 · Abstract (English)
Vision Transformers with various attention modules have demonstrated superior performance on vision tasks. While using sparsity-adaptive attention, such as in DAT, has yielded strong results in image classification, the key-value pairs selected by deformable points lack semantic relevance when fine-tuning for semantic segmentation tasks. The query-aware sparsity attention in BiFormer seeks to focus each query on top-k routed regions. However, during attention calculation, the selected key-value pairs are influenced by too many irrelevant queries, reducing attention on the more important ones. To address these issues, we propose the Deformable Bi-level Routing Attention (DBRA) module, which optimizes the selection of key-value pairs using agent queries and enhances the interpretability of queries in attention maps. Based on this, we introduce the Deformable Bi-level Routing Attention Transformer (DeBiFormer), a novel general-purpose vision transformer built with the DBRA module. DeBiFormer has been validated on various computer vision tasks, including image classification, object detection, and semantic segmentation, providing strong evidence of its effectiveness.Code is available at {https://github.com/maclong01/DeBiFormer}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。