针对NPU优化多尺度可变形注意力，实现训练加速近8倍。
Towards Efficient Multi-Scale Deformable Attention on NPU
- 重构内存访问与计算策略，适配昇腾NPU架构
- 前向/反向计算加速最高达8.9倍，端到端训练快7.3倍
- 适合需要高效视觉模型训练的硬件研发人员
多尺度可变形注意力（MSDA）是视觉任务中灵活且强大的特征提取机制，但其随机网格采样策略在专用加速器（如NPU）上带来显著优化挑战。本文提出一种协同设计方法，系统重思昇腾NPU架构下的内存访问与计算策略。所提实现支持高效的前向与反向计算，完全适配训练负载，并集成一系列硬件感知优化。大量实验表明，该方案相较基于网格采样的基线，在前向、反向及端到端训练中分别获得最高5.9倍、8.9倍和7.3倍加速；相较于最新厂商库，分别提速1.9倍、2.4倍和2.0倍。
原文摘要 · Abstract (English)
Multi-scale deformable attention (MSDA) is a flexible and powerful feature extraction mechanism for visual tasks, but its random-access grid sampling strategy poses significant optimization challenges, especially on domain-specific accelerators such as NPUs. In this work, we present a co-design approach that systematically rethinks memory access and computation strategies for MSDA on the Ascend NPU architecture. With this co-design approach, our implementation supports both efficient forward and backward computation, is fully adapted for training workloads, and incorporates a suite of hardware-aware optimizations. Extensive experiments show that our solution achieves up to $5.9\times$ (forward), $8.9\times$ (backward), and $7.3\times$ (end-to-end training) speedup over the grid sample-based baseline, and $1.9\times$, $2.4\times$, and $2.0\times$ acceleration over the latest vendor library, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。