提出轻量框架优化可变形注意力，提升硬件部署效率
A Memory-Efficient Framework for Deformable Transformer with Neural Architecture Search
- 用NAS自动切分特征图，避免内存冲突且不改模型结构
- 图像识别任务中精度仅损失0.2%，远低于现有方法
- 在FPGA上将显存访问次数降至18%，适合边缘设备
可变形注意力变换器（DAT）在计算机视觉任务中表现出色，通过自适应关注图像关键区域实现高性能。然而其依赖数据的采样机制导致内存访问模式不规则，给硬件高效部署带来挑战。现有加速方法或增加硬件开销，或牺牲模型精度。本文提出一种面向硬件优化的DAT框架：首先采用基于神经架构搜索（NAS）的新切片策略，在推理时自动将输入特征划分为均匀块，避免内存冲突且无需修改模型结构；该方法联合优化硬件成本与推理精度，寻找最优切片配置。其次，设计基于FPGA的验证系统，在边缘硬件上测试性能。算法在ImageNet-1K数据集上的实验表明，该框架仅造成0.2%精度下降；硬件实验在Xilinx FPGA上显示，相比现有加速方法，本方法将DRAM访问次数降低至18%。
原文摘要 · Abstract (English)
Deformable Attention Transformers (DAT) have shown remarkable performance in computer vision tasks by adaptively focusing on informative image regions. However, their data-dependent sampling mechanism introduces irregular memory access patterns, posing significant challenges for efficient hardware deployment. Existing acceleration methods either incur high hardware overhead or compromise model accuracy. To address these issues, this paper proposes a hardware-friendly optimization framework for DAT. First, a neural architecture search (NAS)-based method with a new slicing strategy is proposed to automatically divide the input feature into uniform patches during the inference process, avoiding memory conflicts without modifying model architecture. The method explores the optimal slice configuration by jointly optimizing hardware cost and inference accuracy. Secondly, an FPGA-based verification system is designed to test the performance of this framework on edge-side hardware. Algorithm experiments on the ImageNet-1K dataset demonstrate that our hardware-friendly framework can maintain have only 0.2% accuracy drop compared to the baseline DAT. Hardware experiments on Xilinx FPGA show the proposed method reduces DRAM access times to 18% compared with existing DAT acceleration methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。