为设备端Transformer设计可硬件部署的自适应分段激活函数
DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training
- 根据前激活数据分布，动态分配不同精度的分段线性近似
- 相比GELU加速16倍，逻辑单元占用减少16倍，性能相当或更优
- 适合资源受限设备上的Transformer推理与训练部署
非线性激活函数在设备端推理与训练中至关重要，不仅消耗大量硬件资源，还显著影响系统性能与能效。本文提出分布感知分段激活函数（DAPA），通过利用前激活数据的分布特性，构建可微且硬件友好的变换器激活函数。DAPA采用非均匀分段逼近，将更高精度的分段分配给分布概率高的区域，提升了对以往分段线性方法的泛化能力。进一步通过分布加权均方误差对结果进行量化，以降低延迟和资源开销。硬件实现显示，DAPA使GELU计算提速16倍，DSP资源使用降低16倍，同时在视觉变换器与GPT-2模型上保持相当或更优的性能。
原文摘要 · Abstract (English)
Non-linear activation functions play a pivotal role in on-device inference and training, as they not only consume substantial hardware resources but also impose a significant impact on system performance and energy efficiency. In this work, we propose Distribution-Aware Piecewise Activation (DAPA), a differentiable and hardware-friendly activation function for Transformer architectures by exploiting the distribution of pre-activation data. DAPA employs a non-uniform piecewise approximation that allocates finer segments to high-probability regions of the distribution, improving generalizability over prior piecewise linear methods. The resulting approximation is further quantized using Distribution-Weighted Mean Square Error to reduce latency and resource utilization for hardware deployment. Our HLS implementation demonstrates that DAPA speeds up GELU computation by 16$\times$ and decreases DSP utilization by 16$\times$ while maintaining comparable or better performance across vision Transformers and GPT-2 models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。