提出全整数注意力机制，显著提升边缘设备推理速度与能效。
IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference
- 用整数域的IndexSoftmax替代浮点指数运算，避免数据类型转换开销。
- 在Armv8 CPU上实现最高3.7倍加速和61%能耗降低，比FP16快2倍。
- 无需训练即可部署，适合资源受限的边缘设备应用。
将Transformer模型部署到边缘设备受延迟和能耗预算限制。尽管INT8量化可加速主要矩阵乘法，但softmax相关路径成为主要瓶颈，需经历耗时的反量化→softmax→重量化流程，占总注意力延迟高达65%，破坏边缘硬件所需的端到端整数数据流。为此，我们提出IntAttention，首个无需训练的全整数注意力流水线，可直接替换现有结构。核心是IndexSoftmax，一种完全在整数域内替代浮点指数的硬件友好算子。IntAttention结合稀疏感知裁剪、32项查表近似和直接整数归一化，彻底消除注意力路径中的数据类型转换开销。在Armv8 CPU上的实验表明，相比FP16基线，最多提速3.7倍、能耗降低61%；相比传统INT8注意力，最快提速2.0倍。在多种语言、视觉模型及推理、长上下文任务中，保持强整体保真度，优于现有基于查找表的softmax近似方法。代码已开源：https://github.com/WanliZhong/IntAttention
原文摘要 · Abstract (English)
Deploying Transformer models on edge devices is limited by latency and energy budgets. While INT8 quantization effectively accelerates the primary matrix multiplications, it exposes the softmax-related path as the dominant bottleneck. This stage incurs a costly dequantize -> softmax -> requantize detour, which can account for up to 65% of total attention latency and disrupts the end-to-end integer dataflow critical for edge hardware efficiency. To address this limitation, we present IntAttention, the first fully integer attention pipeline that serves as a training-free drop-in replacement. At the core of our approach lies IndexSoftmax, a hardware-friendly operator that replaces floating-point exponentials entirely within the integer domain. IntAttention integrates sparsity-aware clipping, a 32-entry lookup table approximation, and direct integer normalization, thereby eliminating datatype conversion overhead along the attention path. Experiments on Armv8 CPUs show that our method achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines, and up to 2.0x speedup over conventional INT8 attention pipelines. Across diverse language and vision models, as well as additional reasoning and long-context evaluations, IntAttention maintains strong overall fidelity and demonstrates a more favorable trade-off than existing LUT-based softmax approximations. Code is available at https://github.com/WanliZhong/IntAttention
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。