首个全整数运算的轻量级Transformer检测模型,适合边缘设备部署。
Enabling Fully Integer-Only Inference for Lightweight Detection Transformers

- 全链路整数运算,支持变形注意力、特征融合等复杂操作
- 模型体积缩小3.6倍,计算量降低超10倍,精度损失小
- 适合NPU和微控制器等资源受限场景
视觉Transformer检测器已接近CNN的准确率,但因可变形注意力、特征融合及非线性激活函数不兼容整数运算,难以在NPU和微控制器上部署。现有量化检测器要么保留Softmax、GELU、LayerNorm等操作,要么聚焦于重型骨干网络,导致轻量级检测Transformer缺乏端到端整数实现。本文提出I-LW-DETR,首个全整数运算的轻量级DETR,前向传播中所有操作(包括变换器非线性)均使用整数算术。其核心为:保持尺度的分裂卷积,为多尺度投影器各分支分配独立激活尺度;SD-ShiftGELU,一种符号依赖的GELU近似,避免精度下降;约束版Shiftmax,维持Softmax归一化稳定性。实验表明,该量化流程在不同模型规模下均生成高效全整数模型,所有规模下仅造成适度精度损失,模型大小压缩约3.6倍,计算成本降低超过一个数量级。
原文摘要 · Abstract (English)
Vision Transformer detectors now approach the accuracy of CNNs but remain difficult to deploy on NPUs and microcontrollers because key components, including deformable attention, feature fusion, and nonlinear activation functions, are not natively compatible with integer arithmetic. Existing quantized detectors either retain operators such as Softmax, GELU, and LayerNorm or focus on heavyweight backbones, leaving lightweight detection transformers without an end-to-end integer implementation. We address this gap with I-LW-DETR, the first fully integer-only lightweight DETR, in which every operation in the forward pass, including transformer nonlinearities, is executed in integer arithmetic. I-LW-DETR is built upon three key components: a scale-preserving split convolution that assigns independent activation scale to each branch of the multi-scale projector; SD-ShiftGELU, a sign-dependent GELU approximation that preserves element-wise behavior while avoiding the accuracy degradation; and a constrained Shiftmax that maintains stable Softmax normalization. Experimental results demonstrate that the proposed quantization pipeline consistently produces efficient fully integer-only models across different model scales. Across all model scales, the proposed pipeline incurs only a moderate accuracy degradation while reducing the model size by approximately $3.6\times$ and the computational cost by more than one order of magnitude.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。