用查表法让Transformer更省电，性能几乎不变。
Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference
- 用查表替代乘法运算，实现低功耗推理
- 在CIFAR-10上保持95.64%准确率，减少55%乘法
- 适合边缘设备部署的高效Transformer模型
Transformer正广泛应用于聊天机器人、视觉识别等场景，但其日益增长的计算与内存需求导致能耗上升。为实现快速且节能的推理，本文提出基于查找表（LUT）的轻量级Transformer架构——准无权重Transformer（QuWeiT），将LUT方法扩展至Transformer中的多层感知机（MLP）模块，并与原模型融合。该方法仅需少量查表操作即可完成推理。在I-ViT-T模型上，于CIFAR-10数据集上达到95.64%的准确率，同时替换约55%的乘法运算，能效提升2.2倍；在nanoGPT框架中也观察到类似节能效果。
原文摘要 · Abstract (English)
Transformers are set to become ubiquitous with applications ranging from chatbots and educational assistants to visual recognition and remote sensing. However, their increasing computational and memory demands is resulting in growing energy consumption. Building models with fast and energy-efficient inference is imperative to enable a variety of transformer-based applications. Look Up Table (LUT) based Weightless Neural Networks are faster than the conventional neural networks as their inference only involves a few lookup operations. Recently, an approach for learning LUT networks directly via an Extended Finite Difference method was proposed. We build on this idea, extending it for performing the functions of the Multi Layer Perceptron (MLP) layers in transformer models and integrating them with transformers to propose Quasi Weightless Transformers (QuWeiT). This allows for a computational and energy-efficient inference solution for transformer-based models. On I-ViT-T, we achieve a comparable accuracy of 95.64% on CIFAR-10 dataset while replacing approximately 55% of all the multiplications in the entire model and achieving a 2.2x energy efficiency. We also observe similar savings on experiments with the nanoGPT framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。