轻量级Transformer通过降维与剪枝，大幅降低内存与推理时间。
Resource-Efficient Transformer Architecture: Optimizing Memory and Execution Time for Real-Time Applications
- 减半嵌入维度，结合剪枝与量化压缩模型
- 内存减少52%，推理速度提升33%且精度损失小
- 适合边缘设备上的实时应用，如移动端或物联网
本文提出一种内存高效的Transformer模型，通过将嵌入维度减半，并结合参数剪枝与量化等技术，在几乎不牺牲性能的前提下，显著降低内存占用和执行时间。尽管近期已有多种注重参数效率与计算优化的Transformer架构，但在真实场景中部署于边缘设备时仍需大量硬件资源。本方法在保持接近原模型性能的同时，实现52%的内存减少和33%的执行时间下降，优于当前最先进的模型。实验对比了MobileBERT、DistilBERT等主流架构，验证了其在资源受限环境下的可行性，尤其适用于实时性要求高的轻量级深度学习应用。
原文摘要 · Abstract (English)
This paper describes a memory-efficient transformer model designed to drive a reduction in memory usage and execution time by substantial orders of magnitude without impairing the model's performance near that of the original model. Recently, new architectures of transformers were presented, focused on parameter efficiency and computational optimization; however, such models usually require considerable resources in terms of hardware when deployed in real-world applications on edge devices. This approach addresses this concern by halving embedding size and applying targeted techniques such as parameter pruning and quantization to optimize the memory footprint with minimum sacrifices in terms of accuracy. Experimental results include a 52% reduction in memory usage and a 33% decrease in execution time, resulting in better efficiency than state-of-the-art models. This work compared our model with existing compelling architectures, such as MobileBERT and DistilBERT, and proved its feasibility in the domain of resource-friendly deep learning architectures, mainly for applications in real-time and in resource-constrained applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。