用张量分解压缩小模型词嵌入，让手机等设备运行更省电。
TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices
- 不训练直接用张量分解压缩词向量，降低存储和计算开销。
- 在树莓派上实现嵌入层2倍压缩，性能几乎不变，单次查询耗能减半。
- 适合资源受限的移动端部署，尤其关注能效比的场景。
小型语言模型（SLMs）参数量远小于大模型（LLMs），常部署于手机、单板计算机等低功耗设备。与依赖规模提升泛化能力的数据中心大模型不同，边缘部署的SLMs需具备环境适应性和能效优势。本文提出一种无需训练的词嵌入压缩方法，基于张量列车分解（TTD）将每个词向量转换为低秩矩阵乘积态（MPS）。我们在典型低功耗设备（如树莓派）上全面评估了压缩率、语言任务性能、延迟和能耗。以子十亿参数级别的GPT-2/Cerebres-GPT和OPT模型为例，该方法实现约2.0倍的嵌入层压缩，任务性能接近原模型，单次查询能耗降低50%。
原文摘要 · Abstract (English)
Small Language Models (SLMs, or on-device LMs) have significantly fewer parameters than Large Language Models (LLMs). They are typically deployed on low-end devices, like mobile phones and single-board computers. Unlike LLMs, which rely on increasing model size for better generalisation, SLMs designed for edge applications are expected to have adaptivity to the deployment environments and energy efficiency given the device battery life constraints, which are not addressed in datacenter-deployed LLMs. This paper addresses these two requirements by proposing a training-free token embedding compression approach using Tensor-Train Decomposition (TTD). Each pre-trained token embedding vector is converted into a lower-dimensional Matrix Product State (MPS). We comprehensively evaluate the extracted low-rank structures across compression ratio, language task performance, latency, and energy consumption on a typical low-end device, i.e. Raspberry Pi. Taking the sub-billion parameter versions of GPT-2/Cerebres-GPT and OPT models as examples, our approach achieves a comparable language task performance to the original model with around $2.0\times$ embedding layer compression, while the energy consumption of a single query drops by half.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。