不依赖矩阵乘法的高效语言模型，训练推理更快更省显存。
Reservoir Computing inspired Matrix Multiplication-free Language Model
- 用类储层计算结构固定部分权重，减少训练开销。
- 参数量降19%，训练时间快9.9%，推理快8.0%。
- 适合资源受限场景下的大模型部署与快速迭代。
大型语言模型在自然语言处理中表现优异，但计算成本过高仍是主要瓶颈。本文提出一种无矩阵乘法的语言模型（MatMul-free LM），并借鉴储层计算思想优化架构以降低训练成本。通过部分固定并共享选定层的权重，插入储层层以生成丰富动态表征，且无需额外训练开销；同时合并多个操作以减少内存访问。实验表明，该模型参数量最多减少19%，训练时间缩短9.9%，推理时间降低8.0%，性能与基线模型相当。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved state-of-the-art performance in natural language processing; however, their high computational cost remains a major bottleneck. In this study, we target computational efficiency by focusing on a matrix multiplication free language model (MatMul-free LM) and further reducing the training cost through an architecture inspired by reservoir computing. Specifically, we partially fix and share the weights of selected layers in the MatMul-free LM and insert reservoir layers to obtain rich dynamic representations without additional training overhead. Additionally, several operations are combined to reduce memory accesses. Experimental results show that the proposed architecture reduces the number of parameters by up to 19%, training time by 9.9%, and inference time by 8.0%, while maintaining comparable performance to the baseline model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。