用超稀疏内存层提升Transformer推理速度，兼顾性能与效率
Ultra-Sparse Memory Network
- 引入超稀疏内存层，降低推理时的内存访问开销
- 2000万内存槽模型在计算预算内实现最优推理速度与性能
- 适用于大规模模型部署，支持未来百亿级参数架构
Transformer模型性能与其参数量和计算复杂度呈对数关系。尽管混合专家（MoE）方法将参数量与计算复杂度解耦，但在推理阶段仍面临高昂的内存访问成本。本文提出UltraMem，采用大规模超稀疏内存层以克服上述限制。该方法显著降低推理延迟，同时保持模型性能。我们还研究了该架构的扩展规律，证明其不仅具有优良的扩展特性，且优于MoE。实验中，最大规模的UltraMem模型包含2000万内存槽，结果表明，在给定计算预算下，该方法实现了当前最优的推理速度与模型性能，为支持数十亿内存槽或专家的架构铺平道路。
原文摘要 · Abstract (English)
It is widely acknowledged that the performance of Transformer models is logarithmically related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from computational complexity, they still face challenges in inference due to high memory access costs. This work introduces UltraMem, incorporating large-scale, ultra-sparse memory layer to address these limitations. Our approach significantly reduces inference latency while maintaining model performance. We also investigate the scaling laws of this new architecture, demonstrating that it not only exhibits favorable scaling properties but outperforms MoE. In experiments, the largest UltraMem we train has 20 million memory slots. The results show that our method achieves state-of-the-art inference speed and model performance within a given computational budget, paving the way for billions of slots or experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。