ECHO-LLaMA通过共享键值缓存提升训练速度与推理效率。
ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training
- 将部分层的键值缓存共享，降低计算复杂度。
- 训练时吞吐量最高提升77%，模型算力利用率提高16%。
- 适合大规模语言模型的高效预训练与微调场景。
本文提出ECHO-LLaMA，一种高效的LLaMA架构，旨在提升LLaMA模型的训练速度与推理吞吐量，同时保持其学习能力。ECHO-LLaMA通过在特定层间共享键值（KV）缓存，显著降低键值计算复杂度，同时维持或提升语言性能。实验表明，该方法在相同训练令牌数下，训练阶段最高实现77%的每秒令牌吞吐量提升，模型浮点运算利用率（MFU）最高提升16%，损失降低最多14%。对于1.1B参数模型，测试阶段吞吐量较基线高出约7%。通过引入计算高效的适配机制，ECHO-LLaMA为大规模语言模型的预训练与微调提供了可扩展且低成本的解决方案，实现更快、更节省资源的训练而不牺牲性能。
原文摘要 · Abstract (English)
This paper introduces ECHO-LLaMA, an efficient LLaMA architecture designed to improve both the training speed and inference throughput of LLaMA architectures while maintaining its learning capacity. ECHO-LLaMA transforms LLaMA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. Experimental results demonstrate that ECHO-LLaMA achieves up to 77\% higher token-per-second throughput during training, up to 16\% higher Model FLOPs Utilization (MFU), and up to 14\% lower loss when trained on an equal number of tokens. Furthermore, on the 1.1B model, ECHO-LLaMA delivers approximately 7\% higher test-time throughput compared to the baseline. By introducing a computationally efficient adaptation mechanism, ECHO-LLaMA offers a scalable and cost-effective solution for pretraining and finetuning large language models, enabling faster and more resource-efficient training without compromising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。