将Llama模型蒸馏到Mamba架构，实现高效语言处理。
Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
- 用Mamba架构蒸馏Llama-3.x，构建轻量级序列模型。
- 在相同性能下，推理吞吐量更高，支持更大批量处理。
- 仅需0.1%数据训练，适合手机等边缘设备部署。
我们提出Llamba，一系列从Llama-3.x蒸馏至Mamba架构的高效循环语言模型。该系列包括Llamba-1B、Llamba-3B和Llamba-8B，相比基于Transformer的模型，在保持相当基准性能的同时，实现了更高的推理吞吐量并能处理显著更大的批处理规模。此外，Llamba通过MOHAWK（Bick et al., 2024）实现了跨架构蒸馏，仅使用通常相似规模模型所需训练数据的不到0.1%便达成此成果。为充分发挥其效率优势，我们还为资源受限设备如智能手机和边缘平台提供了优化实现，提供了一种实用且内存高效的Transformer替代方案。总体而言，Llamba提升了速度、内存效率与性能之间的平衡，使高质量语言模型更具可及性。
原文摘要 · Abstract (English)
We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B, which achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Furthermore, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK (Bick et al., 2024), achieving these results with less than 0.1% of the training data typically used for models of similar size. To take full advantage of their efficiency, we provide an optimized implementation of Llamba for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers. Overall, Llamba improves the tradeoff between speed, memory efficiency, and performance, making high-quality language models more accessible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。