纯Mamba架构的70亿参数模型，性能超越主流Transformer模型。
Falcon Mamba: The First Competitive Attention-free 7B Language Model
- 采用纯Mamba架构，无注意力机制，基于5.8万亿词训练
- 在长序列生成中推理速度更快、内存占用更低，性能超Mistral 7B等模型
- 适合追求高效推理与低资源部署的开发者，开源可商用
本文介绍Falcon Mamba 7B，一种基于新型Mamba架构的70亿参数基础大语言模型。该模型在5.8万亿个标记的数据上训练，不使用注意力机制，纯基于Mamba结构。在多项评测中，其表现优于Mistral 7B、Llama3.1 8B和Falcon2 11B等主流开源模型，与Gemma 7B相当，并超越RecurrentGemma 9B和RWKV-v6 Finch 7B/14B等异构设计模型。根据Open LLM Leaderboard，Falcon Mamba 7B是当前同规模下性能最佳的Mamba模型，显著优于现有Mamba及混合模型。得益于架构优势，其推理速度更快，长序列生成所需内存大幅减少。尽管近期研究认为混合架构更优，本工作证明纯Mamba也能达到甚至超越Transformer与混合架构的效果。模型权重已通过Hugging Face开源，许可宽松。
原文摘要 · Abstract (English)
In this technical report, we present Falcon Mamba 7B, a new base large language model based on the novel Mamba architecture. Falcon Mamba 7B is trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure Mamba-based model, Falcon Mamba 7B surpasses leading open-weight models based on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. It is on par with Gemma 7B and outperforms models with different architecture designs, such as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B. Currently, Falcon Mamba 7B is the best-performing Mamba model in the literature at this scale, surpassing both existing Mamba and hybrid Mamba-Transformer models, according to the Open LLM Leaderboard. Due to its architecture, Falcon Mamba 7B is significantly faster at inference and requires substantially less memory for long sequence generation. Despite recent studies suggesting that hybrid Mamba-Transformer models outperform pure architecture designs, we demonstrate that even the pure Mamba design can achieve similar, or even superior results compared to the Transformer and hybrid designs. We make the weights of our implementation of Falcon Mamba 7B publicly available on https://huggingface.co/tiiuae/falcon-mamba-7b, under a permissive license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。