分析Mamba模型在GPU训练中的行为，为性能优化提供依据
Characterizing the Behavior of Training Mamba-based State Space Models on GPUs
- 构建代表性Mamba模型工作负载套件
- 揭示其在GPU上的计算特征与资源需求
- 适合关注模型高效训练的硬件设计者
基于Mamba的状态空间模型（SSM）已成为取代主流Transformer的有前景方案。尽管Transformer表达能力强，但其注意力机制的二次复杂度限制了序列长度扩展。SSM通过新型架构，在视频、文本生成和图等领域实现了更低的计算复杂度。因此，刻画此类新兴工作负载在GPU上的运行行为，对理解其微架构需求至关重要。本文评估了基于Mamba的SSM在GPU上的训练行为，构建了覆盖不同架构的代表性模型工作负载套件，并利用该套件分析其对GPU架构的影响。研究结果为持续提升此类模型的性能提供了新的优化方向。
原文摘要 · Abstract (English)
Mamba-based State Space Models (SSM) have emerged as a promising alternative to the ubiquitous transformers. Despite the expressive power of transformers, the quadratic complexity of computing attention is a major impediment to scaling performance as we increase the sequence length. SSMs provide an alternative path that addresses this problem, reducing the computational complexity requirements of self-attention with novel model architectures for different domains and fields such as video, text generation and graphs. Thus, it is important to characterize the behavior of these emerging workloads on GPUs and understand their requirements during GPU microarchitectural design. In this work we evaluate Mamba-based SSMs and characterize their behavior during training on GPUs. We construct a workload suite that offers representative models that span different model architectures. We then use this suite to analyze the architectural implications of running Mamba-based SSMs on GPUs. Our work sheds new light on potential optimizations to continue scaling the performance for such models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。