对比Mamba与LLaMA在长文本中的效率，发现SSM更省资源。
Benchmarking the Computational and Representational Efficiency of State Space Models against Transformers on Long-Context Dyadic Sessions
- 用Mamba和LLaMA对比长序列处理，测试512到8192词元。
- SSM内存占用更低，推理速度更快，尤其在长序列时优势明显。
- 适合处理医疗对话等长上下文场景的开发者参考。
状态空间模型（SSMs）作为长序列建模的潜在替代方案，相比Transformer的二次复杂度 $O(N^2)$，具有线性复杂度 $O(N)$。本文对Mamba SSM与LLaMA Transformer在长序列上的表现进行了全面基准测试，以双人治疗会话为典型应用场景。评估涵盖两个维度：(1) 计算效率，测量从512至8,192个词元的内存占用与推理速度；(2) 表示效率，分析隐藏状态动态与注意力模式。研究结果为长上下文应用的实践者提供了可操作的洞见，明确了在何种条件下SSM相较Transformer更具优势。
原文摘要 · Abstract (English)
State Space Models (SSMs) have emerged as a promising alternative to Transformers for long-context sequence modeling, offering linear $O(N)$ computational complexity compared to the Transformer's quadratic $O(N^2)$ scaling. This paper presents a comprehensive benchmarking study comparing the Mamba SSM against the LLaMA Transformer on long-context sequences, using dyadic therapy sessions as a representative test case. We evaluate both architectures across two dimensions: (1) computational efficiency, where we measure memory usage and inference speed from 512 to 8,192 tokens, and (2) representational efficiency, where we analyze hidden state dynamics and attention patterns. Our findings provide actionable insights for practitioners working with long-context applications, establishing precise conditions under which SSMs offer advantages over Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。