提出LaTIM方法，解析Mamba模型中令牌间的交互机制。
LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
- 设计细粒度令牌分解方法，揭示Mamba内部交互模式。
- 在机器翻译、复制任务等场景中验证有效性。
- 适合关注模型可解释性的研究者与开发者。
状态空间模型（SSMs），如Mamba，已成为长序列建模中替代Transformer的高效方案。然而，尽管其应用日益广泛，现有方法缺乏类似注意力机制的可解释性工具。尽管近期工作提供了对Mamba内部机制的洞见，但未明确分解每个令牌的贡献，导致难以理解其跨层选择性处理序列的过程。本文提出LaTIM，一种适用于Mamba-1和Mamba-2的新型令牌级分解方法，实现细粒度可解释性。我们在多种任务上进行广泛评估，包括机器翻译、复制任务和基于检索的生成，结果表明该方法能有效揭示Mamba的令牌间交互模式。
原文摘要 · Abstract (English)
State space models (SSMs), such as Mamba, have emerged as an efficient alternative to transformers for long-context sequence modeling. However, despite their growing adoption, SSMs lack the interpretability tools that have been crucial for understanding and improving attention-based architectures. While recent efforts provide insights into Mamba's internal mechanisms, they do not explicitly decompose token-wise contributions, leaving gaps in understanding how Mamba selectively processes sequences across layers. In this work, we introduce LaTIM, a novel token-level decomposition method for both Mamba-1 and Mamba-2 that enables fine-grained interpretability. We extensively evaluate our method across diverse tasks, including machine translation, copying, and retrieval-based generation, demonstrating its effectiveness in revealing Mamba's token-to-token interaction patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。