提出新指标解析Mamba模型中令牌的影响力，揭示其内在工作机制。
Controllability Analysis of State Space-based Language Model
- 通过反向递推计算令牌影响力得分，量化其对后续状态的作用。
- 模型越大、数据越多，影响力越强；中后层影响集中，存在最近性偏见。
- 大模型才出现内容词优先等新行为，适合研究SSM机制的学者参考。
状态空间模型（SSMs），尤其是Mamba，已成为序列建模的强大架构，但其内部动态仍远不如注意力模型清晰。本文引入并验证了基于可控制性的影响力得分，该指标源自Mamba离散化状态空间参数，通过类似系统可观测性的反向递推计算得出。该得分衡量位置k的令牌对所有后续状态和输出的强烈程度。我们在三种Mamba变体（mamba-130m、mamba-2.8b、mamba-2.8b-slimpj）上进行六项实验，测试其对温度、提示复杂度、令牌类型、层数、位置及输入扰动的敏感性。结果揭示三大发现：(1) 影响力得分随模型规模与训练数据增加而上升，反映模型容量；(2) Mamba具有稳定架构特征，包括最近性偏见及中后层影响集中；(3) 仅在大规模下出现涌现行为，mamba-2.8b-slimpj独特地优先处理内容词，并在噪声中降低内部影响。这些发现确立了影响力得分为解读与比较基于SSM的语言模型的实用诊断工具。
原文摘要 · Abstract (English)
State-space models (SSMs), particularly Mamba, have become powerful architectures for sequence modeling, yet their internal dynamics remain poorly understood compared to attention-based models. We introduce and validate the Influence Score, a controllability-based metric derived from the discretized state-space parameters of Mamba and computed through a backward recurrence analogous to system observability. The score quantifies how strongly a token at position k affects all later states and outputs. We evaluate this measure across three Mamba variants: mamba-130m, mamba-2.8b, and mamba-2.8b-slimpj, using six experiments that test its sensitivity to temperature, prompt complexity, token type, layer depth, token position, and input perturbations. The results show three main insights: (1) the Influence Score increases with model size and training data, reflecting model capacity; (2) Mamba exhibits consistent architectural patterns, including recency bias and concentrated influence in mid-to-late layers; and (3) emergent behaviors appear only at scale, with mamba-2.8b-slimpj uniquely prioritizing content words and reducing internal influence in the presence of noise. These findings establish the Influence Score as a practical diagnostic tool for interpreting and comparing SSM-based language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。