提出精确测量Mamba模型状态使用的方法，发现输入会动态迁移模式分配。
An Exact Instrument for State Usage in Selective State-Space Models, and the Input-Driven Migration It Reveals
- 基于对角状态矩阵，构建可精确计算删减任意模式误差的Gram张量。
- 实测显示70亿参数模型中输入驱动的模式迁移使误差减半，且在低预算下性能媲美全模型。
- 适用于模型压缩与理解，尤其适合研究Mamba类架构的动态机制。
选择性状态空间模型如Mamba通过一组一阶模式传递信息，其输入耦合由学习到的选择机制决定。本文提出一种精确测量训练后模型如何使用这些模式的工具。由于状态矩阵为对角阵，每个通道输出可精确分解为各模式贡献，每层、通道、窗口的Gram张量能离线计算删减任意模式子集的精确输出误差。在Mamba-1系列上验证，相对误差仅2.3×10⁻⁷；对4,464种配置预测部署剪枝误差，中位相对偏差5×10⁻⁷，下限由重建精度决定。应用于Mamba-1（130M–2.8B）、7B Falcon-Mamba及Mamba-2，发现训练模型会随输入重分配状态空间：信号携带模式随上下文迁移，在最敏感层，基于输入的最优模式设置可将输出误差减半。冻结信号反事实分析表明迁移主要来自输入依赖的写入映射B_t，而通常被认为具选择性的时刻贡献极小。基于该测量的输入调度模式剪枝，在130M至70亿规模上均优于静态、Hankel基及层自适应排名，且在半状态预算下达到未剪枝模型性能。因调度器需先读取每窗口模式使用情况，证明存在实际可利用的冗余空间；但不宣称计算或内存节省。
原文摘要 · Abstract (English)
Selective state-space models such as Mamba route information through a bank of first-order modes whose input coupling is set by a learned selection mechanism. We give an exact instrument for measuring how a trained model uses these modes. Because the state matrix is diagonal, each channel's output decomposes exactly into per-mode contributions, and a per-(layer, channel, window) Gram tensor yields the exact output error of dropping any subset of modes, offline, at any budget. Validated against the reference implementation to a relative error of $2.3\times10^{-7}$ on the Mamba-1 family where it is exact, the instrument predicts a layer's deployed pruning error to a median relative deviation of $5\times10^{-7}$ over $4{,}464$ configurations, its floor set by the reconstruction. Applying the instrument across the Mamba-1 family (130M--2.8B), the deployed 7B Falcon-Mamba, and Mamba-2, we find that trained models re-allocate their state space with the input: which modes carry the signal migrates across contexts, and at the most affected layers a per-input oracle roughly halves the output error of a fixed mode set. Frozen-signal counterfactuals attribute the migration primarily to the input-dependent write map $B_t$; the timestep usually identified with selectivity carries almost none of it. Input-scheduled mode pruning on this measurement outperforms static, Hankel-based, and layer-adaptive rankings at every scale from 130M to the deployed 7B Falcon-Mamba, and at half the state budget it matches the unpruned model. Because the scheduler reads each window's mode usage from a first pass, this demonstrates realizable headroom; we claim no deployed compute or memory saving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。