揭示Mamba模型中先验与近期效应的机制，解释信息如何被记住或遗忘。
Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View
- 通过选择性状态空间块中的稀疏通道实现长期记忆，支持开头内容保留。
- 近期输入因指数衰减获更高权重，但干扰项会破坏其优势，限制记忆深度。
- 语义规律性动态调节遗忘倾向，重复关系导致中间内容更易被遗忘。
我们利用先验与近期效应作为行为工具，研究状态空间语言模型中的记忆机制。在Mamba架构上施加结构化回忆任务，观察到准确率呈现典型的U形曲线，表明模型在序列开头和末尾表现更优。我们识别出三个核心机制：第一，长期记忆由模型选择性状态空间块中的稀疏通道支撑，这些通道持续编码早期输入，与先验效应有因果关联；第二，短期记忆受增量调制循环控制，近期输入因指数衰减获得更高权重，但引入干扰项后该优势消失，揭示了记忆深度的明确极限；第三，记忆分配受语义规律性动态调节：输入序列中重复关系改变增量门控行为，增强对中间内容的遗忘倾向。我们在两个大规模Mamba模型（1.4B和7B参数）上通过针对性消融和输入扰动验证了这些发现。
原文摘要 · Abstract (English)
We study memory in state-space language models using primacy and recency effects as behavioral tools to uncover how information is retained and forgotten over time. Applying structured recall tasks to the Mamba architecture, we observe a consistent U-shaped accuracy profile, indicating strong performance at the beginning and end of input sequences. We identify three mechanisms that give rise to this pattern. First, long-term memory is supported by a sparse subset of channels within the model's selective state space block, which persistently encode early input tokens and are causally linked to primacy effects. Second, short-term memory is governed by delta-modulated recurrence: recent inputs receive more weight due to exponential decay, but this recency advantage collapses when distractor items are introduced, revealing a clear limit to memory depth. Third, we find that memory allocation is dynamically modulated by semantic regularity: repeated relations in the input sequence shift the delta gating behavior, increasing the tendency to forget intermediate items. We validate these findings via targeted ablations and input perturbations on two large-scale Mamba-based language models: one with 1.4B and another with 7B parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。