arXiv:2604.05923cs.LGcs.CL2026-04

提出可逆状态管理新任务,揭示SSM模型学习能力的局限性。

The UNDO Flip-Flop: A Controlled Probe for Reversible Semantic State Management in State Space Model

论文配图:The UNDO Flip-Flop: A Controlled Probe for Reversible Semantic State Management in State Space Model
图 1 · 摘自论文原文
  • 设计带UNDO功能的翻转任务,测试模型回溯历史状态能力
  • 两层Mamba-2模型在对抗性测试中准确率仅41.10%,低于随机水平
  • 失败根源在于状态检索而非存储,暴露训练机制缺陷

状态空间模型(SSMs)理论上具备建模无循环序列和有限层级结构的能力(Sarrof et al., 2024)。然而,理论表达能力并不保证梯度优化能稳定发现对应解。现有基准仅测试单调状态追踪(如标准翻转任务)或结构嵌套(如Dyck语言),但均未考察可逆语义状态恢复。本文引入UNDO Flip-Flop任务以填补空白:在标准翻转任务基础上增加UNDO操作,要求模型维护隐式有界栈,并在非单调更新序列下恢复历史状态。在该框架下评估单层与双层Mamba-2模型,二者均未能获得理论可实现的基于栈的回滚机制,而是收敛到局部切换启发式策略——仅反转当前状态,而非检索存储的历史。在训练长度分布内的对抗性回退压力测试中,双层模型准确率降至41.10%,低于随机猜测水平。结果表明此为系统性失败而非偶然。因果消融分析显示问题出在状态检索环节,而非存储。这清晰划清了架构的理论表达能力与梯度下降实际学习能力之间的界限,而仅靠理论表达性分析无法捕捉这一差异。

原文摘要 · Abstract (English)

State space models (SSMs) have been shown to possess the theoretical capacity to model both star-free sequential tasks and bounded hierarchical structures Sarrof et al. (2024). However, formal expressivity results do not guarantee that gradient-based optimisation will reliably discover the corresponding solutions. Existing benchmarks probe either monotonic state tracking, as in the standard Flip-Flop task, or structural nesting, as in the Dyck languages, but neither isolates reversible semantic state retrieval. We introduce the UNDO Flip-Flop task to fill this gap. By extending the standard Flip-Flop with an UNDO, the task requires a model to maintain an implicit bounded stack and recover historical states under non-monotonic update sequences. We evaluate one-layer and two-layer Mamba-2 under this framework. Both variants fail to acquire the provably expressible stack-based rollback mechanism, converging instead on a local toggle heuristic that inverts the current state rather than retrieving stored history. Under an adversarial retraction pressure test held within the training length distribution, the two-layer model collapses to 41.10% accuracy, which is below random chance. The results confirm systematic rather than incidental failure. Causal ablation shows that the bottleneck lies in retrieval, not storage. These results draw a clear line between what an architecture can in principle represent and what gradient descent reliably learns, a distinction that theoretical expressivity analyses alone cannot capture.

状态空间模型可逆计算模型评估Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。