用状态反馈让序列模型更高效，长程依赖抓得更好
Context-Selective State Space Models: Feedback is All You Need
- 引入状态反馈机制，使模型动态随上下文自适应调节
- 在诱导头任务中参数量少100倍，准确率接近完美
- 仅3585参数就达MNIST 97%准确率，适合资源受限场景
Transformer依赖注意力机制，虽为大模型核心，但存在复杂度二次增长及长程依赖处理困难的问题。状态空间模型（SSM）是潜在替代方案。本文提出COFFEE（COntext From FEEdback）模型，一种时变状态空间模型，通过引入状态反馈实现上下文相关的选择性，同时保持并行计算能力。该机制使模型能根据内部状态所蕴含的输入历史紧凑表示动态调整行为。状态反馈显著提升长程依赖建模能力：在诱导头任务中，参数量仅为S6的百分之一，训练序列少两个数量级，仍达近似完美准确率；在MNIST上，相同架构下性能大幅超越S6，仅用3585个参数即达到97%准确率。结果表明，状态反馈是构建可扩展、高效率序列模型的关键机制。
原文摘要 · Abstract (English)
Transformers, powered by the attention mechanism, are the backbone of most foundation models, yet they suffer from quadratic complexity and difficulties in dealing with long-range dependencies in the input sequence. Recent work has shown that state space models (SSMs) provide a promising alternative. In this paper, we introduce the COFFEE (COntext From FEEdback) model, a novel time-varying SSM that incorporates state feedback to enable context-dependent selectivity, while still allowing for parallel implementation. This idea allows the model to regulate its dynamics based on the context described by the internal state, which embodies a compact representation of the input history. State feedback allows COFFEE to improve its ability to capture long-range dependencies: on the induction head task, it achieves near-perfect accuracy with two orders of magnitude fewer parameters and training sequences compared to S6 (the SSM of Mamba). On MNIST, COFFEE largely outperforms S6 within the same architecture, reaching 97% accuracy with only 3585 parameters. These results showcase the role of state feedback as a key mechanism for building scalable and efficient sequence models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。