用网络拓扑启发的低秩结构提升长序列建模能力
Motif-Mamba: network motif improved mamba for long-range sequence modeling
- 引入三节点网络基序约束的低秩递归路径，增强状态维度间交互
- 在长序列外推与语言建模任务中优于Mamba基线模型
- 适合需要高效长程依赖建模的研究与应用
长序列建模仍是大语言模型的核心挑战，因自注意力随序列长度呈平方级增长。Mamba通过选择性状态空间递归提供线性时间替代方案，但其主要采用对角状态转移，限制了状态维度间的显式交互。本文提出Motif-Mamba，一种结构化状态空间模型，在Mamba基础上引入基序约束的低秩递归路径。受三节点网络基序动态启发，该路径将隐藏状态投影至紧凑动力学子空间，施加基序引导的交互，并将结果映射回原始状态空间。此设计增强了跨维度通信，同时保持Mamba的线性时间递归结构。在长序列外推、语言建模基准及脑机接口解码任务中的实验均显示，相较于Mamba基线模型有持续提升，表明基序引导的低秩动力学为长程序列建模提供了有效结构先验。
原文摘要 · Abstract (English)
Efficient long-sequence modeling remains a central challenge for large language models, as self-attention scales quadratically with sequence length. Mamba offers a linear-time alternative through selective state space recurrence, but its predominantly diagonal state transitions restrict explicit interactions among state dimensions. We propose Motif-Mamba, a structured state space model that augments Mamba with a motif-constrained low-rank recurrent pathway. Inspired by the dynamics of three-node network motifs, the proposed pathway projects hidden states into a compact dynamical subspace, imposes motif-guided interactions, and maps the resulting dynamics back to the original state space. This design enhances cross-dimensional communication while preserving the linear-time recurrent structure of Mamba. Experiments on long-sequence extrapolation, language modeling benchmarks, and brain--computer interface decoding show consistent improvements over Mamba backbones, suggesting that motif-guided low-rank dynamics provide an effective structural prior for long-range sequence modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。