改进Mamba模型,提升时间序列建模的长程依赖与计算效率。
Attention Mamba: Time Series Modeling with Adaptive Pooling Acceleration and Receptive Field Enhancements
- 引入自适应池化块加速注意力计算并扩展感受野。
- 结合双向Mamba块捕捉长短时特征,增强非线性建模能力。
- 在多个数据集上超越主流模型,适合时序预测任务应用。
时间序列建模是天气预报、交通管理等实际应用的核心。近年来,Mamba模型凭借近线性计算复杂度与高精度表现成为有前景的时序建模方法,但存在注意力机制对非线性依赖建模不足、卷积导致感受野受限等问题。为此,本文提出Attention Mamba框架,引入新型自适应池化模块,加速注意力计算并融合全局信息,有效突破感受野限制。同时,集成双向Mamba块,高效捕获长短时特征,并将输入转换为注意力机制中的Value表示。在多个数据集上的大量实验表明,该模型在提取非线性依赖和增强感受野方面表现优异,性能优于当前主流方法。代码将开源于GitHub。
原文摘要 · Abstract (English)
"This work has been submitted to the lEEE for possible publication. Copyright may be transferred without noticeafter which this version may no longer be accessible." Time series modeling serves as the cornerstone of real-world applications, such as weather forecasting and transportation management. Recently, Mamba has become a promising model that combines near-linear computational complexity with high prediction accuracy in time series modeling, while facing challenges such as insufficient modeling of nonlinear dependencies in attention and restricted receptive fields caused by convolutions. To overcome these limitations, this paper introduces an innovative framework, Attention Mamba, featuring a novel Adaptive Pooling block that accelerates attention computation and incorporates global information, effectively overcoming the constraints of limited receptive fields. Furthermore, Attention Mamba integrates a bidirectional Mamba block, efficiently capturing long-short features and transforming inputs into the Value representations for attention mechanisms. Extensive experiments conducted on diverse datasets underscore the effectiveness of Attention Mamba in extracting nonlinear dependencies and enhancing receptive fields, establishing superior performance among leading counterparts. Our codes will be available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。