InfoMamba用线性复杂度实现全局建模,替代了注意力机制。
InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
- 用线性滤波层替代注意力,构建轻量级全局信息接口。
- 在分类、密集预测等任务上超越Transformer和Mamba基线模型。
- 通过信息最大化融合,动态注入全局上下文,适合高效序列建模场景。
在计算资源受限下,如何平衡细粒度局部建模与长程依赖捕捉仍是序列建模的核心挑战。尽管Transformer具备强大的跨标记混合能力,但其复杂度为二次方;而类似Mamba的选通状态空间模型(SSMs)虽呈线性增长,却难以捕捉高秩且同步的全局交互。我们提出一致性边界分析,揭示对角短时记忆SSMs何时可近似因果注意力,并识别出仍存在的结构缺陷。受此启发,我们提出InfoMamba——一种无注意力的混合架构。它以概念瓶颈线性滤波层取代令牌级自注意力,作为最小带宽的全局接口,与选通递归流通过信息最大化融合(IMF)集成。IMF动态向SSM动力学注入全局上下文,并通过互信息启发的目标促进互补信息利用。在分类、密集预测及非视觉任务上的大量实验表明,InfoMamba持续优于强基线模型,在保持近线性扩展的同时实现具有竞争力的准确率-效率权衡。
原文摘要 · Abstract (English)
Balancing fine-grained local modeling with long-range dependency capture under computational constraints remains a central challenge in sequence modeling. While Transformers provide strong token mixing, they suffer from quadratic complexity, whereas Mamba-style selective state-space models (SSMs) scale linearly but often struggle to capture high-rank and synchronous global interactions. We present a consistency boundary analysis that characterizes when diagonal short-memory SSMs can approximate causal attention and identifies structural gaps that remain. Motivated by this analysis, we propose InfoMamba, an attention-free hybrid architecture. InfoMamba replaces token-level self-attention with a concept bottleneck linear filtering layer that serves as a minimal-bandwidth global interface and integrates it with a selective recurrent stream through information-maximizing fusion (IMF). IMF dynamically injects global context into the SSM dynamics and encourages complementary information usage through a mutual-information-inspired objective. Extensive experiments on classification, dense prediction, and non-vision tasks show that InfoMamba consistently outperforms strong Transformer and SSM baselines, achieving competitive accuracy-efficiency trade-offs while maintaining near-linear scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。