将Mamba2重构为可解释的自适应滤波器，参数减少41%仍保持性能。
Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
- 用图信号处理视角重构建模，分层设计共享与专家滤波器
- 在多个任务上性能持平，参数量仅为原模型58.9%
- 适合关注效率与模型可解释性的序列建模研究者
状态空间模型（SSMs）以线性时间复杂度提供注意力机制的高效替代。Mamba2是一种基于SSM的语言模型，采用选择性输入门控和多头结构，支持并行计算并在基准测试中表现优异。然而其多头递归独立运行，缺乏结构化利用与分析。本文提出一种名为层级自适应滤波器组用于高效SSM（HADES）的新方法，受图信号处理（GSP）启发，将Mamba2重新诠释为线图上的自适应滤波器组。其分层架构引入两类滤波器：用于全局低通行为的共享滤波器和用于局部高通行为的专家滤波器，通过参数Δ的结构化偏置实现。HADES在语言建模、常识推理和长上下文检索等多个基准上表现与基线模型相当，仅需原模型58.9%的参数。该工作连接了图信号处理与神经序列建模，实现了状态空间模型中的高效、分层且可解释的滤波机制。
原文摘要 · Abstract (English)
State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter Δ. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。