无需扫描的视觉状态空间模型，用频域计算提升效率与精度。
HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet
- 直接在频域处理图像，用FFT替代传统扫描策略。
- ImageNet-1K上达85.7%准确率，推理速度比ViT快2.2倍。
- 适合追求高效视觉建模的开发者和部署场景。
视觉状态空间模型（SSM）如Vim、VMamba和SiMBA依赖复杂的扫描策略将序列化SSM适配至二维图像处理，引入计算开销与架构复杂性。本文提出HAMSA，一种无需扫描的频域直接运算的SSM。其核心创新包括：(1) 简化核参数化——使用单个高斯初始化的复数核替代传统(A, B, C)矩阵，消除离散化不稳定性；(2) SpectralPulseNet（SPN）——输入依赖的频率门控机制，实现自适应频谱调制；(3) Spectral Adaptive Gating Unit（SAGU）——基于幅值的门控，保障频域梯度稳定流动。通过基于FFT的卷积，HAMSA消除了序列扫描，实现O(L log L)复杂度，兼具简洁性与高效性。在ImageNet-1K上达到85.7%的top-1准确率（当前SSM中最高），推理速度比DeiT-S快2.2倍（4.2ms vs 9.2ms），较扫描型SSM提速1.4–1.9倍，内存占用更低（2.1GB vs 3.2–4.5GB），能耗更少（12.5J vs 18–25J）。HAMSA在迁移学习与密集预测任务中展现出强泛化能力。
原文摘要 · Abstract (English)
Vision State Space Models (SSMs) like Vim, VMamba, and SiMBA rely on complex scanning strategies to adapt sequential SSMs to process 2D images, introducing computational overhead and architectural complexity. We propose HAMSA, a scanning-free SSM operating directly in the spectral domain. HAMSA introduces three key innovations: (1) simplified kernel parameterization-a single Gaussian-initialized complex kernel replacing traditional (A, B, C) matrices, eliminating discretization instabilities; (2) SpectralPulseNet (SPN)-an input-dependent frequency gating mechanism enabling adaptive spectral modulation; and (3) Spectral Adaptive Gating Unit (SAGU)-magnitude-based gating for stable gradient flow in the frequency domain. By leveraging FFT-based convolution, HAMSA eliminates sequential scanning while achieving O(L log L) complexity with superior simplicity and efficiency. On ImageNet-1K, HAMSA reaches 85.7% top-1 accuracy (state-of-the-art among SSMs), with 2.2 X faster inference than transformers (4.2ms vs 9.2ms for DeiT-S) and 1.4-1.9X speedup over scanning-based SSMs, while using less memory (2.1GB vs 3.2-4.5GB) and energy (12.5J vs 18-25J). HAMSA demonstrates strong generalization across transfer learning and dense prediction tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。