对比Mamba、注意力与混合模型，发现长音频下注意力更优
HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit
- 构建参数对齐的框架，比较纯Mamba、纯注意力与混合模型
- 5分钟语音识别任务中，混合模型比纯Mamba提升11.5分
- 短音频时注意力反而降低性能，长序列才显现优势
音频表征学习通常孤立评估输入前端、序列主干和序列长度等设计选择。我们发现这些因素相互耦合，单一设置下的结论难以推广。本文提出HELIX，一个受控框架，用于比较纯Mamba、纯注意力以及仅含单个注意力瓶颈的最小混合模型。所有模型参数量约830万，以隔离架构差异。在六个数据集上，我们发现最优输入表示依赖于主干结构；注意力在短而平稳的音频上损害性能，但在长序列中变得重要。在包含30,000个标记的5分钟说话人识别任务中，纯注意力因显存溢出失败，而HELIX相比纯Mamba提升了11.5个百分点。
原文摘要 · Abstract (English)
Audio representation learning typically evaluates design choices such as input frontend, sequence backbone, and sequence length in isolation. We show that these axes are coupled, and conclusions from one setting often do not transfer to others. We introduce HELIX, a controlled framework comparing pure Mamba, pure attention, and a minimal hybrid with a single attention bottleneck. All models are parameter-matched at about 8.3M parameters to isolate architectural effects. Across six datasets, we find that the preferred input representation depends on the backbone, and that attention hurts performance on short, stationary audio but becomes important at longer sequence lengths. On a 5-minute speaker identification task with 30,000 tokens, pure attention fails with out-of-memory errors, while HELIX closes an 11.5-point gap over pure Mamba.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。