用Mamba替代Transformer，提升语音自监督模型的长序列与实时处理能力
An Exploration of Mamba for Speech Self-Supervised Models
- 基于Mamba的HuBERT模型利用线性时序选择状态空间，实现高效长上下文建模
- 在流式语音识别上性能优于传统Transformer，且计算开销更低
- 适合需要低延迟、高效率的语音处理场景，如实时转录与声纹提取
尽管Mamba在语言建模中表现优异，其在语音自监督学习（SSL）中的潜力仍待挖掘，以往研究仅限于单一任务。本文探索基于Mamba的HuBERT模型作为Transformer架构的替代方案。利用线性时间选择状态空间机制，该模型可在长上下文语音识别任务中以显著更低的计算成本进行微调，并在流式语音识别任务中表现出更优性能。此外，在SUPERB探针基准测试中，尤其在因果设置下表现良好。分析显示，该模型生成的量化表示质量更高，对说话人特征的捕捉更为清晰。这些结果表明，基于Mamba的语音自监督学习是长序列建模、实时语音处理及语音单元提取的有前景且互补的方向。代码已开源：https://github.com/hckuo145/Mamba-based-HuBERT。
原文摘要 · Abstract (English)
While Mamba has demonstrated strong performance in language modeling, its potential as a speech self-supervised learning (SSL) model remains underexplored, with prior studies limited to isolated tasks. To address this, we explore Mamba-based HuBERT models as alternatives to Transformer-based SSL architectures. Leveraging the linear-time Selective State Space, these models enable fine-tuning on long-context ASR with significantly lower compute. Moreover, they show superior performance when fine-tuned for streaming ASR. Beyond fine-tuning, these models show competitive performance on SUPERB probing benchmarks, particularly in causal settings. Our analysis shows that they yield higher-quality quantized representations and capture speaker-related features more distinctly than Transformer-based models. These findings highlight Mamba-based SSL as a promising and complementary direction for long-sequence modeling, real-time speech modeling, and speech unit extraction. The codebase is available at https://github.com/hckuo145/Mamba-based-HuBERT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。