用Mamba做纯解码器语音识别，效果接近Transformer。
Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognition
- 纯解码器架构，直接从语音标记生成文本。
- 结合双向语音前缀，大模型上性能媲美Transformer。
- 适合追求高效推理的语音识别研究者。
以Mamba为代表的选通状态空间模型（SSMs）在自动语音识别（ASR）等任务中展现出计算效率高、效果优异的特点。现有工作将Mamba用于基于注意力的编码器-解码器框架，仍保留编码器与解码器间的交叉注意力机制。本文探索将Mamba作为纯解码器架构在ASR中的应用，提出MAmba-based DEcoder-ONly方法（MADEON），其仅包含一个解码器，以语音标记为条件，自回归地预测文本标记。为进一步提升MADEON性能，我们提出语音前缀（speech prefixing）策略，对语音标记进行双向处理，从而丰富隐藏状态中的上下文信息。实验表明，MADEON显著优于非选通的SSM；结合近期提出的Mamba-2，该方法在大规模数据集上的表现可与基于Transformer的模型相媲美。
原文摘要 · Abstract (English)
Selective state space models (SSMs) represented by Mamba have demonstrated their computational efficiency and promising outcomes in various tasks, including automatic speech recognition (ASR). Mamba has been applied to ASR task with the attention-based encoder-decoder framework, where the cross-attention mechanism between encoder and decoder remains. This paper explores the capability of Mamba as the decoder-only architecture in ASR task. Our MAmba-based DEcoder-ONly approach (MADEON) consists of a single decoder that takes speech tokens as a condition and predicts text tokens in an autoregressive manner. To enhance MADEON, we further propose speech prefixing that performs bidirectional processing on speech tokens, which enriches the contextual information in the hidden states. Our experiments show that MADEON significantly outperforms a non-selective SSM. The combination of speech prefixing and the recently proposed Mamba-2 yields comparable performance to Transformer-based models on large datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。