用多项式混合机制替代自注意力，提升语音编码器效率
Polynomial Mixing for Efficient Self-supervised Speech Encoders
- 提出多项式混合机制（PoM），线性复杂度替代二次复杂度的自注意力
- 在语音识别任务中达到与全自注意力相当的词错误率
- 适合追求高效语音模型的开发者和研究者
当前先进的语音转文本模型普遍采用基于Transformer的编码器，通过自注意力机制建模词元依赖关系。然而，自注意力在内存和计算上的二次复杂度严重制约了模型可扩展性。本文提出一种新型词元混合机制——多项式混合器（PoM），作为多头自注意力的即插即用替代方案。PoM以线性复杂度计算输入的多项式表示，与输入序列长度成线性关系。我们将PoM集成至基于BEST-RQ的自监督语音表征学习框架，并在下游语音识别任务上进行评估。实验结果表明，PoM在词错误率上与完整自注意力及其他线性复杂度方法相比具有竞争力，同时在时间和内存效率上实现了更优的性能-效率权衡。
原文摘要 · Abstract (English)
State-of-the-art speech-to-text models typically employ Transformer-based encoders that model token dependencies via self-attention mechanisms. However, the quadratic complexity of self-attention in both memory and computation imposes significant constraints on scalability. In this work, we propose a novel token-mixing mechanism, the Polynomial Mixer (PoM), as a drop-in replacement for multi-head self-attention. PoM computes a polynomial representation of the input with linear complexity with respect to the input sequence length. We integrate PoM into a self-supervised speech representation learning framework based on BEST-RQ and evaluate its performance on downstream speech recognition tasks. Experimental results demonstrate that PoM achieves a competitive word error rate compared to full self-attention and other linear-complexity alternatives, offering an improved trade-off between performance and efficiency in time and memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。