用线性注意力替代自注意力，提速降显存且性能不降。
An Analysis of Linear Complexity Attention Substitutes with BEST-RQ
- 用四种线性复杂度注意力替代原版多头自注意力
- 显存减少20%~60%，速度提升7%~65%，性能接近原模型
- 适合追求高效训练的语音自监督学习研究者
自监督学习（SSL）在语音处理等众多领域表现优异，但计算和内存开销大，部分原因在于多头自注意力（MHSA）具有二次复杂度。尽管已有多种线性复杂度替代方法被提出并应用于语音领域，但在完整SSL设置下的系统评估仍不足。本文研究了将MHSA替换为最新线性复杂度方法——HyperMixing、Fastformer、SummaryMixing和Mamba的效果，评估指标包括速度、显存占用及在SSL MP3S基准上的性能。结果表明，这些线性替代方案在输入序列长度为20至80秒时，平均显存消耗降低20%至60%，推理速度提升7%至65%，同时保持与原模型相当的性能水平。
原文摘要 · Abstract (English)
Self-Supervised Learning (SSL) has proven to be effective in various domains, including speech processing. However, SSL is computationally and memory expensive. This is in part due the quadratic complexity of multi-head self-attention (MHSA). Alternatives for MHSA have been proposed and used in the speech domain, but have yet to be investigated properly in an SSL setting. In this work, we study the effects of replacing MHSA with recent state-of-the-art alternatives that have linear complexity, namely, HyperMixing, Fastformer, SummaryMixing, and Mamba. We evaluate these methods by looking at the speed, the amount of VRAM consumed, and the performance on the SSL MP3S benchmark. Results show that these linear alternatives maintain competitive performance compared to MHSA while, on average, decreasing VRAM consumption by around 20% to 60% and increasing speed from 7% to 65% for input sequences ranging from 20 to 80 seconds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。