优化自监督语音识别模型，提升准确率并加快训练收敛。
Optimized Self-supervised Training with BEST-RQ for Speech Recognition
- 引入KL散度正则化和多码本机制改进预训练
- 在LibriSpeech上测试集词错误率降低超23%
- 适合追求高效高精度语音识别的开发者
自监督学习已在语音识别等任务中取得成功。基于BERT的语音预训练模型BEST-RQ已达到顶尖性能。本文通过引入KL散度作为额外正则损失，并基于低层特征聚类构建每簇多码本机制，进一步优化BEST-RQ。在LibriSpeech的train-100数据集上，使用多码本使test-clean的词错误率相对降低11.2%；结合交叉熵与KL散度后，词错误率再降4.5%。在完整LibriSpeech数据集上，采用6个码本的预训练与微调方案，在test-clean上实现最高23.8%的相对词错误率下降，test-other上达30.6%。此外，该方法加速了预训练与微调的收敛，并提升了训练稳定性。
原文摘要 · Abstract (English)
Self-supervised learning has been successfully used for various speech related tasks, including automatic speech recognition. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ) has achieved state-of-the-art results in speech recognition. In this work, we further optimize the BEST-RQ approach using Kullback-Leibler divergence as an additional regularizing loss and multi-codebook extension per cluster derived from low-level feature clustering. Preliminary experiments on train-100 split of LibriSpeech result in a relative improvement of 11.2% on test-clean by using multiple codebooks, utilizing a combination of cross-entropy and Kullback-Leibler divergence further reduces the word error rate by 4.5%. The proposed optimizations on full LibriSpeech pre-training and fine-tuning result in relative word error rate improvements of up to 23.8% on test-clean and 30.6% on test-other using 6 codebooks. Furthermore, the proposed setup leads to faster convergence in pre-training and fine-tuning and additionally stabilizes the pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。