用未来预测训练语音模型，更适合实时语音识别。
NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training
- 采用因果编码器和未来词预测任务，支持流式语音处理。
- 在非流式识别上性能接近BEST-RQ，流式识别更优。
- 为语音自监督学习提供新思路，适合实时语音场景研究。
语音自监督预训练能有效提升下游任务性能。然而，以往的语音自监督学习方法（如HuBERT和BEST-RQ）多采用双向上下文的非因果编码器，难以支持下游流式模型。为此，我们提出基于随机投影量化器的未来词预测语音预训练方法（NEST-RQ）。该方法使用仅依赖左文的因果编码器，并以未来词预测（NTP）作为训练目标。在大规模数据集上，相比BEST-RQ，NEST-RQ在非流式自动语音识别（ASR）上表现相当，在流式ASR上表现更优。我们还分析了流式ASR中未来上下文大小、自监督编码器码本质量及编码器规模的影响。结果表明，未来词预测在语音自监督学习中具有可行性，并提供了实证依据与研究洞见。
原文摘要 · Abstract (English)
Speech self-supervised pre-training can effectively improve the performance of downstream tasks. However, previous self-supervised learning (SSL) methods for speech, such as HuBERT and BEST-RQ, focus on utilizing non-causal encoders with bidirectional context, and lack sufficient support for downstream streaming models. To address this issue, we introduce the next token prediction based speech pre-training method with random-projection quantizer (NEST-RQ). NEST-RQ employs causal encoders with only left context and uses next token prediction (NTP) as the training task. On the large-scale dataset, compared to BEST-RQ, the proposed NEST-RQ achieves comparable performance on non-streaming automatic speech recognition (ASR) and better performance on streaming ASR. We also conduct analytical experiments in terms of the future context size of streaming ASR, the codebook quality of SSL and the model size of the encoder. In summary, the paper demonstrates the feasibility of the NTP in speech SSL and provides empirical evidence and insights for speech SSL research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。