对比离散与连续语音表示在大模型语音识别中的效果
Comparing Discrete and Continuous Space LLMs for Speech Recognition
- 按特征连续性与训练方式分四类，构建连续/离散空间大模型框架
- 在LibriSpeech上达1.69%词错误率，为当前最优结果
- 开源成果适合语音识别与自然语言处理研究者参考
本文研究基于大语言模型（LLM）的自动语音识别（ASR）中离散与连续语音表示的差异，根据特征连续性和训练方法将模型分为四类：有监督与无监督的离散及连续类型。进一步依据输入和自回归反馈方式，将LLM划分为连续空间与离散空间模型。通过专用编码器，并与联合训练的从头开始语言模型（JTFS LM）及预训练的LLaMA2-7b对比，深入分析其有效性。本工作首次系统比较了基于LLM的语音表示方法，探索多种建模技术。我们实现了使用HuBERT编码器在LibriSpeech上的1.69%词错误率（WER），达到当前最先进水平，为提升语音识别与自然语言处理研究提供重要参考。
原文摘要 · Abstract (English)
This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69\% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。