用软量化让语音与大模型更好对接，提升语音识别效果。
Bridging the Modality Gap: Softly Discretizing Audio Representation for LLM-based Automatic Speech Recognition
- 用大模型的词表做向量量化代码本，实现语音到离散符号的映射。
- 通过加权求和实现软量化，使语音表示更贴近语言结构。
- 在域外数据上表现显著更好，适合语音识别场景的模型融合。
将语音输入与大语言模型（LLM）结合面临语音数据连续性与LLM离散标记范式之间的差异。为缓解这一差距，我们提出一种将向量量化（VQ）融入基于大语言模型的自动语音识别（ASR）的方法。利用LLM的嵌入表作为VQ代码本，使音频编码器产生的连续表示与LLM的离散输入对齐，从而让LLM处理更符合语言结构的离散化语音表示。我们进一步通过更新代码本并对其嵌入进行加权求和，实现语音表示的软量化。实验结果表明，该方法显著优于基于LLM的ASR基线，尤其在域外条件下表现更佳。本工作展示了软量化作为大模型语音识别中模态桥梁的潜力。
原文摘要 · Abstract (English)
One challenge of integrating speech input with large language models (LLMs) stems from the discrepancy between the continuous nature of audio data and the discrete token-based paradigm of LLMs. To mitigate this gap, we propose a method for integrating vector quantization (VQ) into LLM-based automatic speech recognition (ASR). Using the LLM embedding table as the VQ codebook, the VQ module aligns the continuous representations from the audio encoder with the discrete LLM inputs, enabling the LLM to operate on a discretized audio representation that better reflects the linguistic structure. We further create a soft "discretization" of the audio representation by updating the codebook and performing a weighted sum over the codebook embeddings. Empirical results demonstrate that our proposed method significantly improves upon the LLM-based ASR baseline, particularly in out-of-domain conditions. This work highlights the potential of soft discretization as a modality bridge in LLM-based ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。