让大模型先推理再转录，提升语音识别准确率。
Speech LLMs are Contextual Reasoning Transcribers
- 引入思维链机制，让模型先分析语境再转录。
- 相比传统方法，词错误率降低8.7%，实体错误率降16.9%。
- 支持用户输入提示引导转录，适合复杂场景应用。
尽管已拓展至语音输入，有效利用大语言模型（LLM）在自动语音识别（ASR）中的丰富知识与上下文理解能力仍具挑战，因该任务主要依赖直接的语音到文本映射。为此，本文提出思维链语音识别（CoT-ASR），构建推理链使LLM能先分析输入语音并生成上下文分析，从而充分释放其生成能力。借助此上下文推理，CoT-ASR实现更智能的语音识别，并在单次处理中完成推理与转录。此外，CoT-ASR天然支持用户引导转录：虽设计为自动生成推理，也可无缝整合用户提供的上下文以指导转录，进一步扩展了ASR功能。为减少模态差距，本文引入基于CTC的模态适配器，利用CTC非空白令牌概率对LLM嵌入加权，高效对齐语音编码器输出与LLM的文本潜在空间。实验表明，相较于标准LLM-based ASR，CoT-ASR在词错误率（WER）上相对降低8.7%，实体错误率（EER）降低16.9%。
原文摘要 · Abstract (English)
Despite extensions to speech inputs, effectively leveraging the rich knowledge and contextual understanding of large language models (LLMs) in automatic speech recognition (ASR) remains non-trivial, as the task primarily involves direct speech-to-text mapping. To address this, this paper proposes chain-of-thought ASR (CoT-ASR), which constructs a reasoning chain that enables LLMs to first analyze the input speech and generate contextual analysis, thereby fully exploiting their generative capabilities. With this contextual reasoning, CoT-ASR then performs more informed speech recognition and completes both reasoning and transcription in a single pass. Moreover, CoT-ASR naturally supports user-guided transcription: while designed to self-generate reasoning, it can also seamlessly incorporate user-provided context to guide transcription, further extending ASR functionality. To reduce the modality gap, this paper introduces a CTC-guided Modality Adapter, which uses CTC non-blank token probabilities to weight LLM embeddings, efficiently aligning speech encoder outputs with the LLM's textual latent space. Experiments show that, compared to standard LLM-based ASR, CoT-ASR achieves a relative reduction of 8.7% in word error rate (WER) and 16.9% in entity error rate (EER).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。