让语音模型自己修正识别错误，通过隐藏状态互动提升准确性。
Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions
- 利用语音模型与基础大模型的隐藏状态交互，定位需修正的词元。
- 针对高语义依赖词元进行局部修正，显著优于全局重打分方法。
- 适合追求高精度语音识别的开发者或研究者使用。
近期自动语音识别（ASR）系统越来越多地整合大语言模型（LLM）以利用其语义知识，通常通过外部对数融合或内部暖初始化实现。然而，如何有效结合这两种策略仍缺乏探索。本文提出一种基于隐藏状态交互的自纠正方法——混合搜索（Hybrid Search），聚焦于保留基础LLM的LoRA适配设置。核心思路是：首先，语音识别模型与基础大模型之间的隐藏状态交互特征能反映词元的语义依赖程度；其次，仅对高语义依赖词元进行选择性修正，可显著提升性能，远超传统的全局重打分和后期融合方法。分析表明，即使经过暖初始化完成语义知识迁移，基于LLM的ASR模型仍可通过其基础模型在推理阶段进一步优化表现。
原文摘要 · Abstract (English)
Recent automatic speech recognition (ASR) systems increasingly integrate large language models (LLMs) to leverage their semantic knowledge, either externally through logit fusion or internally through warm initialization. However, how to effectively combine these two strategies remains underexplored. In this work, we refine warm-initialized LLM-based ASR models by leveraging their own pre-adaptation base LLMs, focusing on LoRA-adapted settings where the base LLM is preserved. To achieve this, we propose Hybrid Search, a targeted correction strategy motivated by two observations. First, interaction features that characterize the relationship between LLM-based ASR hidden states and base-LLM hidden states provide informative signals about a token's degree of semantic dependence. Second, selectively refining targeted tokens with high semantic dependence improves ASR performance far beyond naive global LLM-correction methods including rescoring and late fusion. Our analysis suggests that, even after semantic knowledge transfer through warm initialization, LLM-based ASR models can still leverage their base LLM to further improve inference-time performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。