arXiv:2507.12252cs.CLcs.AI2025-07被引 1

用大模型融合细粒度与整体词组信息,提升语音识别中专有名词准确率

Improving Contextual ASR via Multi-grained Fusion with Large Language Models

  • 分粒度融合:同时结合词级和短语级信息增强上下文理解
  • 在中英文数据集上关键词识别指标达最新水平,普通文本准确率不降
  • 适合需要精准识别人名地名等实体的语音应用,如智能客服、会议转录

端到端语音识别模型在通用语音转录上表现优异,但在识别专有名词或用户特定实体等上下文相关关键词时仍存在困难。此前方法通过文本模态中的关键词词典,分别采用词级融合(引导逐词生成)或短语级融合(直接复制关键词短语),但两者粒度不同,各有局限。本文提出一种新型多粒度融合方法,联合利用大语言模型(LLM)在词级与短语级上的优势。采用后融合策略,巧妙结合语音识别的声学信息与大模型的丰富上下文知识,在细粒度词级精度与整体短语级理解间取得平衡。在中文和英文数据集上的实验表明,该方法在关键词相关指标上达到当前最优性能,同时保持非关键词文本的高准确率。消融实验进一步验证,词级与短语级组件均对性能提升有显著贡献,彼此互补。代码与模型将公开于 https://github.com/。

原文摘要 · Abstract (English)

While end-to-end Automatic Speech Recognition (ASR) models have shown impressive performance in transcribing general speech, they often struggle to accurately recognize contextually relevant keywords, such as proper nouns or user-specific entities. Previous approaches have explored leveraging keyword dictionaries in the textual modality to improve keyword recognition, either through token-level fusion that guides token-by-token generation or phrase-level fusion that enables direct copying of keyword phrases. However, these methods operate at different granularities and have their own limitations. In this paper, we propose a novel multi-grained fusion approach that jointly leverages the strengths of both token-level and phrase-level fusion with Large Language Models (LLMs). Our approach incorporates a late-fusion strategy that elegantly combines ASR's acoustic information with LLM's rich contextual knowledge, balancing fine-grained token precision with holistic phrase-level understanding. Experiments on Chinese and English datasets demonstrate that our approach achieves state-of-the-art performance on keyword-related metrics while preserving high accuracy on non-keyword text. Ablation studies further confirm that the token-level and phrase-level components both contribute significantly to the performance gains, complementing each other in our joint multi-grained framework. The code and models will be publicly available at https://github.com/.

语音识别大模型关键词识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。