用语音检索增强大模型，提升语音助手对人名的识别准确率
Contextualization of ASR with LLM using phonetic retrieval-based augmentation
- 通过语音相似性从个人数据库中检索候选人名，供大模型参考
- 在语音助手任务中，人名识别错误率降低73.6%，词错率降30.2%
- 无需加载全部人名库，高效适配大规模命名实体场景
大语言模型（LLM）在建模多模态信号方面表现出色，能够根据语音输入生成语音或文本响应。然而，在语音输入场景下，模型仍难以识别个人命名实体（如电话簿中的联系人）。本文针对语音识别任务，提出一种基于检索的上下文增强方法：先让LLM在无上下文情况下检测语音中的命名实体，再以该实体为查询，从个人数据库中检索语音上相似的候选实体，并将其作为上下文输入给LLM，最后进行上下文感知的解码。在语音助手任务中，该方案相较无上下文基准系统，实现了最高达30.2%的相对词错误率降低和73.6%的相对命名实体错误率降低。值得注意的是，该方法避免向LLM注入完整命名实体数据库，因此具有高效率，适用于大规模命名实体场景。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown superb capability of modeling multimodal signals including audio and text, allowing the model to generate spoken or textual response given a speech input. However, it remains a challenge for the model to recognize personal named entities, such as contacts in a phone book, when the input modality is speech. In this work, we start with a speech recognition task and propose a retrieval-based solution to contextualize the LLM: we first let the LLM detect named entities in speech without any context, then use this named entity as a query to retrieve phonetically similar named entities from a personal database and feed them to the LLM, and finally run context-aware LLM decoding. In a voice assistant task, our solution achieved up to 30.2% relative word error rate reduction and 73.6% relative named entity error rate reduction compared to a baseline system without contextualization. Notably, our solution by design avoids prompting the LLM with the full named entity database, making it highly efficient and applicable to large named entity databases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。