无需语音识别，直接用语音提问也能精准检索答案。
A Multimodal Dense Retrieval Approach for Speech-Based Open-Domain Question Answering
- 直接处理语音输入，跳过语音转文字步骤。
- 在关键词误识别时表现优于传统流水线方法。
- 适合低资源语言和语音数据稀缺的场景。
语音驱动的开放域问答（基于大量文本段落回答语音提问）因语音交互用户增多而变得重要。段落检索是其中关键任务。以往方法依赖语音识别（ASR）将语音转为文本后再由稠密文本检索器处理，存在局限：需大量标注语音数据，限制低资源语言和专业领域应用；且ASR错误会传递给检索器。本文提出一种免ASR、端到端训练的多模态稠密检索模型，可直接处理语音问题。实验表明，在短问题场景下，该模型在ASR易出错（如关键词误识别或高词错误率）时，检索性能优于传统流水线，展现出良好替代潜力。
原文摘要 · Abstract (English)
Speech-based open-domain question answering (QA over a large corpus of text passages with spoken questions) has emerged as an important task due to the increasing number of users interacting with QA systems via speech interfaces. Passage retrieval is a key task in speech-based open-domain QA. So far, previous works adopted pipelines consisting of an automatic speech recognition (ASR) model that transcribes the spoken question before feeding it to a dense text retriever. Such pipelines have several limitations. The need for an ASR model limits the applicability to low-resource languages and specialized domains with no annotated speech data. Furthermore, the ASR model propagates its errors to the retriever. In this work, we try to alleviate these limitations by proposing an ASR-free, end-to-end trained multimodal dense retriever that can work directly on spoken questions. Our experimental results showed that, on shorter questions, our retriever is a promising alternative to the \textit{ASR and Retriever} pipeline, achieving better retrieval performance in cases where ASR would have mistranscribed important words in the question or have produced a transcription with a high word error rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。