WavRAG让语音对话模型直接用音频检索知识，速度提升10倍。
WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models
- 跳过语音转文字，直接用原始音频做检索和嵌入
- 在相同效果下，推理速度比传统方法快10倍
- 适合需要低延迟语音交互的智能助手场景
检索增强生成（RAG）因能为大语言模型注入外部知识而广泛应用。然而现有RAG框架主要面向文本模型，依赖自动语音识别处理语音输入，会丢失关键音频信息、产生识别错误并增加计算开销。为此，我们提出首个原生支持端到端音频的RAG框架WavRAG。其核心特性包括：1）绕过ASR，直接使用原始音频进行嵌入与检索；2）将音频与文本融合为统一知识表征。我们设计了WavRetriever，实现从图文混合知识库中检索，并通过思维链推理增强语音对话模型的上下文理解能力。相比最先进的ASR-文本RAG流水线，WavRAG在检索性能相当的前提下，实现了10倍加速。此外，其独特的音文混合检索能力，将RAG的应用边界拓展至音频模态。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) has gained widespread adoption owing to its capacity to empower large language models (LLMs) to integrate external knowledge. However, existing RAG frameworks are primarily designed for text-based LLMs and rely on Automatic Speech Recognition to process speech input, which discards crucial audio information, risks transcription errors, and increases computational overhead. Therefore, we introduce WavRAG, the first retrieval augmented generation framework with native, end-to-end audio support. WavRAG offers two key features: 1) Bypassing ASR, WavRAG directly processes raw audio for both embedding and retrieval. 2) WavRAG integrates audio and text into a unified knowledge representation. Specifically, we propose the WavRetriever to facilitate the retrieval from a text-audio hybrid knowledge base, and further enhance the in-context capabilities of spoken dialogue models through the integration of chain-of-thought reasoning. In comparison to state-of-the-art ASR-Text RAG pipelines, WavRAG achieves comparable retrieval performance while delivering a 10x acceleration. Furthermore, WavRAG's unique text-audio hybrid retrieval capability extends the boundaries of RAG to the audio modality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。