让语音直接检索文档，速度翻倍且更准。
SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation
- 语音和文本共用嵌入空间，跳过中间转写环节。
- 延迟降低50%，在不同语音条件下准确率更高。
- 适合做语音大模型的检索系统，尤其对低质量语音有效。
基于嵌入的检索模型在文本和多模态大语言模型（LLMs）的检索增强生成（RAG）中取得了显著进展。然而，针对语音大语言模型（SLLMs），现有方法仍依赖两阶段流程：先通过自动语音识别（ASR）转写语音，再进行文本检索。这种串行架构导致延迟高且错误传播严重。为此，我们提出一种统一的嵌入框架，无需中间文本表示。该框架包含独立的语音与文本编码器，后接共享缩放层，将双模态映射至统一嵌入空间。实验表明，本模型相较传统两阶段方法降低50%流水线延迟，并实现更高检索准确率。我们还从理论上分析了端到端语音检索的挑战，提出了有效的语音-文档匹配架构原则。大量实验证明，该方法在多样声学条件与说话人差异下均具鲁棒性，为多模态SLLMs检索系统开辟新范式。
原文摘要 · Abstract (English)
Embedding-based retrieval models have made significant strides in retrieval-augmented generation (RAG) techniques for text and multimodal large language models (LLMs) applications. However, when it comes to speech larage language models (SLLMs), these methods are limited to a two-stage process, where automatic speech recognition (ASR) is combined with text-based retrieval. This sequential architecture suffers from high latency and error propagation. To address these limitations, we propose a unified embedding framework that eliminates the need for intermediate text representations. Specifically, the framework includes separate speech and text encoders, followed by a shared scaling layer that maps both modalities into a common embedding space. Our model reduces pipeline latency by 50\% while achieving higher retrieval accuracy compared to traditional two-stage methods. We also provide a theoretical analysis of the challenges inherent in end-to-end speech retrieval and introduce architectural principles for effective speech-to-document matching. Extensive experiments demonstrate the robustness of our approach across diverse acoustic conditions and speaker variations, paving the way for a new paradigm in multimodal SLLMs retrieval systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。