无需转写语音，直接用声音检索音频片段，提升口语问答效率。
VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering
- 用音频嵌入和声学特征直接匹配语音查询与候选片段。
- 相关片段召回率达0.60（部分相关），信息相关性指标达0.27。
- 适合语音交互、无障碍访问等场景，减少转写误差影响。
我们提出VoxRAG，一种模块化语音到语音的检索增强生成系统，无需文本转写即可从语音查询中直接检索语义相关的音频段。该系统采用静音感知分段、说话人分离、CLAP音频嵌入，并使用基于L2归一化余弦相似度的FAISS检索。我们构建了一个由母语英语者录制的50个语音查询测试集，通过大模型作为评判者评估检索质量。对于高度相关的音频段，余弦相似度在Recall@10上达到0.34；对于部分相关的音频段，Recall@10提升至0.60，nDCG@10达0.27，显示良好的主题一致性。答案质量按相关性、准确性、完整性与精确性四维度评分，平均得分分别为0.84、0.58、0.56和0.46。尽管精确性和检索性能仍是主要挑战，但实验表明，在RAG系统中实现无转写语音到语音检索是可行的。
原文摘要 · Abstract (English)
We introduce VoxRAG, a modular speech-to-speech retrieval-augmented generation system that bypasses transcription to retrieve semantically relevant audio segments directly from spoken queries. VoxRAG employs silence-aware segmentation, speaker diarization, CLAP audio embeddings, and FAISS retrieval using L2-normalized cosine similarity. We construct a 50-query test set recorded as spoken input by a native English speaker. Retrieval quality was evaluated using LLM-as-a-judge annotations. For very relevant segments, cosine similarity achieved a Recall@10 of 0.34. For somewhat relevant segments, Recall@10 rose to 0.60 and nDCG@10 to 0.27, highlighting strong topical alignment. Answer quality was judged on a 0--2 scale across relevance, accuracy, completeness, and precision, with mean scores of 0.84, 0.58, 0.56, and 0.46 respectively. While precision and retrieval quality remain key limitations, VoxRAG shows that transcription-free speech-to-speech retrieval is feasible in RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。