无需文字识别，直接用语音问答文档图像。
TextlessRAG: End-to-End Visual Document RAG by Speech Without Text
- 语音输入直接处理，全程不依赖文本转换
- 在中英文双语数据集上准确率显著提升
- 适合移动端、无障碍场景的智能文档助手
文档图像包含丰富知识，而语音查询具有更好的便携性与应用灵活性。然而，此前尚无研究探索基于语音查询对视觉文档图像进行知识库问答。本文提出 TextlessRAG，首个端到端的语音驱动文档图像问答框架。该框架摒弃 ASR、TTS 和 OCR，构建完全无文本的流水线，直接解析语音、检索相关视觉知识并生成答案。为进一步提升性能，引入布局感知重排序机制优化检索结果。实验表明，在效率与准确率方面均有显著提升。为推动该方向研究，我们发布了首个中英双语语音-文档 RAG 数据集，包含中英文语音查询与多模态文档内容。代码与数据集将开源于：https://github.com/xiepeijinhit-hue/textlessrag
原文摘要 · Abstract (English)
Document images encapsulate a wealth of knowledge, while the portability of spoken queries enables broader and flexible application scenarios. Yet, no prior work has explored knowledge base question answering over visual document images with queries provided directly in speech. We propose TextlessRAG, the first end-to-end framework for speech-based question answering over large-scale document images. Unlike prior methods, TextlessRAG eliminates ASR, TTS and OCR, directly interpreting speech, retrieving relevant visual knowledge, and generating answers in a fully textless pipeline. To further boost performance, we integrate a layout-aware reranking mechanism to refine retrieval. Experiments demonstrate substantial improvements in both efficiency and accuracy. To advance research in this direction, we also release the first bilingual speech--document RAG dataset, featuring Chinese and English voice queries paired with multimodal document content. Both the dataset and our pipeline will be made available at repository:https://github.com/xiepeijinhit-hue/textlessrag
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。