用检索增强生成提升虚拟现实中的问答准确率
RAG-VR: Leveraging Retrieval-Augmented Generation for 3D Question Answering in VR Environments
- 将外部知识库检索与大模型结合，动态补充场景信息
- 相比基线系统，准确率提升17.9%~41.8%，延迟降低34.5%~47.3%
- 适合需要高精度实时问答的虚拟现实应用开发者
大型语言模型（LLMs）为虚拟现实（VR）环境中的上下文理解带来了新机遇。然而，VR上下文通常高度局部化且个性化，限制了通用大模型的效果。为此，我们提出RAG-VR，首个基于检索增强生成（RAG）的3D VR问答系统，通过从本地知识库中检索外部知识来增强大模型，从而提升答案质量。RAG-VR包含一套提取虚拟环境与用户状态完整知识的流水线，以实现精准回答生成。为确保高效检索，系统将检索过程卸载至邻近边缘服务器，并仅使用关键信息进行检索。此外，我们训练检索器以有效区分相关、无关及难以区分的信息。相比两种基线系统，RAG-VR在端到端延迟上减少34.5%~47.3%，准确率提升17.9%~41.8%。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) provide new opportunities for context understanding in virtual reality (VR). However, VR contexts are often highly localized and personalized, limiting the effectiveness of general-purpose LLMs. To address this challenge, we present RAG-VR, the first 3D question-answering system for VR that incorporates retrieval-augmented generation (RAG), which augments an LLM with external knowledge retrieved from a localized knowledge database to improve the answer quality. RAG-VR includes a pipeline for extracting comprehensive knowledge about virtual environments and user conditions for accurate answer generation. To ensure efficient retrieval, RAG-VR offloads the retrieval process to a nearby edge server and uses only essential information during retrieval. Moreover, we train the retriever to effectively distinguish among relevant, irrelevant, and hard-to-differentiate information in relation to questions. RAG-VR improves answer accuracy by 17.9%-41.8% and reduces end-to-end latency by 34.5%-47.3% compared with two baseline systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。