arXiv:2501.13880cs.CL2025-01被引 6

用检索增强生成技术,打造专为巴西圣保罗大学服务的智能助手。

A RAG-Based Institutional Assistant

  • 采用检索+生成双模块架构,结合不同模型与参数优化
  • 准确率最高达54.02%,依赖正确文档时提升超30个百分点
  • 揭示当前语义检索精度不足,适合高校知识服务场景

尽管大语言模型具备强大的文本生成能力,但在需要访问结构化知识库或特定文档的场景下表现不佳,限制了其在知识密集型任务中的应用。为此,本文设计并评估了一个面向巴西圣保罗大学的基于检索增强生成(RAG)的虚拟助手。系统由检索器与生成模型两部分构成,实验对比了多种模型组合及超参数(如分块大小、召回文档数量)。最优检索模型达到Top-5准确率30%,最优生成模型在与真实答案比对时得分为22.04%。值得注意的是,当提供正确文档片段时,准确率显著提升至54.02%,增幅超过30个百分点;反之,无上下文输入时性能降至13.68%。结果表明数据库访问对提升大模型表现至关重要,也暴露了现有语义搜索在精准定位相关文档方面的局限性,凸显大模型生成精确回答仍面临挑战。

原文摘要 · Abstract (English)

Although large language models (LLMs) demonstrate strong text generation capabilities, they struggle in scenarios requiring access to structured knowledge bases or specific documents, limiting their effectiveness in knowledge-intensive tasks. To address this limitation, retrieval-augmented generation (RAG) models have been developed, enabling generative models to incorporate relevant document fragments into their inputs. In this paper, we design and evaluate a RAG-based virtual assistant specifically tailored for the University of São Paulo. Our system architecture comprises two key modules: a retriever and a generative model. We experiment with different types of models for both components, adjusting hyperparameters such as chunk size and the number of retrieved documents. Our optimal retriever model achieves a Top-5 accuracy of 30%, while our most effective generative model scores 22.04\% against ground truth answers. Notably, when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%, an increase of over 30 percentage points. Conversely, without contextual input, performance declines to 13.68%. These findings highlight the critical role of database access in enhancing LLM performance. They also reveal the limitations of current semantic search methods in accurately identifying relevant documents and underscore the ongoing challenges LLMs face in generating precise responses.

RAG知识问答智能助手大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。