用检索增强生成技术提升匹兹堡与卡内基梅隆大学问答准确率
Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU
- 结合网页抓取与人工+AI生成标注，构建领域专用问答数据集
- 在时效性和复杂问题上F1提升至42.21%，召回率达56.18%
- 适合需要高精度领域问答的教育、城市研究场景
我们设计了一种检索增强生成（RAG）系统，用于回答关于匹兹堡和卡内基梅隆大学（CMU）的领域特定问题。通过贪婪爬取策略提取了超过1,800个子页面，并采用混合标注方法，结合人工与Mistral生成的问答对，达成0.7625的标注者间一致性（IAA）评分。RAG框架融合了BM25与FAISS检索器，并引入重排序模块以提升文档检索准确性。实验结果表明，该系统显著优于非RAG基线，在时间敏感和复杂查询任务中表现突出，F1得分从5.45%提升至42.21%，召回率达到56.18%。本研究验证了RAG在提升答案精确性与相关性方面的潜力，同时指出了文档检索与模型训练仍有优化空间。
原文摘要 · Abstract (English)
We designed a Retrieval-Augmented Generation (RAG) system to provide large language models with relevant documents for answering domain-specific questions about Pittsburgh and Carnegie Mellon University (CMU). We extracted over 1,800 subpages using a greedy scraping strategy and employed a hybrid annotation process, combining manual and Mistral-generated question-answer pairs, achieving an inter-annotator agreement (IAA) score of 0.7625. Our RAG framework integrates BM25 and FAISS retrievers, enhanced with a reranker for improved document retrieval accuracy. Experimental results show that the RAG system significantly outperforms a non-RAG baseline, particularly in time-sensitive and complex queries, with an F1 score improvement from 5.45% to 42.21% and recall of 56.18%. This study demonstrates the potential of RAG systems in enhancing answer precision and relevance, while identifying areas for further optimization in document retrieval and model training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。