用生成式AI构建中文知识检索系统,提升问答准确率与数据安全。
Knowledge Retrieval Based on Generative AI
- 结合BGE-M3与BGE-reranker实现精准检索与重排序。
- 自动评估准确率提升,人工辅助下答题正确率显著改善。
- 本地化部署保障隐私,适合金融等敏感领域应用。
本研究基于检索增强生成(RAG)构建问答系统,采用中文维基百科和法务数据库作为检索源。使用TTQA和TMMLU+作为评估数据集,通过BGE-M3进行密集向量检索,再由BGE-reranker根据查询相关性重新排序结果,将最相关的检索内容作为大语言模型(LLM)的参考知识,增强其问答能力,建立基于生成式AI的知识检索系统。评估分为两阶段:自动评估通过比较模型自动生成答案与真实答案的准确率,在无人员干预下衡量性能;辅助评估让20名无金融背景的参与者回答20道金融相关选择题,先独立作答,再接收系统生成的参考信息协助作答,检验系统是否提升准确性。主要贡献包括:(1) 增强LLM能力:通过BGE-M3与BGE-reranker实现高相关性检索与重排序,减少幻觉,动态访问授权或公开知识源;(2) 提升数据隐私:定制化RAG架构支持本地运行LLM,无需将私密数据上传外部服务器,增强安全性,降低对商业服务依赖与运营成本,缓解隐私风险。
原文摘要 · Abstract (English)
This study develops a question-answering system based on Retrieval-Augmented Generation (RAG) using Chinese Wikipedia and Lawbank as retrieval sources. Using TTQA and TMMLU+ as evaluation datasets, the system employs BGE-M3 for dense vector retrieval to obtain highly relevant search results and BGE-reranker to reorder these results based on query relevance. The most pertinent retrieval outcomes serve as reference knowledge for a Large Language Model (LLM), enhancing its ability to answer questions and establishing a knowledge retrieval system grounded in generative AI. The system's effectiveness is assessed through a two-stage evaluation: automatic and assisted performance evaluations. The automatic evaluation calculates accuracy by comparing the model's auto-generated labels with ground truth answers, measuring performance under standardized conditions without human intervention. The assisted performance evaluation involves 20 finance-related multiple-choice questions answered by 20 participants without financial backgrounds. Initially, participants answer independently. Later, they receive system-generated reference information to assist in answering, examining whether the system improves accuracy when assistance is provided. The main contributions of this research are: (1) Enhanced LLM Capability: By integrating BGE-M3 and BGE-reranker, the system retrieves and reorders highly relevant results, reduces hallucinations, and dynamically accesses authorized or public knowledge sources. (2) Improved Data Privacy: A customized RAG architecture enables local operation of the LLM, eliminating the need to send private data to external servers. This approach enhances data security, reduces reliance on commercial services, lowers operational costs, and mitigates privacy risks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。