通过后处理修正引用错误,提升RAG系统引用准确率。
CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction
- 用关键词+语义匹配、BERTScore微调模型和轻量LLM交叉验证引用
- 引用准确率相对提升15.46%,达到接近90%的高水平
- 可替换大模型为更小模型,降低成本与延迟3倍以上
检索增强生成(RAG)结合传统搜索与大语言模型,用于生成带准确引用的答案。但在实际产品开发中发现,主流生成式搜索引擎的引用准确率仅约74%。为此,本文提出高效后处理算法,通过关键词与语义匹配、微调的BERTScore模型以及轻量级LLM方法,对生成结果中的引用进行交叉验证。实验表明,该方法使RAG系统的整体引用准确率提升15.46%。这一改进使得在保持相近性能的前提下,可将原本依赖的大模型替换为约12倍更低成本、3倍更快推理速度的小模型,显著提升商业产品中AI内容的可信度与实用性。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) has emerged as a powerful application of Large Language Models (LLMs), revolutionizing information search and consumption. RAG systems combine traditional search capabilities with LLMs to generate comprehensive answers to user queries, ideally with accurate citations. However, in our experience of developing a RAG product, LLMs often struggle with source attribution, aligning with other industry studies reporting citation accuracy rates of only about 74% for popular generative search engines. To address this, we present efficient post-processing algorithms to improve citation accuracy in LLM-generated responses, with minimal impact on latency and cost. Our approaches cross-check generated citations against retrieved articles using methods including keyword + semantic matching, fine tuned model with BERTScore, and a lightweight LLM-based technique. Our experimental results demonstrate a relative improvement of 15.46% in the overall accuracy metrics of our RAG system. This significant enhancement potentially enables a shift from our current larger language model to a relatively smaller model that is approximately 12x more cost-effective and 3x faster in inference time, while maintaining comparable performance. This research contributes to enhancing the reliability and trustworthiness of AI-generated content in information retrieval and summarization tasks which is critical to gain customer trust especially in commercial products.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。