EncouRAGe框架实现RAG系统本地高效评估，支持可复现实验。
EncouRAGe: Evaluating RAG Local, Fast, and Reliable
- 模块化设计支持快速搭建和测试RAG系统
- 在25k问答对上验证，混合BM25效果最佳
- 适合需要本地部署与可靠评估的研究者
我们提出EncouRAGe，一个全面的Python框架，用于使用大语言模型（LLM）和嵌入模型开发与评估检索增强生成（RAG）系统。该框架包含五个可扩展模块：类型声明、RAG工厂、推理、向量存储和指标，支持灵活实验与可扩展开发。框架强调科学可复现性、多样评估指标和本地部署，使研究者能高效评估数据集在RAG流程中的表现。本文详述实现细节，并在多个基准数据集上进行广泛评估，涵盖25,000个问答对和超过51,000篇文档。结果表明，相比理想上下文（Oracle Context），RAG仍存在性能差距；而混合BM25在四个数据集上始终表现最优。我们进一步考察重排序影响，发现性能提升有限且响应延迟显著增加。
原文摘要 · Abstract (English)
We introduce EncouRAGe, a comprehensive Python framework designed to streamline the development and evaluation of Retrieval-Augmented Generation (RAG) systems using Large Language Models (LLMs) and Embedding Models. EncouRAGe comprises five modular and extensible components: Type Manifest, RAG Factory, Inference, Vector Store, and Metrics, facilitating flexible experimentation and extensible development. The framework emphasizes scientific reproducibility, diverse evaluation metrics, and local deployment, enabling researchers to efficiently assess datasets within RAG workflows. This paper presents implementation details and an extensive evaluation across multiple benchmark datasets, including 25k QA pairs and over 51k documents. Our results show that RAG still underperforms compared to the Oracle Context, while Hybrid BM25 consistently achieves the best results across all four datasets. We further examine the effects of reranking, observing only marginal performance improvements accompanied by higher response latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。