提出新评估指标OPI，精准衡量RAG系统处理深层逻辑问题的能力。
Intrinsic Evaluation of RAG Systems for Deep-Logic Questions
- 用逻辑关系准确率与答案相似度的调和平均构建OPI评估指标。
- BERT嵌入相似度与外部评估得分高度相关，验证了指标有效性。
- 多检索器融合优于单一检索器，尤其以余弦相似度检索表现最佳。
我们提出了总体性能指数（OPI），一种用于评估涉及深层逻辑查询的检索增强生成（RAG）机制的内在指标。OPI是逻辑关系正确率与真实答案与生成答案之间BERT嵌入相似度平均值的调和平均。我们利用在Hugging Face的RAG-Dataset-12000上微调的GPT-4o逻辑关系分类器，对流行的RAG工具LangChain进行评估。结果表明，BERT嵌入相似度与外部评估分数存在强相关性。在常用检索器中，基于BERT嵌入的余弦相似度检索器表现最优，而基于欧氏距离的检索器最差。此外，通过算法组合或合并检索句子的方式融合多个检索器，性能优于单一检索器。
原文摘要 · Abstract (English)
We introduce the Overall Performance Index (OPI), an intrinsic metric to evaluate retrieval-augmented generation (RAG) mechanisms for applications involving deep-logic queries. OPI is computed as the harmonic mean of two key metrics: the Logical-Relation Correctness Ratio and the average of BERT embedding similarity scores between ground-truth and generated answers. We apply OPI to assess the performance of LangChain, a popular RAG tool, using a logical relations classifier fine-tuned from GPT-4o on the RAG-Dataset-12000 from Hugging Face. Our findings show a strong correlation between BERT embedding similarity scores and extrinsic evaluation scores. Among the commonly used retrievers, the cosine similarity retriever using BERT-based embeddings outperforms others, while the Euclidean distance-based retriever exhibits the weakest performance. Furthermore, we demonstrate that combining multiple retrievers, either algorithmically or by merging retrieved sentences, yields superior performance compared to using any single retriever alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。