用大模型生成数据评估RAG系统，部分有效但有局限。
Can we Evaluate RAGs with Synthetic Data?
- 用LLM生成问答对替代真人标注数据做评估
- 在检索器配置变化时，合成数据排名与人工数据一致
- 不同生成器比较时结果不可靠，因任务错配和风格偏见
我们研究大型语言模型（LLMs）生成的合成问题-答案（QA）数据，能否在缺乏人工标注基准时作为有效代理。通过两个实验评估合成基准的可靠性：一个固定生成器，改变检索器参数；另一个固定检索器，改变生成器参数。在四个数据集上（两个开放域，两个专有），结果显示，当检索器配置变化时，合成基准能可靠地对RAG系统进行排序，与人工标注基准高度一致。然而，在比较不同生成器架构时，合成基准未能持续产生可靠排序。这可能源于合成数据与人工数据之间的任务不匹配，以及对某些生成器的风格偏好。
原文摘要 · Abstract (English)
We investigate whether synthetic question-answer (QA) data generated by large language models (LLMs) can serve as an effective proxy for human-labeled benchmarks when the latter is unavailable. We assess the reliability of synthetic benchmarks across two experiments: one varying retriever parameters while keeping the generator fixed, and another varying the generator with fixed retriever parameters. Across four datasets, of which two open-domain and two proprietary, we find that synthetic benchmarks reliably rank the RAGs varying in terms of retriever configuration, aligning well with human-labeled benchmark baselines. However, they do not consistently produce reliable RAG rankings when comparing generator architectures. The breakdown possibly arises from a combination of task mismatch between the synthetic and human benchmarks, and stylistic bias favoring certain generators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。