测试发现，泄露评估细节会让RAG系统虚假得分接近完美。
Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
- 用评估标准反向优化输出，让系统假装表现优异
- 在提示模板或黄金答案泄露时，得分接近100%
- 适合关注评估公平性的AI研发者和评测人员
当前RAG系统普遍采用大模型评判（LLM judges）进行评估与优化，基于关键信息片段（nugget-based）的方法不仅用于评估框架，还被嵌入系统架构中。然而这种集成可能引发循环评估风险。本文通过对比Ginger、Crucible等基于nugget的RAG系统与GPT-Researcher等强基线，在故意修改Crucible以适应LLM评判标准的实验中发现：当评估细节如提示模板或黄金答案可预测时，系统能获得近乎完美的评分。结果表明，必须采用盲评设置和多样化的评估方法，才能避免将对评估指标的过拟合误判为真实性能提升。
原文摘要 · Abstract (English)
RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in particular are now embedded not only in evaluation frameworks but also in the architectures of RAG systems themselves. While this integration can lead to genuine improvements, it also creates a risk of faulty measurements due to circularity. In this paper, we investigate this risk through comparative experiments with nugget-based RAG systems, including Ginger and Crucible, against strong baselines such as GPT-Researcher. By deliberately modifying Crucible to generate outputs optimized for an LLM judge, we show that near-perfect evaluation scores can be achieved when elements of the evaluation - such as prompt templates or gold nuggets - are leaked or can be predicted. Our results highlight the importance of blind evaluation settings and methodological diversity to guard against mistaking metric overfitting for genuine system progress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。