用大模型批量评估检索结果，新系统加入时只需评新增文档，节省97%成本。
Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection
- 大模型统一评估所有候选系统返回的文档池，增量扩展时只评新增部分。
- 在4个基准上与人工标注相关性高，97%的系统排序结果保持一致。
- 金融问答系统中实现65%-80%评价复用，成本降低4.9倍，适合持续迭代的生产系统。
为生产级RAG系统选择检索模型需要可靠的对比评估，但大规模获取相关性标注成本高且难以重复。本文研究了联合式大模型评估（pooled LLM evaluation），即由大模型对当前所有候选系统返回文档的并集进行打分，并在新系统引入时仅对新增文档重新评分，旧评分可复用于所有系统评估。我们在包含11种不同配置（密集、稀疏、混合）的4个检索基准上验证该方法，并在金融新闻问答系统中部署以比较62种检索配置。结果显示，联合评估排名与人工标准高度相关，考虑查询相关性置信度后，97%的系统成对排序保持一致。实际应用中，文档重叠带来65%-80%的评分复用率，评估成本最高降低4.9倍，使团队可在不重评旧文档的前提下持续评估新候选模型。这表明联合式大模型评估是生产环境中高效、可持续的检索模型迭代方案。
原文摘要 · Abstract (English)
Selecting a retrieval model for a production RAG system requires reliable comparative evaluation, but obtaining relevance judgments at scale is expensive and difficult to repeat as new candidate systems arrive. We study pooled LLM evaluation, in which an LLM judges the union of documents retrieved by the current set of candidate systems, and the pool is then expanded incrementally as new systems are introduced by judging only the new documents they contribute. These judgments are reused to evaluate all systems on a common basis. We validate this approach on four retrieval benchmarks with 11 systems spanning dense, sparse, and hybrid configurations, and deploy it to compare 62 retrieval configurations for a financial news QA system. Pooled LLM rankings correlate strongly with gold-standard evaluation across datasets, and 97% of pairwise system orderings are preserved once bootstrap uncertainty in the qrels is taken into account. In production, document overlap yields 65-80% judgment reuse and up to 4.9x lower evaluation cost, allowing teams to benchmark new retrieval candidates without re-judging previously assessed documents. These results suggest pooled LLM evaluation is a practical and cost-effective workflow for incremental retrieval model selection in deployed systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。