为生产级RAG评估设计新指标与基准，解决传统方法不适用、结果不可复现的问题。
Practical RAG Evaluation: A Rarity-Aware Set-Based Metric and Cost-Latency-Quality Trade-offs
- 提出RA-nWG@K新指标，考虑稀有性与查询归一化，更符合LLM实际使用场景
- 构建可复现的黄金数据集流水线，支持迭代优化与端到端性能评估
- 揭示嵌入模型对专有名词和对话噪声的敏感度，指导工程调优
本文针对生产级RAG构建中的评估困境：传统以排序为中心的指标（nDCG/MAP/MRR）不适用于RAG，因LLM处理的是候选段落集合而非有序列表；位置折扣和忽略出现频率的聚合方式，无法反映截止到第K个片段时提示中是否包含决定性证据。同时，缺乏标准化、可复现的黄金数据集构建与审计方法，且现有排行榜未涵盖端到端的、真实语料上的成本-延迟-质量权衡。此外，先进嵌入模型如何处理专有名词身份信号及对话噪声仍不透明。为此，本文贡献：(1) RA-nWG@K——一种稀有性感知、每查询归一化的集合评分指标，结合池受限的最优解（PROC）与PROC占比（%PROC），在成本-延迟-质量（CLQ）框架下分离检索与排序的提升空间；(2) rag-gs（MIT）——轻量级黄金数据集生成流水线，采用Plackett-Luce列表级精炼，迭代更新优于单次大模型排序；(3) 在科学论文语料上的全面基准测试，覆盖密集检索、混合稠密+BM25、不同嵌入模型与维度、交叉编码器重排序、ANN（HNSW）及量化方案；(4) 通过身份破坏与格式消融实验，定量分析专有名词信号保留与对话噪声敏感性。上述组件共同为从业者提供帕累托优化指导与可审计的决策护栏，支持预算与SLA约束下的可靠选择。
原文摘要 · Abstract (English)
This paper addresses the guessing game in building production RAG. Classical rank-centric IR metrics (nDCG/MAP/MRR) are a poor fit for RAG, where LLMs consume a set of passages rather than a browsed list; position discounts and prevalence-blind aggregation miss what matters: whether the prompt at cutoff K contains the decisive evidence. Second, there is no standardized, reproducible way to build and audit golden sets. Third, leaderboards exist but lack end-to-end, on-corpus benchmarking that reflects production trade-offs. Fourth, how state-of-the-art embedding models handle proper-name identity signals and conversational noise remains opaque. To address these, we contribute: (1) RA-nWG@K, a rarity-aware, per-query-normalized set score, and operational ceilings via the pool-restricted oracle ceiling (PROC) and the percentage of PROC (%PROC) to separate retrieval from ordering headroom within a Cost-Latency-Quality (CLQ) lens; (2) rag-gs (MIT), a lean golden-set pipeline with Plackett-Luce listwise refinement whose iterative updates outperform single-shot LLM ranking; (3) a comprehensive benchmark on a production RAG (scientific-papers corpus) spanning dense retrieval, hybrid dense+BM25, embedding models and dimensions, cross-encoder rerankers, ANN (HNSW), and quantization; and (4) targeted diagnostics that quantify proper-name identity signal and conversational-noise sensitivity via identity-destroying and formatting ablations. Together, these components provide practitioner Pareto guidance and auditable guardrails to support reproducible, budget/SLA-aware decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。