提出无需标注的评估方法，精准衡量检索文本对大模型生成的帮助程度。
Evaluating the Utility of Grounding Documents with Reference-Free LLM-based Metrics
- 基于大模型生成置信度（熵）定义内容实用性，适配具体模型特性。
- 在无标注条件下仍能准确区分真实文档，性能比通用指标提升18.2点MRR。
- 可用于筛选高质量重写数据，提升RAG系统问答准确率9.4点。
检索增强生成（RAG）的成功依赖于大模型从用于定位的文本中获取的实际帮助。目前尚无明确的内容效用量化标准，现有度量方法或忽略模型特性，或依赖昂贵的人工标注。本文提出一种模型专属、无需参考的评估指标GroGU，将效用定义为下游大模型生成结果置信度（基于熵）的函数。尽管无需人工标注，GroGU仍能有效区分真实文档，捕捉到传统无模型指标忽略的细微差异。我们利用GroGU训练查询重写器，通过识别高实用性偏好数据进行直接偏好优化。实验表明，该方法在平均倒数排名（MRR）上最高提升18.2点，在答案准确率上最高提升9.4点。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG)'s success depends on the utility the LLM derives from the content used for grounding. Quantifying content utility does not have a definitive specification and existing metrics ignore model-specific capabilities and/or rely on costly annotations. In this paper, we propose Grounding Generation Utility (GroGU), a model-specific and reference-free metric that defines utility as a function of the downstream LLM's generation confidence based on entropy. Despite having no annotation requirements, GroGU is largely faithful in distinguishing ground-truth documents while capturing nuances ignored by LLM-agnostic metrics. We apply GroGU to train a query-rewriter for RAG by identifying high-utility preference data for Direct Preference Optimization. Experiments show improvements by up to 18.2 points in Mean Reciprocal Rank and up to 9.4 points in answer accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。