提出兼顾精确与覆盖的生成评估新方法,解决只测对错不看全的缺陷。
Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle
- 用完备事实库同时测生成内容的精确率和覆盖率
- 小模型微调后精确率与覆盖率均超0.98,超越大模型零样本表现
- 适合评估需要全面性的对话、决策类生成任务
无参考的忠实度指标仅衡量生成内容中每个断言是否被真值支持,即精确率,却忽略覆盖率——即是否涵盖相关事实。这导致模型可通过少说来获得高分。本文利用公式1遥测数据(含7,253个决策实例,覆盖157场赛事)这一具有确定性且完全的事实源,首次可精确测量召回率(覆盖度)。结果显示,当前最精准的前沿模型覆盖不足一半相关事实,F1排名垫底;在另一完全事实域(NOAA天气预报)中同样出现此现象。对1B-7B规模的小模型在该完备事实上微调,可使精确率与召回率均达约0.98,超越所有零样本大模型。本文提出融合精确与覆盖的统一评分,并通过受控扰动、不同提取器一致性验证其有效性(系统级斯皮尔曼相关系数1.0)。此外提供基于验证器引导的生成方法,无需参考即可同时提升两者。数据集、标注、基准、工具及交互演示已开源。
原文摘要 · Abstract (English)
Reference-free faithfulness metrics verify each atomic claim a model makes against ground truth, and are increasingly used to evaluate grounded generation. We show they share a blind spot: they measure only precision -- are the stated claims supported? -- and therefore reward abstention, since a model can score near-perfect faithfulness by saying almost nothing. We make this measurable using Formula 1 telemetry, a domain where strategic ground truth is derived deterministically and, crucially, completely: for each decision we know the full set of facts that mattered. This completeness -- absent in open-domain faithfulness benchmarks -- lets us measure recall (coverage of the relevant facts) exactly, alongside precision. On a multilingual (EN/ES/PT) benchmark of 7,253 decision instances spanning 157 races, the most precise frontier model covers under half of the relevant facts and ranks last by F1, so requiring coverage reorders the systems; the same effect reappears in a second complete-oracle domain (NOAA weather forecasts). Fine-tuning small models (1B-7B) on the complete oracle closes the precision-recall gap entirely (F1 ~0.98), beating every zero-shot frontier system regardless of scale. We pair faithfulness with coverage into a single score, validate the metric (controlled perturbation; agreement across a model-free regex extractor and a cross-family LLM extractor, system-level Spearman 1.0), and give a verifier-guided generation method that improves precision and recall without references. We release the benchmark, structured annotations, metric, baselines, and an interactive demo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。