不同数据集上，LLM归因评估指标表现差异大，不能通用。
Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs
- 测试8种自动评分器在三类任务中的表现，发现指标间排名反转
- 短文本上表现好的模型在长文本上随机猜测，准确率从0.90降至0.53
- 需在目标数据集验证指标，不可直接借用其他数据集结果
实践中常将大语言模型检索增强生成的自动归因指标视为可互换。我们审计了八种评分器——词法、嵌入和BERTScore基线，以及基于蕴含/真实性训练的模型（clean和FEVER NLI，检查器MiniCheck）——在三个评估构型（出处/主题性、生成答案归因、事实核查蕴含）中的表现，检验是否存在可迁移的评分器：即在所有多数据集构型中，其表现始终落在最佳评分器95%置信区间内。在人类标注覆盖最广的构型——生成答案归因（AttributionBench四个源数据集，n=1,610，独立HAGRID，n=2,150）中，无一评分器满足条件：各数据集上的评分器排名发生反转（AttributedQA vs. LFQA，Kendall tau = -0.64，p = 0.031），一个在短句任务中表现最佳的NLI评分器（AUROC 0.90）在长文档LFQA上骤降至0.53（随机水平），而BERTScore则胜出（0.91）；这一反转并非长度或截断所致。这种不稳定性带来实际决策成本：简单采用‘平均最优’规则选择评估器，在留一数据集测试中平均损失达0.172 AUROC，甚至劣于固定单一评分器，表明指标选择必须在目标数据集上验证，而非从其他数据集学习。基于提示的LLM判官避免了自动评分器的随机崩溃（无LFQA崩溃），但成本高约100倍，且非确定性，仅转移而非消除验证负担。
原文摘要 · Abstract (English)
Practice often treats automatic metrics for attribution in LLM retrieval-augmented generation as interchangeable. We audit eight automatic scorers -- lexical, embedding, and BERTScore baselines alongside entailment/grounding-trained models (clean and FEVER NLI, the checker MiniCheck) -- across three evaluation constructs (provenance/topicality, generated-answer attribution, and fact-check entailment), asking whether any scorer transfers: stays within the 95% confidence interval of the best audited scorer on every dataset of a multi-dataset construct. In the construct with the most multi-dataset human-labeled coverage -- generated-answer attribution (AttributionBench's four source datasets, n = 1,610, with independent HAGRID, n = 2,150) -- none does: the per-dataset metric rankings invert (Kendall tau = -0.64, p = 0.031 on AttributedQA vs. LFQA), and an off-the-shelf NLI scorer that is best on short-claim AttributedQA (AUROC 0.90) collapses to AUROC 0.53 (chance) on long-form LFQA, where BERTScore wins (0.91); the flip is not a length or truncation artifact. This instability has a concrete decision cost: a naive "best-on-average" rule for choosing an evaluator fails leave-one-dataset-out (mean held-out regret 0.172 AUROC, worse than fixing one scorer), so metric choice must be validated on the target dataset rather than learned from others. A prompt-based LLM judge avoids the chance-level collapses the automatic scorers suffer (no LFQA collapse) but is not uniformly best, ~100x costlier, and non-deterministic -- relocating, not removing, the validation burden.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。