评测大模型生成文本是否遗漏关键信息，发现简单方法反而更有效。
Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation
- 用大模型直接比对内容，检测缺失信息，无需复杂分解。
- 实测表明，最简单的端到端方法效果最好，但解释性较弱。
- 适合关注事实完整性、需评估多源问答质量的研究者使用。
尽管大型语言模型在诸多任务中表现优异，却常出现信息不完整或选择性遗漏关键内容的问题。在敏感领域，这种遗漏可能造成与事实错误相当的严重后果。本文聚焦于评估大模型生成文本的全面性，重点检测缺失信息或被弱化的观点。研究了三种自动化评估指标：(1) 基于自然语言推理（NLI）的方法，将文本拆分为原子语句并用NLI识别缺失事实；(2) 基于问答对的方法，提取问题-答案对并在不同来源间对比响应；(3) 端到端方法，直接利用大模型识别缺失内容。实验表明，看似简单的端到端方法效果出人意料地优秀，尽管其鲁棒性、可解释性和结果粒度有所下降。进一步评估了多个主流开源大模型在多源信息基础上回答用户查询时的全面性表现。
原文摘要 · Abstract (English)
Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce outputs that are incomplete or selectively omit key information. In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies, including hallucinations. In this study, we address the challenge of evaluating the comprehensiveness of LLM-generated texts, focusing on the detection of missing information or underrepresented viewpoints. We investigate three automated evaluation metrics: (1) an NLI-based method that decomposes texts into atomic statements and uses natural language inference (NLI) to identify missing facts, (2) a Q&A-based metric that extracts question-answer pairs and compares responses across sources, and (3) an end-to-end approach that directly identifies missing content using LLMs. Our experiments demonstrate the surprising effectiveness of the simple end-to-end metric compared to more complex metrics, though at the cost of reduced robustness, interpretability and result granularity. We further assess the comprehensiveness of responses from several popular open-weight LLMs when answering user queries based on multiple sources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。