提出新方法评估摘要事实一致性，无需训练即可超越现有主流指标。
Using Similarity to Evaluate Factual Consistency in Summaries
- 用句子级相似度比对摘要与原文，改进传统词级比较
- 在基准测试中表现优于BERTScore等主流指标
- 适合需要零样本评估的摘要质量分析场景
当前先进的抽象式摘要模型生成的文本流畅，但事实性无法保证。早期基于n-gram重叠和嵌入相似性的评估指标与人工标注不一致。因此，许多检测事实错误的方法依赖自然语言推理（NLI）或问答（QA）模型，并引入额外的监督学习步骤。本文重新审视基于相似度的指标，指出其失败源于对比文本的选择及其粒度。我们提出一种新的零样本事实一致性评估指标——句向量相似度得分（SBERTScore），通过比较摘要与源文档中的句子进行匹配。该方法在基准测试上表现优于广泛使用的词-词指标（如BERTScore），且可与现有的NLI和QA基事实一致性指标相媲美，无需任何微调。实验表明，不同技术各有优势，其中SBERTScore在识别正确摘要方面尤为有效。我们还证明，多种技术组合能更有效地检测各类错误。
原文摘要 · Abstract (English)
Cutting-edge abstractive summarisers generate fluent summaries, but the factuality of the generated text is not guaranteed. Early summary factuality evaluation metrics are usually based on n-gram overlap and embedding similarity, but are reported fail to align with human annotations. Therefore, many techniques for detecting factual inconsistencies build pipelines around natural language inference (NLI) or question-answering (QA) models with additional supervised learning steps. In this paper, we revisit similarity-based metrics, showing that this failure stems from the comparison text selection and its granularity. We propose a new zero-shot factuality evaluation metric, Sentence-BERT Score (SBERTScore), which compares sentences between the summary and the source document. It outperforms widely-used word-word metrics including BERTScore and can compete with existing NLI and QA-based factuality metrics on the benchmark without needing any fine-tuning. Our experiments indicate that each technique has different strengths, with SBERTScore particularly effective in identifying correct summaries. We demonstrate how a combination of techniques is more effective in detecting various types of error.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。