提出新指标衡量摘要抽象程度，区分提取与生成式摘要。
Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
- 用调和平均与非重叠因子设计抽象度量化方法
- 实验显示生成模型抽象度达0.96-1.77，提取模型仅0.12-0.26
- 可识别高幻觉风险摘要，适合模型评估与人工审核
量化生成摘要的抽象程度对于超越ROUGE等表面指标评估摘要模型至关重要。本文提出参考抽象（RA）、摘要抽象（SA）和抽象比率（AR）——一组基于启发式的严谨度量，用于衡量摘要与源文本逐字复制之间的偏离程度。该公式采用文档长度的调和平均，并通过三次非重叠因子调节，实现量纲一致、取值有界，且对提取-生成边界具有非线性敏感性。在100个XSUM文档上对四种模型（BART-large-cnn、Pegasus-xsum、DistilBart、MT5-small）的评估表明，该指标能有效区分提取型模型（SA ~ 0.12-0.26）与生成型模型（SA ~ 0.96-1.77），且抽象比率可识别需人工评估以检测潜在幻觉的摘要。代码与结果已公开于https://github.com/katweNLP/AbstractionStudy。
原文摘要 · Abstract (English)
Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。