研究发现参考摘要集变化会显著影响评估结果，建议纳入变异性以提升可靠性。
References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation
- 分析三种数据集上参考集变化对评估指标的影响
- 发现ROUGE等指标在不同参考集下排名波动大，结果不可靠
- 建议评估时考虑参考集多样性，尤其适用于LLM生成文本
人类语言表达具有丰富的多样性和差异性，反映不同的交流风格与意图。然而，当前摘要评估常忽视这种变异。尽管多参考摘要已知可提升与人工判断的相关性，但参考集选择对基于参考的评估指标的影响尚未系统研究。本文分析了三种多参考摘要数据集（SummEval、GUMSum、DUC2004）中参考集变化对主流评估指标的敏感性，发现多数常用指标存在显著不稳定性。这一问题在基于n-gram的指标（如ROUGE）中尤为突出，模型排名随参考集变化而波动，削弱了模型比较的可信度。此外，我们收集了跨文体生成内容的人工判断，发现其与现有指标相关性弱至无相关性，补充了仅基于新闻摘要的研究。综合来看，建议在摘要评估中引入参考集变异，以增强一致性并提高与人工判断的相关性，尤其是在评估大语言模型时。
原文摘要 · Abstract (English)
Human language production exhibits remarkable richness and variation, reflecting diverse communication styles and intents. However, this variation is often overlooked in summarization evaluation. While having multiple reference summaries is known to improve correlation with human judgments, the impact of the reference set on reference-based metrics has not been systematically investigated. This work examines the sensitivity of widely used reference-based metrics in relation to the choice of reference sets, analyzing three diverse multi-reference summarization datasets: SummEval, GUMSum, and DUC2004. We demonstrate that many popular metrics exhibit significant instability. This instability is particularly concerning for n-gram-based metrics like ROUGE, where model rankings vary depending on the reference sets, undermining the reliability of model comparisons. We also collect human judgments on LLM outputs for genre-diverse data and examine their correlation with metrics to supplement existing findings beyond newswire summaries, finding weak-to-no correlation. Taken together, we recommend incorporating reference set variation into summarization evaluation to enhance consistency alongside correlation with human judgments, especially when evaluating LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。