不生成摘要就能预测总结质量,提升自动摘要效率。
PreSumm: Predicting Summarization Performance Without Summarizing
- 基于原文直接预测摘要效果,无需实际生成摘要。
- 低分文档常存在连贯性差、内容复杂或主题不清问题。
- 可用于优化人工介入流程和清理数据集噪声。
尽管自动摘要技术取得进展,但现有模型对不同文档的总结效果差异显著,引发关键问题:为何如此?以往研究多关注模型本身,却忽视文档特征对摘要质量的影响。本文探索两个核心问题:文档在多个系统中是否表现出一致的摘要质量?能否仅凭原文预测摘要表现?我们肯定回答,并提出预摘要(PreSumm)新任务——仅根据源文档预测摘要性能。分析发现,低分文档普遍存在连贯性差、内容复杂或缺乏明确主题等问题。此外,我们验证了PreSumm在两类应用中的价值:通过识别需人工处理的文档改进混合摘要流程;通过过滤异常值和噪声文档提升数据集质量。结果表明,文档特性在摘要表现中起关键作用,揭示了当前系统局限性,为未来改进提供依据。
原文摘要 · Abstract (English)
Despite recent advancements in automatic summarization, state-of-the-art models do not summarize all documents equally well, raising the question: why? While prior research has extensively analyzed summarization models, little attention has been given to the role of document characteristics in influencing summarization performance. In this work, we explore two key research questions. First, do documents exhibit consistent summarization quality across multiple systems? If so, can we predict a document's summarization performance without generating a summary? We answer both questions affirmatively and introduce PreSumm, a novel task in which a system predicts summarization performance based solely on the source document. Our analysis sheds light on common properties of documents with low PreSumm scores, revealing that they often suffer from coherence issues, complex content, or a lack of a clear main theme. In addition, we demonstrate PreSumm's practical utility in two key applications: improving hybrid summarization workflows by identifying documents that require manual summarization and enhancing dataset quality by filtering outliers and noisy documents. Overall, our findings highlight the critical role of document properties in summarization performance and offer insights into the limitations of current systems that could serve as the basis for future improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。