用BART模型生成更像人写的摘要,发现现有评价指标会忽略事实错误。
Assessment of Transformer-Based Encoder-Decoder Model for Human-Like Summarization
- 基于BART的编码器-解码器模型实现类人摘要生成
- 人工评估显示模型摘要比人类标准低17%的事实一致性
- 传统指标如ROUGE无法检测事实性错误,需结合WeCheck等新方法
近年来,从大量文本中提取有价值信息取得了显著进展。在社交媒体时代,人们期望快速获取关键信息。自动文本摘要旨在通过压缩长文本生成更易处理的摘要,帮助决策者挖掘重要内容。随着深度学习的发展,编码器-解码器框架已成为自动摘要的核心方法。本研究采用基于Transformer的BART模型进行类人摘要生成,该任务具有开放性与诸多挑战。在训练与微调后,使用多样样本文章测试摘要质量,并基于人类评估参数进行评估。进一步对比了微调模型与基线预训练模型在ROUGE和BERTScore等指标上的表现。此外,为提升对话类摘要的抽象生成性能,需进行领域适配。研究发现,现有主流评价指标对事实性错误不敏感。通过WeCheck和SummaC等新型事实一致性评估工具分析微调模型生成的摘要,实证结果表明:在BBC新闻数据集上,人类撰写的黄金标准摘要在事实一致性上比模型生成摘要高出17%。
原文摘要 · Abstract (English)
In recent times, extracting valuable information from large text is making significant progress. Especially in the current era of social media, people expect quick bites of information. Automatic text summarization seeks to tackle this by slimming large texts down into more manageable summaries. This important research area can aid in decision-making by digging out salient content from large text. With the progress in deep learning models, significant work in language models has emerged. The encoder-decoder framework in deep learning has become the central approach for automatic text summarization. This work leverages transformer-based BART model for human-like summarization which is an open-ended problem with many challenges. On training and fine-tuning the encoder-decoder model, it is tested with diverse sample articles and the quality of summaries of diverse samples is assessed based on human evaluation parameters. Further, the finetuned model performance is compared with the baseline pretrained model based on evaluation metrics like ROUGE score and BERTScore. Additionally, domain adaptation of the model is required for improved performance of abstractive summarization of dialogues between interlocutors. On investigating, the above popular evaluation metrics are found to be insensitive to factual errors. Further investigation of the summaries generated by finetuned model is done using the contemporary evaluation metrics of factual consistency like WeCheck and SummaC. Empirical results on BBC News articles highlight that the gold standard summaries written by humans are more factually consistent by 17% than the abstractive summaries generated by finetuned model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。