arXiv:2410.04087cs.CLcs.AI2024-10EMNLP被引 20

构建多语言新闻摘要统一基准,解决跨语言多文档冲突问题

GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization

  • 将多语言、跨语言、多文档摘要整合为新任务MCMS
  • 构建事件中心的GLOBESUMM数据集,含真实报道冲突与冗余
  • 适合多语言AI评估与大模型跨模态理解研究者使用

当今全球新闻环境面临多语言内容泛滥与多方观点并存的挑战,但现有研究多局限于单一语言或单文档任务。为此,本文提出统一多语言、跨语言和多文档摘要的新任务MCMS,以契合真实世界需求。由于缺乏基准数据集,研究进展受限。为此,我们精心构建了GLOBESUMM数据集,通过收集大量多语言新闻并重构为事件中心格式,同时采用协议引导提示法实现高质量低成本参考标注。在MCMS中,除冗余与遗漏外,特别强调报道间观点冲突带来的挑战,显著提升任务复杂性。通过广泛实验验证了数据集质量,并揭示任务内在难点。我们坚信,该基准将有力推动多语言社区发展与大模型评估。

原文摘要 · Abstract (English)

News summarization in today's global scene can be daunting with its flood of multilingual content and varied viewpoints from different sources. However, current studies often neglect such real-world scenarios as they tend to focus solely on either single-language or single-document tasks. To bridge this gap, we aim to unify Multi-lingual, Cross-lingual and Multi-document Summarization into a novel task, i.e., MCMS, which encapsulates the real-world requirements all-in-one. Nevertheless, the lack of a benchmark inhibits researchers from adequately studying this invaluable problem. To tackle this, we have meticulously constructed the GLOBESUMM dataset by first collecting a wealth of multilingual news reports and restructuring them into event-centric format. Additionally, we introduce the method of protocol-guided prompting for high-quality and cost-effective reference annotation. In MCMS, we also highlight the challenge of conflicts between news reports, in addition to the issues of redundancies and omissions, further enhancing the complexity of GLOBESUMM. Through extensive experimental analysis, we validate the quality of our dataset and elucidate the inherent challenges of the task. We firmly believe that GLOBESUMM, given its challenging nature, will greatly contribute to the multilingual communities and the evaluation of LLMs.

多语言新闻摘要大模型评估数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。