构建医疗领域可追溯的摘要基准,提升医学摘要准确性与可信度。
TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain
- 为500篇医学摘要标注7个关键维度,生成3500组带句子级引用的摘要对。
- 提出细粒度评估框架,用4项指标衡量内容完整性和一致性。
- 验证先追踪后生成的流程能提升准确率和完整性,适合医疗应用。
尽管大语言模型已提升文本信息获取效率,但在医疗领域其摘要的真实性仍受关注。通过追溯摘要来源句子,可帮助用户评估其准确性。本文提出TracSum,一个面向可追溯、基于方面(aspect-based)的医学摘要新基准,生成的摘要附带句子级引用,支持溯源。首先,我们对500篇医学摘要标注了7个关键医学方面,共生成3500组摘要-引用对。随后,提出一种细粒度评估框架,采用4个指标评估生成内容的完整性和一致性。最后,设计了一种名为Track-Then-Sum的摘要流水线作为基线方法。在实验中,评估该基线及多个大语言模型在TracSum上的表现,并开展人工评估验证结果。结果表明,TracSum可有效支撑可追溯的方面摘要任务。同时发现,先进行句子级追踪再生成,能显著提升生成准确性;而引入完整上下文进一步增强摘要完整性。
原文摘要 · Abstract (English)
While document summarization with LLMs has enhanced access to textual information, concerns about the factual accuracy of these summaries persist, especially in the medical domain. Tracing evidence from which summaries are derived enables users to assess their accuracy, thereby alleviating this concern. In this paper, we introduce TracSum, a novel benchmark for traceable, aspect-based summarization, in which generated summaries are paired with sentence-level citations, enabling users to trace back to the original context. First, we annotate 500 medical abstracts for seven key medical aspects, yielding 3.5K summary-citation pairs. We then propose a fine-grained evaluation framework for this new task, designed to assess the completeness and consistency of generated content using four metrics. Finally, we introduce a summarization pipeline, Track-Then-Sum, which serves as a baseline method for comparison. In experiments, we evaluate both this baseline and a set of LLMs on TracSum, and conduct a human evaluation to assess the evaluation results. The findings demonstrate that TracSum can serve as an effective benchmark for traceable, aspect-based summarization tasks. We also observe that explicitly performing sentence-level tracking prior to summarization enhances generation accuracy, while incorporating the full context further improves completeness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。