复现对话摘要模型，揭示实验差异并提升评估可靠性
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
- 复现多种对话摘要模型，基于AMI数据集对比效果
- 人类评估显示摘要质量存在中等程度波动（标准差0.656）
- 适合关注NLP可复现性与摘要评估方法的研究者
自然语言处理领域的科学可复现性对验证实验结果的稳健性至关重要。本文系统复现并评估了对话摘要模型，重点分析原始研究与本研究复现结果之间的差异。对话摘要旨在将对话内容浓缩为简洁且信息丰富的摘要，以支持高效的信息检索与决策。研究使用AMI（Augmented Multi-party Interaction）数据集，评估了分层记忆网络（HMNet）及多种指针-生成网络（PGN）变体，包括PGN(DKE)、PGN(DRD)、PGN(DTS)和PGN(DALL)。通过人工评估方式衡量摘要的丰富性与质量，该方法引入主观性与评估波动。初步分析显示，数据集1中样本标准偏差为0.656，表明评分存在中等程度离散。
原文摘要 · Abstract (English)
Reproducibility in scientific research, particularly within the realm of natural language processing (NLP), is essential for validating and verifying the robustness of experimental findings. This paper delves into the reproduction and evaluation of dialogue summarization models, focusing specifically on the discrepancies observed between original studies and our reproduction efforts. Dialogue summarization is a critical aspect of NLP, aiming to condense conversational content into concise and informative summaries, thus aiding in efficient information retrieval and decision-making processes. Our research involved a thorough examination of several dialogue summarization models using the AMI (Augmented Multi-party Interaction) dataset. The models assessed include Hierarchical Memory Networks (HMNet) and various versions of Pointer-Generator Networks (PGN), namely PGN(DKE), PGN(DRD), PGN(DTS), and PGN(DALL). The primary objective was to evaluate the informativeness and quality of the summaries generated by these models through human assessment, a method that introduces subjectivity and variability in the evaluation process. The analysis began with Dataset 1, where the sample standard deviation of 0.656 indicated a moderate dispersion of data points around the mean.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。