arXiv:2409.09249cs.CL2024-09被引 6

NovAScore自动评估文档新颖性,比人类判断更准。

NovAScore: A New Automated Metric for Evaluating Document Level Novelty

  • 基于原子信息的新颖性与重要性得分,动态加权聚合
  • 在TAP-DLND数据集上与人工判断相关性达0.626
  • 适合需要高效评估新闻、报告等文本新颖性的场景

在线内容的快速膨胀加剧了信息冗余问题,亟需能识别真正新信息的解决方案。然而,随着大语言模型的兴起,学术界对新颖性检测的关注度下降,且以往方法严重依赖人工标注,耗时费力,尤其当需对比海量历史文档时更为困难。本文提出NovAScore(原子性新颖性评估分数),一种用于评估文档级新颖性的自动化指标。该方法通过聚合原子信息的新颖性与显著性得分,实现高可解释性,并提供对文档新颖性的细致分析。其动态权重调整机制增强了灵活性,额外提供了评估信息新颖程度与重要性的维度。实验表明,NovAScore与人工判断高度相关,在TAP-DLND 1.0数据集上达到0.626的点二列相关系数,在内部人工标注数据集上达到0.920的皮尔逊相关系数。

原文摘要 · Abstract (English)

The rapid expansion of online content has intensified the issue of information redundancy, underscoring the need for solutions that can identify genuinely new information. Despite this challenge, the research community has seen a decline in focus on novelty detection, particularly with the rise of large language models (LLMs). Additionally, previous approaches have relied heavily on human annotation, which is time-consuming, costly, and particularly challenging when annotators must compare a target document against a vast number of historical documents. In this work, we introduce NovAScore (Novelty Evaluation in Atomicity Score), an automated metric for evaluating document-level novelty. NovAScore aggregates the novelty and salience scores of atomic information, providing high interpretability and a detailed analysis of a document's novelty. With its dynamic weight adjustment scheme, NovAScore offers enhanced flexibility and an additional dimension to assess both the novelty level and the importance of information within a document. Our experiments show that NovAScore strongly correlates with human judgments of novelty, achieving a 0.626 Point-Biserial correlation on the TAP-DLND 1.0 dataset and a 0.920 Pearson correlation on an internal human-annotated dataset.

新颖性评估自动化指标信息冗余文本分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。