构建首个事实核查的声明抽取数据集与评估框架。
Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics
- 提出基于一对多生成的声明抽取方法，对比大模型与小模型表现。
- 释放包含1.7万条原子事实声明的FEVERFact数据集，覆盖4000个上下文句子。
- 设计六项可量化评估指标，逼近人工评分，适合研究事实核查系统者参考。
本文探讨利用一到多文本生成方法进行声明抽取的问题，比较大语言模型、微调的小型摘要模型以及先前以命名实体识别为中心的基线模型QACG。由于当前关于声明抽取、事实提取、声明生成和值得核查的声明检测的研究在方法和术语上分散不清，我们整合其共同目标，发布新的FEVERFact数据集，包含从4000个带上下文的维基百科句子中提取出的1.7万条原子事实声明，源自原始FEVER数据集。我们构建了评估框架，分别针对每个生成声明评估原子性、流畅性、去上下文化、忠实度，以及针对整组预测声明评估焦点与覆盖率。每项指标均通过转化为已有的自然语言处理任务来实现量化。通过与人类对通用声明的评分对比验证，模型在最困难的$F_{fact}$指标上的排名未发生变化，且整体框架在$F_1$和RMSE上与人工评分高度一致。
原文摘要 · Abstract (English)
In this paper, we explore the problem of Claim Extraction using one-to-many text generation methods, comparing LLMs, small summarization models finetuned for the task, and a previous NER-centric baseline QACG. As the current publications on Claim Extraction, Fact Extraction, Claim Generation and Check-worthy Claim Detection are quite scattered in their means and terminology, we compile their common objectives, releasing the FEVERFact dataset, with 17K atomic factual claims extracted from 4K contextualised Wikipedia sentences, adapted from the original FEVER. We compile the known objectives into an Evaluation framework of: Atomicity, Fluency, Decontextualization, Faithfulness checked for each generated claim separately, and Focus and Coverage measured against the full set of predicted claims for a single input. For each metric, we implement a scale using a reduction to an already-explored NLP task. We validate our metrics against human grading of generic claims, to see that the model ranking on $F_{fact}$, our hardest metric, did not change and the evaluation framework approximates human grading very closely in terms of $F_1$ and RMSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。