通过检索上下文提升可验证性判断准确率
ContextClaim: A Context-Driven Paradigm for Verifiable Claim Detection
- 结合维基百科检索实体信息,生成上下文摘要辅助判断
- 在多个数据集上显著提升可验证性检测效果,最高增益达6.3%
- 适合需要高精度事实核查的系统集成与研究者使用
自动化事实核查流程通常从过滤阶段开始,判断哪些陈述值得验证,因为后续的证据检索与验证成本高昂。核心任务是可验证性判断,即判断一个陈述是否原则上可由外部证据验证。以往工作仅依赖声明本身进行判断,但我们认为这限制了效果,因为判断可验证性往往需识别提及的实体和事件,并确认外部信息是否存在。受下游验证系统依赖检索证据的启发,我们将检索环节提前至检测阶段,提出ContextClaim。给定一条声明,该方法识别实体,查询维基百科作为结构化背景源,并利用大语言模型将检索内容压缩为简短上下文摘要,再交由分类器处理。实验在两个领域和语料上进行:CheckThat! 2022 Twitter 数据集和政治辩论语料库PoliClaim,涵盖编码器与解码器模型,在微调、零样本和少样本设置下评估。引入上下文后,多种配置下的可验证性检测性能均获提升,增幅随数据集、模型架构和训练设置变化。进一步发现,相同检索摘要对下游验证也有帮助:将其输入FEVER验证模型,使验证F1提升。组件分析、人工标注和错误检查揭示了上下文有效性的边界条件。
原文摘要 · Abstract (English)
Automated fact-checking pipelines typically begin with a filtering stage that decides which claims are worth verifying, given that the later evidence retrieval and verification components are expensive to apply at scale. A central task in this stage is verifiable claim detection, which asks whether a statement is in principle checkable against external evidence. Prior work on this task, as well as on the closely related notion of check-worthiness, conditions its decisions only on the claim sentence itself. We argue that this is restrictive, because deciding whether a statement is checkable often depends on identifying the entities and events it mentions, and on whether external information about them is actually available in the first place. Motivated by how downstream verification systems rely on retrieved evidence, we move retrieval upstream into the detection stage and introduce ContextClaim. Given an input claim, the approach identifies entity mentions, queries Wikipedia as a structured background source, and uses large language models to compress the retrieved material into short contextual summaries that are then passed to a classifier. Experiments are conducted on two domains and genres, namely the CheckThat! 2022 Twitter collection and the PoliClaim corpus of political debates, and cover both encoder and decoder only models under fine-tuning, zero-shot, and few-shot settings. The added context yields gains on verifiable claim detection in several configurations, although the size of the improvement varies with the dataset, the backbone model, and the training setup. We further find that the same retrieved summaries are useful beyond detection. Feeding them into a downstream verification model on FEVER improves verification F1. Component level analyses, human annotation, and error inspection further clarify the conditions under which retrieved context helps, and where it does not.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。