首个动态多模态事实核查基准,应对大模型训练导致的评测失效问题。
VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking
- 构建全自动七阶段流水线,持续更新真实世界多模态核查数据。
- 涵盖54种语言、2.5万条真实声明,覆盖104家专业机构的核查结果。
- 采用标准化评分体系,适合评估大模型时代的事实核查能力。
在线虚假信息规模不断扩大,亟需自动化事实核查(AFC)系统。然而,现有评测基准在任务范围、模态、领域、语言多样性、真实性或虚假信息类型覆盖方面均存在局限,且均为静态数据集,其内容可能已进入大模型预训练语料,导致评测性能失真。为此,我们提出首个动态多模态事实核查基准VeriTaS,以应对基础模型持续预训练带来的数据泄露问题。VeriTaS目前包含来自54种语言、104家专业机构的25,000条真实世界声明,涵盖文本与音视频内容。通过全自动七阶段流水线,实现声明规范化、原始媒体检索及异构专家结论到新型标准化、解耦评分体系的映射,并附带文本化依据。人工评估表明自动标注与人类判断高度一致。我们承诺持续更新,建立抗数据泄露的评测基准,支持大模型时代有意义的事实核查评估。代码与数据公开于 https://veritas.mai.informatik.tu-darmstadt.de。
原文摘要 · Abstract (English)
The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, however, are largely limited in terms of task scope, modalities, domain, language diversity, realism, or coverage of misinformation types. Critically, they are static, thus subject to data leakage as their claims enter the pretraining corpora of LLMs. As a result, benchmark performance no longer reliably reflects the actual ability to verify claims. We introduce Verified Theses and Statements (VeriTaS), the first dynamic benchmark for multimodal AFC, designed to remain robust under ongoing large-scale pretraining of foundation models. VeriTaS currently comprises 25,000 real-world claims from 104 professional fact-checking organizations across 54 languages, covering textual and audiovisual content. Claims are added quarterly via a fully automated seven-stage pipeline that normalizes claim formulation, retrieves original media, and maps heterogeneous expert verdicts to a novel, standardized, and disentangled scoring scheme with textual justifications. Through human evaluation, we demonstrate that the automated annotations closely match human judgments. We commit to updating VeriTaS in the future, establishing a leakage-resistant benchmark, supporting meaningful AFC evaluation in the era of rapidly evolving foundation models. The code and data are publicly available under https://veritas.mai.informatik.tu-darmstadt.de .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。