构建首个跨领域新闻论断提取数据集与模型，支持离线部署且准确率接近GPT-4。
NewsScope: Schema-Grounded Cross-Domain News Claim Extraction with Open Models
- 基于LLaMA 3.1 8B微调，用结构化模式提取跨领域新闻论断
- 在政治类论断上准确率达94.3%，整体人类评估准确率89.4%
- 开源模型可低成本部署，适合需要隐私保护的验证场景
自动化新闻验证需结构化论断提取，但现有方法或不遵循模式，或跨领域泛化差。本文提出NewsScope，包含跨政治、健康、科技/环境、商业四领域的数据集、基准测试及微调模型。数据集共455篇文章，含395篇域内与60篇域外测试文章。使用LoRA对LLaMA 3.1 8B进行微调，训练集315例，测试集为80篇域内和60篇域外文章。400个论断的人工评估显示，NewsScope准确率达89.4%，略低于GPT-4o-mini的93.7%（p=0.07）。在政治类论断上表现更优（94.3% vs. 87.8%）。引入数值锚定过滤器后准确率提升至91.6%，差距缩小至2.1个百分点。160个论断的标注者一致性研究显示，支持性判断的正向一致性达94.6%。模型开源，可实现约15美元按需计算成本（或免费版零成本）部署。代码与基准已公开。
原文摘要 · Abstract (English)
Automated news verification requires structured claim extraction, but existing approaches either lack schema compliance or generalize poorly across domains. This paper presents NewsScope, a cross-domain dataset, benchmark, and fine-tuned model for schema-grounded news claim extraction. The dataset contains 455 articles across politics, health, science/environment, and business, consisting of 395 in-domain articles and 60 out-of-source articles for generalization testing. LLaMA 3.1 8B was fine-tuned using LoRA on 315 training examples and evaluated on held-out in-domain (80 articles) and out-of-source (60 articles) test sets. Human evaluation on 400 claims shows NewsScope achieves 89.4% human-evaluated accuracy compared to GPT-4o-mini's 93.7% (p=0.07). NewsScope outperforms GPT-4o-mini on political claims (94.3% vs. 87.8%). A numeric grounding filter further improves accuracy to 91.6%, narrowing the gap to 2.1 percentage points. Inter-annotator agreement studies (160 claims) confirm labeling reliability (94.6% positive agreement on SUPPORTED judgments). The open-weight model enables offline deployment at approximately $15 on-demand compute (or $0 on free tiers). Code and benchmark are publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。