构建10万条多语言事实核查解释数据集，提升社区内容可信度判断能力
COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations
- 构建包含104,000条帖子的多语言数据集，标注解释内容的有用性与原因
- 通过自动提示优化改进原因定义，提升有用性与原因预测准确率
- 为平台事实核查系统提供可复用的有用性评估能力，适合内容安全研究者
主流平台如X、Meta和TikTok正将事实核查从专家主导转向社区参与模式，用户通过添加解释性备注说明信息为何具有误导性。然而，如何判断这些解释是否真正有助于理解真实主张及其原因，仍是研究空白。实践中，多数社区备注因标注速度慢而未被发布，且有用性的标准尚不明确。为此，我们提出预测解释有用性及其原因的任务，并构建了包含10.4万条帖子的大型多语言数据集COMMUNITYNOTES，附带用户提供的备注与有用性标签。我们提出一种框架，通过自动提示优化生成并改进原因定义，并将其整合到预测模型中。实验表明，优化后的原因定义显著提升了有用性与原因预测效果。最后，我们验证了有用性信息对现有事实核查系统的实际增益。
原文摘要 · Abstract (English)
Fact-checking on major platforms, such as X, Meta, and TikTok, is shifting from expert-driven verification to a community-based setup, where users contribute explanatory notes to clarify why a post might be misleading. An important challenge here is determining whether an explanation is helpful for understanding real-world claims and the reasons why, which remains largely underexplored in prior research. In practice, most community notes remain unpublished due to slow community annotation, and the reasons for helpfulness lack clear definitions. To bridge these gaps, we introduce the task of predicting both the helpfulness of explanatory notes and the reason for this. We present COMMUNITYNOTES, a large-scale multilingual dataset of 104k posts with user-provided notes and helpfulness labels. We further propose a framework that automatically generates and improves reason definitions via automatic prompt optimization, and integrate them into prediction. Our experiments show that the optimized definitions can improve both helpfulness and reason prediction. Finally, we show that the helpfulness information is beneficial for existing fact-checking systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。