arXiv:2412.15308cs.CLcs.IR2024-12中稿 · AAAI被引 7

首个越南语多领域新闻真伪核查数据集,助力低资源语言信息可信度提升。

ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese

  • 构建首个公开越南语多领域事实核查数据集,含7232条人工标注的论点-证据对。
  • Gemma模型在该数据集上取得89.90%宏平均F1分数,表现最佳。
  • 适合研究低资源语言事实核查、数字媒体可信度与AI辅助审核的学者与开发者。

数字时代信息传播迅速,对有效事实核查工具的需求日益迫切,尤其针对资源有限的语言如越南语。为此,我们推出ViFactCheck,首个专为越南语多领域在线新闻事实核查设计的公开基准数据集。该数据集包含7,232条来自权威越南在线新闻的人工标注论点-证据对,覆盖12个不同主题,经严格标注流程确保质量,达成0.83的Fleiss Kappa一致性评分。我们利用先进的预训练与大语言模型,通过微调和提示技术评估性能。显著的是,Gemma模型表现出色,宏平均F1得分为89.90%,确立了新的基准水平。该结果凸显了Gemma在准确识别越南语事实方面的强大能力。为推动事实核查技术进步并提升数字媒体可靠性,我们已将ViFactCheck数据集、模型检查点、核查流水线及源代码免费发布于GitHub。这一举措旨在激发更多研究,提升低资源语言的信息准确性。

原文摘要 · Abstract (English)

The rapid spread of information in the digital age highlights the critical need for effective fact-checking tools, particularly for languages with limited resources, such as Vietnamese. In response to this challenge, we introduce ViFactCheck, the first publicly available benchmark dataset designed specifically for Vietnamese fact-checking across multiple online news domains. This dataset contains 7,232 human-annotated pairs of claim-evidence combinations sourced from reputable Vietnamese online news, covering 12 diverse topics. It has been subjected to a meticulous annotation process to ensure high quality and reliability, achieving a Fleiss Kappa inter-annotator agreement score of 0.83. Our evaluation leverages state-of-the-art pre-trained and large language models, employing fine-tuning and prompting techniques to assess performance. Notably, the Gemma model demonstrated superior effectiveness, with an impressive macro F1 score of 89.90%, thereby establishing a new standard for fact-checking benchmarks. This result highlights the robust capabilities of Gemma in accurately identifying and verifying facts in Vietnamese. To further promote advances in fact-checking technology and improve the reliability of digital media, we have made the ViFactCheck dataset, model checkpoints, fact-checking pipelines, and source code freely available on GitHub. This initiative aims to inspire further research and enhance the accuracy of information in low-resource languages.

事实核查越南语多领域低资源语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。