通过标注者可信度加权提升虚假信息检测准确率
Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media
- 基于标注者间与自身一致性评估可信度，动态加权样本
- 使用Llama-3.2-1B模型达0.757的宏平均F1
- 适合需高精度虚假信息识别的平台和研究者
虚假信息在社交媒体上快速传播，误导公众并影响脆弱群体。为有效缓解其负面影响，需先精准检测，再采取如X平台社区笔记等干预措施——当前仍依赖人工。本研究采用基于知识的虚假信息检测方法，将问题建模为自然语言推理任务。提出EffiARA标注框架，利用标注者间的共识与内部一致性评估其可靠性，并据此调整大模型训练中的样本权重。为此构建了公开的俄乌冲突知识型虚假信息分类数据集（RUC-MCD）。实验表明，结合标注者可靠性的样本加权策略表现最佳，融合了跨标注者与同标注者一致性及软标签训练。使用Llama-3.2-1B模型获得最高宏平均F1为0.757，而TwHIN-BERT-large达到0.740。
原文摘要 · Abstract (English)
Misinformation spreads rapidly on social media, confusing the truth and targeting potentially vulnerable people. To effectively mitigate the negative impact of misinformation, it must first be accurately detected before applying a mitigation strategy, such as X's community notes, which is currently a manual process. This study takes a knowledge-based approach to misinformation detection, modelling the problem similarly to one of natural language inference. The EffiARA annotation framework is introduced, aiming to utilise inter- and intra-annotator agreement to understand the reliability of each annotator and influence the training of large language models for classification based on annotator reliability. In assessing the EffiARA annotation framework, the Russo-Ukrainian Conflict Knowledge-Based Misinformation Classification Dataset (RUC-MCD) was developed and made publicly available. This study finds that sample weighting using annotator reliability performs the best, utilising both inter- and intra-annotator agreement and soft-label training. The highest classification performance achieved using Llama-3.2-1B was a macro-F1 of 0.757 and 0.740 using TwHIN-BERT-large.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。