arXiv:2507.18343cs.CL2025-07被引 7

用大模型预标注+人工校验,提升谣言检测效率与一致性

Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence

  • 构建三级分类体系,用大模型预标注传播手法并生成解释
  • 人工验证显示标注一致率显著提升,耗时减少40%以上
  • 小模型通过知识蒸馏学习大模型生成的高质量标注数据

社交媒体中的谣言检测因任务复杂且高质量标注数据稀缺而困难。本文提出一种融合人类专家与大语言模型(LLM)的新框架,提升标注一致性和可扩展性。构建包含14种细粒度传播手法的分层分类体系,基于HQP数据集的人工标注研究发现细粒度标签间一致性较低;设计基于LLM的预标注流程,自动提取传播片段、生成简明解释,并分配局部与全局标签。二次人工验证表明,该方法在标注一致性与效率上均有显著提升。进一步地,通过知识蒸馏,使用高质量的LLM生成数据微调小型语言模型(SLMs),实现结构化标注。本工作推动可扩展、鲁棒的谣言检测系统发展,支持透明、负责任的媒体生态建设,符合可持续发展目标16。代码已开源。

原文摘要 · Abstract (English)

Propaganda detection on social media remains challenging due to task complexity and limited high-quality labeled data. This paper introduces a novel framework that combines human expertise with Large Language Model (LLM) assistance to improve both annotation consistency and scalability. We propose a hierarchical taxonomy that organizes 14 fine-grained propaganda techniques into three broader categories, conduct a human annotation study on the HQP dataset that reveals low inter-annotator agreement for fine-grained labels, and implement an LLM-assisted pre-annotation pipeline that extracts propagandistic spans, generates concise explanations, and assigns local labels as well as a global label. A secondary human verification study shows significant improvements in both agreement and time-efficiency. Building on this, we fine-tune smaller language models (SLMs) to perform structured annotation. Instead of fine-tuning on human annotations, we train on high-quality LLM-generated data, allowing a large model to produce these annotations and a smaller model to learn to generate them via knowledge distillation. Our work contributes towards the development of scalable and robust propaganda detection systems, supporting the idea of transparent and accountable media ecosystems in line with SDG 16. The code is publicly available at our GitHub repository.

谣言检测大模型应用人机协作知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。