构建多模型多角色疫苗伪信息数据集,助力识别AI生成的假消息
VaxGuard: A Multi-Generator, Multi-Type, and Multi-Role Dataset for Detecting LLM-Generated Vaccine Misinformation
- 设计VaxGuard数据集,涵盖多类大模型生成的疫苗伪信息
- GPT-3.5和GPT-4o在检测情感化内容上表现最优,长文本检测能力下降
- 揭示角色差异对检测效果的影响,适合安全与舆情研究者使用
大型语言模型(LLMs)在文本生成方面取得显著进展,但也带来疫苗相关伪信息生成的风险。现有研究多关注人工撰写的内容,缺乏对LLM生成伪信息的系统性分析及检测方法。本文提出VaxGuard,一个面向多模型、多类型、多角色的疫苗伪信息数据集,旨在填补这一空白。实验表明,GPT-3.5与GPT-4o在检测复杂或情绪化叙事时优于其他模型;而PHI3与Mistral在恐惧驱动语境中精度与召回率均较低。同时,随着输入文本长度增加,检测性能普遍下降,凸显对长文本处理机制的改进需求。结果强调角色特异性检测策略的重要性,表明VaxGuard可作为提升LLM生成疫苗伪信息检测能力的关键资源。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have significantly improved text generation capabilities. However, they also present challenges, particularly in generating vaccine-related misinformation, which poses risks to public health. Despite research on human-authored misinformation, a notable gap remains in understanding how LLMs contribute to vaccine misinformation and how best to detect it. Existing benchmarks often overlook vaccine-specific misinformation and the diverse roles of misinformation spreaders. This paper introduces VaxGuard, a novel dataset designed to address these challenges. VaxGuard includes vaccine-related misinformation generated by multiple LLMs and provides a comprehensive framework for detecting misinformation across various roles. Our findings show that GPT-3.5 and GPT-4o consistently outperform other LLMs in detecting misinformation, especially when dealing with subtle or emotionally charged narratives. On the other hand, PHI3 and Mistral show lower performance, struggling with precision and recall in fear-driven contexts. Additionally, detection performance tends to decline as input text length increases, indicating the need for improved methods to handle larger content. These results highlight the importance of role-specific detection strategies and suggest that VaxGuard can serve as a key resource for improving the detection of LLM-generated vaccine misinformation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。