微调模型比零样本大模型更擅长识别社交媒体中的错误信息传播
Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit
- 用微调的RoBERTa模型分类,准确率比最强零样本模型高12%
- 零样本模型普遍低估情感化、隐含立场的传播类评论
- 适合需要精准识别错误信息传播场景的研究者和安全团队
随着大语言模型成为在线信息验证的主流工具,一个隐含假设是:模型规模和通用能力足以应对错误信息话语的细微分类。我们在此对900条来自Reddit的评论进行测试,涵盖三个经PolitiFact验证的虚假信息话题(环境、健康、移民),标签分为信念(传播主张)、事实核查(纠正)或其他。比较了九种模型在通用与主题特定标签体系下的表现:BART-MNLI、三种Llama变体、三种商业前沿大模型(Claude Haiku 4.5、Gemini Flash Lite 2.5、Claude Sonnet 4.6),以及微调的DistilBERT和RoBERTa。结果表明该假设不成立:微调的RoBERTa达到0.62的宏平均F1,优于最佳零样本模型(Claude Haiku 4.5,0.50),且查询成本极低;监督优势集中在信念类别,所有零样本模型均显著低估该类。模型规模未带来提升:Llama-3-8B与Llama-3-70B表现相当,而更大的Claude Sonnet 4.6在通用标签下反而表现更差,信念检测下降至0.17,并对部分敏感评论拒绝响应。这反映的是安全对齐效应而非能力限制。标签体系与话题共同影响零样本性能,同一模型在不同话题间宏平均F1差异超过0.13。在信息验证中,漏检信念类是代价更高的错误,因此即使大模型普及,任务特定微调仍是更可靠的选择。
原文摘要 · Abstract (English)
As large language models (LLMs) become default tools for online information verification, an implicit assumption follows them: that scale and general capability are sufficient for nuanced classification of misinformation discourse. We test this assumption directly on 900 Reddit comments spanning three PolitiFact-verified misinformation claims (environment, health, immigration), labelled as belief (propagates the claim), fact-check (corrects it), or other. We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs (Claude Haiku 4.5, Gemini Flash Lite 2.5, Claude Sonnet 4.6), and fine-tuned DistilBERT and RoBERTa -- under universal and topic-specific label schemas. The assumption does not hold. Fine-tuned RoBERTa reaches 0.62 macro-$F_1$ against a best zero-shot result of 0.50 (Claude Haiku 4.5), at a fraction of the per-query cost; the supervised advantage is concentrated on the belief class, the implicit, affective category every zero-shot model under-detects. Scaling does not help: Llama-3-8B matches Llama-3-70B, and Claude Sonnet 4.6 underperforms the smaller Haiku under generic labels, collapsing belief detection to 0.17 and refusing outright on a subset of comments flagged as sensitive. This is a safety-alignment artefact, not a capacity limit. Label schema and topic jointly shape zero-shot performance, with the same model varying by more than 0.13 macro-$F_1$ across topics under matched labels. In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice despite the proliferation of large generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。