arXiv:2512.18533cs.CLcs.LG2025-12

政治假新闻检测的泛化能力有限,简单模型表现优于复杂模型。

Generalization Gaps in Political Fake News Detection: An Empirical Study on the LIAR Dataset

  • 对比九种模型,发现词汇和语义特征均有性能上限
  • 细粒度分类加权F1最高仅0.32,训练准确率与测试准确率差距超75%
  • 复杂模型不比线性SVM强,说明瓶颈在语义理解而非模型容量

政治虚假信息的语言微妙性给自动化事实核查带来挑战。尽管神经网络架构日益复杂,但纯文本建模的实证极限仍不清晰。我们在LIAR基准上系统评估了九种机器学习算法。通过分离词汇特征(词袋、TF-IDF)和语义嵌入(GloVe),发现细粒度分类的性能存在硬上限,加权F1得分不超过0.32。关键的是,简单线性SVM(准确率0.624)表现与预训练Transformer如RoBERTa(准确率0.620)相当,表明模型容量并非主要瓶颈。树集成模型在训练集上准确率超99%,测试集骤降至约25%,显示其依赖词汇记忆而非语义推理。通过SMOTE进行合成数据增强未带来显著提升,证实限制源于语义特征模糊性而非分布差异。结果表明,在政治事实核查中,若不引入外部知识,单纯增加模型复杂度将导致回报递减。

原文摘要 · Abstract (English)

The proliferation of linguistically subtle political disinformation poses a significant challenge to automated fact-checking systems. Despite increasing emphasis on complex neural architectures, the empirical limits of text-only linguistic modeling remain underexplored. We present a systematic diagnostic evaluation of nine machine learning algorithms on the LIAR benchmark. By isolating lexical features (Bag-of-Words, TF-IDF) and semantic embeddings (GloVe), we uncover a hard "Performance Ceiling", with fine-grained classification not exceeding a Weighted F1-score of 0.32 across models. Crucially, a simple linear SVM (Accuracy: 0.624) matches the performance of pre-trained Transformers such as RoBERTa (Accuracy: 0.620), suggesting that model capacity is not the primary bottleneck. We further diagnose a massive "Generalization Gap" in tree-based ensembles, which achieve more than 99% training accuracy but collapse to approximately 25% on test data, indicating reliance on lexical memorization rather than semantic inference. Synthetic data augmentation via SMOTE yields no meaningful gains, confirming that the limitation is semantic (feature ambiguity) rather than distributional. These findings indicate that for political fact-checking, increasing model complexity without incorporating external knowledge yields diminishing returns.

假新闻检测泛化能力语言模型性能天花板

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。