跨数据集检测虚假新闻和讽刺时，文本长度陷阱会导致模型失效。
When Benchmarks Mislead: Shortcut Learning, Length Confounds, and the Limits of Cross-Dataset Generalization in Multilingual Fake News and Sarcasm Detection
- 通过分析文本长度差异，发现数据集间分布偏差导致模型误判。
- 在乌尔都语虚假新闻任务中，长文本误导使跨数据集准确率暴跌至0.005。
- 提出可复用的诊断方法，适用于多种二分类文本任务。
跨数据集泛化是部署文本分类器于真实场景的基本要求，但在虚假新闻检测中系统性跨源评估仍不常见，讽刺检测研究则几乎空白。本文对三种领域（乌尔都语虚假新闻、英语虚假新闻、讽刺检测）开展统一实证研究，使用xlm-roberta-base在某一数据集上微调并在另一来源数据集上评估，对比TF-IDF+LR与SVM基线。在乌尔都语虚假新闻（Ax-to-Grind vs. Notri-Fact）中，发现Ax-to-Grind数据集中假新闻平均字数为真新闻的3.4倍，导致A→B迁移性能崩溃（宏平均F1=0.005），而反向迁移达F1=0.771。扩展至英语虚假新闻（WELFake vs. ISOT）与讽刺检测（TweetEval Irony vs. Sarcasm Corpus V2）也证实此类失败模式跨语言、跨领域普遍存在，表明对分布伪象的捷径学习是二分类文本识别中的普遍挑战。论文提供一套可复用的诊断方法，包含条件长度分析、双向迁移不对称性及预测标签坍塌检验，适用于任意二分类文本任务。
原文摘要 · Abstract (English)
Cross-dataset generalisation is a fundamental requirement for deploying text classifiers in real-world settings, yet systematic evaluation across corpora from different sources remains uncommon in fake news detection and virtually absent in sarcasm detection research. This paper presents a unified empirical study of zero-shot cross-dataset transfer in three domains: Urdu fake news detection (FND), English FND, and sarcasm detection. For each domain, we fine-tune xlm-roberta-base on one corpus and evaluate it on a second corpus from a different source, comparing against TF-IDF baselines with Logistic Regression (LR) and Support Vector Machines (SVM). In Urdu FND (Ax-to-Grind vs. Notri-Fact), we identify a severe length confound in the Ax-to-Grind dataset, fake articles average 3.4 times more words than real articles, causing catastrophic A to B transfer collapse (macro F1 = 0.005) while B to A achieves F1 = 0.771. Extension to English FND (WELFake vs. ISOT) and sarcasm detection (TweetEval Irony vs. Sarcasm Corpus V2) reveals that such failure modes extend beyond Urdu, confirming that shortcut learning from distributional artefacts is a cross-lingual, cross-domain challenge in binary text classification. We provide a reusable diagnostic methodology, combining class-conditional length analysis, bidirectional transfer asymmetry, and predicted label collapse inspection, applicable across any binary text classification setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。