arXiv:2607.18429cs.CRcs.AI2026-07

对比两种钓鱼邮件检测方法在对抗攻击下的表现,发现准确率高不等于抗干扰强。

Adversarial Robustness of Phishing Email Detection: A Comparative Study of TF-IDF + Logistic Regression and Fine-Tuned DistilBERT

  • 用TF-IDF+逻辑回归和微调的DistilBERT模型对比检测钓鱼邮件
  • 对抗测试下两者准确率均降至63.6%~64.0%,下降超35个百分点
  • 模型错误不重复,说明防御策略需互补,不能只看干净数据表现

钓鱼邮件仍是持续存在的网络安全威胁,机器学习分类器被广泛用于检测。然而,多数报告的检测准确率基于干净、分布内测试数据,而非刻意篡改以逃避检测的邮件。本文在三种条件下对两种检测方法进行了受控对比:标准分布内、合成钓鱼邮件和对抗钓鱼邮件。模型使用来自六个公开数据集的82,255封邮件统一训练。两模型在干净数据上均超过98%准确率,但在对抗测试下显著下降:TF-IDF + LR降至64.00%(下降34.59个百分点),DistilBERT降至63.64%(下降35.40个百分点),差距仅0.36个百分点(相当于275样本测试集中1封邮件)。LIME、SHAP与注意力可视化分析显示,两模型依赖不同证据,但表现出相似脆弱性。成对错误分析表明,模型在54.9%的对抗样本上一致出错,各自独立错误数分别为24和25,说明失败模式部分互补。结果表明,干净数据准确率无法预测对抗鲁棒性,对抗测试应成为钓鱼邮件检测评估的标准环节。

原文摘要 · Abstract (English)

Phishing emails remain one of the most persistent cybersecurity threats, and machine-learning classifiers are widely used to detect them. Most reported detection accuracies, however, are measured on clean, in-distribution test data rather than on emails deliberately altered to evade detection. This paper reports a controlled, pairwise comparison of two phishing-detection approaches a TF-IDF + Logistic Regression baseline and a fine-tuned DistilBERT transformer trained on a unified corpus of 82,255 emails drawn from six public datasets and evaluated under three conditions: normal in-distribution, synthetic phishing, and adversarial phishing. Both models exceeded 98% accuracy on clean data yet degraded sharply under adversarial testing: TF-IDF + LR fell to 64.00% (a 34.59-percentage-point drop) and DistilBERT fell to 63.64% (a 35.40-percentage-point drop) a gap of only 0.36 percentage points, equivalent to a single email in the 275-sample adversarial test set. LIME, SHAP, and attention-rollout analysis indicate the two models relied on different evidence yet showed similar vulnerability. Pairwise error analysis shows the models agreed on 54.9% of adversarial samples but each made a similar number of exclusive errors (24 and 25 respectively), indicating partly complementary rather than identical failure modes. The results show that clean-data accuracy does not predict adversarial robustness, and that adversarial testing should be a standard part of phishing-detection evaluation.

钓鱼检测对抗攻击模型鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。