扩充对话数据集让Transformer模型在诈骗识别中超越传统方法。
An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection
- 用改进生成流程构建1万+条新对话数据,解决原有数据污染与标签错误问题。
- 长短期记忆模型(Longformer)在新数据上准确率达79.71%,超过XGBoost的78.43%。
- 适合关注对话式诈骗检测、模型训练数据质量的研究者和从业者。
此前工作提出COVA数据集,包含3201条标注对话,建立多轮诈骗对话检测基准。尽管基于TF-IDF的XGBoost表现最佳(准确率72.5%,宏平均F1 0.691),但Transformer模型表现不佳,归因于输入截断与训练数据不足。本文提出扩展版COVA-X,涵盖10,985条对话,覆盖八类针对老年人的诈骗场景,通过改进生成流程解决了数据污染、标签错配、阶段信息泄露及提示设计缺陷问题。在新数据上重新训练所有分类器后发现:Longformer在各项指标上均超越XGBoost,达到79.71%准确率与0.7786宏平均F1,而XGBoost为78.43%与0.7563。结果证实Transformer需更大对话语料才能发挥上下文优势。此外,数据清洗使标签修正率提升12.7倍(从49.8%降至3.9%),架构优化将虚拟绑架类伪影率从67.1%降至46.5%;按诈骗类型分析显示结果具机制一致性。前后对比敏感性分析表明,数据精炼有效恢复了各类模型的标签相关信号。
原文摘要 · Abstract (English)
Our prior work introduced COVA, a synthetically generated multi-turn conversational smishing dataset of 3,201 labeled conversations, establishing baseline detection benchmarks across eight models. While XGBoost with TF-IDF features achieved the best performance, with 72.5\% accuracy and 0.691 macro F1, transformer models underperformed, which was attributed to input truncation and insufficient training data. We present COVA-X, an expanded dataset of 10,985 conversations spanning eight elder-targeted scam categories, produced by an improved generation pipeline addressing contamination, label mismatch, stage-direction bleed, and prompt-design failures from the first iteration. Retraining all classifiers on the expanded dataset yields the central finding of this work: Longformer now surpasses XGBoost on all evaluation metrics, achieving 79.71\% accuracy and 0.7786 macro F1 compared with 78.43\% and 0.7563 for XGBoost. This directly confirms that transformer models require larger conversational corpora to realize their contextual advantages. We additionally document a quality life-cycle including a 12.7$\times$ improvement in label correction rate, from 49.8\% to 3.9\%, an architectural intervention reducing virtual-kidnapping artifact rates from 67.1\% to 46.5\%, and a per-scam-type outcome analysis showing that scam categories modulate results in mechanism-consistent ways. A pre/post-cleanup sensitivity analysis confirms that dataset refinement recovers genuine label-relevant signal across all three classifier architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。