提出新方法提升诈骗检测模型在未知攻击场景下的可靠性
Evidence-Consistent Generative Detection under Scenario-Level Distribution Shift

- 用证据一致性训练框架,避免模型依赖特定场景线索
- 在未知攻击场景下,宏平均F1提升3.22点,推理理由与标签不一致率下降4.22点
- 适合需要高鲁棒性的反欺诈系统研发者使用
传统分布内评估在训练与测试数据共享特定任务模式时会过度估计模型鲁棒性。这一风险在社交工程欺诈检测中尤为突出,攻击者可改变场景、冒用身份或措辞,但保持恶意意图不变。本文研究短信与语音钓鱼场景下的场景级分布外(SL-OOD)检测问题,即在训练时完全排除某些完整攻击场景,而标签空间保持不变。该设定检验模型是否能基于决策相关证据而非熟悉场景线索实现泛化。实验发现,高分布内性能无法可靠预测分布外鲁棒性,各类基于特征、编码器和解码器的基线模型均存在此差距,归因于场景记忆:过度依赖特定场景的词汇或实体线索。为此,提出ECoG框架,结合证据跨度监督与理由-标签一致性目标进行训练。在0.5B解码器上,相较未使用一致性正则化的同架构模型,ECoG使分布外挑战样本的宏平均F1提升3.22点,生成理由支持相反标签的比例降低4.22点,与参考证据跨度的词粒度重叠度提升8.38点;该不一致率下降效果在四个解码器主干网络中保持一致。结果表明,在社交工程分布偏移下,紧凑的生成式检测器可通过证据监督与理由-标签一致性获益。
原文摘要 · Abstract (English)
Conventional in-distribution evaluation can overestimate robustness when training and test data share recurring task-specific patterns or surface cues. This risk is especially relevant in social-engineering fraud detection, where attackers can preserve malicious intent while changing the scenario, impersonated entity, or wording. We study this problem as scenario-level out-of-distribution (SL-OOD) detection for SMS and voice phishing, where entire attack scenarios are held out from training while the label space remains fixed. This setting tests whether models can generalize to unseen attack scenarios using decision-relevant evidence rather than familiar scenario-specific cues. Using this SL-OOD evaluation, we find that high in-distribution performance does not reliably predict held-out robustness across feature-, encoder-, and decoder-based baselines. We interpret this gap as scenario memorization: reliance on recurring scenario-specific lexical or entity cues rather than decision-relevant evidence. We propose ECoG, an evidence-consistent generative framework that combines evidence-span supervision with a rationale-label consistency objective during training. On the 0.5B decoder, relative to the same backbone trained without consistency regularization, ECoG raises Macro-F1 on OOD challenging instances by 3.22 points, reduces the share of predictions whose generated rationale supports the opposite label by 4.22 points, and increases token-level overlap with reference evidence spans by 8.38 points; the reduction in prediction-rationale inconsistency is consistent across four decoder backbones. These results suggest that compact generative detectors can benefit from evidence supervision and rationale-label consistency under social-engineering shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。