用认知模拟提升大模型识骗能力，让防诈系统更懂人心。
SCRIPTMIND: Crime Script Inference and Cognitive Evaluation for LLM-based Social Engineering Scam Detection System
- 构建犯罪脚本推理任务与数据集，指导小模型学骗术逻辑。
- 微调后模型在识别准确率上比GPT-4o高13%，误报率更低。
- 能真实提升用户警惕性，适合反诈系统与安全教育场景。
社交工程诈骗日益采用个性化、多轮欺骗策略，传统检测方法已难应对。尽管大语言模型（LLMs）在识别欺骗方面展现潜力，但其认知辅助价值尚未被充分挖掘。本文提出ScriptMind框架，整合自动化推理与人类认知评估，包含三项核心组件：犯罪脚本推理任务（CSIT）、面向小模型微调的犯罪脚本感知数据集（CSID），以及基于认知模拟的社会工程防御评估（CSED）。基于571个韩国电话诈骗案例，构建了22,712个结构化诈骗者行为序列训练样本。实验表明，使用ScriptMind微调的110亿参数小模型在检测准确率上超越GPT-4o达13%，在误报控制、诈骗者话术预测和推理合理性方面均优于商用模型。此外，在电话诈骗模拟测试中，该模型显著提升了用户的怀疑水平并维持了认知警觉性。ScriptMind为实现以人为中心、具备认知适应性的防诈大模型迈出了关键一步。
原文摘要 · Abstract (English)
Social engineering scams increasingly employ personalized, multi-turn deception, exposing the limits of traditional detection methods. While Large Language Models (LLMs) show promise in identifying deception, their cognitive assistance potential remains underexplored. We propose ScriptMind, an integrated framework for LLM-based scam detection that bridges automated reasoning and human cognition. It comprises three components: the Crime Script Inference Task (CSIT) for scam reasoning, the Crime Script-Aware Inference Dataset (CSID) for fine-tuning small LLMs, and the Cognitive Simulation-based Evaluation of Social Engineering Defense (CSED) for assessing real-time cognitive impact. Using 571 Korean phone scam cases, we built 22,712 structured scammer-sequence training instances. Experimental results show that the 11B small LLM fine-tuned with ScriptMind outperformed GPT-4o by 13%, achieving superior performance over commercial models in detection accuracy, false-positive reduction, scammer utterance prediction, and rationale quality. Moreover, in phone scam simulation experiments, it significantly enhanced and sustained users' suspicion levels, improving their cognitive awareness of scams. ScriptMind represents a step toward human-centered, cognitively adaptive LLMs for scam defense.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。