用大模型零样本识别钓鱼网址并生成人类可读解释
LLMs are One-Shot URL Classifiers and Explainers
- 大模型结合思维链推理，零样本判断网址是否恶意
- 性能接近有监督模型，GPT-4-Turbo表现最佳
- 解释内容与传统模型一致，且清晰易懂
恶意网址分类是网络安全的关键环节。现有基于机器学习和深度学习的网址分类模型普遍面临泛化能力差、领域适应性弱的问题，主要源于缺乏代表性训练数据。此外，这些模型无法以自然语言提供分类解释。本文研究并验证了大语言模型（LLMs）在该任务中的应用。提出一种基于大模型的零样本学习框架，利用思维链（CoT）推理判断给定网址是否为良性或钓鱼。在三个网址数据集上，使用五种主流大模型进行评估，结果表明零样本提示下的大模型性能已接近有监督模型，其中 GPT-4-Turbo 表现最优，其次为 Claude 3 Opus。我们对大模型生成的解释进行了定量分析，发现其解释大多与有监督分类器的后验解释一致，且具备高可读性、连贯性和信息量。
原文摘要 · Abstract (English)
Malicious URL classification represents a crucial aspect of cyber security. Although existing work comprises numerous machine learning and deep learning-based URL classification models, most suffer from generalisation and domain-adaptation issues arising from the lack of representative training datasets. Furthermore, these models fail to provide explanations for a given URL classification in natural human language. In this work, we investigate and demonstrate the use of Large Language Models (LLMs) to address this issue. Specifically, we propose an LLM-based one-shot learning framework that uses Chain-of-Thought (CoT) reasoning to predict whether a given URL is benign or phishing. We evaluate our framework using three URL datasets and five state-of-the-art LLMs and show that one-shot LLM prompting indeed provides performances close to supervised models, with GPT 4-Turbo being the best model, followed by Claude 3 Opus. We conduct a quantitative analysis of the LLM explanations and show that most of the explanations provided by LLMs align with the post-hoc explanations of the supervised classifiers, and the explanations have high readability, coherency, and informativeness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。