arXiv:2501.14737cs.SEcs.AI2025-01被引 8

用多智能体模拟专家评审,自动评估软件漏洞。

EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment

  • 多大模型协作,自主讨论漏洞细节。
  • 在三种语言上准确率超44%,F1达43.29%。
  • 适合需要解释性漏洞评估的开发者和安全团队。

软件漏洞(SV)评估是确定漏洞攻击路径、影响范围等关键信息的重要过程,有助于开发人员有效优先修复。由于漏洞复杂且标注数据稀缺,该过程极具挑战性。为此,我们提出EvalSVA,一种多智能体评估系统,可自主模拟真实场景中的漏洞评估策略。该框架将多个大型语言模型(LLMs)整合为协作团队,在数据有限的情况下提升评估效果,并设计多样通信机制以自主讨论漏洞各维度。此外,我们基于新版CVSS标准构建了多语言漏洞评估数据集,涵盖C++、Python和Java的699、888和1,310个相关提交。实验表明,EvalSVA平均准确率达44.12%,F1值为43.29%,优于以往方法。系统不仅能生成答案,还能提供类似人类的推理过程,增强可解释性,辅助人工专家进行更细致的评估。

原文摘要 · Abstract (English)

Software Vulnerability (SV) assessment is a crucial process of determining different aspects of SVs (e.g., attack vectors and scope) for developers to effectively prioritize efforts in vulnerability mitigation. It presents a challenging and laborious process due to the complexity of SVs and the scarcity of labeled data. To mitigate the above challenges, we introduce EvalSVA, a multi-agent evaluators team to autonomously deliberate and evaluate various aspects of SV assessment. Specifically, we propose a multi-agent-based framework to simulate vulnerability assessment strategies in real-world scenarios, which employs multiple Large Language Models (LLMs) into an integrated group to enhance the effectiveness of SV assessment in the limited data. We also design diverse communication strategies to autonomously discuss and assess different aspects of SV. Furthermore, we construct a multi-lingual SV assessment dataset based on the new standard of CVSS, comprising 699, 888, and 1,310 vulnerability-related commits in C++, Python, and Java, respectively. Our experimental results demonstrate that EvalSVA averagely outperforms the 44.12\% accuracy and 43.29\% F1 for SV assessment compared with the previous methods. It shows that EvalSVA offers a human-like process and generates both reason and answer for SV assessment. EvalSVA can also aid human experts in SV assessment, which provides more explanation and details for SV assessment.

漏洞评估多智能体LLM应用代码安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。