评测大模型在漏洞检测、解释和修复上的综合能力，发现当前最佳模型准确率仅54.7%。
VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation
- 构建涵盖174个真实漏洞的多维度人类评估基准，覆盖识别、分类、解释与修复全过程。
- 六款主流大模型平均准确率49%-54.7%，修复质量与漏洞分类准确性高度相关（r>0.97）。
- 公开数据集、评分标准与可视化工具，助力安全领域大模型研究可复现、可改进。
确保大语言模型（LLMs）能有效评估、检测、解释并修复软件漏洞，对构建健壮安全的软件系统至关重要。我们提出VADER，一个专为评估LLM在漏洞处理四大核心维度——评估、检测、解释与修复——表现而设计的人类评估基准。VADER包含174个从GitHub仓库中精心筛选的真实世界漏洞，均由安全专家标注。每个漏洞案例要求模型识别缺陷，使用通用弱点枚举（CWE）进行分类，解释根本原因，提出补丁代码，并制定测试方案。我们采用单次提示策略，对六款前沿大模型（Claude 3.7 Sonnet、Gemini 2.5 Pro、GPT-4.1、GPT-4.5、Grok 3 Beta、o3）进行评测，由人类安全专家依据严格评分量表打分，其中修复质量占50%，解释占20%，分类与测试计划占30%。结果显示，当前最先进模型在VADER上仅达中等水平：OpenAI的o3总体准确率为54.7%，其余模型在49%-54%之间，仍有巨大提升空间。值得注意的是，修复质量与准确分类及测试计划间存在强相关性（皮尔逊相关系数r > 0.97），表明良好分类能力有助于高质量修复。VADER的完整数据集、详细评分体系、评分工具及带置信区间的结果可视化均已公开，为社区提供可解释、可复现的基准，推动具备漏洞感知能力的大模型发展。所有代码与数据详见：https://github.com/AfterQuery/vader
原文摘要 · Abstract (English)
Ensuring that large language models (LLMs) can effectively assess, detect, explain, and remediate software vulnerabilities is critical for building robust and secure software systems. We introduce VADER, a human-evaluated benchmark designed explicitly to assess LLM performance across four key vulnerability-handling dimensions: assessment, detection, explanation, and remediation. VADER comprises 174 real-world software vulnerabilities, each carefully curated from GitHub repositories and annotated by security experts. For each vulnerability case, models are tasked with identifying the flaw, classifying it using Common Weakness Enumeration (CWE), explaining its underlying cause, proposing a patch, and formulating a test plan. Using a one-shot prompting strategy, we benchmark six state-of-the-art LLMs (Claude 3.7 Sonnet, Gemini 2.5 Pro, GPT-4.1, GPT-4.5, Grok 3 Beta, and o3) on VADER, and human security experts evaluated each response according to a rigorous scoring rubric emphasizing remediation (quality of the code fix, 50%), explanation (20%), and classification and test plan (30%) according to a standardized rubric. Our results show that current state-of-the-art LLMs achieve only moderate success on VADER - OpenAI's o3 attained 54.7% accuracy overall, with others in the 49-54% range, indicating ample room for improvement. Notably, remediation quality is strongly correlated (Pearson r > 0.97) with accurate classification and test plans, suggesting that models that effectively categorize vulnerabilities also tend to fix them well. VADER's comprehensive dataset, detailed evaluation rubrics, scoring tools, and visualized results with confidence intervals are publicly released, providing the community with an interpretable, reproducible benchmark to advance vulnerability-aware LLMs. All code and data are available at: https://github.com/AfterQuery/vader
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。