首个针对Python函数级漏洞预测的分层评估基准，区分漏洞类型精准反馈。
Adaptive Hierarchical Evaluation of LLMs and SAST tools for CWE Prediction in Python
- 提出分层惩罚机制，区分过泛、过度具体和横向错误，提升诊断价值。
- 7个LLM与2个SAST工具对比显示，LLM整体表现更优但一致性波动大（8.26%-81.87%）。
- 适用于希望改进代码安全检测与反馈系统的开发者与研究者。
大型语言模型在软件开发中日益重要，却常生成存在漏洞的代码。现有漏洞检测基准采用二分类，缺乏细粒度的通用缺陷类别（CWE）信息，无法为迭代修正系统提供有效反馈。本文提出ALPHA（基于层级评估的自适应学习），首个面向Python函数级的评估基准，同时评估LLM与SAST工具，引入分层感知的CWE特定惩罚机制。ALPHA可区分过泛化、过度具体及横向错误，反映实际诊断差异。对7个LLM和2个SAST工具的评估表明，LLM整体表现显著优于SAST，但当检测发生时，SAST精度更高。关键发现：不同模型间预测一致性差异巨大（8.26%-81.87%），对反馈驱动系统有重大影响。文章进一步提出将ALPHA惩罚机制纳入监督微调的路径，有望实现原则性层次感知的漏洞检测，待实证验证。
原文摘要 · Abstract (English)
Large Language Models have become integral to software development, yet they frequently generate vulnerable code. Existing code vulnerability detection benchmarks employ binary classification, lacking the CWE-level specificity required for actionable feedback in iterative correction systems. We present ALPHA (Adaptive Learning via Penalty in Hierarchical Assessment), the first function-level Python benchmark that evaluates both LLMs and SAST tools using hierarchically aware, CWE-specific penalties. ALPHA distinguishes between over-generalisation, over-specification, and lateral errors, reflecting practical differences in diagnostic utility. Evaluating seven LLMs and two SAST tools, we find LLMs substantially outperform SAST, though SAST demonstrates higher precision when detections occur. Critically, prediction consistency varies dramatically across models (8.26%-81.87% agreement), with significant implications for feedback-driven systems. We further outline a pathway for future work incorporating ALPHA penalties into supervised fine-tuning, which could provide principled hierarchy-aware vulnerability detection pending empirical validation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。