arXiv:2412.14306cs.SEcs.CR2024-12中稿 · ICSE 2025 research…

实测AI漏洞检测工具在真实开发中误报多、修复不适用,仍难落地。

Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE

  • 集成主流模型的IDE插件,自动扫描漏洞并生成修复建议。
  • 17名开发者测试24个真实项目,发现170个告警但50%修复建议无效。
  • 用户反馈缺乏上下文理解与定制能力,需提升可信度和工作流融合。

本研究首次针对专业开发者在真实项目上使用漏洞检测与修复工具进行实证分析。我们基于前沿检测与修复模型开发了集成于IDE的DeepVulGuard工具,可识别漏洞类型与代码位置,提出修复方案,并通过自然语言解释和聊天界面提供说明。招募17名微软开发者,在其真实代码上使用该工具,观察其使用情况并访谈评估工具的实用性、速度、信任度、相关性及工作流融合度。共扫描24个项目、6.9千个文件、超过170万行源码,生成170个告警和50个修复建议。结果表明,尽管现有AI工具在历史数据集上表现良好,但在实际应用中因误报率高、修复建议不适用,尚不具备实用性。用户反馈指出关键痛点,如上下文信息不足、无法适配个人代码库。同时探讨了置信度分数、解释说明和聊天交互等AI功能在漏洞处理中的作用。基于这些发现,提出评估与部署AI检测修复模型的实际建议。代码与数据已公开于https://doi.org/10.6084/m9.figshare.26367139。

原文摘要 · Abstract (English)

This paper presents the first empirical study of a vulnerability detection and fix tool with professional software developers on real projects that they own. We implemented DeepVulGuard, an IDE-integrated tool based on state-of-the-art detection and fix models, and show that it has promising performance on benchmarks of historic vulnerability data. DeepVulGuard scans code for vulnerabilities (including identifying the vulnerability type and vulnerable region of code), suggests fixes, provides natural-language explanations for alerts and fixes, leveraging chat interfaces. We recruited 17 professional software developers at Microsoft, observed their usage of the tool on their code, and conducted interviews to assess the tool's usefulness, speed, trust, relevance, and workflow integration. We also gathered detailed qualitative feedback on users' perceptions and their desired features. Study participants scanned a total of 24 projects, 6.9k files, and over 1.7 million lines of source code, and generated 170 alerts and 50 fix suggestions. We find that although state-of-the-art AI-powered detection and fix tools show promise, they are not yet practical for real-world use due to a high rate of false positives and non-applicable fixes. User feedback reveals several actionable pain points, ranging from incomplete context to lack of customization for the user's codebase. Additionally, we explore how AI features, including confidence scores, explanations, and chat interaction, can apply to vulnerability detection and fixing. Based on these insights, we offer practical recommendations for evaluating and deploying AI detection and fix models. Our code and data are available at https://doi.org/10.6084/m9.figshare.26367139.

漏洞检测AI开发实证研究IDE工具

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。