测试LLM辅助修复漏洞的实际效果，发现可能提速但易引入安全漏洞。
Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study
- 用交叉实验设计对比人类手动与LLM辅助修复
- LLM修复虽快但常通过功能测试却通不过安全验证
- 通过隐藏测试用例检测修复质量，适合安全研究者参考
软件漏洞修复是需要专业安全知识的高认知负荷任务，普通开发者往往缺乏相关能力。大语言模型（LLMs）在漏洞检测、定位和修复方面展现出潜力。本文假设：尽管LLM能加速修复过程，但也可能引入幻觉或不安全代码，导致生成表面修复——即通过功能测试但无法通过安全验证。为验证该假设，我们开展一项控制实验，采用平衡交叉设计，开发用于代码执行的WebApp，并集成隐藏的幽灵测试（Ghost Tests）以检验修复完整性。实验涵盖训练与评估阶段，评估修复速度、标准功能测试与安全测试的修复成效，以及参与者的主观感受。前期小样本试点实验已提供关键设计依据。
原文摘要 · Abstract (English)
Software vulnerability remediation is a cognitively demanding task that requires specialized security expertise often lacking in general developers. In the meantime, Large Language Models (LLMs) assisted tools show potential in vulnerability detection, location, and repair tasks. [Hypothesis:] While LLM-assistance is hypothesized to accelerate patching, it also risks introducing hallucinations or insecure code, leading to a higher likelihood of generating superficial repairs that bypass the standard functionality checks but fail the security validation. [Objective:] We aim to present an empirical experiment, unveiling the capability of LLM-assisted vulnerability patching compared to manual debugging on human participants in real-world scenarios. [Method:] We plan to conduct a controlled experiment using a Balanced Crossover design. For that, we have developed a WebApp for code execution and integrated hidden Ghost Tests to verify patch integrity beyond visible functional requirements. The experiment involves training and evaluation scenarios. The remediation speed, remediation efficacy for both standard functionality tests and security tests, and participant perception will be evaluated. [Pilot Study:] A pilot experiment with a small sample of participants has been conducted, providing insights for the following study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。