测试大模型修复漏洞时，是否依赖记忆而非真正理解。
Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs
- 故意在提示中错误定位漏洞，检验模型能否正确修复。
- 即使错位10行，仍能修复约65%的漏洞，说明依赖记忆。
- 适合关注LLM真实能力、安全可信性的研究者参考。
自动化漏洞修复（AVR）是程序修复领域的新兴方向。近期研究表明，大型语言模型（LLMs）在该任务上优于传统方法，其成功可能源于训练数据泄露或精确的故障定位等“看不见的手”。为验证这一假设，本研究在受控条件下复现了先前的AVR实验：通过将提示中的漏洞位置人为偏移n行，模拟不准确的故障定位。使用Vul4J和VJTrans基准测试，先由一个LLM生成补丁，再由第二个LLM审查，最后通过回归测试与漏洞证明测试验证结果。随机抽样手动审计补丁并用Agresti-Coull-Wilson方法估算错误率。结果显示，即使偏移10行，仍有约65%的修复成功，表明模型更依赖对已知修复方案的记忆，而非真正的推理能力。
原文摘要 · Abstract (English)
Background: Automated Vulnerability Repair (AVR) is a fast-growing branch of program repair. Recent studies show that large language models (LLMs) outperform traditional techniques, extending their success beyond code generation and fault detection. Hypothesis: These gains may be driven by hidden factors -- "invisible hands" such as training-data leakage or perfect fault localization -- that let an LLM reproduce human-authored fixes for the same code. Objective: We replicate prior AVR studies under controlled conditions by deliberately adding errors to the reported vulnerability location in the prompt. If LLMs merely regurgitate memorized fixes, both small and large localization errors should yield the same number of correct patches, because any offset should divert the model from the original fix. Method: Our pipeline repairs vulnerabilities from the Vul4J and VJTrans benchmarks after shifting the fault location by n lines from the ground truth. A first LLM generates a patch, a second LLM reviews it, and we validate the result with regression and proof-of-vulnerability tests. Finally, we manually audit a sample of patches and estimate the error rate with the Agresti-Coull-Wilson method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。