研究代码注释如何欺骗AI安全审查，发现攻击效果有限。
Can Adversarial Code Comments Fool AI Security Reviewers -- Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis
- 构建跨语言注释攻击基准，测试8种对抗策略。
- 对抗注释对检测准确率影响微小，95%置信区间包含零。
- 静态分析交叉验证是最佳防御，适合高风险代码审查场景。
AI辅助代码审查广泛用于生产前漏洞检测。已有研究显示对抗性提示可降低大模型在代码生成中的表现。本文检验类似注释操纵是否会影响大模型在漏洞检测中的能力。构建包含Python、JavaScript、Java的100样本基准，每份代码配以8种注释变体（从无注释到权威伪装、技术欺骗等对抗策略）。评估8个前沿模型（5个商用、3个开源），共9,366次试验。对抗注释对检测准确率影响极小且统计不显著（McNemar精确p > 0.21；所有95%置信区间包含零）。该结论适用于商用模型（基线检测率89%-96%）和开源模型（53%-72%），尽管性能差距显著。复杂对抗策略并未优于简单操纵。进一步在4,646次额外试验中测试四种自动化防御（总计14,012次）。静态分析交叉验证表现最佳，检测率达96.9%，挽回47%基线遗漏。注释删除会降低弱模型检测能力，因去除有益上下文。失败主要集中在固有难度高的漏洞类型，如竞争条件、时序侧信道、复杂权限逻辑，而非对抗注释本身。
原文摘要 · Abstract (English)
AI-assisted code review is widely used to detect vulnerabilities before production release. Prior work shows that adversarial prompt manipulation can degrade large language model (LLM) performance in code generation. We test whether similar comment-based manipulation misleads LLMs during vulnerability detection. We build a 100-sample benchmark across Python, JavaScript, and Java, each paired with eight comment variants ranging from no comments to adversarial strategies such as authority spoofing and technical deception. Eight frontier models, five commercial and three open-source, are evaluated in 9,366 trials. Adversarial comments produce small, statistically non-significant effects on detection accuracy (McNemar exact p > 0.21; all 95 percent confidence intervals include zero). This holds for commercial models with 89 to 96 percent baseline detection and open-source models with 53 to 72 percent, despite large absolute performance gaps. Unlike generation settings where comment manipulation achieves high attack success, detection performance does not meaningfully degrade. More complex adversarial strategies offer no advantage over simple manipulative comments. We test four automated defenses across 4,646 additional trials (14,012 total). Static analysis cross-referencing performs best at 96.9 percent detection and recovers 47 percent of baseline misses. Comment stripping reduces detection for weaker models by removing helpful context. Failures concentrate on inherently difficult vulnerability classes, including race conditions, timing side channels, and complex authorization logic, rather than on adversarial comments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。