用大模型评判代码自然度，自动识别并修复对抗攻击。
Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge
- 让大模型当裁判，判断代码是否自然来识别对抗样本。
- 80%以上攻击代码可被检测，净化后模型准确率提升超83%。
- 轻量设计仅70亿参数，效果媲美GPT-4，适合实际部署。
代码语言模型在软件工程中的广泛应用暴露了对抗攻击的漏洞，尤其是标识符替换攻击。尽管现有攻击方法成功率高，但常生成不自然的代码模式。本文通过大模型作为评判者系统评估对抗样本质量，发现超过80%的先进攻击（如ALERT）生成的代码可被检测。基于此，提出EP-Shield统一框架，通过自然度感知推理实现评估与净化：先判断代码自然性并定位被篡改部分，再修复使目标模型恢复正确预测。大量实验表明，该框架优于对抗微调（最高提升83.36%），且仅需70亿参数，性能接近GPT-4。
原文摘要 · Abstract (English)
The widespread adoption of code language models in software engineering tasks has exposed vulnerabilities to adversarial attacks, especially the identifier substitution attacks. Although existing identifier substitution attackers demonstrate high success rates, they often produce adversarial examples with unnatural code patterns. In this paper, we systematically assess the quality of adversarial examples using LLM-as-a-Judge. Our analysis reveals that over 80% of adversarial examples generated by state-of-the-art identifier substitution attackers (e.g., ALERT) are actually detectable. Based on this insight, we propose EP-Shield, a unified framework for evaluating and purifying identifier substitution attacks via naturalness-aware reasoning. Specifically, we first evaluate the naturalness of code and identify the perturbed adversarial code, then purify it so that the victim model can restore correct prediction. Extensive experiments demonstrate the superiority of EP-Shield over adversarial fine-tuning (up to 83.36% improvement) and its lightweight design 7B parameters) with GPT-4-level performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。