用对抗扰动让大模型在编程作业中作弊失效
Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation
- 对大模型生成代码施加对抗扰动，降低其正确性
- 平均正确率下降77%，有效抑制作弊行为
- 适合想防大模型作弊的课程教师参考
大型语言模型（如 CoPilot、ChatGPT）虽能提升专业开发效率，却也助长初学编程课程中的作弊行为。本文评估了5个主流LLM在初学者编程题上的基础表现，研究了对抗扰动对其性能的降级效果，并通过用户实验分析扰动在真实编程任务中阻碍代码生成的有效性。实验表明：1）扰动使平均正确率下降77%；2）扰动的可检测性影响其降效程度。
原文摘要 · Abstract (English)
While Large language model (LLM)-based programming assistants such as CoPilot and ChatGPT can help improve the productivity of professional software developers, they can also facilitate cheating in introductory computer programming courses. Assuming instructors have limited control over the industrial-strength models, this paper investigates the baseline performance of 5 widely used LLMs on a collection of introductory programming problems, examines adversarial perturbations to degrade their performance, and describes the results of a user study aimed at understanding the efficacy of such perturbations in hindering actual code generation for introductory programming assignments. The user study suggests that i) perturbations combinedly reduced the average correctness score by 77%, ii) the drop in correctness caused by these perturbations was affected based on their detectability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。