为编程教育中的大模型设计安全防护框架，防止恶意提示干扰学习。
CodeGuard: Improving LLM Guardrails in CS Education
- 构建首个面向教育场景的提示分类体系
- 提出实时检测工具PromptShield，F1达0.93
- 可减少30%-65%有害代码生成，不影响正常教学
大型语言模型（LLMs）正被广泛应用于计算机科学教育中，以自动化代码生成、反馈和评估。然而，其对对抗性或恶意提示的脆弱性威胁学生学习与学术诚信。为此，我们评估了现有通用LLM在处理教育领域不安全及无关提示时的表现，发现现有安全机制存在显著不足，由此提出CodeGuard——一个面向教育AI系统的综合性安全防护框架。该框架包含：(i) 首个针对教育场景的提示分类体系；(ii) 包含8,000条提示的CodeGuard数据集，覆盖该分类体系；(iii) PromptShield，一种轻量级句向量模型，经微调后可实时检测不安全提示。实验表明，PromptShield取得0.93的F1分数，优于现有方法。进一步测试显示，CodeGuard可降低30%-65%潜在有害或违规代码生成，同时保持合法教育任务性能不变。代码、数据集与评估脚本已开源共享。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly embedded in Computer Science (CS) classrooms to automate code generation, feedback, and assessment. However, their susceptibility to adversarial or ill-intentioned prompts threatens student learning and academic integrity. To cope with this important issue, we evaluate existing off-the-shelf LLMs in handling unsafe and irrelevant prompts within the domain of CS education. We identify important shortcomings in existing LLM guardrails which motivates us to propose CodeGuard, a comprehensive guardrail framework for educational AI systems. CodeGuard includes (i) a first-of-its-kind taxonomy for classifying prompts; (ii) the CodeGuard dataset, a collection of 8,000 prompts spanning the taxonomy; and (iii) PromptShield, a lightweight sentence-encoder model fine-tuned to detect unsafe prompts in real time. Experiments show that PromptShield achieves 0.93 F1 score, surpassing existing guardrail methods. Additionally, further experimentation reveals that CodeGuard reduces potentially harmful or policy-violating code completions by 30-65% without degrading performance on legitimate educational tasks. The code, datasets, and evaluation scripts are made freely available to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。