用对话式AI验证学生编程理解,防止只写对代码却不懂原理。
Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems
- 设计双代理对话层结合代码分析与追问机制
- 通过实时执行状态引导推理,减少模型幻觉
- 适合需要深度评估理解的编程教学场景
大型语言模型(LLMs)正挑战传统自动化编程评估方式,因学生可写出功能正确的代码却未展现真实理解。本文首先开展基于饱和度的系统性文献综述,梳理出三类主流对话式评估架构:基于规则/模板的系统、基于LLM的系统以及混合系统。现有研究显示,对话代理在提供可扩展反馈和深入探查代码理解方面具有潜力,但仍存在幻觉、过度依赖、隐私、完整性及部署限制等关键问题。其次,本文提出一种混合苏格拉底框架,将对话验证机制融入自动化编程评估系统(APAS)。该框架结合确定性代码分析、双代理对话层、知识追踪、分步提问与基于运行时事实的提示约束。同时提出多项实际防护措施,包括监考部署模式、随机化跟踪问题、与具体执行状态绑定的逐步推理,以及本地化模型部署以应对隐私敏感场景。该框架并非取代传统测试,而是作为补充层,用于验证学生提交代码的真实理解程度。
原文摘要 · Abstract (English)
Large Language Models (LLMs) challenge conventional automated programming assessment because students can now produce functionally correct code without demonstrating corresponding understanding. This paper makes two contributions. First, it reports a saturation-based scoping review of conversational assessment approaches in programming education. The review identifies three dominant architectural families: rule-based or template-driven systems, LLM-based systems, and hybrid systems. Across the literature, conversational agents appear promising for scalable feedback and deeper probing of code understanding, but important limitations remain around hallucinations, over-reliance, privacy, integrity, and deployment constraints. Second, the paper synthesizes these findings into a Hybrid Socratic Framework for integrating conversational verification into Automated Programming Assessment Systems (APASs). The framework combines deterministic code analysis with a dual-agent conversational layer, knowledge tracking, scaffolded questioning, and guardrails that tie prompts to runtime facts. The paper also discusses practical safeguards against LLM-generated explanations, including proctored deployment modes, randomized trace questions, stepwise reasoning tied to concrete execution states, and local-model deployment options for privacy-sensitive settings. Rather than replacing conventional testing, the framework is intended as a complementary layer for verifying whether students understand the code they submit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。