用AI评估编程作业,新方法能更精细地打分。
Evaluating Generative AI for CS1 Code Grading: Direct vs Reverse Methods
- AI直接评分或先改错再估分,后者更细致。
- 改错法在逻辑错误上评分更准,但需精心设计提示词。
- 适合想提升批改效率与公平性的计算机课程教师。
初学编程课程的作业人工评分耗时且易不一致。虽然单元测试常用于自动评估,但多为通过/失败二值判断,无法给部分分数。大语言模型(LLMs)为自动化、可扩展且更客观的评分提供了可能。本文比较了两种AI评分方法: extit{Direct}(AI直接根据评分标准评估学生代码)和 extit{Reverse}(新提出的方法,先修复代码错误,再依据修改类型和数量推断分数)。两种方法在原始评分尺度及十倍扩展尺度下均进行评估,以检验评分范围对准确率的影响。通过与人类助教评分对比,在多种编程问题和错误类型上验证了效果。初步结果显示,尽管直接法更快,但逆向法因聚焦于修正难度,能提供更细粒度的评价。两者均需仔细设计提示词,尤其在分配部分分数和处理逻辑错误时。为进一步测试一致性,还使用Gemini Flash 2.0生成合成学生代码,涵盖更多受控错误类型与难度等级。本文讨论了各方法优劣、提示词设计要点,以及未来混合人机评分系统的发展方向,旨在提升计算机课程中评分的一致性、效率与公平性。
原文摘要 · Abstract (English)
Manual grading of programming assignments in introductory computer science courses can be time-consuming and prone to inconsistencies. While unit testing is commonly used for automatic evaluation, it typically follows a binary pass/fail model and does not give partial marks. Recent advances in large language models (LLMs) offer the potential for automated, scalable, and more objective grading. This paper compares two AI-based grading techniques: \textit{Direct}, where the AI model applies a rubric directly to student code, and \textit{Reverse} (a newly proposed approach), where the AI first fixes errors, then deduces a grade based on the nature and number of fixes. Each method was evaluated on both the instructor's original grading scale and a tenfold expanded scale to assess the impact of range on AI grading accuracy. To assess their effectiveness, AI-assigned scores were evaluated against human tutor evaluations on a range of coding problems and error types. Initial findings suggest that while the Direct approach is faster and straightforward, the Reverse technique often provides a more fine-grained assessment by focusing on correction effort. Both methods require careful prompt engineering, particularly for allocating partial credit and handling logic errors. To further test consistency, we also used synthetic student code generated using Gemini Flash 2.0, which allowed us to evaluate AI graders on a wider range of controlled error types and difficulty levels. We discuss the strengths and limitations of each approach, practical considerations for prompt design, and future directions for hybrid human-AI grading systems that aim to improve consistency, efficiency, and fairness in CS courses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。