评测主流AI在大学编程考试中的表现,发现其代码生成能力已超普通学生。
Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026
- 用真实课程题目测试5款AI,按学生评分标准评估表现
- 多数AI得分高于学生平均分,长题常拿满分,但仍有编译错误
- 在接口、抽象类等高级概念上仍易出错,适合教育研究与教学设计参考
近年来生成式人工智能(GenAI)的进展显著提升了大语言模型(LLM)生成和解释源代码的能力。然而,其在真实面向对象编程(OOP)评估中的表现仍不明确。本研究评估了五款广泛使用的GenAI系统:ChatGPT-5.2、DeepSeek-V3、Gemini 2.5 Flash、Claude Sonnet 4.5 和 M365 Copilot,使用某大学入门级OOP课程的真实编程测试与考试题。生成代码采用与学生相同的评分标准进行评估,并与往年学生结果对比。分析发现,所有被测系统得分均高于平均水平,且在较长编程任务中常获满分。然而,它们偶尔生成无法编译的代码,对高级OOP概念(如接口、抽象类及特定继承任务)仍存在困难,且在涉及图像理解的图形类问题上表现受限。相比去年,各系统在多数任务上均有明显提升,但重复性错误模式依然存在。研究为当前GenAI在真实入门级OOP评估中的能力与局限提供了更新依据,可支持编程评估设计、负责任地融入软件工程教育,以及未来对AI辅助编程演进的研究。
原文摘要 · Abstract (English)
Recent advances in Generative Artificial Intelligence (GenAI) have substantially improved the ability of large language models (LLMs) to generate and explain source code. However, their performance on authentic object-oriented programming (OOP) assessments remains insufficiently understood. This study evaluates five widely used GenAI systems, ChatGPT-5.2, DeepSeek-V3, Gemini 2.5 Flash, Claude Sonnet 4.5, and M365 Copilot, using programming tests and examination tasks from an introductory university OOP course. The generated solutions were assessed using the same grading criteria applied to students and compared with historical student results from the same course, as well as findings from the previous year. Common errors were also analyzed to identify recurring limitations across models. All evaluated GenAI systems achieved higher scores than the average student cohort and frequently obtained full marks on longer programming tasks. Nevertheless, they occasionally produced non-compiling code and continued to struggle with advanced OOP concepts, particularly interfaces, abstract classes, and certain inheritance-related tasks. Performance was also limited on graphics-related questions involving image interpretation. Compared with the previous year, the evaluated systems demonstrated noticeable improvements across most assessments while exhibiting several recurring error patterns. The findings provide an updated evaluation of the capabilities and limitations of contemporary GenAI systems on authentic introductory OOP assessments. They also offer evidence that can inform the design of programming assessments, the responsible integration of GenAI tools into software engineering education, and future studies evaluating the evolution of AI-assisted programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。