评估大模型在高阶编程题中的代码生成能力，发现其对进阶题目挑战更大但仍有参考价值。
Evaluating Code Generation of LLMs in Advanced Computer Science Problems
- 选取12道高阶编程题，每题生成1000个测试用例验证代码正确性。
- 大模型在入门题中表现优秀，但在高阶题中成功率显著下降。
- 结果可为高年级编程课程设计提供参考，适合教学研究者和课程设计者。
大型语言模型（如GitHub Copilot和ChatGPT）在编程学习中日益流行。以往研究主要评估模型在入门级计算机科学课程作业中的代码生成能力，结果显示模型在基础编程任务中表现优异。然而，针对高阶编程作业的评估仍存在空白。本文评估了四种LLM工具在三种主流编程语言（Java、Python、C）的高年级课程编程题上的表现，手动选取12道题目，其中3道为入门级作为基线，9道来自二、三年级课程。针对每道题构建包含1000个测试用例的测试套件，分析程序输出。结果显示，尽管大模型在入门级任务中高效，但在高阶任务中面临更大挑战；但仍能识别核心问题并提供部分有效解决方案，对计算机科学学生具有辅助价值。研究结果可为高年级编程课程的教学设计提供指导。
原文摘要 · Abstract (English)
Large Language Models (LLMs), such as GitHub Copilot and ChatGPT have become popular among programming students. Students use LLMs to assist them in programming courses, including generating source code. Previous work has evaluated the ability of LLMs in solving introductory-course programming assignments. The results have shown that LLMs are highly effective in generating code for introductory Computer Science (CS) courses. However, there is a gap in research on evaluating LLMs' ability to generate code that solves advanced programming assignments. In this work, we evaluate the ability of four LLM tools to solve programming assignments from advanced CS courses in three popular programming languages, Java, Python, and C. We manually select 12 problems, three problems from introductory courses as the baseline and nine programming assignments from second- and third-year CS courses. To evaluate the LLM-generated code, we generate a test suite of 1000 test cases per problem and analyze the program output. Our evaluation shows that although LLMs are highly effective in generating source code for introductory programming courses, solving advanced programming assignments is more challenging. Nonetheless, in many cases, LLMs identify the base problem and provide partial solutions that may be useful to CS students. Furthermore, our results may provide useful guidance for teachers of advanced programming courses on how to design programming assignments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。