用大模型自动生成编程作业测试用例，提升批改效率与覆盖面。
Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
- 输入题目和参考答案，让GPT-4生成可直接用于自动评分的测试集。
- 在26道题、超2.5万次提交中验证，生成测试集覆盖度不低于人工设计。
- 能发现题目描述歧义，助力教学设计优化，适合教育技术研究者。
自动批改编程作业可为学生提供即时反馈，显著降低教师批改负担。然而，为自动评分系统构建全面的测试用例耗时且复杂，可能导致部分教师不愿增设题目或测试覆盖不足，从而给出误导性反馈，削弱学生及时获益的机会。本文评估了将大语言模型（LLMs）融入工作流，自动生成入门级编程题测试用例的有效性。将题目描述与参考解法输入GPT-4，生成可用于自动评分的测试集。我们基于26个问题及超过25,000份学生提交的代码进行评估，对比模型生成与教师手动编写的测试集表现。结果表明，大模型生成的测试集能准确识别多数正确解法，多数问题的覆盖程度不逊于人工测试集。此外，生成的测试集暴露了部分题目描述中的模糊点，凸显其在提升自动评分与教学设计方面的潜力。
原文摘要 · Abstract (English)
Automatically graded programming assignments provide instant feedback to students and significantly reduce manual grading time for instructors. However, creating comprehensive suites of test cases for programming problems within automatic graders can be time-consuming and complex. The effort needed to define test suites may deter some instructors from creating additional problems or lead to inadequate test coverage, potentially resulting in misleading feedback on student solutions. Such limitations may reduce student access to the well-documented benefits of timely feedback when learning programming. In this work, we evaluate the effectiveness of using Large Language Models (LLMs), as part of a larger workflow, to automatically generate test suites for CS1-level programming problems. Each problem's statement and reference solution are provided to GPT-4 to produce a test suite that can be used by an autograder. We evaluate our proposed approach using a sample of 26 problems, and more than 25,000 attempted solutions to those problems, submitted by students in an introductory programming course. We compare the performance of the LLM-generated test suites against the instructor-created test suites for each problem. Our findings reveal that LLM-generated test suites can correctly identify most valid solutions, and for most problems are at least as comprehensive as the instructor test suites. Additionally, the LLM-generated test suites exposed ambiguities in some problem statements, underscoring their potential to improve both autograding and instructional design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。