用题目定制评分标准,让大模型更懂代码逻辑对错
Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
- 为每道题设计专属评分标准,而非通用规则
- 在150份算法题和80份面向对象题中提升评估准确率
- 适合教育场景下想获得精准反馈的教师与学生
自GPT-3和ChatGPT发布以来,大语言模型(LLMs)在编程任务中展现出巨大潜力。尽管代码生成研究活跃,但基于LLM的代码评估仍处于探索阶段。本文聚焦于该方向,提出一种多智能体方法,采用针对具体问题的‘题目专属评分标准’,认为其比通用评分标准更利于逻辑判断。为弥补数据集不足,构建了两个新数据集:包含150份学生提交的《数据结构与算法》题库,以及涵盖80份本科生课程作业的面向对象编程数据集。除使用标准指标(斯皮尔曼相关系数、科恩卡帕值)外,还提出新指标“宽松度”,量化评估严格性相对于专家标准的程度。实验表明,题目专属评分标准显著提升教育场景中代码逻辑评估效果,反馈更契合教学目标,超越语法正确性。
原文摘要 · Abstract (English)
Since the emergence of Large Language Models (LLMs) popularized by the release of GPT-3 and ChatGPT, LLMs have shown remarkable promise in programming-related tasks. While code generation using LLMs has become a popular field of research, code evaluation using LLMs remains under-explored. In this paper, we focus on LLM-based code evaluation and attempt to fill in the existing gaps. We propose multi-agentic novel approaches using \emph{question-specific rubrics} tailored to the problem statement, arguing that these perform better for logical assessment than the existing approaches that use \emph{question-agnostic rubrics}. To address the lack of suitable evaluation datasets, we introduce two datasets: a Data Structures and Algorithms dataset containing 150 student submissions from a popular Data Structures and Algorithms practice website, and an Object Oriented Programming dataset comprising 80 student submissions from undergraduate computer science courses. In addition to using standard metrics (Spearman Correlation, Cohen's Kappa), we additionally propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment. Our comprehensive analysis demonstrates that \emph{question-specific rubrics} significantly enhance logical assessment of code in educational settings, providing better feedback aligned with instructional goals beyond mere syntactic correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。