评测大模型对大学生手写理科学习题的理解能力，发现其识别错误严重，影响自动评分可靠性。
EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions
- 构建1300+份真实手写解题数据集，含专家校验的转录与评分报告
- 发现大模型在识别公式、图示和推理逻辑时存在严重错误，影响自动评分
- 提出通过识别错误模式预纠正，仅需3.3%人工介入即可提升系统鲁棒性
多模态大语言模型（MLLMs）有望革新传统教育并减轻教师负担。然而，准确解析包含数学公式、图表和文本推理的自由格式理科学习题手写解答，因缺乏真实且领域特定的基准而面临挑战。现有评估主要依赖下游任务结果（如自动评分），仅覆盖部分内容，难以全面反映模型对复杂手写逻辑的整体理解。为此，我们发布EDU-CIRCUIT-HW数据集，包含1300+份大学理学课程的真实学生手写解答。基于专家验证的逐字转录和评分报告，我们同时评估多种MLLM在上游识别保真度与下游自动评分性能上的表现。评估揭示了模型在识别学生手写内容中存在惊人的隐性失败，表明其在高风险教育场景中的自动评分等应用不可靠。作为潜在解决方案，我们展示案例研究：通过利用识别出的错误模式预先检测并修正识别错误，仅需将3.3%的作业转交人工评分，其余由GPT-5.1完成，即可有效提升部署式AI评分系统的鲁棒性。代码与数据集见GitHub：https://gt-learning-innovation.github.io/CIRCUIT_EDU_HW_ACL。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) hold significant promise for revolutionizing traditional education and reducing teachers' workload. However, accurately interpreting unconstrained STEM student handwritten solutions with intertwined mathematical formulas, diagrams, and textual reasoning poses a significant challenge due to the lack of authentic and domain-specific benchmarks. Additionally, current evaluation paradigms predominantly rely on the outcomes of downstream tasks (e.g., auto-grading), which often probe only a subset of the recognized content, thereby failing to capture the MLLMs' understanding of complex handwritten logic as a whole. To bridge this gap, we release EDU-CIRCUIT-HW, a dataset consisting of 1,300+ authentic student handwritten solutions from a university-level STEM course. Utilizing the expert-verified verbatim transcriptions and grading reports of student solutions, we simultaneously evaluate various MLLMs' upstream recognition fidelity and downstream auto-grading performance. Our evaluation uncovers an astonishing scale of latent failures within MLLM-recognized student handwritten content, highlighting the models' insufficient reliability for auto-grading and other understanding-oriented applications in high-stakes educational settings. As a potential solution, we present a case study demonstrating that leveraging identified error patterns to preemptively detect and correct recognition errors, while requiring only minimal human intervention (e.g., routing 3.3% of assignments to human graders and the remainder to the GPT-5.1 grader), can effectively enhance the robustness of the deployed AI-enabled grading system. Code and dataset are available in this GitHub repo: https://gt-learning-innovation.github.io/CIRCUIT_EDU_HW_ACL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。