用大模型自动评阅命令行考试，按认知难度分级提升评分准确率。
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
- 基于四层认知分类法设计评分框架，区分命令复杂度。
- 谷歌Gemini在提示优化下与专家评分一致率达0.888，误差仅0.10。
- 提示词质量比模型选择更重要，高阶题更难精准评分。
可扩展且可靠的命令行考试评分在计算教育中仍是挑战，学生人数增加使人工批改困难，而规则型自动评分器无法处理部分得分、等效答案或语法变体。本文评估了四种前沿大语言模型（GPT、Claude Opus、Gemini、GLM）在评分简短的Linux/bash命令响应时是否能接近专家判断。研究采用四层认知分类法，结合认知复杂度与操作影响，涵盖信息检索（L1）、基础文件操作（L2）、结构化操作（L3）和高级系统管理（L4）。模型在两种提示策略下测试：最小基线与基于评分标准的增强版本，数据来自200名大二计算机工程专业学生的1200份真实作答，由三位专家独立评分。结果显示，使用评分标准引导提示的Gemini 3.0 Pro达到最高人类-模型一致性（ICC(3,1) = 0.888，MAE = 0.10，Bland-Altman偏差 = -0.014）。一致性随分类层级升高而下降，高阶任务差异最大。所有模型中，评分标准质量的影响大于模型提供商选择，结构化提示始终提升一致性。结果表明，题目复杂度是大模型评分准确性的可靠预测因子，为确定哪些题目适合AI辅助评分、哪些需人工评审提供了原则性框架，并提供可迁移的评估协议与提示模板。
原文摘要 · Abstract (English)
Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。