arXiv:2607.05571cs.AIcs.HC2026-07

为小模型当编程导师设计评估基准,发现调优方法比参数大小更重要。

CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming

论文配图:CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming
图 1 · 摘自论文原文
  • 构建面向积木编程的教育场景评测集,含17个带教学评分标准的问题
  • 多数模型在术语和语气上表现好,但难以避免答案泄露或复盘学生调试过程
  • 针对性提示优化让11个模型中10个得分提升,适合教育AI部署选型参考

大语言模型被广泛探索用于人工智能辅导,但在中小学场景中部署面临隐私、成本及对专有模型依赖等问题。小语言模型(SLMs)提供了有前景的替代方案,但针对特定教育领域(如积木式编程)选择合适模型仍具挑战性,因该领域数据通常未出现在模型训练集中。本文提出CSTutorBench,一个评估语言模型在VEX VR(一种积木式机器人环境)中担任计算机科学导师能力的基准。该基准包含17个基于场景的问题,采用基于教学与反馈研究的评分标准,并通过人机协同的LLM判官流程进行评估。对11个模型(4B–120B参数)的初步测试显示,模型在词汇和语调等表层指标上表现良好,但在深层教学行为上存在不足,尤其在避免答案泄露和利用学生调试历史方面。结果显示,模型家族与指令微调方式比参数量更能预测辅导质量,尽管样本数量有限使结论有待验证。基于最新教育提示工程研究的针对性提示优化,使11个模型中有10个得分提升。这些结果强调了在教育部署中使用情境化、教学理论驱动的基准的重要性。

原文摘要 · Abstract (English)

Large language models are increasingly explored as AI tutors, yet deploying them in K-12 settings raises concerns around privacy, cost, and reliance on proprietary models. Small language models (SLMs) offer a promising alternative, but selecting the right model for a specific educational context remains difficult, particularly when the target domain, such as block-based programming, is largely absent from model training data. We introduce CSTutorBench, a benchmark for evaluating language models as CS tutors in VEX VR, a block-based robotics environment. The benchmark comprises 17 scenario-based questions scored against a pedagogical rubric grounded in established tutoring and feedback research, with a human-in-the-loop LLM-as-judge pipeline for evaluation. Preliminary findings across 11 models (4B-120B parameters) reveal that models perform well on surface-level criteria such as vocabulary and tone but struggle with deeper pedagogical behaviors, particularly avoiding answer leakage and engaging with student debugging histories. In our sample, model family and instruction-tuning approach appear to be better predictors of tutoring quality than parameter count alone, though the small number of models limits the strength of this conclusion. A targeted prompt revision grounded in recent educational prompt engineering research improved scores for 10 of 11 models. These results underscore the value of context-specific, pedagogically grounded benchmarks for SLM selection in educational deployment.

小模型编程教育评测基准提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。