构建首个开源评测框架，评估大模型在数学辅导中的教学能力。
MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
- 设计多维度数据集与评分体系，覆盖对话式教学核心能力
- 训练奖励模型区分专家与新手教师回复，准确率高
- 发现解题能力强不等于会教学，长期对话中提问策略易失效
评估基于AI的辅导模型的教学能力对推动该领域发展至关重要。然而，目前缺乏可靠、易用且可快速运行的评估方法来反映模型的真实教学水平。为此，我们提出MathTutorBench——一个开源基准，用于全面评估辅导型大模型。该基准包含涵盖对话教学中学习科学定义的关键能力的数据集与指标。为评估开放式教师回复的教学质量，我们训练了一个奖励模型，其能以高准确率区分专家与新手教师的回答。我们在多种闭源与开源模型上进行了评估，发现仅具备学科解题能力并不等同于优秀教学；教学能力与学科专长之间存在权衡，这一权衡取决于模型的辅导专业化程度。此外，在较长对话中，简单提问策略逐渐失效，教学难度显著上升。我们已开源该基准、代码与排行榜，以支持未来模型的快速评测。
原文摘要 · Abstract (English)
Evaluating the pedagogical capabilities of AI-based tutoring models is critical for making guided progress in the field. Yet, we lack a reliable, easy-to-use, and simple-to-run evaluation that reflects the pedagogical abilities of models. To fill this gap, we present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation. MathTutorBench contains a collection of datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching. To score the pedagogical quality of open-ended teacher responses, we train a reward model and show it can discriminate expert from novice teacher responses with high accuracy. We evaluate a wide set of closed- and open-weight models on MathTutorBench and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching. Rather, pedagogy and subject expertise appear to form a trade-off that is navigated by the degree of tutoring specialization of the model. Furthermore, tutoring appears to become more challenging in longer dialogs, where simpler questioning strategies begin to fail. We release the benchmark, code, and leaderboard openly to enable rapid benchmarking of future models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。