测试大模型是否像人一样会揣摩学习者知识来决定教什么。
Do Large Language Models Mentalize When They Teach?
- 用逆向规划建模，看模型能否推断学习者缺了哪些知识节点。
- 多数模型表现接近人类，且策略稳定，最优贝叶斯模型解释力最强。
- 提示引导不等于教学变好，反而可能让模型在复杂任务中表现下降。
我们通过一个受控任务测试大语言模型（LLM）在教学时是否基于对学习者知识状态的推理，而非简单规则。每轮任务中，教师模型需根据虚拟学习者在带奖励标注的有向图中的路径，选择揭示一条边，以帮助学习者重新规划后走向更优路径。实验中使用多种LLM作为模拟教师，用与人类研究相同的认知模型拟合其逐次决策：贝叶斯最优教师（反向规划）、较弱的贝叶斯变体、启发式基线（如仅基于奖励）以及非心理化效用模型。在与人类被试相同刺激条件下，多数模型表现良好，策略随时间变化小，整体表现与人类相似。贝叶斯信息准则（BIC）分析显示，贝叶斯最优教学模型能最好解释大多数模型的选择。当引入辅助提示（如要求关注推理或奖励），模型虽表面服从，但在启发式不一致的测试图上并未提升表现，甚至有时降低。结果表明，认知模型可揭示大模型的教学策略，但提示合规不等于教学优化。
原文摘要 · Abstract (English)
How do LLMs decide what to teach next: by reasoning about a learner's knowledge, or by using simpler rules of thumb? We test this in a controlled task previously used to study human teaching strategies. On each trial, a teacher LLM sees a hypothetical learner's trajectory through a reward-annotated directed graph and must reveal a single edge so the learner would choose a better path if they replanned. We run a range of LLMs as simulated teachers and fit their trial-by-trial choices with the same cognitive models used for humans: a Bayes-Optimal teacher that infers which transitions the learner is missing (inverse planning), weaker Bayesian variants, heuristic baselines (e.g., reward based), and non-mentalizing utility models. In a baseline experiment matched to the stimuli presented to human subjects, most LLMs perform well, show little change in strategy over trials, and their graph-by-graph performance is similar to that of humans. Model comparison (BIC) shows that Bayes-Optimal teaching best explains most models' choices. When given a scaffolding intervention, models follow auxiliary inference- or reward-focused prompts, but these scaffolds do not reliably improve later teaching on heuristic-incongruent test graphs and can sometimes reduce performance. Overall, cognitive model fits provide insight into LLM tutoring policies and show that prompt compliance does not guarantee better teaching decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。