让大模型当苏格拉底导师，用启发式强化学习减少直接给答案。
Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning
- 分两阶段训练：先监督微调，再用群体相对策略优化对齐苏格拉底式引导
- 苏格拉底效果提升至63.3%，关键词泄露从30.0%降到13.3%
- 模型规模大也不自动具备引导能力，需专门训练
将大语言模型应用于教育场景时，常表现为直接给出答案，违背苏格拉底式渐进式提问的教学原则。本文提出HeuristicEdu，一种两阶段管道，通过监督预热与群体相对策略优化（GRPO）对Qwen2.5-7B进行对齐，使其扮演苏格拉底导师。训练基于SocraticEdu数据集——797条来自真实平台的中文儿童科学多轮对话，采用包含认知深度（R_cog）、好奇心参与度（R_eng）和直接性（R_dir）的启发式奖励，并引入K_query修正学生提出术语。引入支架有效性（SE）与对话深度（CD）评估超越表面流畅性的教学效果。在30个保留问题上，最优GRPO变体使SE从30.0%提升至63.3%，关键词泄露从30.0%降至13.3%。值得注意的是，优化中去除直接性惩罚反而更优，表明显式反泄漏约束可能与梯度对齐冲突。未对齐的Qwen-72B基线仅得0% SE与96.7%泄露，证明规模无法自动产生苏格拉底行为。
原文摘要 · Abstract (English)
Large language models (LLMs) deployed in educational settings often behave as direct answerers: they disclose target concepts in the opening turn instead of guiding students through progressive inquiry, as Socratic pedagogy prescribes. We present HeuristicEdu, a two-phase pipeline that aligns Qwen2.5-7B toward Socratic tutoring via supervised warm-up and Group Relative Policy Optimization (GRPO). Training uses SocraticEdu, 797 multi-turn Chinese children's science dialogues reconstructed from a live platform, with a heuristic reward over cognitive depth (R_cog), curiosity engagement (R_eng), and directness (R_dir), together with a K_query correction for student-introduced terms. We introduce Scaffolding Effectiveness (SE) and Conversation Depth (CD) to evaluate outcomes beyond surface fluency. On 30 held-out questions, the best GRPO variant improves SE from 30.0% to 63.3% and lowers keyword leakage from 30.0% to 13.3%. Notably, this best variant omits the directness penalty during optimization, suggesting that explicit anti-leakage terms can conflict with gradient-based behavioral alignment. An unaligned Qwen-72B baseline reaches 0% SE and 96.7% leakage, showing that scale alone does not induce Socratic behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。