用隐状态优化对话策略，让AI tutor更会长期引导学生解题。
Efficient RL for optimizing conversation level outcomes with an LLM-based tutor
- 用低维隐状态表征学生对话历史，指导长期教学决策。
- 在模拟数学辅导中，显著提升学生独立解题的成功率。
- 轻量级设计，比端到端训练更省算力，适合实际部署。
基于现有基于人类反馈的强化学习（RLHF）框架的大语言模型通常仅优化单轮对话中的即时回应偏好，但在多轮对话（如在线数学辅导）中表现不足。本文提出一种新方法：通过低维隐状态表示学生对话历史，并基于该状态优化长期策略，以决定高阶教学动作。目标是使AI导师行为更契合长期目标——引导学生自主解决目标数学问题。该模型轻量化，相比以往端到端训练导师策略直接输出下一句的方式，计算资源需求更低。实验表明，在LLM模拟的辅导任务中，该方法显著提升了长期教学效果。
原文摘要 · Abstract (English)
Large language models (LLMs) built on existing reinforcement learning with human feedback (RLHF) frameworks typically optimize responses based on immediate turn-level human preferences. However, this approach falls short in multi-turn dialogue settings, such as online math tutoring. We propose a method to enhance LLM-based tutors by representing the dialogue history with a lower-dimensional latent state representation of a student and optimizing a long-term policy to determine high-level actions based on the latent state. The goal is to better align the tutor's behavior with the long-term objective of guiding the student towards solving a target math problem on their own. Our model is lightweight, requiring less computational resources than prior work of training the tutor policy end-to-end to directly output the tutor's next utterance. Our experiment results demonstrate that these modifications lead to improved long-term outcomes compared to prompting in LLM-simulated tutoring tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。