用对话游戏训练大模型，让模型通过互动自我提升。
Playpen: An Environment for Exploring Learning Through Conversational Interaction
- 用规则化对话游戏作为反馈信号，实现模型自玩自学。
- GRPO方法在未见游戏上表现更好且不损害其他能力。
- 适合对交互式学习和自适应训练感兴趣的开发者。
近期，通过奖励模型评估大语言模型回复适当性的交互式后训练受到关注。本文探讨对话游戏——以语言行为为主导的目标导向、规则驱动活动——是否可作为学习的反馈信号。我们提出Playpen环境，支持离线与在线的对话游戏自博弈学习，并测试了三种典型后训练方法：监督微调（SFT）、直接对齐（DPO）及基于GRPO的强化学习。在小规模模型Llama-3.1-8B-Instruct上进行实验，评估其在训练游戏未见实例、未见游戏以及标准基准上的表现。结果表明，通过SFT的模仿学习能提升未见实例表现，但会损害其他技能；而使用GRPO的交互学习则实现平衡提升且无技能损失。我们开源该框架与基线训练设置，推动合成互动学习这一新兴方向的研究。
原文摘要 · Abstract (English)
Interaction between learner and feedback-giver has come into focus recently for post-training of Large Language Models (LLMs), through the use of reward models that judge the appropriateness of a model's response. In this paper, we investigate whether Dialogue Games -- goal-directed and rule-governed activities driven predominantly by verbal actions -- can also serve as a source of feedback signals for learning. We introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning; direct alignment (DPO); and reinforcement learning with GRPO. We experiment with post-training a small LLM (Llama-3.1-8B-Instruct), evaluating performance on unseen instances of training games as well as unseen games, and on standard benchmarks. We find that imitation learning through SFT improves performance on unseen instances, but negatively impacts other skills, while interactive learning with GRPO shows balanced improvements without loss of skills. We release the framework and the baseline training setups to foster research in the promising new direction of learning in (synthetic) interaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。