用游戏训练大模型,提升规划与社交等通用能力。
GIFT: Games as Informal Training for Generalizable LLMs
- 用棋类、猜谜等游戏替代传统指令,无标注地训练模型
- 结合数学与游戏任务,使模型在多能力上泛化效果提升23%以上
- 适合想提升模型综合智能的研究者或开发者
当前大语言模型在数学推理和代码生成等正式任务上表现优异,但在规划、创造性和社交智能等更广泛能力上仍显不足。受人类学习方式启发——正式教学与非正式经验共同塑造智能——我们引入非正式学习,以游戏作为无需标注、反馈驱动的训练环境。为覆盖抽象推理、规划、创造力和社交互动等多元能力,将正式数学任务与三种代表性游戏任务(矩阵博弈、井字棋、谁是卧底)结合。然而,直接统一强化学习目标会模糊任务特异性学习信号,且缺乏对任务梯度方向的协调指导。为此,提出协同子任务训练(CST),用序列化的子任务更新替代单一混合更新,分离异构强化学习信号,同时隐式促进子任务间协调。在面向能力的基准测试中,基于游戏的非正式学习显著提升模型泛化能力,而CST进一步增强多任务强化学习性能,在保持领域内子任务表现的同时,有效提升整体通用能力。代码与数据已公开。
原文摘要 · Abstract (English)
Recent LLMs excel at formal tasks such as mathematical reasoning and code generation, but still struggle with broader abilities such as planning, creativity, and social intelligence. Inspired by human learning, where formal instruction and informal experience jointly shape intelligence, we introduce informal learning into LLM training and use games as annotation-free, feedback-driven environments. To cover diverse abilities including abstract reasoning, planning, creativity, and social interaction, we combine formal math tasks with three representative game tasks, including Matrix Games, TicTacToe, and Who's the Spy. However, directly mixing these tasks under a unified RL objective can blur task-specific learning signals and provides no explicit guidance for coordinating task-gradient directions. To combat these, we propose Coordinated Subtask Training (CST), which replaces a single mixed update with sequential subtask-specific updates, separating heterogeneous RL signals while implicitly promoting coordination among subtasks. Experiments on ability-oriented benchmarks show that game-based informal learning improves generalization beyond formal training alone, while CST further enhances multi-task RL by preserving in-domain subtask performance and improving broader general abilities. Code and data are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。