通过计数游戏研究强化学习如何教会大模型组合新技能。
How Does RL Post-training Induce Skill Composition? A Case Study on Countdown
- 用表达式树分析模型解题结构，识别可复用的子任务技能。
- 模型能泛化到更大规模和未见树结构，展现组合能力。
- 浅层平衡结构先学会，右重结构即使深度相同也难掌握。
尽管强化学习（RL）有效提升大语言模型的推理能力，但其在促进组合泛化（即从已知组件合成新技能）方面的作用常被误认为仅是长度泛化。为此，我们研究了RL后训练如何影响技能组合，并分析组合结构对技能迁移的影响。聚焦于计数任务（给定n个数字和目标值，构造一个求值为目标的表达式），将模型解视为表达式树，其中每个子树对应一个可复用的子任务，即一种“技能”。通过追踪训练过程中树形结构及其成功概率，发现：(i) 模型在更大n值和未见树结构上实现分布外（OOD）泛化，表明子任务可组合复用；(ii) 学习能力具有结构依赖性——模型优先掌握浅层平衡树（子任务负载均衡），而对深层非平衡树中的右重结构存在持续脆弱性（即使与某些左重结构深度相同）。诊断揭示了所学内容、学习顺序及泛化失败点，澄清了仅靠强化学习的后训练如何实现超越标准指标（如pass@k）的分布外泛化。
原文摘要 · Abstract (English)
While reinforcement learning (RL) successfully enhances reasoning in large language models, its role in fostering compositional generalization (the ability to synthesize novel skills from known components) is often conflated with mere length generalization. To this end, we study what RL post-training teaches about skill composition and how the structure of the composition affects the skill transfer. We focus on the Countdown task (given n numbers and a target, form an expression that evaluates to the target) and analyze model solutions as expression trees, where each subtree corresponds to a reusable subtask and thus can be viewed as a ``skill.'' Tracking tree shapes and their success rates over training, we find: (i) out-of-distribution (OOD) generalization to larger n and to unseen tree shapes, indicating compositional reuse of subtasks; (ii) a structure-dependent hierarchy of learnability -- models master shallow balanced trees (workload is balanced between subtasks) before deep unbalanced ones, with persistent fragility on right-heavy structures (even when the composition depth is the same as some left-heavy structures). Our diagnostic reveals what is learned, in what order, and where generalization fails, clarifying how RL-only post-training induces OOD generalization beyond what standard metrics such as pass@k reveal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。