arXiv:2606.18089cs.LG2026-06被引 2

揭示大模型推理的可组合泛化机制,提出模块化训练新范式。

From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

论文配图:From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
图 1 · 摘自论文原文
  • 用层级潜变量模型解析推理过程,识别可复用的原子模块。
  • 强化学习从合成轨迹中提取模块,实现新配置下的泛化能力提升。
  • 适合研究模型推理机制与高效训练策略的读者。

后训练流程中,监督微调(SFT)与强化学习(RL)的结合已成为将大语言模型转化为强推理系统的关键方法。本文认为其成功源于可组合泛化,并通过分层潜选择模型形式化这一机制。该模型将推理轨迹建模为一系列离散潜变量选择,对应可复用的原子模块,包括技能(局部操作)和路由机制(中间信息的选择、复用与组合方式)。理论上证明:SFT 提供复合轨迹中的原始模块材料,而 RL 则分解这些轨迹以识别潜原子模块,从而实现可组合泛化。我们设计受控实验验证该理论:结果表明,RL 能从 SFT 提供的复合轨迹中提取原子模块并重新组合解决新问题;且在复合轨迹上训练比在孤立原子模块上训练具有更强泛化能力。最后,我们发现一种有效协议:SFT 通过复合轨迹覆盖所有原子模块,而 RL 专注于探索超出 SFT 支撑范围的新组合以驱动探索。

原文摘要 · Abstract (English)

Post-training pipelines that combine supervised fine-tuning (SFT) with reinforcement learning (RL) have emerged as the key recipe for transforming large language models (LLMs) into robust reasoners. We argue that this combined success is driven by compositional generalization, which we formalize through a hierarchical latent selection model. In this framework, reasoning traces are generated by a cascade of discrete latent selection variables corresponding to reusable atomic modules, including both skills (local operations) and routing mechanisms (how intermediate information is selected, reused, and composed). Within this model, we theoretically show that SFT and RL play asymmetric, complementary roles: SFT supplies the raw module materials in compositional traces, and RL decomposes those traces to identify the latent atomic modules and enable compositional generalization. We design controlled experiments to validate this theory. Our results demonstrate that RL can extract atomic modules from compound traces supplied by SFT and recombine them to solve new configurations. Moreover, we find that training on compound traces yields stronger generalization than training on isolated atomic modules. Finally, we investigate the relationship between SFT and RL data and identify an effective protocol in which SFT ensures coverage of all atomic modules through compositional traces, while RL focuses on novel compositions outside the SFT support to drive exploration.

大模型推理可组合泛化强化学习模块化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。