arXiv:2606.27721cs.LG2026-06被引 1

通过分步训练,让模型用更少数据学会复杂推理。

Learning to Reason with Curriculum II: Compositional Generalization

  • 将长问题拆成短子问题递归求解,逐步构建答案
  • 只需约2的√(log T)次方量级数据,远低于直接学习的T量级
  • 适合研究高效学习机制或强化学习中模型优化的人

组合泛化是自然与人工智能解决复杂问题的核心能力,依赖于将大问题分解为小问题并组合求解。本文以模拟半自动机(预测序列计算T步结果)为基准任务,研究其理论基础。基于前作提出的自适应课程学习方法,通过递归拆分长序列、逐层求解子问题并组合结果,显著降低统计复杂度。(i)在类监督微调场景下,模型仅需2^O(√(log T))量级的标注数据即可学习,远低于直接模拟所需的Ω(T);(ii)在类可验证奖励强化学习场景下,对预训练模型的要求从覆盖完整长度T降至仅需覆盖较短块长B≪T,条件要求指数级减弱。

原文摘要 · Abstract (English)

Compositional generalization, the ability to solve complex problems by combining solutions to simpler sub-problems, is a fundamental capability of both natural and artificial intelligence, and a key mechanism underlying chain-of-thought reasoning. However, the theoretical underpinnings of compositional generalization remain poorly understood: when and why does decomposing a problem into parts yield more efficient learning than solving it directly? We study this question through the canonical problem of learning to simulate semiautomata (predicting the outcome of $T$ steps of sequential computation), a model that captures state tracking, regular language recognition, and modular arithmetic. We show that an autocurriculum-based approach building on Part I of this series, recursively decomposing longer sequences into shorter sub-problems, learning to solve them, and composing the solutions, achieves dramatically better statistical complexity than direct methods. (i) For a setting inspired by supervised fine-tuning (SFT) where the learner receives interactive feedback on intermediate states of the computation, curriculum facilitates learning from only $2^{\mathcal{O}(\sqrt{\log T})}$ tokens of supervision; i.e., subpolynomial in the sequence length $T$, overcoming the $Ω(T)$ token barrier required by direct simulation. (ii) For a setting inspired by reinforcement learning with verifiable rewards (RLVR), where the learner improves a pre-trained reference model using an outcome verifier, we show that curriculum reduces the requirement on the reference model from coverage at the full sequence length $T$ to coverage at a shorter block length $B \ll T$, an exponentially weaker condition.

组合泛化课程学习推理能力高效学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。