通过分步任务训练，模型能学会组合计算并零样本泛化。
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
- 用分步子任务引导模型学习复合算法
- 带子任务训练的模型在相同上下文长度下更鲁棒
- 不同训练策略导致模型采用不同计算方式
在上下文学习（ICL）中，通常通过均匀采样的输入输出对来学习函数。本文研究在上下文中引入组合子任务课程是否会影响Transformer模型的学习计算方式。设计了一个基于模块化指数的算法任务——由两个单指数子任务构成的双指数任务，并训练Transformer模型在上下文中学习该任务。对比两种训练方式：(a) 使用仅包含单指数子任务的上下文课程；(b) 直接在双指数任务上训练，无课程引导。结果表明，使用子任务课程训练的模型可对未见过的组合任务进行零样本推理，且在相同上下文长度下更具鲁棒性。我们分析了两种训练方式中任务与子任务的表征差异，发现模型会根据具体课程设计采用多样化的策略。
原文摘要 · Abstract (English)
In-context learning (ICL) research often considers learning a function in-context through a uniform sample of input-output pairs. Here, we investigate how presenting a compositional subtask curriculum in context may alter the computations a transformer learns. We design a compositional algorithmic task based on the modular exponential-a double exponential task composed of two single exponential subtasks and train transformer models to learn the task in-context. We compare (a) models trained using an in-context curriculum consisting of single exponential subtasks and, (b) models trained directly on the double exponential task without such a curriculum. We show that models trained with a subtask curriculum can perform zero-shot inference on unseen compositional tasks and are more robust given the same context length. We study how the task and subtasks are represented across the two training regimes. We find that the models employ diverse strategies modulated by the specific curriculum design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。