用自回归组合结构，让模型从少量任务学会解决大量新任务。
Task Generalization With AutoRegressive Compositional Structure: Can Learning From $D$ Tasks Generalize to $D^{T}$ Tasks?
- 将任务拆解为有限子任务的组合，构建可扩展的任务空间。
- 仅训练约D个任务，就能在理论上泛化到D^T个任务。
- 在奇偶校验、算术和翻译任务中验证了指数级泛化能力。
大型语言模型展现出惊人任务泛化能力，仅需少量示例即可解决未显式训练过的任务。本文从自回归组合结构视角探讨：当仅从少量任务学习时，能否泛化到庞大的任务家族？我们假设每个任务由T个操作构成，每个操作来自有限的D个子任务，共形成D^T个任务。理论上证明，仅需训练约˜O(D)个任务即可实现对所有D^T个任务的泛化。实验上，我们发现变压器模型通过上下文学习（ICL）与思维链（CoT）推理，在稀疏奇偶校验函数上实现了这种指数级任务泛化，并进一步在算术和翻译任务中验证了泛化效果。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit remarkable task generalization, solving tasks they were never explicitly trained on with only a few demonstrations. This raises a fundamental question: When can learning from a small set of tasks generalize to a large task family? In this paper, we investigate task generalization through the lens of autoregressive compositional structure, where each task is a composition of $T$ operations, and each operation is among a finite family of $D$ subtasks. This yields a total class of size $D^T$. We first show that generalization to all $D^T$ tasks is theoretically achievable by training on only $\widetilde{O}(D)$ tasks. Empirically, we demonstrate that Transformers achieve such exponential task generalization on sparse parity functions via In-context Learning (ICL) and chain-of-thought (CoT) reasoning. We further show generalization in arithmetic and translation, beyond parity functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。