大模型能隐式完成多跳推理，但需海量数据支撑
Language models can learn implicit multi-hop reasoning, but only if they have lots of training data
- 用GPT2风格模型从零训练，直接学习多跳推理机制
- 每多一跳所需数据量呈指数级增长，层深线性增加
- 课程学习可缓解数据需求，但无法彻底消除
隐式推理是指语言模型在单次前向传播中解决多跳推理任务，无需显式思维链。我们使用从零训练的GPT2风格模型，在控制的k跳推理数据集（k=2,3,4）上研究该能力。结果表明，尽管模型确实能学习隐式k跳推理，但所需训练数据量随k呈指数增长，所需Transformer层数则随k线性增加。我们提出了理论解释，说明为何深度增长是必要的。此外，课程学习可部分缓解数据需求，但无法完全消除。
原文摘要 · Abstract (English)
Implicit reasoning is the ability of a language model to solve multi-hop reasoning tasks in a single forward pass, without chain of thought. We investigate this capability using GPT2-style language models trained from scratch on controlled $k$-hop reasoning datasets ($k = 2, 3, 4$). We show that while such models can indeed learn implicit $k$-hop reasoning, the required training data grows exponentially in $k$, and the required number of transformer layers grows linearly in $k$. We offer a theoretical explanation for why this depth growth is necessary. We further find that the data requirement can be mitigated, but not eliminated, through curriculum learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。