构建合成数据集，系统研究思维链提示对语言模型的影响。
CoT-ICL Lab: A Synthetic Framework for Studying Chain-of-Thought Learning from In-Context Demonstrations
- 解耦因果结构与令牌处理函数，精细控制提示复杂度。
- 深度模型更易利用少量提示实现高准确率，浅层模型需更多示例。
- 限制处理函数多样性可提升模型对因果关系的学习能力。
我们提出 CoT-ICL Lab，一个生成合成分词数据集并系统研究语言模型中思维链（CoT）提示学习（ICL）的框架与方法。该框架通过解耦（1）链式令牌生成中的因果结构与（2）底层令牌处理函数，实现对上下文示例复杂度的细粒度控制。我们在最大700M参数的仅解码器变换器上训练模型，发现思维链能加速模型在不同规模下准确率向更高值跃迁。特别地，模型深度对利用有限提示至关重要；而更多示例则有助于浅层模型达到深层模型性能。此外，训练期间限制令牌处理函数多样性可促进通过ICL学习因果结构。我们还通过分析变换器嵌入与注意力图解释这些跃迁现象。总体而言，CoT-ICL Lab为理解提示学习与思维链提供了简洁而有力的实验平台。
原文摘要 · Abstract (English)
We introduce CoT-ICL Lab, a framework and methodology to generate synthetic tokenized datasets and systematically study chain-of-thought (CoT) in-context learning (ICL) in language models. CoT-ICL Lab allows fine grained control over the complexity of in-context examples by decoupling (1) the causal structure involved in chain token generation from (2) the underlying token processing functions. We train decoder-only transformers (up to 700M parameters) on these datasets and show that CoT accelerates the accuracy transition to higher values across model sizes. In particular, we find that model depth is crucial for leveraging CoT with limited in-context examples, while more examples help shallow models match deeper model performance. Additionally, limiting the diversity of token processing functions throughout training improves causal structure learning via ICL. We also interpret these transitions by analyzing transformer embeddings and attention maps. Overall, CoT-ICL Lab serves as a simple yet powerful testbed for theoretical and empirical insights into ICL and CoT in language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。