细粒度思维链训练能显著提升大模型在新任务上的泛化能力
Beyond In-Distribution Success: Scaling Curves of CoT Granularity for Language Model Generalization
- 用细粒度思维链数据训练，让模型学会真正推理
- 只需80%数据就达到传统问答模型的性能
- 适合需要跨任务泛化的AI系统研发人员
在分布外的新复合任务上实现泛化，对部署基于Transformer的语言模型至关重要。本文研究思维链（CoT）推理对模型泛化能力的影响。通过在多个复合任务上的受控实验发现：(1) 仅在问答数据上训练的模型虽在分布内表现接近完美，但其分布外性能仍会急剧下降，即使使用超过10000k的训练样本；(2) 思维链数据的粒度与泛化性能强相关，越细粒度的思维链数据越有助于泛化；(3) 思维链具有极强的数据效率，仅需较少（甚至80%）数据即可达到问答模型的性能水平。理论上，我们证明复合任务中问答数据天然存在捷径，与真实推理原则不符，而思维链迫使模型内化正确的依赖结构，从而实现更好泛化。此外，我们发现变压器的位置编码可通过强调长思维链序列中子任务条件的重复性来增强泛化。理论与实证分析共同表明，思维链推理是应对真实世界分布偏移下复合任务泛化问题的关键训练范式。
原文摘要 · Abstract (English)
Generalization to novel compound tasks under distribution shift is important for deploying transformer-based language models (LMs). This work investigates Chain-of-Thought (CoT) reasoning as a means to enhance OOD generalization. Through controlled experiments across several compound tasks, we reveal three key insights: (1) While QA-trained models achieve near-perfect in-distribution accuracy, their OOD performance degrades catastrophically, even with 10000k+ training examples; (2) the granularity of CoT data strongly correlates with generalization performance; finer-grained CoT data leads to better generalization; (3) CoT exhibits remarkable sample efficiency, matching QA performance with much less (even 80%) data. Theoretically, we demonstrate that compound tasks inherently permit shortcuts in Q-A data that misalign with true reasoning principles, while CoT forces internalization of valid dependency structures, and thus can achieve better generalization. Further, we show that transformer positional embeddings can amplify generalization by emphasizing subtask condition recurrence in long CoT sequences. Our combined theoretical and empirical analysis provides compelling evidence for CoT reasoning as a crucial training paradigm for enabling LM generalization under real-world distributional shifts for compound tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。