CoT让Transformer用更少数据学会推理，关键在注意力变稀疏。
From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
- 用思维链引导模型建立稀疏的输入依赖关系，降低学习难度。
- 有CoT时只需多项式样本即可学好任务，无CoT则需指数级样本。
- 揭示了思维链提升效率的核心机制：注意力结构变稀疏且可解释。
思维链（Chain-of-thought, CoT）显著提升大语言模型的推理能力。尽管现有理论多归因于表达能力和计算容量的增强，我们指出，在大模型阶段，表达力并非主要瓶颈——当前大模型仍会失败于简单任务。通过奇偶性学习任务设置，我们证明即使表征能力充足，引入CoT仍能大幅提升样本效率：有CoT时，Transformer可在多项式样本内完成学习；无CoT时，所需样本量为指数级。进一步发现，CoT通过在输入词元间引入稀疏的序列依赖，简化学习过程，并导致注意力分布稀疏且可解释。合成与真实世界实验验证了理论分析，确认注意力稀疏性是CoT带来性能提升的关键因素。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) significantly enhances the reasoning performance of large language models (LLM). While current theoretical studies often attribute this improvement to increased expressiveness and computational capacity, we argue that expressiveness is not the primary limitation in the LLM regime, as current large models will fail on simple tasks. Using a parity-learning setup, we demonstrate that CoT can substantially improve sample efficiency even when the representation power is sufficient. Specifically, with CoT, a transformer can learn the function within polynomial samples, whereas without CoT, the required sample size is exponential. Additionally, we show that CoT simplifies the learning process by introducing sparse sequential dependencies among input tokens, and leads to a sparse and interpretable attention. We validate our theoretical analysis with both synthetic and real-world experiments, confirming that sparsity in attention layers is a key factor of the improvement induced by CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。