让模型为每个词动态决定思考长度，提升效率与效果。
Pretraining with Token-Level Adaptive Latent Chain-of-Thought
- 每词生成可变长度的隐式思维链，难词多想，易词少想。
- 在相同计算量下，语言建模困惑度更低，下游任务更准。
- 无需额外训练，自动适配，适合资源受限场景。
通过增加每词计算量而非扩展参数或数据量，缓解大模型训练的高质量语料和通信成本瓶颈。本文提出在预训练中引入逐词自适应隐式思维链（adaptive latent CoT），模型在生成每个词前会生成一段长度可变的隐式推理轨迹——对难词分配更长轨迹，对易词则缩短甚至省略。该行为在单阶段通用文本预训练中自然涌现，通过逐词自适应停止机制，在训练与推理阶段均减少计算开销。基于 Llama 架构的实验表明，该方法在训练 FLOPs 更少的情况下，仍持续降低语言建模困惑度，并提升广泛下游任务的准确率，优于以往循环基线。
原文摘要 · Abstract (English)
Scaling large language models by increasing parameters and training data is increasingly constrained by limited high-quality corpora and rising communication costs. This work explores an alternative axis: increasing per-token computation without expanding parameters, by internalizing latent Chain-of-Thought (CoT) into pretraining. We propose Pretraining with Token-Level Adaptive Latent CoT (adaptive latent CoT), where the model generates a variable-length latent CoT trajectory before emitting each token -- allocating longer trajectories to difficult tokens and shorter (or even zero) trajectories to easy ones. Importantly, this behavior emerges naturally from one-stage pretraining on general text and reduces computation in both training and inference via token-wise adaptive halting. Experiments with Llama architectures show that adaptive latent CoT consistently improves language modeling perplexity and broad downstream accuracy, even with fewer training FLOPs than prior recurrent baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。