让大模型在预训练时多思考一步，生成更优文本。
PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space
- 预训练阶段增加隐式思考步骤，用连续空间优化生成。
- 每字多一次思考，效果超过参数翻倍的标准模型。
- 适合追求高效高质生成的开发者和研究者。
Chain-of-Thought（CoT）通过测试时扩展生成步骤显著提升性能，启发我们思考：能否在预训练阶段也通过增加计算步骤来提升单个词元的生成质量？为此，我们提出一种新预训练方法——潜思预训练（PonderLM-2）。该方法让语言模型（LM）在生成每个实际词元前，先生成一个中间隐式思考（即当前位置的最后隐藏状态），再以此为输入预测下一个词元。这一额外计算步骤使模型能在无约束的连续空间中优化预测。实验表明，在相同推理成本下，每词元多生成一次隐式思考的模型，性能优于参数量翻倍的标准模型。例如，基于300B词元训练的PonderLM-2-Pythia-1.4B，在相同数据上表现显著超越未加思考的Pythia-2.8B，涵盖语言建模与多项通用下游任务。进一步增加每词元前的隐式思考次数（类比CoT链），模型性能持续提升。代码已开源。
原文摘要 · Abstract (English)
The remarkable success of Chain-of-Thought (CoT), which enhances performance by scaling generation steps at test-time, inspires us to ask: can we leverage a similar scaling of computational steps during pretraining to improve the generation of each individual token? To address this, we propose a novel pre-training methodology: Pretraining Language Models with Latent Thoughts (PonderLM-2). Our approach pretrains a language model (LM) to first generate an intermediate latent thought-the last hidden state of the current position-which is then used as input to predict the actual subsequent token. This additional computational step enables the LM to refine its prediction within unconstrained continuous space. Our experiments demonstrate that, at an identical inference cost, a LM that generates one additional latent thought per token outperforms a standard model with double the parameters. For instance, our PonderLM-2-Pythia-1.4B, pretrained on 300B tokens from the Pile, significantly surpasses the vanilla Pythia-2.8B trained on the same data on both language modeling and a range of general downstream tasks. Furthermore, increasing the number of latent thoughts generated before each actual token-forming a chain analogous to CoT-consistently improves the model's performance. The code is available at https://github.com/LUMIA-Group/PonderLM-2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。