让大模型在推理时并行思考，提升效率与效果。
Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space
- 通过克隆或删除残差流，在隐空间实现并行计算。
- 用一半训练预算，困惑度和零样本表现超越传统模型。
- 适合希望在预训练阶段就学会自适应计算的研究者。
当前基于transformer的推理扩展方法依赖显式的思维链标记，仅能在推理阶段使用且无法在预训练中应用。本文提出Thoughtbubbles，一种原生支持隐空间并行自适应计算的transformer变体，通过学习克隆或删除残差流，使需更多计算的路径形成‘气泡’。该行为在仅使用语言建模损失的预训练中学习。使用一半训练预算，Thoughtbubbles在150M至1.9B模型规模上均优于标准解码器语言模型及非自适应并行计算方法，在困惑度和零样本评估上表现更优。其在GSM8K任务上以一半的令牌预算达到竞争力结果。该隐式机制使模型可在预训练阶段即开始学习自适应计算，为训练与推理阶段统一计算扩展铺平道路。
原文摘要 · Abstract (English)
Current approaches for scaling inference-time compute in transformers train them to emit explicit chain-of-thought tokens before producing an answer. While these methods are powerful, they are limited because they cannot be applied during pretraining and rely solely on serially-generated, natural-language verbalization. In this work, we propose Thoughtbubbles, a transformer variant that natively performs parallel adaptive computation in latent space by learning to fork or delete residual streams. Thus, tokens requiring more computation can form a "bubble" of cloned residuals in the middle of the network. Crucially, this behavior is learned during pretraining with only language modeling loss. Using half of the training budget, Thoughtbubbles outperforms the perplexity and zero-shot evals of both standard decoder LMs and those using non-adaptive parallel computation approaches. These results hold across model sizes from 150M to 1.9B. Thoughtbubbles achieves competitive GSM8K results using half of the baseline's token budget. The implicit nature of our method enables models to begin learning adaptive computation at pretraining time, paving the way to unified train-time and test-time scaling behaviors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。