让大模型推理过程在连续空间进行，不依赖离散词元。
A Model with No Head and Many Thoughts
- 用轻量投影器替代解码头，在嵌入空间持续推理
- 所有k值下通过率均提升，单步计算量下降
- 适合追求高效推理与连续思维的场景
大型语言模型在每一步解码时通过大型词汇头投影隐藏状态，这一操作计算开销大，且迫使所有推理必须以离散词元表达。本文提出软潜思（Soft Latent Thinking），在推理阶段用轻量级投影器替代语言模型头，实现嵌入空间中的自回归展开，使推理步骤保持连续而非离散化。在DeepSeek-Qwen-1.5B和LLaMA-3.2-3B上的实验表明，该方法在所有pass@k指标上均有提升，同时降低链式思考的每步计算成本。其在所有软思维方法中达到最高的pass@32表现，证明有效推理可在无离散词元生成的情况下于连续空间完成。
原文摘要 · Abstract (English)
Large language models decode by projecting hidden states through a large vocabulary head at every step. This operation is computationally costly and forces all reasoning to be expressed in discrete tokens. We introduce Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector, enabling autoregressive rollout in embedding space where reasoning steps remain continuous rather than tokenized. Experiments on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B show that Soft Latent Thinking consistently improves pass@k across all k while reducing per-step compute during chain-of-thought. Our method achieves the highest pass@32 among all soft-thinking approaches, demonstrating that effective reasoning can be carried out in continuous space without discrete token generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。