让大模型推理更快：一次生成两个思维步骤，不降精度。
SuperThoughts: Reasoning Tokens in Superposition
- 将连续两个思维令牌压缩为一个潜变量，用轻量模块一次解码两个。
- 推理速度翻倍，思维链长度减少20%~30%，准确率下降仅1-2点。
- 适合需要长推理链但又追求高效的应用，如数学求解和复杂问答。
长链思维（CoT）推理能提升大模型的问题求解能力，但因逐个生成标记而计算开销大。现有工作尝试在连续潜空间中进行推理以绕过离散标记生成，但常面临训练不稳定、难以扩展至复杂长程任务的问题，因缺乏监督信号。本文提出 SuperThoughts，将连续的两个 CoT 标记对压缩为单一潜在表示，并通过轻量级多标记预测（MTP）模块实现每步解码两个标记。该方法在训练时保留离散标记监督，推理时吞吐量翻倍。我们微调了 Qwen2.5-Math-1.5B-Instruct、Qwen2.5-Math-7B-Instruct、Qwen2.5-Math-14B-Instruct 模型，在 MATH500、AMC、OlympiadBench、GPQA-Diamond 数据集上评估。采用基于置信度的自适应机制，不确定时回退到标准解码，使思维链长度减少约20%~30%，同时保持高准确率，多数任务仅出现1-2分准确率下降。
原文摘要 · Abstract (English)
Long Chain-of-Thought (CoT) reasoning improves LLM problem-solving but is computationally expensive due to sequential token generation. While recent works explore reasoning in continuous latent spaces to bypass discrete token generation, they often struggle with training stability and fail to scale to complex, long-horizon tasks due to lack of supervision signal. We propose SuperThoughts, which compresses pairs of consecutive CoT tokens into single latent representations and decodes two tokens per step via a lightweight Multi-Token Prediction (MTP) module. This preserves discrete token supervision at training time while doubling throughput at inference time. We finetune Qwen2.5-Math-1.5B-Instruct, Qwen2.5-Math-7B-Instruct, Qwen2.5-Math-14B-Instruct, and evaluate on MATH500, AMC, OlympiadBench, and GPQA-Diamond. With a confidence-based adaptive mechanism that falls back to standard decoding when uncertain, SuperThoughts achieves $\sim$20--30\% CoT length reduction while maintaining accuracy with minimal degradation (1-2 points accuracy drop on most tasks).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。