让大模型在推理时持续思考,显著提升复杂问题解答能力。
State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning

- 用非线性状态流实现连续潜空间推理,每层通过可学习融合机制传递状态。
- 训练时采用双阶段并行方法,突破递归依赖瓶颈,大幅降低计算开销。
- 仅用少量数据微调270亿参数模型,就在难题上超越更大模型,适合追求推理效率的场景。
当前Transformer在位置间丢弃丰富的潜空间残差流,每次新位置都重新构建推理上下文,浪费了潜在推理能力。SST V2通过解码器每一层的前馈网络驱动的非线性递归,在连续潜空间中实现参数高效的推理,利用可学习融合机制将潜状态横向传播至整个序列。该机制同样支持推理时对每个位置进行持续潜空间推敲,额外消耗浮点运算以探索抽象推理后再决定输出词元。采用两阶段并行训练流程解决递归的顺序依赖问题,实现高效训练。隐藏状态分析显示,状态流通过在连续潜空间中探索不同的语义盆地,使内容相关位置的状态转移进入显著不同的贝叶斯后验,直接影响后续位置的潜空间。通过可学习探测器发现,在首个生成词元位置,潜状态已能预测最终答案是否能在后续潜空间计算中存活。将SST与现有270亿参数主干联合训练,仅使用少量GSM8K数据,即在分布外的GPQA-Diamond上比微调基线提升15.15分,并减少基线46%的GSM8K剩余错误,证明推理提升源于架构机制而非规模或数据。在GPQA-Diamond上,该270亿参数SST模型精度超过多个更大规模的开源及专有系统,包括最大达其25倍的开源模型。
原文摘要 · Abstract (English)
Current transformers discard their rich latent residual stream between positions, reconstructing latent reasoning context at each new position and leaving potential reasoning capacity untapped. The State Stream Transformer (SST) V2 enables parameter-efficient reasoning in continuous latent space through an FFN-driven nonlinear recurrence at each decoder layer, where latent states are streamed horizontally across the full sequence via a learned blend. This same mechanism supports continuous latent deliberation per position at inference time, dedicating additional FLOPs to exploring abstract reasoning before committing to a token. A two-pass parallel training procedure resolves the sequential dependency of the recurrence to allow compute-efficient training. Hidden state analysis shows the state stream facilitates reasoning through exploration of distinct semantic basins in continuous latent space, where transitions at content-dependent positions move the model into a substantially different Bayesian posterior, directly influencing the latent space at future positions. We also find, via a learned probe, that at the first generated token position, the latent state already predicts whether the eventual answer will survive or break under additional latent computation for every subsequent position. Co-trained into an existing 27B backbone using only a small dataset of GSM8K examples, the SST delivers a +15.15 point gain over a fine-tuning-matched baseline on out-of-distribution GPQA-Diamond and cuts that same baseline's remaining GSM8K errors by 46%, together showing that the reasoning improvement is attributable to the architectural mechanism rather than scale or training data. On GPQA-Diamond, the resulting 27B SST also achieves higher accuracy than several larger open-weight and proprietary systems, including open-weight models up to 25 times larger.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。