arXiv:2602.17993cs.LGcs.AI2026-02

通过跨层连接增强模型推理能力,无需重训练即可显著提效。

Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers

  • 从高层数隐藏状态向低层传递多条残差路径,突破计算深度限制。
  • 在GSM8K等任务上准确率提升0.9%至超10%,Parity任务达100%正确。
  • 可无缝集成到预训练模型,适合需要高效推理优化的场景。

复杂问题如数学、逻辑或规划,人类通过一系列步骤逐步解决,前一步结果影响后一步。本文认为,Transformer的推理能力受限于任意潜在计算路径的最大步数。为此提出Turbo Connection(TurboConn)架构,通过将每个标记t的高层数隐藏状态的多个残差连接路由至标记t+1的低层,突破固定深度约束。对预训练大模型微调后,在GSM8K、Parity及多步算术任务上准确率提升0.9%至超过10%。实验表明,这些反向连接的密度至关重要:密集连接显著优于仅传递单一隐藏状态的稀疏方案。值得注意的是,该方法可直接嵌入预训练模型以突破任务瓶颈——原本微调后的Qwen-3-1.7B在Parity任务上仅达53.78%准确率,加入本结构后可提升至100%,且无需从头训练或复杂课程学习。结果有力证明计算路径深度是推理能力的关键因素,并提供一种不显著增加生成延迟的增强机制。

原文摘要 · Abstract (English)

Complex problems, whether in math, logic, or planning, are solved by humans through a sequence of steps where the result of one step informs the next. In this work, we adopt the perspective that the reasoning power of Transformers is fundamentally limited by a fixed maximum number of steps along any latent path of computation. To address this, we introduce Turbo Connection (TurboConn), a novel architecture that overcomes the fixed-depth constraint by routing multiple residual connections from the higher-layer hidden states of each token $t$ to the lower layers of token $t+1$. Fine-tuning pre-trained LLMs with our method not only yields accuracy gains of 0.9% to over 10% on benchmarks like GSM8K, Parity, and multi-step arithmetic, but also demonstrates that the density of these backward connections is critical; our dense interaction significantly outperforms "sparse" alternatives that only pass a single hidden state or vector. Notably, TurboConn can be integrated into pre-trained LLMs to overcome task-specific plateaus: while a fine-tuned Qwen-3-1.7B achieves only 53.78% on Parity, adding our architectural modification enables the model to reach 100% accuracy, all without the necessity to retrain the full model from scratch or sophisticated curriculum learning. Our results provide strong empirical evidence that the depth of the computational path is a key factor in reasoning ability, also offering a new mechanism to enhance LLMs without significantly affecting generation latency.

推理增强模型架构低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。