arXiv:2608.08888cs.AI2026-08被引 1

让Transformer在生成时能回传更多上下文信息,提升推理和生成质量。

Full-bandwidth transformer

论文配图:Full-bandwidth transformer
图 1 · 摘自论文原文
  • 通过潜变量反馈机制,将顶层隐藏状态回传到输入层。
  • 在400B token训练下,验证损失降低,数学与代码生成能力显著提升。
  • 无需额外计算开销,适合需长推理链的任务场景。

自回归Transformer在生成时沿两个方向计算:横向跨生成的词元,纵向通过模型深度。密集注意力使每个词元可广泛访问历史信息,但解码步骤间的纵向反馈通道仍较窄:仅采样的词元返回底部,顶层隐藏状态被丢弃。本文提出全带宽Transformer,引入潜变量反馈机制:每步解码时,将前一时刻顶层隐藏状态与采样词元嵌入通过门控线性单元融合,并作为下一步输入。该机制使非显式计算得以重新进入网络并获得新的深度预算,同时保持标准Transformer架构、键值缓存及语言建模目标。为避免训练中失去并行教师强制,采用调度多轮次目标,在预训练后期引入潜变量反馈,并混合少量深层反馈步骤以保证稳定性。我们训练了10亿参数的全带宽Transformer,最多达4000亿词元,结果表明潜变量反馈改善了验证损失、5样本语言建模评估、数学与编码生成以及指令微调表现。在每词元解码开销可忽略的前提下,其性能达到标准Transformer使用约1.5倍更多数据训练的效果,且能在相同或更优精度下生成更短推理路径。

原文摘要 · Abstract (English)

Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded. We introduce the \emph{full-bandwidth transformer}, which widens this channel with \emph{latent feedback}: at each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input. Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget, while preserving the standard transformer architecture, KV cache, and language-modeling objective. To train full-bandwidth transformers without losing parallel teacher forcing, we use a scheduled multi-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability. We train 1B-parameter full-bandwidth transformers up to 400B tokens and find that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead, full-bandwidth transformers match or approach standard transformers trained with roughly $1.5\times$ more tokens, and manage to produce shorter reasoning traces at equal or better accuracy.

Transformer生成优化推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。