让Transformer在生成时复用隐藏状态，提升效率与效果。
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
- 用前一个词的高层隐藏态作为递归记忆，跨词跨层传递信息。
- 在相同计算量下，比标准Transformer更低的比特数和更优的性能。
- 适合追求低延迟、高效率推理的应用场景，如实时对话系统。
我们研究了潜变量循环Transformer（LRT），这是一种轻量级的自回归Transformer增强方法，通过复用前一令牌的高层源层隐藏状态作为下一令牌的递归记忆。由于该状态已在常规解码中计算，LRT引入了跨令牌、跨层的潜在路径，同时保留标准注意力机制、KV缓存接口及每生成一个令牌仅一次模型前向计算。为在不顺序展开完整序列的情况下预训练此递归机制，我们提出交错并行训练：先进行全序列初始化前向以构建共享缓冲区，随后对不相交的位置子集进行顺序精炼，并在每个子集内并行计算。这使得每个令牌在约2倍理想令牌计算量下获得具备递归记忆感知的监督。在1.3B和2.1B参数的nanochat风格骨干网络及广泛训练预算下，LRT在匹配有效计算量条件下提升了比特每字符（BPB）和核心性能（CORE）。此外，LRT优于两步前向的PonderLM-2，且在BPB上与三重循环Transformer相当，同时保持每令牌一次前向解码，仅比标准Transformer多9%延迟开销。
原文摘要 · Abstract (English)
We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token. Because this state is already computed during ordinary decoding, LRT introduces a cross-token, cross-layer latent pathway while preserving the standard attention mechanism, KV-cache interface, and one model forward per generated token. To pretrain this recurrence without sequentially unrolling the full sequence, we introduce interleaved parallel training: one full-sequence initialization forward constructs a shared buffer, followed by sequential refinement of disjoint position subsets with parallel computation within each subset. This provides every token with recurrent-memory-aware supervision at approximately 2x ideal token compute. Across 1.3B- and 2.1B-parameter nanochat-style backbones and a wide range of training budgets, LRT improves both BPB and CORE under matched effective compute. Additionally, LRT outperforms two-forward PonderLM-2 and matches a three-loop Transformer in BPB, while retaining one-forward-per-token decoding with 9% latency overhead over the standard Transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。