提出低维隐空间接口，让反向传播并行化，提速训练不丢精度。
LBI: Parallel Scan Backpropagation via Latent Bounded Interfaces
- 用低维隐空间接口限制跨区域通信，降低反向传播计算开销。
- 在4种模型上保持0.16~0.35交叉熵误差，接口维度仅需16。
- 适合大规模分布式训练，显著减少设备间反向通信量。
反向传播在深度上天然串行，形成$O(K)$深度依赖链，阻碍并行训练。尽管并行扫描方法理论上可将深度降至$O("log K$)，但因需对整个隐藏状态进行全秩$d imes d$雅可比矩阵组合，导致计算成本高达$O(d^3)$，难以适用于现代架构。本文提出隐空间有界接口（LBI），通过将区域间通信限制在低维隐接口 $m_k \in \mathbb{R}^r$（$r \ll d$）内，使伴随递推变为对$r \times r$雅可比矩阵的后缀扫描，将每步组合成本从$O(d^3)$降至$O(r^3)$，并在有界接口模型下保持精确梯度。实验表明，LBI在四种架构（Mamba-2、Mamba-3、Transformer、Mamba--Transformer混合模型）中维持模型质量，参数量为47–61M；当接口维度$r=16$时，训练误差仅比密集基线高出0.16–0.35交叉熵。该框架为区域并行训练提供算法基础，将跨设备反向通信压缩为一次对$K$个固定大小矩阵的扫描，每次约56 KB。
原文摘要 · Abstract (English)
Backpropagation is inherently sequential across depth, creating an $O(K)$-deep dependency chain that bottlenecks parallel training. While parallel-scan formulations theoretically reduce this depth to $O(\log K)$, they are computationally prohibitive for modern architectures due to the $O(d^3)$ cost of composing full-rank $d\times d$ Jacobians over the entire hidden state. We introduce Latent Bounded Interfaces (LBI), an algorithmic formulation that makes scan-based backpropagation tractable by restricting inter-region communication to a low-dimensional latent interface, $ m_k \in \mathbb{R}^{r}$, where $r \ll d$. This reduces the adjoint recursion to a suffix scan over $r \times r$ Jacobians, cutting per-combine cost from $O(d^3)$ to $O(r^3)$ while preserving exact gradients under the bounded-interface model. We demonstrate that LBI maintains model quality across four architectures (Mamba-2, Mamba-3, Transformer, and a Mamba--Transformer hybrid) at 47--61M block parameters. Interfaces of dimension $r=16$ suffice to preserve training quality within 0.16--0.35 cross entropy of dense baselines. The resulting framework provides an algorithmic foundation for region-parallel training, reducing cross-device backward communication to a single scan over $K$ fixed-size matrices, of approximately 56 KB for our experimental configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。