提出ResBM架构，实现128倍激活压缩，支持低带宽下高效训练。
ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism
- 在流水线边界引入可端到端训练的残差编码解码瓶颈模块
- 实现128倍激活压缩，收敛速度与标准模型相当
- 适合大规模分布式训练中通信带宽受限的场景
解锁大规模低带宽去中心化训练可利用未被开发的计算资源。在集中式环境中，大规模多节点训练主要依赖数据并行和流水线并行，这两项技术需超高带宽通信。尽管去中心化数据并行已有高效方法，流水线并行仍是主要挑战。近期工作如子空间模型（Subspace Models, SM）声称可达100倍激活压缩，但依赖复杂约束优化且偏离端到端训练。本文提出新方法：基于从零设计、适配低带宽通信环境的架构——残差瓶颈模型（Residual Bottleneck Model, ResBM）。该模型在流水线边界引入残差编码解码瓶颈模块，可作为模型参数端到端训练，并保留显式的低秩恒等路径。实验表明，ResBM实现128倍激活压缩，收敛速率无显著损失，且内存与计算开销几乎不变。
原文摘要 · Abstract (English)
Unlocking large-scale low-bandwidth decentralized training has the potential to utilize otherwise untapped compute resources. In centralized settings, large-scale multi-node training is primarily enabled by data and pipeline parallelism, two techniques that require ultra-high-bandwidth communication. While efficient methods now exist for decentralized data parallelism, pipeline parallelism remains the primary challenge. Recent efforts, such as Subspace Models (SM), have claimed up to 100x activation compression but rely on complex constrained optimization and diverge from true end-to-end training. In this paper, we propose a different approach, based on an architecture designed from the ground up to be native to low-bandwidth communication environments while still applicable to any standard transformer-based architecture. We call this architecture the Residual Bottleneck Model or ResBM, it introduces a residual encoder-decoder bottleneck module across pipeline boundaries that can be trained end-to-end as part of the model's parameters while preserving an explicit low-rank identity path. We show that ResBMs achieve state-of-the-art 128x activation compression without significant loss in convergence rates and without significant memory or compute overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。