通过子空间混合压缩激活值，实现低带宽下的高效上下文并行训练。
Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training
- 用可学习的子空间混合动态压缩激活输出，降低通信量。
- 在300Mbps网络上训练超10万词长上下文模型，压缩率超95%。
- 适合分布式低带宽场景，对大模型训练有实用价值。
在预训练语言模型时扩展上下文窗口能增强生成能力。现有方法将输入序列分块，在多设备间广播并逐块计算注意力，导致显著通信开销。虽然在高速集群中可行，但在低带宽分布式环境中不切实际。本文提出一种通信高效的上下文并行压缩方法，在去中心化设置下实现超过95%的压缩率，几乎无额外开销且不损失收敛性。核心思路是利用激活输出的内在低秩结构，通过高效重参数化动态将其约束到学习得到的子空间混合。实验表明，可在300Mbps网络上将数十亿参数模型扩展至超过10万词的上下文长度，其墙钟收敛速度与使用100Gbps互连的集中式模型相当。
原文摘要 · Abstract (English)
Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for communication-efficient context parallelism in decentralized settings, achieving a remarkable compression rate of over 95\% with negligible overhead and no loss in convergence. Our key insight is to exploit the intrinsic low-rank structure of activation outputs by dynamically constraining them to learned mixtures of subspaces via efficient reparameterizations. We demonstrate scaling billion-parameter decentralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。