通过自适应压缩降低分片学习通信开销，提升带宽受限下的训练效率。
NSC-SL: A Bandwidth-Aware Neural Subspace Compression for Communication-Efficient Split Learning
- 根据实时带宽动态调整低秩近似阶数，实现自适应压缩。
- 采用残差反馈的正交迭代法，有效减少压缩带来的信息损失。
- 适合资源受限场景下的高效分布式模型训练，尤其适用于移动设备。
神经网络规模不断扩大，给分布式机器学习带来了挑战，尤其是在通信资源有限的情况下。尽管分片学习（Split Learning, SL）通过将模型层分布在客户端与服务器之间减轻了客户端计算负担，但频繁传输中间激活值和梯度仍带来巨大通信开销。为此，我们提出NSC-SL，一种面向带宽的自适应压缩算法，用于实现高效的分片学习。NSC-SL首先根据奇异值分布动态确定低秩近似的最优阶数，以适应实时带宽约束；随后，通过带有残差反馈的交替正交迭代进行误差补偿的张量分解，有效降低截断损失。协同机制使NSC-SL在保持语义丰富信息的同时实现高压缩比，保障模型收敛。大量实验表明，该方法性能优越。
原文摘要 · Abstract (English)
The expanding scale of neural networks poses a major challenge for distributed machine learning, particularly under limited communication resources. While split learning (SL) alleviates client computational burden by distributing model layers between clients and server, it incurs substantial communication overhead from frequent transmission of intermediate activations and gradients. To tackle this issue, we propose NSC-SL, a bandwidth-aware adaptive compression algorithm for communication-efficient SL. NSC-SL first dynamically determines the optimal rank of low-rank approximation based on the singular value distribution for adapting real-time bandwidth constraints. Then, NSC-SL performs error-compensated tensor factorization using alternating orthogonal iteration with residual feedback, effectively minimizing truncation loss. The collaborative mechanisms enable NSC-SL to achieve high compression ratios while preserving semantic-rich information essential for convergence. Extensive experiments demonstrate the superb performance of NSC-SL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。