通过拆分梯度流提升训练吞吐，解决梯度消失问题。
DeInfoReg: A Decoupled Learning Framework for Better Training Throughput
- 将长梯度流拆成多段短流，缓解梯度消失。
- 在多GPU上实现模型并行，训练吞吐显著提升。
- 抗噪性强，适合大规模分布式训练场景。
本文提出一种新型解耦监督学习方法——信息正则化解耦学习（DeInfoReg），将长梯度流分解为多个短梯度流，有效缓解梯度消失问题。结合流水线策略，该方法支持跨多张GPU的模型并行，显著提升训练吞吐量。我们在多种任务和数据集上与标准反向传播及其他梯度流分解方法对比，实验表明，DeInfoReg在性能和抗噪声能力上均优于传统反向传播模型，并能高效利用并行计算资源。代码已开源：https://github.com/ianzih/Decoupled-Supervised-Learning-for-Information-Regularization/
原文摘要 · Abstract (English)
This paper introduces Decoupled Supervised Learning with Information Regularization (DeInfoReg), a novel approach that transforms a long gradient flow into multiple shorter ones, thereby mitigating the vanishing gradient problem. Integrating a pipeline strategy, DeInfoReg enables model parallelization across multiple GPUs, significantly improving training throughput. We compare our proposed method with standard backpropagation and other gradient flow decomposition techniques. Extensive experiments on diverse tasks and datasets demonstrate that DeInfoReg achieves superior performance and better noise resistance than traditional BP models and efficiently utilizes parallel computing resources. The code for reproducibility is available at: https://github.com/ianzih/Decoupled-Supervised-Learning-for-Information-Regularization/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。