提出新方法缓解多任务学习中梯度冲突,提升训练稳定性。
Gradient Similarity Surgery in Multi-Task Deep Learning
- 基于梯度大小相似性动态调整优化方向,引导更合理的梯度聚合。
- 在合成任务和多任务基准上显著提升收敛速度与模型性能。
- 适合追求稳定训练的多任务深度学习研究者使用。
多任务学习(MTL)旨在单个模型中同时学习多个任务,捕捉任务间共享的高层次通用特征。在深度学习中,反向传播训练面临优化器设计挑战,尤其在多任务深度学习(MTDL)中,多任务可能产生相互冲突的梯度,导致不同损失函数无法协同收敛。当任务梯度幅度差异大或方向相反时,某些梯度会主导或干扰其他梯度,恶化训练过程。梯度手术方法通过调整整体梯度轨迹来显式处理冲突梯度。本文提出一种新型梯度手术方法——相似性感知动量梯度手术(SAM-GS),基于梯度幅度相似性度量,提供高效且可扩展的优化引导机制。SAM-GS结合梯度归一化与一阶动量调制,在合成问题和MTL基准上验证了其有效性。梯度幅度相似性在正则化MTDL中的梯度聚合、优化学习过程方面起关键作用。
原文摘要 · Abstract (English)
The multi-task learning ($MTL$) paradigm aims to simultaneously learn multiple tasks within a single model capturing higher-level, more general hidden patterns that are shared by the tasks. In deep learning, a significant challenge in the backpropagation training process is the design of advanced optimisers to improve the convergence speed and stability of the gradient descent learning rule. In particular, in multi-task deep learning ($MTDL$) the multitude of tasks may generate potentially conflicting gradients that would hinder the concurrent convergence of the diverse loss functions. This challenge arises when the gradients of the task objectives have either different magnitudes or opposite directions, causing one or a few to dominate or to interfere with each other, thus degrading the training process. Gradient surgery methods address the problem explicitly dealing with conflicting gradients by adjusting the overall gradient trajectory. This work introduces a novel gradient surgery method, the Similarity-Aware Momentum Gradient Surgery (SAM-GS), which provides an effective and scalable approach based on a gradient magnitude similarity measure to guide the optimisation process. The SAM-GS surgery adopts gradient equalisation and modulation of the first-order momentum. A series of experimental tests have shown the effectiveness of SAM-GS on synthetic problems and $MTL$ benchmarks. Gradient magnitude similarity plays a crucial role in regularising gradient aggregation in $MTDL$ for the optimisation of the learning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。