GCond通过梯度累积与自适应仲裁，高效解决大模型多任务学习中的梯度冲突问题。
GCond: Gradient Conflict Resolution via Accumulation-based Stabilization for Large-Scale Multi-Task Learning
- 基于梯度累积和自适应机制，改进PCGrad原理以降低计算开销。
- 在图像与医学影像数据集上，性能优于基线与现有方法，提升4.5%准确率。
- 适用于小型到大型模型，兼容AdamW等现代优化器，适合实际部署。
多任务学习中，梯度冲突是主要挑战。现有方法如PCGrad、CAGrad和GradNorm在原始实现中计算成本高，限制了其在Transformer等大模型中的应用。本文提出梯度导引（GCond），基于PCGrad思想，结合梯度累积与自适应仲裁机制。我们在ImageNet 1K和头颈部CT扫描数据集上，使用MobileNetV3-Small和ConvNeXt架构进行自监督多任务学习评估，对比基线线性组合及前沿梯度冲突解决方法。分析了经典与随机模式的GCond。其随机模式实现两倍计算速度提升，同时保持优化质量，在两个数据集上均低于其他方法的L1与SSIM损失，并在异质场景下展现更优泛化能力：相较基线，提升ImageNet Top-1准确率4.5%，并防止医疗诊断中的置信度过拟合。GCond具备高可扩展性，成功应用于紧凑模型（MobileNetV3-Small、ConvNeXt-tiny）与大模型（ConvNeXtV2-Base），且兼容AdamW、Lion/LARS等现代优化器，为多任务学习中的梯度冲突提供高效可扩展的解决方案。
原文摘要 · Abstract (English)
In multi-task learning (MTL), gradient conflict poses a significant challenge. Effective methods for addressing this problem, including PCGrad, CAGrad, and GradNorm, in their original implementations are computationally demanding, which significantly limits their application in modern large models such as transformers. We propose Gradient Conductor (GCond), a method that builds upon PCGrad principles by combining them with gradient accumulation and an adaptive arbitration mechanism. We evaluated GCond on self-supervised multi-task learning tasks using MobileNetV3-Small and ConvNeXt architectures on the ImageNet 1K dataset and a combined head and neck CT scan dataset, comparing the proposed method against baseline linear combinations and state-of-the-art gradient conflict resolution methods. The classical and stochastic approaches of GCond were analyzed. The stochastic mode of GCond achieved a two-fold computational speedup while maintaining optimization quality, and demonstrated superior performance across all evaluated metrics, achieving lower L1 and SSIM losses compared to other methods on both datasets, and demonstrating superior generalization in heterogeneous scenarios: GCond improved ImageNet Top-1 Accuracy by 4.5% over baselines and prevented confidence overfitting in medical diagnosis tasks. GCond exhibited high scalability, being successfully applied to both compact models: MobileNetV3-Small and ConvNeXt-tiny; and large architecture ConvNeXtV2-Base. It also showed compatibility with modern optimizers such as AdamW and Lion/LARS. Therefore, GCond offers a scalable and efficient solution to the problem of gradient conflicts in multi-task learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。