对比统一损失与专用优化器在多任务学习中的表现
Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning
- 采用大规模实验评估最新专用优化器在复杂多任务问题上的效果
- 固定权重策略表现接近专用优化器，且统一损失在部分场景下表现相当
- 揭示统一损失与专用优化器性能相近的机制，为超参选择提供依据
专用多任务优化器（SMTOs）通过解决冲突梯度和梯度范数差异等问题，在多任务学习中实现任务间的平衡。然而，近期研究质疑其必要性，认为以往结果受不良超参数调优和缺乏正则化影响，同等权重训练也能取得竞争性效果。本文通过在更复杂的多任务问题上对若干最新SMTO方法进行广泛实证评估，澄清该现象。结果表明，SMTOs相比统一损失仍具优势；同时，固定权重策略可达到与SMTOs相当的性能。此外，我们揭示了为何在某些情况下统一损失表现与SMTOs相似。源代码已公开于 https://github.com/Gabriel-SGama/UnitScal_vs_SMTOs。
原文摘要 · Abstract (English)
Specialized Multi-Task Optimizers (SMTOs) balance task learning in Multi-Task Learning by addressing issues like conflicting gradients and differing gradient norms, which hinder equal-weighted task training. However, recent critiques suggest that equally weighted tasks can achieve competitive results compared to SMTOs, arguing that previous SMTO results were influenced by poor hyperparameter optimization and lack of regularization. In this work, we evaluate these claims through an extensive empirical evaluation of SMTOs, including some of the latest methods, on more complex multi-task problems to clarify this behavior. Our findings indicate that SMTOs perform well compared to uniform loss and that fixed weights can achieve competitive performance compared to SMTOs. Furthermore, we demonstrate why uniform loss perform similarly to SMTOs in some instances. The source code is available at https://github.com/Gabriel-SGama/UnitScal_vs_SMTOs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。