通过梯度正则化提升多任务共享表示的通用性,增强模型性能。
Learning Representation for Multitask learning through Self Supervised Auxiliary learning
- 用虚拟任务预测器的梯度范数约束,优化共享编码器表示
- 在多个基准数据集上显著提升多任务预测准确率
- 方法简单高效,可无缝集成到现有多任务学习框架
多任务学习通过同时学习多个相关任务提升算法效率与效果。在硬参数共享架构中,共享编码器生成的数据表示被传递给任务特定的预测器,因此共享编码器生成优质表示至关重要。然而,如何提升共享编码器表示质量仍是未解问题。为此,本文提出一种新方法——虚拟梯度范数正则化(Dummy Gradient norm Regularization, DGR),通过减小损失函数对虚拟任务特定预测器的梯度范数,提升共享编码器表示的通用性。在多个多任务学习基准数据集上的实验表明,DGR有效提升了共享表示质量,从而获得更优的多任务预测性能。该方法在多种分类器上均优于现有方法,且因其结构简洁而具备良好的计算效率,可无缝融入现有多任务学习算法。
原文摘要 · Abstract (English)
Multi-task learning is a popular machine learning approach that enables simultaneous learning of multiple related tasks, improving algorithmic efficiency and effectiveness. In the hard parameter sharing approach, an encoder shared through multiple tasks generates data representations passed to task-specific predictors. Therefore, it is crucial to have a shared encoder that provides decent representations for every and each task. However, despite recent advances in multi-task learning, the question of how to improve the quality of representations generated by the shared encoder remains open. To address this gap, we propose a novel approach called Dummy Gradient norm Regularization that aims to improve the universality of the representations generated by the shared encoder. Specifically, the method decreases the norm of the gradient of the loss function with repect to dummy task-specific predictors to improve the universality of the shared encoder's representations. Through experiments on multiple multi-task learning benchmark datasets, we demonstrate that DGR effectively improves the quality of the shared representations, leading to better multi-task prediction performances. Applied to various classifiers, the shared representations generated by DGR also show superior performance compared to existing multi-task learning methods. Moreover, our approach takes advantage of computational efficiency due to its simplicity. The simplicity also allows us to seamlessly integrate DGR with the existing multi-task learning algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。