通过分析优化景观,快速找到最佳任务组合。
Efficient Task Grouping Through Samplewise Optimisation Landscape Analysis
- 基于样本级优化景观分析推断任务相似性,免去训练开销。
- 在8个数据集上实现五倍加速,性能与顶尖方法相当。
- 适合需要高效多任务学习的场景,尤其资源受限时。
共享训练方法如多任务学习(MTL)和基于梯度的元学习广泛应用于各类机器学习任务,但常因负迁移导致特定任务性能下降。尽管已有优化技术缓解预选任务组的此问题,但识别最优任务组合——即任务分组——仍研究不足且计算成本高昂,因任务组合呈指数增长,需大量训练与评估周期。本文提出一种高效任务分组框架,显著降低现有方法的计算负担。该框架通过样本级优化景观分析推断任务间相似性,无需传统方法中所需的共享模型训练。获得相似性后,采用图聚类算法定位近似最优任务组,为原本NP-hard的问题提供高效可行解。在8个不同数据集上的实证评估表明,该框架相较先前最先进方法实现五倍提速,同时保持相当性能,验证了其卓越的效率与有效性。
原文摘要 · Abstract (English)
Shared training approaches, such as multi-task learning (MTL) and gradient-based meta-learning, are widely used in various machine learning applications, but they often suffer from negative transfer, leading to performance degradation in specific tasks. While several optimisation techniques have been developed to mitigate this issue for pre-selected task cohorts, identifying optimal task combinations for joint learning - known as task grouping - remains underexplored and computationally challenging due to the exponential growth in task combinations and the need for extensive training and evaluation cycles. This paper introduces an efficient task grouping framework designed to reduce these overwhelming computational demands of the existing methods. The proposed framework infers pairwise task similarities through a sample-wise optimisation landscape analysis, eliminating the need for the shared model training required to infer task similarities in existing methods. With task similarities acquired, a graph-based clustering algorithm is employed to pinpoint near-optimal task groups, providing an approximate yet efficient and effective solution to the originally NP-hard problem. Empirical assessments conducted on 8 different datasets highlight the effectiveness of the proposed framework, revealing a five-fold speed enhancement compared to previous state-of-the-art methods. Moreover, the framework consistently demonstrates comparable performance, confirming its remarkable efficiency and effectiveness in task grouping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。