提出一种高效平衡多任务学习的新方法，计算开销仅为常数级。
LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control
- 从双层优化视角设计损失差异控制机制
- 在多个数据集上同时提升准确率与训练效率
- 适合大规模多任务场景，尤其关注计算效率的开发者
多任务学习（MTL）因能同时学习多个任务而被广泛采用。现有梯度调控方法虽比简单标量化方法更均衡，但通常需 $\mathcal{O}(K)$ 的时间和内存开销，其中 $K$ 为任务数。本文提出 LDC-MTL，一种基于双层优化框架的简化且可扩展的损失差异控制方法。该方法包含两个关键组件：(i) 精细控制损失差异的双层公式，(ii) 仅需 $\mathcal{O}(1)$ 时间和内存的可扩展一阶双层算法。理论上，我们证明 LDC-MTL 在温和条件下，不仅能收敛至双层问题的驻点，还能达到所有 $K$ 个损失函数的 $ε$-精确帕累托驻点。在多种多任务数据集上的大量实验表明，LDC-MTL 在准确率和效率方面均表现更优。
原文摘要 · Abstract (English)
Multi-task learning (MTL) has been widely adopted for its ability to simultaneously learn multiple tasks. While existing gradient manipulation methods often yield more balanced solutions than simple scalarization-based approaches, they typically incur a significant computational overhead of $\mathcal{O}(K)$ in both time and memory, where $K$ is the number of tasks. In this paper, we propose LDC-MTL, a simple and scalable loss discrepancy control approach for MTL, formulated from a bilevel optimization perspective. Our method incorporates two key components: (i) a bilevel formulation for fine-grained loss discrepancy control, and (ii) a scalable first-order bilevel algorithm that requires only $\mathcal{O}(1)$ time and memory. Theoretically, we prove that LDC-MTL guarantees convergence not only to a stationary point of the bilevel problem with loss discrepancy control but also to an $ε$-accurate Pareto stationary point for all $K$ loss functions under mild conditions. Extensive experiments on diverse multi-task datasets demonstrate the superior performance of LDC-MTL in both accuracy and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。