提出MOON方法，在矩阵几何下优化多任务学习，提升训练效率和性能。
MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning
- 在矩阵几何下进行梯度操作，利用谱-核范数优化方向。
- 理论证明收敛速度达O(T⁻¹/²)（确定性）和O(T⁻¹/⁴)（随机梯度）。
- 适用于Transformer等具矩阵结构的模型，适合追求高效多任务训练的研究者。
多目标优化（MOO）通过梯度调节在多任务学习中有效缓解任务冲突。然而，现有方法将模型参数展平为向量，在欧氏几何下进行梯度操作，忽略了现代架构（如Transformer）中的矩阵结构。本文指出，在欧氏空间中执行梯度操作并不总能获得矩阵几何下的最速下降方向，可能限制优化效率。基于矩阵参数最速下降理论，我们提出MOON（Multi-Objective OrthoNormalized Updates），在谱-核范数几何下进行梯度操纵，并使用正交化后的操纵梯度更新参数。理论上，对于光滑非凸目标，我们在确定性设置下建立了平均帕累托平稳性收敛率O(T⁻¹/²)，在随机梯度下为O(T⁻¹/⁴)。大量实验表明，MOON在多个基准上持续提升优化效率与最终多任务性能。代码已开源：https://github.com/KunlinLyu/MOON。
原文摘要 · Abstract (English)
Multi-objective optimization (MOO) has demonstrated significant success in multi-task learning by mitigating task conflicts through gradient manipulation. However, most existing methods flatten model parameters into vectors and perform gradient manipulation under Euclidean geometry, thereby overlooking the matrix structure prevalent in modern architectures such as Transformers. In this paper, we show that gradient manipulation in Euclidean space does not generally yield the steepest descent direction under matrix geometry, potentially limiting optimization efficiency. Drawing from the theory of steepest descent for matrix-valued parameters, we propose MOON (Multi-Objective OrthoNormalized Updates), which performs gradient manipulation under spectral--nuclear norm geometry and uses the orthonormalized manipulated gradient for parameter updates. Theoretically, for smooth non-convex objectives, we establish convergence of the averaged Pareto-stationarity measure at rates of $\mathcal{O}(T^{-1/2})$ in the deterministic setting and $\mathcal{O}(T^{-1/4})$ under stochastic gradients. Empirical results across various benchmarks show that MOON consistently improves both optimization efficiency and final multi-task performance. Our code is available at https://github.com/KunlinLyu/MOON.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。