改进Adam优化器在多目标学习中的表现,解决权重与几何失配问题。
MAdam: Metric-Aware Multi-Objective Adam

- 提出MAdam,通过偏好感知的曲率预处理来修正优化方向。
- 在多任务学习等场景中,显著提升各类多目标求解器性能。
- 无需修改现有框架,可直接替换Adam使用,适合多目标学习研究者。
多目标优化(MOO)是许多机器学习问题的核心,但当前主流的损失平衡、梯度平衡和基于帕累托的方法几乎都依赖Adam优化器。我们发现这种耦合导致两个系统性偏差:一是权重不匹配——Adam的二阶矩分母将时变偏好向量与梯度统计混杂,使偏好被历史平均化,不同帕累托权衡趋于近似均匀混合;二是几何不匹配——Adam的自适应度量扭曲了MOO求解器假设的欧氏几何,使原本对齐的目标产生表观冲突。为此,我们提出MAdam(Metric-Aware Multi-Objective Adam),一个无需修改求解器和优化器的即插即用封装。MAdam通过偏好条件化的标量目标曲率对协调方向进行预处理,使在该白化输入下Adam的二阶矩退化为单位矩阵,从而实现由偏好条件度量驱动的真实更新。在多任务学习、帕累托前沿恢复、物理信息神经网络及医学影像等多个任务中,MAdam在所有求解器家族上均一致优于标准Adam。
原文摘要 · Abstract (English)
Multi-objective optimization (MOO) underlies many machine learning problems, yet MOO solvers across the loss-balancing, gradient-balancing, and Pareto-based families almost universally hand their reconciled directions to Adam~\cite{kingma2015adam}. We show this coupling introduces two systematic gaps between the solver's intent and the optimizer's execution. The first is a \emph{weighting mismatch}: Adam's second-moment denominator entangles the time-varying preference vector with gradient statistics, marginalizing the preference into a history average and collapsing distinct Pareto trade-offs toward a near-uniform mixture. The second is a \emph{geometric mismatch}: Adam's adaptive metric distorts the Euclidean geometry MOO solvers assume, turning aligned objectives into apparent conflicts. To resolve both jointly, we introduce \textbf{MAdam} (Metric-Aware Multi-Objective Adam), a drop-in wrapper that leaves both solver and optimizer unchanged. MAdam preconditions the reconciled direction by the preference-conditioned curvature of the scalarized objective; on this whitened input, Adam's second moment collapses to identity, so the realized update is governed by the preference-conditioned metric. Across multi-task learning, Pareto-front recovery, physics-informed neural networks, and medical imaging, MAdam consistently improves over Adam for every solver family.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。