提出统一鲁棒训练方法,有效应对数据中的异常值。
Outlier-Robust Training of Machine Learning Models
- 基于改进的对偶理论统一两类鲁棒损失设计。
- 提出自适应交替算法,无需复杂调参即可收敛到无异常值最优解。
- 在回归、分类和神经场景重建中验证效果,适合含异常值的数据集。
在存在异常值的情况下,机器学习模型的鲁棒训练受到广泛关注。目前主流方法是使用鲁棒损失函数以减轻异常值的影响。本文揭示了两个长期分离的研究方向:一个基于M估计(常见于机器人与计算机视觉),另一个基于风险最小化框架(常见于深度学习)。我们首先通过改进的Black-Rangarajan对偶性,建立统一视角,并定义了一类满足两类方法的鲁棒损失核σ。其次,基于该对偶性,提出自适应交替算法(AAA),通过迭代更新非鲁棒损失的权重进行训练,且将权重解释为内点概率,引入新颖参数更新规则,避免复杂调参。第三,我们证明,在任意异常值假设下(无需分布假设),使用鲁棒损失核σ可扩大收敛区域。实验表明,该算法在回归、分类及神经场景重建任务中均表现优异。代码已开源:https://github.com/MIT-SPARK/ORT。
原文摘要 · Abstract (English)
Robust training of machine learning models in the presence of outliers has garnered attention across various domains. The use of robust losses is a popular approach and is known to mitigate the impact of outliers. We bring to light two literatures that have diverged in their ways of designing robust losses: one using M-estimation, which is popular in robotics and computer vision, and another using a risk-minimization framework, which is popular in deep learning. We first show that a simple modification of the Black-Rangarajan duality provides a unifying view. The modified duality brings out a definition of a robust loss kernel $σ$ that is satisfied by robust losses in both the literatures. Secondly, using the modified duality, we propose an Adaptive Alternation Algorithm (AAA) for training machine learning models with outliers. The algorithm iteratively trains the model by using a weighted version of the non-robust loss, while updating the weights at each iteration. The algorithm is augmented with a novel parameter update rule by interpreting the weights as inlier probabilities, and obviates the need for complex parameter tuning. Thirdly, we investigate convergence of the adaptive alternation algorithm to outlier-free optima. Considering arbitrary outliers (i.e., with no distributional assumption on the outliers), we show that the use of robust loss kernels σ increases the region of convergence. We experimentally show the efficacy of our algorithm on regression, classification, and neural scene reconstruction problems. We release our implementation code: https://github.com/MIT-SPARK/ORT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。