通过调节学习率增强模型对异常值的鲁棒性
Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate
- 用平滑损失函数动态降低学习率以抑制异常值影响
- SRL和SMAE在多个数据集上显著优于Huber等经典损失
- 适合处理含噪声数据的回归任务，支持GPU高效计算
大多数用于训练监督学习模型的真实世界数据集都包含噪声和异常值，导致预测误差较大。本文提出一种新方法，通过受异常值敏感的因子调节学习率来实现鲁棒性。该方法利用无限可导、严格凸或拟凸的交替损失函数，更接近绝对误差，从而实现学习率的降低。通过在多种基准和数据集上的对比实验，验证了本文提出的平方根损失（SRL）和光滑平均绝对误差（SMAE）损失的优越性能。本文还提出了两种新的鲁棒线性回归模型，并给出了高度向量化的参数更新公式，充分利用现代GPU实现随机与批量梯度下降的高效计算。
原文摘要 · Abstract (English)
Most real-world datasets used for training supervised learning models are contaminated with noisy data and outliers leading to large prediction errors. This paper proposes a new approach for achieving robustness where the learning rate is modulated by a factor that is sensitive to outliers. In this approach a reduction of the learning rate is shown to be achieved by using alternate loss functions that are infinitely differentiable, strictly convex or quasiconvex and more closely approximate the absolute error than Huber and log-cosh losses. A comparison of the performance of regression models trained with different loss functions on a wide variety of benchmarks and datasets is presented to demonstrate the superior performance of the Square Root Loss (SRL) and Smooth Mean Absolute Error (SMAE) losses proposed in this paper. Two new robust linear regression models are presented. Highly vectorized robust parameter update formulae that take advantage of modern GPUs for both stochastic and batch gradient descent are presented.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。