arXiv:2411.15876cs.LGcs.AI2024-11

动态加权子模型,让深度学习更抗过拟合。

DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learning

  • 按验证集表现动态分配子模型权重,智能融合预测结果。
  • 在多个数据集上提升泛化能力,显著降低过拟合风险。
  • 适合数据少、噪声大或需强化鲁棒性的模型训练场景。

过拟合仍是深度学习中的重大挑战,常由数据异常值、噪声和训练数据有限引发。此前提出的 Divide2Conquer(D2C)方法将训练数据划分为多个子集,对每个子集独立训练相同模型,从而学习更一致的模式并减少个别异常值与噪声的影响。然而,标准 D2C 的聚合策略通常对所有子模型等权处理或依赖固定启发式规则(如数据量大小),可能未能充分利用各子模型在泛化能力上的差异。为此,本文提出动态不确定性感知的 Divide2Conquer(DUA-D2C),通过共享验证集上模型的表现(包括准确率与预测不确定性)动态调整子模型的贡献权重。该智能聚合机制使中心模型优先采纳更具泛化性与置信度的子模型输出,更有效地缓解过拟合。在跨多个领域的基准数据集上的实证评估表明,DUA-D2C 显著提升了模型泛化性能。分析涵盖决策边界、损失曲线等指标,验证其有效性。即使在已有正则化方法基础上应用,仍能进一步改善性能,证明 DUA-D2C 是一种理论扎实且高效的抗过拟合方案。代码已公开于:https://github.com/Saiful185/DUA-D2C。

原文摘要 · Abstract (English)

Overfitting remains a significant challenge in deep learning, often arising from data outliers, noise, and limited training data. To address this, the Divide2Conquer (D2C) method was previously proposed, which partitions training data into multiple subsets and trains identical models independently on each. This strategy enables learning more consistent patterns while minimizing the influence of individual outliers and noise. However, D2C's standard aggregation typically treats all subset models equally or based on fixed heuristics (like data size), potentially underutilizing information about their varying generalization capabilities. Building upon this foundation, we introduce Dynamic Uncertainty-Aware Divide2Conquer (DUA-D2C), an advanced technique that refines the aggregation process. DUA-D2C dynamically weights the contributions of subset models based on their performance on a shared validation set, considering both accuracy and prediction uncertainty. This intelligent aggregation allows the central model to preferentially learn from subsets yielding more generalizable and confident edge models, thereby more effectively combating overfitting. Empirical evaluations on benchmark datasets spanning multiple domains demonstrate that DUA-D2C significantly improves generalization. Our analysis includes evaluations of decision boundaries, loss curves, and other performance metrics, highlighting the effectiveness of DUA-D2C. This study demonstrates that DUA-D2C improves generalization performance even when applied on top of other regularization methods, establishing it as a theoretically grounded and effective approach to combating overfitting in modern deep learning. Our codes are publicly available at: https://github.com/Saiful185/DUA-D2C.

过拟合集成学习动态加权泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。