数据多样性通过改变权重谱分布，隐式正则化模型。
Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?
- 用随机矩阵理论分析数据多样性对权重空间的影响。
- 多样性使权重谱分布趋近于丢弃法，优于权重衰减。
- 提出新指标比较真实与合成数据的多样性收益。
数据增强通过引入输入数据多样性，长期被用于提升深度学习模型的鲁棒性与泛化能力，实际效果与丢弃法、权重衰减等正则化策略相当。然而，数据多样性如何促进模型性能的内在机制仍不明确。本文基于随机矩阵理论，通过谱分析对比使用数据增强、丢弃法和权重衰减训练的模型，发现增加数据多样性会以类似正则化的方式改变权重谱分布，其模式更接近丢弃法而非权重衰减。基于此，我们提出一个量化指标，用于解释并比较传统数据增强与合成数据带来的多样性效益。
原文摘要 · Abstract (English)
Data augmentation that introduces diversity into the input data has long been used in training deep learning models. It has demonstrated benefits in improving robustness and generalization, practically aligning well with other regularization strategies such as dropout and weight decay. However, the underlying mechanism of how diverse training data contributes to model improvements remains unknown. In this paper, we investigate the impact of data diversity on the weight space of deep neural networks using Random Matrix Theory. Through spectral analysis and comparing models trained with data augmentation, dropout, and weight decay, we reveal that increasing data diversity alters the weight spectral distribution similarly to other regularization techniques, while displaying a pattern more closely aligned with dropout than with weight decay. Building on these insights, we propose a metric to explain and compare the benefits of diversity introduced by traditional data augmentations and those achieved through synthetic data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。