用随机标签学习量化模型复杂度，发现正则化未必提升泛化能力。
Learned Random Label Predictions as a Neural Network Complexity Metric
- 设计多头网络并行学习真实与随机标签，评估模型复杂度。
- 随机标签学习能降低样本记忆，但不改善测试准确率。
- 挑战传统认知：复杂度降低未必带来更好泛化效果。
我们实证研究在监督学习中同时学习随机生成标签对深度神经网络过拟合、模型复杂度和泛化能力的影响。为此，我们提出一种多头网络架构，作为标准CNN的扩展。受公平性算法启发，该方法可实现随机标签的‘遗忘’，防止网络记忆单个样本。基于Rademacher复杂度，我们首先将该方法用作复杂度度量，分析常见正则化技术的效果，并挑战了卷积神经网络中特征提取与分类的传统理解。其次，我们提出一种新正则项，有效抑制样本记忆。然而，与经典统计学习理论预测相反，我们未观察到泛化性能的提升。
原文摘要 · Abstract (English)
We empirically investigate the impact of learning randomly generated labels in parallel to class labels in supervised learning on memorization, model complexity, and generalization in deep neural networks. To this end, we introduce a multi-head network architecture as an extension of standard CNN architectures. Inspired by methods used in fair AI, our approach allows for the unlearning of random labels, preventing the network from memorizing individual samples. Based on the concept of Rademacher complexity, we first use our proposed method as a complexity metric to analyze the effects of common regularization techniques and challenge the traditional understanding of feature extraction and classification in CNNs. Second, we propose a novel regularizer that effectively reduces sample memorization. However, contrary to the predictions of classical statistical learning theory, we do not observe improvements in generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。