通过约束数据子集间特征差异，有效抑制过拟合。
ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization
- 在同一批训练数据的随机子集间约束特征差异。
- 显著降低过拟合，提升准确率并减少验证损失。
- 无需调参、通用性强，适合各类模型和任务。
过参数化神经网络常导致训练与测试性能差距显著，即过拟合现象。为应对这一问题，研究者提出了多种针对不同任务和模型架构的正则化方法。本文从新视角看待过拟合：模型在不同独立同分布（i.i.d.）数据集上学习到不同的表示。基于此，我们提出一种自适应方法 ConsistentFeature，通过约束同一训练集随机子集间的特征差异来正则化模型。由于假设极少，该方法几乎适用于任何模型架构与任务。实验表明，该方法能有效缓解过拟合，对超参数不敏感且计算开销极低。尤其在模型已开始过拟合时，仍能显著抑制记忆偏差并促进正常收敛。即使无明显过拟合，该方法也持续提升精度并降低验证损失。
原文摘要 · Abstract (English)
Over-parameterized neural network models often lead to significant performance discrepancies between training and test sets, a phenomenon known as overfitting. To address this, researchers have proposed numerous regularization techniques tailored to various tasks and model architectures. In this paper, we introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets. Based on this viewpoint, we propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set. Due to minimal prior assumptions, this approach is applicable to almost any architecture and task. Our experiments show that it effectively reduces overfitting, with low sensitivity to hyperparameters and minimal computational cost. It demonstrates particularly strong memory suppression and promotes normal convergence, even when the model has already started to overfit. Even in the absence of significant overfitting, our method consistently improves accuracy and reduces validation loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。