提出新型指数损失函数PENEX，提升神经网络在小样本下的泛化能力。
PENEX: AdaBoost-Inspired Neural Network Regularization
- 基于指数损失设计可梯度优化的正则化方法
- 在低数据场景下显著提高模型泛化性能
- 适合追求高效正则化的机器学习研究者
AdaBoost通过逐次拟合弱学习器来最小化指数损失，该损失对误分类样本惩罚更重，但实际中随着弱学习器数量增加仍具良好泛化性。本文提出一种理论严谨的新多分类指数损失形式PENEX，不同于传统形式，其可通过一阶优化方法有效训练神经网络。实验表明，PENEX能有效增大数据点的间隔，进而导出泛化界。在计算机视觉与自然语言处理任务中，该方法在低数据条件下显著提升神经网络泛化能力，计算成本与现有正则化方法相当，部分场景表现更优。结果表明，指数损失在超越AdaBoost应用方面具有潜力。
原文摘要 · Abstract (English)
AdaBoost sequentially fits so-called weak learners to minimize an exponential loss, which penalizes misclassified data points more severely than other loss functions like cross-entropy. Paradoxically, AdaBoost generalizes well in practice as the number of weak learners grows. In the present work, we introduce Penalized Exponential Loss (PENEX), a new formulation of the multi-class exponential loss that is theoretically grounded and, in contrast to the existing formulation, amenable to optimization via first-order methods, making it a practical objective for training neural networks. We demonstrate that PENEX effectively increases margins of data points, which can be translated into a generalization bound. Empirically, across computer vision and language tasks, PENEX improves neural network generalization in low-data regimes, matching and in some settings outperforming established regularizers at comparable computational cost. Our results highlight the potential of the exponential loss beyond its application in AdaBoost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。