揭示梯度下降在多分类数据中的隐式偏好，统一了二分类与多分类的理论框架。
The Implicit Bias of Gradient Descent on Separable Multiclass Data
- 基于PERM损失框架，扩展指数尾部性质到多分类场景。
- 证明梯度下降会收敛到最大化类别间距的解，类似二分类结果。
- 适用于研究优化算法偏好问题的研究者，尤其关注多分类模型设计。
隐式偏差描述了优化算法在无显式正则化的情况下，倾向于选择简单估计器的现象，即使更复杂的模型具有相同的损失值。已有研究在二分类中基于损失函数满足指数尾部性质的假设建立了理论，但多分类分析仍存在明显空白，现有成果也仅限于交叉熵损失。本文采用周等（2024）提出的排列对称与相对间隔（PERM）损失框架，将指数尾部性质推广至多分类场景。该类损失不仅包含交叉熵，还涵盖其他损失形式。利用此框架，我们将Soudry等（2018）的隐式偏差结果拓展至多分类情形。此外，我们的证明方法与二分类情况高度相似，凸显了PERM框架在弥合二分类与多分类理论差距方面的强大能力。
原文摘要 · Abstract (English)
Implicit bias describes the phenomenon where optimization-based training algorithms, without explicit regularization, show a preference for simple estimators even when more complex estimators have equal objective values. Multiple works have developed the theory of implicit bias for binary classification under the assumption that the loss satisfies an exponential tail property. However, there is a noticeable gap in analysis for multiclass classification, with only a handful of results which themselves are restricted to the cross-entropy loss. In this work, we employ the framework of Permutation Equivariant and Relative Margin-based (PERM) losses [Wang and Scott, 2024] to introduce a multiclass extension of the exponential tail property. This class of losses includes not only cross-entropy but also other losses. Using this framework, we extend the implicit bias result of Soudry et al. [2018] to multiclass classification. Furthermore, our proof techniques closely mirror those of the binary case, thus illustrating the power of the PERM framework for bridging the binary-multiclass gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。