logit正则化让分类器权重对齐最优方向，显著降低数据需求。
The Implicit Bias of Logit Regularization
- 在线性分类中，logit正则化使输出聚类到特定目标值
- 对高斯数据，可实现权重与Fisher判别分析完全对齐
- 实验显示样本量减半即可达到相同性能，且抗噪声能力强
logit正则化通过在logit空间添加凸惩罚项广泛应用于现代分类器，标签平滑是典型例子。尽管此类方法常提升校准性和泛化性能，其机制仍不明确。本文在线性分类框架下分析了一类通用的logit正则化器，证明其诱导出每个样本的logit向有限目标值聚类的隐式偏差。对于高斯数据或当logits充分聚类的情况，我们证明该聚类会使权重向量精确对齐Fisher线性判别（Fisher's Linear Discriminant）。通过一个简单的信号加噪声模型，我们发现这种转变具有显著影响：logit正则化将临界样本复杂度减半，并在小噪声极限下引发grokking现象，同时使泛化对噪声更加鲁棒。研究结果拓展了对标签平滑的理论理解，凸显了更广泛logit正则化方法的有效性。
原文摘要 · Abstract (English)
Logit regularization, the addition of a convex penalty directly in logit space, is widely used in modern classifiers, with label smoothing as a prominent example. While such methods often improve calibration and generalization, their mechanism remains under-explored. In this work, we analyze a general class of such logit regularizers in the context of linear classification, and demonstrate that they induce an implicit bias of logit clustering around finite per-sample targets. For Gaussian data, or whenever logits are sufficiently clustered, we prove that logit clustering drives the weight vector to align exactly with Fisher's Linear Discriminant. To demonstrate the consequences, we study a simple signal-plus-noise model in which this transition has dramatic effects: Logit regularization halves the critical sample complexity and induces grokking in the small-noise limit, while making generalization robust to noise. Our results extend the theoretical understanding of label smoothing and highlight the efficacy of a broader class of logit-regularization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。