arXiv:2604.06689cs.LGstat.ML2026-04

用生成式原理改进交叉熵,让少样本分类更准更可靠。

Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification

  • 将生成模型思想融入判别网络,不改架构直接替换损失函数。
  • 在小样本和类别不平衡场景下,准确率显著高于传统交叉熵。
  • 提升概率校准度与分布外检测能力,适合数据稀缺场景。

交叉熵是监督分类的标准训练损失,但在标签稀缺时样本效率有限。现有方法多从数据侧入手,如增强、合成或迁移学习,很少重新审视训练目标本身。本文提出生成式交叉熵(GenCE),作为交叉熵的即插即用替代方案,将生成学习原理引入标准判别网络,无需修改架构或训练独立密度模型。GenCE基于类条件似然的贝叶斯重写,在小批量近似下,将每个样本的softmax分数归一化为批次内同类预测的相对值,实现同类别样本间的训练信号耦合。我们扩展了严格得分规则框架以涵盖此类非局部损失,并证明在温和完备性条件下,GenCE的总体风险仅在真实后验处被最小化。在三个数据集上,两种架构下,无论平衡小样本还是类别不平衡情形,GenCE均优于交叉熵及其他常用损失,同时产生更优的概率校准结果与更强的分布外检测能力。

原文摘要 · Abstract (English)

Cross-entropy (CE) is the default training loss for supervised classification, but its sample efficiency is limited when labels are scarce. Existing remedies primarily act on the data side, via augmentation, synthesis, or transfer from pretrained models; the training objective itself is rarely revisited. We revisit it here. Drawing on the classical observation that generative classifiers reach their asymptotic error with fewer samples than discriminative ones, we propose Generative Cross-Entropy (GenCE), a drop-in replacement for CE that introduces a generative learning principle into a standard discriminative network without altering the architecture or fitting a separate density model. GenCE follows from a Bayesian rewrite of the class-conditional likelihood and, in the mini-batch approximation, reduces to normalizing each sample's softmax score against the model's predictions on the batch, coupling the training signal across examples sharing a class. We extend the proper-scoring-rule framework to such non-local losses and prove that GenCE is strictly proper under a mild completeness condition: its population risk is uniquely minimized at the true posterior. Across three datasets, on two architectures and in both balanced small-data and class-imbalanced regimes, GenCE outperforms CE and other widely used losses, while also producing better-calibrated probabilities and stronger out-of-distribution detection.

分类少样本损失函数概率校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。