约束类均值与协方差,让深度线性判别分析更稳定且可解释。
Simplex Deep Linear Discriminant Analysis
- 固定类均值为正单纯形顶点,协方差为球形,仅优化先验与方差。
- 在多个数据集上达到与Softmax相当的准确率,且潜空间聚类清晰分离。
- 适合需要可解释潜空间结构的分类任务,如可视化与模型调试。
我们从似然角度重新审视深度线性判别分析(Deep LDA)。经典LDA是具有线性决策边界的简单高斯模型,但将LDA头接在神经编码器后,如何通过最大似然估计(MLE)训练这一深层分类器成为问题。我们首先发现,联合学习未约束的Deep LDA模型参数会导致退化解:部分类别簇重叠甚至坍缩,分类性能下降。批量矩重估也无法消除此失败模式。为此,我们提出一种约束型Deep LDA:将类均值固定为潜空间中正单纯形的顶点,共享协方差设为球形,仅学习先验和一个方差参数。在此几何约束下,MLE变得稳定,生成清晰分离的类别簇。在图像(Fashion-MNIST、CIFAR-10、CIFAR-100)与文本(AG News、CLINC150)数据集上,该模型达到与Softmax基线相当的准确率,同时提供直观可解释的潜空间几何结构,二维投影中明显可见。
原文摘要 · Abstract (English)
We revisit Deep Linear Discriminant Analysis (Deep LDA) from a likelihood-based perspective. While classical LDA is a simple Gaussian model with linear decision boundaries, attaching an LDA head to a neural encoder raises the question of how to train the resulting deep classifier by maximum likelihood estimation (MLE). We first show that end-to-end MLE training of an unconstrained Deep LDA model ignores discrimination: when both the LDA parameters and the encoder parameters are learned jointly, the likelihood admits a degenerate solution in which some of the class clusters may heavily overlap or even collapse, and classification performance deteriorates. Batchwise moment re-estimation of the LDA parameters does not remove this failure mode. We then propose a constrained Deep LDA formulation that fixes the class means to the vertices of a regular simplex in the latent space and restricts the shared covariance to be spherical, leaving only the priors and a single variance parameter to be learned along with the encoder. Under these geometric constraints, MLE becomes stable and yields well-separated class clusters in the latent space. On images (Fashion-MNIST, CIFAR-10, CIFAR-100) and texts (AG News, CLINC150), the resulting Deep LDA models achieve accuracy competitive with softmax baselines while offering a simple, interpretable latent geometry that is clearly visible in two-dimensional projections.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。