用熵识别隐变量模型中的活跃与休眠状态，不依赖高斯先验。
Entropy-Based Characterisation of the Polarised Regime in Latent Variable Models
- 基于均值表示的熵构建信息论分类方法，摆脱对高斯先验的依赖。
- 实验验证在多种变分模型中能稳定识别极化结构，熵可区分活跃与混合维度。
- 发现休眠维度经归一化后仍可提升下游任务性能，提示崩溃本质是尺度问题。
变分自编码器（VAEs）常表现出隐变量的极化现象，即隐变量分为活跃、休眠和混合三类。现有活跃维度判别标准依赖高斯先验，限制了其在变分模型及特定先验下的适用性。本文提出一种基于均值表示熵的信息论分类方法，理论上揭示该熵通过熵-方差界与KL散度最小化耦合，并关联到Bonheme的活跃/休眠条件。同时指出：仅靠均值熵无法可靠区分活跃与混合维度，需结合方差信息。实证上，在β-VAE、可辨识VAE、最小体积自编码器及L2正则化自编码器中，该熵准则能一致恢复极化结构。最后发现，经恰当归一化后，休眠维度仍能带来微小但稳定的下游任务性能提升，表明崩溃更多是尺度问题而非信息完全丢失。
原文摘要 · Abstract (English)
Variational Autoencoders (VAEs) often exhibit a polarised regime in which latent variables separate into active, passive, and mixed subsets. Existing criteria for identifying active dimensions depend on a Gaussian prior, limiting their applicability to variational models and specific priors. We propose a simple information-theoretic classification of the polarised regime based on the entropy of the mean representation. We show theoretically how this entropy couples to KL minimisation through entropy--variance bounds, and we relate the resulting criterion to Bonheme's active/passive conditions. We also clarify a key limitation: entropy of the mean alone cannot reliably distinguish active from mixed dimensions without additional signals from the variance representation. Empirically, we evaluate the entropy criterion on $β$-VAEs, identifiable VAEs, Least-Volume Autoencoders, and L2-regularised autoencoders, and find that it consistently recovers a polarised regime when such a regime is present across the model classes studied. Finally, we show that passive dimensions can yield small but consistent improvements on downstream tasks when latent codes are appropriately normalised, suggesting that collapse is often a matter of scale rather than absolute information removal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。