提出可验证的判据,防止教师引导的VAE出现恒定隐变量崩溃。
A Testable Certificate for Constant Collapse in Teacher-Guided VAEs

- 通过教师分布与隐变量的互信息建立崩溃阈值。
- 实验表明训练保持在安全边界内,崩溃后重启可恢复证书。
- 适用于关注隐变量有效性的生成模型研究者。
变分自编码器中的后验崩溃常通过小KL项、强解码器或弱隐变量使用来诊断,但这些信号无法定义崩溃边界。本文研究输入无关的恒定崩溃这一具体失败模式,发现其存在精确阈值:对于任意非恒定教师分布𝑇(⋅∣𝑥),最优常数学生为数据集平均教师分布,其对齐代价即为教师互信息𝐼𝑇(𝑋;𝑇)。若严格仅依赖隐变量的原始见证(raw witness)对齐损失低于该值且带安全裕度,则该见证不可能为输入恒定。此关系将定性故障转为可度量问题。在CIFAR-100上基于种子搜索教师的实验显示,完整训练始终位于认证边界内;移除对齐驱动会使其进入恒定学生区间,而从崩溃检查点重启并启用对齐可恢复证书。Tiny-ImageNet-200固定目标实验中,三种独立搜索的教师均呈现相同预防-崩溃-恢复模式。标准VAE基线(包括保持重建质量或事後可预测性的方法)在原始证书下始终为负。该保证有意局限:仅认证匹配非恒定教师相对变化通过隐层路径,而非排除所有形式的后验崩溃。
原文摘要 · Abstract (English)
Posterior collapse in variational autoencoders is often diagnosed by its symptoms: a small KL term, a strong decoder, or weak use of the latent code. These signals are useful, but they do not define a collapse boundary. We study a concrete failure mode, input-independent constant collapse, and show that this case admits an exact threshold. For any fixed nonconstant teacher distribution \(T(\cdot\mid x)\), the best constant student is the dataset-average teacher distribution, and its alignment cost is the teacher mutual information \(I_T(X;T)\). Therefore, if a strictly latent-only raw witness achieves alignment loss below this value, with a safety margin, the witness cannot be constant in the input. This identity turns a qualitative failure mode into a measurable one. In CIFAR-100 experiments with per-seed teacher search, full training stays on the certified side of the boundary, removing alignment drives the raw witness into the constant-student regime, and restarting from a collapsed checkpoint with alignment enabled restores the certificate. Tiny-ImageNet-200 fixed-target runs show the same prevention--collapse--rescue pattern across three independently searched teachers. Standard VAE-style baselines, including methods that preserve reconstruction quality or post-hoc predictability, remain negative under the raw certificate. The guarantee is intentionally narrow: it certifies that the matched nonconstant teacher-relative variation passes through the latent pathway, rather than claiming that all forms of posterior collapse have been ruled out.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。