将SVD降秩与变分自编码结合,提升生成效果并防止后验崩溃。
Variational Rank Reduction Autoencoders
- 在潜在空间中引入截断SVD并结合KL散度正则化,实现稳定生成。
- 在MNIST、CelebA和CIFAR-10上FID得分优于VAE和RRAE。
- 对小规模合成数据也表现出强抗崩溃能力,适合生成任务研究者。
确定性秩减自编码器(RRAEs)通过截断SVD强制施加潜在空间正则化,增强模型能力,但其确定性特性使其生成性能不直观。而变分自编码器(VAEs)因学习概率潜在空间,在生成方面表现优异。本文提出变分秩减自编码器(VRRAEs),融合两者优势。结果表明,当对RRAEs的潜在空间进行谨慎采样,并进一步通过KL散度正则化(类似VAEs)时,VRRAEs在生成性能上超越RRAEs和VAEs。此外,SVD诱导的正则化不仅使VRRAEs生成效果优于VAEs,还降低了后验崩溃风险。实验包括一个小规模合成数据集,展示其对抗崩溃的鲁棒性,以及三个真实数据集:MNIST、CelebA和CIFAR-10。在多种随机生成与插值任务中,基于FID分数,VRRAEs均优于VAEs和RRAEs。代码基于JAX(Equinox)开源,地址为https://github.com/JadM133/RRAEs.git。
原文摘要 · Abstract (English)
Deterministic Rank Reduction Autoencoders (RRAEs) enforce by construction a regularization on the latent space by applying a truncated SVD. While this regularization makes Autoencoders more powerful, using them for generative purposes is counter-intuitive due to their deterministic nature. On the other hand, Variational Autoencoders (VAEs) are well known for their generative abilities by learning a probabilistic latent space. In this paper, we present Variational Rank Reduction Autoencoders (VRRAEs), a model that leverages the advantages of both RRAEs and VAEs. Our claims and results show that when carefully sampling the latent space of RRAEs and further regularizing with the Kullback-Leibler (KL) divergence (similarly to VAEs), VRRAEs outperform RRAEs and VAEs. Additionally, we show that the regularization induced by the SVD not only makes VRRAEs better generators than VAEs, but also reduces the possibility of posterior collapse. Our results include a synthetic dataset of a small size that showcases the robustness of VRRAEs against collapse, and three real-world datasets; the MNIST, CelebA, and CIFAR-10, over which VRRAEs are shown to outperform both VAEs and RRAEs on many random generation and interpolation tasks based on the FID score. We developed an open-source implementation of VRRAEs in JAX (Equinox), available at https://github.com/JadM133/RRAEs.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。