破解向量量化代码本坍塌难题，提出新方法提升模型利用率
Beyond Stationarity: Rethinking Codebook Collapse in Vector Quantization
- 发现编码器非平稳更新是代码本坍塌根源
- 两种新方法使代码本使用率接近100%，重建质量更优
- 适合研究生成模型与向量量化机制的学者
向量量化（VQ）是VQ-VAE、VQ-GAN和潜在扩散模型等生成框架的核心。然而，代码本坍塌问题长期存在：训练中大量码向量未被使用。本文揭示编码器更新的非平稳性是根本原因——当编码器漂移时，未被选中的码向量无法获得更新，逐渐失效。为此提出两种新方法：非平稳向量量化（NSVQ），通过核函数规则将编码器漂移传播至未选码向量；基于Transformer的向量量化（TransVQ），采用轻量映射自适应变换整个码本，同时保持收敛至k-means解。在CelebA-HQ数据集上的实验表明，两者均实现近乎完整的码本利用和更优重建质量，为未来VQ生成模型提供可扩展的理论基础。代码已开源。
原文摘要 · Abstract (English)
Vector Quantization (VQ) underpins many modern generative frameworks such as VQ-VAE, VQ-GAN, and latent diffusion models. Yet, it suffers from the persistent problem of codebook collapse, where a large fraction of code vectors remains unused during training. This work provides a new theoretical explanation by identifying the nonstationary nature of encoder updates as the fundamental cause of this phenomenon. We show that as the encoder drifts, unselected code vectors fail to receive updates and gradually become inactive. To address this, we propose two new methods: Non-Stationary Vector Quantization (NSVQ), which propagates encoder drift to non-selected codes through a kernel-based rule, and Transformer-based Vector Quantization (TransVQ), which employs a lightweight mapping to adaptively transform the entire codebook while preserving convergence to the k-means solution. Experiments on the CelebA-HQ dataset demonstrate that both methods achieve near-complete codebook utilization and superior reconstruction quality compared to baseline VQ variants, providing a principled and scalable foundation for future VQ-based generative models. The code is available at: https://github.com/CAIR- LAB- WFUSM/NSVQ-TransVQ.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。