解决向量量化中代码本崩溃问题，提升生成质量与代码本利用率。
NSVQ: Mitigating Codebook Collapse by Stabilizing Encoder Drift in Vector Quantization
- 通过动态嵌入损失和代码本替换，跟踪编码器漂移。
- 在128×128图像上，rFID从2.39降至2.10，代码本使用率达100%。
- 适合需要高稳定性和全代码本利用的生成模型训练场景。
向量量化是现代生成建模的核心，但大代码本VQ模型常出现代码本崩溃。本文识别出编码器漂移是关键原因：随着编码器移动潜在分布，稀疏更新的代码向量会滞后、失去分配，导致量化误差增大，并通过直通估计器形成反馈循环。为此提出NSVQ，一种非平稳感知的VQ训练策略，结合密集非平稳嵌入损失、代码本替换和分阶段编码器冻结。NSVQ先在早期训练中帮助代码本跟踪编码器漂移，再冻结编码器以在固定潜在几何下固化代码本，最后引入对抗性精修。ImageNet-1k实验显示，NSVQ在保持100%代码本利用率的同时提升了重建质量；在128×128分辨率下使用65,536个代码时，rFID由2.39降至2.10（对比SimVQ）。下游潜变量扩散实验也表明，NSVQ显著改善ImageNet生成的FID表现。
原文摘要 · Abstract (English)
Vector quantization is central to modern generative modeling pipelines, but large-codebook VQ models often suffer from codebook collapse. We identify encoder drift as a key driver of this failure: as the encoder moves the latent distribution, sparsely updated code vectors can lag behind, lose assignments, and increase quantization error, creating a feedback loop through the straight-through estimator. We propose NSVQ, a non-stationary-aware VQ training strategy that combines a dense non-stationary embedding loss, codebook replacement, and stage-wise encoder freezing. NSVQ first helps the codebook track encoder drift during early training, then freezes the encoder to consolidate the codebook under a fixed latent geometry, and finally reintroduces adversarial refinement. Experiments on ImageNet-1k show that NSVQ improves reconstruction quality while maintaining full codebook utilization. On ImageNet-1k at 128$\times$128 with 65,536 codes, NSVQ reduces rFID from 2.39 to 2.10 compared with SimVQ, while both methods maintain 100\% utilization. Additional latent diffusion experiments show that NSVQ also improves downstream ImageNet generation FID.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。