分组优化代码本,提升向量量化模型的重建效果与灵活性
Group-Wise Optimization for Self-Extensible Codebooks in Vector Quantized Models
- 将代码本分组,组内联合优化,组间独立更新
- 在图像重建任务中显著提升重构质量,代码本利用率更高
- 无需重新训练即可调整代码本大小,适合部署调优
向量量化变分自编码器(VQ-VAEs)通过重建任务实现自监督学习,用代码本中的最近向量表示连续向量。然而,代码本坍塌等问题仍存在。现有方法采用隐式静态代码本或整体联合优化,限制了代码本的学习能力,导致重建质量下降。本文提出Group-VQ,对代码本进行分组优化:每组独立更新,组内联合优化,提升了代码本利用与重建性能的权衡。此外,提出一种无需训练的代码本重采样方法,支持训练后动态调整代码本大小。在多种设置下的图像重建实验中,Group-VQ在重构指标上表现更优,且后训练采样方法实现了代码本规模的灵活调节。
原文摘要 · Abstract (English)
Vector Quantized Variational Autoencoders (VQ-VAEs) leverage self-supervised learning through reconstruction tasks to represent continuous vectors using the closest vectors in a codebook. However, issues such as codebook collapse persist in the VQ model. To address these issues, existing approaches employ implicit static codebooks or jointly optimize the entire codebook, but these methods constrain the codebook's learning capability, leading to reduced reconstruction quality. In this paper, we propose Group-VQ, which performs group-wise optimization on the codebook. Each group is optimized independently, with joint optimization performed within groups. This approach improves the trade-off between codebook utilization and reconstruction performance. Additionally, we introduce a training-free codebook resampling method, allowing post-training adjustment of the codebook size. In image reconstruction experiments under various settings, Group-VQ demonstrates improved performance on reconstruction metrics. And the post-training codebook sampling method achieves the desired flexibility in adjusting the codebook size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。