提出可扩展的图像分块量化方法，解决代码本不稳定问题。
Scalable Image Tokenization with Index Backpropagation Quantization
- 用索引反向传播实现代码本与编码器联合优化，保持潜在空间一致。
- 首次实现2^18规模、256维高利用率代码本，支持大规模训练。
- 适用于需要高效视觉分块的生成模型，如自回归图像生成。
现有向量量化（VQ）方法在可扩展性方面表现不佳，主要源于训练过程中代码本部分更新导致的不稳定性。随着未激活代码与视觉特征之间的分布差距不断增大，代码本易发生坍缩。为此，我们提出索引反向传播量化（IBQ），一种用于联合优化所有代码本嵌入和视觉编码器的新方法。通过在编码特征与代码本间的一次性独热分布上应用直通估计器，所有代码均可微，且与视觉编码器保持一致的潜在空间。IBQ实现了视觉分块器的大规模训练，首次在2^18规模、256维高维度下实现高利用率代码本。ImageNet标准基准测试表明，IBQ在重建性能和自回归视觉生成任务中均表现优异，达到竞争性结果。代码与模型已公开于https://github.com/TencentARC/SEED-Voken。
原文摘要 · Abstract (English)
Existing vector quantization (VQ) methods struggle with scalability, largely attributed to the instability of the codebook that undergoes partial updates during training. The codebook is prone to collapse as utilization decreases, due to the progressively widening distribution gap between non-activated codes and visual features. To solve the problem, we propose Index Backpropagation Quantization (IBQ), a new VQ method for the joint optimization of all codebook embeddings and the visual encoder. Applying a straight-through estimator on the one-hot categorical distribution between the encoded feature and codebook, all codes are differentiable and maintain a consistent latent space with the visual encoder. IBQ enables scalable training of visual tokenizers and, for the first time, achieves a large-scale codebook ($2^{18}$) with high dimension ($256$) and high utilization. Experiments on the standard ImageNet benchmark demonstrate the scalability and superiority of IBQ, achieving competitive results on reconstruction and the application of autoregressive visual generation. The code and models are available at https://github.com/TencentARC/SEED-Voken.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。