提出渐进量化方法，解决向量量化过早离散化问题。
Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization
- 通过渐进式软硬量化，让编码器先学习数据流形再逐步离散化
- 在ImageNet和蛋白质结构任务上显著提升重建与生成性能
- 适合需要高质量向量令牌化的多模态模型研究者
向量量化（VQ）已成为多模态大语言模型和扩散生成的基石。然而，现有VQ范式存在根本性矛盾：在编码器尚未捕捉数据流形之前就强制离散化。我们称此为过早离散化。为此，提出渐进量化（ProVQ），将量化硬度的动态变化作为训练中的关键维度。通过将量化过程设计为从连续潜在空间平滑过渡到离散空间的课程学习，ProVQ有效引导码本对充分展开的数据流形进行建模。大量实验证明ProVQ在多种模态下均具广泛有效性。在ImageNet-1K和ImageNet-100基准上实现更优的重建与生成性能，凸显其对生成建模的提升作用。此外，ProVQ在复杂生物序列建模中表现卓越，在StrutTokenBench基准上为蛋白质结构令牌化设立新性能上限。
原文摘要 · Abstract (English)
Vector Quantization (VQ) has become the cornerstone of tokenization for many multimodal Large Language Models and diffusion synthesis. However, existing VQ paradigms suffer from a fundamental conflict: they enforce discretization before the encoder has captured the underlying data manifold. We term this phenomenon Premature Discretization. To resolve this, we propose Progressive Quantization (ProVQ), which incorporates the dynamics of quantization hardness as a fundamental yet previously overlooked axis in VQ training. By treating quantization as a curriculum that smoothly anneals from a continuous latent space to a discrete one, ProVQ effectively guides the codebook toward the well-expanded manifolds. Extensive experimental results demonstrate the broad effectiveness of ProVQ across diverse modalities. We report improved reconstruction and generative performance on the ImageNet-1K and ImageNet-100 benchmarks, highlighting the ProVQ's boost for generative modeling. Furthermore, ProVQ proves highly effective for modeling complex biological sequences, establishing a new performance ceiling for protein structure tokenization on the StrutTokenBench leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。