提出可变码本量化,让图像生成逐步提升精度,突破传统固定码本瓶颈。
Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation

- 码本大小随序列位置递增,从最小2开始逐步增大
- 在ImageNet上将生成质量指标gFID从27.98降至1.71
- 无需额外训练技巧,自然形成粗到细的语义层次
多数离散视觉分词器采用统一码本设计,即序列中所有位置共享相同码本。研究者试图通过扩大码本规模K来提升重建性能,但这种固定码本设计存在信息论根本限制。我们发现,训练集各位置条件熵沿序列迅速下降,在ImageNet上当K=16384时,仅前2个位置仍具不确定性,其余254个位置近乎确定性,形成记忆难题。我们称此为熵悬崖现象,并给出公式:t* = ⌈log₂N / log₂K⌉。该现象在语言中不显著,因其天然结构维持较低有效熵。为此,我们提出可变码本大小量化(VCQ),码本大小K_t沿序列单调增长,从K_min=2到K_max,保持损失函数、参数量和AR训练流程不变。使用基础自回归Transformer与标准下一个词预测,基线版VCQ使ImageNet 256×256上的gFID(无CFG)从27.98降至14.80;放大后达gFID 1.71,仅用684M自回归参数,无需语义正则化或因果对齐等额外技术。极小码本(K_min=2)带来的极端信息瓶颈自然诱导出粗到细语义层次:仅前10个令牌的线性探测在ImageNet上达到43.8% top-1准确率,优于统一码本的27.1%。结果表明,关键不仅在于总容量,更在于容量的分布与组织方式。
原文摘要 · Abstract (English)
Most discrete visual tokenizers rely on a default design: every position in the sequence shares the same codebook. Researchers try to scale the codebook size $K$ to get better reconstruction performance. Such a constant-codebook design hits a fundamental information-theoretic limit. We observe that the per-position conditional entropy of the training set decays so quickly along the sequence that, after a few positions, the conditional distribution becomes essentially deterministic. On ImageNet with $K=16384$, this happens within only 2 out of 256 positions, turning the remaining 254 into a memorization problem. We call this phenomenon the Entropy Cliff and formalize it with a simple expression: $t^{*} = \lceil \log_2 N / \log_2 K \rceil$. Interestingly, this phenomenon is not observed in language, as its natural structure keeps the effective entropy per position well below the codebook capacity. To address this, we propose Variable Codebook Size Quantization (VCQ), where the codebook size $K_t$ grows monotonically along the sequence from $K_{\min}=2$ to $K_{\max}$, leaving the loss function, parameter count, and AR training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, a base version of VCQ reduces gFID w/o CFG from 27.98 to 14.80 on ImageNet $256\times256$ over the baseline. Scaled up, it reaches gFID 1.71 with 684M autoregressive parameters, without any extra training techniques such as semantic regularization or causal alignment. The extreme information bottleneck at $K_{\min}=2$ naturally induces a coarse-to-fine semantic hierarchy: a linear probe on only the first 10 tokens reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks. Ultimately, these results show that what matters is not only the total capacity of the codebook, but also how that capacity is distributed and organized.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。