用新方法挖掘代码本中的相似性信息,让图像自回归生成更快更准。
Exploiting Discriminative Codebook Prior for Autoregressive Image Generation
- 用实例距离替代中心点距离,更好捕捉代码本特征空间的相似性。
- 在LlamaGen-B上训练速度提升42%,最终图像质量(FID/IS)也更好。
- 可直接接入现有模型,适合做自回归图像生成的研究者使用。
基于离散标记的自回归图像生成系统先通过代码本将图像转为标记序列,再在自回归框架下建模。尽管生成模型仅训练于标记值,但代码本中蕴含的丰富标记相似性先验未被利用。近期研究尝试用朴素k-means聚类挖掘此先验,以减小代码本规模并辅助训练,但我们发现k-means在代码本特征空间中表现不佳,存在标记空间失衡与质心距离不准确等问题。为此,本文提出判别性代码本先验提取器(DCPE),替代k-means以更有效地挖掘代码本中的标记相似性信息。DCPE摒弃依赖中心点的距离度量,改用更适配标记特征空间的实例距离;并采用凝聚合并策略,避免分割高密度区域、聚合低密度区域,缓解标记空间失衡问题。大量实验表明,DCPE即插即用,可无缝集成至现有基于代码本先验的范式中。借助所提取的判别性先验,DCPE在LlamaGen-B上使训练速度提升42%,并显著改善最终的FID和IS指标。
原文摘要 · Abstract (English)
Advanced discrete token-based autoregressive image generation systems first tokenize images into sequences of token indices with a codebook, and then model these sequences in an autoregressive paradigm. While autoregressive generative models are trained only on index values, the prior encoded in the codebook, which contains rich token similarity information, is not exploited. Recent studies have attempted to incorporate this prior by performing naive k-means clustering on the tokens, helping to facilitate the training of generative models with a reduced codebook. However, we reveal that k-means clustering performs poorly in the codebook feature space due to inherent issues, including token space disparity and centroid distance inaccuracy. In this work, we propose the Discriminative Codebook Prior Extractor (DCPE) as an alternative to k-means clustering for more effectively mining and utilizing the token similarity information embedded in the codebook. DCPE replaces the commonly used centroid-based distance, which is found to be unsuitable and inaccurate for the token feature space, with a more reasonable instance-based distance. Using an agglomerative merging technique, it further addresses the token space disparity issue by avoiding splitting high-density regions and aggregating low-density ones. Extensive experiments demonstrate that DCPE is plug-and-play and integrates seamlessly with existing codebook prior-based paradigms. With the discriminative prior extracted, DCPE accelerates the training of autoregressive models by 42% on LlamaGen-B and improves final FID and IS performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。