arXiv:2503.17760cs.CVcs.AI2025-03ICCV被引 9

用连续VAE改造离散分词器,训练更稳、效果更好。

CODA: Repurposing Continuous VAEs for Discrete Tokenization

  • 分离压缩与离散化,先用现成连续VAE再适配
  • 仅需1/6训练预算,代码本利用率100%、重建质量高
  • 适合想高效构建高质量视觉分词器的研究者

离散视觉分词器将图像转为词元序列,支持类语言模型的视觉生成。但该过程需同时完成信号压缩和离散化,传统方法联合训练常导致训练不稳定、代码本利用率低、重建质量差。本文提出CODA(连续到离散自适应)框架,将预训练的连续变分自编码器(VAE)解耦压缩与离散化任务,通过精心设计的离散化流程转化为离散分词器。相比从头训练,该方法聚焦离散化,确保训练稳定高效,并保留连续VAE的强视觉保真度。实验表明,相较于标准VQGAN,CODA仅需6倍少的训练预算,在ImageNet 256×256上实现8×和16×压缩率下100%代码本利用率,重建FID分别达0.43和1.34。

原文摘要 · Abstract (English)

Discrete visual tokenizers transform images into a sequence of tokens, enabling token-based visual generation akin to language models. However, this process is inherently challenging, as it requires both compressing visual signals into a compact representation and discretizing them into a fixed set of codes. Traditional discrete tokenizers typically learn the two tasks jointly, often leading to unstable training, low codebook utilization, and limited reconstruction quality. In this paper, we introduce \textbf{CODA}(\textbf{CO}ntinuous-to-\textbf{D}iscrete \textbf{A}daptation), a framework that decouples compression and discretization. Instead of training discrete tokenizers from scratch, CODA adapts off-the-shelf continuous VAEs -- already optimized for perceptual compression -- into discrete tokenizers via a carefully designed discretization process. By primarily focusing on discretization, CODA ensures stable and efficient training while retaining the strong visual fidelity of continuous VAEs. Empirically, with $\mathbf{6 \times}$ less training budget than standard VQGAN, our approach achieves a remarkable codebook utilization of 100% and notable reconstruction FID (rFID) of $\mathbf{0.43}$ and $\mathbf{1.34}$ for $8 \times$ and $16 \times$ compression on ImageNet 256$\times$ 256 benchmark.

视觉分词VAE离散化图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。