统一连续与离散视觉分词,实现高质量图像生成。
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging

- 通过分词合并构建语义桥梁,联合优化连续与离散分词器。
- 在相同分词量下,重建误差(rFID)显著低于主流VAE与VQ模型。
- 生成结果兼具语义组织性,适配自回归与扩散模型。
当前图像生成的视觉分词器分为两类:连续的变分自编码器(VAE)虽能高保真重建,但潜在表示密集且纠缠,难以实现语义控制;离散的向量量化(VQ)模型支持自回归生成,却面临梯度稀疏、训练不稳定与码本坍缩问题。本文提出MergeTok,一种在编码器-解码器架构中联合优化连续(VAE)与离散(VQ)分词器的统一方法,利用分词合并技术作为语义桥梁。编码过程中对相似分词进行聚类,建立结构先验,提供双重监督信号:(i) 在VAE分支中强制合并分词的语义对齐,使潜在空间更解耦、更具语义感知;(ii) 生成组内多样性与组间排他性的约束,稳定VQ训练。在ImageNet-256上,MergeTok表现出竞争力的重建与生成性能,在相同分词预算下,相比强基线模型,显著降低rFID,同时生成具有语义组织的分词表示,兼容自回归与扩散生成器。这表明单一架构可同时实现稳健的语义组织与生成友好型离散性。
原文摘要 · Abstract (English)
Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging token merging techniques as a semantic bridge. By clustering similar tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-token semantic alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets, while producing semantically-organized token representations compatible with both autoregressive and diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。