arXiv:2511.20565cs.CV2025-11被引 5

用DINO模型构建高保真视觉分词器,兼顾语义与细节。

DINO-Tok: Adapting DINO for Visual Tokenizers

  • 基于冻结DINO编码器,融合浅层细节与深层语义生成完整潜在空间。
  • 连续与离散分词在ImageNet上分别达0.28和1.10 rFID,生成效果优越。
  • 提出主子空间量化,解决高维语义空间中的码本崩溃问题,适合生成模型研究者。

视觉生成的进展强调了潜在生成模型(LGM)的重要性,其关键在于有效的视觉分词器以连接像素与语义表示。然而,基于预训练视觉基础模型(VFMs)的分词器常难以在高维潜在空间中平衡语义丰富性与重建保真度。本文提出DINO-Tok,一个基于冻结DINO编码器的视觉分词器,支持连续自编码(DINO-Tok-AE)与离散向量量化(DINO-Tok-VQ)。通过整合浅层细粒度特征与深层全局语义的分层表示,构建信息完备的潜在空间,保留纹理细节并保持生成时的语义一致性。针对高维冻结语义空间中常见的信息稀释与码本崩溃问题,提出主子空间量化(DSQ),利用全局PCA筛选主成分并抑制噪声维度,稳定码本优化,提升重建与生成质量。在ImageNet 256x256上,DINO-Tok在连续自编码下达到0.28 rFID,离散VQ为1.10 rFID,扩散生成1.82 gFID,自回归生成2.44 gFID。结果表明,如DINO等预训练视觉模型可直接适配为下一代潜伏生成模型的高保真、语义对齐分词器。代码将公开于https://github.com/MKJia/DINO-Tok。

原文摘要 · Abstract (English)

Recent advances in visual generation have emphasized the importance of Latent Generative Models (LGMs), which critically depend on effective visual tokenizers to bridge pixels and semantic representations. However, tokenizers constructed on pre-trained vision foundation models (VFMs) often struggle to balance semantic richness and reconstruction fidelity in high-dimensional latent spaces. In this paper, we introduce DINO-Tok, a visual tokenizer built upon a frozen DINO encoder that supports both continuous autoencoding (DINO-Tok-AE) and discrete vector-quantization (DINO-Tok-VQ). By unifying hierarchical representations from both shallow fine-grained features and deep global semantics into an information-complete latent space, DINO-Tok preserves texture details while maintaining \textit{semantic consistency} for generation. We further investigate VQ in frozen semantic feature spaces of high dimensionality, where information dilution and codebook collapse frequently arise. To address this issue, we propose Dominant-Subspace Quantization (DSQ), which leverages a global PCA analysis to select principal components while suppressing noisy dimensions, thereby stabilizing codebook optimization and improving reconstruction and generation quality. On ImageNet 256x256, DINO-Tok achieves strong reconstruction performance, achieving 0.28 rFID for continuous autoencoding and 1.10 rFID for discrete VQ, as well as strong few-step generation performance 1.82 gFID for diffusion and 2.44 gFID for autoregressive generation. These results demonstrate that pre-trained VFMs such as DINO can be directly adapted into high-fidelity, semantically aligned visual tokenizers for next-generation latent generative models. Code will be publicly available at https://github.com/MKJia/DINO-Tok.

视觉分词器DINO生成模型向量量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。