用自监督模型的单个隐向量实现高效图像生成。
Adapting Self-Supervised Representations as a Latent Space for Efficient Generation
- 用预训练视觉变压器提取单个连续隐向量，仅微调语义嵌入。
- 在ImageNet上达到与主流方法相当的生成质量，且训练成本大幅降低。
- 适合资源受限场景，可零样本迁移至文生图任务。
我们提出Representation Tokenizer（RepTok），一种生成建模框架，通过自监督视觉变换器提取单个连续隐向量表示图像。基于预训练的SSL编码器，仅微调语义向量嵌入，并与联合训练的生成解码器搭配，使用标准流匹配目标进行优化。该适配过程为向量注入低层重建细节，实现高保真图像重构。为保持原始SSL空间的良好几何结构，引入余弦相似性损失以正则化适配后的向量，确保隐空间平滑且适合生成。单向量设计消除了二维隐空间中的空间冗余，显著降低训练开销。尽管结构简单高效，RepTok在类别条件图像生成（ImageNet）上表现竞争力，并自然扩展至文本到图像合成，在极低训练预算下于MS-COCO数据集达到有竞争力的零样本性能。结果表明，微调后的SSL表征可作为紧凑高效的生成潜空间。
原文摘要 · Abstract (English)
We introduce Representation Tokenizer (RepTok), a generative modeling framework that represents an image using a single continuous latent token obtained from self-supervised vision transformers. Building on a pre-trained SSL encoder, we fine-tune only the semantic token embedding and pair it with a generative decoder trained jointly using a standard flow matching objective. This adaptation enriches the token with low-level, reconstruction-relevant details, enabling faithful image reconstruction. To preserve the favorable geometry of the original SSL space, we add a cosine-similarity loss that regularizes the adapted token, ensuring the latent space remains smooth and suitable for generation. Our single-token formulation resolves spatial redundancies of 2D latent spaces and significantly reduces training costs. Despite its simplicity and efficiency, RepTok achieves competitive results on class-conditional ImageNet generation and naturally extends to text-to-image synthesis, reaching competitive zero-shot performance on MS-COCO under extremely limited training budgets. Our findings highlight the potential of fine-tuned SSL representations as compact and effective latent spaces for efficient generative modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。