arXiv:2605.25012cs.CV2026-05中稿 · CVPR

用语义词典训练统一视觉模型,生成与理解能力兼备。

Learning from Semantic Dictionaries: Discriminative Codebook Contrastive Learning for Unified Visual Representation and Generation

论文配图:Learning from Semantic Dictionaries: Discriminative Codebook Contrastive Learning for Unified Visual Representation and Generation
图 1 · 摘自论文原文
  • 双码本设计在离散令牌空间中联合优化生成与判别任务。
  • 图像分类、生成质量、少样本迁移等指标全面超越现有方法。
  • 无需数据增强或教师模型,适合快速部署的通用视觉系统。

判别式与生成式视觉模型虽各自表现优异,但语义上存在错位,阻碍了统一视觉学习的发展。我们提出 LEASE(LEArning from SEmantic Dictionaries),一种基于成对生成-判别码本设计的自监督框架,通过一次预计算生成离散令牌空间,在无需数据增强、教师模型或在线分词器的情况下实现高效训练。该框架融合两种互补目标:掩码令牌重建损失用于捕捉精细生成细节,码本对比损失通过自适应中心加权将编码器特征与判别语义对齐。双重监督使潜在空间兼具高质量生成与强表征能力。在 ImageNet-1K 上,LEASE 达到统一性能新纪录,线性探测提升最高 +1.7%,无条件生成(相较 MAGE:FID 降低 -1.26,IS 提升 +10.19),少样本学习平均提升 +0.56%,迁移能力平均提高 +0.75%,鲁棒性测试平均提升 +5.86% 和 +4.25%。其无监督模型还可扩展至条件生成,表现媲美专用基线。整体上,LEASE 为兼顾理解与生成的通用视觉模型提供了高效有效路径。

原文摘要 · Abstract (English)

Discriminative and generative vision models excel in their respective domains but remain semantically misaligned, hindering progress toward unified visual learning. We introduce LEASE (LEArning from SEmantic Dictionaries), a self-supervised framework that bridges this gap using a paired generative-discriminative codebook design. LEASE operates entirely in a discrete token space produced through a one-time precomputation step, enabling efficient training without data augmentations, teacher models, or online tokenizers. LEASE integrates two complementary objectives: a masked token reconstruction loss that captures fine-grained generative detail, and a codebook contrast loss that aligns encoder features with discriminative semantics via adaptive centroid weighting. This dual supervision yields a unified latent space that supports both high-quality generation and strong representation learning. On ImageNet-1K, LEASE achieves state-of-the-art unified performance, outperforming prior VQGAN-based methods such as MAGE and Sorcen across linear probing (up to +1.7%), unconditional generation (-1.26 FID and +10.19 IS w.r.t MAGE), few-shot learning (+0.56% on average against Sorcen), transfer (+0.75% average improvement against MAGE and Sorcen), and robustness benchmarks (+5.86% and +4.25% average improvement against MAGE and Sorcen, respectively). It also competes favorably with domain-specialized contrastive and generative models while surpassing previous MIM methods. The unsupervised LEASE model can also be extended to conditional generation by building upon its learned representations, proving competitive with specialized baselines. Overall, LEASE provides an efficient and effective step toward general-purpose vision models that jointly understand and generate visual content.

统一视觉生成模型自监督码本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。