用视觉理解先验提升图像生成质量，不依赖复杂编码器。
VUGEN: Visual Understanding priors for GENeration
- 将视觉语言模型的视觉表征降维，保留关键信息用于生成。
- 在降低维度的潜空间中采样，生成图像性能提升显著。
- 无需变分自编码器，适合追求高效高质量生成的研究者。
视觉语言模型（VLM）虽实现了文本与图像的统一理解，但赋予其稳健的图像生成能力仍具挑战。现有方法多依赖重建导向的自编码器或复杂桥接机制，导致理解与生成表征不一致或结构冗余。本文提出VUGEN框架，直接利用预训练VLM的视觉理解先验，实现高效高质的图像生成。首先将VLM原生视觉编码器的高维潜空间转换为低维、可处理的分布，最大程度保留视觉信息；随后训练VLM在此简化潜空间中采样，确保与理解能力对齐；最后使用专用像素解码器将生成潜变量映射回图像空间。实验表明，无需变分自编码器的像素扩散解码器性能达到或优于依赖VAE潜空间的复杂解码器。大量实验证明，VUGEN显著提升生成效果：在COCO数据集上，DPG Bench从71.17提升至74.32，FID从11.86降至9.06，且完全保留了VLM原有的理解能力。
原文摘要 · Abstract (English)
Recent advances in Vision-Language Models (VLMs) have enabled unified understanding across text and images, yet equipping these models with robust image generation capabilities remains challenging. Existing approaches often rely on reconstruction-oriented autoencoders or complex bridging mechanisms, leading to misalignment between understanding and generation representations, or architectural complexity. In this work, we propose VUGEN, a novel framework that explicitly leverages VLM's pretrained visual understanding priors for efficient and high-quality image generation. Our approach first transforms the high-dimensional latent space of the VLM's native vision encoder into a lower-dimensional, tractable distribution that maximally preserves visual information. The VLM is then trained to sample within this reduced latent space, ensuring alignment with its visual understanding capabilities. Finally, a dedicated pixel decoder maps these generated latents back to the image space. We find that a VAE-free pixel diffusion decoder to be on par or better than commonly used complex latent diffusion decoders that internally rely on VAE latents. Extensive experiments demonstrate that VUGEN achieves superior image generation performance, improving DPG Bench from 71.17 to 74.32 and FID from 11.86 to 9.06 on COCO, while fully preserving the VLM's original understanding capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。