用统一视觉空间实现理解、生成与编辑,保留细节并提升图像重建质量。
UniSpace: Unified Visual Representation and Scalable Multimodal Modeling

- 通过重参数化补丁嵌入,在不训练ViT的前提下保留视觉细节。
- 80亿参数的UniSpace模型实现高质量图像生成与指令编辑。
- 适合需要高保真图像重建的多模态应用,如图像生成与编辑系统。
语义视觉编码器已成为图像生成中多模态理解与语义条件化的核心视觉接口。然而,其最终标记会丢失细粒度视觉细节,导致像素重建效果差,限制了在图像生成和编辑等重建敏感任务中的应用。本文探讨是否可基于预训练语义ViT构建一个统一的视觉表示空间,以同时支持理解、生成与编辑。研究发现,冻结的Transformer块本身并非无法保留视觉细节,而是原始补丁参数化方式使表示趋向语义抽象,难以恢复细粒度信息。为此,我们提出『补丁重参数化』,在保持原有语义路径的同时,引入具备重建感知的补丁嵌入,向同一冻结的ViT块提供细粒度视觉信息。所得统一表示既保留多模态理解能力,又支持高保真图像重建,并实现良好的重建-生成权衡。进一步扩展为80亿参数的多专家变压器模型UniSpace,可在同一视觉空间内完成理解、生成与编辑,无需独立的VAE路径。系统级评估表明,该模型实现了实用的文本到图像生成与基于指令的图像编辑,证明重参数化的预训练ViT可作为可扩展多模态建模的统一视觉接口。
原文摘要 · Abstract (English)
Semantic vision encoders have become a central visual interface for multimodal understanding and semantic conditioning in image generation. However, their final tokens discard fine-grained visual details, leading to poor pixel reconstruction and limiting their use in reconstruction-sensitive tasks such as image generation and editing. In this work, we ask whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrained semantic ViT. We show that the frozen Transformer blocks of a semantic ViT are not intrinsically unable to preserve visual details. Instead, the original patch parameterization drives the representation toward semantic abstraction, making fine-grained information difficult to recover from the final tokens. Based on this observation, we introduce \emph{Patch Reparameterization}, which preserves the original semantic pathway while adding a reconstruction-aware patch embedding that provides fine-grained visual information to the same frozen ViT blocks. The resulting unified representation preserves multimodal understanding while enabling high-fidelity image reconstruction and a favorable reconstruction--generation trade-off. We further scale this representation into \emph{UniSpace}, an 8B Mixture-of-Transformer-Experts model that performs understanding, generation, and editing in the same visual space without a separate VAE pathway. System-level evaluations demonstrate practical text-to-image generation and instruction-based image editing, showing that a reparameterized pretrained ViT can serve as a unified visual interface for scalable multimodal modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。