用统一编码器实现图文理解生成与重建,兼具语义连续性和生成离散性。
VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
- 基于向量量化自编码器,用高维离散码本统一处理视觉理解与生成。
- 1536维码本利用率达100%,在多个基准上表现优异且可扩展。
- 适合需要统一多模态建模的科研与工业场景,尤其关注生成质量与精度。
统一多模态理解、生成与重建的表示学习仍面临挑战。以往工作多采用双编码器范式,如分别使用编码器进行理解与生成,或通过对比损失平衡语义表示与低级特征。本文提出VQRAE,即向量量化表示自编码器,首次在统一分词器中实现连续语义特征用于图像理解,同时生成离散标记用于视觉生成。我们基于预训练视觉基础模型,采用对称ViT解码器,并设计两阶段训练策略:首先冻结编码器,以像素重建为目标学习高维语义向量量化码本;随后联合优化编码器并引入自蒸馏约束。该设计使语义信息几乎无损保留,生成所用离散标记兼容生成与精细重建。此外,我们发现语义编码器使用高维码本(如1536维)时,可实现100%码本利用率,优于传统低维码本在图像重建中的做法。VQRAE在多个视觉理解、生成与重建基准上表现优异,且在自回归范式下展现出良好的可扩展性。
原文摘要 · Abstract (English)
Unifying multimodal understanding, generation and reconstruction representation in a single tokenizer remains a key challenge in building unified models. Previous research predominantly attempts to address this in a dual encoder paradigm, e.g., utilizing the separate encoders for understanding and generation respectively or balancing semantic representations and low-level features with contrastive loss. In this paper, we propose VQRAE, a Vector Quantization version of Representation AutoEncoders, which pioneers the first exploration in unified representation to produce Continuous semantic features for image understanding and Discrete tokens for visual generation within a unified tokenizer. Specifically, we build upon pretrained vision foundation models with a symmetric ViT decoder and adopt a two-stage training strategy: first, it freezes the encoder and learns a high-dimensional semantic VQ codebook with pixel reconstruction objective; then jointly optimizes the encoder with self-distillation constraints. This design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens that are compatible for generation and fine-grained reconstruction. Besides, we identify the intriguing property in quantizing semantic encoders that rely on high-dimensional codebook in contrast to the previous common practice of low-dimensional codebook in image reconstruction. The semantic VQ codebook can achieve a 100% utilization ratio at a dimension of 1536. VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction with promising scaling property in the autoregressive paradigm for its discrete merits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。