arXiv:2503.14324cs.CVcs.CL2025-03被引 41

用双视觉词表统一图像理解与生成,性能全面超越现有方法。

DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies

  • 分离高阶语义与低阶视觉细节的双词典设计,避免训练冲突。
  • 在ImageNet上实现0.25 rFID与82.0%零样本准确率。
  • 适用于多模态大模型的理解与生成任务,尤其适合需兼顾双端的场景。

视觉理解与生成在自回归大语言模型中面临表示空间差异的挑战。用于重建的视觉分词器擅长捕捉低层视觉外观,适合生成但缺乏高层语义;而通过对比学习训练的视觉编码器虽与语言对齐良好,却难以解码回像素空间。为此,我们提出DualToken,通过分离高阶语义与低阶视觉细节的双词典,统一两者表示。直接融合重建与语义目标会导致性能下降,因此我们采用独立词典策略。结果表明,DualToken在ImageNet上达成0.25 rFID和82.0%零样本准确率,在十项视觉理解基准上平均超越VILA-U 5.8点,在GenAI-Bench上提升13%。使用双视觉令牌显著优于单一令牌类型,验证了双词典设计的有效性。

原文摘要 · Abstract (English)

The differing representation spaces required for visual understanding and generation pose a challenge in unifying them within the autoregressive paradigm of large language models. A vision tokenizer trained for reconstruction excels at capturing low-level visual appearance, making it well-suited for visual generation but lacking high-level semantic representations for understanding tasks. Conversely, a vision encoder trained via contrastive learning aligns well with language but struggles to decode back into the pixel space for generation tasks. To bridge this gap, we propose DualToken, a method that unifies representations for both understanding and generation within a single tokenizer. However, directly integrating reconstruction and semantic objectives creates conflicts, leading to degraded performance in both reconstruction fidelity and semantic accuracy. Instead of forcing a single codebook to capture both visual appearance and semantics, DualToken disentangles them by introducing separate codebooks for high-level semantics and low-level visual details. As a result, DualToken achieves 0.25 rFID and 82.0% zero-shot accuracy on ImageNet, and demonstrates strong effectiveness in downstream MLLM tasks for both understanding and generation. Specifically, our method surpasses VILA-U by 5.8 points on average across ten visual understanding benchmarks and delivers a 13% improvement on GenAI-Bench. Notably, incorporating dual visual tokens outperforms using a single token type on both understanding and generation tasks. We hope our research offers a new perspective on leveraging dual visual vocabularies for building unified vision-language models. Project page is available at https://songweii.github.io/dualtoken-project-page.

视觉生成双词典多模态模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。