用语义约束提升视觉与语言对齐，让统一模型更高效更强
MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding
- 给视觉分词器加入语义约束，实现视觉与语言令牌对齐
- 仅用更少数据就比SOTA模型理解能力高4.8%，超越专用模型3.7%
- 既适合多模态理解也适合生成任务，适合研究统一模型的学者
我们提出MUSE-VL，一种通过语义离散编码实现统一多模态理解与生成的视觉-语言模型。当前研究虽探索统一模型用于视觉生成与理解，但现有视觉分词器（如VQGAN）仅关注低层信息，难以与语言令牌对齐，导致训练复杂度高，需大量数据才能达最优性能，且表现仍落后于专用理解模型。本文提出语义离散编码（SDE），通过在视觉分词器中引入语义约束，有效对齐视觉与语言信息，大幅减少训练数据需求，并提升统一模型性能。在相同LLM规模下，本方法理解性能比先前SOTA Emu3提升4.8%，超越专用理解模型LLaVA-NeXT 34B达3.7%。模型在视觉生成基准上也优于现有统一模型。
原文摘要 · Abstract (English)
We introduce MUSE-VL, a Unified Vision-Language Model through Semantic discrete Encoding for multimodal understanding and generation. Recently, the research community has begun exploring unified models for visual generation and understanding. However, existing vision tokenizers (e.g., VQGAN) only consider low-level information, which makes it difficult to align with language tokens. This results in high training complexity and necessitates a large amount of training data to achieve optimal performance. Additionally, their performance is still far from dedicated understanding models. This paper proposes Semantic Discrete Encoding (SDE), which effectively aligns the information of visual tokens and language tokens by adding semantic constraints to the visual tokenizer. This greatly reduces the amount of training data and improves the performance of the unified model. With the same LLM size, our method improved the understanding performance by 4.8% compared to the previous SOTA Emu3 and surpassed the dedicated understanding model LLaVA-NeXT 34B by 3.7%. Our model also surpasses the existing unified models on visual generation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。