Tuna-2直接用像素嵌入实现多模态理解与生成,无需预训练视觉编码器。
Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

- 抛弃视觉编码器,直接用像素嵌入进行多模态建模
- 在多模态基准上达到顶尖性能,生成质量媲美潜空间方法
- 适合追求端到端优化和细粒度视觉感知的研究者
统一的多模态模型通常依赖预训练的视觉编码器,并为理解和生成任务使用独立的视觉表示,导致两任务间存在错位,阻碍从原始像素进行端到端优化。我们提出Tuna-2,一种原生统一的多模态模型,直接基于像素嵌入完成视觉理解与生成。Tuna-2通过简单的图像块嵌入层编码视觉输入,彻底摒弃了如VAE或表征编码器等模块化设计,极大简化模型结构。实验表明,Tuna-2在多模态基准上达到业界领先性能,证明像素空间建模可完全媲美潜空间方法,在高质量图像生成中表现优异。尽管基于编码器的变体在早期预训练中收敛更快,但Tuna-2的无编码器设计在大规模下展现出更强的多模态理解能力,尤其在需要细粒度视觉感知的任务上优势显著。结果表明,预训练视觉编码器并非多模态建模必需,端到端像素空间学习为生成与感知提供了可扩展的强视觉表征路径。
原文摘要 · Abstract (English)
Unified multimodal models typically rely on pretrained vision encoders and use separate visual representations for understanding and generation, creating misalignment between the two tasks and preventing fully end-to-end optimization from raw pixels. We introduce Tuna-2, a native unified multimodal model that performs visual understanding and generation directly based on pixel embeddings. Tuna-2 drastically simplifies the model architecture by employing simple patch embedding layers to encode visual input, completely discarding the modular vision encoder designs such as the VAE or the representation encoder. Experiments show that Tuna-2 achieves state-of-the-art performance in multimodal benchmarks, demonstrating that unified pixel-space modelling can fully compete with latent-space approaches for high-quality image generation. Moreover, while the encoder-based variant converges faster in early pretraining, Tuna-2's encoder-free design achieves stronger multimodal understanding at scale, particularly on tasks requiring fine-grained visual perception. These results show that pretrained vision encoders are not necessary for multimodal modelling, and end-to-end pixel-space learning offers a scalable path toward stronger visual representations for both generation and perception.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。