用中间表示预测替代冻结VAE，实现端到端多模态模型生成与理解。
Representation Forcing for Bottleneck-Free Unified Multimodal Models
- 让解码器先预测视觉表示，再生成像素，统一生成流程。
- 图像生成质量媲美顶尖基于VAE的模型，理解任务表现更优。
- 无需外部生成隐空间，适合追求端到端架构的研究者。
统一多模态模型（UMMs）旨在单一模型中完成感知与生成任务。然而现有方法仍依赖独立预训练的冻结变分自编码器（VAE）进行图像生成，形成结构瓶颈。直接移除该组件会引入质量差距，因模型需从原始像素中同时学习高层结构与低层细节。本文提出表示强制（Representation Forcing, RF），通过使表示预测成为模型的原生能力来弥合这一差距。具体而言，RF 强制解码器在生成像素前，以自回归方式预测视觉表示作为中间标记；这些标记保留在上下文中，引导同一骨干网络内的像素扩散过程。通过将表示从感知输出转变为生成目标，RF 消除了对外部生成隐空间的需求。实验表明，RF 同时提升理解和生成性能：在图像生成方面，基于像素空间的模型在使用 RF 时达到与最先进的基于 VAE 的统一模型相当的水平；在图像理解任务中，像素空间的 RF 模型普遍优于其基于 VAE 的变体。这些结果为实现端到端、无瓶颈的统一多模态模型提供了有效路径。
原文摘要 · Abstract (English)
Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。