纯解码器架构实现图文理解生成编辑统一，效率更高。
OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
- 用单一自回归目标训练，无需额外视觉组件。
- 多尺度视觉自回归减少解码步骤，性能达顶尖水平。
- 适合需要高效统一模态处理的场景，如实时生成。
我们提出OneCAT，一种统一的多模态模型，通过创新的纯解码器仅变压器架构，无缝融合理解、生成与编辑功能。该框架在推理时完全避免使用外部组件（如ViT或视觉分词器），尤其在高分辨率输入下显著提升效率。其核心是基于模态特定的专家混合（MoE）结构，以单一自回归目标训练，并天然支持动态分辨率。此外，我们在大语言模型中首创多尺度视觉自回归机制，相比扩散方法大幅减少解码步数，同时保持领先性能。实验表明，纯自回归建模足以成为统一多模态智能的坚实基础。OneCAT在多模态生成、编辑和理解基准上超越现有开源统一模型，树立新标准。
原文摘要 · Abstract (English)
We introduce OneCAT, a unified multimodal model that seamlessly integrates understanding, generation, and editing within a novel, pure decoder-only transformer architecture. Our framework uniquely eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizer during inference, leading to significant efficiency gains, especially for high-resolution inputs. This is achieved through a modality-specific Mixture-of-Experts (MoE) structure trained with a single autoregressive (AR) objective, which also natively supports dynamic resolutions. Furthermore, we pioneer a multi-scale visual autoregressive mechanism within the Large Language Model (LLM) that drastically reduces decoding steps compared to diffusion-based methods while maintaining state-of-the-art performance. Our findings demonstrate the powerful potential of pure autoregressive modeling as a sufficient and elegant foundation for unified multimodal intelligence. As a result, OneCAT sets a new performance standard, outperforming existing open-source unified multimodal models across benchmarks for multimodal generation, editing, and understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。