一个简单模型统一理解、生成与编辑图像，支持中英文和多分辨率。
A Simple Baseline for Unifying Understanding, Generation, and Editing via Vanilla Next-token Prediction
- 用基础的下一个词预测统一多模态任务
- 在多个基准上表现优于或媲美现有统一模型
- 适合需要轻量级多任务统一方案的研究者
本文提出 Wallaroo，一种基于下一个词预测的简单自回归基线模型，可同时实现多模态理解、图像生成与编辑。Wallaroo 支持多分辨率输入输出及中英文双语处理。通过将视觉编码解耦为独立路径，并采用四阶段训练策略，重塑模型能力。在多个基准上的实验表明，Wallaroo 性能达到竞争力或超越现有统一模型，表明自回归模型在统一多模态理解与生成方面具有巨大潜力。代码已开源：https://github.com/JiePKU/Wallaroo。
原文摘要 · Abstract (English)
In this work, we introduce Wallaroo, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time. Moreover, Wallaroo supports multi-resolution image input and output, as well as bilingual support for both Chinese and English. We decouple the visual encoding into separate pathways and apply a four-stage training strategy to reshape the model's capabilities. Experiments are conducted on various benchmarks where Wallaroo produces competitive performance or exceeds other unified models, suggesting the great potential of autoregressive models in unifying multi-modality understanding and generation. Our code is available at https://github.com/JiePKU/Wallaroo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。