OmniGen2统一生成多模态任务,支持文本到图像、编辑和上下文生成。
OmniGen2: Towards Instruction-Aligned Multimodal Generation
- 分路径解码文本与图像,参数不共享,保留原生文本生成能力
- 在多个基准上表现优异,上下文生成任务达开源模型最佳水平
- 开源模型、代码、数据集,支持后续研究
本文提出OmniGen2,一个多功能开源生成模型,可统一处理文本到图像、图像编辑及上下文生成等多样化任务。相比OmniGen v1,OmniGen2采用独立的文本与图像解码路径,使用非共享参数和解耦图像分词器,无需重新适配VAE输入即可继承现有多模态理解模型能力,保持原始文本生成性能。为训练该模型,我们构建了完整的数据构造流程,涵盖图像编辑和上下文生成数据。同时引入专用于图像生成的反思机制,并基于OmniGen2构建专用反思数据集。尽管参数量较小,OmniGen2在多个任务基准上表现竞争力,包括文本到图像和图像编辑。为进一步评估上下文生成(即主题驱动任务),我们提出了新基准OmniContext。OmniGen2在一致性方面达到开源模型最优水平。我们将发布模型、训练代码、数据集及数据构建流程,以支持该领域后续研究。项目页面:https://vectorspacelab.github.io/OmniGen2;GitHub链接:https://github.com/VectorSpaceLab/OmniGen2
原文摘要 · Abstract (English)
In this work, we introduce OmniGen2, a versatile and open-source generative model designed to provide a unified solution for diverse generation tasks, including text-to-image, image editing, and in-context generation. Unlike OmniGen v1, OmniGen2 features two distinct decoding pathways for text and image modalities, utilizing unshared parameters and a decoupled image tokenizer. This design enables OmniGen2 to build upon existing multimodal understanding models without the need to re-adapt VAE inputs, thereby preserving the original text generation capabilities. To facilitate the training of OmniGen2, we developed comprehensive data construction pipelines, encompassing image editing and in-context generation data. Additionally, we introduce a reflection mechanism tailored for image generation tasks and curate a dedicated reflection dataset based on OmniGen2. Despite its relatively modest parameter size, OmniGen2 achieves competitive results on multiple task benchmarks, including text-to-image and image editing. To further evaluate in-context generation, also referred to as subject-driven tasks, we introduce a new benchmark named OmniContext. OmniGen2 achieves state-of-the-art performance among open-source models in terms of consistency. We will release our models, training code, datasets, and data construction pipeline to support future research in this field. Project Page: https://vectorspacelab.github.io/OmniGen2; GitHub Link: https://github.com/VectorSpaceLab/OmniGen2
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。