一个模型同时搞定图像生成与多标签理解，让机器像人一样看图说话。
Jodi: Unification of Visual Generation and Understanding via Joint Modeling
- 用线性扩散Transformer+角色切换机制，统一建模图像和多个标签域。
- 支持联合生成、可控生成、图像感知三类任务，性能超越单一任务模型。
- 适合需要跨视觉理解与生成的场景，如内容创作、智能助手等。
视觉生成与理解是人类智能的两个紧密关联方面，但在机器学习中长期被当作独立任务处理。本文提出Jodi，一种基于线性扩散Transformer与角色切换机制的扩散框架，通过联合建模图像域与多个标签域，实现视觉生成与理解的统一。Jodi可执行三类任务：（1）联合生成，同时输出图像与多个标签；（2）可控生成，根据任意标签组合生成图像；（3）图像感知，从给定图像中一次性预测多个标签。我们构建了Joint-1.6M数据集，包含20万张高质量图像、7个视觉领域的自动标注标签及大模型生成的描述文本。大量实验表明，Jodi在生成与理解任务上均表现优异，并具备向更广泛视觉领域扩展的强泛化能力。代码已公开于https://github.com/VIPL-GENUN/Jodi。
原文摘要 · Abstract (English)
Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。