用视频模型统一图像生成，实现条件帧预测的类上下文学习。
RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Models
- 将图像生成转为条件帧预测，模仿大语言模型的上下文学习机制。
- 在定制化生成中主体相似度提升14.5%，在边缘转图像任务中质量提升10%。
- 适合需要多任务统一生成的开发者与研究者，尤其关注跨模态对齐。
统一多种图像生成任务于单一框架仍是视觉生成中的核心挑战。尽管大语言模型通过无任务特定数据和生成方式实现了统一，现有视觉生成模型未能满足此原则。当前方法要么依赖任务专属数据集和大规模训练，要么通过任务特异性修改预训练图像模型，限制了泛化能力。本文探索以视频模型为基础实现统一图像生成，利用其建模时序相关性的内在优势。提出RealGeneral框架，将图像生成重构为条件帧预测任务，类比大语言模型的上下文学习。为弥合视频模型与条件-图像对之间的差距，设计(1)统一条件嵌入模块以实现多模态对齐，(2)统一流式DiT块，采用解耦自适应层归一化与注意力掩码以缓解跨模态干扰。RealGeneral在多个重要视觉生成任务中表现出色：在定制化生成任务中主体相似度提升14.5%，在canny-to-image任务中图像质量提升10%。
原文摘要 · Abstract (English)
Unifying diverse image generation tasks within a single framework remains a fundamental challenge in visual generation. While large language models (LLMs) achieve unification through task-agnostic data and generation, existing visual generation models fail to meet these principles. Current approaches either rely on per-task datasets and large-scale training or adapt pre-trained image models with task-specific modifications, limiting their generalizability. In this work, we explore video models as a foundation for unified image generation, leveraging their inherent ability to model temporal correlations. We introduce RealGeneral, a novel framework that reformulates image generation as a conditional frame prediction task, analogous to in-context learning in LLMs. To bridge the gap between video models and condition-image pairs, we propose (1) a Unified Conditional Embedding module for multi-modal alignment and (2) a Unified Stream DiT Block with decoupled adaptive LayerNorm and attention mask to mitigate cross-modal interference. RealGeneral demonstrates effectiveness in multiple important visual generation tasks, e.g., it achieves a 14.5% improvement in subject similarity for customized generation and a 10% enhancement in image quality for canny-to-image task. Project page: https://lyne1.github.io/realgeneral_web/; GitHub Link: https://github.com/Lyne1/RealGeneral
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。