让理解力直接指导生成,提升图文模型的创作能力
Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generation
- 在生成过程中嵌入理解能力,用图像编辑作为桥梁
- 在TIIF长提示测试中提升3.92%生成质量
- 适合想改进图文生成模型的研究者和开发者
近期研究通过思维链(CoT)提升了统一模型在文本到图像生成中的表现,但这些方法将理解与生成分离,难以有效引导模型弥补生成缺陷。为此,我们提出一种新框架Understanding-in-Generation(UiG),利用统一模型强大的理解能力来增强其图像生成性能。核心思想是在推理过程中融入理解能力以指导生成,从而缓解生成能力不足的问题。我们引入‘图像编辑’作为桥梁:先验证生成图像,再结合统一模型的理解生成编辑指令,并逐步优化图像,实现理解向生成过程的渐进式注入。实验表明,该框架在TIIF基准的长提示设置下相比现有方法提升3.92%,显著改善了文本到图像生成效果。项目代码已开源:https://github.com/QC-LY/UiG。
原文摘要 · Abstract (English)
Recent works have made notable advancements in enhancing unified models for text-to-image generation through the Chain-of-Thought (CoT). However, these reasoning methods separate the processes of understanding and generation, which limits their ability to guide the reasoning of unified models in addressing the deficiencies of their generative capabilities. To this end, we propose a novel reasoning framework for unified models, Understanding-in-Generation (UiG), which harnesses the robust understanding capabilities of unified models to reinforce their performance in image generation. The core insight of our UiG is to integrate generative guidance by the strong understanding capabilities during the reasoning process, thereby mitigating the limitations of generative abilities. To achieve this, we introduce "Image Editing" as a bridge to infuse understanding into the generation process. Initially, we verify the generated image and incorporate the understanding of unified models into the editing instructions. Subsequently, we enhance the generated image step by step, gradually infusing the understanding into the generation process. Our UiG framework demonstrates a significant performance improvement in text-to-image generation over existing text-to-image reasoning methods, e.g., a 3.92% gain on the long prompt setting of the TIIF benchmark. The project code: https://github.com/QC-LY/UiG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。