通过理解模型与生成模型互动,让文生图更美观。
ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction
- 用理解模型隐式学习人类偏好,指导生成模型改进图像
- 迭代融合优化结果,使生成图像更符合审美
- 无需额外计算开销,适合实际部署
扩散模型的出现显著推进了图像生成技术。近期大型语言模型中的交互与自修正推理研究为提升文生图模型提供了新思路。受此启发,本文提出一种名为 ArtAug 的新方法,旨在通过生成模型与理解模型的交互来增强文生图模型。据我们所知,ArtAug 是首个通过模型交互提升图像生成质量的方法。在交互过程中,利用理解模型隐式学习的人类偏好,为生成模型提供细粒度建议,如调整曝光、改变拍摄角度、添加氛围效果等,以提升图像美感。这些改进通过额外的增强模块逐步融入生成模型本身,使其能直接输出更具审美价值的图像,且不增加额外计算成本。实验表明,在现有文生图模型上训练 ArtAug 增强模块后,多种评估指标一致显示其有效提升了生成能力,且无额外计算开销。源代码与模型将公开发布。
原文摘要 · Abstract (English)
The emergence of diffusion models has significantly advanced image synthesis. The recent studies of model interaction and self-corrective reasoning approach in large language models offer new insights for enhancing text-to-image models. Inspired by these studies, we propose a novel method called ArtAug for enhancing text-to-image models in this paper. To the best of our knowledge, ArtAug is the first one that improves image synthesis models via model interactions with understanding models. In the interactions, we leverage human preferences implicitly learned by image understanding models to provide fine-grained suggestions for image synthesis models. The interactions can modify the image content to make it aesthetically pleasing, such as adjusting exposure, changing shooting angles, and adding atmospheric effects. The enhancements brought by the interaction are iteratively fused into the synthesis model itself through an additional enhancement module. This enables the synthesis model to directly produce aesthetically pleasing images without any extra computational cost. In the experiments, we train the ArtAug enhancement module on existing text-to-image models. Various evaluation metrics consistently demonstrate that ArtAug enhances the generative capabilities of text-to-image models without incurring additional computational costs. The source code and models will be released publicly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。