让图像生成模型像人类一样快速学会新指令。
Dual-Process Image Generation
- 用视觉语言模型评分生成图，反向传播优化图像生成器
- 支持颜色、线条、深度等多模态控制，几分钟内完成设定
- 无需重训练，通过文本图像接口即可实现新任务
以往图像生成控制方法难以学习新任务。相比之下，视觉语言模型（VLM）可在上下文中学习并生成正确输出。本文提出一种双过程蒸馏框架，使前馈式图像生成器可从推理型VLM中学习新任务。该方法利用VLM对生成图像进行评分，并将梯度反向传播以更新图像生成器权重。该通用框架通过统一的文本-图像接口支持多种新控制任务，如常识推理和视觉提示。我们展示了该技术在颜色配色、线条粗细、地平线位置和相对深度等多种属性上的应用，用户可在数分钟内实现多模态控制。项目页面：https://dual-process.github.io。
原文摘要 · Abstract (English)
Prior methods for controlling image generation are limited in their ability to be taught new tasks. In contrast, vision-language models, or VLMs, can learn tasks in-context and produce the correct outputs for a given input. We propose a dual-process distillation scheme that allows feed-forward image generators to learn new tasks from deliberative VLMs. Our scheme uses a VLM to rate the generated images and backpropagates this gradient to update the weights of the image generator. Our general framework enables a wide variety of new control tasks through the same text-and-image based interface. We showcase a handful of applications of this technique for different types of control signals, such as commonsense inferences and visual prompts. With our method, users can implement multimodal controls for properties such as color palette, line weight, horizon position, and relative depth within a matter of minutes. Project page: https://dual-process.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。