VersaGen让文本生成图像支持多类型视觉控制，自由组合主体、背景等元素。
VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis
- 在冻结的文本图像模型上训练适配器，融合多种视觉控制输入
- 支持单个/多个主体、背景或任意组合控制，覆盖创作全流程需求
- 推理阶段引入三项优化策略，提升生成质量与用户体验
尽管文本到图像（T2I）合成技术快速进步，但实现精确视觉控制仍是重大挑战。现有方法尝试结合文本和草图等多种控制方式，以增强图像生成的创意控制力。然而我们的初步研究发现，人类表达能力远超当前方法。用户需要更灵活的方式，以控制个体对象到整体场景布局的多样化创作意图。为此，我们提出VersaGen，一种支持多类型视觉控制的生成式AI代理。VersaGen可接受四种视觉控制：单一视觉主体、多个视觉主体、场景背景，以及上述三者的任意组合或完全无控制。我们在一个冻结的T2I模型基础上训练适配器，将视觉信息融入以文本为主导的扩散过程，并在推理阶段引入三种优化策略，以提升生成效果并改善用户体验。在COCO和Sketchy数据集上的综合实验表明，VersaGen在定性和定量结果上均展现出卓越的有效性与灵活性。
原文摘要 · Abstract (English)
Despite the rapid advancements in text-to-image (T2I) synthesis, enabling precise visual control remains a significant challenge. Existing works attempted to incorporate multi-facet controls (text and sketch), aiming to enhance the creative control over generated images. However, our pilot study reveals that the expressive power of humans far surpasses the capabilities of current methods. Users desire a more versatile approach that can accommodate their diverse creative intents, ranging from controlling individual subjects to manipulating the entire scene composition. We present VersaGen, a generative AI agent that enables versatile visual control in T2I synthesis. VersaGen admits four types of visual controls: i) single visual subject; ii) multiple visual subjects; iii) scene background; iv) any combination of the three above or merely no control at all. We train an adaptor upon a frozen T2I model to accommodate the visual information into the text-dominated diffusion process. We introduce three optimization strategies during the inference phase of VersaGen to improve generation results and enhance user experience. Comprehensive experiments on COCO and Sketchy validate the effectiveness and flexibility of VersaGen, as evidenced by both qualitative and quantitative results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。