用一张图生成包含复杂事件的定制化图像,无需重新训练模型。
Event-Customized Image Generation
- 通过双路径设计,分离实体替换与事件迁移,实现无训练定制。
- 在两个新数据集上验证,生成图像在动作和关系上与参考图高度一致。
- 适合需要快速生成复杂场景图像的应用,如创意设计、虚拟拍摄。
定制化图像生成因其创造性和新颖性受到广泛关注。尽管主体定制已取得显著进展,部分前沿工作进一步探索了超越实体外观的动作与交互定制,但这些方法仅限于简单动作和双实体间交互,且受限于缺乏完全相同的参考图像。为拓展定制化图像生成至更复杂的现实场景,我们提出新任务:事件定制化图像生成。给定单张参考图像,将“事件”定义为场景中所有具体动作、姿态、关系或实体间的交互。该任务旨在准确捕捉复杂事件,并生成包含不同目标实体的定制图像。为此,我们提出一种无需训练的事件定制方法:FreeEvent。其在通用扩散去噪过程中引入两条额外路径:1)实体切换路径,通过交叉注意力引导与调控目标实体生成;2)事件迁移路径,将参考图像的空间特征与自注意力图注入目标图像以生成事件。为推动此新任务,我们构建了两个评估基准:SWiG-Event 和 Real-Event。大量实验与消融分析证明了 FreeEvent 的有效性。
原文摘要 · Abstract (English)
Customized Image Generation, generating customized images with user-specified concepts, has raised significant attention due to its creativity and novelty. With impressive progress achieved in subject customization, some pioneer works further explored the customization of action and interaction beyond entity (i.e., human, animal, and object) appearance. However, these approaches only focus on basic actions and interactions between two entities, and their effects are limited by insufficient ''exactly same'' reference images. To extend customized image generation to more complex scenes for general real-world applications, we propose a new task: event-customized image generation. Given a single reference image, we define the ''event'' as all specific actions, poses, relations, or interactions between different entities in the scene. This task aims at accurately capturing the complex event and generating customized images with various target entities. To solve this task, we proposed a novel training-free event customization method: FreeEvent. Specifically, FreeEvent introduces two extra paths alongside the general diffusion denoising process: 1) Entity switching path: it applies cross-attention guidance and regulation for target entity generation. 2) Event transferring path: it injects the spatial feature and self-attention maps from the reference image to the target image for event generation. To further facilitate this new task, we collected two evaluation benchmarks: SWiG-Event and Real-Event. Extensive experiments and ablations have demonstrated the effectiveness of FreeEvent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。