用一张图+一段文字,让任意人物做指定动作,无需标注数据。
ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models
- 基于扩散模型,通过频域引导实现人物与动作的联合控制。
- 仅需少量微调,就能生成符合描述的个性化图像,效果优于基线。
- 适合想快速生成特定人物做特定动作图像的研究者或创作者。
我们提出ImPoster,一种新型算法,可生成目标图像:让源图像中的主体执行驱动图像中任意类别的主体所完成的动作。输入包括一对图像(源图像和驱动图像)以及两者的文本描述。该方法完全无监督,无需关键点或姿态标注。基于预训练的文本到图像潜空间扩散模型,通过少量迭代微调学习源图像与驱动图像的特征。推理时,采用分步文本提示策略:先沿驱动图像的图像流形方向去噪,再沿目标图像文本描述的流形方向去噪。我们提出一种新颖的图像频域引导机制,在每一步去噪过程中引导生成结果趋近于源主体与驱动动作的流形。该频域引导方法基于图像的频域特性设计。我们在多种源-驱动图像对上进行了广泛评估,结果表明优于现有基线。据我们所知,ImPoster是首个同时实现主体驱动与动作驱动图像个性化的方案。代码与数据见https://github.com/divyakraman/ImPosterDiffusion2024。
原文摘要 · Abstract (English)
We present ImPoster, a novel algorithm for generating a target image of a 'source' subject performing a 'driving' action. The inputs to our algorithm are a single pair of a source image with the subject that we wish to edit and a driving image with a subject of an arbitrary class performing the driving action, along with the text descriptions of the two images. Our approach is completely unsupervised and does not require any access to additional annotations like keypoints or pose. Our approach builds on a pretrained text-to-image latent diffusion model and learns the characteristics of the source and the driving image by finetuning the diffusion model for a small number of iterations. At inference time, ImPoster performs step-wise text prompting i.e. it denoises by first moving in the direction of the image manifold corresponding to the driving image followed by the direction of the image manifold corresponding to the text description of the desired target image. We propose a novel diffusion guidance formulation, image frequency guidance, to steer the generation towards the manifold of the source subject and the driving action at every step of the inference denoising. Our frequency guidance formulations are derived from the frequency domain properties of images. We extensively evaluate ImPoster on a diverse set of source-driving image pairs to demonstrate improvements over baselines. To the best of our knowledge, ImPoster is the first approach towards achieving both subject-driven as well as action-driven image personalization. Code and data is available at https://github.com/divyakraman/ImPosterDiffusion2024.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。