arXiv:2409.02653cs.CV2024-09被引 3

用深度图控制生成任意物体的姿势,且不改变物体形状。

Skip-and-Play: Depth-Driven Pose-Preserved Image Generation for Any Objects

  • 通过分析深度条件ControlNet的三部分影响,选择性跳过组件
  • 实现姿势控制同时保持物体形状不变
  • 支持不同对象间的跨物种生成,适合复杂场景应用

扩散模型的兴起使得仅通过文本即可生成高质量图像,但其可控性仍有待提升。现有方法依赖相机参数或关键点控制姿势,但仅适用于特定物体或视角(如正面)。相机参数法因3D数据集小导致生成失真;关键点法难以获取教堂等物体或背面视角的可靠关键点。为此,本文提出基于深度图的姿势控制,因其可由单模型获得,不受物体和视角限制。然而深度图会同时影响姿态与形状。为此,我们设计了跳过-播放(Skip-and-Play, SnP),通过对深度条件ControlNet三个组件的影响分析,有选择地跳过部分模块,在保留姿势信息的同时缓解形状依赖。实验表明SnP优于基线,能生成多样化物体与姿势的图像。尤其值得注意的是,即使条件中的物体(如马)与提示中的物体(如刺猬)不同,SnP仍可成功生成。

原文摘要 · Abstract (English)

The emergence of diffusion models has enabled the generation of diverse high-quality images solely from text, prompting subsequent efforts to enhance the controllability of these models. Despite the improvement in controllability, pose control remains limited to specific objects (e.g., humans) or poses (e.g., frontal view) due to the fact that pose is generally controlled via camera parameters (e.g., rotation angle) or keypoints (e.g., eyes, nose). Specifically, camera parameters-conditional pose control models generate unrealistic images depending on the object, owing to the small size of 3D datasets for training. Also, keypoint-based approaches encounter challenges in acquiring reliable keypoints for various objects (e.g., church) or poses (e.g., back view). To address these limitations, we propose depth-based pose control, as depth maps are easily obtainable from a single depth estimation model regardless of objects and poses, unlike camera parameters and keypoints. However, depth-based pose control confronts issues of shape dependency, as depth maps influence not only the pose but also the shape of the generated images. To tackle this issue, we propose Skip-and-Play (SnP), designed via analysis of the impact of three components of depth-conditional ControlNet on the pose and the shape of the generated images. To be specific, based on the analysis, we selectively skip parts of the components to mitigate shape dependency on the depth map while preserving the pose. Through various experiments, we demonstrate the superiority of SnP over baselines and showcase the ability of SnP to generate images of diverse objects and poses. Remarkably, SnP exhibits the ability to generate images even when the objects in the condition (e.g., a horse) and the prompt (e.g., a hedgehog) differ from each other.

图像生成深度图姿势控制扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。