arXiv:2412.14531cs.CV2024-12被引 1

用空间条件扩散模型实现人像生成一致性,无需逐个微调新身份。

Consistent Human Image and Video Generation with Spatially Conditioned Diffusion

  • 将生成任务转为带空间条件的修复问题,统一网络处理外观与姿态。
  • 在未见人物和姿势上表现良好,无需额外实例微调。
  • 双阶段设计提升效率,适合需要灵活控制的人像视频创作。

一致的人像与视频生成旨在生成新姿态图像或视频的同时保持参考图像的外观一致性,对低成本视觉内容创作至关重要。现有基于扩散模型的方法通常依赖独立网络提取参考外观特征并生成目标视觉内容,导致参考与目标间存在不一致的域差距。本文将该任务建模为空间条件修复问题,通过在统一去噪网络中以参考特征指导姿态一致的目标生成,缓解了域差距。此外,引入因果特征交互机制:参考特征仅可自查询,目标特征可同时访问参考与自身信息。为提升计算效率与灵活性,实际实现中将生成过程分解为两阶段:参考外观提取与条件化目标生成,二者共享单一去噪网络,交互仅限于自注意力层。该方法实现了对生成人像外观的灵活控制。通过对现有基础扩散模型在人体视频数据上微调,本方法展现出对未见人物身份与姿态的强大泛化能力,无需额外实例微调。实验验证了其有效性,在一致人像与视频生成任务中达到与现有方法相当甚至更优的性能。

原文摘要 · Abstract (English)

Consistent human-centric image and video synthesis aims to generate images or videos with new poses while preserving appearance consistency with a given reference image, which is crucial for low-cost visual content creation. Recent advances based on diffusion models typically rely on separate networks for reference appearance feature extraction and target visual generation, leading to inconsistent domain gaps between references and targets. In this paper, we frame the task as a spatially-conditioned inpainting problem, where the target image is inpainted to maintain appearance consistency with the reference. This approach enables the reference features to guide the generation of pose-compliant targets within a unified denoising network, thereby mitigating domain gaps. Additionally, to better maintain the reference appearance information, we impose a causal feature interaction framework, in which reference features can only query from themselves, while target features can query appearance information from both the reference and the target. To further enhance computational efficiency and flexibility, in practical implementation, we decompose the spatially-conditioned generation process into two stages: reference appearance extraction and conditioned target generation. Both stages share a single denoising network, with interactions restricted to self-attention layers. This proposed method ensures flexible control over the appearance of generated human images and videos. By fine-tuning existing base diffusion models on human video data, our method demonstrates strong generalization to unseen human identities and poses without requiring additional per-instance fine-tuning. Experimental results validate the effectiveness of our approach, showing competitive performance compared to existing methods for consistent human image and video synthesis.

扩散模型人像生成视频生成一致性控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。