arXiv:2411.17440cs.CVcs.MM2024-11CVPR被引 164

用频率分解让生成视频保持人脸一致,无需微调。

Identity-Preserving Text-to-Video Generation by Frequency Decomposition

论文配图:Identity-Preserving Text-to-Video Generation by Frequency Decomposition
图 1 · 摘自论文原文
  • 在频域分离人脸全局与细节特征,分别控制。
  • 无需微调即可生成高保真、身份一致的视频。
  • 适合需要稳定人物形象的视频生成场景。

身份保持的文本到视频生成旨在创建高保真且人物身份一致的视频,是视频生成中的重要任务,但仍是开放难题。本文在两个未被解决的方向上推进:(1) 无需繁琐微调的免调参流程;(2) 基于DiT的频域感知身份控制机制。提出ConsisID,一种免调参的基于DiT的可控身份保持文本到视频模型。受扩散变压器频域分析研究启发,将身份控制信号引入频域,将人脸特征分解为低频全局特征与高频局部特征。首先,从低频视角出发,设计全局人脸提取器,将参考图像和面部关键点编码至潜在空间,生成富含低频信息的特征,并注入网络浅层以缓解DiT训练挑战。其次,从高频视角出发,设计局部人脸提取器,捕捉高频细节并注入到Transformer块中,增强模型对细微特征的保留能力。提出分层训练策略,利用频域信息实现身份保持,将预训练视频生成模型转化为IPT2V模型。大量实验表明,该频域感知启发式方案为DiT模型提供了最优控制解。得益于该方案,ConsisID可生成高质量、身份一致的视频,推动了更有效的IPT2V发展。代码:https://github.com/PKU-YuanGroup/ConsisID。

原文摘要 · Abstract (English)

Identity-preserving text-to-video (IPT2V) generation aims to create high-fidelity videos with consistent human identity. It is an important task in video generation but remains an open problem for generative models. This paper pushes the technical frontier of IPT2V in two directions that have not been resolved in literature: (1) A tuning-free pipeline without tedious case-by-case finetuning, and (2) A frequency-aware heuristic identity-preserving DiT-based control scheme. We propose ConsisID, a tuning-free DiT-based controllable IPT2V model to keep human identity consistent in the generated video. Inspired by prior findings in frequency analysis of diffusion transformers, it employs identity-control signals in the frequency domain, where facial features can be decomposed into low-frequency global features and high-frequency intrinsic features. First, from a low-frequency perspective, we introduce a global facial extractor, which encodes reference images and facial key points into a latent space, generating features enriched with low-frequency information. These features are then integrated into shallow layers of the network to alleviate training challenges associated with DiT. Second, from a high-frequency perspective, we design a local facial extractor to capture high-frequency details and inject them into transformer blocks, enhancing the model's ability to preserve fine-grained features. We propose a hierarchical training strategy to leverage frequency information for identity preservation, transforming a vanilla pre-trained video generation model into an IPT2V model. Extensive experiments demonstrate that our frequency-aware heuristic scheme provides an optimal control solution for DiT-based models. Thanks to this scheme, our ConsisID generates high-quality, identity-preserving videos, making strides towards more effective IPT2V. Code: https://github.com/PKU-YuanGroup/ConsisID.

视频生成身份保持频域控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。