用两张参考图实现可控人脸视频编辑,稳定生成任意长度视频。
Qffusion: Controllable Portrait Video Editing via Quadrant-Grid Attention Learning
- 通过四格网格排列参考图与面部条件的潜在编码,融合双模态特征。
- 仅修改Stable Diffusion输入格式,无需额外网络即可稳定生成视频。
- 支持任意长度视频生成,适合需要灵活控制的人脸编辑场景。
本文提出Qffusion,一种基于双帧引导的人脸视频编辑框架。受‘为编辑而动画’设计原则启发,模型从两张静态参考图中训练而成,推理时通过调整起始与结束帧作为参考,即可轻松实现人脸视频编辑。利用Stable Diffusion的强大生成能力,提出四格网格排列(QGA)方案,将两张参考图及四种面部条件的潜在编码以四格形式组织。随后在该结构下进行自注意力融合,联合建模外观与时间信息。Qffusion无需额外网络或复杂训练阶段,仅通过修改Stable Diffusion输入格式即可实现稳定视频生成。进一步提出四格传播(QGP)推理策略,通过递归处理参考帧与条件帧,具备生成任意长度视频的独特优势。大量实验表明,Qffusion在人脸视频编辑任务上持续优于现有最优方法。
原文摘要 · Abstract (English)
This paper presents Qffusion, a dual-frame-guided framework for portrait video editing. Specifically, we consider a design principle of ``animation for editing'', and train Qffusion as a general animation framework from two still reference images while we can use it for portrait video editing easily by applying modified start and end frames as references during inference. Leveraging the powerful generative power of Stable Diffusion, we propose a Quadrant-grid Arrangement (QGA) scheme for latent re-arrangement, which arranges the latent codes of two reference images and that of four facial conditions into a four-grid fashion, separately. Then, we fuse features of these two modalities and use self-attention for both appearance and temporal learning, where representations at different times are jointly modeled under QGA. Our Qffusion can achieve stable video editing without additional networks or complex training stages, where only the input format of Stable Diffusion is modified. Further, we propose a Quadrant-grid Propagation (QGP) inference strategy, which enjoys a unique advantage on stable arbitrary-length video generation by processing reference and condition frames recursively. Through extensive experiments, Qffusion consistently outperforms state-of-the-art techniques on portrait video editing. Project page: https://qffusion.github.io/page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。