arXiv:2409.09292cs.CV2024-09TPAMI被引 26

仅用一张人脸图和音频,就能生成个性化说话风格的视频。

StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads

  • 通过参考视频提取风格编码,统一控制表情与头部动作
  • 仅需一张人像图和一段音频,生成多样且逼真的说话视频
  • 适合需要个性化虚拟主播或角色配音的场景

每个人都有独特的面部表情和头部姿态风格,体现其个性化的说话方式。现有的一次性说话头生成方法无法捕捉这些个性化特征,因此在最终视频中难以呈现多样化的说话风格。为解决此问题,我们提出一种一次性风格可控的说话头生成方法,能够从参考说话视频中获取说话风格,并驱动单张肖像图以该风格配合另一段音频进行说话。我们的方法旨在统一框架下合成3D可变形模型(3DMM)的风格可控系数,包括面部表情和头部运动。具体而言,首先利用风格编码器从参考视频中提取期望的说话风格并转化为风格码;随后,采用风格感知解码器基于音频输入和风格码合成3DMM系数。解码阶段采用双分支结构,分别生成风格化面部表情系数与风格化头部运动系数。获得3DMM系数后,图像渲染器将其转换为特定人物的说话头视频。大量实验表明,本方法仅需一张肖像图和一个音频片段,即可生成视觉真实、风格多样的说话头视频。

原文摘要 · Abstract (English)

Individuals have unique facial expression and head pose styles that reflect their personalized speaking styles. Existing one-shot talking head methods cannot capture such personalized characteristics and therefore fail to produce diverse speaking styles in the final videos. To address this challenge, we propose a one-shot style-controllable talking face generation method that can obtain speaking styles from reference speaking videos and drive the one-shot portrait to speak with the reference speaking styles and another piece of audio. Our method aims to synthesize the style-controllable coefficients of a 3D Morphable Model (3DMM), including facial expressions and head movements, in a unified framework. Specifically, the proposed framework first leverages a style encoder to extract the desired speaking styles from the reference videos and transform them into style codes. Then, the framework uses a style-aware decoder to synthesize the coefficients of 3DMM from the audio input and style codes. During decoding, our framework adopts a two-branch architecture, which generates the stylized facial expression coefficients and stylized head movement coefficients, respectively. After obtaining the coefficients of 3DMM, an image renderer renders the expression coefficients into a specific person's talking-head video. Extensive experiments demonstrate that our method generates visually authentic talking head videos with diverse speaking styles from only one portrait image and an audio clip.

说话头生成风格控制3DMM个性化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。