输入一段视频和文字描述,实时生成可驱动的卡通头像。
TextToon: Real-Time Text Toonify Head Avatar from Single Video
- 用条件嵌入三平面建模面部风格,结合高斯变形场实现真实与卡通融合。
- 在48帧/秒(GPU)和15-18帧/秒(手机)下实现实时驱动,支持任意身份动画。
- 通过自适应像素迁移网络和对比学习提升图像质量,适合消费级应用。
我们提出TextToon,一种从单段视频和文字指令生成可驱动卡通头像的方法。给定一个短的单目视频序列和关于头像风格的文字描述,模型可生成高保真、可实时驱动的卡通化头像,其动画可由任意身份的视频控制。现有方法严重依赖多视角建模恢复几何,以静态纹理嵌入呈现,导致控制受限,且多视角输入难以部署于实际场景。为此,我们采用条件嵌入三平面,在高斯变形场中学习真实而风格化的面部表征。同时,通过引入自适应像素迁移神经网络并利用块感知对比学习,扩展了3D高斯溅射的风格化能力,实现高质量图像生成。为推动应用落地,我们构建了一个实时系统,在GPU上达到48 FPS,移动端达15-18 FPS。大量实验表明,本方法在生成质量与实时动画方面优于现有方法。
原文摘要 · Abstract (English)
We propose TextToon, a method to generate a drivable toonified avatar. Given a short monocular video sequence and a written instruction about the avatar style, our model can generate a high-fidelity toonified avatar that can be driven in real-time by another video with arbitrary identities. Existing related works heavily rely on multi-view modeling to recover geometry via texture embeddings, presented in a static manner, leading to control limitations. The multi-view video input also makes it difficult to deploy these models in real-world applications. To address these issues, we adopt a conditional embedding Tri-plane to learn realistic and stylized facial representations in a Gaussian deformation field. Additionally, we expand the stylization capabilities of 3D Gaussian Splatting by introducing an adaptive pixel-translation neural network and leveraging patch-aware contrastive learning to achieve high-quality images. To push our work into consumer applications, we develop a real-time system that can operate at 48 FPS on a GPU machine and 15-18 FPS on a mobile machine. Extensive experiments demonstrate the efficacy of our approach in generating textual avatars over existing methods in terms of quality and real-time animation. Please refer to our project page for more details: https://songluchuan.github.io/TextToon/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。