用3D高斯点云实现多说话人音频驱动的唇同步视频生成
GenSync: A Generalized Talking Head Framework for Audio-driven Multi-Subject Lip-Sync using 3D Gaussian Splatting
- 通过解耦身份特征与音频表示，统一建模多个说话人
- 训练速度比顶尖模型快6.8倍，保持高唇同步精度
- 适合需要快速生成多角色口型同步视频的场景
我们提出GenSync，一种基于3D高斯点云的多身份音频驱动唇同步视频生成框架。与大多数需为每个身份单独训练模型的现有3D方法不同，GenSync学习一个统一网络，可为多个说话人生成唇同步视频。通过引入解耦模块，该方法将身份特异性特征与音频表征分离，实现高效的多身份视频合成。这一设计显著降低计算开销，在保持高唇同步准确率和视觉质量的同时，训练速度相较当前最优模型提升6.8倍。
原文摘要 · Abstract (English)
We introduce GenSync, a novel framework for multi-identity lip-synced video synthesis using 3D Gaussian Splatting. Unlike most existing 3D methods that require training a new model for each identity , GenSync learns a unified network that synthesizes lip-synced videos for multiple speakers. By incorporating a Disentanglement Module, our approach separates identity-specific features from audio representations, enabling efficient multi-identity video synthesis. This design reduces computational overhead and achieves 6.8x faster training compared to state-of-the-art models, while maintaining high lip-sync accuracy and visual quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。