用重建代替生成，100帧实时无遮罩唇动同步
FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANs
- 两阶段设计：先重构图像，再由语音预测嘴型向量
- 单卡运行超100FPS，视觉质量媲美大模型
- 无需遮罩和对抗训练，稳定且适合实时应用
我们提出FlashLips，一种两阶段、无遮罩的唇同步系统，将嘴部控制与图像渲染解耦，实现近实时性能。其基于改进的U-Net在单张GPU上运行速度超过100 FPS，同时保持与更大规模先进模型相当的视觉质量。第一阶段为轻量级单步潜空间编辑器，仅使用重建损失，通过参考身份、带掩码的目标帧和低维嘴型向量重构图像；为避免推理时显式遮罩，采用自监督策略，利用嘴部变形的伪真值训练网络定位嘴部修改区域并保留其余内容。第二阶段为音频到嘴型向量的Transformer，使用流匹配目标进行训练。整体流程结合确定性重建与鲁棒音频控制，实现高感知质量与超实时速度。
原文摘要 · Abstract (English)
We present FlashLips, a two-stage, mask-free lip-sync system that decouples lips control from rendering and achieves real-time performance, with our U-Net variant running at over 100 FPS on a single GPU, while matching the visual quality of larger state-of-the-art models. Stage 1 is a compact, one-step latent-space editor that reconstructs an image using a reference identity, a masked target frame, and a low-dimensional lips-pose vector, trained purely with reconstruction losses - no GANs or diffusion. To remove explicit masks at inference, we use self-supervision via mouth-altered target variants as pseudo ground truth, teaching the network to localize lip edits while preserving the rest. Stage 2 is an audio-to-pose transformer trained with a flow-matching objective to predict lips-pose vectors from speech. Together, these stages form a simple and stable pipeline that combines deterministic reconstruction with robust audio control, delivering high perceptual quality and faster-than-real-time speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。