arXiv:2606.11180cs.CV2026-06

用两步去噪实现实时唇同步,速度比现有方法快39倍。

Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization

论文配图:Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization
图 1 · 摘自论文原文
  • 提出自回归扩散框架,仅需两步去噪生成每段视频。
  • 1.3B模型达31帧/秒,比同类模型快17.6倍,14B模型快39.8倍。
  • 通过轨迹分析设计奖励机制,兼顾真实度与同步性,适合实时应用。

基于扩散的唇同步模型虽视觉质量高、音画对齐好,但全序列双向注意力和多步去噪使其难以实时推理。本文提出Lip Forcing,据我们所知首个用于视频到视频唇同步的自回归扩散方法,将140亿参数的音频条件双向视频扩散教师模型蒸馏为因果学生模型。推理时,学生仅需两步去噪即可生成每段视频,无需推理时的分类引导(CFG),实现真正实时唇同步。唇同步专用的教师轨迹分析揭示了无CFG预测更保参考真实度,而CFG引导预测则在中轨迹区间更优同步性。Lip Forcing据此设计三个分析驱动组件:同步窗口去噪模块(Sync-Window DMD)、两步推理调度和基于SyncNet的奖励函数。我们在两个学生规模上验证该方法,均来自140亿教师模型。13亿参数学生达到31帧/秒,较同规模双向模型快17.6倍;140亿参数学生是目前报道的最大扩散模型用于视频到视频唇同步,其运行速度较教师快39.8倍,且保持相近参考真实度。两种规模下首帧生成时间均低于毫秒级,远低于所有扩散基线。

原文摘要 · Abstract (English)

Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.

唇同步扩散模型实时生成自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。