arXiv:2608.18832cs.CV2026-08

用参考帧纹理直接拼接,实现实时高保真口型同步。

EfficientSync: Real-Time Lip Synchronization via Deformation-Based Reference Texture Mixing

论文配图:EfficientSync: Real-Time Lip Synchronization via Deformation-Based Reference Texture Mixing
图 1 · 摘自论文原文
  • 通过通道加权选择参考帧纹理,避免重生成
  • 166帧每秒,保持身份一致性和真实唇纹细节
  • 适合需要低延迟口型同步的影视/虚拟人应用

音频驱动的口型同步需使说话人脸视频的嘴部动作匹配输入音频,同时保持头部姿态、身份和背景不变。现有方法多采用全脸重建,依赖重型GAN或扩散模型解码器,导致延迟高,并常虚构口腔内部细节(如牙齿、唇纹)。我们指出,身份保持不佳的关键并非参考帧不足,而是缺乏有效保留已有真实纹理的机制。为此提出EfficientSync:一种基于形变的实时框架,直接保留参考帧纹理而非重新生成。首先,动态纹理混合器将多参考融合重构为通道级选择,在全局上下文中评估空间对齐的参考帧,通过通道加权求和聚合,低成本保持纹理完整性;其次,时空偏移自适应掩码将源帧分解为唇部生成条件与独立背景先验,抑制下脸信息泄露,实现无缝融合;第三,零开销的STAR采样预先筛选最清晰且拓扑多样性最高的参考帧。在HDTF和VFHQ数据集上,达到166 FPS(单卡),视觉质量与身份保持均达当前最优。视频演示:https://alunaticat.github.io/EfficientSync/index.html。

原文摘要 · Abstract (English)

Audio-driven lip synchronization manipulates the mouth region of a talking-face video to match the driving audio while preserving head pose, identity, and background. Although the task is inherently local editing, prevailing approaches reconstruct the entire lower face with heavy GAN- or diffusion-based decoders, incurring substantial latency and, more critically, hallucinating intra-oral details such as teeth and lip wrinkles instead of preserving authentic textures. We contend that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain. We therefore present EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them. First, the Dynamic Texture Mixer reformulates multi-reference fusion as channel-wise selection, evaluating each spatially aligned reference in a global context and aggregating them by channel-wise weighted summation, preserving textural integrity at low cost. Second, Spatio-Temporal Shifted Adaptive Masking decomposes the source frame into lip-generation conditions and an independent background prior, suppressing lower-face leakage while blending the synthesized mouth seamlessly into the background. Third, STAR Sampling, a zero-overhead pre-processing step, retrieves the sharpest and most topologically diverse reference frames. Experiments on HDTF and VFHQ show state-of-the-art visual quality and identity preservation at 166 FPS on a single GPU. Video demos: https://alunaticat.github.io/EfficientSync/index.html.

口型同步实时生成纹理保留轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。