实时生成双人对话视频，延迟低于100毫秒，支持即时非语言反馈。
DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Model
- 基于流匹配的自回归框架，支持流式处理音频输入。
- 引入前瞻模块，仅用60毫秒未来信息提升画质，帧延迟仅34毫秒。
- 适合需要实时交互的虚拟会议、数字人对话等场景。
生成逼真的双人对话头像视频需要极低延迟。现有分块方法需完整非因果上下文窗口，造成显著延迟，严重影响真实听众的即时非语言反馈。为此，我们提出DyStream，一种基于流匹配的自回归模型，可从说话者和听者音频实时生成视频。方法包含两个关键设计：(1) 采用适配流式的自回归框架，结合流匹配头部进行概率建模；(2) 提出一种带前瞻模块的因果编码器，融入短时未来上下文（如60毫秒）以提升质量，同时保持低延迟。分析表明，该方法显著优于其他因果策略，包括蒸馏和生成式编码器。大量实验显示，DyStream每帧生成耗时34毫秒，系统总延迟控制在100毫秒以内。此外，在HDTF数据集上，离线与在线唇形同步得分分别达到8.13和7.61，达到当前最优水平。模型、权重与代码均已公开。
原文摘要 · Abstract (English)
Generating realistic, dyadic talking head video requires ultra-low latency. Existing chunk-based methods require full non-causal context windows, introducing significant delays. This high latency critically prevents the immediate, non-verbal feedback required for a realistic listener. To address this, we present DyStream, a flow matching-based autoregressive model that could generate video in real-time from both speaker and listener audio. Our method contains two key designs: (1) we adopt a stream-friendly autoregressive framework with flow-matching heads for probabilistic modeling, and (2) We propose a causal encoder enhanced by a lookahead module to incorporate short future context (e.g., 60 ms) to improve quality while maintaining low latency. Our analysis shows this simple-and-effective method significantly surpass alternative causal strategies, including distillation and generative encoder. Extensive experiments show that DyStream could generate video within 34 ms per frame, guaranteeing the entire system latency remains under 100 ms. Besides, it achieves state-of-the-art lip-sync quality, with offline and online LipSync Confidence scores of 8.13 and 7.61 on HDTF, respectively. The model, weights and codes are available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。