解决视频生成延迟与连贯性难题,实现高保真实时人脸动画
Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation
- 分块生成+缓存参考图像的键值状态,保持身份一致
- 通过重叠块传递时空线索,消除帧间运动断裂
- 动态提前更新参考帧时间坐标,保障长期一致性
实时人脸动画对虚拟助手和直播虚拟形象等交互应用至关重要,需具备高视觉保真度、时间连贯性、极低延迟及对动态输入(如参考图、驱动信号)的响应能力。尽管基于扩散模型的方法质量优异,但其非因果特性阻碍流式部署。因果自回归视频生成虽支持逐帧高效生成,却存在误差累积、分块边界运动不连续、长期一致性下降等问题。本文提出名为Knot Forcing的新流式框架,通过三项关键设计解决上述挑战:(1) 基于参考图像缓存键值状态的分块生成策略,结合滑动窗口注意力实现局部时序建模;(2) 时序结点模块通过相邻分块重叠与图像到视频条件引导,平滑分块间运动过渡;(3) “提前运行”机制在推理中动态更新参考帧的时间坐标,使其语义上下文始终领先当前生成帧,以维持长期一致性。Knot Forcing 实现了高保真、时间连贯且可无限序列交互的人脸动画,在消费级GPU上达到实时性能,具有强视觉稳定性。
原文摘要 · Abstract (English)
Real-time portrait animation is essential for interactive applications such as virtual assistants and live avatars, requiring high visual fidelity, temporal coherence, ultra-low latency, and responsive control from dynamic inputs like reference images and driving signals. While diffusion-based models achieve strong quality, their non-causal nature hinders streaming deployment. Causal autoregressive video generation approaches enable efficient frame-by-frame generation but suffer from error accumulation, motion discontinuities at chunk boundaries, and degraded long-term consistency. In this work, we present a novel streaming framework named Knot Forcing for real-time portrait animation that addresses these challenges through three key designs: (1) a chunk-wise generation strategy with global identity preservation via cached KV states of the reference image and local temporal modeling using sliding window attention; (2) a temporal knot module that overlaps adjacent chunks and propagates spatio-temporal cues via image-to-video conditioning to smooth inter-chunk motion transitions; and (3) A "running ahead" mechanism that dynamically updates the reference frame's temporal coordinate during inference, keeping its semantic context ahead of the current rollout frame to support long-term coherence. Knot Forcing enables high-fidelity, temporally consistent, and interactive portrait animation over infinite sequences, achieving real-time performance with strong visual stability on consumer-grade GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。