让视频生成模型实时运行，同时保持高质量控制能力。
Adapting VACE for Real-Time Autoregressive Video Diffusion
- 将参考帧移入独立条件路径，支持流式处理和因果注意力
- 1.3B与14B模型延迟仅增加20%-30%，显存占用几乎不变
- 适合需要低延迟视频生成的场景，如直播或交互应用
我们描述了对VACE（Video All-in-one Creation and Editing）的改进，使其适用于实时自回归视频生成。VACE提供统一的视频控制能力（参考引导、结构条件、图像修复和时序扩展），但依赖于全序列双向注意力，无法适配需固定分块大小和因果注意力的流式处理管道。关键改进是将参考帧从扩散潜空间转移到并行条件路径，保留固定分块大小和键值缓存机制。该方法复用预训练VACE权重，无需额外训练。在1.3B和14B模型规模下，结构控制和图像修复带来的延迟增加为20%-30%，相对于基础模型的显存开销可忽略。由于因果注意力限制，参考帧到视频的保真度相比批处理VACE显著下降。参考代码已开源：https://github.com/daydreamlive/scope。
原文摘要 · Abstract (English)
We describe an adaptation of VACE (Video All-in-one Creation and Editing) for real-time autoregressive video generation. VACE provides unified video control (reference guidance, structural conditioning, inpainting, and temporal extension) but assumes bidirectional attention over full sequences, making it incompatible with streaming pipelines that require fixed chunk sizes and causal attention. The key modification moves reference frames from the diffusion latent space into a parallel conditioning pathway, preserving the fixed chunk sizes and KV caching that autoregressive models require. This adaptation reuses existing pretrained VACE weights without additional training. Across 1.3B and 14B model scales, VACE adds 20-30% latency overhead for structural control and inpainting, with negligible VRAM cost relative to the base model. Reference-to-video fidelity is severely degraded compared to batch VACE due to causal attention constraints. A reference implementation is available at https://github.com/daydreamlive/scope.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。