通过时空位置编码优化，实现视频生成近实时推理。
Accelerating Video Generation Inference with Sequential-Parallel 3D Positional Encoding Using a Global Time Index
- 提出序列并行3D位置编码，降低跨显卡通信开销。
- 八卡A800集群下首帧延迟低于1秒，5秒480P视频生成提速1.58倍。
- 适合需要实时交互的视频生成应用，如AI创作工具。
基于DiT的视频生成模型在长视频合成和实时推理中存在固有瓶颈，主要源于全时空注意力机制带来的O(N²)内存爆炸和高首帧延迟。为此，我们对因果自回归视频生成流水线实施系统级推理优化，将Self-Forcing框架适配至序列并行推理，并提出一种新的因果旋转位置编码序列并行变体（Causal-RoPE SP），实现局部化计算与减少跨节点通信。同时，通过算子融合和位置编码预计算优化计算与通信流水线。在八卡A800集群上的实验表明，该优化系统在保持生成质量的同时，实现亚秒级首帧延迟和接近实时的推理速度。生成5秒480P视频时，速度提升达1.58倍，为实时交互式应用提供有效支持。
原文摘要 · Abstract (English)
Diffusion Transformer (DiT)-based video generation models inherently suffer from bottlenecks in long video synthesis and real-time inference, which can be attributed to the use of full spatiotemporal attention. Specifically, this mechanism leads to explosive O(N^2) memory consumption and high first-frame latency. To address these issues, we implement system-level inference optimizations for a causal autoregressive video generation pipeline. We adapt the Self-Forcing causal autoregressive framework to sequence parallel inference and implement a sequence-parallel variant of the causal rotary position embedding which we refer to as Causal-RoPE SP. This adaptation enables localized computation and reduces cross-rank communication in sequence parallel execution. In addition, computation and communication pipelines are optimized through operator fusion and RoPE precomputation. Experiments conducted on an eight GPU A800 cluster show that the optimized system achieves comparable generation quality, sub-second first-frame latency, and near real-time inference speed. For generating five second 480P videos, a 1.58x speedup is achieved, thereby providing effective support for real-time interactive applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。