StreamDiT实现每秒16帧的实时文本生成视频，支持交互式应用。
StreamDiT: Real-Time Streaming Text-to-Video Generation
- 采用可变时间嵌入与窗口注意力的adaLN DiT架构，结合流动匹配训练。
- 40亿参数模型经多步蒸馏后，在单卡上达16 FPS，512p分辨率实时生成。
- 适用于流式、交互式生成及视频转视频等实时场景，效果获人工评估验证。
近期基于大规模参数变压器扩散模型在文本到视频（T2V）生成方面取得显著进展，能够生成高质量视频。然而，现有模型通常仅能离线生成短片段，限制了其在交互式和实时应用中的使用。本文提出StreamDiT，一种流式视频生成模型。其训练基于添加移动缓存的流动匹配方法，并设计了不同帧分块方案的混合训练策略，以提升内容一致性和视觉质量。模型采用具有可变时间嵌入和窗口注意力的adaLN DiT结构。为实践该方法，我们训练了一个40亿参数的StreamDiT模型，并提出了针对StreamDiT定制的多步蒸馏方法。采样蒸馏在选定分块方案的每个片段中进行，蒸馏后总函数求值次数（NFEs）降至缓存块数。最终，蒸馏模型在单张GPU上实现16 FPS的实时性能，可生成512p分辨率的视频流。我们通过定量指标与人工评估验证方法有效性。该模型支持流式生成、交互式生成及视频到视频等实时应用场景。更多视频示例见项目主页：https://cumulo-autumn.github.io/StreamDiT/
原文摘要 · Abstract (English)
Recently, great progress has been achieved in text-to-video (T2V) generation by scaling transformer-based diffusion models to billions of parameters, which can generate high-quality videos. However, existing models typically produce only short clips offline, restricting their use cases in interactive and real-time applications. This paper addresses these challenges by proposing StreamDiT, a streaming video generation model. StreamDiT training is based on flow matching by adding a moving buffer. We design mixed training with different partitioning schemes of buffered frames to boost both content consistency and visual quality. StreamDiT modeling is based on adaLN DiT with varying time embedding and window attention. To practice the proposed method, we train a StreamDiT model with 4B parameters. In addition, we propose a multistep distillation method tailored for StreamDiT. Sampling distillation is performed in each segment of a chosen partitioning scheme. After distillation, the total number of function evaluations (NFEs) is reduced to the number of chunks in a buffer. Finally, our distilled model reaches real-time performance at 16 FPS on one GPU, which can generate video streams at 512p resolution. We evaluate our method through both quantitative metrics and human evaluation. Our model enables real-time applications, e.g. streaming generation, interactive generation, and video-to-video. We provide video results and more examples in our project website: https://cumulo-autumn.github.io/StreamDiT/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。