14B模型实现实时长视频生成,无需加速技巧且不漂移。
Helios: Real Real-Time Long Video Generation Model
- 14B自回归扩散模型,统一支持文本/图像/视频生成
- 单卡H100达19.5帧/秒,生成分钟级视频无漂移
- 训练无需分布式框架,80GB显存可装4个模型
我们提出Helios,首个在单张NVIDIA H100 GPU上以19.5 FPS运行的140亿参数视频生成模型,支持分钟级生成且质量媲美强基线。突破体现在三方面:(1)无需自强迫、误差池或关键帧采样等抗漂移技巧,即可稳定生成长视频;(2)不依赖KV缓存、稀疏/线性注意力或量化等加速技术,实现真正实时生成;(3)训练无需并行或分片框架,可达到图像扩散模型级别的批量大小,80GB显存下最多容纳4个14B模型。Helios为140亿参数自回归扩散模型,采用统一输入表示,原生支持文本到视频、图像到视频、视频到视频任务。为缓解长视频漂移,我们分析典型失败模式,提出在训练中显式模拟漂移的简单有效策略,并从源头消除重复运动。为提升效率,大幅压缩历史与噪声上下文,减少采样步数,计算开销低于甚至相当于13亿参数模型。此外,引入基础设施级优化,加速推理与训练同时降低内存消耗。大量实验表明,Helios在短/长视频生成上持续优于此前方法。计划开源代码、基础模型及蒸馏模型,以推动社区发展。
原文摘要 · Abstract (English)
We introduce Helios, the first 14B video generation model that runs at 19.5 FPS on a single NVIDIA H100 GPU and supports minute-scale generation while matching the quality of a strong baseline. We make breakthroughs along three key dimensions: (1) robustness to long-video drifting without commonly used anti-drifting heuristics such as self-forcing, error-banks, or keyframe sampling; (2) real-time generation without standard acceleration techniques such as KV-cache, sparse/linear attention, or quantization; and (3) training without parallelism or sharding frameworks, enabling image-diffusion-scale batch sizes while fitting up to four 14B models within 80 GB of GPU memory. Specifically, Helios is a 14B autoregressive diffusion model with a unified input representation that natively supports T2V, I2V, and V2V tasks. To mitigate drifting in long-video generation, we characterize typical failure modes and propose simple yet effective training strategies that explicitly simulate drifting during training, while eliminating repetitive motion at its source. For efficiency, we heavily compress the historical and noisy context and reduce the number of sampling steps, yielding computational costs comparable to -- or lower than -- those of 1.3B video generative models. Moreover, we introduce infrastructure-level optimizations that accelerate both inference and training while reducing memory consumption. Extensive experiments demonstrate that Helios consistently outperforms prior methods on both short- and long-video generation. We plan to release the code, base model, and distilled model to support further development by the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。