Astraea通过智能选择视频帧令牌,实现高效加速且保持画质。
Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformers

- 基于进化算法自动分配令牌预算,动态优化每一步的计算量。
- 单卡提速2.4倍,8卡可达13.2倍,画质损失低于0.5%。
- 适合需要快速部署高分辨率视频生成的开发者与研究者。
视频扩散变换器(vDiTs)在文本到视频生成中取得显著进展,但其高计算需求严重制约实际部署。现有加速方法多依赖启发式规则,适用性受限。我们提出Astraea框架,可在性能目标下搜索vDiT生成任务的近优配置。核心包括轻量级令牌选择机制和内存高效、GPU友好的稀疏注意力策略,实现执行时间线性节省,对生成质量影响极小。为确定不同时间步的最优令牌缩减比例,进一步设计基于经典进化算法的搜索框架,有效分配令牌预算。实验表明,Astraea在单张GPU上实现最高2.4倍推理加速,在8张GPU上达13.2倍加速,同时相比当前最佳方法视频质量提升超10 dB(VBench评分损失小于0.5%)。
原文摘要 · Abstract (English)
Video diffusion transformers (vDiTs) have made tremendous progress in text-to-video generation, but their high compute demands pose a major challenge for practical deployment. While studies propose acceleration methods to reduce workload at various granularities, they often rely on heuristics, limiting their applicability. We introduce Astraea, a framework that searches for near-optimal configurations for vDiT-based video generation under a performance target. At its core, Astraea proposes a lightweight token selection mechanism and a memory-efficient, GPU-friendly sparse attention strategy, enabling linear savings on execution time with minimal impact on generation quality. Meanwhile, to determine optimal token reduction for different timesteps, we further design a search framework that leverages a classic evolutionary algorithm to automatically determine the distribution of the token budget effectively. Together, Astraea achieves up to 2.4$\times$ inference speedup on a single GPU with great scalability (up to 13.2$\times$ speedup on 8 GPUs) while achieving up to over 10~dB video quality compared to the state-of-the-art methods ($<$0.5\% loss on VBench compared to baselines).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。