通过时空剪枝加速自回归视频生成,提升效率且几乎不损失质量。
FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesis
- 基于时空相似性识别冗余区域,跳过无需细化的计算。
- 在InfinityStar上实现2.01倍加速,PSNR达28.29,性能下降不足1%。
- 无需训练,适合追求高效高质视频生成的研究与应用者。
视觉自回归建模(VAR)已成为扩散模型的高效替代方案,合成质量相当。然而,当该范式扩展至时空自回归建模(STAR)用于视频生成时,分辨率和帧数的提升导致“标记爆炸”,在最终精炼阶段造成巨大计算瓶颈。为此,我们提出FastSTAR,一种无需训练的加速框架,专为高质量视频生成设计。核心方法为时空标记剪枝,通过整合两项专用机制:(1) 空间相似性,评估多尺度结构收敛性,跳过冗余区域的计算;(2) 时间相似性,通过特征级变化分析识别活跃运动轨迹。结合局部更新机制,FastSTAR仅对未收敛区域进行精炼,保持流畅运动的同时规避冗余计算。在InfinityStar上的实验表明,FastSTAR实现最高2.01倍加速,PSNR达28.29,性能下降低于1%,验证了其在STAR视频生成中卓越的效率-质量平衡。
原文摘要 · Abstract (English)
Visual Autoregressive modeling (VAR) has emerged as a highly efficient alternative to diffusion-based frameworks, achieving comparable synthesis quality. However, as this paradigm extends to Spacetime Autoregressive modeling (STAR) for video generation, scaling resolution and frame counts leads to a "token explosion" that creates a massive computational bottleneck in the final refinement stages. To address this, we propose FastSTAR, a training-free acceleration framework designed for high-quality video generation. Our core method, Spatiotemporal Token Pruning, identifies essential tokens by integrating two specialized terms: (1) Spatial similarity, which evaluates structural convergence across hierarchical scales to skip computations in regions where further refinement becomes redundant, and (2) Temporal similarity, which identifies active motion trajectories by assessing feature-level variations relative to the preceding clip. Combined with a Partial Update mechanism, FastSTAR ensures that only non-converged regions are refined, maintaining fluid motion while bypassing redundant computations. Experimental results on InfinityStar demonstrate that FastSTAR achieves up to a 2.01x speedup with a PSNR of 28.29 and less than 1% performance degradation, proving a superior efficiency-quality trade-off for STAR-based video synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。