通过缓存剪枝加速视觉自回归模型，大幅降低高分辨率生成耗时。
FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning
- 仅前向传播关键令牌，其余用历史缓存恢复
- 2K图像生成仅需1.5秒，内存占用15GB
- 适配高分辨率零样本生成，适合部署与推理优化
视觉自回归（VAR）建模因其面向多尺度预测的特性而受到关注。然而，现有方法在每个尺度步骤中处理完整的令牌图，导致复杂度和运行时间随图像分辨率显著上升。为解决此问题，我们提出FastVAR，一种用于VAR高效分辨率扩展的后训练加速方法。关键发现是：大部分延迟来自多数令牌已收敛的大尺度步骤。基于此，我们设计缓存令牌剪枝策略，仅对特定尺度的关键令牌进行前向传播，并利用前一尺度的缓存令牌恢复被剪枝的位置。该方法大幅减少前向传播的令牌数量，提升大分辨率下的效率。实验表明，FastVAR可在不损失性能（<1%）的前提下，使FlashAttention加速的VAR提速2.7倍。进一步拓展至高分辨率图像的零样本生成：单块NVIDIA 3090 GPU上，1.5秒内生成一张2K图像，内存占用15GB。代码已开源。
原文摘要 · Abstract (English)
Visual Autoregressive (VAR) modeling has gained popularity for its shift towards next-scale prediction. However, existing VAR paradigms process the entire token map at each scale step, leading to the complexity and runtime scaling dramatically with image resolution. To address this challenge, we propose FastVAR, a post-training acceleration method for efficient resolution scaling with VARs. Our key finding is that the majority of latency arises from the large-scale step where most tokens have already converged. Leveraging this observation, we develop the cached token pruning strategy that only forwards pivotal tokens for scale-specific modeling while using cached tokens from previous scale steps to restore the pruned slots. This significantly reduces the number of forwarded tokens and improves the efficiency at larger resolutions. Experiments show the proposed FastVAR can further speedup FlashAttention-accelerated VAR by 2.7$\times$ with negligible performance drop of <1%. We further extend FastVAR to zero-shot generation of higher resolution images. In particular, FastVAR can generate one 2K image with 15GB memory footprints in 1.5s on a single NVIDIA 3090 GPU. Code is available at https://github.com/csguoh/FastVAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。