通过动态剔除低频冗余像素,实现高分辨率图像生成的高效加速。
Frequency-Aware Autoregressive Modeling for Efficient High-Resolution Image Synthesis
- 基于低频区域相似性与影响小的特性,动态过滤冗余像素。
- 在Infinity-2B模型上实现最高2倍提速,质量损失极小。
- 无需训练,可直接接入现有生成框架,适合实时高分辨率应用。
基于多尺度预测范式的视觉自回归建模在图像质量和模型可扩展性上优于传统自回归与扩散模型,通过逐级提升分辨率生成图像。然而,高分辨率阶段因大量像素令牌带来的计算开销仍是关键挑战。本文提出SparseVAR,一种无需额外训练的即插即用加速框架,在推理时动态排除低频令牌。该方法基于观察:高分辨率阶段中低频区域的令牌对图像质量影响微乎其微,且与邻近令牌高度相似。同时发现模型不同模块关注不同区域,部分集中于高频细节。SparseVAR利用轻量级MSE度量识别低频令牌,并通过均匀采样的少量锚点令牌保持被排除区域的保真度。在HART和Infinity模型上均实现显著加速,尤其在Infinity-2B中达2倍速度提升,质量损失极小。
原文摘要 · Abstract (English)
Visual autoregressive modeling, based on the next-scale prediction paradigm, exhibits notable advantages in image quality and model scalability over traditional autoregressive and diffusion models. It generates images by progressively refining resolution across multiple stages. However, the computational overhead in high-resolution stages remains a critical challenge due to the substantial number of tokens involved. In this paper, we introduce SparseVAR, a plug-and-play acceleration framework for next-scale prediction that dynamically excludes low-frequency tokens during inference without requiring additional training. Our approach is motivated by the observation that tokens in low-frequency regions have a negligible impact on image quality in high-resolution stages and exhibit strong similarity with neighboring tokens. Additionally, we observe that different blocks in the next-scale prediction model focus on distinct regions, with some concentrating on high-frequency areas. SparseVAR leverages these insights by employing lightweight MSE-based metrics to identify low-frequency tokens while preserving the fidelity of excluded regions through a small set of uniformly sampled anchor tokens. By significantly reducing the computational cost while maintaining high image generation quality, SparseVAR achieves notable acceleration in both HART and Infinity. Specifically, SparseVAR achieves up to a 2 times speedup with minimal quality degradation in Infinity-2B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。