提出无需训练的并行解码方法,让文本生成图像更快更多样。
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
- 用概率收敛准则替代确定性判断,实现采样解码下的并行生成。
- 在不损失图像质量前提下,推理步数减少30%以上,加速比达2.5倍。
- 适合追求高效生成且需多样性的文生图应用,如设计、创作工具。
当前大规模自回归模型虽能生成高质量高分辨率图像,但推理需数百甚至上千步逐词预测,耗时严重。现有雅可比解码虽可无训练加速,却依赖确定性收敛判据,仅适用于贪婪解码,无法兼容采样解码——而后者对图像质量和多样性至关重要。本文提出无需训练的概率化并行解码算法:推测雅可比解码(SJD),通过引入概率收敛准则,在保持采样随机性的同时实现多标记并行预测,显著减少生成步数。SJD在每步预测多个标记,并依据概率标准接受结果,支持模型以更少步骤生成多样化图像。我们还研究了利用视觉数据空间局部性的标记初始化策略,进一步提升特定场景下的加速比。在多个自回归文生图模型上验证,SJD有效加速推理且不牺牲视觉质量。代码已开源:https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/
原文摘要 · Abstract (English)
The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next-token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on a deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-based decoding which is crucial for visual quality and diversity in the current auto-regressive text-to-image generation. In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in sampling-based token decoding and allowing the model to generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token-prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to-image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality. The code of our work is available here: https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。