arXiv:2411.11925cs.CV2024-11被引 22

提出连续推测解码,让图像生成速度提升2倍以上。

Continuous Speculative Decoding for Autoregressive Image Generation

  • 用重参数化对齐策略提升采样接受率
  • 设计无需计算积分的拒绝-接受采样算法
  • 适合需要高速图像生成的研究与应用

连续视觉自回归(AR)模型在图像生成中表现优异,但其固有的序列特性导致推理速度慢。推测解码是加速大语言模型的成功技术,已用于离散视觉AR模型,但缺乏适用于连续分布的理论支持,难以应用于连续AR模型。本文提出连续推测解码,解决两大挑战:1)因目标模型与草稿模型输出分布不一致导致的低接受率;2)因复杂积分带来的分布修改且无解析表达式。针对挑战1),引入近似准则、基于重参数化邻近性的去噪轨迹对齐策略及标记预填充;针对挑战2),提出带合适上界的拒绝-接受采样算法,避免显式计算积分。此外,去噪轨迹对齐策略复用于采样过程,有效避免重复扩散模型推断。在256x256和512x512分辨率下多种模型上的实验表明,该方法实现超过2倍的墙时(wall-time)加速,同时保持图像生成质量。代码已公开于:https://github.com/MarkXCloud/CSpD。

原文摘要 · Abstract (English)

Continuous visual autoregressive (AR) models have demonstrated promising performance in image generation, but their inherently sequential nature results in slow inference speed. Speculative decoding, a successful acceleration technique for large language models (LLMs), has effectively accelerated discrete visual AR models. However, the absence of an analogous theory for continuous distributions precludes its use in accelerating continuous AR models. To fill this gap, this work presents continuous speculative decoding, and addresses challenges from: 1) low acceptance rate, caused by inconsistent output distribution modeled by target and draft models, and 2) modified distribution without analytic expression, caused by a complex integral. For challenge 1), we address low acceptance rates through an approximated criterion, a novel denoising trajectory alignment strategy based on reparameterization proximity, and token pre-filling. For challenge 2), we introduce acceptance-rejection sampling algorithm with an appropriate upper bound, thereby avoiding explicitly calculating the integral. Furthermore, our denoising trajectory alignment is also reused in acceptance-rejection sampling, effectively avoiding repetitive diffusion model inference. Extensive experiments on various models at 256x256 and 512x512 resolutions demonstrate that our approach achieves over 2x wall-time speedup while preserving the image generation quality. Codes is available at: https://github.com/MarkXCloud/CSpD

图像生成扩散模型加速推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。