通过多尺度局部推测生成,让图像自回归模型提速5倍且保持质量。
Multi-Scale Local Speculative Decoding for Image Generation
- 用低分辨率草稿+上采样生成候选图像,高分辨率模型并行验证
- 局部拒绝与重采样机制使错误修正效率提升,最高加速5倍
- 适合追求生成速度又不牺牲画质的图像生成研究者
自回归(AR)模型在图像生成中表现优异,但其串行特性带来显著延迟。推测解码为加速提供了可能,但现有方法受限于词元级模糊和缺乏空间感知。本文提出多尺度局部推测解码(MuLo-SD),结合多分辨率草稿与空间感知验证,利用低分辨率草稿配合上采样生成候选图像,并由高分辨率目标模型并行验证。关键创新在于引入局部拒绝与重采样机制,仅在空间邻域内修正错误,避免全图重采样。结合并行解码重采样,MuLo-SD实现最高5倍加速,在MS-COCO 5k验证集上优于基线方法,同时保持相近的语义一致性和感知质量。实验基于GenEval、DPG-Bench、FID/HPSv2评估,消融分析揭示了上采样设计、概率池化及邻域扩展对性能的影响。该方法在图像生成推测解码中达到新基准。
原文摘要 · Abstract (English)
Autoregressive (AR) models have achieved remarkable success in image synthesis, yet their sequential nature imposes significant latency constraints. Speculative Decoding offers a promising avenue for acceleration, but existing approaches are limited by token-level ambiguity and lack of spatial awareness. In this work, we introduce Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that combines multi-resolution drafting with spatially informed verification to accelerate AR image generation. Our method leverages a low-resolution drafter paired with an up-sampling step to propose candidate image tokens, which are then verified in parallel by a high-resolution target model. Crucially, we incorporate a local rejection and resampling mechanism, enabling efficient correction of draft errors by focusing on spatial neighborhoods rather than raster-scan resampling after the first rejection. When integrated with parallel decoding resampling, MuLo-SD achieves substantial speedups -- up to $\mathbf{5\times}$ -- outperforming both speculative decoding and parallel decoding baselines in terms of acceleration, while maintaining comparable semantic alignment and perceptual quality. These results are validated using GenEval, DPG-Bench, and FID/HPSv2 on the MS-COCO 5k validation split. Extensive ablations highlight the impact of up-sampling design, probability pooling, and local rejection and resampling with neighborhood expansion. Our approach sets a new state-of-the-art in speculative decoding for image synthesis, bridging the gap between efficiency and fidelity. Project page is available at https://qualcomm-ai-research.github.io/mulo-sd-webpage/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。