通过跳过部分验证步骤,让视觉自回归生成更快
VVS: Accelerating Speculative Decoding for Visual Autoregressive Generation via Partial Verification Skipping
- 提出部分验证跳过机制,减少目标模型前向传播次数
- 相比原始方法减少2.8倍前向计算,保持生成质量
- 适合需要快速生成图像的实时应用开发者
视觉自回归(AR)生成模型在图像生成方面展现出强大潜力,但其逐标记预测范式带来显著推理延迟。尽管推测解码(SD)已被证明能加速视觉AR模型,但其‘草稿一步,验证一步’的模式无法直接减少前向传播次数,限制了加速效果。受视觉标记可互换性的启发,我们首次探索在SD过程中跳过部分验证,以明确减少目标模型的前向传播次数,从而降低推理延迟。通过分析草稿阶段特性,发现验证冗余和陈旧特征复用是维持生成质量的同时提升无验证步骤速度的关键因素。受此启发,我们提出新型SD框架VVS,通过三个互补模块实现加速:(1) 带动态截断的无验证标记选择器,(2) 标记级特征缓存与复用,(3) 细粒度跳过步调度。结果表明,VVS相比原始AR解码减少了2.8倍的目标模型前向传播次数,同时保持优异的生成质量,在速度-质量权衡上优于传统SD框架,揭示了重塑推测解码范式的巨大潜力。代码已开源:https://github.com/HyattDD/VVS。
原文摘要 · Abstract (English)
Visual autoregressive (AR) generation models have demonstrated strong potential for image generation, yet their next-token-prediction paradigm introduces considerable inference latency. Although speculative decoding (SD) has been proven effective for accelerating visual AR models, its "draft one step, then verify one step" paradigm prevents a direct reduction in the number of forward passes, limiting its acceleration potential. Motivated by the interchangeability of visual tokens, we explore verification skipping in the SD process for the first time to explicitly cut the number of target model forward passes, thereby reducing inference latency. By analyzing the characteristics of the drafting stage, we observe that verification redundancy and stale feature reusability are key factors to maintain generation quality while improving speed for verification-free steps. Inspired by these two observations, we propose a novel SD framework VVS to accelerate visual AR model via partial verification skipping, which integrates three complementary modules: (1) a verification-free token selector with dynamic truncation, (2) token-level feature caching and reuse, and (3) fine-grained skipped step scheduling. Consequently, VVS reduces the number of target model forward passes by $2.8\times$ relative to vanilla AR decoding while maintaining competitive generation quality, offering a superior speed-quality trade-off over conventional SD frameworks and revealing strong potential to reshape the SD paradigm. Our code is available at https://github.com/HyattDD/VVS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。