通过挖掘图像生成冗余性,实现更高效的推测解码。
CASCADE: Context-Aware Relaxation for Speculative Image Decoding

- 利用目标模型隐状态的语义可互换性和收敛性,放宽采样接受条件。
- 在多个文本到图像模型上实现最高3.6倍加速,保持图像质量。
- 无需额外训练,适合需要高效图像生成的研究与应用。
自回归生成是高保真图像合成的强大方法,但即使在最先进的加速器上仍计算密集且缓慢。尽管推测解码被探索用于缓解此瓶颈,现有方法在图像生成中未能达到文本生成中的效率提升。关键限制在于图像生成过程中目标模型的高不确定性,导致草稿令牌拒绝率高。本文识别出树状推测解码中自然出现的目标模型行为模式。我们形式化了两个特性:语义可互换性和收敛性,源于目标模型隐藏状态表示中的冗余。通过捕捉预测令牌树深度和广度上的这些冗余,我们的方法在无需额外训练的情况下,识别出合理的接受放松机会。此外,通过将目标模型的冗余信号注入草稿模型训练,以最小修改提升独立草稿模型性能。我们在多个文本到图像模型和草稿架构上评估该方法,结果表明,CASCADE 在基于草稿的推测解码中实现了最先进加速,最高达3.6倍,同时保持图像质量和文本提示保真度。
原文摘要 · Abstract (English)
Autoregressive generation is a powerful approach for high-fidelity image synthesis, but it remains computationally demanding and slow even on the most advanced accelerators. While speculative decoding has been explored to mitigate this bottleneck, existing approaches fail to achieve efficiency gains comparable to those observed in text generation. A key limitation is the target model's high uncertainty during image generation, which leads to high draft token rejection rates. In this work, we identify previously overlooked patterns in the target model's behavior that emerge naturally in tree-based speculative decoding. Specifically, we formalize two properties, semantic interchangeability and convergence, arising from the redundancies in the target model's hidden state representations. By capturing these redundancies across the depth and breadth of the predicted token tree, our method identifies principled opportunities for acceptance relaxation without requiring additional training. Additionally, we enhance standalone drafter performance by injecting the redundancy signals from the target model into drafter training with minimal modification. We evaluate our approach across multiple text-to-image models and drafter architectures. Results show that CASCADE achieves state-of-the-art speedups for drafter-based speculative decoding, with up to 3.6x acceleration, while maintaining image quality and text-prompt fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。