通过熵感知策略加速自回归-扩散模型生成,显著降低延迟。
Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
- 用熵感知方法优化自回归草稿生成,减少因信心过强导致的拒绝率。
- 在ImageNet上实现4.3倍无损加速,文本条件生成提速3倍。
- 适合追求快速高质量图像生成的研究者和开发者使用。
自回归-扩散混合框架结合了自回归模型的结构化建模与扩散模型的逼真合成能力,但受限于自回归的串行生成和迭代去噪过程,存在高延迟问题。本文提出统一的Fast-ARDiff框架,联合优化两个组件,在加速自回归推测解码的同时,也加快扩散解码速度。具体而言:(1) 采用熵感知的推测策略,使草稿模型生成更接近目标模型熵特性的高熵表示,缓解熵不匹配问题,降低因草稿过度自信导致的高拒绝率;(2) 将扩散模块融入端到端框架,使用动态调度器优先优化自回归部分,引导后续扩散步骤;通过轨迹与分布匹配的联合蒸馏框架优化扩散部分,确保训练稳定性和极低步数下的高质量合成。推理时,利用自回归模块的浅层特征熵预过滤低熵草稿,避免冗余计算,提升效率。Fast-ARDiff在多种模型上达到业界最优加速效果:在ImageNet 256×256上,TransDiff实现4.3×无损加速;NextStep-1在文本条件生成中实现3×加速。代码将发布于https://github.com/aSleepyTree/Fast-ARDiff。
原文摘要 · Abstract (English)
Autoregressive(AR)-diffusion hybrid paradigms combine AR's structured modeling with diffusion's photorealistic synthesis, yet suffer from high latency due to sequential AR generation and iterative denoising. In this work, we tackle this bottleneck and propose a unified AR-diffusion framework Fast-ARDiff that jointly optimizes both components, accelerating AR speculative decoding while simultaneously facilitating faster diffusion decoding. Specifically: (1) The entropy-informed speculative strategy encourages draft model to produce higher-entropy representations aligned with target model's entropy characteristics, mitigating entropy mismatch and high rejection rates caused by draft overconfidence. (2) For diffusion decoding, rather than treating it as an independent module, we integrate it into the same end-to-end framework using a dynamic scheduler that prioritizes AR optimization to guide the diffusion part in further steps. The diffusion part is optimized through a joint distillation framework combining trajectory and distribution matching, ensuring stable training and high-quality synthesis with extremely few steps. During inference, shallow feature entropy from AR module is used to pre-filter low-entropy drafts, avoiding redundant computation and improving latency. Fast-ARDiff achieves state-of-the-art acceleration across diverse models: on ImageNet 256$\times$256, TransDiff attains 4.3$\times$ lossless speedup, and NextStep-1 achieves 3$\times$ acceleration on text-conditioned generation. Code will be available at https://github.com/aSleepyTree/Fast-ARDiff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。