arXiv:2511.20390cs.CV2025-11被引 1

提出FREE框架,让扩散Transformer并行生成提速2.25倍且不损失质量。

FREE: Uncertainty-Aware Autoregression for Parallel Diffusion Transformers

  • 用轻量级绘图器做特征级自回归,实现并行验证加速。
  • 在ImageNet-512²上达到2.25倍加速,生成质量保持高保真。
  • 引入不确定性引导的动态接受策略,适合对速度敏感的生成场景。

扩散Transformer(DiTs)虽生成质量顶尖,但需长序列去噪,推理延迟高。现有推测推理方法在基于U-Net的模型中可实现无损并行采样,但在DiTs上因绘图准确率不足而加速有限。本文分析发现,DiTs最终变换层(top-block)特征具有强时间一致性与丰富语义抽象性。据此提出FREE框架:利用轻量绘图器进行特征级自回归,结合并行验证,理论与实证均保证无损加速。同时,由于DiTs在后期去噪步骤预测方差(不确定性)自然增大,导致推测采样接受率下降。为此引入不确定性引导的松弛策略,形成FREE(relax),动态调整接受概率以应对不确定性。ImageNet-512²实验表明,FREE实现最高1.86倍加速,FREE(relax)进一步达2.25倍速度提升,生成质量在感知与定量指标上均保持优异。

原文摘要 · Abstract (English)

Diffusion Transformers (DiTs) achieve state-of-the-art generation quality but require long sequential denoising trajectories, leading to high inference latency. Recent speculative inference methods enable lossless parallel sampling in U-Net-based diffusion models via a drafter-verifier scheme, but their acceleration is limited on DiTs due to insufficient draft accuracy during verification. To address this limitation, we analyze the DiTs' feature dynamics and find the features of the final transformer layer (top-block) exhibit strong temporal consistency and rich semantic abstraction. Based on this insight, we propose FREE, a novel framework that employs a lightweight drafter to perform feature-level autoregression with parallel verification, guaranteeing lossless acceleration with theoretical and empirical support. Meanwhile, prediction variance (uncertainty) of DiTs naturally increases in later denoising steps, reducing acceptance rates under speculative sampling. To mitigate this effect, we further introduce an uncertainty-guided relaxation strategy, forming FREE (relax), which dynamically adjusts the acceptance probability in response to uncertainty levels. Experiments on ImageNet-$512^2$ show that FREE achieves up to $1.86 \times$ acceleration, and FREE (relax) further reaches $2.25 \times$ speedup while maintaining high perceptual and quantitative fidelity in generation quality.

扩散模型并行生成不确定性建模Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。