用熵信号统一加速视觉生成的自回归与扩散阶段,实现单步出图。
Drift-AR: Single-Step Visual Autoregressive Generation via Anti-Symmetric Drifting
- 利用预测熵指导草稿生成,减少无效拒绝,提升自回归速度。
- 将熵视为初始状态的物理方差,实现无需迭代的单步解码。
- 全程共享熵信号,无额外计算开销,适合追求高效生成的场景。
自回归与扩散模型融合框架结合了自回归的语义建模能力与扩散模型的高保真合成能力,但存在双重速度瓶颈:自回归阶段的序列化生成和扩散视觉解码阶段的多步去噪。现有方法分别处理各环节,缺乏统一设计原则。本文发现连续空间自回归模型中每个位置的预测熵自然编码了空间变化的生成不确定性,同时影响自回归草稿质量与视觉解码所需的修正强度,这一特性此前未被充分利用。由于熵同时关联两个瓶颈,它成为联合加速的天然信号。为此提出Drift-AR:1)自回归加速方面,引入熵感知的推测解码,通过因果归一化熵损失对齐草稿与目标熵分布,解决熵不匹配导致的过度拒绝问题;2)视觉解码加速方面,将熵重新解释为反对称漂移场的初始物理方差——高熵位置触发强漂移向数据流形,低熵位置漂移趋近于零,从而实现无需迭代去噪或蒸馏的单步(1-NFE)解码。两个阶段共享同一熵信号,仅需一次计算,无额外开销。在MAR、TransDiff和NextStep-1上的实验表明,可实现3.8–5.5倍加速,且保持原质量水平或超越。代码将在https://github.com/aSleepyTree/Drift-AR发布。
原文摘要 · Abstract (English)
Autoregressive (AR)-Diffusion hybrid paradigms combine AR's structured semantic modeling with diffusion's high-fidelity synthesis, yet suffer from a dual speed bottleneck: the sequential AR stage and the iterative multi-step denoising of the diffusion vision decode stage. Existing methods address each in isolation without a unified principle design. We observe that the per-position \emph{prediction entropy} of continuous-space AR models naturally encodes spatially varying generation uncertainty, which simultaneously governing draft prediction quality in the AR stage and reflecting the corrective effort required by vision decoding stage, which is not fully explored before. Since entropy is inherently tied to both bottlenecks, it serves as a natural unifying signal for joint acceleration. In this work, we propose \textbf{Drift-AR}, which leverages entropy signal to accelerate both stages: 1) for AR acceleration, we introduce Entropy-Informed Speculative Decoding that align draft-target entropy distributions via a causal-normalized entropy loss, resolving the entropy mismatch that causes excessive draft rejection; 2) for visual decoder acceleration, we reinterpret entropy as the \emph{physical variance} of the initial state for an anti-symmetric drifting field -- high-entropy positions activate stronger drift toward the data manifold while low-entropy positions yield vanishing drift -- enabling single-step (1-NFE) decoding without iterative denoising or distillation. Moreover, both stages share the same entropy signal, which is computed once with no extra cost. Experiments on MAR, TransDiff, and NextStep-1 demonstrate 3.8-5.5$\times$ speedup with genuine 1-NFE decoding, matching or surpassing original quality. Code will be available at https://github.com/aSleepyTree/Drift-AR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。