通过上下文感知初始化缩短扩散语言模型生成路径,提升推理效率。
Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
- 用轻量辅助模型注入提示相关先验,改进初始状态
- 实验显示减少约35%的去噪迭代次数,降低计算开销
- 适合关注扩散模型加速与高效推理的研究者
扩散大语言模型(DLLMs)支持完全并行的词元解码,但因需大量去噪迭代将信息空白的全掩码初始化逐步修正为连贯文本,实际推理仍不实用。现有加速方法多聚焦于优化生成轨迹的遍历效率,如改进求解器或采样策略。本文提出互补思路:通过上下文感知初始化,使起始点更接近目标分布,从而缩短生成路径。我们设计了一种无需训练的接口,将轻量级辅助模型提供的提示相关先验注入扩散初始化过程,并实现两种机制:离散词元注入与表征级嵌入插值。由于注入先验可能不准确,且仅解掩码可能过早锁定,我们引入基于置信度的重掩码机制作为先验质疑手段。在GSM8K上的初步结果表明,上下文感知初始化可显著减少去噪迭代次数(本设置下约35%的函数评估减少),同时揭示关键挑战:朴素的热启动反而会降低最终精度,相较强基线模型。这些发现推动了一个关于校准、修正机制与表征对齐的可靠热启动扩散解码研究议程。
原文摘要 · Abstract (English)
Diffusion Large Language Models (DLLMs) enable fully parallel token decoding but often remain impractical at inference time due to the many denoising iterations required to refine an information-free, fully masked initialization into coherent text. Most existing acceleration methods focus on traversing this generative trajectory more efficiently via improved solvers or sampling strategies. We advance a complementary perspective: shorten the trajectory itself by starting closer to the target distribution through context-aware initialization. We propose a training-free interface that injects prompt-conditioned priors from a lightweight auxiliary model into the diffusion initialization, and instantiate it with two mechanisms: discrete token injection and representation-level embedding interpolation. Because injected priors can be imperfect and unmask-only decoding can over-commit early, we also introduce a simple confidence-based remasking mechanism as a form of prior skepticism. Preliminary evidence on GSM8K suggests that context-aware initialization can substantially reduce denoising iterations (about 35\% fewer function evaluations in our setting), while also exposing a key open challenge: naive warm-starting can degrade final accuracy relative to strong diffusion baselines. We use these findings to motivate a research agenda around calibration, revision mechanisms, and representation alignment for reliable warm-started diffusion decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。