提出自适应采样方法,让扩散语言模型更省计算、更快生成。
LESS Is More: Mutual-Stability Sampling for Diffusion Language Models

- 根据令牌稳定度动态决定何时解掩码,避免无效计算。
- 在多个模型和任务上,准确率优于现有方法,步数减少72.1%。
- 无需训练,适配多种扩散语言模型,适合追求高效推理的开发者。
扩散大语言模型(dLLMs)通过迭代修正掩码序列提供并行生成能力,但其效率受限于固定步数的采样策略,导致对已稳定位置重复计算,或过早提交不稳定预测。本文提出 extsc{LESS},一种无需训练、通用的自适应采样器,将令牌提交视为在线停止问题。它通过联合稳定性规则实现:仅当某位置的最高置信度预测持续多步且分布稳定(基于前K个预测的跨步JS散度)时,才允许解掩码。我们在Dream-7B、LLaDA-8B和LLaDA-1.5-8B上评估,覆盖全序列扩散与分块半自回归采样,涵盖七项基准任务(通用知识、数学、代码)。 extsc{LESS}在保持高准确率的同时,相较固定预算采样减少72.1%反向去噪步数。每步需一次Transformer前向传播,因此显著降低前向次数、实际延迟与推理算力消耗。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) offer a promising alternative to autoregressive decoding by iteratively refining masked sequences, enabling parallel token updates and bidirectional conditioning. Their practical efficiency, however, is limited by sampling procedures that execute a fixed number of reverse denoising steps selected before decoding, spending computation on already-stable positions and sometimes committing unstable ones too early. We present \textsc{LESS}, a training-free, model-agnostic adaptive sampler that treats token commitment as an online stopping problem. \textsc{LESS} implements mutual-stability sampling through a joint stability rule that makes a masked position eligible for unmasking only when its top-1 prediction has high confidence, its top-1 token persists across recent reverse steps, and its predictive distribution is stable under top-$K$ inter-step Jensen--Shannon divergence. We evaluate \textsc{LESS} on Dream-7B, LLaDA-8B, and LLaDA-1.5-8B, covering full-sequence diffusion and semi-autoregressive blockwise sampling regimes, across seven benchmarks spanning general knowledge, math, and code. \textsc{LESS} improves average accuracy over strong training-free adaptive samplers while using $72.1\%$ fewer reverse steps than fixed-budget decoding. Since each reverse step requires a Transformer forward pass, these step-count reductions translate into fewer forward evaluations, lower measured wall-clock latency, and lower estimated inference compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。