扩散语言模型可自动感知输出长度，节省大量计算。
Diffusion Language Models Are Natively Length-Aware
- 利用隐变量预估生成长度，动态裁剪上下文窗口。
- 四类任务平均减少计算量，性能基本不变甚至提升。
- 适合需要高效推理的对话与推理场景。
与自回归语言模型在预测结束符后终止生成不同，扩散语言模型（DLMs）在固定最大长度上下文中执行预定次数的去噪步骤，但该过程与实际所需输出长度无关，导致多数短响应任务出现计算浪费。我们推测潜在提示表示包含足够信息以估计所需输出长度，并通过实证验证了这一现象。为此，我们提出一种零样本机制，在生成前动态裁剪上下文窗口，从而减少扩散步骤并实现显著计算节省。我们在四个具有多样任务的基准上评估：GSM8K（推理）、HumanEval（代码生成）、IfEval（指令遵循）和LongFormQA（问答），结果表明在几乎无性能损失的前提下实现巨大效率提升。所有任务均显著降低FLOPs，其中2个任务表现反而提升，且无统计显著性能下降。
原文摘要 · Abstract (English)
Unlike autoregressive language models, which terminate variable-length generation upon predicting an End-of-Sequence (EoS) token, Diffusion Language Models (DLMs) operate over a fixed maximum-length context window for a predetermined number of denoising steps. However, this process is independent of the required response length, resulting in computational waste for the majority of short responses common in reasoning and chat tasks. To address this problem, we conjecture that the latent prompt representation contains sufficient information to estimate the required output length. We provide empirical evidence for this phenomenon and propose a zero-shot mechanism to dynamically crop the context window before generation begins, leading to fewer diffusion steps and substantial computational savings. We evaluate our approach on four benchmarks with diverse tasks -- GSM8K (reasoning), HumanEval (code generation), IfEval (instruction following), and LongFormQA (question answering) -- revealing massive efficiency gains at minimal performance impact. We report significant reductions in FLOPs across all tasks, with no statistically significant performance degradation, and significant performance improvements in 2 out of 4 tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。