arXiv:2606.29066cs.CL2026-06

提出连续解码框架,让语言模型更高效生成文本。

$x$-Prediction Flow: Efficient Continuous Decoding for Masked Diffusion Language Models

论文配图:$x$-Prediction Flow: Efficient Continuous Decoding for Masked Diffusion Language Models
图 1 · 摘自论文原文
  • 将掩码预测重释为连续状态预测,支持逐步积累进度
  • 仅用15%步数即达97%全预算准确率,显著提升效率
  • 适合资源受限场景,尤其对长文本生成有优势

掩码扩散语言模型通过迭代解掩码生成文本,但传统解码器将每一步简化为二元决策:某位置要么确定一个词,要么保持完全掩码,忽略丰富的预测信息,强制过早且不可逆的决定,导致在有限解码预算下表现不佳。本文重新将掩码预测视为干净状态预测(x- prediction),并证明其可诱导输入嵌入空间中的连续流动。基于此,我们提出一种连续解码框架,使词元可在每一步积累部分进展并保持可修改性。为应对语言中各位置上下文约束的不均衡性,我们用基于置信度的异步更新替代图像扩散中的全局同步调度,实现逐词推进。此外,引入轻量级策略网络,并将其训练建模为强化学习问题。应用于预训练的LLaDA模型时,该解码器在使用少于15%的扩散步数情况下,仍能保持83–97%的全预算准确率,显著优于同预算下的离散掩码预测解码。

原文摘要 · Abstract (English)

Masked diffusion language models (MDLMs) generate text by iteratively unmasking tokens, but their standard decoder reduces each step to a binary action: a position is either committed to a single token or left fully masked, discarding rich predictive information rather than carrying it forward, and forcing premature, irrevocable commitments that lead to poor performance under a limited decoding budget. In this paper, we reinterpret mask prediction as a clean-state prediction ($x$-prediction) and show that it can be used to induce a continuous flow in the input embedding space. Building on this view, we propose a continuous decoding framework for MDLMs where tokens can accumulate partial progress at each diffusion step and remain revisable. To match the uneven contextual constraints across positions in language, we replace the globally synchronous schedule in image diffusion with a confidence-based asynchronous update in which the diffusion progress is token-wise accumulated. Additionally, we introduce a lightweight policy network and formulate its training as a reinforcement learning problem. Applied to pretrained LLaDA, our decoder retains 83--97% of full-budget accuracy using under 15% of the diffusion steps, largely outperforming discrete mask-prediction decoding at matched budgets.

扩散模型连续解码语言生成高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。