用可演化软令牌替代硬掩码，实现语言模型的可回溯生成。
Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models
- 用软令牌分布替代硬二值掩码，支持逐步优化生成过程。
- 在多个基准上超越现有扩散语言模型，提升显著且稳定。
- 适合需要精细编辑和迭代生成的场景，如文本重写与内容创作。
扩散语言模型（DLMs）通过迭代精炼实现并行解码，是一种有前景的语言建模方法。然而，多数DLM依赖硬二值掩码和离散令牌分配，限制了早期决策的修正，并未充分利用中间概率表示。本文提出EvoToken-DLM，一种基于扩散的语言建模新方法，以可演化的软令牌分布取代硬掩码。该方法实现从掩码状态到离散输出的渐进式过渡，支持可回溯解码。为有效支持这一演化过程，引入连续轨迹监督，使训练目标与迭代概率更新对齐。在多个基准上的广泛实验表明，EvoToken-DLM始终表现优异，优于强基线的扩散语言模型和掩码型DLM。项目主页：https://aim-uofa.github.io/EvoTokenDLM。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) offer a promising alternative for language modeling by enabling parallel decoding through iterative refinement. However, most DLMs rely on hard binary masking and discrete token assignments, which hinder the revision of early decisions and underutilize intermediate probabilistic representations. In this paper, we propose EvoToken-DLM, a novel diffusion-based language modeling approach that replaces hard binary masks with evolving soft token distributions. EvoToken-DLM enables a progressive transition from masked states to discrete outputs, supporting revisable decoding. To effectively support this evolution, we introduce continuous trajectory supervision, which aligns training objectives with iterative probabilistic updates. Extensive experiments across multiple benchmarks show that EvoToken-DLM consistently achieves superior performance, outperforming strong diffusion-based and masked DLM baselines. Project webpage: https://aim-uofa.github.io/EvoTokenDLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。