提出解耦式随机重掩码策略,提升扩散语言模型生成流畅性与多样性。
NAVIRA: Decoupled Stochastic Remasking for Masked Diffusion Language Models
- 分两步推理:先评分再重掩码,避免错误传播
- 随机重掩码降低重复修正,提升文本熵值
- 适合追求高质量生成的文本生成研究者
掩码扩散语言模型通过并行迭代解掩码生成文本,但存在校正难题:同一轮生成的词基于边缘分布预测,早期局部错误会污染上下文。PRISM通过学习词级质量分数并重掩不可靠词来缓解,但其推理规则耦合:同一前向过程既检测低质词又计算替换词的logits,导致错误仍影响再生。我们提出NAVIRA,一种解耦的推理时解码策略,将检测与再生分离,并随机采样重掩码位置。第一轮前向计算得分,选定词被重掩;第二轮从前向清理后的上下文再生。温度控制的随机重掩码减少对同一位置的重复修正,平衡流畅性与多样性。在170M参数的掩码扩散语言模型上,解耦提升流畅性,调度式随机重掩码在更大前向次数预算下维持熵值,获得更强的LLM-judge评分。结果表明,重掩码策略本身比学习到的质量信号更关键。
原文摘要 · Abstract (English)
Masked diffusion language models generate text by iteratively unmasking many tokens in parallel, but this speed comes with a correction problem: tokens generated in the same step are predicted from marginal distributions, and early local dependency errors can later contaminate the context. PRISM addresses this by learning token-level quality scores and remasking unreliable tokens, but its inference rule is coupled: the same forward pass both detects low-quality tokens and computes logits for their replacements, so the erroneous tokens still condition regeneration. We propose NAVIRA, an inference-time decoding policy that separates these two operations and samples remasking positions stochastically. A first forward pass scores tokens; selected tokens are masked; a second forward pass regenerates from the cleaned context. Temperature-controlled remasking reduces repeated correction of the same positions and balances fluency against diversity. In controlled experiments with a 170M masked diffusion language model, decoupling improves fluency, while scheduled stochastic remasking preserves entropy and achieves stronger LLM-judge scores under larger forward-pass budgets. These results show that remasking policy, not only the learned quality signal, is central to reliable masked-diffusion text generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。