arXiv:2510.21961cs.LGcs.CL2025-10被引 11

提出一种无需训练的并行采样方法,让文本生成更快更准。

Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing

  • 通过条件独立性测试识别依赖关系,动态筛选可并行更新的词元。
  • 在IFEval上比基线方法最高提升16%准确率,且对超参数不敏感。
  • 能自发形成先定结构后细化的层级生成策略,适合长文本生成。

掩码扩散模型(MDMs)为离散文本生成提供了替代自回归模型(ARMs)的方案,支持并行采样而非逐个生成,显著加速推理。然而高效并行采样面临两个矛盾需求:(i) 同时更新的词元需条件独立,(ii) 更新应优先高置信度预测。这两者冲突,因高置信度预测常聚集且相互依赖,限制并行机会。本文提出PUNT,一种模型无关的采样器,通过检测词元依赖关系,将低置信度词元从冲突组中剔除,生成满足独立性与置信度双重标准的解封索引集。该方法通过近似条件独立性测试实现更优的并行解封。实验表明,相比其他强训练无关基线,PUNT在准确率与计算开销间取得更优平衡,尤其在长序列生成中表现突出。在IFEval基准上,准确率最高提升16%,涵盖多种超参数设置,减少对敏感调参的依赖。此外,PUNT表现出涌现的分层生成模式,模型先构建高层段落结构,再进行局部精细化,暗示类似规划的生成过程,有助于强对齐性能。

原文摘要 · Abstract (English)

Masked diffusion models (MDMs) offer a compelling alternative to autoregressive models (ARMs) for discrete text generation because they enable parallel token sampling, rather than sequential, left-to-right generation. This means potentially much faster inference. However, effective parallel sampling faces two competing requirements: (i) simultaneously updated tokens must be conditionally independent, and (ii) updates should prioritise high-confidence predictions. These goals conflict because high-confidence predictions often cluster and depend on each other, opportunities for parallel updates. We present PUNT, a model-agnostic sampler that reconciles this trade-off. Our method identifies token dependencies and removes lower-confidence tokens from conflicting groups. This produces sets of indices for unmasking that satisfy both independence and confidence criteria. Our approach ensures improved parallel unmasking through approximate conditional independence testing. Our experiments show that PUNT delivers a superior trade-off between accuracy and compute when compared to other strong training-free baselines, especially for generation of longer sequences. On the IFEval benchmark, it achieves up to 16\% higher accuracy over baseline methods, including sequential generation (one-by-one). These gains hold across different values of hyperparameters, mitigating the need for brittle hyperparameter tuning. Moreover, we observe that PUNT induces an emergent hierarchical generation strategy, where the model first establishes high-level paragraph structure before local refinement, suggesting a planning-like generation process that contributes to strong alignment performance.

文本生成并行采样扩散模型无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。