通过自奖励粒子追踪提升掩码扩散语言模型采样质量
Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models
- 并行运行多个扩散过程，用轨迹置信度动态分配权重
- 无需额外训练即可显著提升生成样本质量与多样性
- 适合需要高质量文本生成的推理场景
本文提出自奖励序贯蒙特卡洛（SMC）算法，用于有效采样掩码扩散语言模型（MDLMs）。现有方法多依赖置信度采样策略，仅保留每步最高置信度的词元，导致生成路径单一、对噪声敏感。为此，我们并行启动多个扩散过程（称作粒子），进行轨迹探索，并引入轨迹级置信度作为自奖励信号，用于分配粒子重要性权重。采样过程中，粒子被迭代加权与重采样，系统性引导生成向全局高置信度、高质量样本收敛。该方法在多个掩码扩散语言模型和基准测试上验证，无需额外训练或奖励引导，即可显著提升采样质量，有效将并行推理能力转化为生成性能。代码已开源：https://github.com/Algolzw/self-rewarding-smc。
原文摘要 · Abstract (English)
This work presents self-rewarding sequential Monte Carlo (SMC), an inference-time scaling algorithm enabling effective sampling of masked diffusion language models (MDLMs). Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy, where only tokens with the highest prediction confidence are preserved at each step. This restricts the generation to a noise-sensitive, greedy decoding paradigm, resulting in an inevitable collapse in the diversity of possible paths. We address this problem by launching multiple interacting diffusion processes in parallel, referred to as particles, for trajectory exploration. Importantly, we introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights. During sampling, particles are iteratively weighted and resampled to systematically steer generation towards globally confident, high-quality samples. Our self-rewarding SMC is verified on various masked diffusion language models and benchmarks, achieving significant improvement without extra training or reward guidance, while effectively converting parallel inference capacity into improved sampling quality. Our code is available at https://github.com/Algolzw/self-rewarding-smc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。