提出无需似然的微调方法，让掩码扩散语言模型更稳定高效。
Discrete Tilt Matching
- 将微调转为局部解码后验的状态匹配，避免不可计算的似然
- 在数独和倒计时任务上显著提升性能，数学推理也保持竞争力
- 适合追求高效训练且需避免模式崩溃的研究者
掩码扩散大语言模型（dLLMs）是自回归生成的一种有前景替代方案。尽管强化学习方法已被用于dLLM微调，但其目标通常依赖于序列级边际似然，这对掩码扩散模型而言是不可计算的。为此，我们推导出离散倾斜匹配（DTM），一种无需似然的方法，将dLLM微调重构为在奖励倾斜下局部解码后验的状态匹配。DTM表现为带显式最小化器的加权交叉熵目标，并支持控制变量以提升训练稳定性。在合成迷宫规划任务中，我们分析了DTM的退火策略和控制变量对训练稳定性和防止模式崩溃的影响。在大规模场景下，使用DTM微调LLaDA-8B-Instruct，在数独（Sudoku）和倒计时（Countdown）任务上取得显著提升，同时在MATH500和GSM8K上保持竞争力。
原文摘要 · Abstract (English)
Masked diffusion large language models (dLLMs) are a promising alternative to autoregressive generation. While reinforcement learning (RL) methods have recently been adapted to dLLM fine-tuning, their objectives typically depend on sequence-level marginal likelihoods, which are intractable for masked diffusion models. To address this, we derive Discrete Tilt Matching (DTM), a likelihood-free method that recasts dLLM fine-tuning as state-level matching of local unmasking posteriors under reward tilting. DTM takes the form of a weighted cross-entropy objective with explicit minimizer, and admits control variates that improve training stability. On a synthetic maze-planning task, we analyze how DTM's annealing schedule and control variates affect training stability and prevent mode collapse. At scale, fine-tuning LLaDA-8B-Instruct with DTM yields strong gains on Sudoku and Countdown while remaining competitive on MATH500 and GSM8K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。