发现扩散模型生成文本时先还原实体，再处理关系和结构。
What Gets Unmasked First? Trajectory Analysis of Diffusion Models for Graph-to-Text Generation
- 分析扩散模型解码轨迹，发现先恢复实体再处理结构
- 微调破坏自然生成顺序，导致信息遗漏或幻觉
- 提出无需训练的修正方法，提升9.4点BLEU-4
我们首次系统研究了用于图到文本生成的掩码扩散语言模型（MDLM）。通过分析生成轨迹——即迭代解码中词元被解掩码的顺序——发现与自回归大模型线性生成不同，MDLM会优先恢复实体，随后是关系词和功能词，最后处理结构化词元。我们进一步发现一种此前未记录的监督微调失败模式：微调会过早锚定句尾结构词元，固定输出长度，导致信息丢失或幻觉。为此，我们提出lambda缩放结构解码，在不改变训练的前提下降低结构词元置信度，使生成性能提升+9.4 BLEU-4。此外，我们引入Graph-LLaDA，将图转换器编码器整合至LLaDA解码流程中，显式建模图结构关系。在LAGRANGE上的跨数据集评估表明，先前基线模型过拟合于特定数据集模式，而基于大模型与MDLM的方法具有显著更优泛化能力。
原文摘要 · Abstract (English)
We present the first systematic study of masked diffusion language models (MDLMs) for graph-to-text generation. We analyze MDLM generation trajectories -- the order in which tokens are unmasked during iterative decoding -- and find that, unlike autoregressive LLMs which generate text linearly, MDLMs naturally prioritize entities first, followed by relational and function words, with structural tokens resolved last. We further identify a previously undocumented failure mode of supervised fine-tuning: SFT disrupts this strategy by prematurely anchoring structural sentence-ending tokens early in the decoding trajectory, effectively fixing the output length which can lead to omitted or hallucinated information. To address this, we propose lambda-scaled structural decoding, a training-free inference-time modification that downweights structural token confidence and recovers +9.4 BLEU-4. Finally, we introduce Graph-LLaDA, which integrates a Graph Transformer encoder into LLaDA's decoding process to explicitly incorporate relational graph structure. Cross-dataset evaluation on LAGRANGE reveals that previous baselines overfit to dataset-specific patterns, while LLM- and MDLM-based approaches generalize significantly better.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。