arXiv:2605.09820cs.LG2026-05

提出无需训练的贝叶斯结构解码,让扩散语言模型自由控制生成长度和结构。

Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference

论文配图:Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference
图 1 · 摘自论文原文
  • 用贝叶斯推理动态推断生成长度、分块边界和解码调度
  • 在多个基准上显著提升生成质量与灵活性,优于固定/可变长度基线
  • 适合需要灵活可控文本生成的应用场景,如对话、摘要

扩散语言模型(DLMs)因其支持并行解码而成为自回归模型的有前途替代方案。然而,现有大多数DLM依赖预设的固定生成长度,限制了实际应用中的灵活性。尽管少数工作尝试支持可变长度生成,但普遍存在缺陷:部分需昂贵重训练,另一些仅依赖局部置信度信号,无法捕捉序列演化结构,导致生成质量不佳。本文提出一种无需训练的贝叶斯结构解码框架,将可变长度生成建模为动态结构推理问题,联合计算扩展长度、块边界和解码调度。每步窗口扩展中,通过统一机制融合局部不确定性与结构信号,支持动态分块扩展与组织,同时保持连贯性。多基准实验表明,该方法在生成质量和灵活性上均显著优于现有固定与可变长度基线,验证了贝叶斯结构解码对扩散语言模型的优势,提供了一种原则性强且高效的结构化文本生成解决方案。

原文摘要 · Abstract (English)

Diffusion language models (DLMs) have recently emerged as a promising alternative to autoregressive models, primarily due to their ability to enable parallel decoding. Despite this advantage, most existing DLMs rely on a fixed generation length specified prior to decoding, which restricts their flexibility in real-world applications. While a few recent works attempt to support flexible-length generation, they typically suffer from notable limitations: some require costly retraining to accommodate variable-length outputs, while others depend solely on local confidence signals during decoding. Such local criteria fail to capture the evolving structure of the sequence, often resulting in suboptimal generation quality. In this paper, we propose a training-free, Bayesian structured decoding framework that formulates flexible-length generation as a dynamic structural inference problem. Our approach formulates flexible-length generation as a dynamic structural inference problem, jointly computing the expansion length, the block boundaries, and the decoding schedule. At each window expansion step, the method integrates local uncertainty with structural signals via a unified mechanism that supports dynamic structured generation, including both flexible block expansion and block organization, while maintaining coherence. Extensive experiments across multiple benchmarks demonstrate that our approach significantly improves generation quality and flexibility over existing fixed-length and flexible-length baselines. These results highlight the advantage of Bayesian structured decoding for diffusion language model, providing a principled and efficient solution for structured text generation.

扩散模型文本生成贝叶斯推理动态结构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。