arXiv:2605.16378cs.LGcs.AI2026-05被引 1

研究掩码语言模型生成时的全局行为,揭示其混合时间特性与温度依赖性。

Mixing Times of Glauber Dynamics on Masked Language Models

论文配图:Mixing Times of Glauber Dynamics on Masked Language Models
图 1 · 摘自论文原文
  • 将掩码生成建模为格劳伯动力学马尔可夫链,分析序列空间演化。
  • 在高温度下混合时间为O(n log n),低温度下存在指数级慢逃逸的稳定态。
  • 发现语义陷阱和循环基域,政治内容是典型实证案例。

掩码语言模型(MLMs)定义了词元上的局部条件分布,但通常不对应任何一致的序列联合分布。这引出一个根本问题:当此类条件被迭代用于生成时,会诱导出怎样的全局分布行为?本文将迭代掩码词元重采样建模为离散序列空间上的格劳伯动力学马尔可夫链。首先证明MLM条件内在不相容:提出矩形检验方法并验证现代MLMs中普遍存在此现象。理论分析表明,在跨词元影响有界的条件下,高温度下链具有收缩性,混合时间为O(n log n),其中n为序列长度;而在均匀局部裕度条件下,链表现出亚稳态,低温度下从语义盆地中逃逸极慢。实验上,我们展示了混合行为随温度和序列长度呈现相变,符合理论预测。通过语义轨迹进一步刻画稳态行为,识别出长期存在的陷阱和循环语义盆地,以政治内容为可量化的案例研究。

原文摘要 · Abstract (English)

Masked language models (MLMs) define local conditional distributions over tokens but do not, in general, correspond to any consistent joint distribution over sequences. This raises a fundamental question: what global distributional behavior is induced when such conditionals are used iteratively for generation? We address this question by modeling iterative masked-token resampling as a Glauber dynamics Markov chain on the discrete space of token sequences. We first show that MLM conditionals are intrinsically incompatible: we introduce a rectangle test that certifies this incompatibility and empirically verify its prevalence across modern MLMs. We then provide a theoretical analysis of the induced Markov chain. Under bounded cross-token influence, we establish a high-temperature contraction result implying $O(n\log n)$ mixing time where $n$ is the sequence length. In contrast, we prove that under a uniform local margin condition, the chain exhibits metastability, with exponentially slow escape from semantic basins at low temperatures. Empirically, we demonstrate a phase transition in mixing behavior as a function of temperature and sequence length, consistent with the theoretical predictions. We further characterize the induced stationary behavior through semantic trajectories, identifying persistent structures such as long-lived traps and recurrent semantic basins, with political content serving as a measurable case study.

生成模型马尔可夫链语义结构混合时间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。