arXiv:2604.26762cs.LGcs.AI2026-04

将Transformer重构为可编程概率图模型,提升时序建模的可解释性与可控性。

Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework

论文配图:Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework
图 1 · 摘自论文原文
  • 将Transformer转化为可编程因子图,通过结构修改注入时序先验知识。
  • 实验证明在数据稀疏和噪声环境下,结构化先验显著提升建模性能。
  • 支持条件生成的结构化控制,适用于需要精准调控的时序预测场景。

概率Transformer(PT)证明了Transformer的自注意力与前馈块在数学上等价于条件随机场(CRF)上的平均场变分推断(MFVI)。在此等价关系下,Transformer不再只是黑箱神经网络,而成为可编程因子图:图结构、因子势能及消息传递调度均为显式可调的构造单元。原生用于自然语言处理的PT,在本报告中被拓展至时序建模领域。我们首先提出时空概率Transformer(ST-PT),以弥补原版PT缺乏通道轴和弱逐步语义的问题,并将其作为统一骨干架构。随后,基于PT/ST-PT作为因子图模型的三个独特性质,提出三个研究问题:RQ1. 图结构与势能是可直接编程的,能否通过结构修改注入符号化时序先验,尤其在数据稀缺与噪声环境下?RQ2. CRF的因子矩阵即为算子势能,能否在样本级外部条件驱动下编程这些矩阵,使条件生成变为结构化而非特征层面的调制?RQ3. 每次MFVI迭代对应因子图上的贝叶斯后验更新,能否将潜在空间自回归(AR)预测的隐层转移从模糊的MLP转变为有原则的后验更新?能否利用CRF教师模型将潜变量蒸馏至AR学生模型以缓解累积误差?每问均提供一项实证研究。三者共同确立了ST-PT作为可编程时序建模范式。

原文摘要 · Abstract (English)

The Probabilistic Transformer (PT) establishes that the Transformer's self-attention plus its feed-forward block is mathematically equivalent to Mean-Field Variational Inference (MFVI) on a Conditional Random Field (CRF). Under this equivalence the Transformer ceases to be a black-box neural network and becomes a programmable factor graph: graph topology, factor potentials, and the message-passing schedule are all explicit and inspectable primitives that can be engineered. PT was originally developed for natural language and in this report we investigate its potential for time series. We first lift PT into the Spatial-Temporal Probabilistic Transformer (ST-PT) to repair PT's missing channel axis and weak per-step semantics, and adopt ST-PT as a shared cornerstone backbone. We then identify three distinct properties that PT/ST-PT offers as a factor-graph model and derive three Research Questions, one per property, that probe how each property can be exploited in time series: RQ1. The graph topology and potentials are direct programmable primitives. Can this be used to inject symbolic time-series priors into ST-PT through structural graph modifications, especially under data scarcity and noise? RQ2. The CRF's factor matrices are the operator's potentials. Can an external condition program these factor matrices on a per-sample basis, so that conditional generation becomes structural rather than feature-level modulation of a fixed one? RQ3. Each MFVI iteration is a Bayesian posterior update on the factor graph. Can this turn the latent transition of latent-space AutoRegressive (AR) forecasting from an opaque MLP into a principled posterior update, and can a CRF teacher distill its latents into the AR student to counter cumulative error? We give one empirical study per question. Together, these three studies position ST-PT as a programmable framework for time-series modeling.

时序建模概率图Transformer可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。