提出自然梯度消息传递,提升变分推断的不确定性校准能力
Information Geometry of Message Passing

- 基于因子图在边缘上约束为指数族,用自然梯度投影构造消息
- 相比变分消息传递,保留更多精确消息信息,不确定性越强效果越显著
- 适用于部分观测链、批量数据过滤等场景,尤其改善预测置信度
我们证明变分推断的自然梯度驻点条件在Forney型因子图上具有边局部形式。从Bethe自由能出发,对选定边边际施加指数族约束。在驻点处,该边的自然参数等于两个投影消息之和,分别来自相邻因子。每个投影消息是接收边际处精确信念传播对数消息的自然梯度投影,或等价于其期望在均值坐标下的梯度。由此提出的方案称为自然梯度消息传递(NGMP)。该规则具有局部性:每条边可携带自身指数族,因子发送的消息依赖于接收边际。相较于变分消息传递,NGMP保留了接收族能表示的精确消息部分,而非对因子进行邻近信念平均。当进入非共轭因子的边不确定性消失时二者一致;当不确定性持续存在时(如部分观测隐变量链或逐批数据参数过滤),NGMP更准确。在泊松平滑、异方差回归和小时级ETTh预测任务上的实验验证了这一点,增益主要体现在不确定性校准上。
原文摘要 · Abstract (English)
We show that the natural-gradient stationary condition of variational inference has an edge-local form on a Forney-style factor graph. We start from the Bethe free energy and constrain a selected edge marginal to an exponential family. At a stationary point, the natural parameter of that edge equals the sum of two projected messages, one from each incident factor. Each projected message is the natural-gradient projection of the exact belief-propagation log-message at the current receiving marginal, or equivalently, the gradient of its expectation in the so-called mean coordinates. We call the resulting scheme natural-gradient message passing (NGMP). The rule is local; each edge may carry its own exponential family, and the message a factor sends depends on the marginal that receives it. Compared with variational message passing, NGMP keeps the part of the exact message that the receiving family can represent instead of averaging the factor under the neighboring beliefs. The two coincide when the uncertainty on the edges entering a non-conjugate factor vanishes, and NGMP is more accurate when that uncertainty persists, for example, along a partially observed latent chain or when parameters are filtered through successive data batches. Experiments on Poisson smoothing, heteroskedastic regression, and hourly ETTh forecasting confirm this and show that the gain appears mainly in uncertainty calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。