arXiv:2502.02463stat.MLcs.LG2025-02中稿 · ICML被引 14

用Transformer实现快速贝叶斯推断,支持实时调整先验。

Distribution Transformers: Fast Approximate Bayesian Inference With On-The-Fly Prior Adaptation

  • 用GMM表示先验,通过自注意力和交叉注意力生成后验分布。
  • 推理速度从分钟级降至毫秒级,性能媲美甚至超越现有方法。
  • 适合需要动态调整先验的实时场景,如传感器融合与量子参数估计。

虽然贝叶斯推断提供了不确定性下的合理推理框架,但精确后验计算的不可行性限制了其广泛应用,通常需依赖近似推断。然而,现有方法往往计算成本高,或在先验变化时需昂贵重训练,尤其在序列推断(如实时传感器融合)中受限严重。为此,我们提出分布Transformer——一种可学习任意分布到分布映射的新架构。该方法可训练为将给定数据集条件下的先验映射为后验,从而实现近似贝叶斯推断。其新颖架构将先验表示为通用逼近的高斯混合模型(GMM),并通过自注意力和交叉注意力将其转化为后验的GMM表示。实验表明,分布Transformer既能灵活改变先验,又将计算时间从分钟级降至毫秒级,在序列推断、量子系统参数推断及带超先验的高斯过程预测后验任务中,对数似然性能与现有方法相当或更优。

原文摘要 · Abstract (English)

While Bayesian inference provides a principled framework for reasoning under uncertainty, its widespread adoption is limited by the intractability of exact posterior computation, necessitating the use of approximate inference. However, existing methods are often computationally expensive, or demand costly retraining when priors change, limiting their utility, particularly in sequential inference problems such as real-time sensor fusion. To address these challenges, we introduce the Distribution Transformer -- a novel architecture that can learn arbitrary distribution-to-distribution mappings. Our method can be trained to map a prior to the corresponding posterior, conditioned on some dataset -- thus performing approximate Bayesian inference. Our novel architecture represents a prior distribution as a (universally-approximating) Gaussian Mixture Model (GMM), and transforms it into a GMM representation of the posterior. The components of the GMM attend to each other via self-attention, and to the datapoints via cross-attention. We demonstrate that Distribution Transformers both maintain flexibility to vary the prior, and significantly reduces computation times-from minutes to milliseconds-while achieving log-likelihood performance on par with or superior to existing approximate inference methods across tasks such as sequential inference, quantum system parameter inference, and Gaussian Process predictive posterior inference with hyperpriors.

贝叶斯推断Transformer快速推理先验适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。