arXiv:2506.09163cs.LGstat.ML2025-06被引 2

新模型让神经过程在百万级数据上快速推理,兼顾精度与效率。

Scalable Spatiotemporal Inference with Biased Scan Attention Transformer Neural Processes

  • 用带偏置的扫描注意力和核回归块提升建模效率
  • 单卡1分钟内完成超百万测试点的时空推理
  • 适合需要高维、多尺度时空建模的研究者

神经过程(NPs)是一类可直接建模随机过程后验预测分布的新兴模型。早期设计主要作为高斯过程(GPs)的可扩展替代方案,如今已应用于地质、流行病学、气候和机器人等复杂数据密集型领域。这些应用对模型可扩展性提出更高要求,许多架构为提升速度牺牲精度。本文证明,在建模全或部分平移不变过程时,此权衡并非必然。我们提出新型架构——有偏扫描注意力变换器神经过程(BSA-TNP),引入核回归块(KRBlocks)、群不变注意力偏置和内存高效的有偏扫描注意力(BSA)。该模型能:(1)在训练时间远少于现有最优模型的同时保持或超越其精度;(2)具备平移不变性,支持多分辨率同步学习;(3)透明建模空间与时间双重演化的过程;(4)支持高维固定效应;(5)实现良好扩展性,在单张24GB GPU上,可在1分钟内完成超过100万测试点与10万上下文点的推理。代码已开源至`dl4bi`包。

原文摘要 · Abstract (English)

Neural Processes (NPs) are a rapidly evolving class of models designed to directly model the posterior predictive distribution of stochastic processes. While early architectures were developed primarily as a scalable alternative to Gaussian Processes (GPs), modern NPs tackle far more complex and data-hungry applications spanning geology, epidemiology, climate, and robotics. These applications have placed increasing pressure on the scalability of these models, with many architectures compromising accuracy for scalability. In this paper, we demonstrate that this trade-off is often unnecessary, particularly when modeling fully or partially translation-invariant processes. We propose a versatile new architecture, the Biased Scan Attention Transformer Neural Process (BSA-TNP), which introduces Kernel Regression Blocks (KRBlocks), group-invariant attention biases, and memory-efficient Biased Scan Attention (BSA). BSA-TNP is able to: (1) match or exceed the accuracy of the best models while often training in a fraction of the time, (2) exhibit translation invariance, enabling learning at multiple resolutions simultaneously, (3) transparently model processes that evolve in both space and time, (4) support high-dimensional fixed effects, and (5) scale gracefully, running inference on over 1M test points and 100K context points in under a minute on a single 24GB GPU. Code is provided as part of the `dl4bi` package.

神经过程时空建模高效推理注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。