用Transformer实现一次训练、任意分布通用的密度与梯度估计。
DiScoFormer: Plug-In Density and Score Estimation with Transformers
- 基于自注意力机制构建可泛化的密度与得分估计器
- 比KDE收敛更快且精度更高，支持任意样本量
- 适合需要快速插件式得分估计的研究场景
从样本中估计概率密度及其得分，是生成建模、贝叶斯推断和动力学理论中的核心问题。现有方法分为两类：经典核密度估计（KDE）泛化能力强但受维度诅咒影响；现代神经得分模型精度高，但需为每种目标分布重新训练。我们提出DiScoFormer（密度与得分Transformer），一种“一次训练、处处推理”的等变Transformer，能将独立同分布样本映射为密度值与得分向量，实现跨分布和跨样本规模的泛化。理论上，我们证明自注意力可恢复归一化KDE，是核方法的功能推广；实证上，各注意力头学习到多尺度核式行为。该模型在密度估计中收敛更快、精度优于KDE，同时提供高保真插件得分预言机，适用于得分去偏KDE、费希尔信息计算及福克-普朗克型偏微分方程求解。
原文摘要 · Abstract (English)
Estimating probability density and its score from samples remains a core problem in generative modeling, Bayesian inference, and kinetic theory. Existing methods are bifurcated: classical kernel density estimators (KDE) generalize across distributions but suffer from the curse of dimensionality, while modern neural score models achieve high precision but require retraining for every target distribution. We introduce DiScoFormer (Density and Score Transformer), a ``train-once, infer-anywhere" equivariant Transformer that maps i.i.d. samples to both density values and score vectors, generalizing across distributions and sample sizes. Analytically, we prove that self-attention can recover normalized KDE, establishing it as a functional generalization of kernel methods; empirically, individual attention heads learn multi-scale, kernel-like behaviors. The model converges faster and achieves higher precision than KDE for density estimation, and provides a high-fidelity plug-in score oracle for score-debiased KDE, Fisher information computation, and Fokker-Planck-type PDEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。