让注意力层直接给出预测可信度,无需额外计算。
Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
- 用泊松随机测度构建新型注意力,一次前向传播即得不确定性
- 输出包含可信度指标,实测在稀疏数据上表现更优
- 适合需要快速可信评估的医疗等实时时序场景
针对不规则采样时序数据的深度模型虽可回答任意连续时间点的查询,却无法提供答案可信度。本文提出Lévy Attention,一种基于非齐次泊松随机测度的交叉注意力算子:查询与键的兼容性构成连续(时间×通道)空间上的强度函数,测度在该强度下散射原子,输出为在这些原子处插值值场的平均。其期望形式等价于平滑余弦核注意力,可替换softmax并使用精确梯度训练。与softmax丢弃的信息不同,泊松构造以闭式保留证据Λ_q(总兼容质量)和分歧trΣ_V(q)(值域扩散),通过精确方差恒等式组合得可信度估计ˆσ(q)=√(trΣ_V(q) φ(Λ_q)),由确定性前向传播直接输出,无需训练额外头。实验显示,分歧携带有效信号,证据因子在密集数据上信息量低,在稀疏数据上显著提升。在t-PatchGNN上,该操作替换仅损失最多5.6%准确率,稀疏数据上无损;自由分歧信号优于20次MC dropout,且ˆσ生成校准高斯,零样本CRPS优于50次采样器;分段共形包装在各置信水平均达名义覆盖,单次前向传播1.4秒内对3,383名未见患者按可信度排序。
原文摘要 · Abstract (English)
Deep models for irregularly-sampled time series answer queries at arbitrary continuous timestamps, yet report nothing about how far each answer should be trusted. We show the attention layer itself can close that gap: with the right stochastic formulation, the pass that makes each prediction also reports, in closed form and at no extra cost, how far it should be trusted. We introduce Lévy Attention, a cross-attention operator whose output is a stochastic integral against an inhomogeneous Poisson random measure: query-key compatibilities assemble an intensity over a continuous (time x channel) index space, the measure scatters atoms under it, and the output averages an interpolated value field at those atoms. In expectation it reduces to a mollified cosine-kernel attention, so it replaces a softmax layer and trains with exact gradients. What softmax discards, the Poisson construction preserves in closed form: the evidence $Λ_q$ (total compatibility mass) and the disagreement $\mathrm{tr}\,Σ_V(q)$ (value spread). An exact variance identity makes their combination $\hatσ(q)=\sqrt{\mathrm{tr}\,Σ_V(q)\,φ(Λ_q)}$ the root-mean-square deviation of the sampled operator, emitted by the deterministic pass with no trained head. Empirically, disagreement carries the signal, while the evidence factor swings from uninformative on dense data to strongly informative on sparse. On t-PatchGNN the operator swap costs at most 5.6% accuracy against a matched control and nothing on the sparsest dataset. The free disagreement signal improves on 20-pass MC dropout across matched five-seed suites, and $\hatσ$ scales a calibrated Gaussian whose zero-sample CRPS beats a fifty-draw sampler; a split-conformal wrapper reaches nominal coverage at every level, and one pass ranks 3,383 unseen patients by trust in 1.4 seconds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。