arXiv:2605.18832cs.LGcs.AI2026-05

用不确定性建模提升Transformer,让模型更懂自己知道什么、不知道什么。

Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise

论文配图:Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise
图 1 · 摘自论文原文
  • 将注意力机制改为基于精度加权的克里金插值,实现动态置信度计算。
  • 在冷启动用户和稀有物品推荐上,性能提升显著,最高达15.3%。
  • 适用于需要鲁棒性的场景,如带噪声的问答和检索增强生成。

Transformer是现代AI的核心组件,但缺乏对真实应用中普遍存在的不确定性的合理处理:序列推荐中的冷启动项、语言模型中的信号质量异质性,以及由无约束softmax引发的注意力坍缩。现有方法对每个词元均赋予相同置信度。本文提出贝叶斯滤波Transformer(BFT),其将注意力机制重构为精度加权的克里金插值,残差连接变为自适应增益的卡尔曼更新,前馈网络则通过雅可比矩阵与过程噪声规则传播精度。观测精度由无需参数的限制最大似然(REML)估计器获得,结合共轭贝叶斯先验。BFT可无缝替换任意Transformer层,开销极低。在序列推荐任务中,对三种主流架构应用BFT,在六个基准上均取得显著提升,尤其在冷启动用户和稀有物品上表现最优。在大语言模型的监督微调中,面对噪声数据时,BFT在两种场景下增强鲁棒性:一是标签噪声(问答任务中的错误标记),二是上下文噪声(真实检索增强问答中的干扰项)。仅通过恢复精度建模这一原则性改进,便在经典序列建模与现代大模型领域均打开巨大提升空间。

原文摘要 · Abstract (English)

The Transformer is the foundational building block of modern AI, yet offers no principled handling of \emph{uncertainty}, which is prevalent in real applications: cold-start tokens with sparse histories in sequential recommendation, heterogeneous signal quality in language models, and attention sinks induced by unconstrained softmax. Every token is treated with uniform confidence. We show this uniformity is a degenerate case of our \emph{Bayesian Filtering Transformer} (BFT): attention becomes precision-weighted kriging, the residual connection becomes a Kalman update with adaptive gain, and the FFN becomes a dynamics model propagating precision via a Jacobian--plus--process-noise rule. Observation precision comes from a parameter-free Restricted Maximum Likelihood (REML) estimator with a conjugate Bayesian prior. BFT replaces any Transformer layer with negligible overhead. On sequential recommendation, BFT applied to three major architectures yields significant gains on six benchmarks, with the largest improvements on cold-start users and rare items where uncertainty is highest. On supervised fine-tuning of large language models with noisy data, BFT improves robustness in two regimes: noisy supervision (token-label corruption in question answering) and noisy context (retrieval-augmented QA with real RAG distractors). A single principled modification -- restoring precision -- unlocks substantial headroom across both classical sequence-modeling and modern LLM regimes.

Transformer不确定性建模推荐系统大模型鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。