用扩散模型思路重构Transformer，让预测不确定性可传导。
Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration
- 将每个特征变换层视为概率映射，构建类扩散路径。
- 在多领域视觉语言任务中，校准效果优于现有方法。
- 适合需要可信预测的高风险场景使用。
预训练Transformer在风险敏感应用中的不确定性校准至关重要，但现有模型缺乏对特征变换过程中不确定性传播的理论机制。本文提出一种受扩散模型启发的Transformer重配置方法，将每个特征变换块建模为概率映射，组合后形成模拟扩散过程的概率路径，将输入分布逐步传递至预训练特征分布。该路径可重新编译为统一转移模型的扩散过程，实现表征不确定性在模型架构内的合理传播，同时保持原始预测性能。在多个视觉与语言基准测试中，本方法在不确定性校准和预测准确率方面均优于现有不确定性感知Transformer。
原文摘要 · Abstract (English)
Uncertainty calibration in pre-trained transformers is critical for their reliable deployment in risk-sensitive applications. Yet, most existing pre-trained transformers do not have a principled mechanism for uncertainty propagation through their feature transformation stack. In this work, we propose a diffusion-inspired reconfiguration of transformers in which each feature transformation block is modeled as a probabilistic mapping. Composing these probabilistic mappings reveals a probability path that mimics the structure of a diffusion process, transporting data mass from the input distribution to the pre-trained feature distribution. This probability path can then be recompiled on a diffusion process with a unified transition model to enable principled propagation of representation uncertainty throughout the pre-trained model's architecture while maintaining its original predictive performance. Empirical results across a variety of vision and language benchmarks demonstrate that our method achieves superior calibration and predictive accuracy compared to existing uncertainty-aware transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。