用可信度令牌提升表格数据的预测性能
The Credibility Transformer
- 引入可信度令牌，融合先验与观测信息加权平均
- 显著稳定训练过程，预测效果超越现有深度学习模型
- 适合处理高不确定性表格数据的建模任务
受大型语言模型中Transformer架构成功的启发，该结构正被越来越多地应用于表格数据。通过将表格数据嵌入低维欧几里得空间，其结构类似于时间序列数据。本文提出一种新型可信度机制，基于一个特殊令牌，可视为一个编码器，该编码器由先验信息和基于观测信息的可信度加权平均构成。实验表明，该机制显著提升了训练稳定性，所提出的可信度Transformer在预测性能上优于当前最先进的深度学习模型。
原文摘要 · Abstract (English)
Inspired by the large success of Transformers in Large Language Models, these architectures are increasingly applied to tabular data. This is achieved by embedding tabular data into low-dimensional Euclidean spaces resulting in similar structures as time-series data. We introduce a novel credibility mechanism to this Transformer architecture. This credibility mechanism is based on a special token that should be seen as an encoder that consists of a credibility weighted average of prior information and observation based information. We demonstrate that this novel credibility mechanism is very beneficial to stabilize training, and our Credibility Transformer leads to predictive models that are superior to state-of-the-art deep learning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。