简单离散化+高斯平滑，让Transformer在表格预测中超越梯度提升树。
Discrete Tokenization Unlocks Transformers for Calibrated Tabular Forecasting
- 用简单离散化处理表格特征，结合自适应高斯平滑标签。
- 在60万实体上，中位MAE比调优的XGBoost低10.8%（35.94秒对40.31秒）。
- 能生成校准概率分布，适合需要可靠置信度的工业预测场景。
梯度提升模型在表格基准上仍占主导。本文提出一种刻意简化的离散化分词方法，以凸显即使基础分词也能释放注意力机制在表格特征上的潜力；结合高斯平滑后，性能已超越调优的梯度提升模型。该方案通过离散化环境上下文，并利用自适应高斯平滑标签，生成校准的概率密度函数。在包含60万实体（500万训练样本）的数据集上，中位平均绝对误差（MAE）优于调优后的XGBoost 10.8%（35.94秒对比40.31秒），并以最小化柯尔莫哥洛夫-斯米尔诺夫统计量（KS=0.0045）选择的自适应方差检查点实现最优校准。消融实验表明：丢失序列顺序导致性能下降约2.0%，移除时间差标记损失约1.8%，分层校准分析揭示了校准偏差的残余位置。
原文摘要 · Abstract (English)
Gradient boosting still dominates Transformers on tabular benchmarks. Our tokenizer uses a deliberately simplistic discretized vocabulary so we can highlight how even basic tokenization unlocks the power of attention on tabular features, yet it already outperforms tuned gradient boosting when combined with Gaussian smoothing. Our solution discretizes environmental context while smoothing labels with adaptive Gaussians, yielding calibrated PDFs. On 600K entities (5M training examples) we outperform tuned XGBoost by 10.8% (35.94s vs 40.31s median MAE) and achieve KS=0.0045 with the adaptive-sigma checkpoint selected to minimize KS rather than median MAE. Ablations confirm architecture matters: losing sequential ordering costs about 2.0%, dropping the time-delta tokens costs about 1.8%, and a stratified calibration analysis reveals where miscalibration persists.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。