为表格基础模型设计了基于概率评分规则的评测基准,解决传统评估忽略分布信息的问题。
ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules
- 引入综合概率评分规则的评测框架,涵盖CRPS、能量评分等五种方法。
- 97个数据集测试显示,不同评分规则下模型排名差异显著,点估计优者未必概率表现好。
- 适合关注误差代价不均的高风险应用,如医疗或金融建模的开发者和研究者。
表格基础模型(如TabPFN和TabICL)已能生成完整的预测分布,但现有回归评测仍主要依赖点估计指标(如RMSE、$R^2$),忽略了这些模型本应提供的分布信息——在高风险领域,各类错误的代价并不相同。我们提出ScoringBench,一个开放可扩展的基准,支持多种合理评分规则(包括CRPS、CRLS、区间评分、能量评分和加权CRPS),同时保留标准点估计指标。该基准覆盖97个来自不同领域的回归数据集,通过Git驱动的排行榜支持社区贡献,并提供两种互补的排序方式:序数Demsar/autorank与保持量级的z-score排名。对多种模型(包括上下文学习、微调的基础模型、梯度提升树、MLP)的评估显示,模型排名随评分规则显著变化:在点估计上表现优异的模型在概率评分中可能落后,且同一模型在不同评分规则下的排名可能相差明显。结果表明,评估指标的选择并非技术细节,而是建模决策;在尾部误差代价更高的场景下,需根据具体领域需求选择合适指标,直接影响模型部署效果。
原文摘要 · Abstract (English)
Tabular foundation models such as TabPFN and TabICL already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics (RMSE, $R^2$). This discards precisely the distributional information these models are designed to provide - a critical gap for high-stakes domains where not all kinds of errors are equally costly. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules - including CRPS, CRLS, interval score, energy score, and weighted CRPS - alongside standard point metrics. ScoringBench covers 97 regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demsar/autorank approach and a magnitude-preserving z-score ranking approach. Evaluating several models - spanning in-context learners, fine-tuned foundation models, gradient-boosted trees, and MLPs - we find that model rankings shift substantially depending on the scoring rule: models that excel on point-estimate metrics can rank poorly on probabilistic ones, and the top-performing model under one proper scoring rule may rank noticeably lower under another. These results demonstrate that the choice of evaluation metric is not a technicality but a modelling decision - and, for applications where e.g. tail errors are disproportionately costly, a domain-specific requirement with direct consequences for model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。