构建多维评估基准,揭示模型在不同数据特征下的表现差异
MultiTab: A Comprehensive Benchmark Suite for Multi-Dimensional Evaluation in Tabular Domains
- 按样本量、标签不平衡等12个维度分类196个数据集,实现数据感知评估
- 13种模型在不同数据条件下表现差异显著,如大样本下相似性模型更优
- 为模型选型与设计提供实证依据,适合数据科学家和算法研究者
尽管表格数据在现实应用中广泛使用,但多数基准仅依赖平均性能指标,无法揭示模型在不同数据场景下的行为变化。为此,我们提出MultiTab,一个用于表格学习算法多维、数据感知分析的基准套件与评估框架。不同于仅做总体对比,MultiTab将196个公开数据集按样本量、标签不平衡、特征交互等关键数据特性进行分类,并评估13种涵盖多种归纳偏置的代表性模型。分析显示,模型性能对数据分布高度敏感:例如,利用样本级相似性的模型在样本量大或特征间相关性强的数据集上表现更佳;而编码特征间依赖关系的模型在弱相关特征下最优。这些发现表明,归纳偏置并不总按预期工作,且需进行场景感知评估以理解并改进模型行为。MultiTab支持更严谨的模型设计,并为根据数据特征选择合适模型提供实用指导。所有数据集、代码与优化日志均公开可获取于 https://huggingface.co/datasets/LGAI-DILab/Multitab。
原文摘要 · Abstract (English)
Despite the widespread use of tabular data in real-world applications, most benchmarks rely on average-case metrics, which fail to reveal how model behavior varies across diverse data regimes. To address this, we propose MultiTab, a benchmark suite and evaluation framework for multi-dimensional, data-aware analysis of tabular learning algorithms. Rather than comparing models only in aggregate, MultiTab categorizes 196 publicly available datasets along key data characteristics, including sample size, label imbalance, and feature interaction, and evaluates 13 representative models spanning a range of inductive biases. Our analysis shows that model performance is highly sensitive to such regimes: for example, models using sample-level similarity excel on datasets with large sample sizes or high inter-feature correlation, while models encoding inter-feature dependencies perform best with weakly correlated features. These findings reveal that inductive biases do not always behave as intended, and that regime-aware evaluation is essential for understanding and improving model behavior. MultiTab enables more principled model design and offers practical guidance for selecting models tailored to specific data characteristics. All datasets, code, and optimization logs are publicly available at https://huggingface.co/datasets/LGAI-DILab/Multitab.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。