基于基向量的变换器模型，高效处理表格回归中的多任务挑战。
Basis Transformers for Multi-Task Tabular Regression
- 设计基向量变换器架构，自动捕捉表格的层级结构与数值表示不变性。
- 在OpenML-CTR23上实现0.338的中位R²提升，34个任务标准差最低。
- 参数量仅为最佳基线的1/5，随机初始化即超越大型预训练语言模型。
由于信息不全、噪声干扰和异构结构，处理表格数据极具挑战。现有方法难以同时应对文本信息、列数可变及缺乏元数据（仅列名）等问题。本文提出一种新架构——基向量变换器（basis transformers），专门解决上述问题，并尊重表格数据的内在不变性，包括层级结构和数值表示特性。我们在多任务表格回归基准OpenML-CTR23上评估该设计，中位R²得分提升0.338，34项任务的标准差最低。此外，模型参数量仅为表现最佳基线的五分之一，即使从随机权重初始化，仍优于预训练大语言模型。
原文摘要 · Abstract (English)
Dealing with tabular data is challenging due to partial information, noise, and heterogeneous structure. Existing techniques often struggle to simultaneously address key aspects of tabular data such as textual information, a variable number of columns, and unseen data without metadata besides column names. We propose a novel architecture, \textit{basis transformers}, specifically designed to tackle these challenges while respecting inherent invariances in tabular data, including hierarchical structure and the representation of numeric values. We evaluate our design on a multi-task tabular regression benchmark, achieving an improvement of 0.338 in the median $R^2$ score and the lowest standard deviation across 34 tasks from the OpenML-CTR23 benchmark. Furthermore, our model has five times fewer parameters than the best-performing baseline and surpasses pretrained large language model baselines -- even when initialized from randomized weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。