一个能统一处理表格数据的通用模型,无需微调即可应对多种任务。
LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence
- 将表格数据建模为变量与缺失值的联合分布,用单一模型实现条件预测。
- 在11个基准上超越强基线,分类、回归、补全等任务均表现优异。
- 适合需要快速适配表格任务的场景,尤其适合资源受限环境使用。
我们主张通向通用智能需依赖语言、物理世界和结构化数据三类基础模型。本文提出LimiX-16M与LimiX-2M两款大规模结构化数据模型(LDMs),将结构化数据视为变量与缺失性的联合分布,通过查询式条件预测支持多种表格任务,仅用单个模型完成。模型采用掩码联合分布建模预训练,结合事件式上下文条件目标,实现推理时零训练快速适应。我们在11个大型结构化数据基准上评估,涵盖样本量、特征维度、类别数、类别/数值特征比、缺失率、样本/特征比等多种场景。LimiX-16M持续优于强基线,各项任务如分类、回归、缺失值填补、数据生成中常有显著优势,且无需任务特化架构或定制训练。值得注意的是,LimiX-2M在严格计算与内存预算下仍表现强劲。我们首次开展LDM的缩放定律研究,揭示数据与模型规模如何协同影响下游性能,并为表格基础建模提供量化指导。所有LimiX模型均开源,许可协议为Apache 2.0。
原文摘要 · Abstract (English)
We argue that progress toward general intelligence requires complementary foundation models grounded in language, the physical world, and structured data. This report presents LimiX-16M and LimiX-2M, two instantiations of our large structured-data models (LDMs). Both models treat structured data as a joint distribution over variables and missingness, thus capable of addressing a wide range of tabular tasks through query-based conditional prediction via a single model. They are pretrained using masked joint-distribution modeling with an episodic, context-conditional objective, supporting rapid, training-free adaptation at inference. We evaluate LimiX models across 11 large structured-data benchmarks with broad regimes of sample size, feature dimensionality, class number, categorical-to-numerical feature ratio, missingness, and sample-to-feature ratios. LimiX-16M consistently surpasses strong baselines, as shown in Figure 1 and Figure 2. The superiority holds across a wide range of tasks, such as classification, regression, missing value imputation, and data generation, often by substantial margins, while avoiding task-specific architectures or bespoke training per task. Notably, LimiX-2M delivers strong results under tight compute and memory budgets. We also present the first scaling law study for LDMs, revealing how data and model scaling jointly influence downstream performance and offering quantitative guidance for tabular foundation modeling. All LimiX models are publicly accessible under Apache 2.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。