用单一模型同时处理混合型时间序列数据，突破传统方法局限。
multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data
- 基于自回归分解与联合嵌入，统一建模类别与数值变量。
- 在心电图和电子健康记录上实现高精度建模与泛化。
- 适合处理采样不规则的多变量时序数据，如医疗监测系统。
现实世界中的过程常生成包含类别与数值变量的混合数据，且采样时间不规则但信息丰富。现有离散令牌方法对数值变量表达能力有限，而神经微分方程对类别数据不适用，且需额外增强以处理特定轨迹。本文提出 multivariateGPT，一种用于混合类别（包括分词文本）与数值数据序列的单一体系架构。通过自回归序列分解、嵌入方案与损失函数，将下一个令牌预测任务扩展为对下一类别的联合分布与数值的似然估计。实验表明，该方法能高效学习简单物理系统的模式，并成功建模复杂时间序列，如心电图与多变量电子健康记录数据。本工作拓展了基于Transformer模型在更多类型数据上的应用能力。
原文摘要 · Abstract (English)
Real-world processes often generate data that are a mix of categorical and numeric values that are recorded at irregular and informative intervals. Discrete token-based approaches are limited in numeric representation capacity while methods like neural ordinary differential equations are not well suited for categorical data or informative sampling and require augmentation to handle certain classes of trajectories. Here, we present multivariateGPT, a single architecture for modeling sequences of mixed categorical (including tokenized text) and numeric data. This is accomplished with an autoregressive sequence decomposition, embedding scheme, and loss function that extend the next token prediction task to likelihood estimation of the joint distribution of next token class and value. We demonstrate how this approach can efficiently learn to generalize patterns in simple physical systems and model complex time series including electrocardiograms and multivariate electronic health record data. This work extends the utility of transformer based models to additional classes of data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。