让大模型同时理解医学文本和时间序列数据，实现更精准的临床推理。
OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data
- 将时间序列作为原生模态融入大模型，通过交叉注意力显式建模序列数据
- 在睡眠分期任务中达到69.9 F1，显著超越仅处理文本的模型
- 代码、数据和模型开源，适合医疗AI研究者和临床决策系统开发者
大型语言模型在多模态数据理解中展现出强大能力，尤其在医学领域可整合海量临床信息生成可操作洞察。然而，其对时间序列数据的处理能力仍有限。为此，我们提出OpenTSLM，一种将时间序列作为原生模态集成到预训练大模型中的时间序列语言模型（TSLM），支持对任意长度多变量时间序列进行推理。我们探索两种架构：OpenTSLM-SoftPrompt通过软提示将时间序列隐式编码为可学习的序列标记；OpenTSLM-Flamingo则通过交叉注意力显式融合时间序列与文本。我们在三组文本-时间序列链式思维（CoT）推理任务上评估，引入HAR-CoT、Sleep-CoT和ECG-QA-CoT数据集。结果表明，OpenTSLM在所有任务中均优于基线，睡眠分期达69.9 F1，HAR任务达65.4，远超微调的纯文本模型（9.05和52.2）。即使是10亿参数的OpenTSLM也超越GPT-4o（15.47和2.95）。OpenTSLM-Flamingo性能相当且在长序列上更优，内存占用稳定；而SoftPrompt随序列长度指数增长，训练需约110 GB VRAM，对比下仅需40 GB。临床专家评审确认OpenTSLMs在心电图问答任务中具备强推理能力。为促进研究，我们开源全部代码、数据集与模型。
原文摘要 · Abstract (English)
LLMs have emerged as powerful tools for interpreting multimodal data. In medicine, they hold particular promise for synthesizing large volumes of clinical information into actionable insights and digital health applications. Yet, a major limitation remains their inability to handle time series. To overcome this gap, we present OpenTSLM, a family of Time Series Language Models (TSLMs) created by integrating time series as a native modality to pretrained LLMs, enabling reasoning over multiple time series of any length. We investigate two architectures for OpenTSLM. The first, OpenTSLM-SoftPrompt, models time series implicitly by concatenating learnable time series tokens with text tokens via soft prompting. Although parameter-efficient, we hypothesize that explicit time series modeling scales better and outperforms implicit approaches. We thus introduce OpenTSLM-Flamingo, which integrates time series with text via cross-attention. We benchmark both variants against baselines that treat time series as text tokens or plots, across a suite of text-time-series Chain-of-Thought (CoT) reasoning tasks. We introduce three datasets: HAR-CoT, Sleep-CoT, and ECG-QA-CoT. Across all, OpenTSLM models outperform baselines, reaching 69.9 F1 in sleep staging and 65.4 in HAR, compared to 9.05 and 52.2 for finetuned text-only models. Notably, even 1B-parameter OpenTSLM models surpass GPT-4o (15.47 and 2.95). OpenTSLM-Flamingo matches OpenTSLM-SoftPrompt in performance and outperforms on longer sequences, while maintaining stable memory requirements. By contrast, SoftPrompt grows exponentially in memory with sequence length, requiring around 110 GB compared to 40 GB VRAM when training on ECG-QA with LLaMA-3B. Expert reviews by clinicians find strong reasoning capabilities exhibited by OpenTSLMs on ECG-QA. To facilitate further research, we provide all code, datasets, and models open-source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。