构建多模态时间序列与文本数据集，探索联合预测方法
Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data
- 构建对齐时间的文本与数值数据集TTC，覆盖气候与医疗领域
- 提出混合多模态模型联合预测文本与时间序列，但效果未超基线
- 揭示多模态预测任务的挑战，适合研究跨模态建模的学者
当前预测方法多为单模态，因缺乏高质量多模态基准数据集而忽略常伴随时间序列的文本信息。本文构建了时间对齐的文本与时间序列数据集TimeText Corpus (TTC)，涵盖气候科学与医疗两个领域，包含数列与文本在时间戳上的对齐数据，是少有的多模态数据资源。我们提出联合预测文本与时间序列的混合多模态模型Hybrid-MMF，通过共享嵌入实现多模态融合。然而实验显示，该模型未优于现有基线，凸显多模态预测的内在难度。代码与数据已开源。
原文摘要 · Abstract (English)
Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。