构建多任务多模态时间序列推理基准,评测通用模型的时序理解能力。
TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models
- 设计4125个跨14领域的时序问题,覆盖感知、推理、预测与决策四维度。
- 30余款模型测试显示:预测能力不随规模提升,语义理解与数值预测常脱节。
- 现有多模态模型无法有效融合文本与图像时序信息,难获协同增益。
时间序列广泛存在于现实场景中,对能源管理、交通控制等应用至关重要。因此,时序推理能力是通用模型解决复杂问题的基础。然而,现有通用模型基准大多忽视这一维度。为此,我们提出TSRBench,一个全面的多模态时间序列推理基准,用于全方位评估时序推理能力。TSRBench包含来自14个领域的4125个问题,分为感知、推理、预测和决策四大维度,并涵盖15项任务,用于评估关键推理能力(如数值推理)。我们对30余款主流私有及开源大语言模型(LLMs)、视觉语言模型(VLMs)和时间序列大模型(TSLLMs)进行了系统评估。结果表明:(1)感知与推理能力符合缩放定律,但预测能力不遵循;(2)强推理能力不能保证准确的上下文感知预测,揭示语义理解与数值预测之间的解耦现象;(3)尽管文本与视觉形式的时间序列输入具有互补性,当前多模态模型未能有效融合二者以实现性能互益。TSRBench提供标准化评估平台,不仅揭示了现存挑战,也为推动通用模型发展提供了宝贵洞见。代码与数据集已开源:https://tsrbench.github.io/。
原文摘要 · Abstract (English)
Time series are ubiquitous in real-world scenarios and crucial for applications ranging from energy management to traffic control. Consequently, the ability to reason over time series is a fundamental skill for generalist models to solve complex problems. However, current benchmarks for generalist models largely overlook this dimension. To bridge this gap, we introduce TSRBench, a comprehensive multi-modal benchmark designed to stress-test the full spectrum of time series reasoning capabilities. TSRBench features: i) a diverse set of 4125 problems from 14 domains, and is categorized into 4 major dimensions: Perception, Reasoning, Prediction, and Decision-Making. ii) 15 tasks from the 4 dimensions evaluating essential reasoning capabilities (e.g., numerical reasoning). Through extensive experiments, we evaluate over 30 leading proprietary and open-source LLMs, VLMs, and TSLLMs within TSRBench. Our findings reveal that: i) scaling laws hold for perception and reasoning but break down for prediction; ii) strong reasoning does not guarantee accurate context-aware forecasting, indicating a decoupling between semantic understanding and numerical prediction; and iii) despite the complementary nature of textual and visual forms of time series as inputs, current multimodal models fail to effectively fuse them for reciprocal performance gains. TSRBench provides a standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance generalist models. Our code and dataset are available at https://tsrbench.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。