构建可扩展时序理解测评集，评估大模型对时序模式的掌握程度。
TimeSeriesExam: A time series understanding exam
- 用700多道程序生成题目，覆盖五类时序理解能力。
- 闭源模型如GPT-4在简单概念上显著优于开源模型。
- 所有模型在因果分析等复杂任务上表现不佳，适合研究者参考。
大型语言模型（LLMs）近期展现出建模时间序列数据的惊人能力，这可能源于其对基本时序概念的理解。然而，我们对这些模型究竟理解哪些时序知识仍知之甚少。为填补这一空白，我们提出TimeSeriesExam，一个可配置、可扩展的多项选择题测评集，用于评估大模型在五个核心时序理解维度上的表现：模式识别、噪声理解、相似性分析、异常检测和因果分析。该测评集包含超过700道题目，通过104个精心设计的模板程序化生成，并经过迭代优化以平衡难度与区分度。我们在7个先进大模型上测试了该测评集，首次全面评估了它们的时序理解能力。结果表明，封闭源代码模型如GPT-4和Gemini在理解简单时序概念方面显著优于开源模型，而所有模型在因果分析等复杂任务上均表现不佳。我们认为，程序化生成题目是评估并提升大模型时序理解与推理能力的关键。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have recently demonstrated a remarkable ability to model time series data. These capabilities can be partly explained if LLMs understand basic time series concepts. However, our knowledge of what these models understand about time series data remains relatively limited. To address this gap, we introduce TimeSeriesExam, a configurable and scalable multiple-choice question exam designed to assess LLMs across five core time series understanding categories: pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality analysis. TimeSeriesExam comprises of over 700 questions, procedurally generated using 104 carefully curated templates and iteratively refined to balance difficulty and their ability to discriminate good from bad models. We test 7 state-of-the-art LLMs on the TimeSeriesExam and provide the first comprehensive evaluation of their time series understanding abilities. Our results suggest that closed-source models such as GPT-4 and Gemini understand simple time series concepts significantly better than their open-source counterparts, while all models struggle with complex concepts such as causality analysis. We believe that the ability to programatically generate questions is fundamental to assessing and improving LLM's ability to understand and reason about time series data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。